When you enter a URL, what exactly happens in the background?

As a software developer, you must have a complete hierarchical understanding of how network applications work, which also includes the technologies used by these applications: browsers, HTTP, HTML, network servers, request processing, etc.

This article will take a closer look at what happens in the background when you enter a URL.

The first step in navigation is to find out the IP address of the domain name you are visiting. The DNS lookup process is as follows:

DNS recursive lookup is shown in the following figure:

Because dynamic pages like the Facebook homepage will expire soon or even immediately in the browser cache after opening, they cannot read from it.

So, the browser will send the following request to the Facebook server:

 GET http://facebook.com/ HTTP/1.1
 Accept: application/x-ms-application, image/jpeg, application/xaml+xml, [...]
 User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; [...]
 Accept-Encoding: gzip, deflate
 Connection: Keep-Alive
 Host: facebook.com
 Cookies: datr=1265876274-[...]; locale=en_US; lsd=WW[...]; c_user=2101[...]

GET This request defines the URL to read: "http://facebook.com/". The browser defines itself ( User-Agent header), and what type of responses it wants to receive (Accept and Accept -Encoding headers). The Connection header asks the server not to close the TCP connection for subsequent requests.

The request also includes the cookies stored by the browser for the domain. As you probably already know, cookies are key-value pairs that track the state of a website across different page requests. In this way, cookies will store the login username, the password assigned by the server and some user settings. Cookies are stored in the client as a text file and sent to the server each time a request is made.

In addition to get requests, there is also a send request, which is often used when submitting forms. Send a request passing its parameters via the URL (eg: http://robozzle.com/puzzle.aspx?id=85). A send request sends its parameters after the request body headers.

Like in "http://facebook.com/" the slash is crucial. In this case, the browser can safely add the slash. For an address like "http://example.com/folderOrFile", the browser cannot automatically add a slash because it does not know whether folderOrFile is a folder or a file. At this point, the browser accesses the address directly without the slash, and the server responds with a redirect, resulting in an unnecessary handshake.

The picture shows the response sent back by the Facebook server to the browser:

 HTTP/1.1 301 Moved Permanently
 Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0,
 pre-check=0
 Expires: Sat, 01 Jan 2000 00:00:00 GMT
 Location: http://www.facebook.com/
 P3P: CP="DSP LAW"
 Pragma: no-cache
 Set-Cookie: made_write_conn=deleted; expires=Thu, 12-Feb-2009 05:09:50 GMT;
 path=/; domain=.facebook.com; httponly
 Content-Type: text/html; charset=utf-8
 X-Cnection: close
 Date: Fri, 12 Feb 2010 05:09:51 GMT
 Content-Length: 0

The server responds to the browser with a 301 Permanent Redirect response, so that the browser visits "http://www.facebook.com/" instead of "http://facebook.com/".

Now the browser knows that “http://www.facebook.com/” is the correct address to visit, so it sends another GET request:

 GET http://www.facebook.com/ HTTP/1.1
 Accept: application/x-ms-application, image/jpeg, application/xaml+xml, [...]
 Accept-Language: en-US
 User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; [...]
 Accept-Encoding: gzip, deflate
 Connection: Keep-Alive
 Cookies: lsd=XW[...]; c_user=21[...]; x-referer=[...]
 Host: www.facebook.com

The header information has the same meaning as in the previous request.

The server receives the GET request, processes it, and returns a response.

On the surface this may seem like a straightforward task, but there are actually a lot of interesting things happening in the process - for a website as simple as the author's blog, let alone a website with a large number of visitors like Facebook!

In the simplest example, demand processing can be stored in a file hierarchy that maps the structure of a website address. For example, the address http://example.com/folder1/page1.aspx will map to the file /httpdocs/folder1/page1.aspx. The web server software can be configured to manually process requests for addresses, so that the publishing address of page1.aspx can be http://example.com/folder1/page1.

The figure shows the response generated and returned by the server:

 HTTP/1.1 200 OK
 Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0,
 pre-check=0
 Expires: Sat, 01 Jan 2000 00:00:00 GMT
 P3P: CP="DSP LAW"
 Pragma: no-cache
 Content-Encoding: gzip
 Content-Type: text/html; charset=utf-8
 X-Cnection: close
 Transfer-Encoding: chunked
 Date: Fri, 12 Feb 2010 09:05:55 GMT

 2b3Tn@[...]

The entire response size is 35kB, most of which is transferred as a blob after being trimmed.

The Content-Encoding header tells the browser that the entire response body should be compressed using the gzip algorithm. After decompressing the blob, you can see the expected HTML:

 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
 lang="en" id="facebook" class="no_js">
 <head>
 <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
 <meta http-equiv="Content-language" content="en" />
 ...

Regarding compression, the header information indicates whether to cache the page, how to do it if cached, what cookies to set (this is not included in the previous response), privacy information, etc.

Before the browser has even finished reading the entire HTML document, it starts displaying this page:

When the browser displays the HTML, it notices the tags that need to fetch the content of other addresses. At this point, the browser sends a get request to retrieve the files.

Here are a few URLs we need to re-fetch when we visit facebook.com:

These addresses go through a process similar to HTML reading. So the browser will look up those domains in DNS, send the request, redirect, etc...

But unlike dynamic pages, static files allow browsers to cache them. Some files may not need to communicate with the server and can be read directly from the cache. The server's response includes information about how long static files should be kept, so the browser knows how long to cache them. In addition, each response may contain an ETag header (the entity value of the requested variable) that works like a version number. If the browser observes that the version ETag information of the file already exists, it will immediately stop transmitting the file.

In the spirit of Web 2.0, the client remains connected to the server after the page is displayed.

Take Facebook's chat function as an example. It will keep in touch with the server to update the status of your bright and gray friends in time. To update the status of these friends whose avatars are lit, the JavaScript code executed in the browser sends an asynchronous request to the server. This asynchronous request is sent to a specific address and is a programmatically constructed get or send request. Still in the Facebook example, the client sends a publish request to http://www.facebook.com/ajax/chat/buddy_list.php to get the online status information of which of your friends are online.

When talking about this pattern, we have to talk about "AJAX" - "Asynchronous JavaScript and XML", although there is no clear reason why the server responds in XML format. As another example, for asynchronous requests, Facebook will return some JavaScript code snippets.