Load Impact Load Testing and Performance Testing

Something strange about Google's logotype

Google's front page must be the most frequently loaded webpage on the Internet today. With Google's heavy focus on performance issues, one would suspect that this page has been obsessively optimized. If your page is loaded hundreds of millions of times per day, not only is there actually money to be saved on eliminating a single byte that needs transferring, but the small difference in user retention that a microscopic speedup in page load time can result in is also worth a lot of trouble. Marissa Mayer said some interesting things about how important page load time is for Google, in her 2007 presentation at the Seattle conference on scalability. It is a 1-hour presentation but well worth watching. If you don't feel like watching a long video presentation, you can check out Steve Souder's recent article Business impact of high performance.

Anyway, when testing our spanking new web page analyzer recently, we noticed that when we analyzed www.google.com and had our analyzer emulate different user agents, the Google logotype image on that page came in different versions, depending on what user agent we pretended to be. Nothing really surprising about this fact. Older web browsers don't support new image formats, for example, so sending the logotype as a GIF image makes sense when Google detects that an older (or unknown) browser is used, while a PNG image might be appropriate for a newer browser.

But, why on earth chop up the GIF logotype into four different pictures?

To someone who isn't interested in performance issues, this may seem like the most boring non-question since... well, ever actually. But to us others it is a bit perplexing. Or is it?  You be the judge. This is what you get when you load www.google.com with Firefox 3.5.

  The image is clickable if you want to view the result interactively and learn more. It shows the page load diagram for www.google.com in our page analyzer, emulating Firefox 3.5 as the user agent. The first thing that happens is that we get redirected to www.google.se, because our request originated from an IP address Google correctly identified as swedish. Then we load www.google.se/ (the HTML) and finally the Google logotype, which is called logo_plain.png and is 7.4KB in size. A single PNG image in this case. You can see this PNG being rendered in the bottom right corner of the screenshot.

Now, if we try and change our user agent emulation and instead tell Google who we really are (the "Load Impact Page Analyzer"), we get something different:

Look at this - Google sends us the logotype as a GIF, chopped up into four different images. The images are called hp0.gif, hp1.gif, hp2.gif and hp3.gif. On the screenshot below the hp0.gif image is shown on the lower right. You can see that it is the "Goo" part of the Google logo.

Why do they do this? Compatibility-wise it makes sense to send GIF images when you don't recognize the user agent, because we might be using some old browser that can't support e.g. PNG images. GIF is probably the most widely supported image file format in existence. But why not send a single GIF?  Why the four different parts?

Is it performance-related? Using more than one TCP connection to fetch objects can result in performance gains if there are many objects to fetch, as more objects can be requested concurrently. With HTTP pipelining this might be less of an issue, but older browsers that can only speak HTTP 1.0 do not support pipelining. On the other hand, the total number of objects on the Google front page, is small. The objects themselves are small in size too. Old browsers might not use more than two concurrent connections. Maybe they will only use one. In that case, dividing an image into several parts is most likely bad for performance.

Also, if it makes sense from a performance, or any other, perspective to divide the logo into four separate images, why isn't logo_plain.png delivered as four separate images too, when we use Firefox 3.5 to retrieve the page? Firefox 3.5 is definitely multi-connection capable - Actually, one could call it a multi-connection fetishist, just like most modern browsers. The same thing goes for if it has something to do with Google's needs to frequently change its logo (i.e. if it wants part of it to be cached on the client side, while still being able to alter other parts of it). Why not do that for all modern browsers too?

Then we start testing other Google sites - google.com (without the redirect to google.se), google.cn and google.jp for instance, all deliver a single GIF image, rather than a PNG image. No matter what browser emulation we use. Why is that?  Why do we get PNGs here in Sweden, while americans and asians get GIFs?

We asked a very technically knowledgeable Google employee about all this, but he had no clue as to the reason, so now we're down to guessing. There must be some kind of strategy behind the decision to serve certain browsers a 4-GIF version of the logo, others a single GIF and yet others a PNG, but what is it? If it had been any site but Google, we would have guessed it was an artifact of how the different logotypes (different countries and regions have different Google logotypes on their localized Google sites) are manufactured, but considering how much effort Google spends on performance optimization, it just doesn't seem likely that this is the cause here. It should not be all that difficult to convert all logotype images to the same format, if it made sense from a performance and compatibility perspective.

Can someone who knows please help shed some light on this mystery?

Implementing CSS support for the Page Analyzer

Our new web page analyzer is getting more popular by the day. Just after its release we even had a call from a well-known Internet organization who wanted to license it for their internal use, to gather general statistics about web site performance. They were impressed with the analyzer, but quickly noticed that it had a flaw - it would always load all resources that were referred to in a CSS file, no matter if the resources were needed to render the web page in question, or not.

Many sites today have a single CSS file that is used for the whole site. That CSS will contain references to e.g. various images that are used for backgrounds on different pages on the site. However, a web browser that loads the CSS file, will parse it and realize that while the background object a.jpg is needed on the current page, another background object b.jpg isn't needed. It is used on some other page on the site, but not on the page that is currently being loaded and rendered. The web browser will see this and refrain from loading b.jpg.

What this all means is that our page analyzer would often load a lot more objects (images etc) than a real browser would, which of course meant that the total page load time would be slower than you'd experience with a real browser. In short, our browser emulation would not be very accurate in many cases.

An interesting side note here is that no other page analyzers seem to fully support CSS. We have looked around and found that all available analyzers seem to have this same problem.

We decided to see what we could do about implementing true CSS support in our analyzer, and started by looking at available CSS libraries for Python (the analyzer is written mainly in Python). We found two contenders - CSSutils and CSSEngine. After looking at both, we decided to go with CSSEngine as the code base seemed to be a better starting point for what we wanted to do. The drawback with that library was that it hasn't been maintained the last 3 years or so.

We cleaned up the code a bit, made it more forgiving so it wouldn't croak and die when it encountered the sometimes horrible mess that constitutes CSS files on the Internet, and made it CSS3-aware (CSS version 3 is fairly new, so the library had no support for it).

Then we started testing our new CSS parsing capabilities, and found to our disappointment that it was slow. For the sites out there with large CSS files, the parser was really slow. Glacial, in fact. What to do? We started out by using Cython to convert the Python code to C and then compile it. This improved speed 3-4 times, but it was still too slow. Parsing a really large CSS file would take 15 seconds or more, and that wasn't acceptable.

So the hunt went on to improve speed. It turned out that the culprit was the CSS parsing, mainly the execution of all the scattered regular expressions. We found Ply, a python implementation of the popular unix programs Lex and Yacc. After some more searching we found css-py, a lexer and parser-grammar written using Ply. Unfortunately css-py seems to have been unmaintained for over a year and both the lexer and parser-grammar needed alot of modification to support our needs, handling broken CSS and CSS3. In the end we used parts of the lexer but had to rewrite the parser-grammar.

With the help of Ply, we managed to speed up execution enough that we were happy with the performance. The end result then was that the parser/cascader became a mix of code from CSSEngine, py-css and newly written code. It supports the following:

• Selectors according to the CSS3 selectors module
• Namespaces according to the CSS3 namespaces module
• Cascading according to CSS2.1, also referred to by CSS3
• Most color spaces (rgb, rgba, hsl, hsv) according to the CSS3 colors module
• Large parts (but not all) of the CSS3 values module

So, while the user agent emulation we had before was very likely better than anything any other analyzer could come up with, it is now better yet. We now emulate most important, performance-affecting characteristics of modern browsers (and other user agents) and can therefore provide a fairly accurate analysis, that tells the user how fast his/her page loads in different browsers. This is something that few, if any, other analyzers can claim being able to do.

The web page analyzer has been updated with more features

Today we installed some updates for the new web page analyzer. These are the major changes we've made:

New top-of-page layout:

The new layout emphasizes the browser emulation feature of the analyzer. This feature makes our analyzer quite unique, as most analyzers have no emulation capability whatsoever, and of those few that do (we know of only one), the number of browser-specific, performance-affecting characteristics they emulate are not many.

New config parameter: client bandwidth limit

The client bandwidth parameter is configurable under the "Advanced settings" panel. Only premium users can change this parameter. The client bandwidth limit emulates a certain client bandwidth. The actual bandwidth used during the analysis will of course also be limited to the available bandwidth to (or from) the web server that is being tested - whichever of the two is lowest. The default setting is 10 Mbit/s bandwidth limitation for a test.

Load diagram:

We also made some small changes to the load diagram presentation. For compressed resources, we added information about the original, uncompressed file size in the mouseover tooltip:

Summary:

For the Summary section (below the load diagram) we added info about what browser emulation a test used, and a "Status:" line that shows whether the analysis ran to completion without any problems, or if it timed out or was aborted for some reason.

Detailed page load analysis of your webpage

The beginnings

A while ago we sat down and thought a bit about who our users were, and what they needed. A large part of our users are web developers, using our service to test and optimize their web code. They need tools to help them do this, and while Load Impact may be an excellent tool to help show how web performance changes with changing load, we felt that it lacked some features to help web developers really debug their web pages.

Developers want to not only apply load to a server and see what happens, they also want to be able to analyze the performance of a single page, in detail, with or without external load applied. We felt that Load Impact was lacking a good web page analyzer feature.

We know this functionality already exists out on the net. Pingdom, for example, has a pretty good analyzer at their site. However, we had two reasons for making our own analyzer. Firstly, we knew we could do better :-) Existing analyzers out there were not very detailed, not very well documented, did not look so good (some are downright ugly), or lacking in features. Secondly, we wanted to be able to integrate the analyzer with our load testing platform in the future, so that load testing configuration settings can be used when analyzing pages, and vice versa.

So, we told one of our developers to spend the summer researching existing web page analyzers out there, and create something better. And that is exactly what he did.

 Try it out

 The analyzer actually speaks for itself, so if you want to you can skip the rest of this article, and just try out the analyzer right away.

If you want to access it later on, it is easy to find through its link on the main menu.

So what is it? How does it work?

The basic idea is pretty simple; You enter the address of a web page, and the Page Analyzer loads the page as if it was a web browser. Then it shows you what objects were loaded, in what order, and how long each object took to load. It also gives you information about return codes, compression ratio, etc. and provides some general statistics about the page in a summary at the end.

Here is a screenshot of the Page Analyzer in action. It has loaded http://sun.com and shows us the objects that were loaded, in the order they were loaded. Click the image for a higher-resolution version.

The colored bars show the load time for each object. The load time is divided into several components, and if you put the mouse pointer over the bar you will get a small lightbox that tells you what these components are. On this screenshot we have put the mouse pointer over an object that spent 1 ms in the load queue (waiting to be loaded), 15 ms was spent establishing a TCP connection to the web server, 469 ms was the time to first byte (the time it took to send the request and until the server started delivering the data), and finally the download took an additional 179 ms after "first byte".

 It shows pretty pictures

Moving the mouse pointer over the URL of an image causes the image to be displayed in a lightbox, like you see on the screenshot here.

And now for the advanced stuff

Unlike any other analyzer out there, our Page Analyzer does not just load a web page. It also makes a pretty good effort at emulating different browsers when it loads things.

The browser emulation settings
control this functionality. You can configure the Page Analyzer to emulate a whole range of different browsers/clients, including search engine bots/crawlers such as Googlebot.

The emulation modifies the HTTP-User-Agent header to mimic a certain browser/client, of course, but it doesn't stop at just doing that. It also emulates a lot of other parameters that vary between browsers and which may affect performance.

To emulate different browsers/clients, the Page Analyzer will:

• Vary limits for max no of concurrent connections to a single webserver, and total number of connections
  
• Use different Accept- headers
• Emulate behaviour of browsers that can't always parallelize downloads of certain resources
• Use different keep-alive timeout settings

The end result is a page analyzer that is smart enough to let you fetch a page as if you were using a specific browser. From the start we emulate all major browsers, Googlebot and other search engine crawlers, etc. In future versions we will likely provide a way for users to modify all settings individually.

 More advanced features - compression ratios

On this screenshot you can see that the Page Analyzer reports compression ratio for downloaded objects.

Click the image for a higher-resolution version

 Over-compression

And on this screenshot the Page Analyzer warns about a couple of very small objects that have actually increased in size as a result of them being compressed.