Load Impact Load Testing and Performance Testing

The art of peak traffic estimation

"How much traffic should I simulate?" is one of the most common questions that we get on load testing. In our previous post, we have already covered how to calculate concurrent users from hourly visitors (you can refer to: http://loadimpact.com/blog/monthly-visitors-and-concurrent-users) but the question still remains - how much traffic should you simulate? 

There are several approaches to do this and you can choose to simulate: 

1)  the average number of hourly v../blog/monthly-visitors-and-concurrent-usersisitors over a period of time

2)  the peak number of hourly visitors over a period of time

3)  increasing the number of visitors until the server breaks

Which method to choose really depends on what you are trying to find out. For 1) the goal would be to ensure that the average load time is at or below a certain level for most visitors on the site. 2) allows you to find out what will happen to your site when there is a sudden surge in traffic. This could happen if the website goes viral online or if there is a marketing campaign. The 3) approach would be to find out the limits of the server, but it is actually not very useful to know when your server will stop responding. That is because it could well be responding to 100,000 concurrent visitors but with an average load time of 30 seconds. Most visitors will probably not bother to load the website in such a situation, and leave. 

Let's focus on finding out how to determine the number of visitors to simulate if you want to prepare for the event of a sudden surge in traffic. 

To do this, I studied the relationship of peak and average traffic from statistics available at Quantcast.com. 

Here is a typical traffic chart from Quantcast:

For the purpose of this study we will only be focusing on traffic from the USA. We will also obtain the average and maximum daily traffic over a 6 month period. This data is was retrieved on 12/02/2011. We also only used data that was MRC accredited. (For more information you can refer to: http://www.quantcast.com/how-we-do-it/mrc-accredited-traffic-measurement). 

The information is summarized here:

*Factor = Maximum daily visitors/Average daily visitors

Notice that "Maximum daily visitors" is 2-8 times higher than "Average daily visitors". This is off monthly traffic of 400,000 visitor's. We compile similar statistics for sites ranked in the 10,000s and 20,000s. The results are as follows:

Note that maximum daily visitors are now 2-10 times higher than average daily visitors. Monthly traffic that each site received was about 195,000. Now let's look at sites ranked in the 20,000s. The results are as follows:

You may notice that maximum daily visitors are greater than average daily visitors by a factor between 2 and 38. The variance has thus increased dramatically. One explanation would be that higher ranking sites with high average monthly visitors are less likely to be affected by a sudden surge in traffic. Large sites like www.cnn.com have high, stable visitorship, and the impact of a popular news story such as protests in Egypt would be much lower as compared to a site with low visitorship. A site like michaelmoore.com can be flooded with traffic if a particular piece of news about Michael Moore becomes viral. So, let's take a look at michaelmoore.com statistics:

It appears that on the 14th of December 2010, Michael Moore decided to contribute $20,000 USD to Julian Assange's bail. 

This news was reported on many major news channels, causing traffic to spike 38 times higher than normal.

According to Scott Galloway Clinical Associate Professor of Marketing at NYU, there are 3 elements of viral content:

1) Authenticity
2) Humor
3) Social Debate

In Michael Moore's case we can see these elements coming into play. What is interesting though, is that most of the time, going viral normally catches people by surprise. They are not prepared for sudden fame. Neither are their websites. Imagine your website being hit with 100,000 visitors in an hour, you should be overjoyed right? But most users actually get a BAD experience because the page is slow loading or worst- unavailable. We think that if your website falls into the category of having low stable traffic with the chance of going viral, you should not hesitate to test more than 30x average traffic.

There also seasonal peak traffic that is easier to estimate. Let's look at another site, www.holidayscentral.com:

Notice that peak traffic occurs in October and December and this pattern probably repeats annually. If your site's traffic is similar to www.holidayscentral.com and experiences surges in traffic that are predictive and repeated year after year, then you just have to look at your past data to find peak traffic. Then proceed to add the % growth expectations for your traffic. For example, if peak traffic last year was 20,000 visitors and average visitors this year is 10 % higher than last year, you should be using 22,000 visitors to test for peak traffic this year. 

=====================================================================================

Conclusion

We can follow these guidelines for estimating peak traffic:

If you do not have prior peak traffic data and...

1)...if your site has low visitorship and contains content that could go viral, test up to 30 times average daily traffic. 

2) ...if your site has high and stable visitorship, test up to 5x average daily traffic.

If you have prior peak traffic data and timing of peak is predictable (seasonal traffic), use past data and add a % growth in traffic to arrive at the final number. 

When uncertain, just remember that testing with more users is (almost) always better than with less. 

/Jack Zhang

P.S. : Michael Moore, if you are reading this (you are probably too busy "taking on" the health insurance industry), we can help you run a load test on your website. 

Animal tussle going on, please hold

As most of you know, Twitter displays a "Fail Whale" when their servers are overloaded and a 503 page is displayed. A 503 page is displayed when the server is down for maintenance or overloaded. A few companies have decided to modify their 503 page to make it more interesting.

Matthew Inman (creator of http://theoatmeal.com/) decided to create a similar page for Tumblr. Its called Tumblbeast and it looks like this:

Update: Tumblr has renamed it Tumbeast and officially adopted it as a 503 page. 

We thought we should make one for our customers....

Just for fun. =). 

You can link to this image at: 

<img src="http://s3.amazonaws.com/loadimpact_us/images/animaltussle2.jpg" border="0"
alt="Animal Tussle" />

By the way, for those who are interested in making a 503 service unavailable page, here's a decent guide: http://webhostinghelpguy.inmotionhosting.com/web-hosting/how-to-make-a-503-service-unavailable-page/

Update: Oatmeal created this image:

Like us, Matthew is taking a jab at the Fail Whale.You can read more about it here: 

http://theoatmeal.com/blog/fail_whale

Learn how to use Google Analytics to derive common user behaviour

Introduction

In addition to finding out how many simultaneous users you need for a load test (for more information you can refer to http://loadimpact.com/blog/search?criteria=analytics), you might also find it challenging to choose visitor behaviors that are representative of all your site visitors. There are many permutations of behaviors possible and it is not practical to simulate everyone of them. We can, however, make use of analytics software to find common user behavior. This guide explains how Google Analytics helps you do the job.

Extracting the data

On the right menu of Google Analytics, select Content -> Top Landing Pages.

Choose an appropriate date range. In this example, we have chosen two months worth of data. Wider date ranges produce more accurate user behaviors, but will not be able to tell you the latest common behavior. To avoid ending up with irrelevant user behaviors, be careful not to choose a start date before any major revisions of your website.

You should now be able to see this  on the right hand side of the page. Click the second option to have the data displayed in pie chart form.

Let's focus on the two most popular landing pages, "/" and "/index.php". Visitor paths will be represented by nodes. We will dive into the concept in more detail in a later analysis, but let's complete the whole diagram first. 

Next, under "Overview", click "Entrance Paths".

Here you will see the various paths of different content.  "Then viewed these pages" shows what was viewed and the corresponding percentages after your visitors went to the selected content. "And ended up here" shows what content was viewed after that. 

Let's take a look on the paths taken by visitors after arriving at the http://loadimpact.com/ page. We will ignore the first two results as there is an auto loading script in the index page (this can be accounted for by increasing sleep time later on). Hence, other probable content that our visitors will go to are "/products.php?light" and "/products.php?basic pages". We will add this information to the diagram:

We click "/products.php?light=" and look under "And ended up here:". Again, we find the two most visited pages and plot them in the diagram together with corresponding precentages of users that came from the previous page.

For the branch starting with "/index.php", we get the following:

The above diagram shows the paths commonly taken by visitors starting from loadimpact.com/index.php. 

We could go on for a few more iterations, but for this example we will stop here. Next, go to "Top Exit Pages" and verify that the top two exit contents appeared in your last iteration.

In this case they did appear and we can safely say that three iterations are sufficient to give an accurate picture of visitor behaviors.

Analysis

In every branch, we multiply the percentages together to obtain a "comparison number". For example, starting from /index.php to /pageanalyzer.php to / yields 0.064*0.0403*0.0221 = 0.000057.

We do that for every branch and then rank the sequence of content from highest to lowest "comparison number".

Hence, the most common user behavior would be to go to our index page, and from there click "Sign up now" under "Load Test Light", and then return to the index page before leaving the website. This is illustrated in the top diagram below.

The other common behavior would be to go to our index page (Note that http://loadimpact.com/ and http://loadimpact.com/index.php point to the same page with the same codes), and then subsequently click "Sign up now" followed by "Proceed to registration" on the products page.

Once common user behaviors are determined, we can proceed to transform it into a load script via the session recorder (available through purchase of any Load Impact Premium account).

It should be noted that this method is only valid for a general load test on your website. You might also want to test specific functionality, in which case this method is not very useful.

New cool stuff on loadimpact.com

Today we performed a site update, with some new functionality and bug fixes. Most of the fixes are behind the scenes stuff that is not immediately visible to users, but we do have one thing that stands out and which can be seen by all registered users and that is our new test validation feature. 

Now, when you create a new test configuration, you will see this on the edit test page:

All tests have to be validated before they can be run. What this means is that when a test configuration is created or changed, we force people to "validate" the configuration before they can start a test using it. The reason we do so is because it allows early detection of load script problems, DNS problems, or quota problems that would prevent the test from being run. We have had many instances where users configure a test and start it, only to find that there is some problem with the test that causes it not to run properly. The test validation feature is meant to allow early warning/prevention of these problems. When you validate a test configuration, what happens behind the scenes is that we run a very short, 1-client test using your configuration. If everything goes well, the test is considered to have validated OK, but if there is an error somewhere, you will see an error message that looks something like this:

We think this feature is going to make it a lot less likely that people misconfigure their tests, and don't notice until they try to run them. The validation feature removes all sleep statements from load scripts when they are being validated, in order to make validation faster. This means that all validations should be fairly quick and painless.

But wait, there is more!

New back end load generator

This is a major change behind the scenes. We have replaced the old load generator, myLoad, with our next-generation load generator rLoad. Both are highly efficient programs, written in C and designed to be able to simulate a lot of clients in a very resource-efficient manner. The main difference between the two is that the new load generator has scripting capabilities and a lot of other features we were missing from the old one. The scripting features are turned off at the moment, because we still need to implement some support systems that are necessary for scripting functionality in Load Impact. However, there are some other features we gain access to immediately. For instance, rLoad will use all available IP addresses when generating load traffic. This means that when you run a 50-client load test on e.g. the Stockholm2 load generator, you will now see traffic from 50 different IP addresses hitting your site. The Stockholm1 and Stockholm2 load generators have 125 IP addresses each, so tests up to 125 simulated clients means the clients will all have unique, individual IPs. If you run larger tests than that, the 125 adresses will be used by the clients in a round-robin fashion. Here is an FAQ entry listing the IP addresses used by all our load generator nodes: http://loadimpact.com/forum/viewtopic.php?id=133

The informed reader will note that Stockholm2 is a new load generator node, and that Stockholm1 has also changed its IP range from before. We recently upgraded the Stockholm nodes and gave them their own class C network where they have plenty of IP addresses to use for load generation. This might mean, however, that some Load Impact users need to upgrade their firewall settings to allow traffic from the new IPs.

Load Impact has been chosen as one of Europe's 100 hottest companies

This week saw the Red Herring Top 100 Europe awards take place in Paris. Load Impact was one of the 200 finalists chosen to present and on thursday, we were declared to be one of the 100 winners of the prestigious award.

The winners were the companies judged to be the most innovative and to be positioned to grow at an explosive rate. Companies were evaluated on both quantitative and qualitative criteria, such as financial performance, technology innovation, quality of management, execution of strategy and industry integration.

We are of course extremely happy to get even further recognition that we are on the right track. Still, the single biggest positive signal of all is of course, and has always been, our astounding usage statistics. Over 114,000 load tests executed, at the time of writing, is pretty telling. Our users are giving us all the confirmation we could ever ask for, and for that we are ever grateful! We will do our best to live up to all the hype and provide you with the world's best load testing service both now and in the future!