Load Impact Load Testing and Performance Testing

How many times can you get slashdotted?

Slashdotting (or "the slashdot effect") is a term coined by the site slashdot.com. It means that some big website/blog/newssite writes an article about you, causing tons of their visitors to take a sudden interest in your website, and causing your site to get overloaded with all the new visitors.

Slashdot has been around for a long time, but now there are many other sites that are big enough to cause a slashdot effect when they publish an article about something. One such site is the Russian web developer site habrahabr.ru where they call the effect the "habr effect".

We have had extensive experience with the "habr effect". We were mentioned on habrahabr.ru in february 2009 the first time, which caused some traffic to come our way. However, it seems habrahabr.ru has increased its number of readers substantially since then - they seem to have around 10x as much traffic today as they did early 2009, according to Alexa.

So, on wednesday they published an article about the importance of load testing as a way to avoid the habr-effect. They suggested people use Load Impact for their load testing (which we think is a splendid idea, of course). This resulted in a lot of people coming to our site to try out our load testing service.

This wouldn't have been a problem under normal circumstances, but an unknown bug in our frontend code had made it possible to start an unlimited number of load tests. We never noticed this under normal traffic conditions, but when several hundred new visitors arrived at the same time from habrahabr, and many of them tried to start a free load test, we suddenly found ourselves executing close to 200 load tests at the same time!

Our system was having problems: We had been habr'd (slashdotted) because of an article about how to avoid getting habr'd.

the effect the habrahabr article had on our (concurrent) site visitors

After some frantic bug-hunting, we found and fixed the frontend bug, and things started working much better.

Then we thought "hey, this was somewhat funny". We decided to write a blog article about the load testing people who got overloaded because of an article saying you should use their load testing service to avoid getting overloaded.

I wrote and published that article on our blog yesterday (thursday).

Today (friday) habrahabr picked it up and published a link to it. Guess what happened?

Yes, same thing (but a longer spike this time, so more traffic)

So today we have achieved something remarkable:

• Today, we were habr'd because of an article about being habr'd because of an article about how to avoid being habr'd!

All programmers out there will love the recursion, I bet.

Note: to be honest we didn't have much problem with the traffic today, even though it peaked at more visitors per hour than we normally get in a whole day.

Russian site habr's Load Impact!

Yesterday, the Russian site habrahabr.ru wrote an article where they warned people about the habr effect (see slashdot effect) and suggested that it was prudent to use Load Impact to load test your site before getting swamped by traffic due to some popular blog or newssite (like habrahabr.ru) publishing an article about you.

This, of course, caused us to get habr'd!

We found that our system was suddenly struggling to keep up, and even though we could see that we had a big traffic spike, we didn't at first understand why the machines were having such a hard time. This is what our concurrent (simultaneous) visitor graph for the past week looks like:

Now let's see, is it possible to determine when the habrahabr article was published? Tricky.

As can be seen, our average traffic this past week has been about 30 concurrent visitors, with a max of around 45 users on the site at the same time. When Habrahabr published the article we suddenly got close to 200 concurrent visitors.

Now, our system is designed to handle more visitors than that. We have had some 300 or so concurrent visitors in the past, when articles have been published about us, but it has not caused a very big problem for our servers. Yesterday, everything slowed down to a crawl, which was very strange.

It all turned out to be due to a malfunction in our test queueing system. As each load test can require quite a lot of system resources to run, we have a queueing system that makes sure we don't try to run too many load tests at the same time. Normally, we wll only allow about a dozen concurrent load tests running. But as it turned out, the queueing system was malfunctioning, and let visitors run as many load tests as they pleased. Under normal traffic conditions, we didn't notice the problem, but when 200 habrahabr.ru visitors all started load tests at the same time, our system suddenly got quite busy.

At one point there were 180 load tests running at the same time - We were load testing close to 200 sites at once!  (must be some kind of new record)

Luckily, practically all of these were small (free) tests and our load generator nodes were actually almost idling despite the excessive number of tests running. The loadimpact.com website, however, had problems. Especially the database had problems keeping up with all the writes caused by test results flowing in from so many concurrent load tests.

This situation went on between about 2 pm (Russian Moscow time - no, we're not russian, but most of the visitors from habrahabr.ru are) and 4 pm, then we found and fixed the problem with the queueing system, causing the number of running tests to go down to normal levels again. So to any of you out there who tried to use Load Impact or run a load test yesterday between 2 and 4 (noon and 2 pm UTC, or early morning in the US), please excuse us and please try again!

Another update

Some people seem to have misunderstood the numbers and what actually happened. I'll try to describe it in other words.

What we initially thought was that we just had a website visitor spike of about 200 concurrent visitors (don't confuse this with visitors per hour, HTTP requests, or visitors per day - see this article for an explanation). We couldn't understand why our system was so slow when it has been designed for up to 300 or maybe even 400 concurrent visitors (10-20x our normal traffic).

As it turned out, it wasn't the number of visitors on our website that caused our system to slow down. It was the number of load tests we were running.

People can run free load tests from our start page, and each free load test we execute means that we start up to 50 concurrent simulated users that access an external website that is to be load tested. Those 50 simulated users might load thousands of objects/resources from the external site, and the load test continuously updates our master database with information about how fast different objects on the external site are delivered to the simulated users. We can get hundreds of such load time results per second from a single load test, all of which go into the database.

Normally, we allow about a dozen concurrent load tests, but in this case a software bug made it possible to start an unlimited number of load tests. As most of the visitors were web developers interested in load testing, most of them started a free load test for their site. This meant that we at one point had about 200 load tests running at the same time, generating probably tens of thousands of database updates per second. This was more than our database server could comfortably handle.

Like everyone else, we have to judge what performance levels our systems should be able to handle, and build things accordingly. We try to make sure our system can handle at least 10 times the normal average traffic, which usually makes us able to handle a Habr or Slashdot effect, but in this case a silly little bug killed one of our most basic performance-protecting features, which kind of put a spanner in the works, so to speak.

Buy 7-day premium accounts

Load Impact is proud to offer the week pass, which allows you to buy premium accounts for just one week. This means we now have three different options:

• the day pass - this gives you 24 hours of access to a premium account level
• the week pass - this gives you 7 days access to a premium account level
• the monthly subscription - this gives you 1 month's access to a premium account level and renews itself automatically unless cancelled

The prices are such that the more time you buy, the less expensive it gets per hour. There are three premium account levels you can purchase:

1. Load Impact BASIC
   Allows you to run tests with up to 250 concurrent simulated clients
2. Load Impact PROFESSIONAL
   Allows you to run tests with up to 1000 concurrent simulated clients
3. Load Impact ADVANCED
   Allows you to run tests with up to 5000 concurrent simulated clients

For more information about account levels, go to our products page or see our FAQ entry: What do I get for free and what do I get if I buy a premium account?

Load Impact passes 50,000 executed load tests

When we launched Load Impact, at the beginning of 2009, we could never dream that the service would be so well received and so widely used. At the time of writing 51,623 load tests have been executed on loadimpact.com, and that number increases at a rate of over 200 tests per day.

In total, we have made over a billion successful HTTP GET requests since we went live, fetching over a million unique URLs.

Our unique and brand new web page analyzer is quickly becoming popular also, with almost 100 analyses run per day, and 5,000 analyses in total so far. Keep an eye on this analyzer in the near future - it is already quite possibly the best one you will find anywhere, but we have plans for some additional major improvements that will make it simply outstanding.

All in all, we are incredibly happy with how we have been received by the Internet community since launch, and we hope that you will continue using Load Impact for your load- and performance testing needs in the future. We urge all our users to get in touch with us and tell us what they like or dislike about the service, so that we may make it even better.

  /The Load Impact team

Load Impact nominated as one of the rising stars of the north

SIME, Northern Europe's largest Internet business conference, has nominated LoadImpact.com as one of the possible winners of their grand prize. The yearly event honors excellence in online communication, business practice and technology, recognizes the individuals and organizations responsible and inspire the Scandinavian Internet industry to world-class performance.

For more information, go to the nomination page: http://sime.nu/09/awards/rising-stars-of-the-north/load-impact/