A while ago, we published a blog entry here about browser behaviour and performance where we told the story of how most browsers seem to transmit AJAX POST requests (i.e. HTTP POST requests originated from Javascript code) as two separate TCP packets, while a small number of browsers - most notably recent versions of Firefox - transmit the same request using only a single TCP packet.
This behaviour was discovered by a Yahoo developer originally, and later on investigated and confirmed by Joseph Scott on his blog. We went further and mapped a wide range of browsers on several different operating systems, to find out in what circumstances the request was sent as two packets. The results can be read in a PDF report we created, if you're interested. (If you are not, then you should probably stop reading now, because for non-geeks, this post is only going to get worse from here)
Because we are such mega nerds when it comes to performance stuff, we decided to dig deeper into this, and find out just how big a performance impact the behaviour had. I.e. we wanted to see how much slower a request would be, on average, when the browser transmitted two instead of one TCP packet. Logically, more packets mean more overhead and a bigger risk of one packet getting lost and having to be resent, causing extra delay. Yahoo actually advice people against using POST in Ajax requests because of this behaviour - see Yahoo's ySlow performance rule #15 - Use GET for Ajax requests - but they don't tell you what it means for performance if you do/do not use POST. To us, just like to Yahoo apparently, it seemed rather obvious that sending two packets was worse than sending one packet, but maybe it shouldn't be a general performance recommendation unless it has a significant impact (if not, then other recommendations would perhaps have been more important to get across to ySlow users than this #15).
So, we wanted to see how big the impact actually was. By sending a large number of AJAX requests using both the GET and POST methods, we reasoned that we should be able to see some statistical difference between the response times of the two. Especially if we also simulated some network delay and packet loss on the link, the difference had to be quite noticeable. Our goal was to be able to say e.g. that using GET instead of POST in an Ajax request can reduce average transaction time by up to X% or more, depending on network conditions.
We set up the test, and spent several days running hundreds of thousands of Ajax GET and POST requests using different browsers, and emulating different network characteristics (i.e. network delay and packet loss). In the end we found something we weren't prepared for - the tests showed that sending two packets seemed to slightly outperform sending one packet. (Again, read the PDF report if you want more details)
After scratching our heads, asking around, double-checking the data, reading up on TCP and its most common implementations, then scratching our heads some more, we still couldn't understand how this came to be. There might have been problems with our setup somehow, but it seems far-fetched that any problems with our setup would give a 2-packet transaction an advantage over a 1-packet transaction. We decided to publish the results and see if anyone could explain what we were seeing, or tell us what we were doing wrong. We have since had a couple of people offer things such as "two packets allow faster resend if the first packet is lost" or "sending more packets might help TCP compute the RTT better". But none of the explanations seem to suggest anything that could outweigh the penalty you get from the fact that the 2-packet transaction suffers twice as much from packet loss as the 1-packet transaction. If it does - maybe that's the assumption we make that is wrong.
At the Velocity 2010 conference, I ran into a well known web performance guru and while talking to him a guy from the Yahoo ySlow team showed up, so I mentioned our findings to them both. The guru said he didn't believe me, and tried to change the subject. The Yahoo guy didn't say anything but walked off as soon as he got the chance. The obvious conclusion from this encounter is that I have to work on my people skills. Or maybe they were just stressed, or suffering from caffeine deficiency (it was a coffee break I think) - who knows? Maybe I was suffering from caffeine deficiency. I hope I don't come off as one of those annoying people who can't stop trying to find faults in stuff other people do, in an attempt to feel superior. Because I am not - I just can't help trying to improve things everywhere, all the time. It's like a drug to me. I have to figure out how things work, and then try to improve them. Have to. I can probably be a pain about it - when people do something really well and expect praise I will usually tell them "Nice! The only thing you might want to change for next time is [whatever]..." and many take that as a sign that I don't like what they've done. For me, constructive criticism is the finest form of praise!
Anyway, enough of the boring introspection and back to the important stuff - investigating the problem! I decided that if people don't believe me, and they don't have time to set up a test bed and run the tests for themselves, the obvious solution is that I set up a test bed they can run tests on remotely. If I can get the critics to observe the strange behaviour first-hand, I might force them to acknowledge that it does happen, if nothing else! Then more people will be likely to try and come up with an explanation. I would be happy to learn that there is some problem with our setup, for instance, because then I would at least know. Not knowing is frustrating. How can I work to improve something unless I understand it first?
So, I have now a test system up and running where anyone can try executing 50 Javascript GET, then 50 Javascript POST operations and see which are faster. I also set up the test on both Apache 2 and Lighttpd, in case this problem is webserver-dependent somehow. I thought, why not name it "Scott's problem", after the Joseph Scott who first bothered to describe it in detail. So the address to the test pages are:
On the page you can see the tests other people have executed using different browsers, and their results. For each unique user agent you will get a summary with average transaction times. Feel free to run multiple tests if you want, you will probably not skew the data as the system only uses the average transaction time seen from a certain IP and a certain browser, when calculating average transaction times for that particular browser. The machine is an Ubuntu server, in case anyone cares.
And yes, I know my web hacker skills are pretty awful. Don't look too closely at the javascript, or the HTML, please. You just might go blind.
