Load testing ajax websites

We've been doing a lot of load testing at Hunch. Like most modern consumer sites, we use a lot of Javascript. We also customize nearly every page we serve to each user: which questions they're asked, what order our recommendations are ranked in, etc are all dynamically computed for each user.

We used to do load testing with jmeter and lots of programmatically generated lists of URLs to fetch. We'd try to generate lists of initial page load and ajax update urls and then run through tens of thousands of them in somewhat random order. We had to make sure that each simulated user was telling the site different things about themselves so that we could really stress all the per-user recommendation systems and not just have a thousands of copies of the same user being simulated. We also had to make sure that the load tests loaded sequences of URLs that would really be loaded if a real user were using the site. If a new ajax call got added on some page, we had to make sure the load test reflected this.

This was obviously a real pain. And very error prone. We found we were constantly testing the site in ways similar to the way a real user would use the site, but not exactly like they used the site. And all it takes is one bad query somewhere that doesn't scale well to bring your site to its knees. Shame if you do all that load testing and miss the one ajax call that invokes that bad query...

Instead recently I've been using a new service called BrowserMob. They run multiple Firefox instances on EC2 to simultaneously put load on your site. Because they're really running Firefox, it's much easier to really simulate the full user experience when visiting your site. Javascript runs, ajax calls get invoked etc. You still have to write detailed javascript test cases to drive the simulation, but I find that it's much easier to really test the site the way a real user would use the site.

We still use jmeter since it's cheap and easy to put a ton of simple load on the site. But I think I'll be using BrowserMob more and more over time.

Disclaimer: I don't have a financial or other interest in BrowserMob, though I do know and like the founder.