Load testing ajax websites

We've been doing a lot of load testing at Hunch. Like most modern consumer sites, we use a lot of Javascript. We also customize nearly every page we serve to each user: which questions they're asked, what order our recommendations are ranked in, etc are all dynamically computed for each user.

We used to do load testing with jmeter and lots of programmatically generated lists of URLs to fetch. We'd try to generate lists of initial page load and ajax update urls and then run through tens of thousands of them in somewhat random order. We had to make sure that each simulated user was telling the site different things about themselves so that we could really stress all the per-user recommendation systems and not just have a thousands of copies of the same user being simulated. We also had to make sure that the load tests loaded sequences of URLs that would really be loaded if a real user were using the site. If a new ajax call got added on some page, we had to make sure the load test reflected this.

This was obviously a real pain. And very error prone. We found we were constantly testing the site in ways similar to the way a real user would use the site, but not exactly like they used the site. And all it takes is one bad query somewhere that doesn't scale well to bring your site to its knees. Shame if you do all that load testing and miss the one ajax call that invokes that bad query...

Instead recently I've been using a new service called BrowserMob. They run multiple Firefox instances on EC2 to simultaneously put load on your site. Because they're really running Firefox, it's much easier to really simulate the full user experience when visiting your site. Javascript runs, ajax calls get invoked etc. You still have to write detailed javascript test cases to drive the simulation, but I find that it's much easier to really test the site the way a real user would use the site.

We still use jmeter since it's cheap and easy to put a ton of simple load on the site. But I think I'll be using BrowserMob more and more over time.

Disclaimer: I don't have a financial or other interest in BrowserMob, though I do know and like the founder.

Competing with large companies for engineers

My theory is that good engineers want to work with people they can learn from, work on interesting problems and see the fruits of their labor appreciated and used. Startups have a huge advantage here over large established companies. Hopefully during the interview process you showed off all the great people in your company, the involvement in product design your engineers have, and the high level of responsibility each engineer has.

At Gigantic Co the average quality of the engineers is not going to be as high (it can't be with 1,000 engineers), the size of the problem each individual engineer gets to work on is smaller (bureaucracy loves slicing problems into ever thinner pieces and distributing them around to too many people), some product manager is telling them exactly what to build, and the chance for personal fame is low.

What Gigantic Co can offer is more cash and the perception of stability. If a candidate just wants cash, you're going to lose them to the big companies and so be it. You didn't want them anyway. You're building equity value for the future and you want engineers who think the same way.

The risk perception argument is a tougher one to win, though it really is a bogus argument. There is effectively 0% unemployment for good engineers. So the only real career risk for an engineer is opportunity cost of not working on something interesting. There's effectively no risk that the engineer would be unemployed for long if your startup blows up.

Assuming you code tested the candidate, you know they are great and so you can invest a lot of effort into personally recruiting them. You can have your CEO or VCs call the candidate -- many candidates will be flattered by the attention. Candidates may also appreciate hearing more about the business strategy for the company, hearing why the VCs invested etc.

RAM is the new hard drive

Like most websites, Hunch uses memcached a lot. The really interesting thing is that the data stored in them is basically as persistent as data written to disk: the machines have never gone down, lost power, or had memcached crash in roughly a year of use.

Hard drive MTBF is a few years in practice. So our memcached is already 20-25% as reliable as a traditional drive. We just need to build all the other things that traditional storage has that memcached lacks: RAIM (reliable arrays of inexpensive memcaches?), backup/restore, replication, etc.

Of course, the downside is that a terabyte of RAM still costs something like $60,000. But that price is going down and I'm not sure that the datasets behind many websites are really that big. For example, Wikipedia is only a little over a terabyte.

Parallel programming is easy

Pundits have been wringing their hands over the difficulty of parallel programming for several decades at this point. Every time a new processor with more cores is announced, I hear some new dire prediction about how we'll never be smart enough to program it. And they're right, it is hard. But we're not seeing huge challenges in building consumer and business applications that scale relatively easily.

Instead, the opposite seems to be happening in the real world. Databases, websites, CRM systems, etc process more and more requests per second using more and more processors, cores and servers. Yes, If I'm writing a complicated algorithm to simulate something or building my own database server then I have to think carefully about locking and synchronization. But luckily, very few people have to do that and those that do can afford to think long and hard about these problems.

Instead, most of what programmers do is write mostly sequential programs that happen to run concurrently with many copies of themselves, largely oblivious to each other. By and large, the software engineering world has developed a few basic abstractions (going back to IBM in 1969 with CICS) that make it actually quite easy to write parallel business-oriented programs.

It also turns out that most real world business applications have tons of data parallelism to them meaning they inherently require very little synchronization. For example, two web requests to render pages about books on Amazon really don't have to coordinate much at all. Roughly, they have to coordinate access to an integer representing inventory levels and coordinate with any backend updates happening to change titles, descriptions etc.

Hunch "Blog Census" widget

We've been working on a "census" widget for blogs. It asks your readers questions about themselves and then tells you (and your readers) interesting statistics about them. As a bonus, it will give your readers predictions about themselves based on the Hunch corpus of data.

Powered by Hunch.com

These aren't available publicly yet, but will be shortly.

Update: You can now add a reader survey to your own blog here (login required).