QA and benefiting from your users' testing

From time to time someone asks me about how I think you should test a website. It depends on the type of website -- if your website is responsible for millions of dollars or peoples lives you need to test it differently than if your website lets you play with animated puppies.

The fundamental challenge in testing your website is the number of possible states your code can be in and your inability to test every one of those states. Every "if", variable, or loop in your code, every record and field value in your db, every API call you make to a third party increases the number of states your system can be. It's impossible to test even a billionth of the possible states a tiny site can be.

QA people have developed all sorts of approaches to deal with this like randomized inputs, testing boundary conditions, black box testing based on requirements, etc. However, consumer web sites in particular can benefit from the thousands to millions of users they have since each of these users is constantly testing different states of your system.

You don't want your users running into serious bugs like accidentally deleting their accounts, buying things they didn't mean to etc. Those are the kinds of features you need to have your QA people focus on to make sure they never happen. Your QA people also need to focus on backend processes that user's don't see. But that still leaves a large body of other problems your users may run into.

The key to benefiting from your users finding bugs is to detect problems immediately and fix them so quickly that (hopefully) no one notices them. Have your site send you email on errors (but rate limit or aggregate them so you don't get flooded!). Even better, send the entire dev team the errors so everyone knows asap when there's a problem and so there can be some social pressure to not check in buggy code. If you're fixing problems asap, you need a basic regression test and an automated deployment system so you can push a button and know you're releasing basically good code.