Code tests when hiring engineers

I also always do some sort of code test after a first round of interviews. At the end of the day you're not hiring programmers based on their ability to talk a good game. They should want to show off what they can do. If they really are 10x better than the average (and yes, good programmers are at least 10x better than then average) they will want to show off their coding since they know it's their competitive advantage. I think a lot of great candidates think better of a company that openly evaluates people based on programming skill -- it's the kind of environment they want to work in.

A good programming project should evaluate whether the programmer will work well in your company. For me, this means working on big underspecified problems, getting something to work quickly and iterating, using any open source tools or code they want, looking up stuff on the web, etc. The only thing I don't allow is someone to talk to another person for help. I do the code test in person just to reduce the possibility of outside help.

A classic coding problem we used back at SiteAdvisor was to write a program that would crawl a website, automatically fill out forms and evaluate whether the form was accepted successfully or not. We'd ask people to do this in about two hours. Obviously, this is a big project but it was extremely interesting to see what people would accomplish in those two hours. Would they have simplified the problem down to a core that worked and which could be expanded on later? Or had they simply drawn up a list of outside vendors they would hire to help with the project (I kid you not one senior candidate actually did this).

The programming test gets at what is one of the most important things to me. Is the candidate a scrappy thinker who finds ways to get something working no matter what? Most startup programming projects are not about inventing new algorithms or big ideas. They're about making daily progress against a nearly insurmountable set of challenges that require quick clever workarounds.