Tuesday, December 8, 2009

How to survive being slashdotted

There's nothing like the pride of having your content featured on slashdot, digg, reddit etc turn into the depression of watching your servers melt down under all the load. Here are some tips to make sure that doesn't happen.

First, the basics:

* if using Apache, use worker MPM with MaxClients and ServerLimit as high as you have RAM for. Better to cap max # of threads and reject some users than let the whole machine go down. Need to benchmark to figure out how high you can go before your machine dies.

* turn off keep-alive for your dynamic content servers

* benchmark your website ahead of time with realistic workloads simulating what real users will do in terms of page flow, click rate, ajax, data values they enter into forms etc.

* run explain on queries to look for queries hitting every row in your db. Looks at MySQL's slow query log!

* follow Yahoo's advice on static content (javascript minification, reduce image sizes, serve static content from several domains, etc).

* make sure you have enough OS file handles (/etc/security/limits.conf on Redhat), MySQL connections (max_connections = 5000), Tomcat threads etc.

* turn off DNS reverse lookups for Apache logging (HostnameLookups Off)

If you think a particular page or set of pages could be slashdotted, consider changing your code to serve the page out of memcache. Alternatively, generate a static version of the page and use mod_rewrite to transparently serve it. Wordpress can do this via the super-cache plugin. If your need to update these pages, use a backend process that periodically re-generates the cached or static version of the page every few minutes.

If all else fails...send traffic from slashdot to a static page asking them to try again later. This allows your normal site visitors to keep using the site, while gracefully shedding the slashdot load.

RewriteCond %{HTTP_REFERER} ^http://slashdot.org [NC]
RewriteCond %{REQUEST_URI} !^/media/slashdot.html$
RewriteRule (.*) http://www.foo.com/media/slashdot.html [NE,R=301,L]
blog comments powered by Disqus