Last week was a very frustrating time for me. For whatever reason, an unusually number of botnets decided to zero in on my Drupal site and created what I call an unintentional  Denial of Service attack (DOS). The attack was actually from spambots looking looking for script vulnerabilities found mainly in older versions of e107 and WordPress. Since the target of these spambots were non-Drupal pages, my Drupal site responded by delivering an unusually large number of "page not found" and "access denied" error pages. Eventually, these requests from a multitude of IPs were too many for my server to handle and for all intents and purposes the botnet attack caused a distributed denial of service that prevented me and my users from accessing the site.

These type of attacks on Drupal sites are nothing new and have been observed and discussed at great length at Drupal.org. However, my search at Drupal.org as well as Google didn't really find a solution that completely addressed my problem. Trying to prevent a DDoS attack isn't easy to begin with and at first the answers alluded me.

I originally looked at Drupal for the solution to my problems. While I've used Mollom for months, Mollom is designed to fight off comment spam while the bots attacking my sight were looking for script vulnerabilities that didn't exist. So with Mollom being the wrong tool to fight off this kind of attack, I decided to take a look at the Drupal contributed model Bad Behavior. Bad Behavior is a set of PHP scripts which prevents spambots from accessing your site by analyzing their actual HTTP requests and comparing them to profiles from known spambots then blocks such access and logs their attempts. I actually installed an "unofficial" version of the Bad Behavior module which packages the Bad Behavior 2.1 scripts and utilizes services from Project Honey Pot.

As I had already suspected, looking for Drupal to solve this botnet attack wasn't the answer. Pretty much all Bad Behavior did for me was to take the time Drupal was spending delivering "page not found" error pages and use it to deliver "access denied" error pages. My Drupal site is likely safer with the Bad Behavior module installed, but it was the wrong tool to help me reduce the botnets from overtaxing Drupal running on my server. Ideally, you would like to prevent the attacks ever reaching your server by taking a look at such things as the firewall, router, and switches. However, since I didn't have access to the hardware, I decided it was time to look at my Apache configuration.

I host my sites on a VPS and use cPanel to help manage the site. While cPanel's defaults will give you a stable server there is definitely room to improve the default configuration. Despite all the places I searched for answers, the Apache documentation itself was the most helpful in helping me find which Apache HTTP Server configuration settings I should look when addressing DoS attacks.

I eventually looked at two directives to help resolve my DoS attacks, MaxClients and TimeOut. For whatever reason, cPanel chooses a default value of 150 for MaxClients even though Apache's default is actually 256. Knowing that whenever the MaxClients were reached, my server wasn't accessible to the client, this was the first httpd directive I wanted to change. Raising this number seemed to delay the effects of the botnet overwhelming my server but it didn't quite solve the problem. Now instead of 150 bot requests being capable of stalling out my server, I could process 256 bot requests. All MaxClients did was invite more disrespectful people to a party that was already getting out of hand.

So I moved on to what was ultimately the solution in my case, I lowered the value given in the Timeout directive. The value configured for the Timeout directive is the amount of time the server will wait for certain events before failing a request. Apache gives the following security tips for how this can be configured to help prevent DoS attacks:

The TimeOut directive should be lowered on sites that are subject to DoS attacks. Setting this to as low as a few seconds may be appropriate. As TimeOut is currently used for several different operations, setting it to a low value introduces problems with long running CGI scripts.

For whatever reason, this directive by default is set for 300 seconds. While I can see a number of reasons why you might need five minutes to run a process before failing the request, that's a value I would be more comfortable to have on an intranet server (fully protected by firewall from the wild wild Web) than on an Internet server. So I lowered the TimeOut directive from 300 seconds to 10 seconds. After the value change, the average requests being processed at any given time dropped from 256 down to around 40.  Most Drupal sites are going to need more processing time than 10 seconds, so you'll find out as I did that this number needs to be higher than 10. So far, I have found a value of 45 for the TimeOut directives allows my site to keep server performance high while handling all those requests from the bots without killing legitimate Drupal related processes.

So in the end, if you find that the spambots are overwhelming your Drupal site and you have the ability to override the httpd configuration file, try lowering the value of your Timeout directive down to 45 or some other low number. Doing this first might just solve your your problem and prevent the need for you to write a long winded blog post about your experience.