Inconsistency sucks (or 'How Apache ticked me off last weekend')

As some of you may know, I own and operate a small web solutions company (hosting, design, custom programming, etc.) and it is primarily powered by the Apache web server. Fair warning, what follows is techie crap and may make your eyes glaze over if you're not so inclined.

Best practice in system administration is to pare down the optional stuff running on your systems so as to free up as many of the resources as possible for the tasks that need to be performed. Uninstalling (or at least not running) services and daemons that aren't needed is a good first step, which I certainly did. My second step was to whittle on the Apache config itself by commenting out modules that I am not (and will never) use.

The httpd.conf file is loaded with stanzas similar to the following:

<IfModule mod_mime_magic.c>
    MIMEMagicFile /usr/local/apache/conf/magic

which basically say, "do this thing if that module is available" because that "thing" - whatever it is - depends on the module being checked for and won't work correctly if it's not there.

So I go about commenting out lines that load modules like mod_negotiation, mod_autoindex, and some of the miscellaneous auth modules. I scheduled a restart for the middle of the night and went on with my day. At 4:00am, my monitoring service started sending me alerts that my sites on that box were down.

Sure enough, Apache wouldn't restart nicely. I knew that I couldn't have screwed up the syntax because I didn't actually change the content of any lines - just added a hash mark in front of them.

A half an hour later, I discovered the issue. Turns out there were a bunch of directives that rely on modules that weren't wrapped in the ubiquitous IfModule stanzas.

An hour after the first alert went out, I stumbled back upstairs and unsuccessfully tried to go back to sleep. So, due to that inconsistency in the default configuration file of the world's leading web server, I not only lost 2.5 hours of sleep but my customer's site was down for an hour.