OK, our high load server went down this evening and Apache was spitting out these “Too many open file” error messages. Then asking me to increase the file descriptors.
Note: We have over 7000 virtualhosts on this server, so that’s a lot of logfiles open! Before anyone says i’m crazy, I’m not. The server load is absolutely fine and it’s not even breaking a sweat. IO times are fine also.
Some background on this issue:
Cpanel do their whole statistics package the WRONG WAY. They log to numerous files simultaneously and as a result tons and tons of files are opened simultaneously also. Apache actually recommend that on high load servers you log everything to a single file, and then run their split-logfile utility to seperate the combined log into individual domain logs. You can then process these files with webalizer or any other statistics package accordingly. This keeps your number of open files down and everything running happily. I submitted this as a question to CPanel and as of yet have had no response. Typical.
So anyway, after spending several hours trawling the internet looking for a solution to this problem, I managed to resolve it using a couple of different suggested methods thrown together. It seems that Apache takes it’s hardcoded descriptor limit from system files, and the only way to resolve the issue is to recompile.
Modify these 2 files:
Find the value for FD_SETSIZE and increase it. I doubled this value, just to keep things simple. Do it for both files. Save files.
Next, login to WHM and select ‘Apache Update’. UNTICK “Raise FD_SETSIZE to 16384 (System Wide)” and then select all relevant modules/options that you need for installation. Compile Apache and your problem should be solved.
ulimit -n 2048 echo Ulimit set to 2048
So far things are now working fine for us, though Apache has over half a million open files it’s still running ok.