Today, my website is down.
|
1 2 |
service httpd status httpd dead but pid file exists |
This problem persists even I restart httpd. I have to look into the apache log to investigate what happened. Where is the apache log? I know the location of the error/access log for the website. They can be specified in the virtual host section of httpd.conf. If you are using sentora, the logs are in /var/sentora/logs/domains/zadmin/. But these log files are apparently not what I’m looking for. This error log I’m looking for is not specific to a website but a global one. After searching, I find the apache error log is /var/log/httpd/error_log. There is an error in this log file,
|
1 |
No space left on device: Couldn't create accept lock |
It seems my server has run out of HDD space. But using the “df” command, I find there is still much space available to use. So what causes httpd dead? In fact, the httpd died of exhausting of semaphores. An account can have a limited semaphores to use. The following command can be used to find the limit:
|
1 2 3 |
sysctl kernel.sem kernel.sem = 250 32000 32 128 |
You can check how many semaphores are being used by apache:
|
1 |
ipcs -s | grep apache |
After using up all available semaphores, httpd will die and cannot be restarted. Knowing the root cause, the solution to fix the problem is easy: delete all semaphores that apache is using:
|
1 2 3 4 5 |
sems=$(ipcs -s | grep apache | awk '{print $2}') for i in $sems do ipcrm -s $i done |
Now restart httpd, apache will work normally.
