It is a common issue for webmasters that a service such as httpd/mysqld crashes now and then. You can not monitor the status of the services manually. Firstly, I solved this problem by writing a monitoring script. The script uses the command like “service httpd status” or “service mysqld status” to check if the service is running, and restarts the service if it is dead. I added an entry in crontab to run the monitoring script every 10 minutes. Things went well until one day I found the service was down and never restarted by the monitoring script. I found the root cause was the crond service was terminated as well for some unknown reason so the monitoring script did not run any longer. This accident made me realize that I should not rely on crond to restart services when they are down. Instead, I should use an init system to do the job. At last, I chose runit to do it. I should have used runit to monitor the status of httpd or mysqld. But since I have written the monitoring script and configured the crontab, I decide to monitor the crond service alone using runit.
install runit in CentOS:
curl -s https://packagecloud.io/install/repositories/imeyer/runit/script.rpm.sh | bash
yum install runit
After installation, runit is actually running in the system:
ps -ef | grep runsvdir
create a directory:
create a sub-directory crond
create a script “run” in that sub-directory, which contains the following content:
change the mode of the script so it can be executed:
chmod +x /etc/sv/crond/run
create a symbolic link in /etc/service/ to point to /etc/sv/crond
ln -s /etc/sv/crond /etc/service/crond
All is done! After several seconds, you will find the dead crond is recovered by the runit.