how to restart a service when it is down?

It is a common issue for webmasters that a service such as httpd/mysqld crashes now and then. You can not monitor the status of the services manually. Firstly, I solved this problem by writing a monitoring script. The script uses the command like “service httpd status” or “service mysqld status” to check if the service is running, and restarts the service if it is dead. I added an entry in crontab to run the monitoring script every 10 minutes. Things went well until one day I found the service was down and never restarted by the monitoring script. I found the root cause was the crond service was terminated as well for some unknown reason so the monitoring script did not run any longer.  This accident made me realize that I should not rely on crond to restart services when they are down. Instead, I should use an init system to do the job. At last, I chose runit to do it. I should have used runit to monitor the status of httpd or mysqld. But since I have written the monitoring script and configured the crontab, I decide to monitor the crond service alone using runit.

install runit in CentOS:

After installation, runit is actually running in the system:

create a directory:

mkdir /etc/sv

create a sub-directory crond

mkdir /etc/sv/crond

create a script “run” in that sub-directory, which contains the following content:

#!/bin/sh
exec /usr/sbin/crond

change the mode of the script so it can be executed:

chmod +x /etc/sv/crond/run

 

create a symbolic link in /etc/service/ to point to /etc/sv/crond

ln -s /etc/sv/crond /etc/service/crond

All is done! After several seconds, you will find the dead crond is recovered by the runit.

 

 

 

 

 

 

 

Posted in tips of hosting