How to automatically recover Tomcat from crashes

Tomcat occasionally crashes if you do frequent hot-deploys or if you are running it on a machine with low memory. Every time tomcat crashes someone has to manually restart it, so I wrote a script which automatically detects that tomcat has crashed and restarts it.

Here’s the pseudo logic:

every few minutes {
  check tomcat status;

  if (status is "not running") {
    start tomcat;
  }
}

Here’s a shell script to implement the above logic. It assumes that you are running on a unix/linux system and have /etc/init.d/tomcat* script setup to manage tomcat.

Adjust the path to “/etc/init.d/tomcat” in the script below to reflect the correct path on your computer. Sometimes it is called /etc/init.d/tomcat5 or /etc/init.d/tomcat6 depending on your tomcat version. Also make sure that the message “Tomcat Servlet Container is not running.” matches with the message that you get when you run the script when tomcat is stopped.

#! /bin/sh
SERVICE=/etc/init.d/tomcat
STOPPED_MESSAGE="Tomcat Servlet Container is not running."

if [ "`$SERVICE status`" == "$STOPPED_MESSAGE"];
then
{
  $SERVICE start
}
fi

To run the script every 10 minutes:

1. Save the above script to “/root/bin/recover-tomcat.sh”.

2. Add execute permission:

chmod +x /root/bin/recover-tomcat.sh

3. Add this to root’s crontab, type the following as root:

crontab -e

4. Add the following lines to the crontab:

# monitor tomcat every 10 minutes
*/10 * * * * /root/bin/recover-tomcat.sh

What if I don’t have /etc/init.d/tomcat* script on my computer?

Tomcat creates a pid file, typically in the TOMCAT_HOME/bin directory. This file contains the process id of the tomcat process running on the machine. The pseudo logic in that case would be:

if (the PID file does not exist) {
  // conclude that tomcat is not running
  start tomcat
}
else {
  read the process id from the PID file
  if (no process that id is running) {
    // conclude that tomcat has crashed
    start tomcat
  }
}

You can implement the above logic as follows. The following is experimental and is merely a suggested way, test it on your computer before using it.

# adjust this to reflect tomcat home on your computer
TOMCAT_HOME=/opt/tomcat5

if [ -f $TOMCAT_HOME/bin/tomcat.pid ]
then
  echo "PID file exists"
  pid="`cat $TOMCAT_HOME/bin/tomcat.pid`"
  if [ "X`ps -p $pid | awk '{print $1}' | tail -1`" = "X"]
  then
    echo "Tomcat is running"
  else
    echo "Tomcat had crashed"
    $TOMCAT_HOME/bin/startup.sh
  fi
else
  echo "PID file does not exist. Restarting..."
  $TOMCAT_HOME/bin/startup.sh
fi

Why would tomcat crash?

The most common reason is low memory. For example, if you have allocated 1024MB of max memory to tomcat and enough memory is not available on that machine. Other reasons may involve repeated hot-deploys causing memory leaks, rare JVM bugs causing the JVM to crash.

Related posts:

  1. How to rotate Tomcat catalina.out
  2. Getting started with Nexus Maven Repo Manager
  3. Java 7: New Feature – automatically close Files and resources in try-catch-finally
  4. Cache Java webapps with Squid Reverse Proxy
  5. How to automate project versioning and release with Maven

6 comments to How to automatically recover Tomcat from crashes

  • sapporo

    Instead of re-inventing the wheel, you could also use Tanuki Java Service Wrapper (open source/commercial), Yet Another Java Service Wrapper (open source), or Apache Commons Daemon (procrun on Windows or jsvc on Linux, both open source).

  • Tom

    Thanks

    You can also test like this:

    return_string = $(jps | grep Bootstrap)
    if [-n "$return_string" ]; then ….

  • I understand the logic of wanting to use a script like this, but I do wonder if in doing so, you end up treating the symptoms rather than addressing the cause of the problem?

    From a business point of view, if the technology I am using in production keeps crashing I would want to know how often this occurs, what I am doing to cause the crashes and whether these two in combination warrant me investigating the use of alternate technologies? Is this script something that you have/would use in production, or is it something you think that is useful for individual developers?

    What are your thoughts about how this could be integrated into a wider monitoring solution so that technical issues are not masked from the business?

  • vineet

    Good points, Rob. I use Tomcat in dev, test and production environments. I use this script in dev and test, but not in production – simply because my production tomcat has never crashed. I suspect the reason for more frequent crashes on is lesser memory on dev and test machines and high frequency of hot deploys.

    I use a version of the script which notifies me when it restarts tomcat.

    SERVICE=/etc/init.d/tomcat
    STOPPED_MESSAGE="Tomcat Servlet Container is not running."

    if [ "`$SERVICE status`" == "$STOPPED_MESSAGE"];
    then
    {
    $SERVICE start
    mail -s "Tomcat was down on `hostname`, restarting..." admin@email.address
    }
    fi

    While this takes care of the symptom right away, it also alerts you so that you can investigate the root cause.

  • ferganil

    Thank Vineet,

    I’m new at linux. And I get this error when execute script:

    [: 10: missing ]

    What is the missing? I use tomcat6 and Ubuntu. This my script:

    #!/bin/sh
    SERVICE=/etc/init.d/tomcat6
    STOPPED_MESSAGE=”Tomcat servlet engine is not running.”

    if [ "`$SERVICE status`" == "$STOPPED_MESSAGE"];
    then
    {
    $SERVICE start
    }
    fi

    Thanks,
    KS.

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Get Adobe Flash playerPlugin by wpburn.com wordpress themes