7/15/2014

Tomcat randomly shuts down without any log or trace - possible solution

Apache Tomcat server shut down

The problem

I ran into a problem couple weeks ago with my Tomcat. I deployed my webapp and it's been up and running for some time and one day it suddenly crashed. My customer reported that the application is not working, and he gets Service unavailable error from Apache.

I started browsing log files: system logs, Apache logs, Tomcat logs, my own application logs, I read everything, and there was no trace of any error, all I could find were these lines:

maj 19, 2014 9:57:50 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-9080
maj 19, 2014 9:57:51 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina

Nothing unusual, it looked like Tomcat was just being properly shut down, just like that. You may guess, that it wasn't me who did this :) After some googling, I found nothing more than "maybe you have System.exit() somewhere in your code?". Damn, no, I don't.

The culprit

A friend of mine looked it all up for me, and he agreed that nothing unusual happens except that Tomcat's process is being normally shut down. We started analyzing bash scripts I used for starting up the server (they're simplified for educational purposes):

The last line was the problem.

The solution

I used to log in to my VPS with PuTTY and execute my scripts, it usually worked fine. But from time to time it happened that after deploying the webapp, I didn't kill my script after it did his job. tail -f shows the logs continuously so the script doesn't finish it's execution automatically. When the ssh connection timed out while my script was still running, shell killed it WITH IT'S CHILD PROCESSES - a Tomcat server among others and that was the whole problem. Simple solution - DO NOT use tail -f in your bash scripts, or at least be aware of it and make sure you kill the script before closing the ssh connection.

9 comments:

  1. Hi man.
    That isn't about tail -f itself.
    All started processed accessing console are killed when ssh terminal is closed.
    You can use nohup tomcat.sh .... to start tomcat.

    ReplyDelete
    Replies
    1. Hi Alexander, thanks for the comment
      I know that "tail -f" was not direct cause, the whole idea was :) Using nohup seems to be a good solution in certain situations, but it may also leave many running unnecessary background processes

      Delete
  2. Thank you Alexander! I had exactly the same problem.

    ReplyDelete
  3. Hi Guys,

    I had also faced the same issue but I didn't found any error logs but came to know that java process keep on consuming 100% utilization this is occurred due to DB server where queries keep on going to que(un-executed). so that apache keep on establishing new connections with out output.

    ReplyDelete
  4. Awesome, I had same exact problem. Yeah, the last line tail -f catalina.out.

    ReplyDelete
  5. super...this was the actual root cause of our application going down.We could not have figured this out if this blog was not there.Thanks a top.
    Curious how did you find this.

    ReplyDelete
    Replies
    1. Thanks for your comment and sorry for the late reply. As far as I remember, after multiple times repeated shut downs I eventually just found a time relation between them and me doing the deployments :)

      Delete
  6. Had this same issue. Thanks for the posting this. Saved me a lot of time.

    ReplyDelete
  7. I have the same issue...running Tomcat 9.0.0.M15

    ReplyDelete