What is the showstopper?

[vw729]

We run into a "not in service" problem yesterday on one of our Spring+Hibernate applications. This application has been running for over two years without any performance issues. There isn't any Hibernate configuration changes. And it only occurs on the deployment box, but not our development box. The site hosting firm said that there is far enough memory for it.

We inserted log code and turned on the log on Hibernate. The log messages shall be the followings:

DEBUG com.abc.xyz.web.MyController.viewHandler:42 - Start fetching ... 
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.getTransaction:319 - Using transaction object [org.springframework.orm.hibernate3.HibernateTransactionManager$HibernateTransactionObject@6a4268]
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.getTransaction:347 - Creating new transaction with name [com.abc.xyz.data.AdEntryDao.getById]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin:430 - Opened new Session [org.hibernate.impl.SessionImpl@2de41d] for Hibernate transaction
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin:441 - Preparing JDBC Connection of Hibernate Session [org.hibernate.impl.SessionImpl@2de41d]
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin:504 - Exposing Hibernate transaction as JDBC transaction [jdbc:postgresql://localhost:5432/myApp, UserName=postgre, PostgreSQL Native Driver]
DEBUG org.springframework.orm.hibernate3.HibernateTemplate.execute:365 - Found thread-bound Session for HibernateTemplate
DEBUG org.springframework.orm.hibernate3.HibernateTemplate.execute:389 - Not closing pre-bound Hibernate Session after HibernateTemplate
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.triggerBeforeCommit:833 - Triggering beforeCommit synchronization
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.triggerBeforeCompletion:846 - Triggering beforeCompletion synchronization
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.processCommit:660 - Initiating transaction commit
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit:554 - Committing Hibernate transaction on Session [org.hibernate.impl.SessionImpl@2de41d]
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.triggerAfterCommit:859 - Triggering afterCommit synchronization
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.triggerAfterCompletion:875 - Triggering afterCompletion synchronization
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion:633 - Closing Hibernate Session [org.hibernate.impl.SessionImpl@2de41d] after transaction
DEBUG org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession:772 - Closing Hibernate Session
DEBUG com.abc.xyz.web.MyController.viewHandler:47 - End of fetching ... 78

After reloading the Tomcat server for a whole, the same fetch won't respond. And the log messages are the following instead:

DEBUG com.abc.xyz.web.MyController.viewHandler:42 - Start fetching ... 
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.getTransaction:319 - Using transaction object [org.springframework.orm.hibernate3.HibernateTransactionManager$HibernateTransactionObject@6a4268]
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.getTransaction:347 - Creating new transaction with name [com.abc.xyz.data.AdEntryDao.getById]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin:430 - Opened new Session [org.hibernate.impl.SessionImpl@2de41d] for Hibernate transaction
DEBUG org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin:441 - Preparing JDBC Connection of Hibernate Session [org.hibernate.impl.SessionImpl@2de41d]

It seems fail to establish the JDBC connection of a Hibernation session. How to fix this issue if that is the cause indeed?

Also, I see the following related messages when I shut down the TC:

...
DEBUG org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(772) - Closing Hibernate Session
DEBUG com.abc.xyz.web.MyController.getExceptionHandler(616) - trying to find handler for exception class [java.lang.ThreadDeath]
...

Thanks very much for your inputs in advance.

[vw729]

Has anyone has encounter this problem before?

[Andreas Senft]

Actually I have not encountered this problem before. However, the log message about the ThreadDeath error indicates the usage of the deprecated Thread.stop() method.
This method is inherently unsafe as it can leave locks open. If this is what happens, than that could be the explanation for the "hanging" of the application: Waiting for locks never to be released again. I suggest searching for occurrences of Thread.stop() and replacing it with more safe means.

Regards,
Andreas

[dejanp]

I'd say you have a connectivity problems with your db server. Hibernate is not trying to get a db connection and it's stuck at that point.

[vw729]

Thanks both for your inputs.

All those messages are from Hibernate or Spring+Hibernate. I won't touch the code for this reason.

If it is caused by a DB connectivity problem, how to explain that it is fine right after I reload the TC. Can a memory usage cause a DB connectivity issue? (It is known the TC has a memory leak issue due to a problem in current JVM)

[Andreas Senft]

Is com.abc.xyz.web.MyController one of your classes? Could it be that shutdown doesn't work as it should and Tomcat just stops the thread (just an idea)?
I would try investigating the shutdown sequence more thoroughly. Maybe that reveals some more information.

Regards,
Andreas

[vw729]

Is com.abc.xyz.web.MyController one of your classes? Could it be that shutdown doesn't work as it should and Tomcat just stops the thread (just an idea)?
I would try investigating the shutdown sequence more thoroughly. Maybe that reveals some more information.

Regards,
Andreas

What I can do with my controller class in this regard? What it does is to call a method in the service layer and to call A DAO Hibernate implementation.

How to shutdown TC in sequence?

[Andreas Senft]

I think you cannot influence the shutdown sequence yourself. I just tried to find an explanation. As of your controller: I was just curious, as in your debug output it has been mentioned.
Can you debug your application to find outh whether really a ThreadDeath error occurs? Maybe try setting an exception breakpoint.