Spring + Hibernate/C3P0 leaving behind threads and pools

[alex314]

Hello!
I've been struggling for some time with a problem during load testing an app and it seems that for some strange reason a SessionFactory with an associated C3P0 datasource will not close properly, leaving several threads and a pool with open connections after each test. I wonder if anyone has come across this situation.

Even more specifically, this only happens if the C3P0 integration is done via a dataSource bean, but NOT if done via hibernate's c3p0 properties.

I managed to narrow the issue to this JUnit test (The dumpThreads method is not relevant to the problem, it's just a debugging aid)

package com.planetalia.webng.hibernate;

import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.Transaction;
import org.hibernate.cfg.AnnotationConfiguration;
import org.hibernate.cfg.Configuration;
import org.junit.Before;
import org.junit.Test;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.FileSystemXmlApplicationContext;

public class ConnectionTest{
	private static final int HIBERNATE = 1;
	private static final int SPRING= 2;
	
	private int mode = HIBERNATE;
	
	private void dumpThreads() {
		StringBuilder sb = new StringBuilder("THREAD COUNT IS  ");
		sb.append(Thread.activeCount());
		sb.append(": ");
		Thread[] liveThreads = new Thread[Thread.activeCount()];
		long id = Thread.currentThread().getId();
		Thread.enumerate(liveThreads);
		for (int i = 0; i < liveThreads.length; i++) {
			long tid = liveThreads[i].getId();
			sb.append(tid);
			sb.append((tid==id)?"*(":"(");
			StackTraceElement[] stack = liveThreads[i].getStackTrace();
			int lastElement = stack.length-1;
			if (lastElement >=0) {
				sb.append(stack[lastElement].getClassName());
				sb.append(".");
				sb.append(stack[lastElement].getMethodName());
				sb.append(")");
				
			}
			sb.append(liveThreads[i].getState());
			sb.append("\t");
		}
		System.out.println(sb);
	}
	
	@Before
	public void start() throws Exception {
		SessionFactory sessionFactory = null;
		switch (mode) {
			case HIBERNATE: {
				Configuration configuration = new AnnotationConfiguration();
				configuration.configure("hibernate.cfg.xml");
				sessionFactory = configuration.buildSessionFactory();
				break;
			}
			case SPRING: {
				ApplicationContext ctx = new FileSystemXmlApplicationContext("wiring.xml");
				sessionFactory = (SessionFactory)ctx.getBean("sessionFactory");
				break;
			}
		}
		Session session = sessionFactory.getCurrentSession();
		Transaction tx = session.beginTransaction();
		tx.commit();
		sessionFactory.close();
		
		dumpThreads();
	}
	
	@Test public void test1() throws Exception {Thread.sleep(500);}
	@Test public void test2() throws Exception {Thread.sleep(500);}
	@Test public void test3() throws Exception {Thread.sleep(500);}
	@Test public void test4() throws Exception {Thread.sleep(500);}
	@Test public void test5() throws Exception {Thread.sleep(500);}
	@Test public void test6() throws Exception {Thread.sleep(500);}
	@Test public void test7() throws Exception {Thread.sleep(500);}
	@Test public void test8() throws Exception {Thread.sleep(500);}
}

This test uses only Hibernate or Spring+Hibernate.

For the first mode, the following configuration file is used (hibernate.cfg.xml):

<hibernate-configuration>
	<session-factory>
		<property name='connection.driver_class'>com.mysql.jdbc.Driver</property>
		<property name='connection.url'>jdbc:mysql://localhost/test</property>
		<property name='connection.username'>test</property>
		<property name='connection.password'>test</property>
		<property name='dialect'>org.hibernate.dialect.MySQLInnoDBDialect</property>
		<property name='current_session_context_class'>thread</property>
		<property name='cache.provider_class'>org.hibernate.cache.NoCacheProvider</property>
		<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
		
		<property name="hibernate.c3p0.min_size">15</property>
		<property name="hibernate.c3p0.max_size">200</property>
		<property name="hibernate.c3p0.max_statements">100</property>
	</session-factory>
</hibernate-configuration>

And for the second (Spring+Hibernate) mode, the following file is used (wiring.xml):

  <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
		<property name="driverClass" value="com.mysql.jdbc.Driver"/>
		<property name="jdbcUrl" value="jdbc:mysql://localhost/test"/>
		<property name="user" value="test"/>
		<property name="password" value="test"/>
		
		<property name="minPoolSize" value="15"/>
		<property name="maxPoolSize" value="200"/>
		<property name="maxStatements" value="100"/>
	</bean>
	
	 <bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
    <property name="dataSource" ref="dataSource"/>		
    <property name="hibernateProperties">
      <value>
      <!-- 
      	hibernate.connection.driver_class = com.mysql.jdbc.Driver
      	hibernate.connection.url=jdbc:mysql://localhost/test
      	hibernate.connection.username=test
      	hibernate.connection.password=test
      	
      	hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider
      	hibernate.c3p0.min_size=15
      	hibernate.c3p0.max_size=200
      	hibernate.c3p0.max_statements=100
      	 -->
        hibernate.dialect = org.hibernate.dialect.MySQLInnoDBDialect
        hibernate.cache.provider_class = org.hibernate.cache.NoCacheProvider
        hibernate.show_sql = false
        hibernate.current_session_context_class=thread
      </value>
    </property>
  </bean>

If the test is run in the SPRING mode, each consecutive test leaves behind 4 threads and a fully built (and connected) pool, regardless of the sessionFactory.close() method. According to the dump, the threads are basically waiting on some monitor. But this only happens if the C3P0 initialization is performed using <property name="dataSource" ref="dataSource"/>. If I remove that property and instead initialize C3P0 using the commented hibernate properties, it works properly.

If the test is run in the HIBERNATE mode, there are no "leftover" threads and/or pools. However, if in this mode I "forget" to call sessionFactory.close(), I get exactly the same behaviour as the SPRING mode WITH sessionFactory.close();

(Versions are: Hibernate: 3, C3P0: 0.9.1, Spring: 2.5.6.SEC01 (2009-04-22) )

Any ideas or similar problems? Thanks!

[Marten Deinum]

Well I suggest the forum search... If you use spring to configure hibernate DON'T set the hibernate.current_session_context_class UNLESS you are using JTA. So remove it...

Next to that make sure that the hibernate.cfg.xml isn't loaded by the sessionfactory, because that will override the settings in spring.

[alex314]

Thanks; from what I've been reading, removing hibernate.current_session_context_class puts me on a slippery slope that forces me to use Spring for transaction management (otherwise I get the typical "No Hibernate session bound to thread"), something that I cannot do.

Although I've read in many places (incl. the docs) that I can in theory use the naked Hibernate API and manage the transactions myself, I haven't still figured out how or found any working example outside of JTA (i.e. plain vanilla JDBC transactions).

Seems to me that an easier way would be to drop the LocalSessionFactory bean altogether and do the job myself using only the Spring container and not the Hibernate integration classes.

[Marten Deinum]

If you use spring to wire your classes then why wouldn't you use springs tx management?! It is far easier and less error prone then managing tx yourself... I rarely do that anymore only in a few corner cases, although I tend to use TransactionTemplate to have more control in those situations.

[alex314]

Well, as I suspected, it turns out that even after using Spring Transaction Management and Hibernate integration according to the "canon", it is still leaking threads...
And this is no longer Spring 2, but the latest Spring (3.0.4) and Hibernate (3.5.5) releases ....

Here's the test class:

import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.junit.Before;
import org.junit.Test;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;

public class ConnectionTest{
	public static class TXClass {
		private SessionFactory sessionFactory;
		
		public SessionFactory getSessionFactory() {
			return sessionFactory;
		}

		public void setSessionFactory(SessionFactory sessionFactory) {
			this.sessionFactory = sessionFactory;
		}

		
		@Transactional(propagation=Propagation.REQUIRES_NEW)
		public void doJob() {
			Session session = sessionFactory.getCurrentSession();
			// Just a stupid test to make sure everything is setup properly
			session.createSQLQuery("select 1 from mysql.db").list();
		}
	}

	
	@Before
	public void start() throws Exception {
		ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext("spring-wiring.xml");
		TXClass tx =(TXClass)ctx.getBean("bean");
		tx.doJob();
		System.out.println("Thread count : "+Thread.activeCount());
	}
	
	@Test public void test1() throws Exception {Thread.sleep(500);}
	@Test public void test2() throws Exception {Thread.sleep(500);}
	@Test public void test3() throws Exception {Thread.sleep(500);}
	@Test public void test4() throws Exception {Thread.sleep(500);}
	@Test public void test5() throws Exception {Thread.sleep(500);}
	@Test public void test6() throws Exception {Thread.sleep(500);}
	@Test public void test7() throws Exception {Thread.sleep(500);}
	@Test public void test8() throws Exception {Thread.sleep(500);}

}

And here's the wiring:

<beans xmlns="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xmlns:tx="http://www.springframework.org/schema/tx"
     xsi:schemaLocation="
     http://www.springframework.org/schema/beans 
     http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
     http://www.springframework.org/schema/tx 
     http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">

 <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
    <property name="driverClass" value="com.mysql.jdbc.Driver"/>
    <property name="jdbcUrl" value="jdbc:mysql://localhost/test"/>
    <property name="user" value="test"/>
    <property name="password" value="test"/>
    <property name="minPoolSize" value="15"/>
    <property name="maxPoolSize" value="200"/>
    <property name="maxStatements" value="100"/>
 </bean>
	
 <bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean" >
    <property name="dataSource" ref="dataSource"/>		
    <property name="hibernateProperties">
      <value>
        hibernate.dialect = org.hibernate.dialect.MySQLInnoDBDialect
        hibernate.cache.provider_class = org.hibernate.cache.NoCacheProvider
        hibernate.show_sql = false
      </value>
    </property>
  </bean>

 <bean id="txManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
	<property name="sessionFactory" ref="sessionFactory"/>
 </bean>

 <tx:annotation-driven transaction-manager="txManager"/>

 <bean id="bean" class="ConnectionTest$TXClass"	>
	<property name="sessionFactory" ref="sessionFactory" />
 </bean>


</beans>

Result:

Thread count : 6
Thread count : 10
Thread count : 14
Thread count : 18
Thread count : 22
Thread count : 26
Thread count : 30
Thread count : 34

[Marten Deinum]

Because what you are doing is something you shouldn't be doing... You are creating a SessionFactory and ApplicationContext for each thread, which you shouldn't be doing...

If you really want to do that you also have to CLOSE the context, which you also aren't doing, which means your connectionpool isn't going to close properly... So basically after each test method you still have an active connection pool, active sessionfactory and active applicationcontext.... Also I'm not sure what you are trying to test, but that is probably me being stupid....

[alex314]

Exactly where am I doing that? "Each thread"? There is only one thread during each of the tests!

[alex314]

One user thread, I mean. Whatever threads Spring+Hibernate+c3P0 create, I don't know but they definitely should NOT leak to the next test.

[Marten Deinum]

Sigh... You don't close the context, so the connection pool doesn't get the signal to close, those threads remain (probaly some monitoring thread etc.). If you use plain hibernate and call close on the sessionfactory that is going to ripple through till the connection pool because well it is managed by hibernate. Now you have a applicationcontext, which you don't properly close and hence those threads hang around...

If you don't stop the threads, or signal the mechanism that should stop those threads those are going to stay around...

You can see for yourself if you do a thread dump.

[alex314]

Marten, I would close() the ApplicationContext if anywhere in the Spring reference documentation it was mentioned that the application context needed closing AND if I could call that method, which I can't since it is not part of the ApplicationContext interface. Of course I will call that method in one way or another (casting, reflection, whatever...), but that's a different issue.

None of the examples in the Spring doc bothers to do it (e.g: pp 49,107,132-137,146,242, etc... of the Spring 3 PDF), not even those dealing with tx examples, and the only place that talks about shutting gracefully the IoC container in nonWeb apps suggests registerShutdownHook() which doesn't work in my case anyway since AFAIK the JVM isn't restarted between test runs in the IDE I'm using.

As an epilogue, I must also say that while I appreciate your answers, I don't appreciate the "Sigh". You may or may not reply to me, that's your choice, but if you do, I expect some respect, just as I respect you and don't "Sigh" on the misguided statements you've made. Having said that, I repeat that I appreciate your effort in helping me with this issue.

Cheers

Alex