CannotCreateTransactionException non-forked integration testing "Session is closed!"

[javapda]

We are converting to using spring-3.1.0.RC2 and hibernate-4.0.0.CR7

We have several hundred integration tests. When they are run in a forked mode (<forkMode>always</forkMode>, which takes much longer) all tests pass. When they are run in non-forked mode (much quicker) they begin to fail after the 28th integration test class is run.

Each of the failing integration tests passes independently.

Here is a stack trace (test result) from the first of the failures:

-------------------------------------------------------------------------------
Test set: com.gs.core.services.impl.ServiceImplIntegrationTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.036 sec <<< FAILURE!
testGetDiseaseByClinicalCode(com.gs.core.services.impl.ServiceImplIntegrationTest)  Time elapsed: 0.018 sec  <<< ERROR!
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.SessionException: Session is closed!
	at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:428)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
	at org.springframework.test.context.transaction.TransactionalTestExecutionListener$TransactionContext.startTransaction(TransactionalTestExecutionListener.java:513)
	at org.springframework.test.context.transaction.TransactionalTestExecutionListener.startNewTransaction(TransactionalTestExecutionListener.java:271)
	at org.springframework.test.context.transaction.TransactionalTestExecutionListener.beforeTestMethod(TransactionalTestExecutionListener.java:164)
	at org.springframework.test.context.TestContextManager.beforeTestMethod(TestContextManager.java:358)
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:73)
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:231)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:174)
	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
	at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
	at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)
	at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
	at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Caused by: org.hibernate.SessionException: Session is closed!
	at org.hibernate.internal.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:126)
	at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:449)
	at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:342)
	... 30 more

Q: Why would forking of the integration tests pass (a new jvm for each test) versus non-forked failing?

Q: Any suggestions on how best to track down the issue?

Thanks in advance. :-)

[javapda]

The test methods of the integration tests make use of the @Transactional annotation. Using @Transactional seems to work fine within individual integration tests.

If, we use a manual approach to the behavior is altered in that the underlying SessionHolder object seems to be cleared of its session upon completion of most persist operations.

    private Session session;
    protected void startTransactionSynchronization() {
        stopTransactionSynchronization();
        session = getCurrentSession();
        session.setFlushMode(FlushMode.AUTO);
        assertTrue(session.isOpen());
        TransactionSynchronizationManager.bindResource(sessionFactory,
                new SessionHolder(session));
    }
    protected void stopTransactionSynchronization() {
        if (TransactionSynchronizationManager.hasResource(sessionFactory)) {
            TransactionSynchronizationManager.unbindResource(sessionFactory);
        }
          if (session == null) {
                return;
            }
       SessionFactoryUtils.closeSession(session);
    }
    private Session getCurrentSession() {
        if(session==null) {
            session = sessionFactory.openSession();
        } else if (!session.isOpen()) {
            session = sessionFactory.openSession();
        }
        return session;
    }

[javapda]

The root cause seems to be related to intermixing the spring Transaction Manager (affected by the @Transactional annotation) with tests that run using a manually choreographed transaction (see startTransaction() above).

Here's how it was tracked down. We run automated tests from jenkins (hudson) via a maven test goal. The actually tests run (and fork mode) are controlled by settings in the configuration setting of the maven-surefire-plugin. (see plugin stanza from pom.xml below)

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>${maven-surefire-plugin.version}</version>
				<configuration>
					<skip>true</skip>
				</configuration>
				<executions>
					<execution>
						<id>all-tests</id>
						<phase>test</phase>
						<goals>
							<goal>test</goal>
						</goals>
						<configuration>
							<forkMode>never</forkMode> 
							<skip>false</skip>
							<includes>
								<include>**/*Test.java</include>
							</includes>
							<excludes>
								<exclude>**/*DataTest.java</exclude>
							</excludes>
							<argLine>-Xms512m -Xmx2024m -XX:PermSize=64M -XX:MaxPermSize=256M
							</argLine>
						</configuration>
					</execution>

				</executions>

			</plugin>

Notice the <include> tags. Here you can use ant-like patterns to constrain which tests are run. We began by narrowing the population of integration tests run.

								<include>**/myspecifictests/*Test.java</include>
								<include>**/otherinteresting/*Test.java</include>

It was mentioned earlier that the test failures occurred later in the test run but individual tests ran successfully. Soon it was discovered that some of the tests run prior to test failure included manual transaction handling by binding the sessionFactory to a session holder object in the TransactionSynchronizationManager (org.springframework.transaction.support.Transacti onSynchronizationManager). As the javadoc states "Central helper that manages resources and transaction synchronizations per thread. To be used by resource management code but not by typical application code." We were obviously not adhering to this advice.

It seems that if you really need to do some discrete hibernate session activity you could simply work directly with Session objects explicitly generated from a session factory, then perform the operations on this session, without the express involvement of the spring transaction manager.