Maven Spring tests fail when run together, but succeed individually

[MojoLA]

We're using Maven/Surefire and Spring/Hibernate transactional tests for a fairly large web application. There are 138 Test* classes, running a total of 1178 tests.

A straightforward "mvn test" will generate 82 errors, the nature of which tend to imply a corrupt application context:

Many of these:
IllegalTransactionStateException: Pre-bound JDBC Connection found!

A few of these:
NoSuchMethodError: org.hibernate.cache.CacheException.(Ljava/lang/ExceptionV

For every failed test, running the test class individually "mvn test -Dtest=TestFailingClass" succeeds. Indeed, using -Dtest=TestClass1,TestClass2,Etc." with various subsets of all my test classes succeeds or fails in different ways. For instance, running only the failing test classes succeeds with 0 errors.

With no apparent means to control the order of classes tested by Surefire, I have a difficult time determining which of my test classes seem to be leaving the context in a bad state.

What I'm looking for is a strategy to help determine what is happening in some kind of deterministic manner. I can certainly see the order of tests run from the log, but I cannot reproduce that order controllably.

And of course, suggestions for what to do about it ...

[Grzegorz Grzybek]

Hi

First - these "NoSuchMethodError" seem strange, they might occur when you run tests both from maven and e.g. Eclipse, but with different maven invocations? Hmm. Try using -X option for maven to see detailed information on classpaths used by Maven.

Second - try (using intuition and experience ) to use @DirtiesContext annotation for some SpringJUnit4ClassRunner managed tests (if you're using Spring 2.5 test annotations and JUnit4 of course!).

Third - do your test run in Eclipse / other IDE properly?

regards
Grzegorz Grzybek

[solid]

It sounds like you have 2 issues working against you here. 1) classloader differences and 2) test data integrity
Classloader issues often manifest as missing method exceptions. This is because you likely have 2 different versions of the same class in 2 different jar files. If that happens, your only saving grace will be that eclipse and maven are loading the classes from different jars so at least one of them will be right. I had that same problem here:
http://forum.springsource.org/showthread.php?t=52499. If you have 2 jars that contain the same class files, it is possible that eclipse and maven will load the class files from different and potentially wrong jars. Whenever I see things like the this: NoSuchMethodError: org.hibernate.cache.CacheException.(Ljava/lang/ExceptionV The first thing I do is check to verify that eclipse and maven are loading the files from the same jar. you can see what jar a class is loaded from by putting the following in your test (at the top):

try
           {
        	String classname="org.hibernate.cache.CacheException";   System.out.println("Jar="+Thread.currentThread().getContextClassLoader().loadClass(classname).getProtectionDomain().getCodeSource().getLocation());
           }
           catch (Exception e)
           {}

Additionally, if you notice that changing the order of tests or grouping them in differing subsets affects your results, it is possible that data is being affected by previous tests that causes subsequent tests to fail. Verify that none of your tests are failing because of data issues.

[MojoLA]

Indeed the problem comes from a corrupt Spring application context. One of the early tests is dirtying the context and causing the following tests to error out.

One difficulty is trying to control the order of the tests while discovering the test causing the trouble.

I was able to accomplish that by using Maven's log to find the order of the test classes run, then excluding tests one at a time from the top. Thirty-four tests in, I found the culprit.

It was a test called TestSpringContexts. Adding @DirtiesContext to these tests solve the problem, but it was also solved by removing calls to context.close() from the tests.

I wrote a blog post about the process here, but that's the gist of the matter: http://mojo.whiteoaks.com/2010/04/27...pts-the-suite/

Mojo