What diagnostic action to take when Maven hangs ?

[stephaneeybert]

Hi,

I have a Maven build that hangs and I wonder what to do to get some feedback like a message or stack trace or dump..

I'm stuck trying to run some integration tests http://forum.springsource.org/showth...d=1#post426888

Thanks

[krimple]

If you're on linux / mac:

1. in the hanging shell, hit CTRL-Z - this will suspend the program and give you a pid
2. type bg [ENTER] - this will send the program to the background
3. Do a kill -3 of the process - it will dump a large amount of text - make sure your command line terminal history window is long enough to capture it all - will be several thousand lines of text
(alternative to #3 - you can try jstack with the pid, it is a Java stack trace generator)
4. You can see what threads are waiting on other threads from there (maybe, sometimes it's a tight CPU loop)

One best practice is to do several thread dumps in a row - and compare where the threads are hanging. Chances are with Maven you only have a few threads running at most since it is a build tool, but if you are running Spring integration tests, then the Maven tool may have fired up Spring and spawned threads in that VM.

Another option, which will run very slowly, but will work, is to run mvnDebug instead of mvn when you run it. It should sit there paused, waiting for you to connect. You could try connecting with jvisualvm or jconsole and watching the heap, cpu, etc., from there. You'd need to make something stop suspending it, so you can always hook up your IDE, like Eclipse, to it using remote debugging on the same JVM port that it exposes.

Ken

[stephaneeybert]

Hi Ken,

First of all, let me say one thing: Lucky were Ken's co-workers.

I ran your commands:

-------------------------------------------------------
T E S T S
-------------------------------------------------------
^Z
[1]+ Stopped mvn clean test
stephane@stephane-ThinkPad-X60:learnintouch> bg
[1]+ mvn clean test &
stephane@stephane-ThinkPad-X60:learnintouch>

But it didn't show a process number.

Here is the process number from another terminal:

stephane 4798 1749 99 00:32 pts/1 00:02:49 /home/stephane/programs/jdk1.6.0_29/bin/java -class
stephane 4822 4798 0 00:33 pts/1 00:00:00 /bin/sh -c cd /home/stephane/dev/java/projects/spri
stephane 4824 4822 95 00:33 pts/1 00:00:22 /home/stephane/programs/jdk1.6.0_29/jre/bin/java -j
stephane 4837 1759 0 00:34 pts/2 00:00:00 ps -ef

And the kill command showed:

stephane@stephane-ThinkPad-X60:learnintouch> kill -3 4824
stephane@stephane-ThinkPad-X60:learnintouch> 2012-10-10 00:36:09
Full thread dump Java HotSpot(TM) Server VM (20.4-b02 mixed mode):
"Low Memory Detector" daemon prio=10 tid=0x08fce400 nid=0x12e3 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x08fcc800 nid=0x12e2 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x08fca800 nid=0x12e1 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x08fc9000 nid=0x12e0 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x08fb7400 nid=0x12df in Object.wait() [0x7fd77000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x84455fa8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:118)
- locked <0x84455fa8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finali zer.java:159)
"Reference Handler" daemon prio=10 tid=0x08fb5c00 nid=0x12de in Object.wait() [0x7fdc8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x84455f88> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Refer ence.java:116)
- locked <0x84455f88> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x08f36400 nid=0x12da runnable [0xb6976000]
java.lang.Thread.State: RUNNABLE
at java.lang.ClassLoader.findLoadedClass0(Native Method)
at java.lang.ClassLoader.findLoadedClass(ClassLoader. java:949)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 91)
- locked <0x84444dd8> (a sun.misc.Launcher$AppClassLoader)
at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:301)
- locked <0x84444dd8> (a sun.misc.Launcher$AppClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 47)
at org.springframework.core.type.classreading.Recursi veAnnotationAttributesVisitor.visitEnd(AnnotationA ttributesReadingVisitor.java:167)
at org.springframework.asm.ClassReader.a(Unknown Source)
at org.springframework.asm.ClassReader.accept(Unknown Source)
at org.springframework.asm.ClassReader.accept(Unknown Source)
at org.springframework.core.type.classreading.SimpleM etadataReader.<init>(SimpleMetadataReader.java:54)

I can see some threads are waiting for another one.

Sorry I haven't done that before so I'm not the best hunter in the wild.

Tomorrow I shall try to have a harder look at it and use the mvn debugger as well.

Thank you.

[stephaneeybert]

I can run the debugger all right with the command:

stephane@stephane-ThinkPad-X60:learnintouch> mvnDebug -DforkMode=never test -Dtest=AddressIntegrationTest
Preparing to Execute Maven in Debug Mode
Listening for transport dt_socket at address: 8000

I can run the debug configuration in Eclipse and it attaches to the remote debugger.

But then nothing much happens within Eclipse and I don't know how to step in the source code.

I attached a screenshot of the Eclipse pane.

Screenshot at 2012-10-17 13:35:13.jpg

[krimple]

Stephane,

Did you set a breakpoint in your test?

You can right-click on the thread in the debugger and suspend it - I'd start with the main thread. My theory (based on absolutely nothing but gut instinct) is that you have a Spring config that is loading and takes forever, perhaps running out of memory on the way up. If you right-click on the main thread and pause it, likely you'll be in some area of the configuration.

[stephaneeybert]

Hi Ken,

I did pause the thread and set a break point in my first test but it was never reached.

I suppose to see what is going on with the Spring Roo configuration I would need to have access to its source code for the debugger...

[stephaneeybert]

I have 3 Gb of ram on a Linux Mint box. Strange that this would not be enough for Maven to build a Spring Roo app.

EDIT

Maybe there is some way to tell Spring to be slow and memory conservative than fast and memory intensive...

[krimple]

Hi Ken,

I did pause the thread and set a break point in my first test but it was never reached.

I suppose to see what is going on with the Spring Roo configuration I would need to have access to its source code for the debugger...

huh... you should be able to download the source - Spring distributes a Roo Java source jar on Maven central so if you get a thread to pause you should be able to double click on the stack frame of interest. Same with the Spring Framework and many others if it asks to download the source just click to make it do so.

Clearly there is something you are triggering that should be looked at.

[stephaneeybert]

If I understood the idea of Roo being a tool and not a framework, the Roo source code is not needed in the debugging process as Roo is simply absent from the application at run time.

Maven has downloaded the source code of Spring and this source code is where I should look to see where it hangs.

The thing is I don't know where to put any break point.. not sure it is even in Spring I should look into.

Sorry for my modest ability on this call.

[krimple]

Aargh. Sorry! I was up late prepping for an add-on presentation at SpringOne and you did ask me a question about source code.... My mind is not so clear at that time of night.

The Spring framework source is definitely available. So if it is happening in the container startup, you should be able to see. The same with JPA, Hibernate, etc.

Can you suspend it? Any info on what methods/classes it stops on?

Don't worry about a breakpoint- just right click on the threads in the debugger and click the option to suspend the thread. I would start with the main one.

Ken

If I understood the idea of Roo being a tool and not a framework, the Roo source code is not needed in the debugging process as Roo is simply absent from the application at run time.

Maven has downloaded the source code of Spring and this source code is where I should look to see where it hangs.

The thing is I don't know where to put any break point.. not sure it is even in Spring I should look into.

Sorry for my modest ability on this call.