Unable to stop DefaultMessageListenerContainer

[john_gregg]

All,

I'm using Spring 2.5.6 sec02. I recently added a unit test that depends on the spring container. Among the beans is a DefaultMessageListenerContainer. However when the test runs in Ant and tries to exit, it hangs forever with this stack trace in the thread dump:

Oracle JRockit(R) R28.2.3-13-149708-1.5.0_34-20120327-1523-windows-ia32

"Main Thread" id=1 idx=0x4 tid=6248 prio=5 alive, waiting, native_blocked
    -- Waiting for notification on: java/lang/Object@0x11DFF950[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at org/springframework/jms/listener/DefaultMessageListenerContainer.doShutdown(DefaultMessageListenerContainer.java:
489)
    ^-- Lock released while waiting: java/lang/Object@0x11DFF950[fat lock]
    at org/springframework/jms/listener/AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:211)
    at org/springframework/jms/listener/AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:149)
    at org/springframework/beans/factory/support/DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:151)
    at org/springframework/beans/factory/support/DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.j
ava:487)
    at org/springframework/beans/factory/support/DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegis
try.java:462)
    at org/springframework/beans/factory/support/DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegi
stry.java:430)
    ^-- Holding lock: java/util/LinkedHashMap@0x109F7078[biased lock]
    at org/springframework/context/support/AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
    at org/springframework/context/support/AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
    at org/springframework/context/support/AbstractApplicationContext.close(AbstractApplicationContext.java:794)

Line 489 is this.lifecycleMonitor.wait();

Interestingly, AbstractApplicationContext.close doesn't get called when I run the test in IntelliJ 11 and the test exits normally. Perhaps it's disabling the shutdown hook somehow. I haven't used Spring JMS before but have done a lot of non-Spring JMS and my config looks pretty vanilla to me.

Any hints on what's going on?

Thanks

Here is my config:

 <bean id="customerListenerContainer"
        class="org.springframework.jms.listener.DefaultMessageListenerContainer">
    <property name="concurrentConsumers" value="1" />
    <!-- never use more than 1 consumer for a topic -->
    <property name="maxConcurrentConsumers" value="1" />
    <property name="sessionTransacted" value="true" />
    <property name="connectionFactory" ref="myConnectionFactory" />
    <property name="destination" ref="customerDestination" />
    <property name="messageListener" ref="customerListener" />
    <property name="pubSubDomain" value="true" />
    <property name="clientId" value="cps" />
    <property name="durableSubscriptionName" value="customerConsumer" />
    <property name="subscriptionDurable" value="true" />
  </bean>

[Gary Russell]

This implies one or more of the listener invokers is hung up.

Do you see any threads in the thread dump with names like "DefaultMessageListenerContainer-0", "...-1" etc

If so, what are they doing?

[john_gregg]

Unfortunately, no. I've done some remote debugging. The container was started on the main thread and the shutdown call is also happening on the main thread. There is 1 active invoker (an instance of AsyncMessageListenerInvoker) in the container. There is no other thread that appears to be doing anything, however. I thought I might see a thread blocked on JMS receive() or something, but I don't. I also don't see any threads with the names you suggested, or any like them.

One other thing that might be relevant is that I didn't want to have an app server running for my tests to pass, so I set lookupOnStartup=false for the queue and cxn factory. That solved my original testing problem (not being able to do a JNDI lookup because the server wasn't running.) Could it also have some effect that prevents the DefaultMessageListenerContainer from starting correctly and therefore from stopping correctly also?

thanks

[Gary Russell]

Can you provide the full thread dump? (Perhaps as a .zip attachment).

[john_gregg]

threadump.zip

Thanks

[Gary Russell]

Strange; by

There is 1 active invoker...

I assume you mean the counter is 1; which shouldn't happen if there's no thread running in it.

			finally {
				synchronized (lifecycleMonitor) {
					decreaseActiveInvokerCount();
					lifecycleMonitor.notifyAll();
				}
...

I see you named your container so the invoker threads should be called customerListenerContainer-x instead of the default name I mentioned.

But, there's still not one of those.

That's a pretty ancient version of Spring - is there any chance you can try with 3.0.7 or 3.1.1?

What if you run with TRACE level logging - do you see anything interesting?

[john_gregg]

Switched to 3.0.7 temporarily. The good news is that the activeInvokerCount is 0, so there is no blocking wait() call. The actual bean initialization is failing because of this:

javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file:  java.naming.factory.initial

That's not too surprising. It isn't a showstopper, however. The stack trace is being printed but not rethrown, so my tests can proceed. I assume the activeInvokerCount is 0 because of the JNDI lookup problems.

Enabling trace didn't show anything interesting with 3.0.7. I'll try with 2.5.6. I'm not sure I want to tackle the upgrade at the moment.

thanks

[john_gregg]

I diff'ed 2.5.6 and 3.0.7. The latter has a finally block in the AsyncMessageListenerInvoker.run() method. The finally block cleans up when the JNDI lookup fails because the server isn't running. Without the finally block in 2.5.6, the uncaught runtime exception skips the normal cleanup. I assume that causes the listener thread to die. Anyway, the activeInvokerCount doesn't get decremented without the finally block, which causes the shutdown process to hang.

I'm going to try reworking my tests to not load the JMS stuff. If that doesn't work I'll look at upgrading.

Thanks for your help.