Server hands on Context stop

[jamescarr99]

When I try to redeploy or undeploy an app that is consuming messages sometimes it seems to hang. This usually requires me to restart the whole tomcat server which brings everything back up and I can stop and start it.


Here are my logs, any ideas?

INFO AnnotationConfigWebApplicationContext - Closing Root WebApplicationContext: startup date [Mon Dec 19 17:08:58 EST 2011]; root of context hierarchy
INFO DefaultLifecycleProcessor - Stopping beans in phase 2147483647
INFO SimpleMessageListenerContainer - Waiting for workers to finish.
INFO SimpleMessageListenerContainer - Workers not finished. Forcing connections to close.
INFO DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultL istableBeanFactory@470a698: defining beans [org.springframework.context.annotation.internalCon figurationAnnotationProcessor,org.springframework. context.annotation.internalAutowiredAnnotationProc essor,org.springframework.context.annotation.inter nalRequiredAnnotationProcessor,org.springframework .context.annotation.internalCommonAnnotationProces sor,appConfig,org.springframework.context.annotati on.ConfigurationClassPostProcessor$ImportAwareBean PostProcessor#0,com.carfax.configuration.amqp.Amqp Context#0,messageConverter,jndiContext,connectionF actory,ampqAdmin,retryInterceptor,errorExchange,er rorQueue,inboundQueue,outboundQueue,deadLetterBind ing,publisher,onContact,inboundMessageListenerCont ainer,companyMessageProcessor,onContactAccountCrea tor,accountSerializationWrapper,messageListener,on ContactDataSource]; root of factory hierarchy

[Dave Syer]

There is a broker bug (discussed on the rabbit mailing list) that causes a race condition when a Connection and its Channels are closed concurrently. You might be tickling that. It's fixed in in the 2.7.1 broker I think. I'm not sure if there is a workaround in Springland - maybe stopping the consumers before you stop the ApplicationContext?

[jamescarr99]

Hmmm... good points. I plan to upgrade to 2.7.1 soon anyway, but I guess in the meantime I'll just setup a @PreDestroy that makes sure the channels get closed first.

[andrewcelerity]

Hi All, sorry to bump an old thread, but I still believe this might be a problem up to including broker 3.0.4. Is it definitely a broker bug and if so, how did you get around it?

Thanks.

[Gary Russell]

What version of Spring-AMQP are you using? Are you using the AMQP log4j appender (there was a bug in it that kept a thread running; fixed in 1.1.4).

Probably the best way to debug this is to take a thread dump (jstack, or VisualVM); take a look at the threads; if you can't figure out why they are "stuck", post the thread dump as an attachment here.

[andrewcelerity]

Current using 1.1.1.RELEASE. I'm not using the appender. I'm hooking Apache Camel to Rabbit for enterprise messaging with spring (using their nifty Spring remoting abstraction).

I'll give VisualVM a shot too if upgrading doesn't help. Thanks!

[andrewcelerity]

I updated to 1.1.4 and still have the issue. Here is a thread dump from a hung up Tomcat server: https://gist.github.com/anonymous/f70986886f3043e6cd8a

[Gary Russell]

It doesn't look like the container was stopped because all the listener threads are in nextMessage().

Can you post a DEBUG log showing the shutdown?

[andrewcelerity]

Here the debug shutdown. It's snipped from the relative section. Some names were redacted out.
https://gist.github.com/anonymous/87d565c5dc3b16fa6119

Thanks for your help!

[Gary Russell]

Strange - it looks like the connection was closed and reopened (hence all the re-declarations).

It looks to me like the container was never shut down.

I am not familiar with the camel wrapper for Spring-AMQP; I suggestion you take it up with them.

11:27:30.045 INFO [pool-2-thread-1][amqp.spring.camel.component.SpringAMQPConsumer] Network connection closed to broker for endpoint Endpoint[REDACTED]
...
11:27:30.186 DEBUG [SpringAMQPConsumer.SpringAMQPExecutor-2][org.springframework.amqp.rabbit.core.RabbitAdmin] declaring Exchange 'REDACTED'
...