InvalidDestinationException using temporary queues in hornetQ cluster

[samhendley]

Hello, I am working on some POC projects using spring integration and hornetQ. I built out my example and it works perfectly when I have a single node serving my requests. As soon as I add another jboss instance with a clustered hornetQ instance I start getting failures.

Here is the exception thrown on the handler side, it appears that it cannot publish back to the temporary queue. This error only occurs on the other node that didn't issue the request.

18:38:04,391 WARN  [org.springframework.jms.listener.DefaultMessageListenerContainer] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-9) Setup of JMS message listener invoker failed for destination 'testQueue2' - trying to recover. Cause: Destination dbe17fa4-587a-46bf-9202-9d381b7c99c7 does not exist
18:38:04,402 INFO  [stdout] (Thread-3 (HornetQ-client-global-threads-15240740)) Removing clusterName=jms.tempqueue.221f172b-9e38-4453-a3d4-a8cf329976bfd53508e9-a9db-11e1-bb73-080027000cb0 on ClusterConnectionImpl@9262213[nodeUUID=199cc33b-aa68-11e1-9329-001aa0397847, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192-168-100-5, address=jms, server=HornetQServerImpl::serverUUID=199cc33b-aa68-11e1-9329-001aa0397847]
18:38:04,408 INFO  [stdout] (Thread-3 (HornetQ-client-global-threads-15240740)) Removing clusterName=jms.tempqueue.157dc876-9911-4a33-ac54-6e6f2aabe641d53508e9-a9db-11e1-bb73-080027000cb0 on ClusterConnectionImpl@9262213[nodeUUID=199cc33b-aa68-11e1-9329-001aa0397847, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192-168-100-5, address=jms, server=HornetQServerImpl::serverUUID=199cc33b-aa68-11e1-9329-001aa0397847]
18:38:04,421 INFO  [stdout] (Thread-3 (HornetQ-client-global-threads-15240740)) Removing clusterName=jms.tempqueue.278b801d-8269-46e5-9658-0d176cab6b8fd53508e9-a9db-11e1-bb73-080027000cb0 on ClusterConnectionImpl@9262213[nodeUUID=199cc33b-aa68-11e1-9329-001aa0397847, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192-168-100-5, address=jms, server=HornetQServerImpl::serverUUID=199cc33b-aa68-11e1-9329-001aa0397847]
18:38:04,422 INFO  [stdout] (http--0.0.0.0-8080-1) GOT REQUEST: [Element: <GetBillSummary [Namespace: http://Mobile.Aclara.com]/>]
18:38:04,428 WARN  [org.springframework.jms.listener.DefaultMessageListenerContainer] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-13) Execution of JMS message listener failed, and no ErrorHandler has been set.: javax.jms.InvalidDestinationException: Destination 07cf88df-4991-4c66-b440-97a177c4ba68 does not exist
        at org.hornetq.jms.client.HornetQSession.createProducer(HornetQSession.java:337)
        at org.springframework.integration.jms.ChannelPublishingJmsMessageListener.sendReply(ChannelPublishingJmsMessageListener.java:390) [spring-integration-jms-2.1.0.RELEASE.jar:]
        at org.springframework.integration.jms.ChannelPublishingJmsMessageListener.onMessage(ChannelPublishingJmsMessageListener.java:294) [spring-integration-jms-2.1.0.RELEASE.jar:]
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535) [spring-jms-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495) [spring-jms-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [spring-jms-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [spring-jms-3.0.7.RELEASE.jar:3.0.7

On the issuing side here is the timeout exception we get (included just for context of how request was getting done)

19:10:52,900 WARN  [org.springframework.integration.ws.SimpleWebServiceInboundGateway] (http--0.0.0.0-8080-28) failure occurred in gateway sendAndReceive: org.springframework.integration.MessageTimeoutException: failed to receive JMS response within timeout of: 5000ms
        at org.springframework.integration.jms.JmsOutboundGateway.handleRequestMessage(JmsOutboundGateway.java:389) [spring-integration-jms-2.1.0.RELEASE.jar:]
        at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:97) [spring-integration-core-2.1.0.RELEASE.jar:]
        at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) [spring-integration-core-2.1.0.RELEASE.jar:]
        at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.6.0_20]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_20]
        at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_20]

It appears to work most of the time, looks like it fails around 5% of the time the request is handled on the other node. I have found this ticket from 2010 that seems like a likely culprit https://issues.jboss.org/browse/HORNETQ-286.

I can't imagine that a bug this obvious is still sitting around so I assume I must be doing something stupid. Any recommendations for what I should be looking for to help figure this out?

[Mark Fisher]

This looks like an issue on the hornet side. Are you sure that your cluster is configured properly? The link you provided does indeed look relevant, but it seems that bug has been fixed (unless you are using an earlier version).

[samhendley]

I also believe its on the HornetQ side, I posted there as well but I wanted to double check that I wasn't missing something on the spring integration side. Thanks for looking at it, I am still very new to spring and there have been enough cases where what I thought was a bug or bad behavior can be fixed with a flag somewhere. Thanks again.

HornetQ post: https://community.jboss.org/message/739138