Messages not getting sent - driving me nuts

[mberg]

Hello all,

I have a small test client which I use to send messages to RabbitMQ. Messages are sent using a spring created gateway proxy, which sends the message to a channel, which the gateway then sends to a Topic exchange and on to a queue inside RabbitMQ.

When the RabbitMQ server runs locally (localhost), the messsage arrives without problems in the expected queue in RabbitMQ.

When the RabbitMQ server runs inside a virtual host, the message does NOT arrive. No error messages are given on the server, and the client produces exactly the same log output, of course with "localhost" being replaced by the ip number of the virtual machine.

I know there is some sort of connection between my client and the server, because if I try to use an incorrect user name, the connection fails with "likely authentication failure..." or whatever the text is. Also if I use an incorrect ip number, the connection fails.

Nothing I do seems to make any difference whatsoever, the messages simply disappear into thin cyberair.

How do I debug this? How do I find out what the heck is going on with these messages?

[Mark Fisher]

Do you have the rabbit management UI running on that server? That's always useful.

Otherwise, it sounds like you have a difference in configuration between the two brokers. Perhaps Queues are not bound to the Topic Exchange on that server. Can you show some of the configuration, including the part you change between the 2 deployments?

[mberg]

Yes I have the management ui running, it's been quite handy.

There might be some configuration differences, but really these are two Rabbit instances that are basically installed and run without any real configuration done. I use a RabbitAdmin instance to auto create queues and exchanges and when I run the test client on a local RabbitMQ instance everything works fine.

I followed the instructions for setting up firehose tracing (http://www.rabbitmq.com/firehose.html) and wrote a small logging consumer to just write out everything that goes into or comes out of the RabbitMQ server. When I manually publish a message using the management ui, the message is forwarded as expected to my logger. But when the test client runs, it doesn't publish anything.

It is as if the test client is connecting, then pretending to send a message, then waiting for a reply that obviously never arrives.

The test client log output says something like this:

2012-06-01 13:53:08,584 [SimpleAsyncTaskExecutor-1] DEBUG CachingConnectionFactory       Creating cached Rabbit Channel from AMQChannel(amqp://guest@172.20.51.117:5672/,1)
2012-06-01 13:53:08,629 [SimpleAsyncTaskExecutor-1] DEBUG RabbitTemplate                 Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1)
2012-06-01 13:53:08,629 [SimpleAsyncTaskExecutor-1] DEBUG RabbitAdmin                    declaring Exchange 'xml-request-delivery'
2012-06-01 13:53:08,632 [SimpleAsyncTaskExecutor-1] DEBUG RabbitAdmin                    declaring Queue 'myproduct.xmlProcessingReplyQueue'
2012-06-01 13:53:08,635 [SimpleAsyncTaskExecutor-1] DEBUG RabbitAdmin                    declaring Queue 'myproduct.xmlProcessingRequestQueue'
2012-06-01 13:53:08,637 [SimpleAsyncTaskExecutor-1] DEBUG RabbitAdmin                    Binding destination [myproduct.xmlProcessingRequestQueue (QUEUE)] to exchange [xml-request-delivery] with routing key [xml-request.*]
2012-06-01 13:53:08,641 [SimpleAsyncTaskExecutor-1] DEBUG RabbitAdmin                    Declarations finished
2012-06-01 13:53:08,650 [SimpleAsyncTaskExecutor-1] DEBUG BlockingQueueConsumer          Started on queue 'myproduct.xmlProcessingReplyQueue': Consumer: tag=[null], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1), acknowledgeMode=AUTO local queue size=0
2012-06-01 13:53:08,650 [main] DEBUG DefaultLifecycleProcessor      Successfully started bean 'amqpTemplate.replyListener'
2012-06-01 13:53:08,652 [main] DEBUG AbstractBeanFactory            Returning cached instance of singleton bean 'org.springframework.integration.config.IdGeneratorConfigurer#0'
2012-06-01 13:53:08,653 [main] DEBUG AbstractBeanFactory            Returning cached instance of singleton bean 'jobServiceGateway'
Fri Jun 01 13:53:08 CEST 2012: Sending message 0
2012-06-01 13:53:08,655 [SimpleAsyncTaskExecutor-1] DEBUG BlockingQueueConsumer          Retrieving delivery for Consumer: tag=[null], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1), acknowledgeMode=AUTO local queue size=0
Fri Jun 01 13:53:08 CEST 2012: Getting response for #0 (isDone=false)
2012-06-01 13:53:08,701 [SimpleAsyncTaskExecutor-1] DEBUG AbstractMessageChannel$ChannelInterceptorList preSend on channel 'xmlProcessingRequestChannel', message: [Payload=<xml test="0"/>][Headers={timestamp=1338551588701, id=0fcd5e69-853c-49b3-bf9f-6af8af22e73d, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@72940b84, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@72940b84, x-myproduct-metadata-id=olsen-0}]
2012-06-01 13:53:08,702 [SimpleAsyncTaskExecutor-1] DEBUG AbstractMessageHandler         org.springframework.integration.amqp.outbound.AmqpOutboundEndpoint#0 received message: [Payload=<xml test="0"/>][Headers={timestamp=1338551588701, id=0fcd5e69-853c-49b3-bf9f-6af8af22e73d, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@72940b84, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@72940b84, x-myproduct-metadata-id=olsen-0}]
2012-06-01 13:53:08,719 [SimpleAsyncTaskExecutor-1] DEBUG AbstractHeaderMapper           headerName=[x-myproduct-metadata-id] WILL be mapped, matched pattern=x-myproduct-*
2012-06-01 13:53:08,723 [SimpleAsyncTaskExecutor-1] DEBUG CachingConnectionFactory       Creating cached Rabbit Channel from AMQChannel(amqp://guest@172.20.51.117:5672/,2)
2012-06-01 13:53:08,723 [SimpleAsyncTaskExecutor-1] DEBUG RabbitTemplate                 Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,2)
2012-06-01 13:53:08,723 [SimpleAsyncTaskExecutor-1] DEBUG RabbitTemplate$4               Sending message with tag 5e76ee71-fc03-49aa-920d-a1a403cabef7
2012-06-01 13:53:08,724 [SimpleAsyncTaskExecutor-1] DEBUG RabbitTemplate                 Publishing message on exchange [xml-request-delivery], routingKey = [xml-request.number]
2012-06-01 13:53:09,659 [SimpleAsyncTaskExecutor-1] DEBUG BlockingQueueConsumer          Retrieving delivery for Consumer: tag=[amq.ctag-wp76PcdYvNnW5_GjoNDceX], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1), acknowledgeMode=AUTO local queue size=0
2012-06-01 13:53:10,659 [SimpleAsyncTaskExecutor-1] DEBUG BlockingQueueConsumer          Retrieving delivery for Consumer: tag=[amq.ctag-wp76PcdYvNnW5_GjoNDceX], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1), acknowledgeMode=AUTO local queue size=0
2012-06-01 13:53:11,660 [SimpleAsyncTaskExecutor-1] DEBUG BlockingQueueConsumer          Retrieving delivery for Consumer: tag=[amq.ctag-wp76PcdYvNnW5_GjoNDceX], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1), acknowledgeMode=AUTO local queue size=0
2012-06-01 13:53:12,661 [SimpleAsyncTaskExecutor-1] DEBUG BlockingQueueConsumer          Retrieving delivery for Consumer: tag=[amq.ctag-wp76PcdYvNnW5_GjoNDceX], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@172.20.51.117:5672/,1), acknowledgeMode=AUTO local queue size=0
...(continues until my preset 60 second timeout)...

[Mark Fisher]

Can you show the actual test client code and config?

[nettad]

I'm not sure how your rabbit configuration is setup but I was having similar problems and my solution was adding a reply-timeout to the amqp template configuration. The default reply timeout is 5 seconds.

in xml config:
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory" reply-timeout="15000"/>

Hope that helps.
Netta