Listeners do not receive messages after a bit of inactivity.

[ashneyderman]

Hi, all!

I setup my listener as I think of it in no particularly special way. But what I am noticing is that initially listeners are receiving messages, then there is a period where I stop sending anything for a while (10-20 minutes). After resuming message sending the listener hears nothing of it (my logs do not show that blockingqueueconsumer picked up anything). I am wondering what is the reason for this?

Here is my setup for the listener:

<bean id="cc.listener.container"class="org.springframewo rk.amqp.rabbit.listener.SimpleMessageListenerConta iner">
<property name="channelTransacted" value="false" />
<property name="concurrentConsumers" value="2" />
<property name="prefetchCount" value="1" />
<property name="connectionFactory" ref="rabbitConnectionFactory" />
<property name="acknowledgeMode" value="MANUAL" />
<property name="queues" value="cc.${instance.name}.queue" />
<property name="autoStartup" value="true" />
<property name="messageConverter" ref="rabbitmq.messageConverter" />
<property name="delegateListener" ref="rabbitmq.ccMessageListener" />
</bean>

does anyone have any ideas of what might be wrong with this? How to debug this problem? On rabbit's web UI I see that messages are un-acknowledged whenever this happens. Rabbit's logs do not indicate any problems. Logs on the client side of things are just regular polling:

Logs on the client side when this happens indicate nothing of interest just the usual polling message:
2012-10-12 15:15:35,251 DEBUG [org.springframework.amqp.rabbit.listener.BlockingQ ueueConsumer] - Retrieving delivery for Consumer: tag=[amq.ctag-A0B0-MDfTy5QiLUpgqXWNT], channel=Cached Rabbit Channel: AMQChannel(amqp://alex@localhost:5672/alex,3), acknowledgeMode=MANUAL local queue size=0

I am using spring amqp/rabbit release 1.1.2.RELEASE, amqp client 2.8.4 and my server is running 2.8.6 of rabbit.

Any help is greatly appreciated.

Cheers,
Alex.

[ashneyderman]

Just a bit of additional information. BTW, I can consistently reproduce this problem (after about 3/4 minutes of inactivity listener stops pumping the messages). I ran rabbitmqctl report on the server and I see this on the queue where message is not being consumed:

<'rabbit@ip-10-32-14-234'.1.23181.13> cc.alex.queue false false [{"x-dead-letter-exchange","Error Messages"},{"x-dead-letter-routing-key","cc.error"}] 0 35024 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,1}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,1}, {next_seq_id,27}, {persistent_count,0}, {avg_ingress_rate,0.003245172815983037}, {avg_egress_rate,0.003245172815983037}, {avg_ack_ingress_rate,0.003245172815983037}, {avg_ack_egress_rate,0.0}]

So, it seems that the message is not being acknowledged hence -> "{pending_acks,1}". Is there a reason anyone can think of that this might be the case? Anyone got any ideas?

Cheers,
Alex.

[Gary Russell]

Best guess is the listener is holding up the container threads.

Can you take a stack dump (using jstack or VisualVM) when it is "stuck" ??

A debug level log would likely be helpful too.

[ashneyderman]

here is the thread dump. http://pastebin.com/Dy8h70HZ

I see nothing suspicious in there. But then again I am not sure what to look for.

BTW, amqp client is 2.8.6 version now. I updated jsut to make sure there is no problem with version incompatibilities.

[Gary Russell]

Yes, looks ok.

I just noticed you have

<property name="acknowledgeMode" value="MANUAL" />

This means you are responsible for the ack. Any reason you are not using the default - AUTO (which is different to rabbit autoAck). AUTO means the container with ack or nack the message after processing - ACK for normal, reject (with requeue by default) when an exception is thrown. NONE means Rabbit autoack.

[ashneyderman]

Yes, looks ok.

I just noticed you have

This means you are responsible for the ack. Any reason you are not using the default - AUTO (which is different to rabbit autoAck). AUTO means the container with ack or nack the message after processing - ACK for normal, reject (with requeue by default) when an exception is thrown. NONE means Rabbit autoack.

Nah, that was the first version. When I noticed the problem, I was thinking well I fucked something up let me change it to let spring manage it for me, just to see if it really was me. Changing that value to AUTO made no difference to the end result. So, at the time thread dump was produced the setting is set to AUTO (and all the rest of the adjustments were made to make that work).

Cheers,
Alex.