Message loss after a netweork disconnect

[rasadoll]

Hi,

If I disconnect my consumer server from the RabbitMQ server (simply unplug the network cable) and then reconnect again, the first message sent after this event is lost but all subsequent messages are received perfect by the consumer. The queue consumers are configured with Auto-Ack property. After sending the first message to the queue, the queue immediately looks empty but it is never received by the consumer.

Am I missing a configuration setting or is it a bug?
I am using RabbitMQ 2.8.4 and spring-amqp 1.1.2

Thanks for your help.

[Gary Russell]

Pulling cables is never a good idea, unless you have heartbeats enabled - so both ends get to know the connection was broken.

That said, I wouldn't expect a complete loss of a message but it's certainly possible to be left with an un-acked message in a queue after a such a failure.

[rasadoll]

Hi Gary,
No message is left in queue after this. This problem is consistently reproducible.

Are you recommending to set heartbeats?
You said "Pulling cables is never a good idea". What is the best way to test network interruption?

Thanks,

[Gary Russell]

All I am saying is, the way TCP works is, if you pull an ethernet cable, depending on the network topology, one or the other end might not know that the socket was disconnected.

The underlying rabbit connection factory supports heartbeats which can be used to detect this condition.

It appears that the new RabbitMQ 3.0.0 release enables heartbeats by default...

https://www.rabbitmq.com/release-notes/README-3.0.0.txt

As I said, I would not expect messages to be lost; but unack'd messages could exist; they wouldn't be in the queue, but the broker won't resubmit it until it finds out the connection was lost.

If you can reproduce it with a simple example, and/or provide a log, I will take a look.

[rasadoll]

The following simple example can reproduce the problem.

<rabbit:connection-factory id="connectionFactory" host="myhost" port="5672" channel-cache-size="10" />
<rabbit:admin connection-factory="connectionFactory" />
<rabbit:queue id="jobExecQ" name="Q_JobExec" queue-arguments="haQ" />
<rabbit:queue-arguments id="haQ">
   	<entry key="x-ha-policy" value="all" />
</rabbit:queue-arguments>
   	
<rabbit:listener-container connection-factory="connectionFactory" concurrency="5" acknowledge="none" >  
     <rabbit:listener ref="jobExecutor" queues="jobExecQ" />
</rabbit:listener-container>
<bean id="jobExecutor" class="test.amqp.JobExecutor" />

And here is the MessageListener code:

public class JobExecutor implements MessageListener {

  private static Log log = LogFactory.getLog(JobExecutor.class);

  public void onMessage(Message rmqMessage)  {
    String messageText = new String(rmqMessage.getBody());
    log.info("Received message: " + messageText );
  }

  public static void main(String[] args) throws Exception {
    ApplicationContext context = new ClassPathXmlApplicationContext("rabbitConfiguration.xml");
  }
}

To reproduce:
1. start the app by the main method in the above class
2. send a message to Q_JobExec queue using the default exchange (I used rabbitmq management console for this)
3. check the log to see the received message
4. unplug the network cable for a few seconds
5. send second message as you did in step 2
6. send third/forth/... messages

The message in step 5 is lost.

Thanks for your help.

[Gary Russell]

Please use [ code ] ... [ /code ] tags (no spaces inside brackets) around code and config.

Did you pull the cable on the consumer, or the server?
Is there a network switch between the consumer and the server? (Likely there is, because the server won't find out the connection was broken until it tries to send the next message).
Do you see anything interesting in the logs on the consumer (with TRACE level) regarding the "lost" message after the connection is re-established?
What about in the Rabbit log on the server?
If you see no evidence of the "lost" message arriving in the consumer's logs, I suggest you raise this question on the RabbitMQ mailing list.