Oct 16th, 2012, 07:40 AM
Failure recovery, total loss of a broker
I'm testing the AMQP/Rabbit code, specifically in failover and recovery situations. My test client uses a MessageListenerContainer and a POJO to receive messages which I send to the broker cluster from a different client or host. My rabbit cluster (3 machines) is behind a load balancer that detects if a broker is up or down using a port check every 10 seconds. The cluster and LB work fine in normal situations.
If my Spring client detects a shutdown of the broker it is connected to, then it logs the shutdown exception, reconnects (back through the same VIP on the LB) to a different cluster member and continues to receive messages - this happens very quickly.
However, if the connection between client and cluster is simply severed (remove a network cable or just power off the broker machine without any server or OS shutdown) then the client doesn't reconnect at all and simply stops receiving messages. It doesn't seem to detect the loss of connectivity ever, even with a restart of the broker or restore the network link. Only if I kill the client app and restart it will it obtain a new connection to the cluster and receive all of the messages that had built up in the cluster.
Is this a limitation of the AMQP or Rabbit Spring code, or the broker, or something that can be configured in the client side?
Tags for this Thread