network problem while reading message from queue

[nima]

Hi,
I'm using spring-amqp within a distributed project which reads messages from a rabbitmq-server in a remote host. the network is very slow (and unreliable I think), but the message rate seems fine to me (till now!). the problem is that sometimes I see some strange behavior from one of the instances; the application stops reading messages from remote rabbitmq (when other instances are usually active);

here's the configuration for MessageListenerContainer ...

final MessageListenerAdapter outgoingMessageListenerAdapter = new MessageListenerAdapter(outgoingMessageConsumer, outgoingMessageConverter);
final SimpleMessageListenerContainer outgoingMessageListenerContainer = new SimpleMessageListenerContainer();
outgoingMessageListenerContainer.setConnectionFactory(connectionFactory);
outgoingMessageListenerContainer.setQueueNames(serviceInfo.getOutgoingQueue());
outgoingMessageListenerContainer.setMessageListener(outgoingMessageListenerAdapter);
outgoingMessageListenerContainer.setErrorHandler(errorHandler);
outgoingMessageListenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
outgoingMessageListenerContainer.setDefaultRequeueRejected(false);
outgoingMessageListenerContainer.setConcurrentConsumers(serviceInfo.getOutgoingConsumerCount());
outgoingMessageListenerContainer.setRecoveryInterval(60000); // recovery when the connection has been lost!
this.outgoingMessageListenerAdapter = outgoingMessageListenerContainer;
outgoingMessageListenerContainer.start();

and the netstat result in this case :

$ netstat -na | grep  REMOTE_RABBIT_HOST
tcp6       0 111096 192.168.0.10:55942      REMOTE_RABBIT_HOST:5672       ESTABLISHED

thanks in advance

P.S: the Send-Q (111096 in the netstat sample) keeps unchanged, or reduces with a very low speed
P.P.S : restarting the application (the box which reads from the rabbit) usually solves the problem (not sure)

[Gary Russell]

WIth an unreliable network, you might want to consider configuring heartbeats on the underlying com.rabbitmq.client.ConnectionFactory ...

    /**
     * Set the requested heartbeat.
     * @param requestedHeartbeat the initially requested heartbeat interval, in seconds; zero for none
     */
    public void setRequestedHeartbeat(int requestedHeartbeat) {
        this.requestedHeartbeat = requestedHeartbeat;
    }

Here's some more information about heartbeats...

http://rabbitmq.1065348.n5.nabble.co...at-td1977.html

[nima]

Hi Gary and thanks again for your help

That problem didn't happen again, and found to be a network problem (firewall issues I guess), but another weird thing is happening on this project, which is that every day I find 50-300 duplicated messages in my log files (based on the traffic count which is vary between 200,000 to 700,000 per day). Everything was okay during the tests and the only thing that is different from the test environment is the "Network Speed" issues on the operational environment ( as I mentioned). do you think I've configured something wrong or may be there's a bug while handling delivery messages which causes duplication, etc ?

thanks again buddy

[nima]

well, let me add more ...
I reduced the number of consumer-boxes to 1 by bring down all the consumer instances (which located on different servers) but one instance, and till now, everything seems ok !
is there any possibility of a bug in consuming/processing acknowledges when multiple consumers is used on separate VMs (or maybe I've configured something wrong for this case) ?
thanks again