DMLC Recovery attempt throttling (Exponential back off)?

[pbray]

Hi All,

Is it possible to throttle the number of connection recovery attempts which are made by the DMLC (Dynamic Message Listener Container)? Or to modify the behavior to use some kind of exponential back off? I understand that we can already configure the recovery interval in milliseconds. Unfortuanatly however this is not an acceptable solution for our infrastructure team :-( .

Our issue is that we have an application which makes heavy use of Spring MDP's and is deployed in WebSphere 7.

As such we have configured our DMLC containers with a cache level of "CACHE_NONE" to delegate JMS connection / session & consumer caching to the IBM MQ JCA adapter pool. This application is deployed into multiple clusters for different environments DEV, TEST, UAT etc and so we end up with up to 200 connections which are all using a single MQ Queue manager.

The problem we have is that the Queue Manager we are using has a weekly scheduled outage. As such each DMLC within our application in each environment detects that the connection has been lost and diligently goes through the process of attempting to reconnect, which it eventually does when the queue manager comes on line 4 hours later.

Ideally we would like to be able to set the recovery interval and some back off multiplier i.e. the first attempt to recover happens after 30 seconds then 1 min, 2 min 4 min, 8 min, 16 etc.

Any assistance which you could provide would be greatly appreciated.

Thanks,
Patrick Bray

[Gary Russell]

There's nothing built-in, but there's no reason you couldn't subclass the DMLC (DefaultMessageListenerContainer) and override the

sleepInbetweenRecoveryAttempts()

method.

You'd have to define the container using <bean/> definitions, rather than with the JMS namespace, though.

[pbray]

Hi Gary,

Thanks for your quick reply.

As you suggested I have tested sub classing the DMLC and overriding both the sleepInbetweenRecoveryAttempts() (to calculate the sleep time) and the refreshConnectionUntilSuccessful() methods to keep track of the retry attempts.

This appears to be working well as per the log output below:

2012-11-02 10:33:49,186 [testListener-1] WARN  com.anz.otc.jms.ExpBackoffMessageListenerContainer - Could not refresh JMS Connection for destination 'DOTC.OUTBOUND.TRADE.FEED' - retrying in 10000 ms. Cause: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect
2012-11-02 10:34:00,154 [testListener-1] WARN  com.anz.otc.jms.NamedMessageListenerContainer - Could not refresh JMS Connection for destination 'DOTC.OUTBOUND.TRADE.FEED' - retrying in 20000 ms. Cause: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect
2012-11-02 10:34:21,280 [testListener-1] WARN  com.anz.otc.jms.ExpBackoffMessageListenerContainer- Could not refresh JMS Connection for destination 'DOTC.OUTBOUND.TRADE.FEED' - retrying in 40000 ms. Cause: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect

Do you see any issues with this approach?

/**
 * Extension of Springs DefaultMessageListener to support an "Exponential back off Algorithm" when
 * performing connection recovery.
 */
public class ExponentialBackoffListenerContainer extends DefaultMessageListenerContainer {

    private long recoveryInterval;
    private int recoveryAttempts = 0;
    private int maximumRecoveryIntervalInSeconds;

    @Override
    public String toString() {
        return getBeanName();
    }

    @Override
    protected void refreshConnectionUntilSuccessful() {

        while (isRunning()) {
            try {

                if (sharedConnectionEnabled()) {
                    refreshSharedConnection();
                }
                else {
                    final Connection con = createConnection();
                    JmsUtils.closeConnection(con);
                }

                logger.info("Successfully refreshed JMS Connection");
                recoveryAttempts = 0;
                break;
            }
            catch (final Exception ex) {

                recoveryAttempts++;

                final StringBuilder msg = new StringBuilder();
                msg.append("Could not refresh JMS Connection for destination '");
                msg.append(getDestinationDescription()).append("' - retrying in ");
                msg.append(calculateRetryDelay()).append(" ms. Cause: ");
                msg.append(ex instanceof JMSException ? JmsUtils.buildExceptionMessage((JMSException) ex) : ex
                        .getMessage());

                if (logger.isDebugEnabled()) {
                    logger.warn(msg, ex);
                }
                else {
                    logger.warn(msg);
                }
            }
            sleepInbetweenRecoveryAttempts();
        }
    }

    @Override
    protected void sleepInbetweenRecoveryAttempts() {
        if (this.recoveryInterval > 0) {
            try {
                Thread.sleep(calculateRetryDelay());
            }
            catch (InterruptedException interEx) {
                // Re-interrupt current thread, to allow other threads to react.
                Thread.currentThread().interrupt();
            }
        }
    }

    private long calculateRetryDelay() {
        if (recoveryAttempts == 0) {
            return recoveryInterval;
        }
        else {
            return Math.min(Math.round(recoveryInterval * Math.pow(2, recoveryAttempts)),
                    (maximumRecoveryIntervalInSeconds * 1000));
        }
    }

    public void setMaximumRecoveryIntervalInSeconds(int maximumRecoveryIntervalInSeconds) {
        this.maximumRecoveryIntervalInSeconds = maximumRecoveryIntervalInSeconds;
    }

    @Override
    public void setRecoveryInterval(long recoveryInterval) {
        super.setRecoveryInterval(recoveryInterval);
        this.recoveryInterval = recoveryInterval;
    }
}

Now as we are using Apache Camel in our application I need to figure out if it is possible to wire a custom MessageListenerContainer bean into the JMSComponent.

Thanks,
Patrick Bray

[Gary Russell]

Looks ok; it would be nice if the DMLC had a hook for successful reconnect, so you wouldn't have to override the entire refresh... method.

I can't speak for Camel, but I can say that if you'd used Spring Integration instead, you would be able to inject a custom container :-)

[pbray]

Excellent thanks Gary.

I agree success and failure reconnect hooks would be great for simplifying this or even something along the lines of the ability to inject a retry delay calculation strategy bean which had access to a retry count. Perhaps in a future version?

I will raise a question on the Camel mailing list and see if it is possible to inject. Unfortunately I don't think we will be able to use Spring Integration in this environment as Camel is already well established.

A bit of a pity really as I have had a look at some of the demos of Spring Integration on youtube and it looks pretty slick.

One quick follow up question? In the scenario where the DMLC looses connectivity to the JMS broker would we expect to see one connection recovery attempt per DMLC or one per concurrent consumer?

Thanks,
Patrick Bray

[Gary Russell]

One quick follow up question? In the scenario where the DMLC looses connectivity to the JMS broker would we expect to see one connection recovery attempt per DMLC or one per concurrent consumer?

You should only have one thread retrying the connection because refreshConnectionUntilSuccessful() is being run under the recoveryMonitor lock. The other consumers are held off trying to acquire the lock; once the connection is re-established, the other threads will see that their consumer is closed, and create a new consumer at that time.

[pbray]

Excellent thanks for all your help Gary, it is much appreciated.

In case anyone else stumbles across this thread and is attempting to do the same thing I can confirm that it is possible to wire a custom MessageListenerContainer into the Apache Camel JMS Component via the messageListenerContainerFactoryRef property:

http://camel.apache.org/jms

Thanks,
Patrick Bray