Problems with RedeliveryPolicy in AMQ while using Spring JMS

[chlebik]

Hello,

it's my first post here so - Welcome Everybody!

I got problem with ActiveMQ (version 5.4.2 and 5.5.1). We decided to introduce RedeliveryPolicy to ensure that our messages would be delivered even when there's some problem with processing (exactly problems with remote SMTP where we are sending messages to).


Our configuration:

<bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory"
		destroy-method="stop">
		<property name="connectionFactory">
			<bean class="org.apache.activemq.ActiveMQConnectionFactory">
				<property name="brokerURL">
					<value>${jms.broker.urls}</value>
				</property>
				<property name="redeliveryPolicy">
					<bean class="org.apache.activemq.RedeliveryPolicy">
						<property name="maximumRedeliveries" value="10"/>
						<property name="initialRedeliveryDelay" value="3000"/>
						<property name="useExponentialBackOff" value="true"/>
						<property name="backOffMultiplier" value="3"/>
					</bean>
				</property>
			</bean>
		</property>
	</bean>

And queue configuration:

<bean id="jmsContainerNotificationIn"
		class="org.springframework.jms.listener.DefaultMessageListenerContainer">
		<property name="connectionFactory" ref="jmsConnectionFactory" />
		<property name="destination" ref="notificationInQueue" />
		<property name="messageListener" ref="notificationReader" />
		<property name="sessionTransacted" value="true" />
		<property name="receiveTimeout" value="1000" />
		<property name="recoveryInterval" value="5000" />
		<property name="concurrentConsumers" value="2" />
		<property name="maxConcurrentConsumers" value="10" />
		<property name="idleTaskExecutionLimit" value="1" />
</bean>

The problem is with large JMS input which occurs every day at night. We are about to put in the queue about 100k messages. But after about 30 minutes of sending process (in testing, production environment has better machine so a couple of minutes more) our AMQ crashes - suddenly PermGen and Old Space reaches 100%. Same thing with Eden Space and Survivor space and GC is going crazy going all over again and again. The only way to deal with it is to shut down AMQ and start over.

When we disable RedeliveryPolicy everything is OK so I assume that's the problem. We already tried to tune AMQ memory settings, system usage and so on. We got broker cluster then we split brokers up and the problem is still there.

So my question is - what's SpringJMS exactly doing with RedeliveryPolicy (using AMQ org.apache.activemq.RedeliveryPolicy) and what can be done to avoid AMQ crashing? Any help would be appreciated.


Have a nice day
Michael

[bsnyder]

Spring JMS does nothing special with the ActiveMQ redelivery policy. This sounds like an issue internally with ActiveMQ. The best thing you can do is ask about it on the ActiveMQ user mailing list. Information on that list is available here:

http://activemq.apache.org/mailing-lists.html

[chlebik]

After tons of tests/config changes it appears that problem lies in Spring and exactly with org.springframework.jms.listener.DefaultMessageLis tenerContainer. Why? I got XML config with configuration of 4 listeners. Everything configured as I wrote in 1 post (with different names of queues and processors of course). And with RedeliveryPolicy and sessionTransacted=TRUE on 2 listeners my AMQ crashes. But when I leave only 1 listener with sessionTransacted=TRUE - everything works like a charm. So I assume there's problem with Spring listener OR there's some transaction problem I am not aware of. I must add that there's still no transactionManager used for ConnectionFactory.

BTW: I've switched to AMQ 5.6.SNAPHOT to force nonBlockingRedelivery (to avoid 'hanging' of threads on one redelivery process).