DMLC stops receiving messages after during loadtests

[ray.frid]

I am using Spring3.1 in a standalone env.

I have configured DMLC(DefaultMessageListenerContainer) which listen to a remote topic on a remote jboss4 server.

I have a terrible situation after 200000 messsages which I get in high rate(could be less)

the listener stopping receiving the messages without error/warning.

But in other hand the application is still running. (what else happens that I need to kill -9 the process while a regular ./stop script wont help)

What have I configured wrong?

this is my configuration:

<bean id="TgwListenerDLMC"
    class="org.springframework.jms.listener.DefaultMessageListenerContainer">
    <property name="connectionFactory" ref="topicConnFactory" />
    <property name="pubSubDomain" value="true" />
    <property name="destinationResolver" ref="destinationResolver" />
    <property name="concurrentConsumers" value="1" />
    <property name="destinationName" value="/topic/TopicRealTime" />
    <property name="messageListener" ref="TgwMDB" />
    <property name="sessionAcknowledgeModeName" value="AUTO_ACKNOWLEDGE" />
    <property name="recoveryInterval" value="5000" />

</bean>

<bean id="topicConnFactory"
        class="org.springframework.jms.connection.CachingConnectionFactory">
        <property name="targetConnectionFactory" ref="jndiTopicConnFactory" />
        <property name="reconnectOnException" value="true" />
        <!-- Number of sessions that will be cached -->
        <property name="sessionCacheSize" value="20" />
    </bean>


    <bean id="jndiTopicConnFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiTemplate" ref="jndiTemplate" />
    <!-- JNDI name of connection factory as defined by provider -->
    <property name="jndiName" value="ConnectionFactory" />
</bean>

<bean id="destinationResolver"
    class="org.springframework.jms.support.destination.JndiDestinationResolver">
    <property name="jndiTemplate" ref="jndiTemplate" />
    <property name="cache" value="true" />
    <!--
        do not create a dynamic destination if the destination name is not
        found in JNDI
    -->
    <property name="fallbackToDynamicDestination" value="false" />
</bean>


<bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
    <property name="environment">
        <props>
            <prop key="java.naming.provider.url">${java.naming.provider.url}</prop>
            <prop key="java.naming.factory.initial">${java.naming.factory.initial}</prop>
            <prop key="java.naming.factory.url.pkgs">${java.naming.factory.url.pkgs}</prop>
        </props>
    </property>
</bean>

Here is ThreadDump print with jstack:

[Loaded sun.reflect.GeneratedConstructorAccessor17 from __JVM_DefineClass__]
2012-08-29 14:45:37,771 org.springframework.jms.connection.CachingConnectionFactory [WARN] Encountered a JMSException - resetting the underlying JMS Connection

javax.jms.JMSException: Failure on underlying remoting connection
        at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:102)
        at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:608)
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor88 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor89 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor90 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor91 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor92 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor93 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor94 from __JVM_DefineClass__]

thanks,
ray.

[rishishehrawat]

You should try taking thread dump (kill -3) of the java process. Will give you an idea about what is happening in the JVM & why it is not processing messages.

[ray.frid]

Will do that next time this situation will occur and post the log here.

But the above configuration seems to be right? sometimes you get lots with all the configuration options.