DMLC - Leaves Stale threads listeneing to Queues

[barathwajram]

Hey all,

I have my application running with MDP's - with concurrent consumers set to 1. But somehow it increased the number of consumers for a Q. When i shutdown the application i see some stale threads listeneing to the Q, which doesnt do anything with the messages, but consumes them. (Looks like). No exceptions, No logs for the messages that went into. I am not sure whats happening.

2. Having the stale threads if i restart the application i see following in my logs which is very wierd.

[5/17/07 11:52:11:518 EDT] 00000042 ApplicationMg A   WSVR0221I: Application started: MyApplication
[5/17/07 11:52:13:096 EDT] 00000c35 DefaultMessag E   Setup of JMS message listener invoker failed - trying to recover
[5/17/07 11:52:13:096 EDT] 00000c34 DefaultMessag E   Setup of JMS message listener invoker failed - trying to recover
[5/17/07 11:52:13:096 EDT] 00000c33 DefaultMessag E   Setup of JMS message listener invoker failed - trying to recover
[5/17/07 11:52:13:096 EDT] 00000c37 DefaultMessag E   Setup of JMS message listener invoker failed - trying to recover
[5/17/07 11:52:13:096 EDT] 00000c36 DefaultMessag E   Setup of JMS message listener invoker failed - trying to recover
[5/17/07 11:52:13:096 EDT] 00000c35 DefaultMessag E   TRAS0014I: The following exception was logged java.lang.IllegalStateException: markInUse: illegal state exception. State = STATE_TRAN_WRAPPER_INUSE
	at com.ibm.ejs.j2c.MCWrapper.markInUse(MCWrapper.java:713)
	at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java:2162)
	at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:769)
	at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:569)
	at com.ibm.ejs.jms.JMSQueueConnectionFactoryHandle.createQueueConnection(JMSQueueConnectionFactoryHandle.java:81)
	at org.springframework.jms.listener.DefaultMessageListenerContainer102.createConnection(DefaultMessageListenerContainer102.java:70)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$MessageListenerContainerResourceFactory.createConnection(DefaultMessageListenerContainer.java:927)
	at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:189)
	at org.springframework.jms.listener.DefaultMessageListenerContainer.doReceiveAndExecute(DefaultMessageListenerContainer.java:483)
	at org.springframework.jms.listener.DefaultMessageListenerContainer.receiveAndExecute(DefaultMessageListenerContainer.java:441)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:871)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:824)
	at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
	at java.lang.Thread.run(Thread.java:813)
.
                                 java.lang.IllegalStateException: markInUse: illegal state exception. State = STATE_TRAN_WRAPPER_INUSE
	at com.ibm.ejs.j2c.MCWrapper.markInUse(MCWrapper.java:713)
	at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java:2162)
	at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:769)
	at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:569)
	at com.ibm.ejs.jms.JMSQueueConnectionFactoryHandle.createQueueConnection(JMSQueueConnectionFactoryHandle.java:81)
	at org.springframework.jms.listener.DefaultMessageListenerContainer102.createConnection(DefaultMessageListenerContainer102.java:70)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$MessageListenerContainerResourceFactory.createConnection(DefaultMessageListenerContainer.java:927)
	at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:189)
	at org.springframework.jms.listener.DefaultMessageListenerContainer.doReceiveAndExecute(DefaultMessageListenerContainer.java:483)
	at org.springframework.jms.listener.DefaultMessageListenerContainer.receiveAndExecute(DefaultMessageListenerContainer.java:441)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:871)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:824)
	at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
	at java.lang.Thread.run(Thread.java:813)

Any idea on whats goin on with those threads?

Thanks
Barath

[Lyserg]

Hello,

as i can see from the stacktrace you are using websphere mq.
Do you run your DMLC inside a Websphere ?
If so, do you use the CommonJ WorkManager abstraction ?

regards
agim

[barathwajram]

Well,

Do you run your DMLC inside a Websphere ?
Yes I am bringing up the spring configuration from a web-app in my application. So yes, DMLC is running inside websphere. Do u see any issues with it?

If so, do you use the CommonJ WorkManager abstraction ?
I am not using CommonJ WorkManager abstraction. Can you throw more insights to this. That will be great.

Thanks
Barath

[Lyserg]

Yes I am bringing up the spring configuration from a web-app in my application. So yes, DMLC is running inside websphere. Do u see any issues with it?

No, absolutely not, if the spring DMLC is configured in the right way.

I am not using CommonJ WorkManager abstraction. Can you throw more insights to this. That will be great.

In websphere application server it is not allowed to create native threads inside your applications (Spring DMLC create native threads if you use the default workmanager). So for that if you need asynchronus processing you should use the CommonJ WorkManager abstraction. The Spring Framework supports the CommonJ Workmanager, so you can use it, with only minimal configuration change.
After that you can configure your thread inside the websphere application server.

Herewith i past you some minimal configuration, with the default workmanager. You should create an own workmanager for productional use.

First you will need to create the CommonJ Workmanager, through this xml snippet

<bean id="taskExecutor"
		class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
		<property name="workManagerName" value="wm/WorkManager" />
		<property name="resourceRef" value="false" />
	</bean>

Note: In this sample i use an direct lookup without any ressource reference, you should create one. "wm/WorkManager" is the default Workmanager in your Application Server.

After defining the WorkManager you have to simply inject the taskExecutor into your DMLC configuration.

Here is the sample about this:

<bean id="container"
	class="org.springframework.jms.listener.DefaultMessageListenerContainer">
           ...
		<property name="taskExecutor" ref="taskExecutor" />
           ...
	</bean>

Hope this helps you out.

regards
agim

[barathwajram]

Thanks. That really helped.
I am not seeing any stale threads after started using the work manager to manage DMLC.

kept my finger crossed though.

Thanks
Barath