java.lang.OutOfMemoryError: unable to create new native thread

[szegedics]

Hi,

After I found the problem ($title) I read in forums that "there is an upper limit on the number of operating system threads that can be opened by one process".
Searching deaper I encountered that the error ocures when I'm writing to the queue many messages in short time.
So I made a stress test, sending ObjectMessages to a queue with Spring Frameworks's JmsTemplate. I also created an MDB to listen for the messages on the specified queue. The queueing is transient and transactional. The application server that I use is jboss-4.0.1sp1 under Windows XP SP2. I count the number of the threads with the following command: typeperf "Process(java)\Thread Count" -si 2, this command will write to the screen the number of threads in the jboss in every 2 seconds (the number of the threads can be seen from the task manager too).
After I start the test the number of the threads are rising in big steps until I get the OutOfMemoryError:
"11/15/2005 09:26:34.540","60.000000"
"11/15/2005 09:26:36.543","60.000000"
"11/15/2005 09:26:38.546","60.000000"
"11/15/2005 09:26:40.609","63.000000"
"11/15/2005 09:26:42.632","229.000000"
"11/15/2005 09:26:44.645","424.000000"
"11/15/2005 09:26:46.658","617.000000"
"11/15/2005 09:26:48.661","804.000000"
"11/15/2005 09:26:50.664","1033.000000"
"11/15/2005 09:26:52.667","1258.000000"
"11/15/2005 09:26:54.670","1472.000000"
"11/15/2005 09:26:56.683","1641.000000"
"11/15/2005 09:26:58.686","1861.000000"
"11/15/2005 09:27:00.689","2085.000000"
"11/15/2005 09:27:02.692","2291.000000"
"11/15/2005 09:27:04.705","2498.000000"
"11/15/2005 09:27:06.708","2717.000000"
"11/15/2005 09:27:08.731","2924.000000"
"11/15/2005 09:27:10.744","3102.000000"
"11/15/2005 09:27:12.747","3301.000000"
"11/15/2005 09:27:14.760","3483.000000"
"11/15/2005 09:27:16.783","3676.000000"
"11/15/2005 09:27:18.796","3847.000000"
"11/15/2005 09:27:20.809","3886.000000"
"11/15/2005 09:27:22.812","3886.000000"
"11/15/2005 09:27:24.815","3886.000000"

So I made the same test with standard jms, without using the JmsTemplate and this problem doesn't occure, actualy the number of threads are not rising.
The other interesting thing is that it's faster then using the JmsTemplate (1000 messages with standard jms was sent in ~3 sec. and with spring framework in ~19 sec).

Here is my code using JmsTemplate:

JmsTemplate jmsTemplate= new JmsTemplate(connectionFactory);

jmsTemplate.setSessionTransacted(true);
jmsTemplate.setDeliveryMode(DeliveryMode.NON_PERSISTENT);

for (int i = 0; i < 5000; i++) {
	jmsTemplate.send(m_queue, new MessageCreator() {
		public Message createMessage(Session session) throws JMSException {
			return session.createObjectMessage("temp");
		}
	});
}

I also tried that the MessageCreator was created outside the for loop but the problem is the same.

Could this be a configuration problem? What am I doing wrong?

I looked after this problem through the net but I didn't find any respons.

Thanks,
szegedics

[ojolly]

I'm not a jms pro, but just to be sure, could you also post the code of your raw jms test code ?
And could you try out this code :
jmsTemplate.execute(new SessionCallback() {
public Object doInJms(Session session) throws JMSException {
Destination destination = resolveDestinationName(session,
m_queue);
doSend(session, destination, new MessageCreator() {
public Message createMessage(Session session)
throws JMSException {
return session.createObjectMessage("temp");
}
});
return null;
}
}, false);

Depending on the type of m_queue, you may remove the resolveDestination Name call and use "m_queue" instead of "destination" in the doSend. Even better, you can also pull up the newMessageCreator outside the loop.

hth

[jstrachan]

It could be that the JmsTemplate version will create & close lots of connection, session and producer objects for *each* send of a message unless you configure it with a pooling aware implementation of ConnectionFactory - whereas I suspect your pure JMS version does not have this issue.

[szegedics]

ojolly, thanx for the code but it didn't helped. First of all there is an error in it. The doSend method doesn't exists in the SessionCallback, so you can't overwrite it. Right?

Here is my raw jms code:

m_queueConnection = connectionFactory.createQueueConnection();
m_queueSession = m_queueConnection.createQueueSession(true, 0);
m_queueSender = m_queueSession.createSender(m_queue);

for (int i = 0; i < 1000; i++) {
	m_message = m_queueSession.createObjectMessage();
	m_message.setObject("temp");
	m_queueSender.send(m_message, DeliveryMode.NON_PERSISTENT, 0,
						3600000);
	m_queueSession.commit();
}

[szegedics]

jstrachan, thanx for you reply to, but still didn't helped, but here the problem could be me. Could you be more specific on "configure it with a pooling aware implementation of ConnectionFactory".

TIA

[drewcox]

I too have ended up on this post facing a similar problem sending a large number of messages with the JmsTemplate view BEA Weblogic 9.1. At about message 7000 I get the dreaded OutOfMemoryError in Spring 2.0M2:

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at weblogic.rmi.extensions.AbstractDisconnectMonitorD elegate.startPinger(AbstractDisconnectM
onitorDelegate.java:106)
at weblogic.rmi.extensions.AbstractDisconnectMonitorD elegate.addDisconnectListener(AbstractD
isconnectMonitorDelegate.java:80)
at weblogic.rmi.extensions.DisconnectMonitorListImpl. addDisconnectListener(DisconnectMonitor
ListImpl.java:80)
at weblogic.messaging.dispatcher.DispatcherWrapperSta te.addPeerGoneListener(DispatcherWrappe
rState.java:285)
at weblogic.messaging.dispatcher.DispatcherWrapperSta te.<init>(DispatcherWrapperState.java:7
1)
at weblogic.messaging.dispatcher.DispatcherManager.ad dDispatcherReference(DispatcherManager.
java:115)
at weblogic.jms.dispatcher.JMSDispatcherManager.addDi spatcherReference(JMSDispatcherManager.
java:173)
at weblogic.jms.client.JMSConnection.setupDispatcher( JMSConnection.java:307)
at weblogic.jms.client.JMSConnectionFactory.setupJMSC onnection(JMSConnectionFactory.java:260
)
at weblogic.jms.client.JMSConnectionFactory.createCon nectionInternal(JMSConnectionFactory.ja
va:282)
at weblogic.jms.client.JMSConnectionFactory.createCon nection(JMSConnectionFactory.java:207)
at org.springframework.jms.core.JmsTemplate.createCon nection(JmsTemplate.java:760)
at org.springframework.jms.core.JmsTemplate.execute(J msTemplate.java:420)
at org.springframework.jms.core.JmsTemplate.send(JmsT emplate.java:477)

I have come across a couple of reference to pooling connection factories, but cannot nail down how to configure this. Any

This thread seemed to deal with something similar:

http://forum.springframework.org/arc...p/t-15733.html


help appreciated.

[drewcox]

I have a kind of workaround to this going at the moment. We swtiched to wrapping out Connection Factory in the SingleConnectionFactory, which seems to have the effect of caching the JNDI JMS resources. This appears to have avoided the OutOfMemoryError in the standalone JMS client (and make it quite a bit faster)

I would still like to understand what is really going on here and if there is a bug, resolve it.

[Costin Leau]

Maybe it does not apply but I had issues like that in the past and the problem was that too many connection/file handlers where used and the operating system allowed limit was reached. The error message was somewhat misleading but true - a new thread could not be created because there were no more handlers.
This means that for some reason the operating system resources are not freed - maybe your running everything inside one big transactions and resources linger on until the tx is ended. It's difficult but try profiling your application with 100 or 500 messages - you should be able to see some problems.

[szegedics]

My solution to this problem was that I have switched back to the J2EE JMS.
But once using the J2EE JMS I encountered this problem again. The problem was that before each message sending I opened a new connection and after sending to the queue I closed the connection, but for some reason the thread was still alive. You can imagine that if you send ~300 messages/second that soon you won't get any new available threads. The active threads will die after a while but not soon enough. On the other hand if I started a connection and sent the couple of 1000 messages and I have closed the connection only after this I haven't encountered any problems. So how I see the problem with the spring jms is the same, before each sending it creates a connection and after sending the message it closes the connection, but the thread is still alive. By opening and closing the connection for each message it costs a lot of performance.

HTH,
Csaba