outof memory issue

[leeo]

Hi

I have an app that is using spring integration to connect over tcp to a 3rd party system. Is needs to send the message over tcp get the response and send it back out of my system. I need to be sure that i send a request over tcp and wait for the response before sending another.

To test this i send http requests and use the information to generate an object and send it through the tcp connection.

I am getting an increasing number of threads associated with my applications process that eventually results in this.

19-May-2010 09:40:28 org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet mybenchapp threw exception
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.$$YJP$$start0(Native Method)
    at java.lang.Thread.start0(Thread.java)
    at java.lang.Thread.start(Thread.java:597)
    at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
    at java.util.concurrent.ThreadPoolExecutor.prestartCoreThread(ThreadPoolExecutor.java:1373)
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:223)
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:366)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.schedule(ReschedulingRunnable.java:73)
    at org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.schedule(ThreadPoolTaskScheduler.java:164)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.doStart(AbstractPollingEndpoint.java:187)
    at org.springframework.integration.endpoint.AbstractEndpoint.start(AbstractEndpoint.java:84)
    at org.springframework.integration.config.ConsumerEndpointFactoryBean.start(ConsumerEndpointFactoryBean.java:179)
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:164)
    at org.springframework.context.support.DefaultLifecycleProcessor.access$1(DefaultLifecycleProcessor.java:153)
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:331)
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:142)
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:107)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:885)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:426)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
    at com.securetrading.controllers.HelloController.sendToGateway(HelloController.java:114)
    at com.securetrading.controllers.HelloController.handleRequest(HelloController.java:58)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:619) 

when i profile my app i can see that loads of threads are getting crated with the name task-scheduler-1 through to 9. after looking for a reference to this name in the code i found that DefaultConfigurationBeanFactoryPostProcessor does this work when i dont define my own taskScheduler. what im not sure about is why there are hundereds of these threads with the same name task-scheduler-1 through to 9. from what i can see ThreadPoolExecutor.addIfUnderCorePoolSize is not getting met but the pool size default is 10.

Can anyone tell me why this is happening?

this is my config

              <si:gateway id="gateway"
                service-interface="my.class.GatewayInterface"
                default-request-channel="input"
                default-reply-channel="transformedChannel"/>
    
    <si:channel id="input" />

    
    <si:service-activator input-channel="input"
                          output-channel="requestChannel" 
                          ref="transactionEncoder" 
                          method="processTransaction" />

    <bean id="transactionEncoder"
          class="my.class.EncodingService" />



                 <ip:outbound-gateway id="tcpOutGateway" 
                          host="10.3.21.28"
                         port="40033" 
                         message-format="custom" 
                         request-channel="requestChannel"
                         reply-channel="replyChannel" 
                         custom-socket-writer-class-name="my.class.SocketWriter"
                         custom-socket-reader-class-name="my.class.SocketReader"/>

    <si:channel id="replyChannel">
        <si:queue />
        
    </si:channel>

<si:transformer ref="transactionDecoder" 
                    input-channel="replyChannel"
                    output-channel="transformedChannel"  
                    method="transform"/>
 
 <bean id="transactionDecoder"
          class="my.class.DecodingService" />
    

<si:channel id="transformedChannel">
    <si:queue capacity="5"/>
    </si:channel>


<si:poller max-messages-per-poll="1" 
               id="defaultPoller"
               default="true">
        <si:interval-trigger interval="30000" />
    </si:poller> 

My thoughts are that this might be something to do with the way my poller is working.

Im using 2.0.0M4

[Mark Fisher]

If you only see threads with names between 1 and 9, then it seems like you are indeed staying under 10 threads. Are you seeing the number of threads elsewhere (e.g. JConsole) or just in the log?

Also, what version of Spring Integration are you using?... and is this a production application or something in-development?

Thanks,
Mark

[leeo]

im using 2.0.0M4 this is in developement at the moment so i can change any of my config or even my solution as a whole. when i look at the systems processes i see my tomcat java.exe running the more requests i send to the app the more the number of threads grows and blows up out of memory at about 2500. Its as if no threads are ever released

im using YourKit Java Profiler and i see like 300 threads each with the name task-scheduler-1, another 300 odd for task-scheduler-2 and so on up to 9. should i not just have 9 in total or am i missing something

[Mark Fisher]

Those are thread names, so they are unique, right? There should never be more than one thread named e.g. "task-scheduler-5". Are you sure that you are seeing hundreds of *active* threads?

By the way, you might want to consider our new TCP gateway support:
http://static.springsource.org/sprin...l#tcp-gateways

[Marten Deinum]

Judging from the stacktrace you are creating your own instance of an ApplicationContext... For EVERY request that comes in... Why? If you want to run out of resources that is the way to go...

    
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:885)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:426)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

[leeo]

As far as i am aware i was using the new tcp gateway support i see this in the log when its running

10457 [http-8080-1] DEBUG org.springframework.integration.ip.tcp.SimpleTcpNetOutboundGateway  - handler 'org.springframework.integration.ip.tcp.SimpleTcpNetOutboundGateway#0' sending reply Message: [Payload=[B@23a1fd][Headers={$replyChannel=org.springframework.integration.channel.MessageChannelTemplate$TemporaryReplyChannel@1f025cb, $errorChannel=org.springframework.integration.channel.MessageChannelTemplate$TemporaryReplyChannel@1f025cb, $timestamp=1274273603242, $history=[gateway, channel:input, channel:requestChannel, tcpOutGateway], $id=11abb638-2836-4dc7-9559-b89f79193f8c}] 

as for the threads issue i see them in yourkit java profiler and jconsole

please review the attached images for screen shots of what i see

[Marten Deinum]

See my previous post...

[leeo]

Ahh this could indeed be bad i will try to stop doing that now marten.


what i had in my code is

ApplicationContext context = new ClassPathXmlApplicationContext("si-config.xml");
GatewayInterface gateway =  (GatewayInterface) context.getBean("gateway"); 

i take it i only need to do this once and not for every request