Connection leak on lazy load

[splram]

Hi,

We are facing "connection leaks" issue on the models which are lazily loaded. This happens when there is a heavy traffic to the website. As a stop gap we are not loading the data lazily (i.e. lazy="false") and we stopped getting the connection leaks.

Any pointer in resolving the below issue is highly appreciated.

Below is the exception that is occurring when a model that is lazily loaded is accessed:
<A JDBC pool connection leak was detected. A connection leak occurs when a connection obtained from the pool was not closed explicitly by calling close() and then was disposed by the garbage collector and returned to the connection pool. The following stack trace at create shows where the leaked connection was created. Stack trace at connection create:

at weblogic.jdbc.wrapper.PoolConnection.init(PoolConn ection.java:56)
at weblogic.jdbc.pool.Driver.allocateConnection(Drive r.java:254)
at weblogic.jdbc.pool.Driver.connect(Driver.java:164)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver .java:509)
at weblogic.jdbc.jts.Driver.connect(Driver.java:139)
at weblogic.jdbc.common.internal.RmiDataSource.getCon nection(RmiDataSource.java:305)
at org.springframework.orm.hibernate3.LocalDataSource ConnectionProvider.getConnection(LocalDataSourceCo nnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnectio n(ConnectionManager.java:423)
at org.hibernate.jdbc.ConnectionManager.getConnection (ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQuerySta tement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement( Loader.java:1547)
at org.hibernate.loader.Loader.doQuery(Loader.java:67 3)
at org.hibernate.loader.Loader.doQueryAndInitializeNo nLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.loadCollection(Loader. java:1994)
at org.hibernate.loader.collection.CollectionLoader.i nitialize(CollectionLoader.java:36)
at org.hibernate.persister.collection.AbstractCollect ionPersister.initialize(AbstractCollectionPersiste r.java:565)
at org.hibernate.event.def.DefaultInitializeCollectio nEventListener.onInitializeCollection(DefaultIniti alizeCollectionEventListener.java:60)
at org.hibernate.impl.SessionImpl.initializeCollectio n(SessionImpl.java:1716)
at org.hibernate.collection.AbstractPersistentCollect ion.forceInitialization(AbstractPersistentCollecti on.java:454)
at org.hibernate.engine.StatefulPersistenceContext.in itializeNonLazyCollections(StatefulPersistenceCont ext.java:785)
at org.hibernate.loader.Loader.doQueryAndInitializeNo nLazyCollections(Loader.java:241)
at org.hibernate.loader.Loader.loadEntity(Loader.java :1860)
at org.hibernate.loader.entity.AbstractEntityLoader.l oad(AbstractEntityLoader.java:48)
at org.hibernate.loader.entity.AbstractEntityLoader.l oad(AbstractEntityLoader.java:42)
at org.hibernate.persister.entity.AbstractEntityPersi ster.load(AbstractEntityPersister.java:3042)
at org.hibernate.event.def.DefaultLoadEventListener.l oadFromDatasource(DefaultLoadEventListener.java:39 5)
at org.hibernate.event.def.DefaultLoadEventListener.d oLoad(DefaultLoadEventListener.java:375)
at org.hibernate.event.def.DefaultLoadEventListener.l oad(DefaultLoadEventListener.java:139)
at org.hibernate.event.def.DefaultLoadEventListener.o nLoad(DefaultLoadEventListener.java:98)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImp l.java:878)
at org.hibernate.impl.SessionImpl.immediateLoad(Sessi onImpl.java:836)
at org.hibernate.proxy.AbstractLazyInitializer.initia lize(AbstractLazyInitializer.java:66)
at org.hibernate.proxy.AbstractLazyInitializer.getImp lementation(AbstractLazyInitializer.java:111)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitialize r.invoke(CGLIBLazyInitializer.java:150)
at com.jnj.hp2.ehealth.cms.model.NavigationRootItem$$ EnhancerByCGLIB$$e5620ca0.getNavigation(<generated >)
at com.jnj.hp2.ehealth.janssencilag.service.impl.JCUr lPatternNavItemIdResolver.resolveNavigationItemId( JCUrlPatternNavItemIdResolver.java:244)
at com.jnj.hp2.ehealth.janssencilag.web.interceptor.N avigationInterceptor.postHandle(NavigationIntercep tor.java:137)
at org.springframework.web.servlet.DispatcherServlet. doDispatch(DispatcherServlet.java:


Thanks in advance,
Ram

[Marten Deinum]

Use [ code][/code ] tags when posting code/xml/stacktraces.

Judging from the stacktrace, improper transactionmanagement/configuration. Also I don't see the full stacktrace or you don't use something like the OpenSessionInViewFilter to allow for lazy loading and correct handling of sessions in the web layer.

[splram]

Hi Marten,

Thanks for looking into this.

We indeed use the OpenSessionInViewInterceptor, I thought I had pasted the complete stack trace. However, I am pasting the trace from where last I had left (The editor is not allowing me to paste the full trace in single post).

I am also posting the configurations that we have used.

     <bean id="dataSource"
        class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiName">
            <value>${ehealth.cms.db.datasource}</value>
        </property>
        <property name="jndiEnvironment">
            <props>
                <prop key="java.naming.factory.initial">
                    weblogic.jndi.WLInitialContextFactory
                </prop>
            </props>
        </property>
    </bean>

    <bean id="sessionFactory"
        class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />
        <property name="mappingLocations">
            <list>
                <value>classpath:com/jnj/hp2/ehealth/cms/model/AbstractContent.hbm.xml</value>
            </list>
        </property>
    </bean>

    <bean id="txManagerCMS"
		class="org.springframework.orm.hibernate3.HibernateTransactionManager">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>

	<bean name="openSessionInViewInterceptor"
		class="org.springframework.orm.hibernate3.support.OpenSessionInViewInterceptor">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>


	<bean id="handlerMapping"
		class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping">
		<property name="order" value="1"/>
		<property name="interceptors">
			<list>
				<ref bean="openSessionInViewInterceptor" />

Below is the full stack trace

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:865)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:793)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:476)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:431)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1006)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:130)
	at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:107)
	at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:78)
	at com.jnj.hp2.core.web.filter.HpUrlRewriteFilter.doFilter(HpUrlRewriteFilter.java:268)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at com.jnj.hp2.security.web.filter.ChangePasswordFilter.doFilterInternal(ChangePasswordFilter.java:112)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
	at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.securechannel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:138)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at com.jnj.hp2.ehealth.core.web.filter.SiteKeyFilter.doFilterInternal(SiteKeyFilter.java:155)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.securechannel.ChannelFromOnHostEntryPointFilter.doFilterInternal(ChannelFromOnHostEntryPointFilter.java:28)
	at com.jnj.hp2.ehealth.core.wrapper.HttpsSupportRequestWrapperFilter.doFilterInternal(HttpsSupportRequestWrapperFilter.java:125)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at com.jnj.hp2.core.web.filter.BootstrapFilter.doFilterInternal(BootstrapFilter.java:71)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at com.jnj.hp2.core.web.filter.BootstrapFilter.doFilterInternal(BootstrapFilter.java:71)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
	at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.jnj.hp2.core.web.filter.SimpleSingleSignOnFilter.doFilter(SimpleSingleSignOnFilter.java:92)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6724)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3764)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2644)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
>

Best Regards,
Ram

[splram]

Hi Marten,

Another thing worth noting is that, when the data is accessed using lazy loading, the data is being retrieved as non-transactional retrieval, this is evident by looking into the below part of the stack trace:

I am wondering when a model is accessed in a transactional way, what is the possibility (or how) the other data of the same model (that is to be lazily loaded) gets accessed non-transactional?

I am not able to re-produce this scenario and hence I am going by some hypothesis and I would appreciate any help in this regard.

weblogic.jdbc.wrapper.PoolConnection.init(PoolConnection.java:56)
	at weblogic.jdbc.pool.Driver.allocateConnection(Driver.java:254)
	at weblogic.jdbc.pool.Driver.connect(Driver.java:164)
	at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:509)
	at weblogic.jdbc.jts.Driver.connect(Driver.java:139)
	at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:305)
	at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)

[Marten Deinum]

You are using the interceptor instead of the filter. Judging from the earlier stacktrace your issues arise in a Interceptor. I suggest switching from the OpenSessionInViewInterceptor to the OpenSessionInViewFilter which opens a session on the first request and closes it after view rendering. That might solve some of the issues.

Also what is it that you are doing inside the NavigationInterceptor and JCUrlPatternNavItemIdResolver?

I am wondering when a model is accessed in a transactional way, what is the possibility (or how) the other data of the same model (that is to be lazily loaded) gets accessed non-transactional?

Well basically when your service method is over the transaction ends, however the session remains open so the remained of the data (the lazy data) can be retrieved outside of a transaction. However because there already is a session it shouldn't need a new connection (that should still be open).

[splram]

Hi Marten,

The JCUrlPatternNavItemIdResolver is a simple class that gets invoked from NavigationInterceptor. The NavigationInterceptor is configured in the handlerMapping bean after the openSessionInViewInterceptor bean is configured.

In the handler mapping there are other references of OSIVI e.g. productOpenSessionInViewInterceptor and roleUserTypeInterceptor.

The productOpenSessionInViewInterceptor operates on a different session factory where as the roleUserTypeInterceptor and navigationInterceptor accesses the data that is retrieved using OSIVI.

	<bean id="handlerMapping"
		class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping">
		<property name="order" value="1"/>
		<property name="interceptors">
			<list>
				<ref bean="openSessionInViewInterceptor" />
				<ref bean="productOpenSessionInViewInterceptor" />
				<ref bean="roleUserTypeInterceptor" />
				<ref bean="navigationInterceptor" />

[Marten Deinum]

The JCUrlPatternNavItemIdResolver is a simple class that gets invoked from NavigationInterceptor. The NavigationInterceptor is configured in the handlerMapping bean after the openSessionInViewInterceptor bean is configured.

It seems that this class is for some reason the culprit (at least that is where the stacktrace occurs). Is possible to see the code?

Another thing I notice is that you have set the order attribute on the HandlerMapping, if you have multiple HandlerMappings you need also specify all the interceptors on the other HandlerMappings (or at least the OpenSessionInView ones).

[splram]

I agree, the JCUrlPatternNavItemIdResolver is the place where the exception starts, however, there were other places (controllers) where the connection leak has occurred, but when we specified lazy="false" for the model (data) used in those controllers the connection leak error disappeared.

As far as the order is concerned I have only one handlerMapping in my configuration.

The place where I get the connection leak in the JCUrlPatternNavItemIdResolver is in the below code

if ("leftnavigation".equals(topic) && navItm.getAncestor().getNavItemId().longValue() != navItm.getRootItem().getNavItemId().longValue()) {

As you can see in the stack trace the above piece of code in bold is performing a lazy load and at this place the connection leak is occurring.

Thanks,
Ram

[Marten Deinum]

The navItems are loaded in the same request/session or do they come from somewhere else?

You could try setting the useTransactionAwareDataSource to true, that might help in discovering the already ongoing connection for the current session.

Another question is how are you declaring your transactions, you have multiple SessionFactories so I wonder how you declare them and if you have multiple SessionFactories you should also have multiple HibernateTransactionManagers (or use JTA if 2 session factories can be used in a single transaction).

[splram]

Yeah, the navItems are loaded in the same request i.e. in navigationInterceptor.

"You could try setting the useTransactionAwareDataSource to true, that might help in discovering the already ongoing connection for the current session."

Can you please elaborate how the above will help in discovering?

Yes we have multiple session factories and also multiple hibernate transaction managers. And yes, in a request there are two session factories that are getting used. Do you think using two session factories (two OSIVI) can cause connection leaks? If "yes" how can we best handle the situation where multiple session factories can be used in one request flow?

Once again thanks for your patience.