RMI + Hibernate + dbcp results in hanging

[wengweng]

Hi,

I have a spring-webapp using RMI webservice in combination with Hibernate and dbcp/c3p0. The problem is, when I enter the webapp through rmi a few times (5 aprox) to access database info via hibernate, the webapp hangs. Probably because it cannot access the database anymore.

This problem only occurs when I use rmi. When I access the website using the same backend, there is no problem.

mvc->manager->hibernateDao: This works fine
rmi->manager->hibernateDao: Works about 5 times, then all data access is blocked

I have tried several settings with dbcp/c3p0 (even default), but nothing seems to help.

There is no problem when I use spring's DriverManagerDataSource. Then rmi works fine.

I'm kind of lost here so I hope someone can help me with this problem.

[sjivan]

Try getting a thread dump when your server hangs (Crtl + Break on windows). Then analyze the thread dump to narrow down where the app is hanging.

Sanjiv

[wengweng]

Below you will find a stacktrace using c3p0. It seems that it hangs when it tries to acquire a pooled connection.

Edit:
I have debugged a bit further and it looks like the opened connections are not returned to the pool. How could this be?

System Thread [RMI TCP Connection(29)-10.31.100.105] (Stepping)
	BasicResourcePool.awaitAcquire(long) line: 968
	BasicResourcePool.checkoutResource(long) line: 208
	C3P0PooledConnectionPool.checkoutPooledConnection() line: 260
	PoolBackedDataSource.getConnection() line: 94
	ComboPooledDataSource.getConnection() line: 521
	LocalDataSourceConnectionProvider.getConnection() line: 80
	ConnectionManager.openConnection() line: 417
	ConnectionManager.getConnection() line: 144
	BatchingBatcher(AbstractBatcher).prepareQueryStatement(String, boolean, ScrollMode) line: 105
	CriteriaLoader(Loader).prepareQueryStatement(QueryParameters, boolean, SessionImplementor) line: 1561
	CriteriaLoader(Loader).doQuery(SessionImplementor, QueryParameters, boolean) line: 661
	CriteriaLoader(Loader).doQueryAndInitializeNonLazyCollections(SessionImplementor, QueryParameters, boolean) line: 224
	CriteriaLoader(Loader).doList(SessionImplementor, QueryParameters) line: 2145
	CriteriaLoader(Loader).listIgnoreQueryCache(SessionImplementor, QueryParameters) line: 2029
	CriteriaLoader(Loader).list(SessionImplementor, QueryParameters, Set, Type[]) line: 2024
	CriteriaLoader.list(SessionImplementor) line: 94
	SessionImpl.list(CriteriaImpl) line: 1533
	CriteriaImpl.list() line: 283
	DeviceDao(BaseDaoHibernate<T,TID,TSearchCommand>).getBySearchCommand(TSearchCommand) line: 111
	DeviceManagerImpl.getDeviceBySearchCommand(DeviceSearchCommand) line: 469
	DeviceManagerImpl.identifyDeviceByUserAgent(String) line: 72
	GeneratedMethodAccessor54.invoke(Object, Object[]) line: not available
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
	Method.invoke(Object, Object...) line: 585
	AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 335
	ReflectiveMethodInvocation.invokeJoinpoint() line: 181
	ReflectiveMethodInvocation.proceed() line: 148
	RemoteInvocationTraceInterceptor.invoke(MethodInvocation) line: 68
	ReflectiveMethodInvocation.proceed() line: 170
	JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 176
	$Proxy1.identifyDeviceByUserAgent(String) line: not available
	NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
	NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
	Method.invoke(Object, Object...) line: 585
	RemoteInvocation.invoke(Object) line: 179
	DefaultRemoteInvocationExecutor.invoke(RemoteInvocation, Object) line: 33
	RmiServiceExporter(RemoteInvocationBasedExporter).invoke(RemoteInvocation, Object) line: 76
	RmiServiceExporter(RmiBasedExporter).invoke(RemoteInvocation, Object) line: 72
	RmiInvocationWrapper.invoke(RemoteInvocation) line: 62
	GeneratedMethodAccessor51.invoke(Object, Object[]) line: not available
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
	Method.invoke(Object, Object...) line: 585
	UnicastServerRef.dispatch(Remote, RemoteCall) line: 294
	Transport$1.run() line: 153
	AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]
	TCPTransport(Transport).serviceCall(RemoteCall) line: 149
	TCPTransport.handleMessages(Connection, boolean) line: 460
	TCPTransport$ConnectionHandler.run() line: 701
	Thread.run() line: 595

[sjivan]

Are you closing the Hibernate sessions?

[wengweng]

From what i've read, Spring's hibernate template should manage this for me. And please note, this problem only occurs when I use RMI.

[wengweng]

After further debugging I've found a difference between normal web/mvc access and rmi-webservice.

The difference can be found in Spring's hibernateTemplate execute function. When I access via rmi it doesn't see it as an existing transaction, while it does when I access via web/mvc.

org.springframework.orm.hibernate3.HibernateTempla te:

public Object execute(HibernateCallback action, boolean exposeNativeSession) throws DataAccessException {
	Session session = getSession();
	boolean existingTransaction = SessionFactoryUtils.isSessionTransactional(session, getSessionFactory());

.....
	finally {
		if (existingTransaction) {
			//Web/mvc
			logger.debug("Not closing pre-bound Hibernate Session after HibernateTemplate");
			disableFilters(session);
			if (previousFlushMode != null) {
				session.setFlushMode(previousFlushMode);
			}
		}
		else {
			//rmi
			SessionFactoryUtils.releaseSession(session, getSessionFactory());
		}
	}
}

So when using rmi it releases the session. What could be the cause/reason for this difference?

[Andreas Senft]

I guess the reason is, that the Session is held in a ThreadLocal variable. On using RMI the threadlocal context is not being reestablished, as it is dependent on the current VM and a RMI call usually points to another process.

Regards,
Andreas

[wengweng]

I guess the reason is, that the Session is held in a ThreadLocal variable. On using RMI the threadlocal context is not being reestablished, as it is dependent on the current VM and a RMI call usually points to another process.

Regards,
Andreas

Then why is it, that this results in not releasing the connections of the connection pool?

[Andreas Senft]

Then why is it, that this results in not releasing the connections of the connection pool?

Good question. I would also expect the connections to be released again.
Maybe try debugging the session closing initiated from the HibernateTemplate to see what actually happens.

Regards,
Andreas

[wengweng]

See Edit below
[not true]
I'm a few steps closer now. I have discovered that this occurs when I perform 2 RMI calls right after each other. I see 3 connections being opened and only 2 of them are closed. When I remove one of the functions, the sessions/connections are closed correctly.

//manager is a RmiProxyFactoryBean
//Connection pool trouble. 2 of 3 connections are closed
void someFunction(){
	Object result1 = manager.doRmiCall1();
	Object result2 = manager.doRmiCall2();
}

//No problem here
void someFunction2(){
	Object result1 = manager.doRmiCall1();
}

Could it be a thread synchronization problem? Anyway, I will look further into this...
[/not true]

Edit: In second example Hibernate sessions are closed correctly, but connections are still leaking...