Problem with Transaction Management in multithreaded environment.

[sudhirshinde]

Hi

I am using programatic transaction management and it works fine when I test it in single user environment. But in production when thousands of users hitting the same method at same time, it fails while doing 'commit' and gives me following error:

"Transaction is already completed - do not call commit or rollback more than once per transaction"


Here is the code which gets me a transaction:

DefaultTransactionDefinition def = new DefaultTransactionDefinition();
def.setPropagationBehavior(TransactionDefinition.P ROPAGATION_REQUIRED);
status = getTransactionManager().getTransaction(def);


Here is my xml showing datasource and transaction manager:

<bean id="campaignDataSource" class="org.springframework.jndi.JndiObjectFactoryB ean" singleton="true">
<property name="jndiName" value="java:comp/env/campaignRef" />
</bean>


<bean id="campaignTransactionManager" class="org.springframework.jdbc.datasource.DataSou rceTransactionManager" singleton="false">
<property name="dataSource">
<ref bean="campaignDataSource"/>
</property>
</bean>



This error comes when I call

getTransactionManager().commit(status);


And this works fine with few users hitting my site, but when load increases I start getting this error.

Here is the complete error stack.

2007-08-16 10:35:12,446 ERROR - SVC ID[221503638] CLIENT[PARTS] METHOD[getLead] service failed with an unexpected exception[Transaction is already completed - do not call commit or rollback more than once per transaction]
org.springframework.transaction.IllegalTransaction StateException: Transaction is already completed - do not call commit or rollback more than once per transaction
at org.springframework.transaction.support.AbstractPl atformTransactionManager.rollback(AbstractPlatform TransactionManager.java(Inlined Compiled Code))
at com.sears.os.business.ABaseBusinessBean.rollbackWo rk(ABaseBusinessBean.java(Inlined Compiled Code))
at com.sears.os.campaign.business.CampaignBusinessBea nNew.getLead(CampaignBusinessBeanNew.java(Compiled Code))
at com.sears.os.campaign.service.CampaignService.getL ead(CampaignService.java(Compiled Code))
at sun.reflect.GeneratedMethodAccessor119.invoke(Unkn own Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compil ed Code))
at org.apache.soap.server.RPCRouter.invoke(Unknown Source)
at org.apache.soap.providers.RPCJavaProvider.invoke(U nknown Source)
at org.apache.soap.server.http.RPCRouterServlet.doPos t(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet .java(Compiled Code))
at javax.servlet.http.HttpServlet.service(HttpServlet .java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.StrictServletInsta nce.doService(StrictServletInstance.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.StrictLifecycleSer vlet._service(StrictLifecycleServlet.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServicingServletSt ate.service(StrictLifecycleServlet.java:333)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleSer vlet.service(StrictLifecycleServlet.java(Inlined Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletInstance.se rvice(ServletInstance.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ValidServletRefere nceState.dispatch(ValidServletReferenceState.java( Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletInstanceRef erence.dispatch(ServletInstanceReference.java(Inli ned Compiled Code))
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispat cher.handleWebAppDispatch(WebAppRequestDispatcher. java(Compiled Code))
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispat cher.dispatch(WebAppRequestDispatcher.java(Compile d Code))
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispat cher.forward(WebAppRequestDispatcher.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.WebAppInvoker.doForwar d(WebAppInvoker.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.WebAppInvoker.handleIn vocationHook(WebAppInvoker.java(Compiled Code))
at com.ibm.ws.webcontainer.cache.invocation.CachedInv ocation.handleInvocation(CachedInvocation.java(Com piled Code))
at com.ibm.ws.webcontainer.cache.invocation.Cacheable InvocationContext.invoke(CacheableInvocationContex t.java(Compiled Code))
at com.ibm.ws.webcontainer.srp.ServletRequestProcesso r.dispatchByURI(ServletRequestProcessor.java(Compi led Code))
at com.ibm.ws.webcontainer.oselistener.OSEListenerDis patcher.service(OSEListener.java(Compiled Code))
at com.ibm.ws.webcontainer.http.HttpConnection.handle Request(HttpConnection.java(Compiled Code))
at com.ibm.ws.http.HttpConnection.readAndHandleReques t(HttpConnection.java(Compiled Code))
at com.ibm.ws.http.HttpConnection.run(HttpConnection. java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.j ava(Compiled Code))



I would appreciate any help on this.

Thanks
Sudhir

[Alarmnummer]

Can you post the complete code of the method/class (you can omit business related details) of creating and committing?

Maybe you are sharing variables between threads (like the status.. you are not declaring it when you use it.. so this could be the cause).

And why is the transactionmanager not a singleton?

[sudhirshinde]

Here is the code from business method:

try {
beginWork();


//........some inserts and updates....


commitWork();


} catch (DataAccessException dae) {
rollbackWork();
throw dae;
}


And here is the abstract class which is extended by my business bean having above method.


public abstract class ABaseBusinessBean {

private String userId;
private DataSourceTransactionManager transactionManager;
protected final Log logger = LogFactory.getLog(getClass());
private TransactionStatus status;

/**
* Returns the transactionMgr.
* @return DataSourceTransactionManager
*/
public DataSourceTransactionManager getTransactionManager() {
return transactionManager;
}

/**
* Sets the transactionMgr.
* @param transactionMgr The transactionMgr to set
*/
public void setTransactionManager(DataSourceTransactionManager transactionManager) {
this.transactionManager = transactionManager;
}

public void beginWork() {
DefaultTransactionDefinition def = new DefaultTransactionDefinition();
def.setPropagationBehavior(TransactionDefinition.P ROPAGATION_REQUIRED);
status = getTransactionManager().getTransaction(def);
}

public void commitWork() {
getTransactionManager().commit(status);
}

public void rollbackWork() {
getTransactionManager().rollback(status);
}

}


And why is the transactionmanager not a singleton?
I tried to resolve the issue by removing singleton for all objects.

Thanks
Sudhir

Can you post the complete code of the method/class (you can omit business related details) of creating and committing?

Maybe you are sharing variables between threads (like the status.. you are not declaring it when you use it.. so this could be the cause).

And why is the transactionmanager not a singleton?

[Alarmnummer]

The cause of your (race) problem is that Status is shared between threads. So multiple threads could commit on the same status object, and in some cases a commit is never executed. So that is why you have problems.

Solution?
- use a threadlocal to store the status
- don't leave the method, so no seperate commit and rollback methods.. but create and commit the status in a single method
- return the status to the callee so that it can send it when it is going to commit.

The issue all solutions have in common is that no state is stored in that service, so no sharing of mutable objects. So no race problems.
So loose the status member-field in your object..

I would go for the second one for most cases. Passing transaction status objects normally is not something you want.

[sudhirshinde]

Thanks Alarmnummer for looking into this issue.

I still have one question. I am not understanding why "Status" will get shared across threads. It's a private NON-static object defined in class ABusinessBean which is extended by MyBusinessBean and MyBusinessBean is a NON-singleton class. So I guess every request will get their own copy of MyBusinessBean and hence own copy of "Status".

sorry I am asking a basic java question now.

thanks
sudhir

[Alarmnummer]

Thanks Alarmnummer for looking into this issue.

I still have one question. I am not understanding why "Status" will get shared across threads. It's a private NON-static object defined in class ABusinessBean which is extended by MyBusinessBean and MyBusinessBean is a NON-singleton class. So I guess every request will get their own copy of MyBusinessBean and hence own copy of "Status".

Ok.. that changes the situation. Normally services are stateless, so that is what I assumed here (to assume is the mother of all fuck ups )

My gut feeling still says that somehow the MyBusinessBeans are shared between threads. Because that would explain the issue you have.

[Jörg Heinicke]

This error comes when I call getTransactionManager().commit(status);

Your stacktrace says ABaseBusinessBean.rollbackWork()! Can it be that the commit goes through successfully, but afterwards it's tried to rollback the work though?

Joerg

[dejanp]

I'd suggest using a TransactionTemplate instead of try..begin...commit...catch...rollback. It's too error prone.