Server failing under load using transaction attributes.

[ctran]

Hello,

We are seeing our server (resin) failing under heavy load. CPU reaches 100% and never recovers. I believe that we have isolated the problem to something with Spring attribute autoprxy. When we remove the bean config file, the server scales as expected. The file is below. Also a thread dump after the server locks always contains a get from the attribute cache HashMap. I have attached the dump below.

Any help would be appreciated.

Thanks You

Thread Dump ============
"tcpConnection-6802-19" daemon prio=5 tid=0x3513e818 nid=0xa60 runnable [372ff000..372ffdb0]
at java.util.HashMap.get(HashMap.java:325)
at org.springframework.transaction.interceptor.Abstra ctFallbackTransactionAttributeSource.getTransactio nAttribute(AbstractFallbackTransactionAttributeSou rce.java:77)
at org.springframework.transaction.interceptor.Transa ctionAttributeSourceAdvisor.matches(TransactionAtt ributeSourceAdvisor.java:51)
at org.springframework.aop.support.AopUtils.canApply( AopUtils.java:189)
at org.springframework.aop.support.AopUtils.canApply( AopUtils.java:201)
at org.springframework.aop.framework.autoproxy.Abstra ctAdvisorAutoProxyCreator.findEligibleAdvisors(Abs tractAdvisorAutoProxyCreator.java:83)
at org.springframework.aop.framework.autoproxy.Abstra ctAdvisorAutoProxyCreator.getInterceptorsAndAdviso rsForBean(AbstractAdvisorAutoProxyCreator.java:64)
at org.springframework.aop.framework.autoproxy.Abstra ctAutoProxyCreator.postProcessAfterInitialization( AbstractAutoProxyCreator.java:209)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.applyBeanPostProcessors AfterInitialization(AbstractAutowireCapableBeanFac tory.java:182)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactory.java:285)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactory.java:204)
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:212)
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:136)
at org.springframework.context.support.AbstractApplic ationContext.getBean(AbstractApplicationContext.ja va:426)
at com.opensymphony.xwork.spring.SpringObjectFactory. buildBean(SpringObjectFactory.java:56)
at com.opensymphony.xwork.ObjectFactory.buildAction(O bjectFactory.java:77)
at com.opensymphony.xwork.DefaultActionInvocation.cre ateAction(DefaultActionInvocation.java:199)
at com.opensymphony.xwork.DefaultActionInvocation.ini t(DefaultActionInvocation.java:272)
at com.opensymphony.xwork.DefaultActionInvocation.<in it>(DefaultActionInvocation.java:65)
at com.opensymphony.xwork.DefaultActionInvocation.<in it>(DefaultActionInvocation.java:58)
at com.opensymphony.xwork.DefaultActionProxyFactory.c reateActionInvocation(DefaultActionProxyFactory.ja va:32)
at com.opensymphony.xwork.DefaultActionProxy.prepare( DefaultActionProxy.java:124)
at com.opensymphony.xwork.DefaultActionProxy.<init>(D efaultActionProxy.java:75)
at com.opensymphony.xwork.DefaultActionProxyFactory.c reateActionProxy(DefaultActionProxyFactory.java:45 )
at com.opensymphony.webwork.dispatcher.ServletDispatc her.serviceAction(ServletDispatcher.java:271)
at com.opensymphony.webwork.dispatcher.ServletDispatc her.service(ServletDispatcher.java:243)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:103)
at com.caucho.server.http.FilterChainServlet.doFilter (FilterChainServlet.java:96)
at com.opensymphony.webwork.lifecycle.RequestLifecycl eFilter.doFilter(RequestLifecycleFilter.java:67)
at com.caucho.server.http.FilterChainFilter.doFilter( FilterChainFilter.java:88)
at com.gearworks.servlet.SecurityFilter.doFilter(Secu rityFilter.java:76)
at com.caucho.server.http.FilterChainFilter.doFilter( FilterChainFilter.java:88)
at com.caucho.server.http.Invocation.service(Invocati on.java:315)
at com.caucho.server.http.CacheInvocation.service(Cac heInvocation.java:135)
at com.caucho.server.http.RunnerRequest.handleRequest (RunnerRequest.java:346)
at com.caucho.server.http.RunnerRequest.handleConnect ion(RunnerRequest.java:274)
at com.caucho.server.TcpConnection.run(TcpConnection. java:139)
at java.lang.Thread.run(Thread.java:



attribute bean config file ================
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">

<beans>

<description>
Generic autoproxy definitions enabling declarative transaction management,
based on Commons Attributes attributes. The application context must define
a single PlatformTransactionManager for autoproxying to work.

This file is generic, and not specific to JPetStore. You can use it unchanged
as an application context definition file for your own applications to get
attribute-driven declarative transaction management.

The power of this approach is more apparent when you have many transactional
business objects, not just one as in this simple application.

Add more Advisor definitions if you want, for additional declarative services.
</description>

<!--
This bean is a postprocessor that will automatically apply relevant advisors
to any bean in child factories.
-->
<bean id="autoproxy"
class="org.springframework.aop.framework.autoproxy .DefaultAdvisorAutoProxyCreator">
<property name="proxyTargetClass"><value>true</value></property>
</bean>

<!--
Commons Attributes Attributes implementation. Replace with another
implementation of org.springframework.metadata.Attributes to source
attributes from a different source.
-->
<bean id="attributes"
class="org.springframework.metadata.commons.Common sAttributes"/>

<bean id="transactionAttributeSource"
class="org.springframework.transaction.interceptor .AttributesTransactionAttributeSource">
<constructor-arg>
<ref bean="attributes"/>
</constructor-arg>
</bean>

<bean id="transactionInterceptor"
class="org.springframework.transaction.interceptor .TransactionInterceptor">
<property name="transactionManager">
<ref bean="transactionManager"/>
</property>
<property name="transactionAttributeSource">
<ref bean="transactionAttributeSource"/>
</property>
</bean>

<!--
AOP advisor that will provide declarative transaction management
based on attributes.
It's possible to add arbitrary custom Advisor implementations as
well, and they will also be evaluated and applied automatically.
-->
<bean id="transactionAdvisor"
class="org.springframework.transaction.interceptor .TransactionAttributeSourceAdvisor">
<constructor-arg>
<ref bean="transactionInterceptor"/>
</constructor-arg>
</bean>
</beans>

[Rod Johnson]

Obviously we'll treat this as a high priority. So if you could give a little more information that would be helpful.

I suspect it's really a usae issue.

It appears from your stack trace that you're getting a prototype (non-singleton) bean. Why are you not using a shared bean instance for your service that needs declarative tx mgt? If it can't be made threadsafe, consider using pooling with one of the pooling TargetSource implementations shipped with Spring.

This way, the autoproxy infrastructure needs to analyse all transaction attributes and iterate over all advisors that could apply to the instance etc. every time one of the these beans is created and invoked.

Rgds
Rod

[ctran]

Rod, Thanks for the quick response.

The bean that is being retrieved is an xWork Action. That is why it is a prototype. We are using Jakarta Attributes to declare our transaction attributes. The xWork actions do not include any attributes.

We seam to have fixed the issue by modifying the AbstractFallbackTransactionAttributeSource class to use a SynchronizedMap for the cache. Should this be synchronized or do you think we have an issue with how we are configuring and/or using our xWork Aciton beans?

[Rod Johnson]

I would suggest that your XWork Actions reference a singleton (or pooled) service object doing tx mgt, rather than themselves be transactionally autoproxied.

[ctran]

We are currently doing what you suggest. Our XWork action contains a service object that is wired as a singleton through the bean factory. We have transaction attributes on the service methods but not the .

The Actions are still being run through the AbstractFallbackTransactionAttributeSource.getTran sactionAttribute() method when they are created but they always return null. We really don't need them to run through there because we know our Actions have no transaction attributes. Am I correct to think that all object go through this method when they are created? Is there a way we can specify a pattern to not do this?

I am still interested in your opinion on synchronizing the cache Map field of the AbstractFallbackTransactionAttributeSource class. When we do this we don't have any more problems under load.

Thank You.
Cuong

[Rod Johnson]

Can you please send me a simplified version of the problem without any of your IP in it? E.g. just something that repeatedly obtains a prototype from JUnit without XWork. If you don't have time for this, perhaps you could send me (offline) a bit more of your code and I'll simplify it myself when I get time. I'll then raise a JIRA issue if necessary, or ensure that it's fixed for 1.1.2. I'm a bit puzzled about why the synching works and would like to investigate more...

[ctran]

We currently have a work-around but I'll try to re-create the problem with some tests.

Thanks for your prompt response to this matter.

[ctran]

Here's what I found out:

- Using TransactionAttributeSourceAdvisor along with prototype objects causes a memory leak since every bean invocation is checked against AttributesTransactionAttributeSource (see the unit below). This is probably the root cause and the jvm was hanging due to excessive gc. We resolved this issue by using BeanNameAutoProxy creator to avoid calling into AbstractFallbackTransactionAttributeSource for prototype objects. Maybe this should be documented in the docs since TransactionAttributeSourceAdvisor was recommended when using attributes.

===============

package com.trantek.spring;

import junit.framework.TestCase;
import org.springframework.metadata.Attributes;
import org.springframework.metadata.commons.CommonsAttrib utes;
import org.springframework.transaction.interceptor.Attrib utesTransactionAttributeSource;
import org.springframework.transaction.interceptor.Abstra ctFallbackTransactionAttributeSource;

import java.lang.reflect.Field;
import java.util.Map;

public class AbstractFallbackTransactionAttributeSourceTest extends TestCase
{
public void testMemoryLeak() throws Exception
{
Attributes attributes = new CommonsAttributes();

AttributesTransactionAttributeSource attributeSource =
new AttributesTransactionAttributeSource(attributes);

for (int i = 0; i < 100; i++)
{
PrototypeBean bean = new PrototypeBean();
attributeSource.getTransactionAttribute(bean.getCl ass().getMethod("doNothing", new Class[0]), bean.getClass());
Map cache = (Map) getPrivateField(attributeSource, AbstractFallbackTransactionAttributeSource.class, "cache");
assertEquals("cache size should not increase", 1, cache.size());
}
}

static class PrototypeBean
{
public void doNothing()
{
}
}

static private Object getPrivateField(Object obj, Class clazz, String fieldName) throws NoSuchFieldException, IllegalAccessException
{
Field field = clazz.getDeclaredField(fieldName);
field.setAccessible(true);
return field.get(obj);
}
}

[Rod Johnson]

Thanks. This is really helpful. I'll look at it tomorrow or Monday.

Rgds
Rod

[Rod Johnson]

Thanks very much for the test. I've now fixed the bug and added this and another new test. The implementation of the cacheKey() method needs to change to

return targetClass + "" + method

The former use of identy hash code was wrong, as there can be multiple instances of the same Method associated with different prototype instances.

The fix is now in CVS. Please test it and let me know if it fixes the problem.

I've also created an issue for this in JIRA: http://opensource.atlassian.com/proj...browse/SPR-418.

Rgds
Rod