Any know leaks with JC/cgilib?

[memelet]

I'm now using JC in somewhat large system. The functional test suite has just under 1000 tests. In the setup for the abstract-test I have:

buyingPowerFactory = BeanFactoryInstance.getBean("com.its.trading.domain.buyingpower.config.BuyingPowerConfiguration");

Where BuyingPowerConfiguration is a JC config loaded in into an ApplicationContext via bean post processing (along with lots of xml configs).

@Configuration
@AnnotationDrivenConfig
@WithBeanNamePrefix("buyingpower.")
public class BuyingPowerConfiguration extends ConfigurationSupport implements BuyingPowerFactory {

    @Autowired
    private InstrumentDataHome instrumentDataHome;

    @Autowired(required=false)
    private Set<BuyingPowerListener> buyingPowerListeners = Collections.emptySet();

    @Bean
    public BuyingPowerHome home() {
        return new BuyingPowerHomeImpl();
    }

    @Bean(scope=BeanDefinition.SCOPE_PROTOTYPE)
    public BuyingPowerImpl buyingPower() {
        BuyingPowerImpl buyingPower = new BuyingPowerImpl();
        addBuyingPowerListeners(buyingPower);
        return buyingPower;
    }
 
    ...
}

Just the calling of that bean lookup method drops the suite to it knees. About half way thru the suite, memory usage starts to sharply rise and test methods take longer and longer, eventually terminating with OutOfMemoryError. Simply commenting out the getBean() call the test runs normally. Note config is not even used in the test suite.

I should note that as the application context is being recreated for every test method! (Not my doing, its what I inherited. I'm trying to correct that, but its gonna take some time.)

I've tried running the system with YourKit, but both with the latest version and their 8.0 EAP the JVM does a hard crash when snapshotting memory.

I've also created a single parameterized test suite that simply creates the same application context as the real test suite and calls getBean() for the JC a thousand times. But this did not reproduce the problem. (I'll keep working on this to try to use more memory to more closely simulate the real suite.)

So, I'm asking (probably way to late in the post to have kept your attention) ... Are there any know issues with memory leaks with JC or its use of gclib that might help me to pinpoint what is going on?

[memelet]

Not a JC or cglib problem at all. It seems that junit does not actually free up the test objects for the duration of the suite. So the test reference to JC config instance was keeping the entire application context in memory.

The existing suite nulled out the the top-level application context, but the introduction of the reference to the JC config instance made that non effective since the JC configure was contains a reference to its containing application context.

As an experiment I added the following as part of the teardown:

    private void nullAllInstanceFields() {
        ReflectionUtils.doWithFields(BaseTestCase.this.getClass(), new FieldCallback() {
            @Override
            public void doWith(Field field) throws IllegalArgumentException, IllegalAccessException {
                field.setAccessible(true);
                field.set(BaseTestCase.this, null);
            }
        }, new FieldFilter() {
            @Override
            public boolean matches(Field field) {
                try {
                    if (Modifier.isStatic(field.getModifiers())) {
                        return false;
                    }
                    if (Modifier.isFinal(field.getModifiers())) {
                        return false;
                    }
                    if (field.isEnumConstant()) {
                        return false;
                    }
                    Class<?> testClazz = field.getDeclaringClass();
                    if (!BaseTestCase.class.isAssignableFrom(testClazz)) {
                        return false;
                    }
                    ReflectionUtils.makeAccessible(field);
                    Object object = ReflectionUtils.getField(field, BaseTestCase.this);
                    if (object == null) {
                        return false;
                    }
                    if (ClassUtils.isPrimitiveOrWrapper(object.getClass())) {
                        return false;
                    }
                    return true;
                } catch (Exception e) {
                    e.printStackTrace();
                    return false;
                }
            }
        });
    }

While this may turn out to be overkill, a suite that sucked up 786M now runs in 64M !!

Here is where I got tipped to this isssue: http://blogs.atlassian.com/developer...ory_usage.html

[Chris Beams]

Interesting.. glad to hear you've got this worked out (or at least worked around).

[memelet]

The root of this problem is that JUnit38ClassRunner is retained by the suite and hence JUnit38ClassRunner.fTest retains a reference to the test instance. Simply clearing this field (via reflection) replaces all the crud in the code above.

For our case, we have a custom suite builder so we are able to replace JUnit38ClassRunner with the following:

    public static class MemoryReleasingJUnit38ClassRunner extends JUnit38ClassRunner {

        private Description description;

        public MemoryReleasingJUnit38ClassRunner(Class<?> klass) {
            super(klass);

        }
        public MemoryReleasingJUnit38ClassRunner(Test test) {
            super(test);
        }

        @Override
        public void run(RunNotifier notifier) {
            super.run(notifier);
            description = getDescription();
            Field testField = ReflectionUtils.findField(this.getClass(), "fTest");
            ReflectionUtils.makeAccessible(testField);
            ReflectionUtils.setField(testField, this, null);
        }

        @Override
        public Description getDescription() {
            if (description == null) {
                return super.getDescription();
            } else {
                return description;
            }
        }
    }

And then inject into junit via:

public class ClasspathSuite extends Suite {

    public ClasspathSuite(Class<?> suiteClass) throws InitializationError {
        super(getRunnerBuilder(), suiteClass, asArray(sortTestClasses(findTestClasses())));
    }

    private static class MemoryReleasingJUnit3Builder extends org.junit.internal.builders.JUnit3Builder {
        @Override
        public Runner runnerForClass(Class<?> testClass) throws Throwable {
            if (isPre4Test_(testClass)) {
                return new MemoryReleasingJUnit38ClassRunner(testClass);
            }
            return null;
        }

        private boolean isPre4Test_(Class<?> testClass) {
            return junit.framework.TestCase.class.isAssignableFrom(testClass);
        }
    }

    private static class AllDefaultPossibilitiesBuilder extends org.junit.internal.builders.AllDefaultPossibilitiesBuilder {
        public AllDefaultPossibilitiesBuilder(boolean canUseSuiteMethod) {
            super(canUseSuiteMethod);
        }
        @Override
        protected MemoryReleasingJUnit3Builder junit3Builder() {
            return new MemoryReleasingJUnit3Builder();
        }
    }

    private static RunnerBuilder getRunnerBuilder() {
        return new AllDefaultPossibilitiesBuilder(true);
    }

    ...

FYI, this bit of complexity is only required if you have very large test suites. Our unit-test suite contains 10k+ tests and our functional-test suite contains 2k+ tests, hence this little hack is critical for us.