ClassLoader leak when using jars

[davidgevorkyan]

Hi Costin,

I browsed around and didn't find a new artifact for the patched version, are these correct places to look?

https://build.springsource.org/brows...Y-311/artifact
http://repo.springsource.org/webapp/...ng-data-hadoop

[Costin Leau]

The second link is correct - see the project homepage for more info [1].
Basically you just add the snapshot repo and use gradle/ant/maven or whatever tool you want to download the latest snapshot.

As for the stacktrace, why not post it somewhere on the gasilion sites that allow file uploads (such as dropbox)?


[1] http://www.springsource.org/spring-data/hadoop#maven

[Costin Leau]

I've pushed another build which should improve the TCCL. From the stacktrace, it seems that you're using Hadoop 2.0 which leaks additional classes - see [1].
The code in SHDP now tries to fix that - could you try the latest snapshot and report back?

Thanks,

[1] https://issues.apache.org/jira/browse/HADOOP-8632

[davidgevorkyan]

Hi Costin,

I run several tests and I saw the following message:

URLs: [file:/project-2.9-SNAPSHOT/WEB-INF/classes/job-jars/test-job.jar]
Parent CL: ContextLoader@conductor
System CL: sun.misc.Launcher$ExtClassLoader@7a9664a1
] in thread Thread[IPC Client (47) connection to jobtracker from dgevorkyan,5,main]

Please note that during a single run our project is submitting 4 jobs and I saw this message showing up only for one of them, hence only classes loaded by that job (from the test-job.jar) were unloaded.

I performed Heap dump and run several reports on it, which you can find attached.
Looking at the Top Consumers report, you will see many instances of ParentLastURLClassLoader.

https://dl.dropbox.com/u/95015919/he..._Consumers.zip
https://dl.dropbox.com/u/95015919/he...k_Suspects.zip

P.S: Note that this test was performed locally, hence some Eclipse/Maven related leaks were detected, such as org.apache.maven.project.DefaultMavenProjectBuilde r.


Sincerely,
David

[Costin Leau]

Unfortunately the reports don't really help - probably because the DefaultMavenProjectBuilder is perceived as the main leak. I don't see the references to the PLUC anymore and can't tell why they are still hanging around. Can you apply some filtering or minimize the scope to focus on PLUC?
The first report you sent was much better in this regard.

As for the messages they don't have to appear for each job - it depends a lot on the runtime and the code path touched by the job submission. Are you using the latest snapshot (313+) or not?

[davidgevorkyan]

Hi Costin,

Yes, I am using the latest snapshot and all of the submitted jobs are Tool implementations.
I will perform more testing in another environment and let you know the results.

Sincerely,
David

[Costin Leau]

That's great.
The first report, even though it didn't have any images and it looked pretty ugly, contained the GC path for the leaking PLUC instances - I couldn't find that in the latest reports.

Also, what version of Hadoop are you using? 1.0.x or 2.0.x? I ask since I'm testing against various 1.0.x versions but not against 2.0.x which has seen to many changes and it's really just an alpha.

[davidgevorkyan]

Yeah, it was easier to do local testing and I was hoping it would give some insight, but anyway I will perform another batch of tests.

We are using Cloudera distribution of hadoop, specifically cdh3u3 on the cluster (hadoop-core version in the client is 0.20.2-cdh3u3).


Sincerely,
David

[Costin Leau]

Right - well it contains some code that's submitted in Hadoop 2.0. The leak unfortunately is not caused by SHDP, but by the Hadoop classes - I've even raised some bug reports on the Hadoop issue tracker to address some of them but time will tell when and if that will happen.
Several classes inside Hadoop literally hold on to the class loader and its classes for 'caching' purposes - in some cases we can try and flush the cache but in others, we cannot fully prevent this from occurring.
An alternative is to upgrade to the latest cloudera distro or raise the bug with them as they might pay more attention to a paying customer.

[davidgevorkyan]

Do you think I should go ahead and do more testing, or the issue is in the CACHE_CLASSES hashmap, which is also used in the hadoop-core-0.20.2-cdh3u3?

Can we also go back and elaborate why did we load classes at all when trying to submit them, I mean in the SDHP?