ClassLoader leak when using jars

[Costin Leau]

1. Of course. Try sending the initial report that focused on PLUCL.
Note, the latest snapshot tries to remedy the CACHE_CLASSES issue - it is being cleared after the Tool is executed. The memory dump should show wether that is indeed the case or not. Note that there might be another leak that we have to patch.

2. SHDP doesn't load the classes - it creates a custom classloader and gives control to the Tool instance. Hadoop however has to load them (when executing the Tool). If there would be no leaks, the classloader and its classes would go away - however Hadoop 'pins' the classloader internally. Thus it cannot be GC'ed.

The test that you devised works - keep sending it back and we'll sort out any leaks that we can. Maybe we can't address them all but for sure it will improve performance and it will help identify the bugs in Hadoop which will get sorted out.

[davidgevorkyan]

Hi Costin,

I have deployed the app in another environment and will run several tests against it, will post the results as soon as I am done.


Sincerely,
David

[Costin Leau]

That's good to know - waiting for your feedback.

[davidgevorkyan]

Hi Costin,

Please find reports from the most recent run.

https://dl.dropbox.com/u/95015919/08...k_Suspects.zip
https://dl.dropbox.com/u/95015919/08..._Consumers.zip

After running several times with the new version I can clearly see that the Perm Gen memory size is still constantly increasing.

Let me know what do you think.


Sincerely,
David

[Costin Leau]

Thanks David - the report confirms that there is a leak but again the report doesn't show what pins the classloaders. The suspects report page 30 only indicates that the PUCL is loaded but that's about it - there's no indication of the GC path which is what I need.
The initial report that you sent (http://forum.springsource.org/showth...446#post420446) had more information on the leak, showing an accumulation path - the following reports don't.

I'm not familiar with MAT but is there a way to make it display more info about the leak?
Additionally, could you send me the HPROF file itself? I could manually import into my tools and poke around for more info?

Thanks!

[Costin Leau]

One more thing - are you doing the heap dump while executing a custom jar, or right after? I ask since I see a thread still holding a reference to PUCL, which should not happen.
I've pushed another minor update [1] that you could try - it contains additional logging (make sure "org.springframework.data.hadoop.mapreduce" is on trace level) and patches another JRE leak.

[1] https://build.springsource.org/brows...OOPNIGHTLY-318

[davidgevorkyan]

Hi Costin,

Tried to do more analysis with MAT, but it didn't show much of detail, however I got a trial license of YourKit analyzer and here are some reports: https://dl.dropbox.com/u/95015919/08...is_Reports.zip
The heap dump was done after execution of custom jars.

Please let me know what do you think.


Sincerely,
David

[davidgevorkyan]

Note that I have also included Possible_Leaks_Objects_Retained_By_Inner_Class_Bac k_References report in the same archive, which might be useful as well.

[Costin Leau]

I know - thanks.
Sorry for the late reply - see the email I sent you.