Specifying a JobJar in the Tool Tasklet.

[davidgevorkyan]

I was also wondering, is it possible to add more logging when running hadoop tool tasklets, such as to show Jar file, Tool Class, Hadoop Arguments, Hadoop Job Configuration Properties (the ones that we are specifying/overriding from the tasklet).
They can be very helpful when having many hadoop jobs and doing manual inspection in the logs.

As you suggested, it can be controlled by specifying verbose flag.

Sincerely,
David

[Costin Leau]

I've raised SHDP-89 and added the option to job-tasklet namespace. Note that the verbose option will only affect the job-tasklet - since a tool takes care of running its own job, it's out of our control.
However since you mentioned you wanted info about the tool execution, I've added various logging on trace and info level.

Try the master and see whether it works for you. If not please comment SHDP-89.

https://jira.springsource.org/browse/SHDP-89

[davidgevorkyan]

Hi Costin,

Thanks for taking look at it, I will try and let you know how it works for me.

One quick question: I just faced an issue in a long running Spring Batch process that submits jobs using SDHP, namely I got an "java.lang.OutOfMemoryError: PermGen space" error and by looking at the monitoring data, seems that the loaded classes aren't unloaded, which means from the top of my head, that the class loader that actually loaded the classes isn't Garbage Collected.

Did you run any tests on that, do you think the ParentLastURLClassLoader is being properly garbage collected and classes that it loads, unloaded?

The issue might be totally in a different part, but the increase is high and since SDHP is the only part that loads jar files with dependencies, my guess is that it might have caused that increase.

Please let me know what do you think.

Sincerely,
David

[Costin Leau]

Hi David,

I've done some debugging and found something. However since this thread is becoming epic (7 pages and 63 replies) I have created a new thread here to keep things clean:
http://forum.springsource.org/showth...hen-using-jars
Let's follow the discussion there.