Specifying a JobJar in the Tool Tasklet.

[davidgevorkyan]

I am actually specifying the classname.

[Costin Leau]

Hi David,

I've updated the tool support so now a jar file (not available on the classpath) can be specified - the loading process is done on a separate classloader so multiple versions and libraries can be used:

<hdp:tool-runner id="tool-jar" tool-class="test.SomeTool" jar="some-tool.jar"/>

Note that currently we don't do any copying or unpacking of the jar so things like nested /libs or /classes won't work - I'll add support for these (legacy) formats after (Ortodox) Easter. Feedback is welcome - also the structure of your jars is useful.

Cheers,

[davidgevorkyan]

Hi Costin,

Thanks for looking into this.

Can I get the latest artifact from somewhere?

Regarding your question about our jar structure: it doesn't have any nested libs, so it only has META-INF directory and compiled classes in the corresponding package directories.

Sincerely,
David

[Costin Leau]

Of course, see [1]. Simply add the snapshot repo in your gradle/maven build and all of SHDP version and its dependencies (including non-SpringSource) will be downloaded from there.

[1] http://www.springsource.org/spring-data/hadoop#maven

[davidgevorkyan]

Thanks Costin,

I downloaded the latest snapshot version and found one issue.

Seems that you have removed Scope parameter from the latest version.
This parameter is required for our cases, since we are constructing arguments based on the jobParameters and these values are returned only in case scope="step" for the tasklet.
See an example of tasklet that uses "jobParameters".

<hdp:tool-tasklet id="taskletId" scope="step" configuration-ref="hadoop-configuration" tool-class="SomeClass">
        <hdp:arg value="${propertyVal1}#{jobParameters['RUN_ID']}${propertyVal2}"/>
</hdp:tool-tasklet>

Sincerely,
David

[Costin Leau]

That was probably an unintended modification (though I don't recall scope being exposed). You should however still be able to use it through the beans namespace (beans XML that is):

<bean class="org.springframework.data.hadoop.mapreduce.T oolTasklet" scope="step" p:tool-class="SomeClass" p:configuration-ref=""/>

I'll fix the scope tomorrow morning (my time) but I'm interested to see whether the classloader update fixes your core issue.

Cheers,

[davidgevorkyan]

Thanks Costin,

I tried to specify the jar's relative and full paths, but none of them worked.
Here is the bean definition:

    <beans:bean id="test_hadoopTasklet" class="org.springframework.data.hadoop.mapreduce.ToolTasklet" scope="step"
            p:tool-class="${test_tool_class}"
            p:jar="test-jobjar.jar"
            p:configuration-ref="hadoop-configuration">
            <beans:property name="arguments">
                <beans:list>
                    <beans:value>${value1}</beans:value>
                    <beans:value>${value2}</beans:value>
                    <beans:value>${part1}#{jobParameters['RUN_ID']}${part2}</beans:value>
                    <beans:value>${part3}#{jobParameters['RUN_ID']}${part4}</beans:value>
                </beans:list>
           </beans:property>
    </beans:bean>

The following exception is being thrown:

org.springframework.beans.TypeMismatchException: Failed to convert property value of type 'java.lang.String' to required type 'java.lang.Class' for property 'toolClass'; nested exception is java.lang.IllegalArgumentException: Cannot find class [package.ClassName] at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFactory.java:527) at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean

Please let me know if I am doing something wrong.


Sincerely,
David

[Costin Leau]

Are you sure you're using the latest 1.0.0.BUILD-SNAPSHOT? Can you post the name of the artifact? toolClass is not a class anymore but a String so there shouldn't be any conversion error.

[Costin Leau]

Pushed an update which adds support for nested libraries (legacy jars). The latest snapshot is [1] 1.0.0.BUILD-20120417.114024-66.jar

[1] http://repo.springsource.org/webapp/...ng-data-hadoop

[Costin Leau]

And another update - the scope attribute is still there for tool-tasklet. That is, assuming you are using the correct SNAPSHOT (as mentioned above).