Specifying a JobJar in the Tool Tasklet.

[Costin Leau]

Great. Could you share more on how you pass the properties to your jobs and the use of Spring Batch?
Do the jobs pass information between each other - and if so, how? Anything that you think it's missing?

Cheers,

[davidgevorkyan]

Hi Costin,

I did extensive testing with property files and here are my findings:

Below is my tasklet configuration:

<hdp:tool-tasklet id="hadoopTasklet" scope="step" configuration-ref="hadoop-configuration"
              tool-class=MyJob" jar="my_job.jar" properties-location="myProp1.properties" files="myProp1.properties">
  ...
</hdp:tool-tasklet>

Here is one of the spring configs in my_job.jar:

    <bean id="propertyPlaceholderConfigurer"
        class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="ignoreUnresolvablePlaceholders" value="true" />
        <property name="ignoreResourceNotFound" value="true" />
        <property name="locations">
            <list>
                <value>classpath:myProp1.properties</value>
            </list>
        </property>
    </bean>

I can tell you for sure that properties-location property is working, since all the values from the property file were loaded and can be seen in the Job Configuration page in Hadoop.

At the same time the files property doesn't work, since I am getting the following errors:

attempt_201107061330_12011_r_000321_0: 2012-05-11 16:59:31,477 WARN  [PropertyPlaceholderConfigurer]
Could not load properties from class path resource [myProp1.properties]: class path resource
[myProp1.properties] cannot be opened because it does not exist
...

I tried to change it to files="classpath:myProp1.properties" but it didn't solve the issue.

When invoking from command line we were using -files fullPath/myProp1.properties


Please let me know if I need to add something else.


Sincerely,
David

[Costin Leau]

It looks like you ran into a bug [1]
The namespace parser currently doesn't properly parse the files/archives/libs properties (even though you did specify them).
I'll try to address that shortly - unfortunately I'm travelling (again) as we speak and will be in transit until next week; in the worse case scenario I'll have an update by next weekend.

Thanks for your feedback and patience.

[1] https://jira.springsource.org/browse/SHDP-74

Cheers,

[davidgevorkyan]

Hi Costin,

Please let me know as soon as you have an update on this.


Sincerely,
David

[Costin Leau]

Haven't forgot this. Due to some hectic problem (and some bad food) I didn't get a chance to take a look at this - but I plan to do this shortly this weekend or early next week as I'm currently on my way home.

[Costin Leau]

Hi David,

The issue has been fixed and the update pushed upstream and a nightly build published.

P.S. The namespace was actually working but the tool wasn't configured properly - which has been now been addressed.

Feedback is welcome!

[davidgevorkyan]

Hi Costin,

Thanks for the update, I haven't been able to test the changes yet, since the cluster under a complete rebuild.
I will do and let you know as soon as possible.

Sincerely,
David

[Costin Leau]

Hi David,

Any update?

[davidgevorkyan]

Hi Costin,

Sorry for the late reply, the cluster was rebuilt on Friday and I only got a chance to test it today.
I just run an end to end test of the project against spring-data-hadoop-1.0.0.BUILD-20120603.231511-119 and everything is working well. I will run another test tomorrow, will let you know if I find something else.


Sincerely,
David

[Costin Leau]

That's good to hear. By the way, the latest snapshot also supports security (the docs should get in there by the end of the day) meaning you can now specify a "user" as an impersonation for running Tool#run.
For what it's worth, this is supported across the board by all MR components (streaming, job & tool) and Pig as well.