Setting Executor on hdb:job-runner

[bajafresh4life]

I am trying to invoke several Hadoop jobs serially using Spring Hadoop, but one problem I'm having is that the main thread is not waiting for the job to complete before proceeding to the next job. The XML I'm using to define each job looks like:

<hdp:job-runner
id="runner"
job-ref="my-job"
run-at-startup="true"
post-action="cleanup-script"
wait-for-completion="true"
/>

although the wait-for-completion attribute doesn't seem to have the desired effect. I've tried setting the "executor" attribute, as stated in the docs:

As the Hadoop job submittion and execution (when wait-for-completion is true) is blocking, JobRunner uses a JDK Executor to start (or stop) a job. The default implementation, SimpleAsyncTaskExecutor creates a new Thread for each new task. Before going into production, it is recommended to double-check whether this strategy is suitable or whether a throttled or pooled implementation is better. One can customize the behaviour through executor parameter.

but I can't get the configuration right -- it's looking for an Executor instance but I can only set executor to a String.

[Costin Leau]

Hi,

There was a bug in the schema (executor should have been executor-ref) which has now been fixed in master. The default executor (the async) might be changed back to the sync approach as that seems to be the behaviour expected by most people...

[bajafresh4life]

Hi,

There was a bug in the schema (executor should have been executor-ref) which has now been fixed in master. The default executor (the async) might be changed back to the sync approach as that seems to be the behaviour expected by most people...

Cool, thanks for your help

[Costin Leau]

Hi.

I've pushed a nightly build with the latest changes - the executor-ref should work properly now. Also the default executor is as before, synchronous, which means jobs are executed on the calling thread.
Could you try it out and report back please?

Thanks