I am successfully executing a Hadoop JobTasklet. I am using Quartz to kick-off the job every few minutes (the job takes about 30 seconds to complete). My problem is that I cannot kick off 2 or more jobs.
My problem appears to be in the JobTasklet.execute() method. I have "wait for job" set to true so the code executes "job.waitForCompletion(false);". This immediately submits a job to the cluster because Job.state=DEFINE.
However, when Quartz kicks off my second job, I see that "Job.state == RUNNING" and this causes execution to block until my previous job is marked as FINISHED (which happens immediately because JobClient.isComplete() determines that the job has completed). Execution then falls back to JobTasklet.execute() and RepeatStatus.FINISHED is returned.
The bottom line is that my second job is never executed. Instead, it simply waits for my previous job to be marked as completed and then nothing else happens.
Can you tell from my description if I am doing something incorrect?*
Thanks in advance for any help you can offer.