Multi-step design question

[bgardner]

Greetings again,

I've successfully completed the implementation of a batch job that loads my input, and it works great, and it's more than speedy enough to handle the million or so records a day we receive. Great job on a great product to Dave and his team!

Now I need to add a FileWatcher that looks for new input files in the Landing Zone - no problem, got it written. Got a cron trigger to call the job, using the JobLauncherDetails class provided in the samples. Now I just need to integrate my FileWatcher as a primary step in my job. That's where it starts to get tricky.

The FileWatcher looks to see if a new file has shown up, and if not, there's no need to go on the next step, but the cron trigger will fire again in 5 minutes, and if there's a file, we go to step 2.

Here's the relevant pieces of the config:

<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
  <property name="triggers">
    <bean id="cronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
      <property name="jobDetail" ref="jobDetail" />
      <property name="cronExpression" value="* 0/5 * * * ?" />
    </bean>
  </property>
</bean>

<bean id="jobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
  <property name="jobClass" value="com.vzw.spa.batch.quartz.JobLauncherDetails" />
  <property name="group" value="spa-batch" />
  <property name="jobDataAsMap">
    <map>
      <entry key="jobName" value="myJob"/>
      <entry key="jobLocator" value-ref="jobRegistry"/>
      <entry key="jobLauncher" value-ref="jobLauncher"/>
    </map>
  </property>
</bean>

<batch:job id="myJob" job-repository="jobRepository">
  <batch:step id="fileWatcher">
    <batch:tasklet transaction-manager="transactionManager"
      ref="fileWatcherTasklet" />
    </batch:step>
    <batch:step id="myStep">
      <batch:tasklet transaction-manager="transactionManager">
        <batch:chunk reader="myReader" processor="myProcessor" 
          writer="myWriter" commit-interval="100"/>
      </batch:tasklet>
  </batch:step>
</batch:job>

How do I control the step logic so that the FileWatcher can do its work, not find a file, and the job not be failed? A JobInstance will be created, but until the FileWatcher has a file to process, there's really no "Job", no parameters, etc. Should the FileWatcher even be part of the Job?

Brian

[hailspring]

What is the business requirement? The job should be completed even if the file is not found?

If you are checking for the existence of a file every 5 minutes and if the file does not show up for 1-2 hours then you would have completed running the same job atleast 12-24 times. If you are persisting your job and step status then are you passing different job parameters to every job? (This will happen in the scenario you have described above)

It really depends on your business requirement, from the information you have provided above, I would prefer to keep the FileWatcher in Quartz and kick-off my spring batch job only when the required condition (arrival of file) is met.

Another scenario you might want to consider in this case is, are you triggering your spring batch job as soon as the file arrives or waiting for the file to arrive completely and then fire the job. You may want to have a check to see whether the file has downloaded completely.

[bgardner]

Hailspring,

You're right. I moved all the quartz and file watching stuff outside the job, so that when a file arrives (and is completely written) I start the job itself, which reads and processes the file. Thanks for getting back to me.

Brian