I have a process that seems like it could be solved nicely with spring-batch but I haven't been able to find an example which shows how to model the following scenario:
Step 1 - Takes 1 day to complete.
Step 2 - Takes 2 days to complete.
= Total time is 3 days if executed in series.
Step 2 consumes the individual items written to the db by Step 1. Further I know that Step 2 will consume the output at a much slower rate than Step 1. If I were to run them in parallel the total process time would be closer to 2 days. The closest configuration I have seen to the ideal parallel solution is to use <split> along with <flow>:
It is my understanding that this will cause step1 to execute in parallel with step2 but how would I prevent Step 2 from completing while Step 1 is still running (in the case that somehow step2 got ahead of step1 and temporarily didn't have input to read)?
<split id="split1" next="step3" task-executor="taskExecutor">
<step id="step3" />
I considered merging these two steps into a single step and performing "Step 2" in the writer however it seems like I lose some of the power of spring-batch by adding retry/backoff/threadpooling in java code instead of configuring it via the spring context.
I also considered separating the steps into two jobs where I kick off job1 before job2 but the original sync problem remains as well as having to maintain the dependency between the two jobs.
Any thoughts on what I am missing?