Nov 22nd, 2010, 12:07 AM
skip total count synchronized in multi-threaded chunk-oriented TaskletStep?
Hi! I am doing some prove of concept for using multi-threaded step in our project. I am using Spring batch 2.0.1 As in the attached batch XML configuration (which also documents the details of each components like reader, processor, writer. etc.), I used multi-threaded chunk-oriented TaskletStep. Multiple threads will execute the Tasklet step concurrently. I synchronized ItemReader.read() and the processor is also thread-safe ( I meant there is no sharing of state other than Spring batch step execution state between executions of ItemProcessor.process())
The key requirement is that the multi-threaded step (and then the job) needs to fail when total count of the skippable exceptions (among ALL threads) exceeds the configured skip limit.
So, is skip total count synchronized between threads in multi-threaded chunk-oriented TaskletStep? I noticed the readSkipCount, processSkipCount, writeSkipCount in StepExecution are all volative. So my understanding after reading code is that skip total count is synchronized between threads and I can trust that step/job will fail when the skip limit is reached even in multi-threaded environment. Please let me know if my understanding is wrong.
After my tests, the persited data in the BATCH.BATCH_STEP_EXECUTION table for this step are:
filter_count: -32 ( why is this a negative number?)
read_count: 110 (expected since read is faster than processing)
I know in a multi-threaded step, those numbers can become weird. What numbers (the above "count" data) can I trust?
Thanks a lot!
Nov 29th, 2010, 05:24 AM
The StepExecution statistics (skip count etc) are synchronized between threads by the TaskletStep. You shouldn't attempt to modify the StepExecution yourself at all.
If you set a skip limit in the step configuration, the step will fail if the limit is exceeded, but since there might be other threads processing the same step, there might be more skips before the step actually stops, and the statistics at the end might show more skips than the limit.