Multi-threaded (partitioned step) batch summary processing advice needed

[jpreston]

Hey guys,

I've been reading over some of the Spring Batch related documentation and have a couple of questions regarding a job flow. Take the following as truth:

--

A batch job where 100,000 database records are to be processed and written into files containing at most 2,000 records. After all records have been exported, all record files must then be encrypted and a "summary" file must be generated containing the new filenames along with record counts for each.

The batch job will contain a partitioned step (for multiple remote execution scalability) that creates the output file(s) with the processed records.

--

Given that quick description, what is the best route to handle it? Should a Tasklet be used to handle encryption and the "summary" file? It almost sounds as if I should add the filenames/record counts for the generated output files to the execution context and use that. Is there another (preferably better) approach?

[snicoll]

I would:

* Create partitions of 2000 elements at most
* Put in the execution context the location of each file to generate (there is something that explains that in the doc [1]
* Have a 2 steps job. The first step generates the files and the second step generates the summary file. Note that you can promote things from the execution context to the job execution context (something like the number of items handled per partition). Your second step would take the directory where the files have been generated as input.

[1] http://static.springsource.org/sprin...putDataToSteps