MultiThreading within Reader!!!

[PieUser]

I am building an Spring Batch application (v2.0.0) to process multiple files(i.e.task) parallely. Each file may have hundreds of lines of records that need to be processed individually. Currently I am able to create threads(Slaves) for each file (i.e. 10 threads for 10 files) by implementing the Partitioner class(partition method) using ThreadPoolTaskExecutor. Creating one Thread per File (i.e. task) seems to a feature of Spring Batch. I have used the partitionJob.xml example from spring batch samples for reference.

But I want to process a individual file with 50 threads. Is there a way to create the additional threads without being dependent on the number of files (or tasks)?

[Dave Syer]

Have you looked at the scalability section in teh user guide? It gives a few the options, including the ones that you mention. If you read from a file in multiple threads you cannot restart the step if it fails (so you have to set it restartable="false" manually and switch off the state saving in the reader).

[PieUser]

Yes. After reading the notes, I guess we will need to use a middleware to achieve this or tweak wit. Also another option that i came across the net, was using the Java concurrent API from within a reader class, although it has some disadvantages in case of Job failures. This link http://www.appliweb.net/index.php/blog/viewpost/78 has a good example on implementing it.

Thanks Dave for referring to the documentation section.