I committed some prototype code using Spring Integration and Spring Batch to show how to use the Enterprise Integration Patterns to scale up batch applications. There are six main use cases implemented, which should be familiar to anyone who has seen me giving a presentation recently (if you haven't then there is a Webinar on Wednesday next week - check the SpringSource website for registration details).
The old spring-batch-integration project has been moved to spring-batch-infrastructure-tests (because that's what it is), and the new stuff is in the spring-batch-integration slot. So SVN co-ordinates are: https://springframework.svn.sourcefo...ch-integration.
Each use case has its own package:
1. Automatic declarative transactional retry and repeat of a MessageHandler (unit tests only in package "retry" using existing features in Spring Integration).
2. Chunk-oriented middleware (package "chunk"). Uses a special ItemWriter to dispatch messages to a channel, which should be durable for restartable production use, and a listener on the worker process that wraps a business ItemWriter.
3. Complex item-processing using a message flow (package "item"). Business logic for item processing can be arbitrarily complex using content-based routing to steer the item through a set of business process stages.
4. Message trigger for a batch job (package "launch"). Spring Integration can be used to bridge between any external trigger (e.g. Quartz, file directory polling, web service, etc.) and a JobLauncher.
5. Non-linear Job flow (package "job"). Provides a Step implementation that acts as a MessageEndpoint, so that instead of a linear series of steps , a job can be composed of an arbitrary message flow.
6. Idempotent file processing (package "file"). A special purpose Job that processes all the files in a directory, treating each one as a source of messages. Spring Batch restartability means that no file will ever be processed more than once, even if the job fails, or is stopped.
There is a seventh use case to be added (Asynchronous Aggregator), as well as much work to be done polishing and finishing off the existing stuff. If anyone has any more suggestions, feel free to ask.
Please note that this code will be distributed and published in the usual repositories, but will not be part of the "official" 1.1 release (mainly because it requires Java 5, and we are not going to go to Java 5 until Spring Batch 2.0). It is work in progress, and subject to heavy refactoring until we get to 2.0 proper. We are publishing it now owing to high demand for samples of how to use Spring Batch in a concurrent and massively parallel setting.
Finally, I would like to thank Jonas Partner for his work writing some of the code.