Mar 5th, 2010, 12:29 PM
Chunk Processing - Listener Query
Hi folks, well done on the excellent work so far...
I'm looking to adopt chunk processing inside one of my Spring Batch Jobs. I need to call out to a remote system to adorn each of my process items with additional data. Do do this for every item would be crazy as there are 100s of thousands. We're using Spring RPC over JMS so while it is quick it will slow down the Jobs significantly.
I'd like to use Chunk Processing. So that for example, I can collate 1000 items and then make the call to the remote service, adorn the items with data and then have them written away.
I've done some tests and implemented a chunk listener which revealed some interesting facts. It seams that the writer is invoked and the transaction commited before I get the 'after chunk' callback. I.E. Too late for me to adorn my items and have them persisted!
Am I missing a trick here or is the only option I have to create my own ChunkProcessor implementation?
Many thanks for any help in advance. BTW I'm a real fan of Spring Batch.. If only you folks could see the Batch Console we've implemented with IceFaces (JSF) and an event queue. True reverse Ajax and a highly interacrive web console to boot :-)
Mar 6th, 2010, 06:21 AM
After chunk is indeed invoked once you have read/processed/written the chunk, which seems normal to me.
your writer is the rendez-vous for that kind of thing even though a failure in there is not something you would want to generalize in practice as it makes error reporting much harder.
Maybe you could use an ItemWriterListener
but error recovery won't be as good as if the error occurred in the processor