Jul 27th, 2011, 05:55 AM
Pass data between steps
I have a job that reads from a database and writes to a flat file. In one step I read the database data, process the data and write registers to the file. While processing I need to collect some totals, group information and cache some data to avoid re-read that from database, that I'll use in further steps to append other kind of registers in the file. So I need to pass some complex objets between steps.
Another requirement is that my job has to allow several threads executing with different parameters. So my problem is where can I store this interstep data? If I use a holder, if singleton I'll have to take care of concurrency and I'll have to clean data after use it to avoid a out-of-memory, with prototype or step scope I can share data. ExecutionContext is place for simple and small data and parameters becase is serialized to the database I think.
Anyone knows a pattern/idea to solve this problem?
Thank you for your help.
Jul 28th, 2011, 06:22 PM
Is the put(String, Object) method in ExecutionContext not good enough for your use case?
The only requirement to use this method is that the object must be serializable.
Aug 18th, 2011, 01:12 AM
What I did is put my data into a threadLocal, but this is only possible with a single threaded job
Aug 18th, 2011, 02:23 AM
Thanks for your replies. Yes, I think ThreadLocal with prototype scoped beans its the solucion to my problem as I have multiple threads executing multiple jobs, but each job is single-threaded. I think It could be useful some kind of interstep, not-persisted context to share this kind of cache structures in batch-processing.
Sep 1st, 2011, 04:10 PM
Agree. Interstep non-persisted context would be useful. We are redesigning a job due to hitting size limits on temporary context data which we don't need to be persisted. Getting MySQL errors on exceeding size on the batch context table(s).