Pass data between steps

[occus3]

I have a job that reads from a database and writes to a flat file. In one step I read the database data, process the data and write registers to the file. While processing I need to collect some totals, group information and cache some data to avoid re-read that from database, that I'll use in further steps to append other kind of registers in the file. So I need to pass some complex objets between steps.

Another requirement is that my job has to allow several threads executing with different parameters. So my problem is where can I store this interstep data? If I use a holder, if singleton I'll have to take care of concurrency and I'll have to clean data after use it to avoid a out-of-memory, with prototype or step scope I can share data. ExecutionContext is place for simple and small data and parameters becase is serialized to the database I think.

Anyone knows a pattern/idea to solve this problem?

Thank you for your help.

[symanco]

Is the put(String, Object) method in ExecutionContext not good enough for your use case?
The only requirement to use this method is that the object must be serializable.

[Reijnemans]

What I did is put my data into a threadLocal, but this is only possible with a single threaded job

[occus3]

Thanks for your replies. Yes, I think ThreadLocal with prototype scoped beans its the solucion to my problem as I have multiple threads executing multiple jobs, but each job is single-threaded. I think It could be useful some kind of interstep, not-persisted context to share this kind of cache structures in batch-processing.

[dvb123]

Agree. Interstep non-persisted context would be useful. We are redesigning a job due to hitting size limits on temporary context data which we don't need to be persisted. Getting MySQL errors on exceeding size on the batch context table(s).