We have a requirement where which would require to read entire input file, group data by account and then process each account as an atomic unit.
Basically delete all the exisisting rows for the account in the database and insert the new rows from the file.
Here's what I am doing(still in dev).
Set commit interval to large number (max records in the file could be 100k, so the commit interval is set to 500k), the processor creates a map<acct,dataList> and stores in the stepcontext, on the writer side I have a wrapper that gets the data map from step context and calls the service layer for each account.
The data that is passed from reader to writer (wrapper) is ignored. So there are basically two sets of data in memory which is not efficient.
Is there a better way to do this?
If I were to return null from processor for all the records, will the writer still be called?
We have several small data files that need to processed with similar pattern (data needs to persisted -all or nothing based on account level)