Oct 11th, 2012, 11:03 AM
Handling Related Data (Orders w/ Lines)
I have a design question... When it comes to processing related data, how is this typically done? I've run into scenarios where related data needs to be processed, but I am not clear on how to preserve restarability and handle read/write/commits properly.
A good example of this is where an order contains multiple line items. In the past, I've created a driving query that fetches all orders to be processed, then inside of a processor, I fetch line items and handle them appropriately. This typically means loading the line item, doing more processing on the line, then saving a status against the line. When all lines are complete for an order, I need to update the orders status. Is there a good general practice to follow here?
Oct 11th, 2012, 01:03 PM
Typically you want to think of the item as the unit to be processed. So in the example you're describing, the item would be the order and all of it's line items. That is what I would expect to come out of the ItemReader and be processed via the processor and writer. This allows for all scenarios to be covered with regards to restartabilty. If you choose to do your own reading/processing/writing from the processor, you could run into issues when using features like remote chunking where date inconsistencies occur on failures.
Oct 13th, 2012, 08:58 AM
Just so I am clear... Are you proposing that the order, with its lines, be read in, processed, and written? Or that the order and lines are treated separately. I ask becuase there are implications with either approach. If the order is read with its lines, then "operation" counts (reads, writes, processed #s) would be off as they would reflect the order and not operations against the line. The implication on restartability would be that restarts would be at order level (failed lines would get rolled back?).
If orders and lines are handled independently, then i need a mechanism to read the order, set it aside, handle related lines (read/write/process), then update the order appropriately (basically process and write it after handling lines). I dont see a clear way to do this though.
Thinking out loud... I guess if I were to set up a reader that handed back a flattened representation of the order line record to inlcude order info, i could have something that payed attention to the order id change, but I'd have to come up with a good way to group lines together and still allow chunking. Maybe there is a pattern for this?
Thanks for the help!
Oct 22nd, 2012, 04:53 PM
I am proposing that the order, with its lines be read in, processed and written. With regards to the "operation counts being off", that depends on your definition of an "operation". The standard batch counts are already not including counts for the line items since you are handling the reads of they from an processor yourself so they should be the same. You are correct (and I would think it's a good thing) that restarts would be at the order level.