Aggregator and CompletionStrategy for unknown line item count

[woodsd]

Hello all,

I'm just getting started looking into Spring Integration and am looking for some suggestions on where to start with the following use case.

I receive files with orders and line items from a 3rd party system with unique order ids but I don't receive any unique key for the line items. The records in these files will get processed multiple times due to data corrections in the originating system but a complete order will always be present in the file. I've looked at using an Aggregator so when the order and all items are received I can update the db with the new line items by purging the old and inserting the new. The problem is that as each row is processed I don't yet know how many line items are present.

My new plan is to not use the Aggregator but send the order and a single line item as they are processed and send the messages with a corresponding batch id or timestamp so I can purge the old data when the new arrives but I'm thinking I'll have to do that for each message and not just the 'first' message for each order. I would use a queue channel but am not certain about using multiple receivers if there will be issues with processing them in the correct order. Is there a feature in Spring Integration that will allow me to not do the purge for each line item?

Below is a mockup of the file structure.

1, "Order 1", "Some line item data"
1, "Order 1", "Some other line item"
2, "Order 2", "Some line item data"
1, "Order 1", "A third line item"

Thanks,
Dustin

[iwein]

Is there any way that you can know (at any point in time or any place in the system) what the whole order consists of?

If yes, you can use a CompletionStrategy and make this information available to it. If no, there is no solution to your problem.

To me it sounds that OrdersFile> [FileSplitter] >Order> [OrderSplitter] >OrderLine> [OrderLineAggregator] >Order> [OrderRepository should just work. Can you try that?

Not having unique identifiers with the line items sounds like a pretty bad idea to me because how can you know if you're getting a duplicate? But it seems that is out of your control.

[woodsd]

Thanks Iwein for the timely response. You've given me a few more ideas on how to proceed.

Is there any way that you can know (at any point in time or any place in the system) what the whole order consists of?

Since I can guarantee that the file will have all line items for each order I could sort the file before processing and queue up the line items until the order changes and then push them through.

To me it sounds that OrdersFile> [FileSplitter] >Order> [OrderSplitter] >OrderLine> [OrderLineAggregator] >Order> [OrderRepository should just work. Can you try that?

I'll look into the FileSplitter more to see if that can effectively re-normalize my file.

Not having unique identifiers with the line items sounds like a pretty bad idea to me because how can you know if you're getting a duplicate? But it seems that is out of your control.

Yes, not having keys is extremely frustrating. I've put in a request to add them but am investigating possible ways to move forward without it.

Thanks,
Dustin