@Splitter memory woes

[qmx]

Hi folks!

I have a scenario like this:

A BIG, BIG file, that has a lot of items to be processed. These items are very good candidates for parallel processing, but they need to arrive on the final pipeline in the same order of input.

The question: If I set the correlation id and the sequence number, they will be elligible for @Aggregator or <integration:resequencer/>? 'Cause I'm a little bit worried of throwing a Collection of Items that big. (Like in cafeDemo)

Any Ideas?

Cheers!

[Mark Fisher]

I would strongly recommend Spring Batch for that type of scenario:
http://static.springsource.org/spring-batch/

I believe it will become quite common to use Spring Integration for coarse-grained messaging (e.g. "A File has arrived and is ready for processing") while Spring Batch will actually process the file (chunking, handling each item across multiple steps within a job, etc). The nice thing about this interaction is that it will work well on both "ends" of a batch job (e.g. send an email after completion). You can expect to see quite a bit more from SpringSource on this Spring Batch + Spring Integration model in the near future, but it should be pretty straightforward to use them together even now.

[qmx]

Hi Mark! Thanks for the fast response!

Actually, We're using spring batch. The main problem is that we need to have these messages processed through jms queues and SI endpoints, that fits very well in our case. (Spring Batch reads the file, we build the messages and send them through the SI channels.

The main question is: What kind of magic data is needed to mimic @Splitter behaviour?

Did I was clear? (English is not my primary language)

Cheers!

[iwein]

If I get this right you're actually looking for a way to use an aggregator without creating messages with a splitter, right?

If so you can use the parameters you mentioned and the sequence size (which is needed by the aggregator to decide when the aggregation is complete). Alternatively you can write a CompletionStrategy. Be sure to carefully consider which parts to solve with batch and which with SI and let us know how it goes