source adapter, big file

[xmatoha]

Hello,

I would like to know your opinion on how to best implement source adapter when I have file which is say 5GB. I can identify records(rows in file) which I can then map to java objects. Polling for file records is not good, there is a problem how to specify period. I need to pull records continuously until I reach eof.

Currently I have java class which takes spring integration channel and iterates over records in file convert them into objects and place them on channel.

My question is, if there is more consistent way how to impelment this, using M4. If not is there a plan to cover this kind of "pattern" with spring integration.

thank you

[Mark Fisher]

Based on your use case, it sounds like you might want to take a a look at Spring Batch: http://static.springframework.org/spring-batch/

At some point, we will be providing integration between the Spring Batch and Spring Integration projects.

[mbogoevici]

As Mark pointed out, the handling of such a large file would be ideally implemented using Spring Batch.
However, it is worth noting that since M4 we support creating messages with a java.io.File payload, so that once such a big file is dropped in the corresponding directory, a message containing a reference to a file is created and passed to a channel.
Thus, your process can react to the change, but not necessarily pass around 5GB of content. Once your Message<File> entity has reached a proper endpoint, the handler can further do a batch processing itself.

--Marius