high volume messages channel

[shaine]

how well does spring integration handle high volumes on a channel.

eg an inbound file adapter listening to a folder with 50000+ files.
is it better to to set a max-messages-per-poll in this case? what would be a good number?

the channel will actually be idle 90% of the time. when it does have work to do, the volume will be high.

[Cleric]

Hello

the channel will actually be idle 90% of the time

It depends of what you are doing with messages in the downstream message-flow.
Try to manipulate with 'max-messages-per-poll' & 'fixed-delay' attributes on your <file:inbound-channel-adapter>.

Please, provide more info about your logic around each polled file.

Take care,
Artem Bilan

[ghillert]

Hi,

Also, what type of channels do you plan on using? (In-memory channels, JMS backed channels, channels backed by group stores for persistence etc.). Furthermore, how to you plan to route the data? Do you want to pass along the actual file references or extract the business data and route it primarily?

Depending on the complexity of your flow, are you planning on using the Claim Check pattern to minimize the amount data your are passing around?

http://static.springsource.org/sprin...ml#claim-check

Moreover, will you be processing the data in your flow synchronously (single-threaded) or will you be using asynchronous queue channels ? There are many more variables that may affect the answer to your question.

However, when using the default channel implementation (org.springframework.integration.channel.DirectCha nnel), the overhead of the channel itself will be minuscule. The sending of a message to a DirectChannel will trigger the invocation of that subscribers handleMessage(Message) method directly in the sender's thread. Thus, as Artem pointed out, it all depends on what you are doing with messages in the downstream message-flow.

You can find some more information regarding the various channel implementation at:

http://static.springsource.org/sprin...mplementations

I hope this gives you some additional information. In order to give you a more specific answer, please provide more information regarding your use-case.

Cheers,

Gunnar