Multithreading and Spring Integration - Design Issue

[Invincible]

Hi,

I am using Spring Integration in my application. I have a design issue. My requirement is that I need to run few similar processes simultaneously and each process eventually does multi processing again. To explain with an example, Lets say I have 5 URLs. Those 5 URLs have links to 6 URLs each. I need to download the content from final 30 URLs in parallel. For the final download, I already have used task-executor and input output channels. From my understanding, task executor uses runnable. So, the tasks given to task executor are multi threaded. Correct me if I am wrong here. Now, for the first 5 URLs, should I create a new thread each or is there a way in SI to achieve this avoiding thread creation by developer?

Appreciate your help.

Thanks,
Jan.

[Gary Russell]

There are a number of ways to do it, but the right choice depends on how the 5 request messages are generated; if you can share how the request messages are created, and maybe a logical description of your flow we can help.

But essentially, yes, the developer generally doesn't get involved with thread creation.

[Invincible]

Thanks Gary.

well, It is like this. The 5 "URLs" are basically strings that I pick from application context. Then I parse the content from that URL and retrieve the 6 URLs from each. Each of these 6 URLs have multiple segment files. So, I need to download those segment files. For eg: I read the m3u8 file name from properties file. then I parse the m3u8 file to get the list of m3u playlists. Each m3u playlist has certain number of segments. I read the segments and download them. I have used task-executor to download the segments so, multi threading is taken care of here. but how do I handle the m3u8 and m3u files? There can be more than 15 m3u8 files and each m3u8 file can have more than 6 m3u files.

Thanks,
Jan.

[Gary Russell]

Not sure what you mean by "I pick from the application context", but you can always go async, by making a <channel/> a queue channel, or by adding a <dispatcher/> to a DirectChannel. Such as...

<channel id="input">
	<dispatcher task-executor="exec"/>
</channel>
	
<task:executor id="exec" />

The thread sending a message to channel "input" essentially hands off the message to a thread managed by the <task:executor />, freeing up the main thread to send the next message; you can see this in operation here....

2012-01-19 18:14:41,212 [main] DEBUG: org.springframework.integration.channel.ExecutorChannel - preSend on channel 'input', message: [Payload=Hello world!][Headers={timestamp=1327014881212, id=3850adcc-2198-40c7-b3bc-6f39b309da2f, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@5a57e787, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@5a57e787}]
2012-01-19 18:14:41,214 [main] DEBUG: org.springframework.integration.channel.ExecutorChannel - postSend (sent=true) on channel 'input', message: [Payload=Hello world!][Headers={timestamp=1327014881212, id=3850adcc-2198-40c7-b3bc-6f39b309da2f, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@5a57e787, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@5a57e787}]
2012-01-19 18:14:41,214 [exec-1] DEBUG: org.springframework.integration.ip.tcp.TcpOutboundGateway - org.springframework.integration.ip.tcp.TcpOutboundGateway#0 received message: [Payload=Hello world!][Headers={timestamp=1327014881212, id=3850adcc-2198-40c7-b3bc-6f39b309da2f, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@5a57e787, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@5a57e787}]
2012-01-19 18:14:41,215 [exec-1] DEBUG: org.springframework.integration.ip.tcp.connection.TcpNetClientConnectionFactory - Opening new socket connection to localhost:11111