Oct 5th, 2011, 08:43 AM
Message buffering and throttling by header
I'm new to Spring Integration, but was hoping it could help me scale up a core application process. The idea is simple: fetch content from servers, but such that - say - only n requests are made to a single server simultaneously. This so that a server which holds a lot of content doesn't get hammered. I'm currently doing this with a subject-limited BlockingQueue implementation - where the subject is the server IP. This approach can't scale beyond a single machine though and so I'd like to send each fetch-request as a JMS message instead.
So far I can see two solutions to implementing this in a message-based system with SI:
- A buffered (pollable?) queue approach, where a message is kept in the queue until < n consumers are working on that same server IP. This is somewhat analogous to what I'm doing now.
- Make some sort of network-wide ReentrantLock store that holds all the server-IPs (could get very big) that consumers can apply to. This obviously brings its own problems, like when servers die off and don't release their locks.
Obviously I'd like to make it so that it scales - and while the amount of messages may be large, the number of server-IPs shouldn't be: maybe several thousand at most. Being able to release the server IP early in the consumer-process would also be very nice to be able to do: fetching the content from the server only takes about 10-20% of the consumer's time for a piece of content - the rest is processing.
I know it's a rather broad question, but, any ideas on how I can do this with SI? Can it be achieved with a router? A gateway? I've searched and read the documentation and forums, but it seems to be a hard question to reduce to a single Google query :-) Any help would be appreciated!
Oct 9th, 2011, 12:36 PM
I think you should post this on the Spring Integration Forum.
Oct 9th, 2011, 01:25 PM
Done! :-) I've also added some new thoughts. Thanks!