Data/node affinity in clustered app

[iangreen]

I have a clustered app which allows for N JVMs to be added. I already have discovery and group management taken care of. I want to introduce Spring Integration to simplify my messaging architecture (now written using custom JMS).

The problem I'm trying to solve is this:

My "worker" nodes are intended to load and cache expensive data. Worker instance "A" may have a cache for NYC, Worker instance "B" may have a cache for Paris, etc. If a request comes into the system for something about NYC, and NYC's cached data is already on worker "A", I want to route the request to that specific worker.

How do I model this kind of system in Spring Integration? Should each live worker have its own channel, and use a router for each channel? If so, should I be I dynamically creating channels when nodes join the system and use a DynamicRouter to route to them, and implement affinity that way?

thanks!

[Cleric]

Hello!
H-m-m...
It looks like you need Hadoop.

Take care,
Artem Bilan

[Amol Nayak]

@Artem

It looks like you need Hadoop.

How can Hadoop fit in here? I'm not a Hadoop expert but what from Hadoop are you suggesting to be used here? MapReduce? Which IMO doesn't fit in here.

@iangreen
You seem to be going the right way. You will have one entry point into the system, a router, which will route your request to the appropriate node for processing.

[Cleric]

Hi, Amol

I don't try to solve the task. I just look at this as a solution.
It reminds me HDFS and distributing tasks between nodes...

[iangreen]

Hello!
H-m-m...
It looks like you need Hadoop.

Take care,
Artem Bilan

Nah, gridgain looks much better to me last I checked. Thanks for your suggestion though.

I decided to use a jms message converter on the local outbound side and selector expression on the remote inbound side. Seems decent enough

Thanks all