Dec 16th, 2011, 11:42 AM
Aggregation as a service(scatter gather), backed by AMQP
With reference to my previous thread of having performance issues while handling group of large messages
I am thinking of changing my design to expose aggregation as a service, in my previous implementation, i was employing a splitter connected to amqp out bound adapter (backed by executor channel) to publish messages parallely to amqp Services Queue. And I was using an amqp inbound adapter connected to a aggregator and a router to receive the reply messages.This did not scale very well. In this design splitting,aggregation happened in client environment itself.
Now i am thinking of making aggregation it self behind an amqp Q, i.e. my client code will always put aggregated requests in an AMQP queue (using amqp outboud gateway), 1 or more set of SI instance which will do the aggreated calls using scatter,gather pattern ( i.e take the message from aggreation request Q(amqp), split it, send it to approprate service Q (Amqp),wait for replies, aggregate the result, and publish it back to aggregation response Q (amqp q).
Please with enough experience in messaging, amqp, EIP patterns, please let me know if this solution will scale.
I am looking for handing multiple parallel request ( upto order of 1000), with typical payload sizes of 400b to 40k, and a group of 100 to 200 such payloads in each aggregated service call.
Any insights will be greatly appreciated.
Please refer to my other posts ,related to same area for a better background of what i am trying :
Tags for this Thread