Oct 2nd, 2012, 04:17 PM
Extreme High Throughput/ Low Latency Aggregation
We have a fire hydrant of streaming quotes at 100k/sec with 17k symbols that need real-time conversion to 1 second quote bars. Oleg demonstrated an excellent usage to build quote bars using an aggregator and reaper in http://forum.springsource.org/showth...highlight=bars. The throughput on Oleg's example is about 10k-20k for 100 symbols on my laptop, which is good but not enough. The latency inherent in the aggregator/ reaper is probably due to having to pass over once again all of the messages that compose the aggregation to create the quote bar instead of building it as you go. If we avoid passing over all the messages a second time we would get a boost. But let me ask an open question.
My question is: how would one scale given this use case with Spring Integration? What have you found with putting a router in front of the aggregators and then allowing different horizontal nodes handle the aggregation? And then if a channel fails?
Tags for this Thread