1 Attachment(s)
performance issue with amqp adapters,aggregator,spring integration
Hi,
I have task, where i need to come up with an API for making aggregated service calls. i.e my clients will need an api, where they will call multiple services concurrently, Basically they will let me know the list of services they want to call and provide the argument for each service, I will need to make concurrent calls to each of the service and provide an aggregated result to them.
Basically my clients makes a blocking call to my api, my aggregated service handler, makes concurrent calls and returns the consolidated result.
As part of my solution, I am using scatter gather pattern of EAI.
I create a message out of the list of arguments and use a splitter down stream to split this into multiple messages, and use an amp outbound gateway to place the messages to appropriate rabbit queues ( exchange,binding key are discovered with help of service name from a service registry).
I have a dedicated AMQP queue where all the reply messages will be sent. I use an AMQP inbound adapter to listen to this queue, and use a aggregator aggregate the message, and then use my custom router ( extended from header value router, uses its own channel resolver instead of the app context file). and route the message back to the message handler from where the message originated.
With this setup, i am able to achieve my requirement, but, i did some performance bench marking and found that my implementation is not able to scale very well.
My environment
linux :
processor :12 cores (24 with hyper threading).
When I have 1 client firing 10,000 requests of various group consisting of ( 1,2,5,10,15,20,30,50 and 45)requests. Each request is of size 400bytes.
I see the system is able to handle groups upto size 45 well , but the performance slows down significantly for a group of size 50.
I have 10 server threads listening the the request queue to serve these requests.
I see that the avg response time for 10,000 requests increases gradually with increase in group size unto 45. with group size of 50, i am not able to handle 10000 request .I am getting java.lang.OutOfMemoryError: GC overhead limit exceeded exception.
Repeated the test for 4k payload,and found very similar results.
repeated the test with 10 clients instead of 1 and found that with10 clients firing 1000 requests each , i was not able to handle a group of size 30 ( runs well for group size =20).
Please find attached the excel for various values of clients, servers threads, listeners threads,group size and payload sizes.
I also found that my resident memory (not heap) was huge and grew unto 6.1 gb before throwing the outofMemory error.
I am not sure if there is a memory leak some where.
I also saw NIO in one of the stack traces, Does amp inbound/outbound adapters uses nio? and could the huge memory footprint (6.1g) be because of the NIOs ?
Answer to any of these questions will be of much help :
-Please let me know if there is some thing fundamentally incorrect with my design?
- Any known performance issues with aggregator ,amp inbound/outbound adapters, rabbit java client api
- Any alternate design which i can consider.
