Jan 11th, 2011, 04:51 PM
Does anyone have any guidance on performance tuning of a spring-amqp application.
We recently put a spring-amqp (glasfish, ehcache, java 1.6) app into production, and we are seeing extremely poor performance out of it. From what we can see initially the performance hit is somewhere between rabbit and out implemenation of the SimpleMessageListener.
How can I get metrics/information on what spring-amqp is doing, and how can I tune what it does? Specifically I am curious about channelCaches, prefetch, etc......
Has anyone put a high-performance app into production using spring-amqp that I can hopefully glean some information from?
As some additional information. I ran visual-vm against the running app and found the following
1 amqp connection set for 8 consumers. Each of the individual SimpleAsync threads was sitting in a "wait" status for ~ 95% of the time.
Profiling CPU show that BlockingCache.nextDelievery was the single most expensive operation happening. No custom application logic even registered in the top 50 operations.
Last edited by jesmith17; Jan 11th, 2011 at 10:22 PM.
Reason: Additional Information
Feb 17th, 2011, 03:31 AM
I kind of hope there are no more production war stories since we are only at 1.0.0.M2, but it's brave of you to go ahead anyway.
The live metrics you quote suggest to me that perhaps you have a bounded queue in the consumer (that may have been the default until recently I forget), so it blocks waiting for the listener to finish processing and pull off another delivery. An unbounded queue is really the only sensible option because it doesn't make sense to block up the connection thread, otherwise it can't be used to send control messages and acks. You do have to beware of overwhelming the memory on the consumer. The best way to throttle is to use a transaction - then the broker will not send more messages until it gets the acks. If you were already transactional that won't be a problem.
If you are transactional you could also try tweaking the prefetch and txSize settings as well (the optimum should depend on the average message size and interval between messages, but usually somewhere in the range 5-20).
The measurements you made were useful, but as an aside I would note that you could have got that information from jstack.
Tags for this Thread