Spring Integration 2.1 request-reply benchmark tests showed very poor performance

[oleg.zhurakousky]

Also could you please post hardware specification of the machine you run the test on, just want to make sure we have fare numbers.
Anyway, i'll run your tests on mine to ensure that they are.

[exto]

http://camel.465427.n5.nabble.com/fyi-SI-td5716049.html

[muellerc]

Hello Oleg!

No, that's not the case and not what I'm saying.
The default request/reply Camel configuration in Camel use temp. queues for the response. In your producer template, you only have to configure the endpoint uri "activemq:jmsCamelQueue" instead of "activemq:jmsCamelQueue?replyTo=bar&receiveTimeout =5&replyToType=Exclusive". With this configuration, I got the numbers below.
This means in the default configuration Camel is almost two times faster than the Spring Integration solution.

Exchanged 1000 messages in 2583 millis
Exchanged 1000 messages in 1710 millis
Exchanged 1000 messages in 1635 millis
Exchanged 1000 messages in 1184 millis
Exchanged 1000 messages in 1189 millis
Exchanged 1000 messages in 3420 millis
Exchanged 1000 messages in 712 millis
Exchanged 1000 messages in 683 millis
Exchanged 1000 messages in 659 millis
Exchanged 1000 messages in 824 millis
Exchanged 1000 messages in 825 millis
Exchanged 1000 messages in 609 millis
Exchanged 1000 messages in 577 millis
Exchanged 1000 messages in 705 millis
Exchanged 1000 messages in 589 millis
Exchanged 1000 messages in 597 millis
Exchanged 1000 messages in 580 millis
Exchanged 1000 messages in 570 millis
Exchanged 1000 messages in 587 millis
Exchanged 1000 messages in 579 millis
Exchanged 1000 messages in 593 millis
Exchanged 1000 messages in 600 millis
Exchanged 1000 messages in 628 millis
Exchanged 1000 messages in 597 millis
Exchanged 1000 messages in 577 millis
Exchanged 1000 messages in 649 millis
Exchanged 1000 messages in 592 millis
Exchanged 1000 messages in 587 millis
Exchanged 1000 messages in 642 millis
Exchanged 1000 messages in 601 millis

Best,
Christian

[Si-Coder]

Checkmate :-)

[oleg.zhurakousky]

@Christian

Sorry, you right i did get that wrong. Didn't realize that I had URL parameters thus overriding whatever default behavior there is

I'll follow up with more
Cheers

[muellerc]

Claus Ibsen also had a good remark:

instead of:
<inOut uri="bean:myService" />

we should use:
<transform><simple>${body}</simple></transform>

because that's more comparable with the SI configuration. It only returns the received payload.
With this configuration, it's again a bit faster:

Exchanged 1000 messages in 2270 millis
Exchanged 1000 messages in 1445 millis
Exchanged 1000 messages in 1414 millis
Exchanged 1000 messages in 916 millis
Exchanged 1000 messages in 984 millis
Exchanged 1000 messages in 652 millis
Exchanged 1000 messages in 612 millis
Exchanged 1000 messages in 611 millis
Exchanged 1000 messages in 780 millis
Exchanged 1000 messages in 881 millis
Exchanged 1000 messages in 606 millis
Exchanged 1000 messages in 550 millis
Exchanged 1000 messages in 541 millis
Exchanged 1000 messages in 590 millis
Exchanged 1000 messages in 551 millis
Exchanged 1000 messages in 530 millis
Exchanged 1000 messages in 535 millis
Exchanged 1000 messages in 532 millis
Exchanged 1000 messages in 535 millis
Exchanged 1000 messages in 528 millis
Exchanged 1000 messages in 574 millis
Exchanged 1000 messages in 571 millis
Exchanged 1000 messages in 561 millis
Exchanged 1000 messages in 596 millis
Exchanged 1000 messages in 572 millis
Exchanged 1000 messages in 583 millis
Exchanged 1000 messages in 610 millis
Exchanged 1000 messages in 601 millis
Exchanged 1000 messages in 549 millis
Exchanged 1000 messages in 555 millis
Exchanged 1000 messages in 584 millis
Exchanged 1000 messages in 534 millis
Exchanged 1000 messages in 742 millis
Exchanged 1000 messages in 560 millis
Exchanged 1000 messages in 566 millis
Exchanged 1000 messages in 546 millis
Exchanged 1000 messages in 535 millis

Best,
Christian

[juliaF]

Sorry for my late reply and thank you all who chipped in in this discussion.

Oleg, after I have changed cacheConsumers to false the performance has become stable, only caching consumers per session and not per selector. Many thanks for pointing it out.
During the tests we found quite challenging finding documentation on “fine” configuration points, which are not covered the your main Spring Integration reference documentation, in particular the mapping of xml elements to real java classes and good examples.

Below are the results of the tests with updated configuration.

The hardware and software setup as follows:

Producers/Consumers/Broker separated via physical network connection.
Producers running within JMeter on 12 64-bit cpus, 8GB RAM
6 consumers as standalone java apps deployed on a cluster of virtual machines each one was allocated 2cpu and 2GB RAM.
Broker deployed on 16 64-bit cpu, 8GB RAM
No other resource consuming apps were running on those machines at the time of stress testing.

Sotware Versions
Linux Ubuntu 10.12 & Java 1.6.0_31 64bit
ActiveMQ 5.5.1-fuse-04-01
Camel 2.10.0
Spring Integration 2.1.0_Release
JMeter 2.6

We ran 10 min sustained load tests with ramp up of 10 sec on variable number of producer threads. Producer/Consumer POJOs didn’t have to do any work to generate messages. The message out was Map<String, String> (64bytes), message back was Map<Integer, String> (64 bytes).

In the request-reply test where Camel & Spring Integration both had 30 producer threads the results were as follows:
Spring Integration throughput – aprox 1000 transactions per second, average response time about 30ms, steady throughout, but has spikes meaning that a small number of individual transactions was taking a very long time (over 200 ms).
Camel – approx. 3000 transactions per second, average response time after 8 min drops to about 14ms. Camel has an interesting change of profile after 4-8 min of sustained load run consistent with all tests we run with different number of producers where performance drops by about half.

Camel degrades slightly when we increased the number of producers. At 100 producers it was still processing about 2000 TPS and the ART was the 8 to 12ms (the same change in profile after 8min)
At 100 producers Spring Integration average response time increased to about 100ms and we saw more long running transactions spikes.

In one-way messaging Spring Integration performed considerably better then Camel. We were sending a 12K Map message.
Spring Integration’s avg transactions per second was about 13K ranging from 9 to 16 K, very consistent.
With client acknowledge mode average response time 0.4ms or even less for up to 20 producers, increasing to 1ms for up to 80 producers and about 1.5 for 90 and over. We reckon that Spring Integration can handle a considerably larger number of producers (100 and more) at high rate than other Camel.
The optimum number of producers for Camel was about 35, transactions per second rate during the first 4 min was around 8K, but then dropped to just over 2K and response time increased from 5 to 12 ms.
This change of profile in Camel performance was consistent for all other test runs on a different number of producers.

Some interesting observations regarding the resources used by both frameworks include:

In request /reply scenario
With Camel’s transaction throughput higher by approx. 70% than Spring Integration Camel used the same amount of memory and only 25% more CPU on the producer, 70% more CPU (comparable with its throughput) and 12% more memory on the consumer. Camel tests resulted in marginally less CPU use on the broker by about 6% (which may not be a meaningful result).

In one-way messaging scenario
For the amount of transactions larger by 65% Spring Integration used 50% less memory than Camel, and Camel used 40% more CPU on the producer. On the consumer Camel used 50% more CPU and 50% more memory. However, Camel tests resulted in 50% less CPU load on the broker.

We only tested JMS with two endpoints, no transformers, filters, splitters aggregators etc, so the throughput may be different for other connectors and other patterns.
Overall it seems that Camel is still faster in request / reply and Spring Integration is faster in one way messaging.
However, we are concerned with Camel’s consistent change in profile after a few minutes when its performance degrades in both request reply and one-way scenarios.

JMeter graphs attached.
Request/Reply test Spring Integration
Request-Reply test Spring Integration.jpg
Request/Reply test Camel
Request-Reply test Camel.jpg
One-Way test Spring Integration
One-way test Spring Integration.jpg
One-Way test Camel
One-way test Camel.jpg

[juliaF]

Request/Reply Camel Configuration
Producer

<camelContext id="camel-htmlprocessor-producer-context" xmlns=http://camel.apache.org/schema/spring trace="false" >
   	   	 <template id="producerTemplate"/>
    		<threadPoolProfile defaultProfile="true" poolSize="10" maxPoolSize="20" maxQueueSize="1000" rejectedPolicy="CallerRuns" id="defaultThreadPoolProfile"/>
    		<route id="htmlProcessorRoute">
        			<from uri="seda:peaches"/>
        			<to uri="activemq:htmlProcessorCamelQueue?deliveryPersistent=true&amp;replyTo=htmlProcessorCamelReplyQueue"/>
    		</route>
</camelContext>

Consumer

<camelContext id="camel-htmlprocessor-consumer-context" xmlns=http://camel.apache.org/schema/spring trace="false" autoStartup="true">
		<route>
		<from uri="activemq:htmlProcessorCamelQueue?concurrentConsumers=10" />
		<bean ref="processorCamelBeanStub" method="reply" />
		</route> 
	</camelContext>

One-way Camel Configuration
Producer

<camelContext id=" camel-impressioncounter-producer-context " xmlns=http://camel.apache.org/schema/spring trace="false" >
     <template id="producerTemplate"/>
    <threadPoolProfile defaultProfile="true" poolSize="10" maxPoolSize="20" maxQueueSize="1000" rejectedPolicy="CallerRuns" id="defaultThreadPoolProfile"/>
    <route id="impressionCounterPersistentRoute">
        <from uri="seda:apples"/>
        <to uri="activemq:impressionCounterCamelQueue?deliveryPersistent=true"/>
    </route> 
</camelContext>

Consumer

<camelContext id="camel-impressioncounter-consumer-context"  xmlns="http://camel.apache.org/schema/spring"	trace="false" autoStartup="true">
		<route>        
			<from uri="activemq:impressionCounterCamelQueue?asyncConsumer=true;&amp;cacheLevelName=CACHE_CONSUMER&amp;cacheLevel=3"/>
			<bean ref="impressionCounterCamelBeanStub" method="logCount"/>
		</route>
</camelContext>
Connection Factory
<bean id="jmsConnectionFactory" 
	   class="org.apache.activemq.ActiveMQConnectionFactory">
	   <property name="brokerURL" value="tcp://activemq:61616?jms.useAsyncSend=true" />
	</bean>
	<bean id="pooledConnectionFactory" 
	   class="org.apache.activemq.pool.PooledConnectionFactory">
	   <property name="maxConnections" value="10" />
	   <property name="maximumActive" value="10" />
	   <property name="connectionFactory" ref="jmsConnectionFactory" />
	</bean>
	<bean id="jmsConfig" 
	   class="org.apache.camel.component.jms.JmsConfiguration">
	   <property name="connectionFactory" ref="pooledConnectionFactory"/>
	   <property name="transacted" value="false"/>
	   <property name="concurrentConsumers" value="10"/>
	</bean>
	<bean id="activemq" 
	    class="org.apache.activemq.camel.component.ActiveMQComponent">
	    <property name="configuration" ref="jmsConfig"/> 
	    <property name="acknowledgementModeName"  value="CLIENT_ACKNOWLEDGE"/> 
	</bean>

Request/Reply SI configuration as above

One-way SI configuration
Producer

<task:executor id="taskExecutor" pool-size="1" queue-capacity="10" rejection-policy="CALLER_RUNS" />
    
    <bean id="persistentTemplate"  class="org.springframework.jms.core.JmsTemplate">
                <property name="connectionFactory" ref="connectionFactory"/>
                <property name="defaultDestinationName" value="siImpressionCounterQueue" />
                <property name="explicitQosEnabled" value="true"/> 
                <property name="deliveryMode">
                    <util:constant static-field="javax.jms.DeliveryMode.PERSISTENT" />
                </property>                                      
                <property name="sessionAcknowledgeMode">
                    <util:constant static-field="javax.jms.Session.CLIENT_ACKNOWLEDGE" />
                </property>
    </bean>
    
    <si:channel id="persistentImpressionCounterChannel" >
        <si:dispatcher task-executor="taskExecutor" />
    </si:channel> 
   
    <int-jms:outbound-channel-adapter id="persistentImpressionCounterChannelAdapter" channel="persistentImpressionCounterChannel" jms-template="persistentTemplate" />

   <bean class="org.apache.activemq.ActiveMQConnectionFactory" id="mq01-jmsCF">
            <property name="brokerURL" value="tcp://activemq:61616?jms.useAsyncSend=true" />
   </bean>

	<bean class="org.apache.activemq.pool.PooledConnectionFactory" id="mq01-pCF">
	    <property name="maxConnections" value="10" />
	    <property name="maximumActive" value="10" />
	    <property name="connectionFactory" ref="mq01-jmsCF" />
   </bean>
  
    <bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
        <constructor-arg ref="mq01-pCF"/>
        <property name="cacheProducers" value="true"/>
        <property name="sessionCacheSize" value="10"/>
    </bean>

Consumer

    <si:channel id="impressionCounterInboundChannel" />
    
  <si:service-activator  input-channel="impressionCounterInboundChannel" ref="impressionCounterServiceActivator" />

  <int-jms:message-driven-channel-adapter id="impressionCounterInbound"
  	destination-name="iImpressionCounterQueue"
  	channel="impressionCounterInboundChannel"
  	connection-factory="connectionFactory" concurrent-consumers="10"
  	max-concurrent-consumers="10" />

    <bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
        <constructor-arg ref="mq01-pCF" />
        <property name="cacheConsumers" value="true" />
        <property name="sessionCacheSize" value="10" />
    </bean>
     <bean class="org.apache.activemq.pool.PooledConnectionFactory" id="mq01-pCF">
	    <property name="maxConnections" value="10" />
	    <property name="maximumActive" value="10" />
	    <property name="connectionFactory" ref="mq01-jmsCF" />
  	</bean>
  
   <bean class="org.apache.activemq.ActiveMQConnectionFactory" id="mq01-jmsCF">
            <property name="brokerURL" value="tcp://activemq:61616" />
   </bean>

[oleg.zhurakousky]

Julia

First, we'd like to thank you for the effort. I am sure it was important to Camel guys as it was to us to see such independent comparison.
It was nice to know that in one-way messaging we were on top, but it also told us (along with your confirmation) that there is definitely room for improvement on the request/reply scenario.
We talked about possible improvements in the past but never got to the point of doing something about it. You can see some of the old discussions here https://jira.springsource.org/browse/SPR-3332 and you can probably find some more. . .
So we went back to see what's going on and here is the summary of it.
Our Outbound Gateway was relying on the most common JMS Message exchange Patterns:
1. Create temporaryReplyQueue if replyTo is not provided
2. Create reply receiving MessageConsumer with selector based on the MessageId of the request message.

The main point I am trying to make here us that these are "the most common JMS Message exchange Patterns" and thus in our opinion the most sensible default behavior. However, having said that we recognized that there are areas that could be improved (Thanks to you) hence the following JIRA https://jira.springsource.org/browse/INT-2683 which is at the final polishing stage and the improvements will be available in SI 2.1.4 (to be released this week) and obviously SI 2.2 to be released later in the summer.
However to foster early and independent confirmation of the figures below I am in the process of setting up a GitHub repo where anyone can run the same tests as I am running.

Also, I want to ask you if you indeed provided me with the correct configuration for Camel. The reason why I am asking is because your configuration for Camel was producing on average 1 message exchange per second (request/reply). That is exactly what I had initially and based on explanation from Christian and Claus (Camel dev team) that was not the optimal configuration so I went ahead and simplified and optimized it based on their recommendations and started to get comparable numbers - the same numbers as were reported by Camel dev team.
So the scenario I am reporting is as follows (for both frameworks)
One producer -> One Consumer with temporary reply queue. Independently run broker. Attempt was made to send and receive 1000000 messages in batches of 20000, so 50 consecutive runs (50*20000=1000000)

Hardware:
Processor 2.53 GHz Intel Core i5
Memory 4 GB 1067 MHz DDR3
Software Mac OS X Lion 10.7.4 (11E53)

Camel:

0. Exchanged 20000  messages in 19370
1. Exchanged 20000  messages in 11857
2. Exchanged 20000  messages in 11865
3. Exchanged 20000  messages in 12114
4. Exchanged 20000  messages in 12279
5. Exchanged 20000  messages in 12473
6. Exchanged 20000  messages in 12745
7. Exchanged 20000  messages in 12956
8. Exchanged 20000  messages in 13282
9. Exchanged 20000  messages in 13790
10. Exchanged 20000  messages in 14177
11. Exchanged 20000  messages in 14953
12. Exchanged 20000  messages in 15394
13. Exchanged 20000  messages in 15740
14. Exchanged 20000  messages in 16502
15. Exchanged 20000  messages in 16679
16. Exchanged 20000  messages in 16796
17. Exchanged 20000  messages in 17298
18. Exchanged 20000  messages in 20308
19. Exchanged 20000  messages in 96861
20. Exchanged 20000  messages in 267895
21. Exchanged 20000  messages in 439834

SI (with recent improvements):

0. Exchanged 20000  messages in 14822
1. Exchanged 20000  messages in 9148
2. Exchanged 20000  messages in 9292
3. Exchanged 20000  messages in 9133
4. Exchanged 20000  messages in 9138
5. Exchanged 20000  messages in 9211
6. Exchanged 20000  messages in 9286
7. Exchanged 20000  messages in 9291
8. Exchanged 20000  messages in 9277
9. Exchanged 20000  messages in 9270
10. Exchanged 20000  messages in 9278
11. Exchanged 20000  messages in 9258
12. Exchanged 20000  messages in 9284
13. Exchanged 20000  messages in 9286
14. Exchanged 20000  messages in 9334
15. Exchanged 20000  messages in 9314
16. Exchanged 20000  messages in 9342
17. Exchanged 20000  messages in 9296
18. Exchanged 20000  messages in 9380
19. Exchanged 20000  messages in 9363
20. Exchanged 20000  messages in 9416
21. Exchanged 20000  messages in 9456
22. Exchanged 20000  messages in 9564
23. Exchanged 20000  messages in 9439
24. Exchanged 20000  messages in 9397
25. Exchanged 20000  messages in 9435
26. Exchanged 20000  messages in 9277
27. Exchanged 20000  messages in 9328
28. Exchanged 20000  messages in 9471
29. Exchanged 20000  messages in 9312

As you can see I never got to 50 runs on Camel and I basically stopped at 30 in SI although you can also see that it moves right along (I did test several million messages before to ensure that it performs consistently).
The interesting thing is, Camel behavior resembles very closely the problem you've observed and reported initially for SI (gradual performance degradation). Although I am not sure what is the problem in Camel I can confirm that it is not the same problem as was observed in SI since our issue, as I described earlier, was consumer caching where for a non-tempReplyQueue scenarios new consumers were created for each message essentially overflowing memory with consumers that were used only once. And you did confirmed that disabling caching stabilized SI. Although just as an extra reminder the figures above are for TemporaryReplyQueue scenario (the named reply queue shows the same figures) and that is what makes me say that its a different problem in Camel then what we saw in SI although the end result is the same.

Also here are JConsole statistics for both which don't really need any comments from me.
CAMEL:

Time: 
2012-07-25 16:22:38
Used: 
   120,131 kbytes
Committed: 
   126,912 kbytes
Max: 
   126,912 kbytes
GC time: 
 9.461 seconds on ParNew (3,435 collections)

     2 minutes on ConcurrentMarkSweep (1,696 collections)

SI:

Time: 
2012-07-25 16:34:55
Used: 
    12,074 kbytes
Committed: 
    83,008 kbytes
Max: 
   126,912 kbytes
GC time: 
 2.063 seconds on ParNew (1,976 collections)

 0.000 seconds on ConcurrentMarkSweep (2 collections)

Anyway, as I said I'll be posting the link to the GitHub repo shortly but for the impatient once here are the configs:
CAMEL:
Producer:

<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
		<route>
			<from uri="direct:invokeCamelOutQueue" />
			<to uri="activemqOut:camelOutQueue" pattern="InOut" />
		</route>
</camelContext>
<bean id="activemqOut" class="org.apache.activemq.camel.component.ActiveMQComponent">
		<property name="connectionFactory" ref="connectionFactory" />
</bean>

Consumer:

<camelContext xmlns="http://camel.apache.org/schema/spring">
	<route>
            <from uri="activemqIn:camelOutQueue"/>
	    <transform><simple>${body}</simple></transform>
        </route>
</camelContext>

<bean id="activemqIn" class="org.apache.activemq.camel.component.ActiveMQComponent">
		<property name="connectionFactory" ref="connectionFactory" />
</bean>

ConnectionFactory:

<bean id="connectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
		<property name="maxConnections" value="10" />
		<property name="maximumActive" value="10" />
		<property name="connectionFactory">
			<bean class="org.apache.activemq.ActiveMQConnectionFactory">
				<property name="brokerURL" value="tcp://localhost:61616" />
			</bean>
		</property>
</bean>

SI:
Producer:

<int:gateway default-request-channel="jmsIn" />

	<int-jms:outbound-gateway request-channel="jmsIn"
	    connection-factory="connectionFactory" 
		request-destination-name="siOutQueue"/>

Consumer:

<int-jms:inbound-gateway request-channel="jmsIn"
		request-destination-name="siOutQueue"
		connection-factory="connectionFactory"/>
  
	<int:transformer input-channel="jmsIn" expression="payload"/>

ConnectionFactory:

<bean id="connectionFactory"
		class="org.springframework.jms.connection.CachingConnectionFactory">
		<property name="targetConnectionFactory">
			<bean class="org.apache.activemq.ActiveMQConnectionFactory">
				<property name="brokerURL" value="tcp://localhost:61616"/>
			</bean>
		</property>
		<property name="cacheProducers" value="true" />
		<property name="cacheConsumers" value="true" />
</bean>

Broker (for both):

<bean id="broker" class="org.apache.activemq.broker.BrokerService"
		init-method="start">
		<property name="persistent" value="false" />
		<property name="transportConnectorURIs">
			<list>
				<value>tcp://localhost:61616</value>
			</list>
		</property>
		<property name="deleteAllMessagesOnStartup" value="true"/>
</bean>

[oleg.zhurakousky]

As promised:
With a bit more investigation we were able to improve the Camel side of our comparison. By configuring Camel to use Spring's CachingConnectionFactory Camel performance improved but most importantly it stabilized and it was able to finish its run.

However, my word is just a word so, I would encourage you and the rest of the community to run the tests locally on your hardware and post the results. Please include the hardware specification as I am sure this exercise will help both Camel and SI team to improve.

To run the tests follow these instructions (you won't need anything other than Java and Git available on your machine - thanks to Gradle!!!)

Open Terminal Window

% git clone git://github.com/olegz/si-jms-perf.git
% cd si-jms-perf

That's it, now you can run the tests;

For Camel:

./gradlew runCamel

For SI:

./gradlew runSi