Performance considerations for JMS Request-Reply pattern.

[mers]

Hi,

I've written a simple request-reply application using Spring Integration and JMS gateways, but I need some help understanding how to configure its channels and gateways to get it to scale.

The application consists of a client and a server. The client sends request messages to a DirectChannel using a MessageChannelTemplate.doSendAndReceive() call. A JMS outbound gateway forwards messages from this request channel out over JMS, where they are read by a JMS inbound gateway on another JVM. A service activator registered on the remote request channel passes the message payload to the server MessageEndPoint. The service activator returns an Object which must be delivered to the Client who originated the request.

The XML I have so far is...

<integration:message-bus />

<integration:channel id="requestChannel" />

<integration:service-activator input-channel="requestChannel" ref="server" />

<bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
	<property name="brokerURL" value="tcp://localhost:61616" />
</bean>

<bean id="requestQueue" class="org.apache.activemq.command.ActiveMQQueue">
	<constructor-arg value="poc.activemq.requestQueue" />
</bean>

<!-- on client only -->
<jms:outbound-gateway id="outboundGateway" jms-queue="requestQueue" request-channel="requestChannel" />

<!-- on server only -->
<jms:inbound-gateway id="inboundGateway" destination="requestQueue" request-channel="requestChannel"
	concurrent-consumers="2" />

This setup works, but I have a few questions I'm hoping someone can help me with.

1. I'm getting a "removing non-serializable header" warning regarding the REPLY_CHANNEL header in the Client request. Should I not use MessageChannelTemplate's doSendAndReceive() method, and instead use a MessageBuilder and call setReplyChannelName()?

2. A new TemporaryQueue is being created on the JMS broker for every outgoing message when using MessageChannelTemplate's doSendAndReceive() method. Looking at JmsOutboundGateway source code, the instantiation of a new QueueRequestor object for each handleRequestMessage() call makes me think this is unavoidable. Can it be avoided?

3. ActiveMQ have a different take on how temporary queues should be used for the Request-Reply pattern, namely to keep a temporary queue per client and use correlation IDs to match responses to requests. (http : // activemq.apache.org/how-should-i-implement-request-response-with-jms.html) Is this approach possible?

4. What is the correct channel type of the request channel on the Server side? I would like the same JmsInboundGateway thread which is pulling messages off the JMS queue to execute the server activator code.

5. Is there anything you would change in the pasted XML?

Thank you very much in advance,
Emerson

[Mark Fisher]

Thanks for the feedback! I've provided some brief answers to your questions. Please let me know if you would like any further clarification or have additional feedback.

1. I'm getting a "removing non-serializable header" warning regarding the REPLY_CHANNEL header in the Client request. Should I not use MessageChannelTemplate's doSendAndReceive() method, and instead use a MessageBuilder and call setReplyChannelName()?

When the MessageHeaders object is serialized, any non-Serializable header values are simply removed. However, the gateway maintains the request headers and copies them back (if not already replaced) before sending the Spring Integration reply Message. Therefore, the "local" replyChannel value will be maintained. For RC2, this log message will be at INFO level instead of WARN.

2. A new TemporaryQueue is being created on the JMS broker for every outgoing message when using MessageChannelTemplate's doSendAndReceive() method. Looking at JmsOutboundGateway source code, the instantiation of a new QueueRequestor object for each handleRequestMessage() call makes me think this is unavoidable. Can it be avoided?

See answer to question #3 below...

3. ActiveMQ have a different take on how temporary queues should be used for the Request-Reply pattern, namely to keep a temporary queue per client and use correlation IDs to match responses to requests. (http : // activemq.apache.org/how-should-i-implement-request-response-with-jms.html) Is this approach possible?

We have been exploring a similar approach. Obviously QueueRequestor was the easiest thing to start with but not ideal as you've pointed out. I've created a JIRA issue so that you can watch the progress: http://jira.springframework.org/browse/INT-466

4. What is the correct channel type of the request channel on the Server side? I would like the same JmsInboundGateway thread which is pulling messages off the JMS queue to execute the server activator code.

When you use the "channel" element without a "queue" sub-element, it will create a DirectChannel instance. That means the service will execute in the same thread that is sending the Message (in your case, the same thread that is receiving from the JMS queue as well).

5. Is there anything you would change in the pasted XML?

The configuration looks good to me.

Regards,
Mark

[mers]

Thanks for the responses.

I've spent another day digging through the documentation and the code. There are a few more questions I've come up with...

1. The JMS QueueRequestor used within JmsOutboundGateway has no timeout parameter, since it's designed for the Request-Reply pattern. Does this play well with the MessageChannelTemplate if the request-channel is a DirectChannel? If the template thread is executing the JmsOutboundGateway code, and nothing picks up the message over JMS, then the QueueRequestor call will block. The doReceive() call following the doSend() call will never be executed, and the client will hang, even though a template timeout is set. Or am I missing something?

2. I looked at JmsSendingMessageHandler as a replacement for the JmsOutboundGateway. Using a DirectChannel, a client thread would execute the JmsTemplate code (which is thread-safe). Hopefully I'd be able to have ThreadLocal Sessions and Producers involved but I'll check. Anyway, this leaves the question of how to get the replies from the server.

From my understanding of the HeaderMappingMessageConverter, the client should be able to specify a REPLY_CHANNEL header which would get translated to a JMS temporary queue. If a <jms:inbound-channel-adapter> is defined for each queue, I should be in business (though I think that creating such adapters programmatically violates separation of concerns if not properly encapsulated). Do you think this is the way to go given the current capabilities of SI?

As a future enhancement, I think it would be brilliant if the client-side outbound gateway or inbound adapter would demultiplex a single temporary queue to multiple channels. So there would be one JMS temporary queue per client, but one SI channel per request. I think this would give the best of both worlds in terms of ActiveMQ performance and client simplicity. This assumes channels are very lightweight. (Are they?)

3. The JmsInboundGateway creates a new producer for each response. These can probably be pooled, especially if temporary queues are later created per client instead of per request.

Sorry for the question/comment overload,
Best regards,
Emerson

[Mark Fisher]

I've committed some changes to the JmsOutboundGateway. Basically, it no longer uses a QueueRequestor. It is now much easier to tune the performance by simply providing a suitable connectionFactory (e.g. Spring's CachingConnectionFactory).

It now supports a 'replyDestination' property. If provided, it will use a MessageSelector to receive each Message with the expected CorrelationID (JmsInboundGateway also now sets the correlationId to match the request Mesasge ID).

If the 'replyDestination' is not set, it will continue to create a TemporaryQueue.

I'll post back with more comments on your recent questions when I have a chance.

Regards,
Mark