On-Demand Message Correlation

[bernerbits]

I have a need to provide realtime per-request auditing in my Spring Integration project. What happens is when a caller invokes the main gateway with a request, a UUID is generated and assigned to the REQUEST_ID header, and passed back to the caller. During processing, various interceptors are used to generate statistics messages on the fly and pass them to an outgoing JMS adapter, which publishes them to a HornetQ topic.

Where I'm stuck is this: the caller then needs to be able to use that request UUID and the HornetQ topic to receive statistics generated by that request on demand (NOT in realtime - I need to be able to invoke the caller over AJAX via a webservice call). In particular, I would like to be able to use the following Gateway interface to pull a message with the given REQUEST_ID off the topic:

com.mycompany.notification.activation.test.gateway;

import org.springframework.integration.annotation.Header;

public interface AuditMessageGateway {
	String take(@Header("REQUEST_ID") String requestId);
}

My initial thought was I could connect this via a direct channel with an outbound JMS gateway:

	<int-jms:outbound-gateway correlation-key="REQUEST_ID" id="auditGateway" 
		request-channel="auditTestChannel" request-destination-name="DLQ"
		reply-destination-name="notify.audit" reply-pub-sub-domain="true" 
		connection-factory="jmsConnectionFactory" />

	<int:channel id="auditTestChannel"/>

	<int:gateway service-interface="com.edo.notification.activation.test.gateway.AuditMessageGateway"  
		default-request-channel="auditTestChannel" 
		default-reply-timeout="7000" />

Since there is no payload, it wouldn't send an outgoing message, but it would still listen on the topic using REQUEST_ID as a correlation key.

However, it doesn't seem like JMS gateways behave quite like SI gateways. According to the documentation they always send out a message. I was able to get around this like so:

	@Payload("''")
	String take(@Header("REQUEST_ID") String requestId);

But that's not good because it fills up the DLQ with a ton of empty messages.

Am I going about this properly? Is there a better way to accomplish my core requirement (that is, polling messages, on demand, from a topic, selected based on the value of a JMS message property)?

[bernerbits]

On reflection, it occurs to me that I am attempting to solve a much harder problem than I initially thought.

To provide realtime statistics on demand to a web client, I'll need to have a durable subscriber open per web session. That leads to further complications, because we need to be able to cluster our tomcat instances.

This is going to require more thought.

[Mark Fisher]

Have you considered using a data store to keep track of those statistics? Something lightweight (MongoDb, Redis) might do the trick.

[bernerbits]

As a senior developer I have some degree of influence here, but I would have to make a very compelling case to go the NoSQL route since we don't use it in any of our existing applications.

The current plan is to store statistics for weekly auditing, and to use our existing PostgreSQL database for that. The system that this project is replacing accomplished near-realtime statistics in this fashion with a 5 second poll that grabs the latest info from the database, and it almost always brought the production environment to its knees during peak load.

My hope was that using a subscribable event model, we could significantly cut back on the amount of communications and DB load required in a traditional polling model, in addition to providing visual feedback with much lower latency.