Sending a message to multiple endpoints with transforms.

[david_geary]

I'm currently trying to send a message (with xml payload) to 3 endpoints (file, email and webservice). Both the email endpoint and webservice endpoint require a transform on the xml payload before sending.

To implement this I'm using a RecipientListRouter to send the message from its incoming channel onto 3 other channels, then performing transforms on the message (using the new transform interface from trunk) on the email and webservice channels. However using this setup everything seems to be acting on the same message object producing unpredictable results as the payload gets transformed.

How should I implement this? Do I need to use a splitter to 'clone' the message? Is there any support for this or examples of it? Or is there a problem with the new transformer implementation in trunk?

Thanks

Dave

[Mark Fisher]

This cloning should probably happen in the router - perhaps as a configurable property of any router that may send to multiple channels (and should probably be 'clone=true' by default). Currently the router is just sending the same instance to each of the channels. It seems that the transformer should not care as long as the cloning is happening at the routing step. Does that make sense for your use case?

-Mark

[david_geary]

Hi Mark,

Thanks for your reply - yes I think that's exactly what I need. It should be up to the router to handle it. Should I raise a JIRA for this?

Dave

[david_geary]

Actually I've looked at what I'm trying to do and think I should be using a publish subscribe channel instead of a router.

The three targets email, webservice and file would be subscribed to this channel (This is really just an example - we are looking at providing a customisable event publisher mechanism on our product where for a particular integration an event may need to be sent to multiple targets)

However the same problem exists - publish-subscribe channels send the same message instance to each subscribed target handler. That means if any of those handlers need to do different transforms they will all be acting on the same object.

There seems to be a generic issue with sending a message to multiple targets and using transforms?

Dave

[Mark Fisher]

Dave,

Would you mind opening a JIRA issue for the addition of a "clone" flag for pub/sub channels and any multi-channel router?

I've started to make the changes for the router - basically anytime a router resolves channels and the set is > 1, then it will consult the flag (which is 'true' by default). To "clone" it simply invokes the GenericMessage constructor that accepts the payload and header (we might want to provide an actual clone operation instead). In general, does that solution sound appropriate for your scenario?

Thanks,
Mark

[david_geary]

Mark,

Yes I'll raise a JIRA for this. The approach you suggest should be what we need assuming the dispatcher for a publish / subscribe channel does a similar thing - (ie an extra attribute setting on the channel in the xml config ie publish-subscribe=true clone=true) and it defaults to true (if publish-subscribe).
The use of the GenericMessage constructor is fine for our needs for now.

In general Spring Integration seems to be shaping up well for what we need. We intend to make use of OXM marshalling also and hence a lot of use of XML payloads. I'll probably raise some other JIRAs for things like XSLT transformers etc.

Thanks

Dave

[iwein]

I get the feeling that Messages should just be immutable. That would make this whole problem go away.

How about putting this in the javadoc:
"Like any handler a transformer should refrain from modifying the incoming message or its payload. If a transformed message needs to be output it is required to create a new message with the transformed payload."

Methods should not have side effects and all that...

[mbogoevici]

Well, the whole point of having Transformers is to modify the payload of an existing Message, rather than swallow-a-message-and-produce-another-one which is the behaviour of a HandlerEndpoint. For example, one might need to preserve the message identity.

What is incorrect here and David pointed out quite well, is that we allow the same Message instance to be processed concurrently by multiple endpoints - which does not happen for linear message processing, but if the messages start being processed in parallel (pub-sub, multiple-destination-routing) this might become a problem when the message is manipulated (not only via payload transformation, but for example by a ContentEnricher). In fact, sometimes it might be desirable to create a copy of the payload itself if the transformers will treat the payload as a value object and modify it directly rather than replacing it (i.e. when the @Transformer method returns the same object as the argument).

David,
Thanks for adding the Jira item.

Cheers,
Marius

[iwein]

Well, the whole point of having Transformers is to modify the payload of an existing Message, rather than swallow-a-message-and-produce-another-one which is the behaviour of a HandlerEndpoint. For example, one might need to preserve the message identity.
Marius

I can see perfectly well the problem that Dave is pointing out would impact users if they don't work with immutable messages. What I don't see is a functional reason to preserve message identity. I don't want to have the whole 'equals()' vs. '==' OR immutable vs. mutable discussion again, but my point is that there is an easy way out: use immutable messages.

Imho a copy of the message is just as good as the message itself (and in fact the solution proposed to dave's problem implies the same). I still don't get it I guess...

Transformation: one thing goes in, another thing comes out. Sounds good to me.

[mbogoevici]

Transformation: one thing goes in, another thing comes out. Sounds good to me.

Iwein,

There are cases when the transformation occurs on the message itself (either payload or header). The typical example for that is Content Enricher. On another note, the identity of the processed message might be necessary to be preserved for a Claim Check based handling ...

The MessageTransformer interface has been designed for this mutable Message case, and that's why transform() does not return anything.

But I'm afraid we moved away from Dave's original issue: this can be solved relatively easy now, by introducing supplemental Endpoints before the e-mail and webservice Targets, which would take charge of the transformation in a MessageHandler - rather than using a MessageTransformer - so the original shared copy is not altered, but rather a new message is created every time. Of course, this should just give some relief until the cloning option will be available.


Cheers,
Marius