Sending a message to multiple endpoints with transforms.

[david_geary]

I'm currently trying to send a message (with xml payload) to 3 endpoints (file, email and webservice). Both the email endpoint and webservice endpoint require a transform on the xml payload before sending.

To implement this I'm using a RecipientListRouter to send the message from its incoming channel onto 3 other channels, then performing transforms on the message (using the new transform interface from trunk) on the email and webservice channels. However using this setup everything seems to be acting on the same message object producing unpredictable results as the payload gets transformed.

How should I implement this? Do I need to use a splitter to 'clone' the message? Is there any support for this or examples of it? Or is there a problem with the new transformer implementation in trunk?

Thanks

Dave

[Mark Fisher]

This cloning should probably happen in the router - perhaps as a configurable property of any router that may send to multiple channels (and should probably be 'clone=true' by default). Currently the router is just sending the same instance to each of the channels. It seems that the transformer should not care as long as the cloning is happening at the routing step. Does that make sense for your use case?

-Mark

[david_geary]

Hi Mark,

Thanks for your reply - yes I think that's exactly what I need. It should be up to the router to handle it. Should I raise a JIRA for this?

Dave

[david_geary]

Actually I've looked at what I'm trying to do and think I should be using a publish subscribe channel instead of a router.

The three targets email, webservice and file would be subscribed to this channel (This is really just an example - we are looking at providing a customisable event publisher mechanism on our product where for a particular integration an event may need to be sent to multiple targets)

However the same problem exists - publish-subscribe channels send the same message instance to each subscribed target handler. That means if any of those handlers need to do different transforms they will all be acting on the same object.

There seems to be a generic issue with sending a message to multiple targets and using transforms?

Dave

[Mark Fisher]

Dave,

Would you mind opening a JIRA issue for the addition of a "clone" flag for pub/sub channels and any multi-channel router?

I've started to make the changes for the router - basically anytime a router resolves channels and the set is > 1, then it will consult the flag (which is 'true' by default). To "clone" it simply invokes the GenericMessage constructor that accepts the payload and header (we might want to provide an actual clone operation instead). In general, does that solution sound appropriate for your scenario?

Thanks,
Mark

[david_geary]

Mark,

Yes I'll raise a JIRA for this. The approach you suggest should be what we need assuming the dispatcher for a publish / subscribe channel does a similar thing - (ie an extra attribute setting on the channel in the xml config ie publish-subscribe=true clone=true) and it defaults to true (if publish-subscribe).
The use of the GenericMessage constructor is fine for our needs for now.

In general Spring Integration seems to be shaping up well for what we need. We intend to make use of OXM marshalling also and hence a lot of use of XML payloads. I'll probably raise some other JIRAs for things like XSLT transformers etc.

Thanks

Dave

[iwein]

I get the feeling that Messages should just be immutable. That would make this whole problem go away.

How about putting this in the javadoc:
"Like any handler a transformer should refrain from modifying the incoming message or its payload. If a transformed message needs to be output it is required to create a new message with the transformed payload."

Methods should not have side effects and all that...

[mbogoevici]

Well, the whole point of having Transformers is to modify the payload of an existing Message, rather than swallow-a-message-and-produce-another-one which is the behaviour of a HandlerEndpoint. For example, one might need to preserve the message identity.

What is incorrect here and David pointed out quite well, is that we allow the same Message instance to be processed concurrently by multiple endpoints - which does not happen for linear message processing, but if the messages start being processed in parallel (pub-sub, multiple-destination-routing) this might become a problem when the message is manipulated (not only via payload transformation, but for example by a ContentEnricher). In fact, sometimes it might be desirable to create a copy of the payload itself if the transformers will treat the payload as a value object and modify it directly rather than replacing it (i.e. when the @Transformer method returns the same object as the argument).

David,
Thanks for adding the Jira item.

Cheers,
Marius

[iwein]

Well, the whole point of having Transformers is to modify the payload of an existing Message, rather than swallow-a-message-and-produce-another-one which is the behaviour of a HandlerEndpoint. For example, one might need to preserve the message identity.
Marius

I can see perfectly well the problem that Dave is pointing out would impact users if they don't work with immutable messages. What I don't see is a functional reason to preserve message identity. I don't want to have the whole 'equals()' vs. '==' OR immutable vs. mutable discussion again, but my point is that there is an easy way out: use immutable messages.

Imho a copy of the message is just as good as the message itself (and in fact the solution proposed to dave's problem implies the same). I still don't get it I guess...

Transformation: one thing goes in, another thing comes out. Sounds good to me.

[mbogoevici]

Transformation: one thing goes in, another thing comes out. Sounds good to me.

Iwein,

There are cases when the transformation occurs on the message itself (either payload or header). The typical example for that is Content Enricher. On another note, the identity of the processed message might be necessary to be preserved for a Claim Check based handling ...

The MessageTransformer interface has been designed for this mutable Message case, and that's why transform() does not return anything.

But I'm afraid we moved away from Dave's original issue: this can be solved relatively easy now, by introducing supplemental Endpoints before the e-mail and webservice Targets, which would take charge of the transformation in a MessageHandler - rather than using a MessageTransformer - so the original shared copy is not altered, but rather a new message is created every time. Of course, this should just give some relief until the cloning option will be available.


Cheers,
Marius

[Mark Fisher]

David,

We do plan to include an OXM-based MessageTransformer implementation in M5. Feel free to watch, vote, or add any comments to this JIRA issue: http://jira.springframework.org/browse/INT-109

If you'd like to open another issue for XSLT-based transformation, please do.

Regards,
Mark

[iwein]

Before I go off topic again, is the solution summarized by Marius workable for you Dave?



There are cases when the transformation occurs on the message itself (either payload or header). The typical example for that is Content Enricher. On another note, the identity of the processed message might be necessary to be preserved for a Claim Check based handling ...
It is perfectly possible to store a key in the message, use that to retrieve the data during the claim and create a new message with the data in it. There is no reason as far as I can see to require message identity preservation.

Relying on identity would be a bad idea imho. The whole point is loose coupling and if you don't know the route of the message you can't be sure that identity is preserved. Some component might persist messages in queues or send them over the network for example.

I'm not saying that mutable messages should be illegal or something, my only point is that I can't see a good reason you would want to use them. This brings us back to the transformer interface.



The MessageTransformer interface has been designed for this mutable Message case, and that's why transform() does not return anything.


It would be no problem to allow both mutable messages an immutable messages if we would return the message from the transform() method. I think that the interface needs to be changed. It doesn't have any advantages to return void.

Anyway, maybe I'm just addicted to immutable objects...

[mbogoevici]

]
There is no reason as far as I can see to require message identity preservation.

The reason for being able to preserve message identity is:

http://www.eaipatterns.com/MessageStore.html

implemented as

org.springframework.integration.message.MessageSto re.


]
It would be no problem to allow both mutable messages an immutable messages if we would return the message from the transform() method. I think that the interface needs to be changed. It doesn't have any advantages to return void.


OK, let's make sure we're on the same page here: first, by discussing the fundamental difference between a Transformer and a Handler: a Transformer modifies the existing message, whereas the Handler grabs the message and potentially returns a new one (it may not return anything, for that matter).

If we would return something from the transform() method, there'd be no difference with a MessageHandler (please compare the two interfaces). Now, that'd be just duplicating existing functionality, which would be nonsensical, right :)? So, the difference must be there for a reason, which is allowing the users to mutate the messages. There's no talk here on what's preferable in any given situation, it is what the framework supports or not.

If you want to use a strict immutable message-based processing model, which, in particular, I very much agree with, just use MessageHandlers instead of MessageTransformers for your handling, that's the whole point. Please understand their different purpose and different functionality.


Regards,
Marius

[iwein]

OK, let's make sure we're on the same page here: first, by discussing the fundamental difference between a Transformer and a Handler: a Transformer modifies the existing message, whereas the Handler grabs the message and potentially returns a new one (it may not return anything, for that matter).

If we would return something from the transform() method, there'd be no difference with a MessageHandler (please compare the two interfaces). Now, that'd be just duplicating existing functionality, which would be nonsensical, right :)? So, the difference must be there for a reason, which is allowing the users to mutate the messages. There's no talk here on what's preferable in any given situation, it is what the framework supports or not.

If you want to use a strict immutable message-based processing model, which, in particular, I very much agree with, just use MessageHandlers instead of MessageTransformers for your handling, that's the whole point. Please understand their different purpose and different functionality.


I do understand, but I still have some arguments with it. In any case we can take it offline. There is a clear understanding of Dave's problem, a solution and a workaround. My further arguments are irrelevant to this thread.

[mbogoevici]

In any case we can take it offline. There is a clear understanding of Dave's problem, a solution and a workaround.

Well, we walked a bit out of the main topic, as I said before. But I hope that it helped in explaining the fundamental differences between MessageTransformer and MessageHandler, as implemented by M4. And of course, let's hope it helps us build a better M5 ;).

[david_geary]

Iwein,

When you say marius' solution do you mean?


But I'm afraid we moved away from Dave's original issue: this can be solved relatively easy now, by introducing supplemental Endpoints before the e-mail and webservice Targets, which would take charge of the transformation in a MessageHandler - rather than using a MessageTransformer - so the original shared copy is not altered, but rather a new message is created every time. Of course, this should just give some relief until the cloning option will be available.

Currently I'm running the message through a clone handler in a chain with the transformer handler adapter to get round the problem. If I did the above I wouldn't be using the transformer interface anymore which I want to try to stick with as I have my own oxm,xslt and xpath implementations of it and this seems more consistent with where the next milestone release is going.

For me doing it either way - either cloning in the pub / sub channel and multi channel routers or having transformers behave more like handlers would work.

Actually originally I was using a handler to do the transform - but wanted to move to the new standard transformer interface and that's when I hit the problem

Although immutable messages is the simpler fix, I can see it's more flexible to have the framework distinguish between transformers and handlers.

Another option would be to put a clone option on the TransformerMessageHandlerAdapter?

Dave

[mbogoevici]

Hi Dave,

Indeed, the idea is to do transformation on a separate copy. Interface-related, the end goal is to be able to use pojos for transformation/handling, case in which switching from one to the other is a matter of modifying an annotation.

The Transformer model will definitely undergo changes in the M5, but this snippet from the unit tests should be illustrative:



private static class HandlerWithTransformers {

@Transformer
@Order(-1)
public String transformBefore(String input) {
return "pre." + input;
}

@Handler
@Order(0)
public String handle(String input) {
return input.toUpperCase();
}

@Transformer
@Order(1)
public String transformAfter(String input) {
return input + ".post";
}
}


Unfortunately, there's no namespace equivalent at this time, yet.

We plan on addressing the issue of cloning messages in a pub-sub scenario before M5.


Regards,
Marius

[iwein]

Iwein,
When you say marius' solution do you mean (quote omitted)?

Yes.


If I did the above I wouldn't be using the transformer interface anymore which I want to try to stick with as I have my own oxm,xslt and xpath implementations of it and this seems more consistent with where the next milestone release is going.

For me doing it either way - either cloning in the pub / sub channel and multi channel routers or having transformers behave more like handlers would work.

Actually originally I was using a handler to do the transform - but wanted to move to the new standard transformer interface and that's when I hit the problem

Although immutable messages is the simpler fix, I can see it's more flexible to have the framework distinguish between transformers and handlers.

One of my points was that we could distinguish and offer the option of doing a copy on transform (by adding a return type to the method). The downside that Marius pointed out is that we couldn't be sure of the identity preservation of the message anymore in that scenario.

One other important point of Marius that I missed is that it doesn't work that way now hence, if you want it to change, you should create an issue for it.

At the moment we're waiting for Mark to shed his light on this discussion (which will take a until after next week). If you want to move forward on the immutable path you could easily implement MessageHandler first and refactor later (to the better method name). You could also glue with something like...


abstract class ImmutableMessageTransformer implements MessageHandler{..}

...which is pretty horrible but very easy to refactor once MessageTransformer is changed (if it ever will be). It's still a bit dodgy, but that means it's good we have this discussion before RC1.

[david_geary]

Thanks for the feedback,




Indeed, the idea is to do transformation on a separate copy. Interface-related, the end goal is to be able to use pojos for transformation/handling, case in which switching from one to the other is a matter of modifying an annotation.



Off topic slightly, but one point to note is that what we are trying to do with spring integration is provide a customisable event publishing mechanism for our product, where we publish event messages with a known (published schema) xml payload on fixed message channels. The integration people will then write custom spring configuration files that get added into the spring context to transform and route the messages to do what ever they need to do.

Hence we are trying to stay away from custom POJOs (although they may occasionally be necessary) and in particular annotations as we are trying to rely on supplied transformers such as xslt and xpath (eg to pull out a bit of content for emails)

Hence the namespace support would be important to us (to make it easier for the integration people to configure) whereas the annotation stuff isn't really at present.

I'm interested to see what the namespace support for transformers / transformers chains will look like. In particular the code you show seems to show the application of pre and post transformers to a message handler. If a pre transformer could be added to message targets also then this could provide a very concise way to make web service calls ie something like




<channel id="eventpublishingchannel" publish-subscribe="true" clone="true"/>

<ws-target id="bpelTarget"
uri="http://localhost:8081/bpel/services/TestService"
channel="eventpublishingchannel"
transformer="bpelRequestTransformer"/>

<xslt-transformer id="bpelRequestTransformer" xslFile="C:\bpelRequest.xsl"/>



In this example the eventpublishingchannel would be the publish subscribe channel with the clone support as already discussed enabling multiple endpoints using the events on this channel.

The transforms could also be done in the target-endpoint tag.

Dave

[mbogoevici]

David,

Thanks for your feedback as well - it's very helpful for us. Namespace support is critical from our point of view as well, it's just that it needs to be ready for M5, as M4 had to be released before we had that part ready.

For target endpoints, it would look something like:



<channel id="eventpublishingchannel" publish-subscribe="true" clone="true"/>

<target-endpoint input-channel="exampleChannel" target="bpelTarget"
transformer="bpelRequestTransformer">
<schedule period="#somePeriod"/>
</target-endpoint>

<ws-target id="bpelTarget"
uri="http://localhost:8081/bpel/services/TestService"
channel="eventpublishingchannel"/>

<xslt-transformer id="bpelRequestTransformer" xslFile="C:\bpelRequest.xsl"/>



It's pretty much what you wrote, except that transformers will be placed on the endpoints themselves. Also, if there's more than one transformer per endpoint (i.e. a chain), they could be placed as sub-elements of <*-endpoint/>, the order being relevant in this case.

Cannot say 100% that this will be the final syntax, but this is by and large how it will look like.

The intent is for the transformers will be hosted in an endpoint (just like handlers) but, in principle, to operate on the same message (again, besides payload handling it can be a matter of header/envelope manipulation). In the case of a target endpoint, it is just pre-processing, in the case of a source-endpoint, it is just post-processing, and for a handler-endpoint they would provide either pre-processing or postprocessing, depending on their placement relative to the handlers.

Regards,
Marius

[kbohnenberger]

Can someone tell me how this all worked out?
I'm using spring-integration-1.0.2.SR1

I have a similar need. I need to essentially "multicast" a message out to many channels. Each of the "many" channels has the need to filter/validate/transform etc.

I was thinking of using the RecipientListRouter but I believe this thread points me to the Publish Subscribe channel.

However, the reference documentation in regards to the Publish Subscribe channel says this: "This is most often used for sending Event Messages whose primary role is notification as opposed to Document Messages which are generally intended to be processed by a single consumer"


Keith

[iwein]

Can you create a documentation issue for this? In my opinion we shouldn't document how things are used, we should document what they do and how you could use them. If you rephrase it for us once you understand that would be nice ;)