Multiple source channels into a destination channel

Hi,

We are currently using spring-integration in one of our projects and we are trying to find an easy way to solve the issue mentioned below. This is the first time we use spring-integration so we may be missing something.

We have a set of publish-subscribe channels (sources) that we use to publish events to. On the other hand, we have component that has the capability to subscribe to a channel and receive events from it. We need to collect all the events published on the source channels and publish them into a destination channel, which the component will subscribe to.

We have taken a look to the Bridge component but it generates a copy of the message and that is something we can not afford. Is there any other way of doing this?

Thanks,
Diego

At the first glance it seems like you need an aggregator to aggregate all the events into a single message, but this raises the question:
How do you know that all of the events were published? Is it a set value? Time intervals?

Also, I am not sure I understand what do you mean by "bridge generates a copy of the message and that is something we can not afford". I am not sure about "affordability" part. Messages are immutable, every time message goes from channel to channel a new Message is generated. . .

Thanks for your answer.

We don't need to aggregate the messages, we just need to "route" messages from different sources into one channel.

Why the message needs to get copied if it is immutable ? I mean, if they really are immutable why to generate a "defensive" copy of it.

As far as I've seen in the code, send method does not generate a copy, neither the Dispatcher (it does a copy when it requires to alter the message), so I am not sure about a "new Message" is generated when it enters or move into channel. I have not found anything in the documentation regarding to this subject.

Diego

Well if you simply want to route from multiple sources to a single channel, then using Bridge is just fine.
I still don't understand your concern with new message creation. Are you experiencing some issues with that?
The creation of a new message for each hop ensures message immutability. It is really a question of understanding Message life cycle as well as the concept of a request and reply message. For example once you created a message and send it to a channel it is a request message. This message will go from the channel to a Message Handler (Bridge in your case) which is responsible to produce a reply message which will be created as a new message and put on the reply-channel and so on. . .

You can read more here:
http://static.springsource.org/sprin...essage-builder

Bear in mind, too, that making a new message does not mean the payload or headers are copied; Spring Integration simply constructs a new message containing the same payload together with a new headers object, which contains the same headers, as well as some additional history content.

It is a very lightweight operation, if performance is your concern.

We have special needs regarding latency, and we need to handle a huge amount of messages, therefore unnecessary message copying is something we need to take care of.

I understand about "cheap" copies, nevertheless this is just to understand the need of copying a message that is not modified since most of the components do not generate copies of them.

For instance, the Router takes messages from one channel and sends them to another channel, and it does NOT copy the message if it does not modify it. It just generates a copy of the message when applySequence is true.

So I may use many "one input" - "one output" routers to route messages from many sources to one destination, avoiding message copying. Of course I don't want to do this, I would prefer in this case to implement a custom endpoint but I need to be sure I would not be breaking any contract.

Thanks

What is the ultimate consuming component? You might want to consider having it subscribe directly to each of the publish-subscribe channels.

Have you run some performance tests to determine that this issue with the BridgeHandler actually does present a problem within your parameters? (on my 4 year old laptop, I sent 1 million messages through a bridge just now in 3 seconds).

Can you describe the use-case in a bit more detail, so we can consider other options? For example, how many publish-subscribe channels are there, and what differentiates those channels from the producer's perspective?

Quote:

---

We have special needs regarding latency, and we need to handle a huge amount of messages, therefore unnecessary message copying is something we need to take care of.

---

Have you profiled your app and determined that message copying takes you beyond your SLAs or this is just your guess that your SLAs will be affected by message copy?
Don't get me wrong, but this is the first time someone ever brought this up as a potential problem and realizing that we have quite a few customers with similar or more stringent performance requirement I am really curious if you have any numbers to show

Quote:

---

the need of copying a message that is not modified

---

Not true. Event with the Bridge message is modified to generate MessageHistory event

The component does not need to know about all the channels, we want to keep it as simple as possible and resolve the channel routing and selection at spring-integration level (configuration).

Before continuing with the performance topic, I would like to state that my intention was to know if there was a component that could route messages from multiple sources channels into one destination channel. We were using Bridges to accomplish this, but since it requires one bridge per source channel, we were looking for a better option.
In addition, while profiling the application we noticed the bridges were copying the messages (creating a reply), and that is something we would like to avoid if possible.
It seems that if we use the Router we avoid the copy of the messages, but we would still need to define a router per source.

Regarding to performance, we need ultra low-latency and we need to keep it as stable as possible. We would like to avoid copying messages (avoid garbage generation) as long as we can and for what we've seen spring-integration does a good job with this. It only copies messages if it requires to alter them. The only issue we have noticed that would break the contract is the "History event" addition that is done in the AbstractMessageHandler class (AbstractMessageHandler#170) which modifies the "immutable" message and does not make a copy of it.

Additionally, since we are expecting to hit spring-integration components millions of time per second, we will need to avoid unnecessary object instantiation in many other places. We have made small changes in the some classes of spring-integration core and we have reduced the number of outliers we had noticed while benchmarking latency. We are open to collaborate our additions if they are welcome by the community.

Best,
Diego

Quote:

---

Originally Posted by oleg.zhurakousky

The creation of a new message for each hop ensures message immutability.

---

If something is immutable, you don't need a copy. Copying does not ensure immutability.

Creating objects is cheap, so I don't see the problem.