publishsubscribechannel when exception gets thrown in one of the subscribers

[Mark Fisher]

It sounds like you have a very good understanding of the behavior.

I don't know the details of your use-case, but typically it seems odd to have a JMS Queue followed by a "publish-subscribe-channel". Have you considered using a JMS Topic, or is there some reason for the Queue?

[kbohnenberger]

The high level picture of what we have going on:
Incoming messages go on the incoming queue.
The spring framework deployment pulls messages off the queue.
A set of "common" steps take place ... like schema validation, security validation, common normalization, etc
From there we need to send the message to several "subscribers" specific channels. Each subscriber path has a flow to transform and manipulate the message specific to that subscriber. The subscriber specific message then goes to a subscriber specific queue. Some subscribers write their own queue listeners and we write others. The ones we write we typically get the messages to the subscriber by file transfer or web service calls or something like that.

The thing I like about our current architecture/design is that I can scale by adding more "spring integration deployements." We also have redundancy because each deployment works off of the same incoming queue and publishes to the same subscriber queues. If one deployment would happen to go down, the message would stay on the incoming queue and another deployment would pick it up.

I thought about replacing the "publish-subscribe-channel" with a topic for the durability but then I lose the "add more deployments to scale" because I can only have one listener pulling messages off of the topic. I could create a topic per deployment, but then I lose the redundancy because each deployment has it's own topic.

I currently have things deployed with the publish-subscribe-channel set to ignore-failures=failse.
Things are working pretty well but I have two problems I'm worried about.
1) If a specific subscriber flow is really really slow, the other subscribers will have to wait for the slow one to get the message.
2) If one subscriber throws an exception it prevents the other subscribers from getting the message.

I was thinking of fixing problem 1 by adding the task-executors but then I run into the durability problem.

I was thinking of fixing problem 2 by adding the ignore-failres=true. I think this is my best path at the moment but I need to put something in place to make sure people get notified of a problem, the problem gets fixed, and the message gets resent.

In our world it is accpetable for our subscribers to get "duplicate messages" so the message that failed for one subsriber can be resent to all of them after the "failed subscriber problem" gets fixed.

I'm new to spring integration and relatively new to "ESBs" so I'm certainly open to other architecture/design ideas.

Thanks

Keith

[Mark Fisher]

Thanks for the detailed information!

Would it be manageable if you introduced another level of JMS Queues on the subscriber side of your "publish-subscribe-channel". That way you could achieve the following:
1) provide a new transaction boundary at the "handoff"
2) allow the subscribers to handle messages concurrently
3) avoid lost messages when one or more subscribers fail (assuming persistent Queues)

I guess it depends on how many such subscribers you have in the system.

[kbohnenberger]

The requirements people have promised my that the number of subscribers won't grow very much. Maybe 10 or so. Also some of the subscribers get what comes out of the "common steps" so I won't need to create another level of queues for those subscribers. I'll experiment and see how it goes.


Thanks for the suggestion.

Keith

[Mark Fisher]

Sounds good, and I was also thinking that this "second" level might be a good place to use JMS Topics after all (maybe even in place of publish-subscribe-channels).

Anyways, if possible, please report back as the project progresses. I'm also interested in any obstacles you encounter or ideas you have that we should consider for the 2.0 roadmap.

Thanks again,
Mark