Reliable delivery and JobManagement with Persistent Queues

[smjain]

First of all Merry Christmas and a very Happy new year..



We want to introduce a notion of job management where each job is composed of processing code behind Channels frontended by a persistent Queue(most probably RabbitMQ)

As an example

Say we have 2 channels each behind a different queue..Now the job is completed when each channel executed successfully.

If say one channel execution fails(it will have a retry mechanism) and finally abort if retry fails..When we abort we push message to a DLQ(Dea letter Queue)

Now from a UI which we will develop we should be able to monitor jobs(I am thinking message itself is the job as it contains the execution context itself)

So if say a job has two Channels(A and B ) and B has failed..From Ui we should be manaully able to restart execution from B itself..

In that case message will be pulled from the DLQ and pushed to the Queue which has listener for processing action B..

I think by using this architecture we achieve reliability as well as dynamic behaviour by introducing more Channels and wiring them through SI

Is this a good/valid idea or do u suggest anything else..



Thanks and Regards

Shashank

[oleg.zhurakousky]

Have you looked at Spring Batch project? It has a notion of job and nicely integrated with spring integration

[smjain]

Yes I am looking into it...

Unable to figure out that if we have different applications to be part of a job execution should they be Steps in Spring Batch or abstracted behind a channel

Or a Spring batch step calls the SI channel which invokes the application/s code.

[Dave Syer]

The Spring Batch domain model more or less mandates that you make everything inside a Job into a Step, but that doesn't mean you can't implement it behind a remote call or send-and-receive (gateway) style messaging. How you should implement your Step depends on what work needs to be done and where - since you say it is an "application" does that mean it is a remote process?