SI - Separation of concerns for single process?

[djharte]

Hi,

I am hoping that the Spring Integration experts may provide some guidance as to whether they would view SI as a solution for single process data flow programming and a good mechanism for separating of concerns (in one process).

I have a use case where there are multiple processes (/services) with dependencies on each other all processing the same units of work, this seems like a pretty vanilla use case for SI, however some of the processes themsleves have sub-processes which are linked together in a graph (with decision points) where these processes process the same units of work as their parent processes. In the example below P1, P2 and P3 would be parent processes and the rest child processes where the same unit of work flows through all of them.

P1-->P2-->P3
-- P3A->P3B1
->P3B2

Separating out these child processes (P3A,P3B1,P3B2) into separate processes is not possible due to the way our data sourcing mechanism works and the overhead of managing them would not make it worth our while. I am planning on multithreading processing of the units of work for each of the processes. But there are still some things which trouble me:

1. Comprehension of the flow between the processes (P1,P2,P3) can only be determined by looking at our job scheduler (Autosys).
2. Processing of P1, P2 and P3 is currently sequential, i.e. P1 terminates, the P2 and then P3.
3. A failure in processing of just unit of work in one process results in that processing failing preventing the others from starting.
4. The internal flow for Process P3 is coded in Java, the code that does this currently is not intuitive i.e. the flow is not easy to determine.
5. It may be possible for us to collapse these processes into one which would allow for easier management.

It seems as if SI would help me address these concerns. But I am unsure of whether it is overkill or not appropriate for single processes were you are looking to use it to separate concerns.

Any thoughts on this would be much appreciated.

David

[djharte]

... to cut a long story short, do you guys think it is valid to use Spring Integration to separate concerns and manage their interaction within one process? Or, is there some other more appropriate framework out there for this use case?

[djharte]

... really, no one has an opinion on this ..

[Gary Russell]

Most of us are off this week.

This forum is really geared to answering specific technical questions; your question is at rather a high level and I (for one) would need to understand much more about the applications before I could make an honest assessment. For example, your point #2 - it's not clear whether you need to retain the sequential nature of these processes or not.

It is common to see an enterprise scheduler (such as autosys) being used to run a number of processes and have failure processing logic contained in its configuration. I see no real point in swapping out a scheduler just for the sake of it, unless there are other drivers.

That said, Spring Integration is a great fit for many scenarios.

If you can provide much more detail, or have specific questions, we're here to help (well, next week, anyway :-)