Mar 26th, 2012, 03:02 PM
Design for Extension and Dynamic Routing in SI
Apologies if this is rambles on a bit - it's a relatively complex issue and I'm wondering if anybody has come across it before.
We've been thinking a lot about how to deal with extension via configuration in our current Spring Integration project and we were wondering if anybody had any thoughts on the best approach or suggestions.
For background we are using SI as the core of a high-volume transaction processing platform - there is a fairly well defined existing transactional integration pipeline (i.e. : accept message, perform basic validation, get context, check config, process message, store results, construct response , reply (with the usual exceptional cases etc)) , conceptually to us there is a "well trodden path of integration" which we'd like to preserve as best as possible. What we are keen to understand is to what extent we could "extend" the functionality of this at load time.
In the past We've used SI as a platform/framework for arbitrarily extending basic service functionality (e.g. new API calls) by sitting new features on filtering channels (e.g. adding a "Foo" call on a service API message channel) which has worked very well. But now we're considering if or how we could do the same for AOP-style cross-cutting of an existing (but well defined) pipeline.
We have a number of features which we know we could integrate by extending the current pipeline in various places - a review of these features has led us to the conclusion that almost all of them can be implemented using one or more of the following patterns inserted at one or more places in the existing pipeline:
* Header enrichment (e.g. early on in the pipeline validating building and storing the context of one feature)
* Later content enrichment from those headers (preserving the existing payload types e.g. consolidating early results into a response)
* Re-routing /"bailing out" out of the existing "well trodden" pipeline to an egress point (there is no need to "re-join" the pipeline) (e.g. something like a pluggable validator which diverted a message into a validation errors channel and on to response in the simplest case)
We've identified a set of extension points in the existing pipeline that everything we're currently planning would fit into, all of the features are essentially isolated (barring one or two clear dependencies (e.g. A, A +B but not B) ) ordering of individual extension is also not likely to be a big issue (simple integer ordering would suffice where needed) .
If possible we'd like to be able to add features by say just including the relevant feature's spring config files in the app context load (which is what we've done before and works pretty well, we think) .
Each feature can currently be expressed as its own spring context (sub to the application context) + set of SI chains "extensions", each of which currently integrates with a single "extension point" on the main pipeline (so one feature might introduce several cooperating extensions) , each extension either passes the message through (possibly with transformation and elaboration) to the next handler , or diverts it out to it's own channel, aborting the main message flow.
The case for us being able to do this is compelling as it looks like it offers the flexibility we think we need, while preserving some or all of the rigidity and understanding of the core pipeline as well as allowing each feature to be tested end-to-end in context without the others. It's conceptually close to AOP in the normal java world but with a limited set of cut points (we are all-too-well aware the pitfalls and risks of trying to use AOP to deliver core functionality) .
The real question is - has anybody tried to do this sort of AOP-like extension to a pipeline before, and what worked?
What I think would work is a sort of "Pluggable" SI primitive which sat at each of these extension points and did the following:
* Was configured for the extension chains/channels relevant to that point
* Could dispatch the message down each of those channels such that:
** If that chain did not "divert" the message the outcome of the chain message was dispatched down the next chain until all chains were visited
** If it did divert the message then processing would stop from the point of view of the remaining chains (and ongoing pipeline)
We've tried a few off-the-shelf compositions of pub/sub and existing routing primitives but nothing has spring out as the right solution.
Tags for this Thread