Spring Integration with Spring Batch - Message Triggered Jobs

[gauravd]

Hi All,

I have a requirement wherein I have to do design J2SE based applications that could do bulk processing operations. Simply put, these applications could do -

1) Process bulk data (from DB to DB , File to DB etc)
2) Have status reporting of the data Jobs (above) - the richer the better.
3) Event driven scenarios using messages.

Due to performance reasons, I was thinking of hosting individual applications inside its own JVM based on business goals. So J2SE Spring batch application at JVM1 could do bulk processing for one big requirement, J2SE Spring batch application at JVM2 could another set of processing and so on...

Spring Batch+Spring Integration+Spring Batch Admin seems to be a great solution for this. For inter JVM communication, I was thinking of having Spring Integration layer hosted at one JVM (master) that could act as a controller for issuing events to Spring Batch applications hosted on different JVMs based on messages (Spring core messaging). I would like to avoid JMS or ActiveMQ or alike for this rather go for Spring core messaging.

But Iam stuck with problem of inter JVM processing through Spring Integration. Logically It seems that the Spring Integration and Spring Batch configuration context needs to be hosted at same JVM in order to be managed by Spring. Is there any way in which I can expose the message channels of Spring Integration at JVM 1 so that Spring batch jobs configured at another JVMs can communicate with Spring Integration layer bidirectionally?

Any other ideas would be immensely useful. Please help ... Iam exhausted googling for it.

[kgrodzicki]

Hi,

Is there any way in which I can expose the message channels of Spring Integration at JVM 1 so that Spring batch jobs configured at another JVMs can communicate with Spring Integration layer bidirectionally?

There are many ways to do it. As I understand you want to use control bus pattern for your application. For communication layer between instances(services) you can use from the simplest tcp/udp, rmi, http based like rest, XML RPC to more advanced protocols like AMQP, of course everything depends on you requirements(security, scalability, performance, reliability etc.).
I think article 'Using RabbitMQ, Spring AMQP and Spring Integration' will be helpful for you.
If this is possible describe the problem which you want to solve in more details, then I will try to help you with architecture and technology.

Krzysiek

[gauravd]

kgrodzicki ,

Thanks a lot for the help. The article is very nice. Ill try my best to describe the solution. However please keep in mind that for various reasons, a solution with lot of technology dependencies would be well...not considered at all.

Problem
--------
There are legacy back end applications based on pure C for which we are thinking of redesigning in Java. There dedicated C executables/programs/jobs for quoting, calculation, reporting, annual bulk processing etc..
All these programs right now rely on daemons (monitoring) , shell scripts (start/stop) and database status to determine the order of executation and all these prgrams do lot of parallel processing through forking and shared memory.

I was thinking of devising dedicated J2SE programs - one for calculation, one for quoting etc... each of these programs to be hosted in different JVMs (memory would be a concern) and utilize the message triggers event driven approach to glue all the JVMs up... Spring Batch + Spring Integration + Spring Admin seemed a good solution to me. However i had one assumption - that Spring core messaging would enough for me.. where i thought a message could trigger a Spring batch job, but i think this solution is only viable if Spring Batch and Spring Integration sits in one JVM.

So for multi JVM solution, I would have to opt for enterprise message brokers... MQ Series...RabbitMQ...JMS providers... etc..

Also as a side note, i am exploring a clustered JVM solution with inter JVM communication.

I would love to hear your opinion or suggestion.

Thank you,

[oleg.zhurakousky]

where i thought a message could trigger a Spring batch job, but i think this solution is only viable if Spring Batch and Spring Integration sits in one JVM

Not true. That is where remote adapters could be applied (HTTP, JMS, AMQP, TCP/IP etc.) to cross not only JVM boundaries but platform boundaries as well.

Also, don't underestimate the power of service activators capability of invoking executable processes with SpEL
For example:

<int:service-activator input-channel="input" output-channel="nullChannel" expression="T(java.lang.Runtime).getRuntime().exec('echo ' + payload)"/>

This means that your C programs don't need to be re-factored right away and could be easily integrated by simply passing some message argument as a program argument etc.

[oleg.zhurakousky]

Of course, I want to follow up with the word of caution.
What i showed above (Runtime and SpEL) is obviously OS/Platform specific and could not be even thought of as a viable solution for enterprise/cloudy application.
The only point I was trying to make is if you look at the migration process iteratively then while thinking about migrating that C program or while migrating it, you don't have to stop thinking about integration. You can as a temporary and mainly for localized tasting use the above approach knowing that its going to change once your refactoring efforts are complete. And then its just a matter of changing the endpoint.

[gauravd]

oleg.zhurakousky - Thanks a lot for your inputs. The service activator SpEL is wonderful. However, due to portability issues, we are trying to do away with C programs(ancient ones really), so this might not help in my scenario.


"Not true. That is where remote adapters could be applied (HTTP, JMS, AMQP, TCP/IP etc.) to cross not only JVM boundaries but platform boundaries as well."

This sounds promising. Do you mean that Spring Integration can be controller and sit on one JVM while individual Java programs (Spring batch jobs) can be listeners to events (triggerred by messages) at other JVMs? I was searching for code similar to this usecase.

Can you please help me out?

Thank you.

[anshumania]

I am also trying to develop a solution to have spring-batch jobs triggered via messages from spring-integration.
You may want to reference this

I am pretty new to spring-integration and batching. However from the looks of it ; if you do not want to use a messaging broker in the middle you can simply use the adapters provided by spring-integration. You can have multiple spring-integration contexts controlled (identified by control buses if you want)
a) have your master jvm have a spring-integration context with all your business intensive processing transforming and routing
b) based on your logic send out messages on any of the adapters spring-integration provides( http, file, db , ...)
c) the message can contain information about the job to launch in anyway you choose ( as a http params, http post, db table, txt file ... )
d) on your slave jvm's have a lightweight spring-integration context listening in on of those channels
e) using the example shown just @ServiceActivator your JoblaunchingMessageHandler and reference your existing spring-batch joblauncher to launch your job.



i may have missed out on some intricacies but that should do it i guess. but yepp i am not an expert so ...
It should( and probably will) help to have spring batch admin deployed in the slave jvms for managing or analyzing batch jobs but i am not sure if its necessary.
Does the spring-batch-admin provide any extra set of (tools?) interfaces to connect to spring-integration ?

[anshumania]

can some one give pointers to anything new in spring (integration?) for launching jobs in a different context but within the same container ?
( same question as accessing beans in a different context deployed in the same container)

[anshumania]

I tried one approach. I took the spring-batch simple template project and used an integration context file to launch a job (using the JobLaunchingMessageHandler) so it would seem that i can seamlessly invoke it from another remote jvm via another spring integration context. However i did notice that the JobLaunchingMessageHandler takes as input a JobLaunchRequest (which would encapsulate the job and the jobparameters).
So i could just create a simple job with an id and pass empty jobparameters and based on the job id fetch the appropriate jobparameters to launch a job in the other jvm.
Is there a better way to do this ?

[Mark Fisher]

You might want to consider if the JobParametersIncrementer strategy in Spring Batch could work for your use case.