How to integrate with Sprint Batch Integration or other Grid Computing technologies

[Saravan]

Hello Gurus/Senior Members,

My requirement is to process millions of records. Each of the record involves some complex processes before updating to database. Hence we thought of using partitioned approach to run on different JVMs.

In Spring Batch Documentation I found that, Spring Batch Integration sub project can be used to accomplish this. However I could not find any other documentation on this.

Could you please guide/direct me to right resource which can help me in setting up batch jobs to run in partitioned mode and make use of multi JVMs.

Your help in this regard is much appreciated.

Best regards,
Saravan.

[Dave Syer]

Spring Batch Integration docs are at http://static.springframework.org/sp...h-integration/. The remote chunking invocation pattern you are talking about that was mentioned in the main User Guide is implemented in the ChunkMessageChannelItemWriter (and friends in teh same package).

Remote chunking is not the only way to go about remote step execution. The PartitionHandler SPI is also available, as per the User Guide There is even an implementation in Spring Batch Integration (MessageChannelPartitionHandler).

I am going to write an article on Spring Batch Integration when I get time to try and raise awareness and get more people to use it. In the meantime, look at the website and the source code (including unit tests) to learn about the project.

[Saravan]

Much Thanks Dave,

Unfortunately I don't see downloadable package for SPI or the source code. Could you please let me know from where I can download the same? Thanks.

Best regards,
Saravan.

[Dave Syer]

PartitionHandler is part of Spring Batch. The Integration project is in SVN alongside the other Batch projects - look for the source control link on the main website.

[Saravan]

Hello Dave,

I downloaded the Spring Batch Integration 1.0.2 from Spring Source download page. I could not find the reference of MessageChannelPartitionHandler. Could you please let me know where I can find it? Thanks.

Best regards,
Saravan.

[Dave Syer]

There's no download zip for 2.x. Look in SVN (https://src.springframework.org/svn/...ch-integration, https://fisheye.springsource.org/bro...ch-integration).

[Saravan]

Hello Dave,

Did you get a chance to write an article on Spring Batch Integration? If yes can you share me article's URL?

Thanks.

[Dave Syer]

Still gestating. I'll post the link when I get round to it.

[snicoll]

Spring Batch Integration docs are at http://static.springframework.org/sp...h-integration/. The remote chunking invocation pattern you are talking about that was mentioned in the main User Guide is implemented in the ChunkMessageChannelItemWriter (and friends in teh same package).

Remote chunking is not the only way to go about remote step execution. The PartitionHandler SPI is also available, as per the User Guide There is even an implementation in Spring Batch Integration (MessageChannelPartitionHandler).

I am going to write an article on Spring Batch Integration when I get time to try and raise awareness and get more people to use it. In the meantime, look at the website and the source code (including unit tests) to learn about the project.

Dave,

We have the same requirements here and we are currently evaluating Spring Batch. I believe that you would get more traction for the Spring batch integration module if it was available as a Maven artifact just the same way as other jars. I am fine with building the source (I did) but integrating this in our PoC requires me to deploy the lib in our corporate repo, checkout the tag corresponding to the version we use (2.0.4 / 2.1M3), etc.

My 2c.

S.

[Dave Syer]

Spring Batch Integration is available as a Maven artifact in 2.1.0 milestones (I only published it in Maven Central style because that's what most people use). It might get folded into Spring Batch Admin before a release of 2.1, but it will be available as a GA release at some point soon.