Spring Integration + Batch + Quartz = Good Solution?

[RonanL]

Hello.

I am at the beginning of a project, which will be implemented in Java, and I am trying to make some decisions about what open source frameworks to make use of.

In summary, the application will process flat files that are FTP'd to the machine for processing, with output files being returned to the client (via (S)FTP also).

The finished application should:
- run continuously
- detect input files
- maintain archive of previous input files
- cater for multiple input/output file formats
- log errors if input files are not recieved when expected
- use seperate xml configuration files for each client
- allow for new clients to be added easily
- cater for input files with embedded "fulfillment times", which will result in multiple jobs (batch?) being scheduled (quartz?)

I have used spring in the past but only for dependancy injection, so spring integration, and spring batch are new to me. I would like to get some opinions on how best to proceed. I'd also be eager to get my hands on some working samples that might help me along the way. I'd especially like to get some tips on how to cater for the multiple fulfillment times mentioned in the last bullet.

Thanks in advance,

Ronan

[chudak]

Yep, I've used these three projects to accomplish something very similar.

[RonanL]

Yep, I've used these three projects to accomplish something very similar.

Thanks for the reply Chudak. It's good to hear that it's possible at least!

As I mentioned, I'm familiar with spring, but only really for dependency injection and I have used quartz. I'm completely new to spring integration and batch, so I have been looking at the samples for both, but I haven't found any that combine the two. Ideally, I'd love if someone could point me towards a sample that combines spring integration, batch, and quartz. Anyone?!

[drumkilling]

... I'd love if someone could point me towards a sample that combines spring integration, batch, and quartz. Anyone?!

Yes me too!

I've two scenarios

spring integration (scheduled): Message Endpoint -> channel -> handler -> staging database
(spring) batch scheduled: staging database -> validation logik -> production database

I'm not sure if spring batch makes sense running inside tomcat reading data from staging tables doing some validation and writing data to production database.
In a clustered environment running two tomcat instances.
There's a constraint that only one instance is allowed to write to production database.
For this reason I thought about using Quartz Clustered Functionallity to establish this say "Singleton" functionality and avoid Scheduler Single Point Of Failure because Quartz is running on both nodes.

Does that make sense or are there any other solutions possible.

In the moment I havn't the right glue between spring integration, quartz and spring batch.

Thanks in advance too!

[chudak]

Samples?

Your guess is as good as mine. I don't know of any samples that put together an end to end example of spring integration->spring batch. These things tend to be kind of meaningless anyways since they are always toy examples that do nothing like what you want to do.

I don't mess around with creating samples...I have software to build for my client; those are the only samples I have and I'm afraid I can't post wholesale examples of IP I've created for my clients. I'd have to waste alot of my time 'anonymizing' the application configuration and coming up with dummy classes to show you how I have it all wired up.

And, don't take this the wrong way, but this is exactly what your employer/client is paying you for: using your creativity and experience to leverage off the shelf FOSS frameworks to solve their business problems. If I show you how to do it end to end maybe they should be hiring me instead of paying you?

There are plenty of examples that show how to kick off a spring batch job from a dynamic input file and in general how to work with Spring Batch.

There are also plenty of examples that show how to configure a spring integration inbound file adapter and move it. If you need to do any file transformation in the middle, there are examples of that two.

The only thing missing is putting the two together; all you have to do is create a message endpoint that is wired up to your spring integration channel that launches spring batch using the incoming file as the input for the batch.

It's all pretty simple...really. And if you get it working yourself, you'll have a much better appreciation for what you did and why it works.

Cheers!

[RonanL]

Thanks for the reply Chudak. In my experience, sentences that begin with "don't take this the wrong way" can only be taken one way. But I appreciate what you are saying. I'm sure I will benefit more by figuring it out myself. On the other hand, it's often a good idea not to waste time re-inventing the wheel.

Anyway, I will continue to look at the samples and work towards putting them together to achieve what I want. Perhaps if I get to a point where I am happy with the solution, I will spend some time doing the "anonymizing" and post the code here for other people to expand on.

In the meantime, if anyone else is willing to contribute on the topic, please feel free.

Thanks,

Ronan

[drumkilling]

@chudak: thank you for your fast and straight response!

I think I have to dive deeper in the spring documentation and I'm sure I will find solutions on my own.

Best regards

[tedjohn09]

Yep, I've used these three projects to accomplish something very similar.

Yeah, me too.

__________________
click here