Batch High Availability and Failover

[klee1703]

I would like to know what mechanisms the Spring Batch framework provides to support high availability and failover for batch jobs. For example, most J2EE servers provide high availability and failover via clusters, load balancers, etc. The Quartz framework can be configured to provide H/A and failover for scheduled jobs via a database. So how can Spring Batch jobs be configured to provide failover throughout the job lifecycle (scheduling, execution, etc.)?

[Dave Syer]

"High Availability" and "Failover" are concepts more relevant to the data that a job might be using, rather than the job itself. There are of course many mechanisms to ensure those features in a clustered environment (e.g. JEE application servers, data grids, hardware configurations).

In a batch job the relevant concepts are re-startability (can a failed job be restarted and pick up where it left off?) and idempotence (can a job be re-run and not need to pick up where it left off?). We provide various mechanisms to help with restartability. And idempotence can be ensured by using common patterns in the data sources (process indicator pattern - see user reference guide for some more description, or the TSE2007 batch presentation slides). Idempotence is called "re-runnability" in one of the TSE2007 slides (it's a term used internally by some folks at Accenture).

If the lights go out on a job, and it is restartable or idempotent, all you need to do it run it again. So your reliability comes from the scheduling tool (we don't do scheduling or triggering in Spring Batch).