Rollback of chunk on exception

[MisterX]

We have the following scenario:

The ItemProcessor processes a dataset and has to update two database tables or other resources. During this processing it might happen that a technical exception - e.g. DataIntegrityViolationException - occurs when the second table is modified. This might be perfectly acceptable for the business logic and could be handled in some way in the ItemProcessor before the exception is rethrown to cause a rollback of the current transaction. After the rollback of the chunk the batch job should continue with the next item.

Unfortunately, this is not the bahaviour we can observe when running a batch job. If an exception is thrown it causes the current transaction to roll back and the job terminates. If an ExceptionHandler is used to handle the exceptions the current transaction is not rolled back which can cause inconsistent data.

What is the right way to handle such a use case?

[Dave Syer]

ExceptionHandler can be used to decide on whether to rethrow or not, but it does not determine (currently at least) whether the batch is complete. You need a CompletionPolicy in the chunkOperations that can ignore the exception. We discussed this internally a couple of weeks ago and decided to consider moving this responsibility into the ExceptionHandler because it is a common mistake to assume that it can do this. I just got round to putting it in JIRA, so you can track it here (http://opensource.atlassian.com/proj...owse/BATCH-201).

[MisterX]

Dave, thanks for your quick reply and for setting up a JIRA issue. Can you estimate when a solution to this problem is available?

[Dave Syer]

There is already a solution - use a CompletionPolicy to decide whether to ignore the exception. The JIRA issue is just saying that this will change if it is addressed (the decision will be an ExceptionHandler concern).

[Dave Syer]

Plus the JIRA issue was marked as resolved so the new ExceptionHandler contract will be in tonight's build if you want to try it.

[MisterX]

We do not want to ignore the exception completely. The exception might be handled in an ExceptionHandler or a CompletionPolicy to log it or to decide whether to rethrow it.

If it is rethrown we would expect the following behaviour:

1. the current transaction should be rolled back
2. the next item is fetched from the ItemProvider
3. if there is a new item a new chunk - and therefore a new transaction - is created
4. the current item is processed by the ItemProvider

This scheme should be continued until all items are processed by the batch.

Currently, rethrowing the exception causes the transaction to roll back as expected. After that the batch job terminates and any remaining items are ignored. It would only be possible to process them if the cause of the exception is corrected (manually) and the batch is started again. However, this is a bit of a problem if the batch jobs are controlled and monitored automatically.

Can you show the configuration of such a scenario in one of the Spring Batch samples, preferably HibernateFailureJobFunctionalTests? Alternatively, I can send you a modified version of the tradeJob batch example or might attach it to the closed JIRA issue.

[Dave Syer]

When you say "currently" you mean "currently with my configuration" not "currently with Spring Batch". Here is a snippet that should work something like you want it to with the current snapshot:

	<bean id="hibernateJob" parent="simpleJob">
		<property name="steps">
			<bean id="step1" class="org.springframework.batch.execution.step.ChunkOperationsStepConfiguration">
				<property name="tasklet">
					<bean
						class="org.springframework.batch.execution.tasklet.RestartableItemProviderTasklet">
						<property name="itemProvider">
							<bean
								class="org.springframework.batch.item.provider.InputSourceItemProvider">
								<property name="inputSource"
									ref="hibernateInputSource" />
							</bean>
						</property>
						<property name="itemProcessor">
							<bean
								class="org.springframework.batch.sample.item.processor.CustomerCreditIncreaseProcessor">
								<property name="outputSource" ref="hibernateOutputSource"/>
							</bean>
						</property>
					</bean>
				</property>
				<property name="chunkOperations">
					<bean class="org.springframework.batch.repeat.support.RepeatTemplate">
						<property name="interceptor" ref="hibernateOutputSource"/>
						<property name="completionPolicy">
							<bean class="org.springframework.batch.repeat.policy.SimpleCompletionPolicy">
								<property name="chunkSize" value="3"/>
							</bean>
						</property>
						<property name="exceptionHandler">
							<bean class="org.springframework.batch.repeat.exception.handler.SimpleLimitExceptionHandler"
								p:limit="5" p:useParent="true" p:type="java.lang.Exception"/>
						</property>
					</bean>
				</property>
			</bean>
		</property>
	</bean>

It will rethrow on the 6th Exception of any type in any chunk. For more control you can use RethrowOnThresholdExceptionHandler.

[MisterX]

By saying "currently" I mean both with my configuration as well as with the current Spring Batch Snapshot. I can see that the ExceptionHandler works as advertised in certain circumstances. Perhaps I should describe in more detail in which circumstances it does not work - at least I can not get it working.

Say you want to update several database tables in the processor. One of these modifications fail but not the first one. Some kind of exception occurs, let's say a DataIntegrityViolationException. We would expect that the modifications made to the other tables are discarded in order to prevent inconsistent data in the database. In the Spring Batch scenario one transaction spans over the whole chunk. So we would expect that all the modifications in this chunk are rolled back to ensure data integrity. Not very nice but we might deal with it for the moment. After the rollback of the last transaction the batch job should continue with the processing of the next item. However, the currently the batch job terminates.

With the ExceptionHandler or CompletionPolicy we would be able to prevent the termination of the batch processing. Unfortunately, there seems to be no way to rollback the last chunk to ensure consistent data, right?

[Dave Syer]

I get it. You want a rollback without the exception propagating to the step level. Then you need an ExceptionHandler in the stepOperations (not the chunkOperations).

If you are using one of the file or SQL InputSources from Spring Batch the current item can be skipped. In the Hibernate case we can't skip the bad record (http://opensource.atlassian.com/proj...owse/BATCH-194) because the Exception comes too late. The same applies to batch JDBC (http://opensource.atlassian.com/proj...rowse/BATCH-76), which we don't provide at the moment anyway.

So use flat files or SQL input sources (for now) to get precisely the behaviour you need.

[MisterX]

Dave, thanks for your help. Configuring an ExceptionHandler for the StepOperations does the trick. But I don't understand what you mean when you say that the items should be skipped.

Additionally, is there a way to repeat the processing of the successfully processed items which have not been committed because of the rollback of the transaction?