Rollback of chunk on exception

[MisterX]

We have the following scenario:

The ItemProcessor processes a dataset and has to update two database tables or other resources. During this processing it might happen that a technical exception - e.g. DataIntegrityViolationException - occurs when the second table is modified. This might be perfectly acceptable for the business logic and could be handled in some way in the ItemProcessor before the exception is rethrown to cause a rollback of the current transaction. After the rollback of the chunk the batch job should continue with the next item.

Unfortunately, this is not the bahaviour we can observe when running a batch job. If an exception is thrown it causes the current transaction to roll back and the job terminates. If an ExceptionHandler is used to handle the exceptions the current transaction is not rolled back which can cause inconsistent data.

What is the right way to handle such a use case?

[Dave Syer]

ExceptionHandler can be used to decide on whether to rethrow or not, but it does not determine (currently at least) whether the batch is complete. You need a CompletionPolicy in the chunkOperations that can ignore the exception. We discussed this internally a couple of weeks ago and decided to consider moving this responsibility into the ExceptionHandler because it is a common mistake to assume that it can do this. I just got round to putting it in JIRA, so you can track it here (http://opensource.atlassian.com/projects/spring/browse/BATCH-201).

[MisterX]

Dave, thanks for your quick reply and for setting up a JIRA issue. Can you estimate when a solution to this problem is available?

[Dave Syer]

There is already a solution - use a CompletionPolicy to decide whether to ignore the exception. The JIRA issue is just saying that this will change if it is addressed (the decision will be an ExceptionHandler concern).

[Dave Syer]

Plus the JIRA issue was marked as resolved so the new ExceptionHandler contract will be in tonight's build if you want to try it.

[MisterX]

We do not want to ignore the exception completely. The exception might be handled in an ExceptionHandler or a CompletionPolicy to log it or to decide whether to rethrow it.

If it is rethrown we would expect the following behaviour:

the current transaction should be rolled back
the next item is fetched from the ItemProvider
if there is a new item a new chunk - and therefore a new transaction - is created
the current item is processed by the ItemProvider

This scheme should be continued until all items are processed by the batch.

Currently, rethrowing the exception causes the transaction to roll back as expected. After that the batch job terminates and any remaining items are ignored. It would only be possible to process them if the cause of the exception is corrected (manually) and the batch is started again. However, this is a bit of a problem if the batch jobs are controlled and monitored automatically.

Can you show the configuration of such a scenario in one of the Spring Batch samples, preferably HibernateFailureJobFunctionalTests? Alternatively, I can send you a modified version of the tradeJob batch example or might attach it to the closed JIRA issue.

[Dave Syer]

When you say "currently" you mean "currently with my configuration" not "currently with Spring Batch". Here is a snippet that should work something like you want it to with the current snapshot:



<bean id="hibernateJob" parent="simpleJob">
<property name="steps">
<bean id="step1" class="org.springframework.batch.execution.step.ChunkOper ationsStepConfiguration">
<property name="tasklet">
<bean
class="org.springframework.batch.execution.tasklet.Restar tableItemProviderTasklet">
<property name="itemProvider">
<bean
class="org.springframework.batch.item.provider.InputSourc eItemProvider">
<property name="inputSource"
ref="hibernateInputSource" />
</bean>
</property>
<property name="itemProcessor">
<bean
class="org.springframework.batch.sample.item.processor.Cu stomerCreditIncreaseProcessor">
<property name="outputSource" ref="hibernateOutputSource"/>
</bean>
</property>
</bean>
</property>
<property name="chunkOperations">
<bean class="org.springframework.batch.repeat.support.RepeatTem plate">
<property name="interceptor" ref="hibernateOutputSource"/>
<property name="completionPolicy">
<bean class="org.springframework.batch.repeat.policy.SimpleComp letionPolicy">
<property name="chunkSize" value="3"/>
</bean>
</property>
<property name="exceptionHandler">
<bean class="org.springframework.batch.repeat.exception.handler .SimpleLimitExceptionHandler"
p:limit="5" p:useParent="true" p:type="java.lang.Exception"/>
</property>
</bean>
</property>
</bean>
</property>
</bean>


It will rethrow on the 6th Exception of any type in any chunk. For more control you can use RethrowOnThresholdExceptionHandler.

[MisterX]

By saying "currently" I mean both with my configuration as well as with the current Spring Batch Snapshot. I can see that the ExceptionHandler works as advertised in certain circumstances. Perhaps I should describe in more detail in which circumstances it does not work - at least I can not get it working.

Say you want to update several database tables in the processor. One of these modifications fail but not the first one. Some kind of exception occurs, let's say a DataIntegrityViolationException. We would expect that the modifications made to the other tables are discarded in order to prevent inconsistent data in the database. In the Spring Batch scenario one transaction spans over the whole chunk. So we would expect that all the modifications in this chunk are rolled back to ensure data integrity. Not very nice but we might deal with it for the moment. After the rollback of the last transaction the batch job should continue with the processing of the next item. However, the currently the batch job terminates.

With the ExceptionHandler or CompletionPolicy we would be able to prevent the termination of the batch processing. Unfortunately, there seems to be no way to rollback the last chunk to ensure consistent data, right?

[Dave Syer]

I get it. You want a rollback without the exception propagating to the step level. Then you need an ExceptionHandler in the stepOperations (not the chunkOperations).

If you are using one of the file or SQL InputSources from Spring Batch the current item can be skipped. In the Hibernate case we can't skip the bad record (http://opensource.atlassian.com/projects/spring/browse/BATCH-194) because the Exception comes too late. The same applies to batch JDBC (http://opensource.atlassian.com/projects/spring/browse/BATCH-76), which we don't provide at the moment anyway.

So use flat files or SQL input sources (for now) to get precisely the behaviour you need.

[MisterX]

Dave, thanks for your help. Configuring an ExceptionHandler for the StepOperations does the trick. But I don't understand what you mean when you say that the items should be skipped.

Additionally, is there a way to repeat the processing of the successfully processed items which have not been committed because of the rollback of the transaction?

[Dave Syer]

Configuring an ExceptionHandler for the StepOperations does the trick.


OK. I made it a bit easier (look in SVN or in tonight's snapshot) by adding the step operations to the step configuration (so you don't have to change the StepExecutor). So ChunkOperationsStepConfiguration is now called RepeatOperationsStepConfiguration and it allows you to set the step operations as well as the chunk operations.

Do you think that the ExceptionHandler on SimpleStepConfiguration should go at the step level (not the chunk level as it does now)? It seems like this is the more likely scenario for common exception handler strategies.



But I don't understand what you mean when you say that the items should be skipped.


If the exception is thrown in the ItemProcessor (or anywhere in the Tasklet if the Tasklet is Skippable and passes the message down to the ItemProvider - this is the case for the ItemProviderProcessorTasklet) the item will be marked in the InputSource as skipped and will not be presented again to the processor.



Additionally, is there a way to repeat the processing of the successfully processed items which have not been committed because of the rollback of the transaction?

That should be automatic if your ExceptionHandler in the step operations does not rethrow - the transaction will be repeated and all the unskipped items (the ones that did not fail) will be processed.

N.B. this is still not true for some failure modes using Hibernate-based ItemProcessors, until we fix or find a workaround for BATCH-194 - the item cannot be skipped because the exception came outside the Tasklet execution.

[MisterX]

Do you think that the ExceptionHandler on SimpleStepConfiguration should go at the step level (not the chunk level as it does now)? It seems like this is the more likely scenario for common exception handler strategies.

Yes, I guess this was the reason for all the confusion on this topic.


That should be automatic if your ExceptionHandler in the step operations does not rethrow - the transaction will be repeated and all the unskipped items (the ones that did not fail) will be processed.

Is there some kind of configuration to enable this behaviour? I'm using a RestartableItemProviderTasklet with a ValidatingItemProvider and I can see that the chunk is not processed again. Instead the chunk is rolled back and the processing of the next chunk begins.

[Dave Syer]

I forgot to add (maybe this is what you are finally missing) - Tasklet exceptions only cause a skip if you are using DefaultStepExecutor, and for some reason the samples are set up to use SimpleStepExecutor. I think the Default* is so little different from Simple* that I might just move that feature into Simple* and remove Default* from the API completely.

[MisterX]

I could not get it running by replacing the SimpleStepExecutor with the DefaultStepExecutor. Can you provide an example in spring-batch-samples, maybe the tradeJob.xml?

[Dave Syer]

How do you mean you "could not get it running"? If you look in SVN all the samples are using DefaultStepExecutor (as of an hour ago), but all I did was change the class name in simple-container-definition.xml. I have been using the hibernateJob.xml as a test because I am working on BATCH-194, so if you ignore the hibernate specific stuff that should work. You must be trying to do something more complicated?

[elukaweski]

Hi:

I was looking for the abstract class mentioned in the note - "added AbstractDrivingQueryInputSource superclass that encapsulates duplications from IbatisDrivingQueryInputSource and SingleKeyDrivingQueryInputSource."

I am curious as to how the framework handles iBatis when is doesn't have the capability of iterating over a result set.

Thanks

[MisterX]

Dave, I took a look at the Hibernate example and it seems to work as expected. However, when I'm using an example based on JDBC it does not show the expected behaviour. I've created a JIRA issue (Batch-244 (http://opensource.atlassian.com/projects/spring/browse/BATCH-244)) and attached an example job to it. It would be nice if you could try to reproduce the problem and give me any hints what I'm doing wrong.

Thanks in advance.

[wxlund]

I was curious about how you're configuring Hibernate in the batch configuration. I've done a lot of Hibernate batch in the past and one of the issues we dealt with was flushing. Hibernate 3.x has added a number of new features that were not available to us but essentially we set Flush to FLUSH.NEVER and then performed the flush at each unit of work (Item).

Our mode of determining where we were at in the chunk processing was using a DrivingQueryInputSource (I'm having to translate because it was pre-Spring-batch) but essentially the DrivingQueryInputSource, because the keys are in memory, allowed us to know where we were at when the Chunk needed to re-process a Chunk. This was necessary because Hibernate requires us to throw out the Session when a transaction is rolled back. At this point you can apply a strategy to change the commit interval to commit from the start of the chunk up until the bad record and then re-adjust at that point.

Regardless, I'd love to know how your configuring the hibernate parts of batch and which, if any, Hibernate 3.x features you're using (e.g. stateless session, etc).

Wayne

[MisterX]

Wayne,

sorry for not answering sooner. Our Hibernate configuration is not any different from the one used in "normal" web applications. We do not use any of the Hibernate configuration used for batch processing. When we had our Hibernate related problems with Spring Batch we had a short look into the StatelessSession but we decided not to go with that because one of our goals is to reuse existing components from web applications in our Spring Batch based jobs.

Our current solution is to flush the Hibernate session automatically after processing each item. This is done by configuring an aspect that is invoked after the process() method of each ItemProcessor. This way we get any Hibernate related errors at least when the processing of an item (during which Hibernate is used) is finished. The Spring Batch framework is able to handle the error and can rerun the failing chunk just as it is expected.

Please have a look at BATCH-194 (http://opensource.atlassian.com/projects/spring/browse/BATCH-194) to see the implementation of the advice and its configuration.

Jens