Default retry policy for Chunk Tasklet

[leogags]

Hi,

I have a chunk tasklet in a step with skippable/non-skippable exceptions and skip limits defined. I have not defined any retry policy or retriable-exception classes.

What I have noticed is that even though I have not defined any retry policy, the chunk retries (twice) when a non-skippable exception occurs in my processor.

Is there a default retry for a chunk tasklet?

Here are extracts from my configs :

		<step id="prepareFileForProcessing" parent="parentStep">
			<tasklet>
				<chunk reader="rawReader" processor="validatingProcessor" writer="processingWriter" commit-interval="1" skip-limit="20">
					<streams>
						<stream ref="fileItemRawReader"/>
						<stream ref="processingWriter"/>
					</streams>
				</chunk>
			</tasklet>
			<next on="FAILED" to="failFlow"/>
			<next on="*" to="processFile"/>
		</step>

	<step id="parentStep" abstract="true">
    	<tasklet allow-start-if-complete="true">
        	<chunk commit-interval="1">
				<skippable-exception-classes>
	           		<include class="java.lang.Exception"/>
	           		<exclude class="java.io.IOException"/>
	           		<exclude class="javax.xml.ws.WebServiceException"/>
	           		<exclude class="org.springframework.batch.item.validator.ValidationException"/>
	       		</skippable-exception-classes>
        	</chunk>
    	</tasklet>
	</step>

Extracts from log indicating retries:

2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Starting repeat context.
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat operation about to start at count=1
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@2e5a2e5a
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Chunk execution starting: queue size=0
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.support.transaction.ResourcelessTransactionManager - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Starting repeat context.
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat operation about to start at count=1

2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat is complete according to policy and result value.
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Retry: count=0

2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Checking for rethrow: count=1
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Rethrow in retry for policy: count=1
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Applying contribution: [StepContribution: read=1, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Rollback for RuntimeException: org.springframework.batch.item.validator.ValidationException: 
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.transaction.support.TransactionTemplate - Initiating transaction rollback on application exception

2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Handling exception: org.springframework.batch.item.validator.ValidationException, caused by: org.springframework.batch.item.validator.ValidationException: Total Grantor lines (2) not equal to total Grantors reported in tralier record (3);Total Create lines (1) not equal to total Creates reported in tralier record (2)
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.item.SimpleRetryExceptionHandler - Handled non-fatal exception

2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat operation about to start at count=2
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@2e5a2e5a
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Chunk execution starting: queue size=0
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.support.transaction.ResourcelessTransactionManager - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Retry failed last attempt: count=1
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Rollback for RuntimeException: org.springframework.batch.retry.RetryException: Non-skippable exception in recoverer while processing; nested exception is org.springframework.batch.item.validator.ValidationException: 
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.transaction.support.TransactionTemplate - Initiating transaction rollback on application exception

Thanks in advance.

[Dave Syer]

Skip is implemented as a retry so I think all you are seeing is a second transaction being started to process the failed items (under control of a retry policy). The default assumption is that an exception invalidates the current chunk transaction, so it has to be rolled back before the skip listener is applied.

[ankkha]

Is it possible that if some one has duplicate records corresponding to a key (Primary Key in table) , Batch will just update the last row with new records , No need to skip the record.

[priyavenkat]

Hi,

I have a chunk tasklet in a step with skippable/non-skippable exceptions and skip limits defined. I have not defined any retry policy or retriable-exception classes.

What I have noticed is that even though I have not defined any retry policy, the chunk retries (twice) when a non-skippable exception occurs in my processor.

Is there a default retry for a chunk tasklet?

Here are extracts from my configs :

		<step id="prepareFileForProcessing" parent="parentStep">
			<tasklet>
				<chunk reader="rawReader" processor="validatingProcessor" writer="processingWriter" commit-interval="1" skip-limit="20">
					<streams>
						<stream ref="fileItemRawReader"/>
						<stream ref="processingWriter"/>
					</streams>
				</chunk>
			</tasklet>
			<next on="FAILED" to="failFlow"/>
			<next on="*" to="processFile"/>
		</step>

	<step id="parentStep" abstract="true">
    	<tasklet allow-start-if-complete="true">
        	<chunk commit-interval="1">
				<skippable-exception-classes>
	           		<include class="java.lang.Exception"/>
	           		<exclude class="java.io.IOException"/>
	           		<exclude class="javax.xml.ws.WebServiceException"/>
	           		<exclude class="org.springframework.batch.item.validator.ValidationException"/>
	       		</skippable-exception-classes>
        	</chunk>
    	</tasklet>
	</step>

Extracts from log indicating retries:

2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Starting repeat context.
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat operation about to start at count=1
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@2e5a2e5a
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Chunk execution starting: queue size=0
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.support.transaction.ResourcelessTransactionManager - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Starting repeat context.
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat operation about to start at count=1

2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat is complete according to policy and result value.
2011-07-19 10:09:50,081 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Retry: count=0

2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Checking for rethrow: count=1
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Rethrow in retry for policy: count=1
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Applying contribution: [StepContribution: read=1, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Rollback for RuntimeException: org.springframework.batch.item.validator.ValidationException: 
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.transaction.support.TransactionTemplate - Initiating transaction rollback on application exception

2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Handling exception: org.springframework.batch.item.validator.ValidationException, caused by: org.springframework.batch.item.validator.ValidationException: Total Grantor lines (2) not equal to total Grantors reported in tralier record (3);Total Create lines (1) not equal to total Creates reported in tralier record (2)
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.item.SimpleRetryExceptionHandler - Handled non-fatal exception

2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.repeat.support.RepeatTemplate - Repeat operation about to start at count=2
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@2e5a2e5a
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.scope.context.StepContextRepeatCallback - Chunk execution starting: queue size=0
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.support.transaction.ResourcelessTransactionManager - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.retry.support.RetryTemplate - Retry failed last attempt: count=1
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.batch.core.step.tasklet.TaskletStep - Rollback for RuntimeException: org.springframework.batch.retry.RetryException: Non-skippable exception in recoverer while processing; nested exception is org.springframework.batch.item.validator.ValidationException: 
2011-07-19 10:09:51,925 DEBUG [task-scheduler-1][] org.springframework.transaction.support.TransactionTemplate - Initiating transaction rollback on application exception

Thanks in advance.

In case of non skippable exceptions, could the experts here please let me know as to how the spring batch retries it one more time - totally 2 times?

[priyavenkat]

Got it answered myself!:-)

[chrishein]

Could you post the answer?

My understanding is that every chunk is retried, at least once, if an exception for one or more items is raised. This is done discarding the failed records.
But if the commit-interval is set to 1, and the only item in the chunk raises an exception, is there a way to avoid the reprocessing of the chunk (single item)?

Thanks

[priyavenkat]

Could you post the answer?

My understanding is that every chunk is retried, at least once, if an exception for one or more items is raised. This is done discarding the failed records.
But if the commit-interval is set to 1, and the only item in the chunk raises an exception, is there a way to avoid the reprocessing of the chunk (single item)?

Thanks

Would you be able to change the status of the DB entry to some ERROR STATUS (using a separate transaction that is not the same as the one that is executing your batch of course) when there is a failure encountered. That case, I assume, the erred record wont get picked again if your reader picks based on status too - Isnt it?