DeadlockLoserDataAccessException

[Dave Syer]

It might be better to upgrade to a full release instead of a milestone. Not that this would explain the issue necessarily, but there isn't much to be gained from using an unsupported distribution. There are two issues here (two stack traces):

To work around the problem in the AbstractStep with retry, you would need to use the retry configuration at the level of the chunk in the declaration of the step (e.g. in XML). Then you have to exclude the updateStepExecution() method from the pointcut that you use to apply retry to the JobReposository.

The launch seems to have failed with a DeadlockLoserException, but I can't see from the logs if was actually retried or not. Assuming it was, then it looks like you were unlucky and it failed after exhausting the retries. But you should be able to verify from the logs if it is retrying (especially with DEBUG logging for org.springframework.batch.retry). You can also tune the retry, e.g. more attempts, exponential backoff.

[Nitty]

Here's the log with debug on for retry..

Is there a easy way to test the retry on repository rather than wait for the event to happen ?

03:52:43,721 DEBUG SimpleAsyncTaskExecutor-119 StepContextRepeatCallback:67 - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@26e26fc2
03:52:43,721 DEBUG SimpleAsyncTaskExecutor-119 StepContextRepeatCallback:75 - Chunk execution starting: queue size=0
03:52:43,722 DEBUG SimpleAsyncTaskExecutor-119 HeaderMapper:61 - parsing busDate in the header :20100628
03:52:43,722 DEBUG SimpleAsyncTaskExecutor-119 StepScope:148 - Creating object in scope=step, name=scopedTarget.metaDataProcessor
03:52:43,722 DEBUG SimpleAsyncTaskExecutor-119 StepScope:148 - Creating object in scope=step, name=scopedTarget.prodTransListStgProcessor
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-119 ChunkOrientedTasklet:87 - Inputs not busy, ended: true
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-118 SimpleFlow:156 - Completed state=faJob.prodSplit.0.pollOtcStg with status=COMPLETED
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-119 TaskletStep:381 - Applying contribution: [StepContribution: read=6, written=0, filtered=6, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-118 SimpleFlow:143 - Handling state=faJob.prodSplit.0.distOtcProd
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-119 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-119 RetryTemplate:234 - Retry: count=0
03:52:43,723 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:234 - Retry: count=0
03:52:43,724 DEBUG SimpleAsyncTaskExecutor-119 TaskletStep:393 - Saving step execution before commit: StepExecution: id=1000000012921, name=loadTransListProd, status=STARTED, exitStatus=EXECUTING, readCount=6, filterCount=6, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
03:52:43,725 DEBUG SimpleAsyncTaskExecutor-119 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:43,725 DEBUG SimpleAsyncTaskExecutor-119 RetryTemplate:234 - Retry: count=0
03:52:44,334 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:44,336 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:234 - Retry: count=0
03:52:44,342 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:44,345 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:234 - Retry: count=0
03:52:44,350  INFO SimpleAsyncTaskExecutor-119 XmlBeanDefinitionReader:315 - Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
03:52:44,352  INFO SimpleAsyncTaskExecutor-118 SimpleStepHandler:113 - Executing step: [TaskletStep: [name=distOtcProd]]
03:52:44,354 DEBUG SimpleAsyncTaskExecutor-118 AbstractStep:180 - Executing: id=1000000012925
03:52:44,357 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:44,358 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:234 - Retry: count=0
03:52:44,364 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:205 - RetryContext retrieved: [RetryContext: count=0, lastException=null, exhausted=false]
03:52:44,365 DEBUG SimpleAsyncTaskExecutor-118 RetryTemplate:234 - Retry: count=0
03:52:44,377  INFO SimpleAsyncTaskExecutor-119 SQLErrorCodesFactory:125 - SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
03:52:44,377 DEBUG SimpleAsyncTaskExecutor-118 StepContextRepeatCallback:67 - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@70b96528
03:52:44,382 DEBUG SimpleAsyncTaskExecutor-118 StepContextRepeatCallback:75 - Chunk execution starting: queue size=0
03:52:44,380 DEBUG SimpleAsyncTaskExecutor-119 RetryTemplate:261 - Checking for rethrow: count=1
03:52:44,385 DEBUG SimpleAsyncTaskExecutor-118 StepScope:148 - Creating object in scope=step, name=scopedTarget.prodDistOtc
03:52:44,389 DEBUG SimpleAsyncTaskExecutor-119 RetryTemplate:234 - Retry: count=1
03:52:44,393 DEBUG SimpleAsyncTaskExecutor-118 ProdDistribution:43 - clearing otc prod step cache for batch =1733
03:52:44,393 DEBUG SimpleAsyncTaskExecutor-118 ProdDistribution:51 - Call product distrubutuin of batchId=1733,source=FA,prodType=O
03:52:44,418 ERROR SimpleAsyncTaskExecutor-119 AbstractStep:213 - Encountered an error executing the step
org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:717)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:147)
        at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:261)
        at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
        at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
        at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
        at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
        at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:247)
        at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:196)
        at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:115)
        at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
        at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
        at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
        at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
        at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:83)
        at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:81)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.lang.Thread.run(Thread.java:619)

[Dave Syer]

Did you move the retry out of the updateStepExecution? Your thread 119 seems to have encountered a problem but it continues immediately to retry something (probably the repository update) and then fails. This wouldn't be the case if the retry was controlled by the step. So make sure you remove your retry config from that method and move it up to the step/chunk. Can you post your new config?

It is quite difficult to simulate a repository failure. You could perhaps add an exception throwing APO advice, but it wouldn't really be the same thing.

[Nitty]

Sorry.. not sure I completly follow this.

Since the problem is around the control (repository) tables, I was thinking the retry should ideally be around the repository rather than the step/chunk level.

If it's moved to the step the entire step would be retried right?
In that case there may be chunks that would already be commited and retring the entire step would try to insert the chunk again which will cause integritiy constraints in our stage tables.

I think in this case it should be done on the chunk level and any error while performing control operation should rollback the entire chunk but we have some steps which are tasklets calling store procedures for these we may have to configure the retry step level.

I'm not sure how to configure chuck & step for specific scenarios mentioned above with pointcuts.

please advice..

[Nitty]

Having it step retry may the workaround.

The failure would retry the entire step but it should start from where it left off from the last successful chunck commit..

Removed the retry config around the repository.

Adding it to the methods of Step..

<aop:config>
    	<aop:pointcut id="stepPointCut" expression="execution(* org.springframework.batch.core.Step.*(..))"/>
    	<aop:advisor pointcut-ref="stepPointCut" advice-ref="retryAdvice" order="-1"/>
</aop:config>

<b:bean id="retryAdvice" class="org.springframework.batch.retry.interceptor.RetryOperationsInterceptor"/>

[Nitty]

Looks like the retry operation interceptor is not working.

As a test I explicitly threw an exception in the processor, based on the logs I don't see any retries

Please find the log attached.

[Dave Syer]

I've lost track now of what your configuration looks like. The exception in the processor will only be retried if you have set up retry config for the step.

[Nitty]

Sorry about that.

Having the interceptor over Step methods doesn't work - below is the configuration for that and the logs are attached in previous post

<aop:config>
    	<aop:pointcut id="stepPointCut" expression="execution(* org.springframework.batch.core.Step.*(..))"/>
    	<aop:advisor pointcut-ref="stepPointCut" advice-ref="retryAdvice" order="-1"/>
</aop:config>

<b:bean id="retryAdvice" class="org.springframework.batch.retry.interceptor.RetryOperationsInterceptor/>

The step retry config from refrerence guide - 5.1.6. works.

We have several steps and its a tedious process to add the retry config to every step , is there easier way to one config that applies across all the step ( I was hoping the interceptor around Step should do it )

[Dave Syer]

Having the interceptor over Step methods doesn't work

I never said it would.

The step retry config from refrerence guide - 5.1.6. works.

That is the correct way to add retry to a step.

We have several steps and its a tedious process to add the retry config to every step , is there easier way to one config that applies across all the step ( I was hoping the interceptor around Step should do it )

You can use a parent step as a template for all the steps that use the same configuration.

[Nitty]

This would work for the step which have chunks.

We have steps which are tasklets either invoking a storeproc and doing some other house keeping work.

How can we retry for these for a deadlock situation on the repository?