Page 1 of 4 123 ... LastLast
Results 1 to 10 of 32

Thread: DeadlockLoserDataAccessException

  1. #1

    Default DeadlockLoserDataAccessException

    Below is my job configuration, job repository isolation level is default - using sybase db.

    Code:
    <job id="myJob">
    <split id="prodSplit" task-executor="taskExecutor" next="loadPosnDlyStg">
        <flow>
    		<step id="loadOtcProd" parent="prodOtcStgLoad" next="pollOtcStg"/>
    		<step id="pollOtcStg" parent="prodOtcStgPoller" next="distOtcProd"/>
    		<step id="distOtcProd" parent="prodOtcDist"/>
        </flow>
        <flow>
    		<step id="loadListProd" parent="prodListStgLoad" next="distListProd"/>
    		<step id="distListProd" parent="prodListDist"/>
        </flow>
    </split>
    <step id="loadPosnDlyStg" parent="posnDlyStgLoad" next="loadPosnDly"/>
    <step id="loadPosnDly" parent="posnDlyLoad"/>
    <listeners>
    	<listener ref="jobListener"/>
    </listeners>
    </job>
    Quite frequently I am getting the below exception (looks like on at some point in time two threads are updating which is the cause of the problem)

    Code:
    org.springframework.dao.DeadlockLoserDataAccessException: PreparedStatementCallback; SQL [UPDATE BATCH_STEP_EXECUTION set START_TIME = ?, END_TIME = ?, STATUS = ?, COMMIT_COUNT = ?, READ_COUNT = ?, FILTER_COUNT = ?, WRITE_COUNT = ?, EXIT_CODE = ?, EXIT_MESSAGE = ?, VERSION = ?, READ_SKIP_COUNT = ?, PROCESS_SKIP_COUNT = ?, WRITE_SKIP_COUNT = ?, ROLLBACK_COUNT = ?, LAST_UPDATED = ? where STEP_EXECUTION_ID = ? and VERSION = ?]; Your server command (family id #0, process id #489) encountered a deadlock situation. Please re-run your command.
    ; nested exception is com.sybase.jdbc3.jdbc.SybSQLException: Your server command (family id #0, process id #489) encountered a deadlock situation. Please re-run your command.
    
    	at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:265)
    	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)
    	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:786)
    	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:842)
    	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:846)
    	at org.springframework.batch.core.repository.dao.JdbcStepExecutionDao.updateStepExecution(JdbcStepExecutionDao.java:172)
    	at org.springframework.batch.core.repository.support.SimpleJobRepository.update(SimpleJobRepository.java:167)
    	at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    	at java.lang.reflect.Method.invoke(Method.java:597)
    ..............

  2. #2
    Join Date
    Jun 2005
    Posts
    4,241

    Default

    I never saw that but I don't use sybase as a rule, and that platform is notorious for unnecessary deadlocks. I suppose you could wrap the JobRepository in a RetryOperationsInterceptor. It might be better to try and understand why it is deadlocked and try to tune the database (e.g. rebuild or add or remove indexes - ask a sybase guru).

  3. #3

    Default

    Dave,

    We did everything possible on the db side, it happens with row level lock too.
    At any point of time several jobs access the same repository.

    As we can configure retry's in application, Is there a way to handle this and configure retry intervals around job repository ?

    Thanks.

  4. #4
    Join Date
    Jun 2005
    Posts
    4,241

    Default

    Yes; "wrap the JobRepository in a RetryOperationsInterceptor".

  5. #5

    Default

    Thanks dave,

    I made the following change and now I get a UnexpectedRollbackException, may be the configuration is wrong - looks like its rolling back the transaction instead of a re-try..

    Code:
    <b:bean id="jobRepository"
    		class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean"
    		p:dataSource-ref="pmdbDataSource" p:transactionManager-ref="transactionManager" />
    	
    	<aop:config>
    		<aop:pointcut id="repositoryPointcut" expression="execution(* org.springframework.batch.core..*Repository+.*(..))"/>
        	<aop:advisor pointcut-ref="repositoryPointcut" advice-ref="txAdvice" order="2"/>
        	<aop:advisor pointcut-ref="repositoryPointcut" advice-ref="retryAdvice" order="1"/>
     	</aop:config>
    
    	<tx:advice id="txAdvice" transaction-manager="transactionManager">
        	<tx:attributes>
            	<tx:method name="*" />
        	</tx:attributes>
    	</tx:advice>
    	
    	<b:bean id="retryAdvice" class="org.springframework.batch.retry.interceptor.RetryOperationsInterceptor"/>
    the exception is

    Code:
    org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:717) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:147) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:261) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at
    Last edited by Nitty; Jun 17th, 2010 at 04:05 PM. Reason: missed out exception msg

  6. #6
    Join Date
    Jun 2005
    Posts
    4,241

    Default

    You don't need the txAdvice because a JobRepository has its own reference to the transaction manager. Not sure why the exception would not lead to a retry. Maybe you could try without txAdvice and post a full stacktrace if it doesn't work?

  7. #7

    Default

    removed txAdvice and here's the full stacktrace..(everytime it happens at different control table)

    Code:
    08:42:51,363 ERROR main BatchController:129 - stack trace of the error ..org.springframework.dao.DataAccessResourceFailureException: Could not increment identity; nested exception is com.sybase.jdbc3.jdbc.SybSQLException: Your server command (family id #0, process id #1064) encountered a deadlock situation. Please re-run your command.
    
            at org.springframework.jdbc.support.incrementer.SybaseMaxValueIncrementer.getNextKey(SybaseMaxValueIncrementer.java:108)
            at org.springframework.jdbc.support.incrementer.AbstractDataFieldMaxValueIncrementer.nextLongValue(AbstractDataFieldMaxValueIncrementer.java:125)
            at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:110)
            at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:127)
            at sun.reflect.GeneratedMethodAccessor630.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
            at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
            at $Proxy0.createJobExecution(Unknown Source)
            at sun.reflect.GeneratedMethodAccessor630.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
            at org.springframework.batch.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:68)
            at org.springframework.batch.retry.support.RetryTemplate.doExecute(RetryTemplate.java:238)
            at org.springframework.batch.retry.support.RetryTemplate.execute(RetryTemplate.java:147)
            at org.springframework.batch.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:55)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
            at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
            at $Proxy1.createJobExecution(Unknown Source)
            at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:111)
            at com.gs.fa.controller.LoadLauncher.launch(LoadLauncher.java:43)
            at com.gs.fa.controller.LoadLauncher.launchJob(LoadLauncher.java:57)
            at com.gs.fa.controller.BatchController.executeBatch(BatchController.java:86)
            at com.gs.fa.controller.BatchController.main(BatchController.java:202)
    Caused by: com.sybase.jdbc3.jdbc.SybSQLException: Your server command (family id #0, process id #1064) encountered a deadlock situation. Please re-run your command.
    
            at com.sybase.jdbc3.tds.Tds.processEed(Tds.java:2942)
            at com.sybase.jdbc3.tds.Tds.nextResult(Tds.java:2246)
            at com.sybase.jdbc3.jdbc.ResultGetter.nextResult(ResultGetter.java:69)
            at com.sybase.jdbc3.jdbc.SybStatement.nextResult(SybStatement.java:220)
            at com.sybase.jdbc3.jdbc.SybStatement.nextResult(SybStatement.java:203)
            at com.sybase.jdbc3.jdbc.SybStatement.updateLoop(SybStatement.java:1804)
            at com.sybase.jdbc3.jdbc.SybStatement.executeUpdate(SybStatement.java:1787)
            at com.sybase.jdbc3.jdbc.SybStatement.executeUpdate(SybStatement.java:434)
            at org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:228)
            at org.springframework.jdbc.support.incrementer.SybaseMaxValueIncrementer.getNextKey(SybaseMaxValueIncrementer.java:92)
            ... 33 more

  8. #8

    Default

    Here's the more detailed description of another instace of deadlock

    Code:
    12:34:30,170 DEBUG SimpleAsyncTaskExecutor-274 TaskletStep:393 - Saving step execution before commit: StepExecution: id=19076, name=loadPosnDly, status=STARTED, exitStatus=EXECUTING, readCount=1, filterCount=0, writeCount=1 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=2, rollbackCount=0, exitDescription=
    12:34:30,891  INFO SimpleAsyncTaskExecutor-274 XmlBeanDefinitionReader:315 - Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
    12:34:30,892  INFO SimpleAsyncTaskExecutor-275 SimpleStepHandler:113 - Executing step: [TaskletStep: [name=distTrans]]
    12:34:30,900 DEBUG SimpleAsyncTaskExecutor-275 AbstractStep:180 - Executing: id=19091
    12:34:30,924  INFO SimpleAsyncTaskExecutor-274 SQLErrorCodesFactory:125 - SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
    12:34:30,927 DEBUG SimpleAsyncTaskExecutor-275 StepContextRepeatCallback:67 - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@2efc8618
    12:34:30,929 DEBUG SimpleAsyncTaskExecutor-275 StepContextRepeatCallback:75 - Chunk execution starting: queue size=0
    12:34:30,933 DEBUG SimpleAsyncTaskExecutor-275 StepScope:148 - Creating object in scope=step, name=scopedTarget.transDist
    12:34:30,936 DEBUG SimpleAsyncTaskExecutor-275 TransDistribution:31 - Call trans distrubtion for batchId=3063
    12:34:30,946 ERROR SimpleAsyncTaskExecutor-274 AbstractStep:213 - Encountered an error executing the step
    org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
            at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:717)
            at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:147)
            at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:261)
            at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
            at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
            at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
            at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
            at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:247)
            at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:196)
            at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:115)
            at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
            at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
            at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
            at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
            at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:83)
            at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:81)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at java.lang.Thread.run(Thread.java:619)
    12:34:30,981 DEBUG SimpleAsyncTaskExecutor-275 TaskletStep:381 - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
    Here's the sybase db log..

    Code:
    01:00000:01354:2010/06/18 12:34:30.84 server Deadlock Id 125 detected 
    Deadlock Id 125: detected. 1 deadlock chain(s) involved. 
      
    Deadlock Id 125: Process (Familyid 0, Spid 1324, Suid 32) was executing a UPDATE command at line 1. 
    Deadlock Id 125: Process 1324 was involved in application ''. 
    Deadlock Id 125: Process 1324 was involved on host name 'xxxxx'. 
    Deadlock Id 125: Process 1324 was involved in transaction '$chained_transaction'. 
    SQL Text:  
    Deadlock Id 125: Process (Familyid 0, Spid 1354, Suid 32) was executing a SELECT command at line 1. 
    Deadlock Id 125: Process 1354 was involved in application ''. 
    Deadlock Id 125: Process 1354 was involved on host name 'xxxx'. 
    Deadlock Id 125: Process 1354 was involved in transaction '$chained_transaction'. 
    SQL Text: SELECT STEP_EXECUTION_ID, STEP_NAME, START_TIME, END_TIME, STATUS, COMMIT_COUNT, READ_COUNT, FILTER_COUNT, WRITE_COUNT, EXIT_CODE, EXIT_MESSAGE, READ_SKIP_COUNT, WRITE_SKIP_COUNT, PROCESS_SKIP_COUNT, ROLLBACK_COUNT, LAST_UPDATED, VERSION from BATCH_STEP_EXECUTION where JOB_EXECUTION_ID = @p0 order by STEP_EXECUTION_IDì 
    Deadlock Id 125: Process (Familyid 0, Spid 1354) was waiting for a 'shared row' lock on row 0 page 976758 of the 'BATCH_STEP_EXECUTION' table in database 'pmdb_load' but process (Familyid 0, Spid 1324) already held a 'exclusive row' lock on it. 
    Deadlock Id 125: Process (Familyid 0, Spid 1324) was waiting for a 'exclusive row' lock on row 1 page 976604 of the 'BATCH_STEP_EXECUTION' table in database 'pmdb_load' but process (Familyid 0, Spid 1354) already held a 'shared row' lock on it. 
      
    Deadlock Id 125: Process (Familyid 0, Spid 1324) was chosen as the victim. End of deadlock information.

  9. #9
    Join Date
    Jun 2005
    Posts
    4,241

    Default

    What version of Spring Batch are you using?

  10. #10

    Default

    Spring Batch - 2.1.0.RC1

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •