Hi All,

My application load the job files and start the job parallely in different threads. Each thread will start the same job with different parameters.
As per business each threads will always get different rows to process. so we are not getting any deadlocks in the business flow.
But while updating the Spring Batch tables we are getting deadlocks.

I am using SpringBatch2.1, Hibernate3, Sql server 2005 and isolation set ISOLATION_READ_COMMITTED.

How To handle the deadlocks in this case?
If we restart the job will it start from the place where it stops?

Please find the job configurations and exception.

Code:
ERROR org.springframework.batch.core.job.AbstractJob  - Encountered fatal error executing job
org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:105)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
	at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)
	at org.springframework.batch.core.launch.support.SimpleJobOperator.start(SimpleJobOperator.java:300)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.batch.core.launch.support.RuntimeExceptionTranslator.invoke(RuntimeExceptionTranslator.java:32)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy3.start(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.reflect.misc.Trampoline.invoke(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
	at javax.management.modelmbean.RequiredModelMBean.invokeMethod(Unknown Source)
	at javax.management.modelmbean.RequiredModelMBean.invoke(Unknown Source)
	at org.springframework.jmx.export.SpringModelMBean.invoke(SpringModelMBean.java:88)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
	at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)
	at org.springframework.jmx.access.MBeanClientInterceptor.doInvoke(MBeanClientInterceptor.java:405)
	at org.springframework.jmx.access.MBeanClientInterceptor.invoke(MBeanClientInterceptor.java:353)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy8.start(Unknown Source)
	at my.application.batch.core.thread.StepControllerThread$1.run(StepControllerThread.java:542)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=Generate_Arrears_ReportJob at state=Generate_Arrears_ReportJob.ValidateXSanPlanMonthCutOffDateStep with exception
	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:152)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:99)
	... 39 more
Caused by: org.springframework.dao.DataAccessResourceFailureException: Could not increment identity; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 135) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at org.springframework.jdbc.support.incrementer.SqlServerMaxValueIncrementer.getNextKey(SqlServerMaxValueIncrementer.java:107)
	at org.springframework.jdbc.support.incrementer.AbstractDataFieldMaxValueIncrementer.nextLongValue(AbstractDataFieldMaxValueIncrementer.java:125)
	at org.springframework.batch.core.repository.dao.JdbcStepExecutionDao.saveStepExecution(JdbcStepExecutionDao.java:117)
	at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:158)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy0.add(Unknown Source)
	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:111)
	at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
	... 41 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 135) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:197)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1493)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:775)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:676)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4575)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1400)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:179)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:154)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeUpdate(SQLServerStatement.java:633)
	at org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:228)
	at org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:228)
	at org.springframework.jdbc.support.incrementer.SqlServerMaxValueIncrementer.getNextKey(SqlServerMaxValueIncrementer.java:104)
	... 59 more

Code:
<job id="Generate_Arrears_ReportJob" restartable="true"
		xmlns="http://www.springframework.org/schema/batch">

		<!--
			The Step for the validation of the XsanPlan Month Cut Off Dates .
		-->
		<step id="ValidateXSanPlanMonthCutOffDateStep">
			<tasklet transaction-manager="transactionManager" ref="validateXSanPlanMonthCutOffDate">
			</tasklet>
			<next on="VALIDATION_SUCCESS" to="Generate_Arrears_ReportStep" />
			<end on="*"/>
		</step>

		<!--
			The Step records the arrears plan in the Arrears Report .
		-->
		<step id="Generate_Arrears_ReportStep">
			<tasklet transaction-manager="transactionManager">
				<chunk reader="commonStepListItemReader" processor="batchRunTaskListProcessorWrapper"
					writer="commonStepListWriter" commit-interval="1" skip-limit="1000" retry-limit="10">
					<skippable-exception-classes>
						<include
							class="my.application.batch.core.exception.EpsilonBatchBusinessException" />
							<include
							class="my.application.batch.core.exception.EpsilonBatchBusinessValidationFailedException" />
							<include class="org.hibernate.exception.LockAcquisitionException" />
					</skippable-exception-classes>
					<retryable-exception-classes>
						<include class="org.hibernate.exception.LockAcquisitionException" />
					</retryable-exception-classes>
				</chunk>
			</tasklet>
		</step>
	</job>
Code:
 
<bean id="jobRepository"
		class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean"
		p:dataSource-ref="dataSource" p:transactionManager-ref="transactionManager" p:isolationLevelForCreate="ISOLATION_READ_COMMITTED"  />