Potential synchronization problem in item oriented step?

[adrianshum]

I am reading 1.0.1's source but seems it is similar in 1.1.

In ItemOrientedStep:

protected ExitStatus doExecute(final StepExecution stepExecution) throws Exception {
    // blablabla

    return stepOperations.iterate(new RepeatCallback() {

        public ExitStatus doInIteration(RepeatContext context) throws Exception {
            // balblabla
            try {
                // Process chunk .... (1)
                try {
                    synchronizer.lock(stepExecution);
                }
                catch (InterruptedException e) {
                    stepExecution.setStatus(BatchStatus.STOPPED);
                    Thread.currentThread().interrupt();
                }
                // step execution persistence

            }
            catch (Error e) {
                processRollback(stepExecution, contribution, fatalException, transaction);
                throw e;
            }
            catch (Exception e) {
                processRollback(stepExecution, contribution, fatalException, transaction);
                throw e;
            }
            finally {
                synchronizer.release(stepExecution);
            }
            //blblabla
            return exitStatus;
        }
    });
}

In case of any exception occured in (1) or during synchronizer.lock(), the internal semaphore in synchornizer is NOT acquired. However, the outer final block releases synchronizer, and hence, releasing the semaphore.

From JDK API of 1.5, it stated that Semaphore can be released by another thread which is not the original acquirer of semaphore.

So, in case of parallel processing, I may goes into some case:

Thread 1 acquired the semaphore and doing those step execution persistence stuff.

Thread 2 have exception in processChunk, and hence releasing the semaphore.

Thread 3, originally waiting for semaphore, is now acquired the semaphore because Thread 2 releases it, and hence, it goes to the peresistence block of code.

In such case, Thread 1 and 3 goes into the should-be-protected block of code for step execution persistence.

I am not sure if it is viable... Can developer take a look, so that I may raise a ticket if it is valid?

(I am using 1.0.1 and facing some optimistic concurrency issue in updating step execution during parallel processing. Have not yet found out if it is related to this issue)

Adrian

[Dave Syer]

I think you are right that there is a problem. By all means create an issue in JIRA. If you find that this is causing your failure I will be surprised, but then if you manage to prove that this is the case, you will have a test case for the issue.

[adrianshum]

After some trials today, I think very likely that it cause my problem... I did a quick fix on that and now no optimistic concurrency exception is caught.

I wonder how should I make a test case for this? As this means much more than a unit test and we should probably need a DB to simulate it.

Make it brief, I extended RepeatOperationStepFactoryBean to make my own factory, which create a ThreadPoolTaskExecutor as step's stepOperations. Then I run my writer parallelly in multiple thread, and when one of them throws exception during its reader/writer logic, it may cause the problem.

I shall raise an JIRA ticket first, and may attach a test case if u can show me a bit on how to do so in this complicated scenario. Or shall I just make a unit test to make sure no concurrent access to JobRepository in the ItemOrientedStep logic?

[cch]

Hi, i face the almost same problem is when i start the two job synchronously there is a deadlock occur. I have no idea how to solve this kind problem. It not everytime occurs.
I currently run 2 different job.

15:28:09,559 ERROR [STDERR] org.springframework.dao.ConcurrencyFailureException: PreparedStatementCallback; SQL [INSERT into BATCH_EXECUTION_CONTEXT(EXECUTION_ID, D
et lock; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock;
15:28:09,565 ERROR [STDERR]     at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.translate(SQLExceptionSubclassTranslator.java:52)
15:28:09,568 ERROR [STDERR]     at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.translate(SQLErrorCodeSQLExceptionTranslator.java:237)
15:28:09,571 ERROR [STDERR]     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:582)
15:28:09,574 ERROR [STDERR]     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:594)
15:28:09,578 ERROR [STDERR]     at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.insertExecutionAttribute(JdbcExecutionContextDao.java:236)
15:28:09,581 ERROR [STDERR]     at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionAttribute(JdbcExecutionContextDao.java:188)
15:28:09,584 ERROR [STDERR]     at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveOrUpdateExecutionContext(JdbcExecutionContextDao.java:1
15:28:09,587 ERROR [STDERR]     at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveOrUpdateExecutionContext(JdbcExecutionContextDao.java:1
15:28:09,590 ERROR [STDERR]     at org.springframework.batch.core.repository.dao.JdbcStepExecutionDao.saveOrUpdateExecutionContext(JdbcStepExecutionDao.java:142)
15:28:09,593 ERROR [STDERR]     at org.springframework.batch.core.repository.support.SimpleJobRepository.saveOrUpdateExecutionContext(SimpleJobRepository.java:254)
15:28:09,595 ERROR [STDERR]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
15:28:09,599 ERROR [STDERR]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
15:28:09,602 ERROR [STDERR]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
15:28:09,604 ERROR [STDERR]     at java.lang.reflect.Method.invoke(Method.java:597)
15:28:09,607 ERROR [STDERR]     at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
15:28:09,611 ERROR [STDERR]     at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
15:28:09,614 ERROR [STDERR]     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
15:28:09,616 ERROR [STDERR]     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
15:28:09,618 ERROR [STDERR]     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
15:28:09,621 ERROR [STDERR]     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
15:28:09,624 ERROR [STDERR]     at $Proxy174.saveOrUpdateExecutionContext(Unknown Source)
15:28:09,626 ERROR [STDERR]     at org.springframework.batch.core.step.item.ItemOrientedStep.doExecute(ItemOrientedStep.java:225)
15:28:09,629 ERROR [STDERR]     at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:172)
15:28:09,635 ERROR [STDERR]     at org.springframework.batch.core.job.SimpleJob.execute(SimpleJob.java:100)
15:28:09,638 ERROR [STDERR]     at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:86)
15:28:09,641 ERROR [STDERR]     at java.lang.Thread.run(Thread.java:619)
15:28:09,644 ERROR [STDERR] Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting tra
15:28:09,647 ERROR [STDERR]     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
15:28:09,649 ERROR [STDERR]     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
15:28:09,651 ERROR [STDERR]     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
15:28:09,654 ERROR [STDERR]     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
15:28:09,656 ERROR [STDERR]     at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
15:28:09,659 ERROR [STDERR]     at com.mysql.jdbc.Util.getInstance(Util.java:381)
15:28:09,661 ERROR [STDERR]     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1045)
15:28:09,664 ERROR [STDERR]     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
15:28:09,666 ERROR [STDERR]     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491)
15:28:09,668 ERROR [STDERR]     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423)
15:28:09,672 ERROR [STDERR]     at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)
15:28:09,674 ERROR [STDERR]     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
15:28:09,677 ERROR [STDERR]     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
15:28:09,681 ERROR [STDERR]     at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
15:28:09,684 ERROR [STDERR]     at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2019)
15:28:09,688 ERROR [STDERR]     at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1937)
15:28:09,691 ERROR [STDERR]     at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1922)
15:28:09,693 ERROR [STDERR]     at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:94)
15:28:09,697 ERROR [STDERR]     at org.springframework.jdbc.core.support.AbstractLobCreatingPreparedStatementCallback.doInPreparedStatement(AbstractLobCreatingPrepa
15:28:09,700 ERROR [STDERR]     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:566)
15:28:09,703 ERROR [STDERR]     ... 23 more

[Dave Syer]

This is a common experience for MySql users apparently (and unrelated to the original post). If you really are launching 2 distinct job instances (different parameters and/or job) then it is entirely possible that the exception you see is due to excessive locking by the database in response to the default isolation level we set in the JobRepositoryFactoryBean. Look at the docos for this class and in the reference guide where we discuss TX isolation. MySql users normally report that the problem goes away if they set the isolationLevelForCreate property of the factory bean to ISOLATION_REPEATABLE_READ.

[cch]

thanks! it work