TaskletStep semaphore.acquire() blocks. Can't migrate batch to 2.1.6.

[mjreged]

Hello Spring Batch Experts.
I came across a use case where my batch job hangs indefinitely.
I am in a middle of batch migration from 1.1.4 to 2.1.6. This issue is a show stopper for us right now.
While i do the refactoring i am running Junit tests to verify correctness.
Job starts the first step, reads one item, writes, reads second item, writes then TaskletSep.java calls semaphore.acquire() on line 390 and hangs.
Our batches are of the simplest uses case. We use readers to call pojo services to fetch next candidate item and we use writers to delegate to another pojo service to process the item. We have commit interval set to 1 and generally we expect to skip not repeat on any exception. Our pojo services have transaction boundries configured, out transaction manager is JPA transaction manager.

Job definition:

<job id="nonRenewal180DayJob" xmlns="http://www.springframework.org/schema/batch">
        <step id="NonRenewStep" next="ReviewStep">
            <tasklet transaction-manager="transactionManager">
                <chunk reader="nonRenew180DayReader" writer="nonRenew180DayWriter"  commit-interval="1"  skip-limit="100000">
                    <skippable-exception-classes>
                        <include class="java.lang.Exception" />
                    </skippable-exception-classes>
                </chunk>
                <listeners>
                    <listener>
                        <beans:bean class="com.bipt.tiva.batch.renewal.listener.NonRenew180DayListener" parent="isfJobListener" />                   
                    </listener>
                </listeners>
            </tasklet>
        </step>
.... more steps exist

simple-job-launcher-context.xml

<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
		<property name="jobRepository" ref="jobRepository" />
</bean>
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"
          p:validateTransactionState="false" p:isolationLevelForCreate="ISOLATION_DEFAULT" p:transactionManager-ref="transactionManager"/>         

<bean id="jobRegistry" class="org.springframework.batch.core.configuration.support.MapJobRegistry" />

Junit Test: snippet

@Resource JobLauncher launcher;    
@Resource  Job job;

@Test 
@Transactional
 public void testLaunchJob() throws Exception {
    launcher.run(job, jobParameters);
}

Our original 1.1.4 batch project did not use database backed JobRepository
and during this migration phase we want to continue this approach for now.

My question is this; did i perhaps miss some configuration either in the job definition itself or generally setting up the framework. Can the use of MapJobRepositoryFactoryBean be the reason for the job hang?
Your input is most appreciated.
Thank you.

[Dave Syer]

The MapJobRepository is not the cause of the problem, but it is preventing you from getting the nice exception message telling you not to use @Transactional in your test method. remove that and it should work.

[mjreged]

Dave,
Your are right, i deliberately set p:validateTransactionState="false" to avoid the exception that I think you are referring too ( AbstractJobRepositoryFactoryBean.java line 160 -167 ).
I thought I could use that so my test can continue to run.

My concern and understanding was that I needed @Transactional in my Junit so that the test can revert database changes that our pojo services might have made during writes. Is my thinking wrong here?

We use Integration Junit testing a lot. We surely rely on these tests to debug and re-run the complex operations of these jobs over and over. Our Batch runs are difficult to setup. If I run the job and not be able to rollback my state I will be spending most of my day manually changing records in the database so that I can re-run the test at most 3 times in a full days of work.

So my question is; How will the test be able to rollback at the end of the test method?

Any insight or alternatives of achieving this use case would help.
Thanks

[arno]

you're not supposed to launch a Spring Batch job from within an ongoing transaction, as the framework handles transactions itself. I don't know automatic solution (e.g. rollback) to go back to the initial state after the execution of a Spring Batch job. I tend to populate the database in the test before the run with a typical, not-too-big dataset.

[Vincent Cornet]

Hello folks,


I'm facing the same issue than mjreged, with pretty much the same use case (integration tests with JUnit and test case-scoped transactions)...

I understand the point of view stating that a batch is not supposed to be triggered within a tx, but :
* Why not? AFAIK, there actually is no strong argument to back up the aforementioned point of view. Or is it so by design indeed?
* I think it would be fairly easy to make it possible, wouldn't it?
* Either way, I don't think waiting for a semaphore indefinitely is a reasonable behaviour (as it sure is not the easiest to recover from). It should not happen IMHO.

So I must admit I'm a bit disappointed with the "don't do it, you'll be ok" answer... :/
If needed, I'm ready to contribute to solve this problem; anyway if I manage to work around this without a patch and in a not-too-ugly way (it's for tests only, not for production code, so there are ways :þ), I'll let you know.


Best regards.