Processor missed call in a retry

[Dave Syer]

I can't follow the log fragment without more detail on the implementation. Can you set up a failing test case and post that?

[sebge2]

The test case is: commit-interval of 2, max-retries = 1, exception thrown every time the item a is written.

This is the behaviour we have seen:

Chunk: a, b
- process of a, b
- write of a, b => ignorable exception
- process of a
- write of a, b => ignorable exception
- process of a
- write of a => ignorable exception
- process of b
- write of b

We're expecting:

Chunk: a, b
- process of a, b
- write of a, b => exception
- process of a
- write of a => exception
- process of a
- write of a => exception
- process of b
- write of b

It seems that when it has tried one times to write a, b (max-retries = 1) according to the exception which is ignorable (the exception is skipped), it begins to process and write one-by-one. But, if the exception cannot be skipped (non-ignorable), it never processes and writes one-by-one and we get the follow result:

Chunk: a, b
- process of a, b
- write of a, b => ignorable exception
- process of a => can be retried + cannot be skipped

We are using templates, I hope the following code is adequate to understand the config file. The exception is thrown by the first partition of step1.partition (the second partition is executed successfully).

    <job id="templateJob" parent="abstractGenericJob"
         xmlns="http://www.springframework.org/schema/batch">
		<step id="preProcessing" next="staging" parent="templateJob.internal.preProcessing"/>
        <step id="staging" next="step1.partition" parent="templateJob.internal.staging"/>
		<step id="step1.partition" next="step2.partition" parent="templateJob.internal.partition1"/>
		<step id="step2.partition" parent="templateJob.internal.partition2">
            <next on="COMPLETED" to="postProcessing"/>
            <end on="COMPLETED WITH *"/>
		</step>
		<step id="postProcessing" parent="templateJob.internal.postProcessing"/>
    </job>

    <!-- Step 2 -->
    <step id="templateJob.internal.partition2" parent="abstractPartitionerStagedStep"
         xmlns="http://www.springframework.org/schema/batch">
        <partition step="step2" partitioner="step2.partitioner">
            <handler grid-size="${step2.grid.size}" task-executor="step2.taskExecutor"/>
        </partition>
    </step>

	<step id="step2" parent="abstractStagedPartitionStep"
         xmlns="http://www.springframework.org/schema/batch">
        <!-- outside a partition must use abstractStagedStep -->
		<tasklet>
			<chunk reader="stagingItemReader" processor="step2.internal.itemProcessor"
				   writer="step2.internal.itemWriter" commit-interval="${step2.commit.interval}"/>
		</tasklet>
	</step>

[sebge2]

Moreover, when I've looked at the log, I've seen that our SkipPolicy receives a negative skipCount for that test (i < 0):

    public boolean shouldSkip(final Throwable throwable, final int i) throws SkipLimitExceededException {
        try {
            final boolean isSkipped = delegateSkipPolicy.shouldSkip(throwable, i);
            if (!isSkipped) {
                logger.debug("Not skip " + throwable + " with number of skipped  [" + i + "]");
                callErrorListeners(throwable);
            } else {
                logger.debug("Skip " + throwable + " with number of skipped [" + i + "]");
            }

            return isSkipped;
        }
        catch (SkipLimitExceededException e) {
            callErrorListeners(throwable);
            throw e;
        }
    }

The log:

2010-03-18 13:41:06 [lazyBindingProxy.step1.taskExecutor#sysinit-2] DefaultSkipPolicy [DEBUG] It's ignorable, so SKIP
2010-03-18 13:41:06 [lazyBindingProxy.step1.taskExecutor#sysinit-2] StagingSkipPolicy [DEBUG] Skip com.bsb.sf.batch.test.exception.MyIgnorableException: Throwing again an exception for [account=11fb0177-526d-4932-8245-d0f6c6a37061-123123-0(#10, number = 0)] (retryCount=1) with number of skipped [-1]

[Dave Syer]

That's expected, and if it is the cause of all the confusion I'm sorry (we can make the documentation of SkipPolicy more explicit). A call with skipCount=-1 means "is this a skipable exception, never mind how many skips?" Your custom skip policy might need to be aware of that I suppose.