onSkipInProcess not called when expected

[Yvory]

Good day to you.

I have had this problem countless time now and I really need someone to help me out on this.

I have a SkipListener with Annotations to log skips into a seperate log file.

I expect it to be called every time a item is skipped but it is not. Usually it works just fine, but now in my integrationtests it is sometimes not called, and it hinders me in my tests, because I get skips which are not logged, when they should be. This is especially dangerous in a live environment when it is vital for me to know why an element has been skipped.

I found out that in the cases when I expect an OnProcessInSkip should be called it calls onProcessError instead, where I do not have a Listener defined for.

If I add a Listener for this then I see the error message.

I have the phenomenon that if I by chance have a whole chunk that is skipped,
then the skipListener is not called at all, only the errorListener, which is just WRONG. It is vital to me to get a skip log message EVERYTIME there is a skip, no matter what.
There is no indication on the API Docu OR the reference guide that the interface is not called in these cases.

If I left the errorListener in my config, I'd get most of the log messages double in my log so how do I fix this? Or is this is a BUG and the DEVs have to fix this?

I cannot work like that if there is no way around that problem I will have to throw away all my SkipListeners and log via AOP or in the places where the Skip exception is raised, which is awkward and not a nice way to handle this.


This is my step config:

    
<batch:job id="job" parent="baseJob">
        <batch:step id="exportStep" parent="basicExportStep">
            <batch:listeners>
                <batch:listener ref="configurationLoader" />
                 <batch:listener ref="itemSkipListener" />
                <batch:listener ref="statisticListener" />
                <batch:listener ref="statisticWriter" />
            </batch:listeners>
        </batch:step>
    </batch:job>

    <batch:job id="baseJob" abstract="true" restartable="true">
        <batch:validator ref="jobParamsValidator" />
    </batch:job>


    <batch:step id="basicExportStep" abstract="true">
        <batch:tasklet>
            <batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="10" skip-limit="9999">
                <batch:skippable-exception-classes>
                    <batch:include class="org.springframework.batch.item.validator.ValidationException" />
                    <batch:exclude class="java.lang.Throwable" />
                </batch:skippable-exception-classes>
            </batch:chunk>
        </batch:tasklet>
        <batch:listeners>
            <batch:listener ref="itemSkipListener" />
            <batch:listener ref="statisticListener" />
            <batch:listener ref="statisticWriter" />
        </batch:listeners>
    </batch:step>

I am grateful for any help.

Sincerly,
Yvory

[mminella]

What is the exception being thrown that triggers the skip?

[Yvory]

What is the exception being thrown that triggers the skip?

Its a org.springframework.batch.item.validator.Validatio nException. It's also in my config above.

[Yvory]

So is there going to be a statement to this ?

IS this normal behaviour and it is just not documented or is this a bug?

I really had hoped to get an answer to this....as I need it to know if I have to refactor my coding....

[mminella]

You said above that this is occuring in an integration test only (not in production environments). Is that integration test method transactional (marked with a @Transactional annotation)?

[Yvory]

I am using almost no annotations and all my readers, processors and writers are non-transactional. I only work on flat files and XML files.

It only is a problem in testing because of the test data, I hope it never occured in Production, but I cannot be sure.

I can reproduce this behaviour, with only one read item which is not valid. I get no exception, the OnSkip method is not called and it returns with Status completed.

I cannot say that this will never happen in our production environment....

edit:

After reading your question I went into debugging the code some more and my problem seeems to lie in this piece of code in the

FaultTolerantChunkProcessor:

				public O doWithRetry(RetryContext context) throws Exception {
					O output = null;
					try {
						count.incrementAndGet();
						O cached = (cacheIterator != null && cacheIterator.hasNext()) ? cacheIterator.next() : null;
						if (cached != null && !processorTransactional) {
							output = cached;
						}
						else {
							output = doProcess(item);
							if (!processorTransactional && !data.scanning()) {
								cache.add(output);
							}
						}
					}
					catch (Exception e) {
						if (rollbackClassifier.classify(e)) {
							// Default is to rollback unless the classifier
							// allows us to continue
							throw e;
						}
						else if (shouldSkip(itemProcessSkipPolicy, e, contribution.getStepSkipCount())) {
							// If we are not re-throwing then we should check if
							// this is skippable
							contribution.incrementProcessSkipCount();
							logger.debug("Skipping after failed process with no rollback", e);
							// If not re-throwing then the listener will not be
							// called in next chunk.
							callProcessSkipListener(item, e);
						}
						else {
							// If it's not skippable that's an error in
							// configuration - it doesn't make sense to not roll
							// back if we are also not allowed to skip
							throw new NonSkippableProcessException(
									"Non-skippable exception in processor.  Make sure any exceptions that do not cause a rollback are skippable.",
									e);
						}
					}

It goes into the coding with the rollbackClassifier not to the next where I need it to go. So far I never had to add any policies or check Classifiers....so maybe I may have some configuration wrong , but what I do not know.

I tried to use a NeverRetryPolicy but that changed nothing. I also tried to make the itemProcessors non transactional, which als serves no purpose for my problem.

this Rollback Classifier is set inside the Processor

	private Classifier<Throwable, Boolean> rollbackClassifier = new BinaryExceptionClassifier(true);

How do I get this to call the skip part of the above code?

[Yvory]

I resolved my problem now. I plunged into the deapth of the Spring Batch code to follow the trail of this classifier that missbehaved for me.

And I found a way to chnge the behaviour of the Classifier, which I never had used or _I even knew this configuration tag was there and more was NEEDED......

what did hte trick for my problem was to add my skip Exception to the no rollback exceptions also.....nothing of this is covered in the reference.

At least now it works with my before failing flatfile that I created to test this.

This is my step new config (addition is bold):

    
<batch:job id="job" parent="baseJob">
        <batch:step id="exportStep" parent="basicExportStep">
            <batch:listeners>
                <batch:listener ref="configurationLoader" />
                 <batch:listener ref="itemSkipListener" />
                <batch:listener ref="statisticListener" />
                <batch:listener ref="statisticWriter" />
            </batch:listeners>
        </batch:step>
    </batch:job>

    <batch:job id="baseJob" abstract="true" restartable="true">
        <batch:validator ref="jobParamsValidator" />
    </batch:job>

    <batch:step id="basicExportStep" abstract="true">
        <batch:tasklet>
            <batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="10" skip-limit="9999">
                <batch:skippable-exception-classes>
                    <batch:include class="org.springframework.batch.item.validator.ValidationException" />
                    <batch:exclude class="java.lang.Throwable" />
                </batch:skippable-exception-classes>
            </batch:chunk>
            <batch:no-rollback-exception-classes>
                <batch:include class="org.springframework.batch.item.validator.ValidationException" />
            </batch:no-rollback-exception-classes>
        </batch:tasklet>
        <batch:listeners>
            <batch:listener ref="itemSkipListener" />
            <batch:listener ref="statisticListener" />
            <batch:listener ref="statisticWriter" />
        </batch:listeners>
    </batch:step>

[Yvory]

Since it now counts a Skipped also as a Filtered item, it's counted double.

I cannot use this solution and I will abandon the use of the SkipListener interface and use AOP, more reliant.

[Yvory]

One last help for all those having the same problems....

I just tried something propsed in another thread which I had thought I had already tried.

Make sure to define your SkipListeners inside the <tasklet> tag. Then it will always be called when appropriate.

This is my configuration that finally worked...

    <batch:step id="basicExportStep" abstract="true">
        <batch:tasklet>
            <batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="30" skip-limit="9999" >
                <batch:skippable-exception-classes>
                    <batch:include class="org.springframework.batch.item.validator.ValidationException" />
                </batch:skippable-exception-classes>
            </batch:chunk>
            <batch:no-rollback-exception-classes>
                <batch:include class="org.springframework.batch.item.validator.ValidationException" />
            </batch:no-rollback-exception-classes>
            <batch:listeners>
                <batch:listener ref="documentSkipListener" />
                <batch:listener ref="statisticListener" />
            </batch:listeners>
        </batch:tasklet>
    </batch:step>