Unable to configure retryPolicy for SkipLimitStepFactoryBean

[Igor Stativkin]

Hello,

My batch application is pretty simple - input is a flat file, outputs: database and error file for wrong records. On write phase I can receive normal DB exception like DataIntegrityViolationException, process it and then rethrow again to rollback DB transaction. I need to skip incorrect records and do not try to retry them (to avoid wrong data in error file and log). Now I have a problem trying to configure retry behavior of SkipLimitStepFactoryBean - every time it retries each wrong record exactly two times. I tried to set retryLimt to 0, 1, even -1, tried to configure retryPolicy as NeverRetryPolicy - everything the magic two attempts were performed.

<bean id="simpleStep" class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"
	abstract="true">
	<property name="itemKeyGenerator"><bean class="my.company.batch.ItemKeyGenerator"/></property>
	<property name="transactionManager" ref="rlTransactionManager" />
	<property name="jobRepository" ref="jobRepository" />
	<property name="commitInterval" value="1" />
	<!-- setting skip limit to a maximum number of record in the file -->
	<property name="skipLimit" value="30000"/>
	<property name="retryPolicy"><bean class="org.springframework.batch.retry.policy.NeverRetryPolicy"/></property>
	<property name="skippableExceptionClasses"><value>org.springframework.dao.DataAccessException</value></property>
</bean>

If you have any idea what's whrong please help. I do not want to keep workaround I have (manual transaction control in writer) to make all the app behavior configurable.

Thanks,
Igor.

[Dave Syer]

What makes you say the items are retried? Is your writer being called twice with the same item?

[Igor Stativkin]

Hello,

I can see duplication in my traces, error file and job log file - the whole read-write cycle is being called second time and only then this item is skipped, not on the first iteration.

ItemKeyGenerator uses lineNumber from original file - so wrong record is identified correctly.

[Dave Syer]

Can you double check that? The read should happen twice, but not the write (assuming that the error was in the write). We have to roll back the transaction if there is an error, so the analysis of whether to retry can only happen in a fresh transaction - otherwise anything you did to recover would be rolled back as well potentially.

The internals are a bit different in 2.0: we buffer the items and do not call read() again for the same item after a failure, but the basic idea about the transaction is the same.

[Igor Stativkin]

I checked it several times with different configuration of step factory. I provided NeverRetryPolicy, I tried to configure retryLimit instead. All the time the same result. I have checked my ItemKeyGenerator - it identifies record correctly - so I thought it should stop and skip record after second read.

I cannot use 2.0 because there is no final release of it.

[Dave Syer]

You shouldn't have to use 2.0 to get the skip to work. You should (as I said) expect that the reader is called twice per item, but that the writer is skipped.

There have been some issues with the retry policy in the factory beans. Can you try it with vanilla settings

<bean id="simpleStep" class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
	<property name="itemKeyGenerator"><bean class="my.company.batch.ItemKeyGenerator"/></property>
	<property name="transactionManager" ref="rlTransactionManager" />
	<property name="jobRepository" ref="jobRepository" />
	<!-- setting skip limit to a maximum number of record in the file -->
	<property name="skipLimit" value="30000"/>
        ... <!-- set tour reader and writer here -->
</bean>

Are you sure that the itemKeyGenerator is doing its job correctly? Remember that if there is an error the transaction will roll back and the input file is read again, so you can't just count the items in the key generator - it needs to be a key based on the data.

[Igor Stativkin]

Hello Dave,

I tried your vanilla settings, as a result for the same multiline item that causes DataIntegrityViolationException there were 3 reads and two writes (one extra-cycle than it's supposed).

As regarding to KeyGenerator - it uses line number of the first line of the item in input flat file, so it's unique for the same record. (I have tests for the generator and it works fine).

Is there anything else what I can try?

Thanks,
Igor.

[Dave Syer]

I'm not clear on the key generator internals, and I'm not sure what you mean by "mulitiline item". If you can post some more code that would probably help, or even an entire test case.

[Igor Stativkin]

I think I found the cause of my problem - you were right - it's KeyGenerator and its test.

I have a question regarding org.springframework.batch.item.file.separator.Line Reader and ResourceLineReader method getPosition(). I thougth it should return position of the first line of the record that we retrying, i.e. first time it's 1, then fault with db and 2nd read where we should receive 1 again, but this method returns line number of the next item in file and not one that we're retrying to read, is it ok?

[Igor Stativkin]

I found the cause of issue:

If you plan to use LineReader.getPosition() as a method that provides you with a unique item's identifier - call this method right before you want to return an item from your ItemReader.read() implementation after reading all the lines related to the item. It will return you a line number of next record and will be the same on next retry for the same item.

Thanks.