Infinite loop during Retry

[veaven]

Hi all,
I am trying the retrySample in Spring-batch-samples, I found out it will be in the infinite loop if the exception is not a transient exception.
I changed code from
public void write(Object data) throws Exception {
counter++;
if (counter == 2 || counter == 3) {
throw new RuntimeException("Temporary error");
}
}
To:
public void write(Object data) throws Exception {
throw new RuntimeException("Temporary error");
}

It looks like that <property name="retryLimit" value="3" /> does not work.

Could anyone please tell me the reason?

[lucasward]

I tried making the same change and the job failed without processing any records, since it will always throw and exception. Did you make any other modifications to the job? Also, are you using 1.0 or another release?

[veaven]

Hi lucasward,
I made a minor change on Trade class of retrySample job to reproduce the issue. and I attached them.
After debugging the code, I found out the issue that might relate to ItemKeyGenerator. Our situation is that we need to generate a new id for trade in ItemWriter.write(), if exception happens when we write code to db, we will generate another id for it during retry.
But the code in ItemWriterRetryPolicy.hasFailed uses the trade itself as the key, that cause the infinite loop since it cannot find out the previous trade from retryContextCache due to trade.id has been changed.

I guess this issue may be the same one of http://jira.springframework.org/browse/BATCH-537

Cheers,
David

	protected boolean hasFailed(FailedItemIdentifier failedItemIdentifier, ItemKeyGenerator keyGenerator, Object item) {
		if (failedItemIdentifier != null) {
			return failedItemIdentifier.hasFailed(item);
		}
		return retryContextCache.containsKey(keyGenerator.getKey(item));
	}

[Dave Syer]

That is indeed the same as BATCH-537. You should find that the latest 1.0.1 snapshot will leave the loop in finite time, and hopefully give you a hint about the fact that your ItemKeyGenerator is not cuttting it.

If you really need your item to have a hashCode based on its ID, which itself is not stable (understandable, but not always a good idea), you will have to provide an ItemKeyGenerator that uses a different key generation algorithm. Or maybe a RetryContextCache that doesn't use HashMap (haven't tried this but it should work).

[veaven]

thank you, Dave.
yep, I can workaround it, but it would be very useful if we can provide another ItemKeyGenerator which does not depend on individual item.

[lucasward]

it would be very useful if we can provide another ItemKeyGenerator which does not depend on individual item.

I'm not quite sure I understand what you mean here? An ItemKeyGenerator is given the item as a parameter to the getKey method, but it could theoretically use anything it wanted to in order to determine equality. The point is, we need some way to recognize an item as one that should be retried, or for that matter skipped. This is because in a rollback scenario we may not be getting back the same item from a memory perspective (if it was recreated) even though it is functionally the same. Because both read and write are in the transaction, we have to go back to the reader, which depending upon implementation, may more may not regenerate the item. (Some item readers buffer the items, i.e. the JdbcCursorItemReader)

The only way to buffer them in the step (which would allow for a memory comparison of equality) would be to put only the write in a transaction, leaving calls to read in a separate transaction.

[veaven]

thank you, lucasward. I got the idea.