Rollback listeners

[sebge2]

Hello,

Is there a simple mechanism allowing to know which items have not been processed & written ?

There is org.springframework.batch.core.listener.ItemListen erSupport which implements three listeners (for reading, processing & writing listening). These listeners are called once an exception is thrown outside them, but an item can be retried, or skipped (according to the context of the batch process). So, how can I know exactly which items are finally not processed & not written ?


Best regards,

Sébastien

[Dave Syer]

What's the use case? Why do you care about items that are rolled back if they are subsequently reqprocessed (as a retry or a skip)?

[snicoll]

We use the progress indicator pattern with a staging table. We want to add the following statuses:

* ERROR: the item could not be processed
* ROLLEDBACK: item in the chunk that were processed successfully but who were rollbacked because of an item in error state

We could listen to the errors and use a requires new transaction but since we can't figure out if the item will be reprocessed anyway, it feels weird to mark it as error while it's being actually reprocessed afterward.

It feels like that the second is useless since Spring Batch will process the items one by one in case of error. Is it something we can configure? What if the chunk must be processed entirely or not at all for instance (and still use the retry feature)?

[Dave Syer]

it feels weird to mark it as error while it's being actually reprocessed afterward.

That's sort of what I was trying to say. I think.

What if the chunk must be processed entirely or not at all for instance (and still use the retry feature)?

That behaviour is not supported out of the box with retry (or skip) because it is not compatible with the contract of the skip listeners (which are called through a retry internally) . Is it important? I guess it would be possible to implement a ChunkProcessor that can retry the whole chunk and not try to identify the failed items for the skip listeners.

[snicoll]

That's sort of what I was trying to say. I think.

Yeah ok but you're not taking the whole problem into account. Of course if the transaction will rollback and the processing will be retried it does not make any sense. However, if the number of retries have been reached (and the batch will fail anyway) we want to be able to know which item causes the issue (preferably with some hint like an error message or something).

Same thing regarding skip. We can skip an item at some point but how do we mark it as skipped?

To me, the progression indicator pattern is kinda useless if the only thing you can say is "done" or "not done" without being able to say why it's not done.

[Dave Syer]

However, if the number of retries have been reached (and the batch will fail anyway) we want to be able to know which item causes the issue (preferably with some hint like an error message or something).

That makes sense in a way, but in the case of a write failure, in general you really can not find the item that caused the issue without scanning for it (and committing partial chunks in the process). You will definitely get more than a hint and an error message in the StepExecution (and in logs) about what went wrong. The item just may not be identifiable in principle.

Same thing regarding skip. We can skip an item at some point but how do we mark it as skipped?

Isn't that what skip listeners do?

I think this discussion just shows why it is often infinitely preferable to have only unexpected, unrecoverable, fatal errors in a writer, and make sure anything that can be skipped or retried happens earlier in the process.