Item processor rollbacks & starts over again if exception occurs in between

Hi,

We have been using spring batch framework for few months now. Yesterday we noticed that batch processing doesnt exactly happen the way we thought it would.

Details:
1. Expected : Read-Process, Read-Process till chunk limit is reached, then write the entire chunk.

Actual Behavior: But ItemReader actually reads the items in one go and then control goes to the processor.
Shouldn't it read one item, process it and then proceed with reading the next item?

2. Assume 4 items were read and now control goes to the processor. Here, lets say an exception occurs while processing the third item and this exception is skippable according to our skipPolicy, then we would expect the system to skip this item and proceed with its processing till the chunk limit is reached and then write them.

But when such a exception occurs, item processor rollsback and starts processing from the first item again, but this time it doesn;t process those items which led to exceptions. Why is this rollback happening?

With chunk limit ,say 10000, exception in 9999th item will lead to a rollback and thus a very bad delay.

http://my.safaribooksonline.com/book...UxODI5NTUvMjMx link doesnt explain why spring batch behaves this way.

Relevant link : http://www.coderanch.com/t/548941/Sp...t-Interval-not

It would be great if someone can enlighten us :)

P.S : We use spring batch version 2.1.8

Thanks,
Gayathri

i think the Spring Batch documentation for Chunk Oriented Processing explains it quite good

that (rollback) bevaviour is mandatory for spring batch to isolate the bad item(s), basically it rollbacks the chunk and processes/writes each item with commit-rate=1 to find the bad one (in either processor or writer)


but i am with you, that the actual solution has its flaws, see a similar comment at stackoverflow*

Quote:

---

right now i could see some potential problems with a more pragmatic approach, things like restartability, deterministic behaviour or complex chunk control in case of multiple errors, in the end you need a dynamic/adapting chunk concept**...which sounds cool btw

---

*shamelessly taken from myself
**emphasis mine

Quote:

---

With chunk limit ,say 10000, exception in 9999th item will lead to a rollback and thus a very bad delay

---

the question is - how to make it better, more adapting, etc. ?

Thanks for the reply Michael.

Quote:

---

that (rollback) bevaviour is mandatory for spring batch to isolate the bad item(s), basically it rollbacks the chunk and processes/writes each item with commit-rate=1 to find the bad one (in either processor or writer)

---

This makes sense for write operation, but why during processing phase also?

the framework does not know, if the exception is thrown in processing or writing

in my opinion there actually is no real processing "phase", as far as i understand, the processing is an optional extension for writing, basically a pluggable "i want to do something with each item individually", but it is part of "writing"