ItemReader read() called before ItemWriter open() Resulting in items not written

[jdihayco]

Hello,

Can somebody please tell me what would cause the read() method of an ItemReader to be called even before the open() method of the corresponding ItemWriter is called?

What I did was to subclass JdbcCursorItemReader and did an override on its read() method, with a logic to call JdbcCursorItemReader's read() method if a list is empty (to query database when list is empty) but to return the list's elements one at a time if it is not.

When there were 4 items on the list, I noticed that my ItemReader's open() method was called first followed by 4 calls to ItemReader.read() and then followed by a call to my ItemWriter's open() method. My ItemWriter's write() method was never called even once. (please see attached logs)

When I tried to increase the list to 65 elements, what happened was that my ItemReader's open() method was called, then 9 calls to ItemReader.read() were made, and then the corresponding ItemWriter's open() method was called. After that, RepeatTemplate and StepContextRepeatCallback got involved, and the ItemReader read() and ItemWriter write() began to be called alternatingly. This resulted in the first 9 elements read before ItemWriter open() to be not written while the remaining items, read after ItemWriter open(), were written successfully.

I tried using another custom ItemReader (not subclassing JdbcCursorItemReader) and the behavior remained the same.

Can anybody explain what would cause Spring Batch to call the read() method of an ItemReader even before the open() method of the corresponding ItemWriter is called, resulting in these items not being sent to ItemWriter.write()? My step's commit interval is 500.

Thanks,

Johannis

NOTE: When I refer to my ItemReader's or ItemWriter's open() method, I am actually referring to the open() method in ItemStream interface which my ItemReader and ItemWriter implementing classes also implement.

[snicoll]

I also tried recently to add additional logic to JdbcCursorItemReader (in my case to wrap it in an ItemReader that does not implement ItemStream because it's not supposed to work properly in a multi-thread environment with the default settings).

Anyway, I got the same problem of getting 0 results and it turned out that my environment was simply wrong. From your log, all I can see is that there is nothing to read.

In your case, you'd better use a delegate pattern where you use the jdbc cursor item reader instead of subclassing it. Can you show your code?

[jdihayco]

Hi snicoll,

I will try that delegate pattern you suggested once I'm back at the office. Anyway, I was not able to mention earlier that the number of items read before the ItemWriter's open() method is called appears to be tied with the number of items successfully read and written in an immediately previous failed run of the same step.

In other words, it appears that my JdbcCursorItemReader subclass appears to be "pre-reading" a number of items corresponding to the number of items successfully read and written in a previous failed step. I suspected the open() method of AbstractItemCountingItemStreamItemReader (JdbcCursorItemReader's abstract superclass) which calls jumpToItem() which in turn calls read() a number of items equal to the read.count value in the ExecutionContext.

However, I am overriding the open() method without calling jumpToItem(). I even tried to override jumpToItem() just to add logging traces but it was not called as well.

Still I am positive that the number of items that are being "pre-read" before ItemWriter open() is called is equal to the number of items successfully read and written in the previous execution minus one.

Thanks,

Johannis

[snicoll]

oh well, If you had said that earlier...

JdbcCursorItemReader saves the state of where it is so that it can reset the cursor to the position where it was in case of failure.

Just call setSaveState(false) and your problem will go away.

[jdihayco]

Hi snicoll,

Thanks for this info. However, I just found out that my current troubles are caused by something else other than failing to set saveState on JdbcCursorItemReader to false.

As previously mentioned, my problem actually happens when I try to return items from an ArrayList on ItemReader.read() when that list is not empty (I call superclass JdbcCursorItemReader's read() method otherwise). The list represents skipped items from a previous failed execution of the same step. Thus in this instance, this is actually not related to resetting the cursor position.

My ItemReader's "pre-reading" was actually caused by a property in the step execution context ("org.springframework.batch.core.step.item.ChunkMon itor.OFFSET") which got set to the number of items successfully read and written in the previous failed execution.

As logically I do not need this OFFSET since I am now working from a list of the skipped items (as opposed to working on the same population of items retrieved from the database in the previous execution), I just resolved the issue by setting ChunkMonitor.OFFSET to 0 in the open() method of my JdbcCursorItemReader subclass as show in snippet below:

@Override
publicvoid open(ExecutionContext executionContext) throws ItemStreamException
{
try {
if(executionContext.containsKey("org.springframework.batch.core.step.item.ChunkMonitor.OFFSET")){
  log.debug("executionContext ChunkMonitor.OFFSET: " + 
   executionContext.getInt("org.springframework.batch.core.step.item.ChunkMonitor.OFFSET"));
   log.debug("setting ChunkMonitor.OFFSET to 0...");
   executionContext.putInt("org.springframework.batch.core.step.item.ChunkMonitor.OFFSET", 0);
}
...

Now I don't have items that are being read before my ItemWriter's open() method is called and so I no longer miss any item.

Thanks for the help,

Johannis

[snicoll]

okay. I still believe you should use a delegate instead of subclassing.