Exception skippable in itemWriter : how manage correctly fields of ItemWriter ?

[KevinF]

Hi,

I tried to skip one type of exception in my ItemWriter (it's a FlatFileItemWriter) by adding my exception like a skippable exception.

All work fine except for my fields of my ItemWriter.

If i'm right, when an exception occurs :

• the transaction rollback
• the items are re-writed with a commitInterval at one.

When the item that thrown the exception :

• the transaction is rollback an other time
• all item are re-writed except the item that thrown the exception.

So it's possible that some items are written twice. The transaction manages this successfully but the fields of my ItemWriter are impacted twice and there aren't transaction to manage my fields.

Below an example :

public class MyItemWriter {

    private Integer mnt = 0;

    public void write (List<? extends Integer> items) {
        for (Integer oneMnt : items) {
            mnt += items;
        }

        // My business
    }

}

In this example, with a commitInterval to 5 and only 5 items to read, if the item n°4 throw an exception skippable, the mnt after step is bad. Is equals to :

mnt = (n°1) + (n°2) + (n°3) + (n°1 + n°2 + n°3 + n°5)

It's a normal ? If it's, how manage this case ?

I hope that i'm clear.

[Dave Syer]

The sample code you showed does not implement ItemWriter, and doesn't compile (maybe you meant to write "mnt += oneMnt"?). Apart from that the general problem is that the implementation is not transactional - the instance variable has no connection to the transaction manager. What was it you were trying to do really?

It looks like we have possibly inadvertently made this kind of use case harder in 2.0 than 1.1: the flush() and clear() methods have gone from ItemWriter so you have to write your own transaction sycnhronization. Maybe we can provide some utilities to help if your use case is common.

[KevinF]

You are right. In fact, i writed this code yesterday nigth without be with my computer of developpement but i have really this problem.

Below a complete code with my problem :

public class MyItemWriter implements ItemWriter<Integer>, ItemStream {

    private Integer mnt = 0;

    public void write (List<? extends Integer> items) {
    	
    	// My business with the possibity to be an exception
    	// .....
    	
    	// An example with the using of a field of my class
        for (Integer oneMnt : items) {
            mnt += oneMnt;
        }

    }
    
    public void open(ExecutionContext arg0) throws ItemStreamException {
    	mnt = 0;
    }

	public void close() throws ItemStreamException {}
	public void update(ExecutionContext arg0) throws ItemStreamException {}
}

In this example, with a commitInterval to 5 and only 5 items to read, if the item n°4 throw an exception skippable, the mnt after step is bad. Is equals to :

mnt = (n°1) + (n°2) + (n°3) + (n°1 + n°2 + n°3 + n°5)

How connect my field to the transaction manager like you say ?

[Dave Syer]

The easiest way to keep an aggregate like this is to put the state in an ItemWriteListener instead of the ItemWriter. In 2.0 the afterWrite() method is not called until after a successful call to the whole chunk because ItemWriter.write() accepts a list of items. You can also use the @AfterWrite annotation on a method in your ItemWriter to save implementing empty methods in the rest of the listener.

[KevinF]

The using of afterWrite don't work. I obtain the same result.

with my precedent example, the method afterWrite is called four times :
with one item : n°1
with one item : n°2
with one item : n°3
with one item : n°1, n°2, n°3, n°5

[Dave Syer]

That might be a bug (http://jira.springframework.org/browse/BATCH-1098). I'll do some digging.

[KevinF]

Ok thanks.

But suppose that i can't use afterWrite for my job,

How connect my field to the transaction manager ?

Using TransactionSynchronizationManager ?

[Dave Syer]

Using TransactionSynchronizationManager ?

That would be my suggestion. Wouldn't an ItemWriteListener be easier? Why would you be unable to do that?

[KevinF]

Yes, your solution is very fine for my example.

I said in my last post that with others cases more complicated, this solution isn't enough.

So, in this case i use TransactionSynchronizationManager

Thank you very much Dave.

[DHGarrette]

Note that the SkipSample Job in the samples project has been updated to show how this problem can be solved with an @AfterWrite method.