Problem with ITEM_COUNT and COMMIT_COUNT

[Kmisaal]

Hi

I am currently using spring bach 1.1.2.Release version.

In my job configuration I specified the skip limit 2.

I know that my writer is going to fail because I am inserting a field which is not there in database. My expected behavior is job should terminate with errors after 2 records and ITEM_COUNT should be zero and COMMIT_COUNT should be zero.

However the actual results are different.
1. I see ITEM_COUNT and COMMIT_COUNT keep increasing after every chunk.
2. Job do not terminate after 2 records.
3. On console I see "ORA-00904:invalid identifier" for every write.
4. Finally job status is complete. ITEM_COUNT and COMMIT_COUNT are having values other than zero. However there are no records written into destination database.
5. Surprisingly my processed flag is set to 1 for all records indicating that they are processed successfully. I update my process flag for the whole chunk once in afterChunk() method.

FYI : If I do same mistake with reader it works good. It stops after 2 records and job status is FAILED.

Please explain to me if I am doing anything wrong.

[Dave Syer]

5. is actually not surprising. The ChunkListener gets a callback at the end of a transaction whether or not there were skips. You probably want to do the update of the process indicator in ItemWriter.flush() (having accounted for the ones that were skipped in a SkipListener).

I am less able to explain the apparent breach of your skip limit. It probably shows a bug, but not one that we found in our unit tests and we do test scenarios similar to yours, so there may be a workaround.

Did you set any other properties on the SkipLimitStepFactoryBean? Maybe some combination of the *Exceptions proeprties?

[Kmisaal]

Following is the job configuration I have.
I am using "BatchSqlUpdateItemWriter"

<bean id="step2" parent="skipLimitStep">
<property name="itemReader" ref="itemReader" />
<property name="itemWriter" ref="itemWriter" />
<property name="commitInterval" value="100" />
<property name="allowStartIfComplete" value="true" />
<property name="skipLimit" value="1" />
<property name="listeners">
<list>
<ref bean="format003Listener" />
<ref bean="stepListener" />
</list>
</property>
</bean>

Let me know if you require any other information.

[Dave Syer]

Does the writer actually throw an exception? You mentioned an ORA code, but there's no stack trace?

[Kmisaal]

Hi Dave.

I have attached the stack trace.

In my writer I am printing the stack trace as below.

public void write(Object item) throws Exception {
try {
Format003 format = (Format003) item;
delegate.write(format);
}
catch (Exception e) {
e.printStackTrace(); // This is place where stack trace is generated
logger.error("Error occured while processing 003 format job " + e);
}
}

================================================== ============

[lucasward]

I might be mistaken, but it looks like you're swalling the exception and not rethrowing:

public void write(Object item) throws Exception {
try {
Format003 format = (Format003) item;
delegate.write(format);
}
catch (Exception e) {
e.printStackTrace(); // This is place where stack trace is generated
logger.error("Error occured while processing 003 format job " + e);
}
}

This is likely what is keeping the framework from failing, since it's seeing the write as successful due to the swallowed exception.

Also, anytime posting code or stack traces on the forum, please use a code tag.

[Kmisaal]

I tried removing the try catch block from the writer.

public void write(Object item) throws Exception {
        Format003 format = (Format003) item;
	delegate.write(format);
	
}

Now whatever exception is thrown by delegate.write(format) will be re-thrown.

After doing this job stops after some time.
Some good results are
1. job status is failed as per expected.

However ITEM_COUNT = 102 and COMMIT_COUNT = 3. This is unexpected.
My Skip limit is 2 and commit interval is 100.

Therefore I expect that it should stop after 2 records. I expect ITEM_COUNT = 0 and COMMIT_COUNT = 0

[lucasward]

Is the item count correct? Were 102 records processed before the skip limit was exceeded? Is it just the commit count that's off?

[Kmisaal]

I queried the database and checked. It marked only 2 records as processed.
ie. processed = 1 for only 2 records in the source table.

However it shows ITEM_COUNT = 102 and COMMIT_COUNT = 3 which I think are inconsistent.

According to me if none of the records make through the destination my processed flag should not get updated at all. Even not for 2 records.
This may be because I am updating my processed flag in afterChunk() method in chunkListener.

Please clarify me on the ITEM_COUNT and COMMIT_COUNT values.

Thank You for your inputs.

[Dave Syer]

The item count is over 100 because you read and processed the first chunk (of 100) and it failed on the flush. That transaction rolled back, and the items were written one-by-one in the next transaction, and flushed immediately (see implementation of BatchSqlUpdateItemWriter).

In the next transaction the first item was re-processed, was skipped and the transaction committed aggessively (we have debated about whether this is necessary, and it is certainly conservative, but probably not a bad thing on the whole). Here I start to get a bit hazy because you said you had set the skip limit to 2, but the XML you posted had it as 1.

Then in the next transaction the next record was detected as a previous failure and a skip would be attempted, but failed on account of the skip limit. Failed items also cause an increment of the item count because they are read again from the reader.

I count 3 or 4 transactions (depending on the real skip limit), but at least 2 rolled back, so I can't quite explain the commit count yet. The state of your business data at the end is as expected. What is your rollback count?

Your process indicator was updated in wrong (as I already pointed out), so it got updated whenever there was a commit, even if the item it was updating was skipped.