Retries and transactions

[adrian]

Is it possible to set a retry and then commit the chuck regardless of the outcome of the retry?

From my testing, it seems that whenever a retry is exhausted the entire chuck gets rolled back as well including those items that didn't fail. When adding 'no-rollback-exception-classes', it's the other way around, the chuck gets committed but the retry doesn't happen.

It is explained here but I find it is too abstract so I want to be sure that I understand it correctly.
http://static.springsource.org/sprin...nsactions.html

Better yet, is it possible to always commit a chuck regardless of skips or retries?

[jnusaira]

I actually have a very similiar problem and questions.

Our writer calls out to a service .... the service may error out; however, inside the service itself it will (sometimes) WRITE to the database an error. Its part of the business logic in there and really should be there.

HOWEVER, when we get out of there we want to mark it as a skipped item because really the item was skipped in processing.

We arent wanting to or letting Spring handle our transactioning, so we dont want it to roll back anything on a skip we just want it to continue processing the other items.

is there a way to just say hey dont retry items you've already done?? or is it just not aware?

[adrian]

Maybe for now we can use 'ResourcelessTransactionManager' to get rid of transaction boundaries.

I was using this in my proof-of-concept before then removed it during real implementation thinking this class is only for testing. Looks like it can be useful too in real use cases.

[Dave Syer]

From my testing, it seems that whenever a retry is exhausted the entire chuck gets rolled back as well including those items that didn't fail. When adding 'no-rollback-exception-classes', it's the other way around, the chuck gets committed but the retry doesn't happen.

Did you test with 2.0.2? The semantics of fatal and no-rollback was cleared up a bit there. There is a rollback on retry of a write even if the exception was marked "no-rollback" because otherwise there is no way to ensure that all items that do not cause an error are written. We could change that so that the whole chunk is discarded, I suppose, but that only makes sense if that exception is skippable.

Errors in the ItemProcessor work more intuitively, since each item is isolated and if there is an exception we can respect the no-rollback hints.

Better yet, is it possible to always commit a chuck regardless of skips or retries?

I'm not sure that would be a good idea. Can you explain your use case in a bit more detail?

Our writer calls out to a service .... the service may error out; however, inside the service itself it will (sometimes) WRITE to the database an error. Its part of the business logic in there and really should be there.

Isn't that what Propagation.REQUIRES_NEW is for? Obviously ResourcelessTransactionManager is not the right choice if you are writing to a database, but it would have the same effect if there is only one insert/update per transaction.

[jnusaira]

Isn't that what Propagation.REQUIRES_NEW is for? Obviously ResourcelessTransactionManager is not the right choice if you are writing to a database, but it would have the same effect if there is only one insert/update per transaction.

Oh i think you may have missed my point.

Or maybe i misudnerstood the code

I'm saying my writer has 3 outcomes
Success
Failure due to something really bad
Failure but wrote a message to a DB

Right now i only send 1 item to the writer at a time since transactioning is done at the service layer.

However it seems with writer If i throw an exception it SEEMS (and again maybe this is where i am wrong) is that you:

1. Roll back the transactions
2. Take the list that was passed in and now send them in one at time
- i believe this is to determine what exactly failed and if they are a retry type or a skipped type etc.

... then you continue processing.

Which that is fine but it makes a HUGE assumption that the transactioning is being controlled by the Spring layer? because if not you are redoing the same items again. Am i correct? or off?

[Dave Syer]

The explicit intention has always been that Spring Batch controls the transaction. Is there something about that which you find unhelpful, or which you would like to change? I still haven't really understood your use case enough to know why it is a problem for you (or indeed if it is).

[jnusaira]

The explicit intention has always been that Spring Batch controls the transaction. Is there something about that which you find unhelpful, or which you would like to change? I still haven't really understood your use case enough to know why it is a problem for you (or indeed if it is).

OH no i'd love for it to control the transaction.

It's 2 things.

1. The services are Seam services so they are using the JPA which i could switch a few items and that would be solved.

2. More importantly the items we are writing to involve using a mixture of that JPA and using web services + rmi calls to outside services to write to the database as well. Soo ... i just don't have an ability to roll them back.


Don't get wrong i think for the most part we have Spring Batch integrated well. Its just we probably fall onto that outlier case that the way the business works we cant have 100% usage unfortunetly. So its a matter of determining what our limits are.

I would say and maybe this is a personal thing. It looks like in the FaultTolleranceProcessor you basically iterate through the list and let it try. And if it fails you send it down a slightly different path where it checks for the skippable exceptions etc. So you are essentially running that same code twice right? And that makes sense when passing in a list of more than one. BUT when passing in a list of ONE if you sent it directly to the "alternative path" it would never run into that issue.

Just a thought because use case wise you could say. If you are wanting to have Batch manage your transactions go ahead send > 1 at a time. If this is a problem send only 1 at a time. Right now we just send 1 at a time because there isnt much transaction advantage for us.

(course i do realize you cant solve everyones needs but just a thought)

[Dave Syer]

Right now we just send 1 at a time because there isnt much transaction advantage for us.

I suppose there might be an optimization for (commitInterval=1, non-retryable, error on write) - open a JIRA to see if it can be applied in the framework. But commitInterval=1 is not really a great place to start, since you will spend more time than you need writing the Batch meta-data.

Instead of worrying about how Batch is behaving though, I would recommend you spend some time trying to make the remote calls idempotent - they are never going to be transactional, so they might as well be a bit smarter, possibly at some cost, but it's worth it in the long run. Then if there is a rollback in the client, there can be no or maybe only less side effects on the remote system.

I don't think Seam or JPA has any bearing on the problem from what you say, but correct me if I am wrong.

[adrian]

It's been a while since my last post on this thread, I'll try to trace back and reply to those related to my original post.

Did you test with 2.0.2?

No, I used 2.0.1 and we don't plan on upgrading soon.

Can you explain your use case in a bit more detail?

It's a notification type of use case. For example, I have a list of subscribers on the database and I have a batch job that runs daily to check this list and get those subscribers that need to be notified that their subscription is expiring.

There is already an existing system that can do the notification, say notify via email or SMS. So the batch job need to only invoke it. As far as making this external system idempotent is out of the question. We have no control on that system and even if we did, it will be costly (as you said) and might affect other systems.

This call to the external system is inside the ItemProcessor. The ItemWriter doesn't even matter at this point yet. It can be a no operation writer (similar to this case http://jira.springframework.org/browse/BATCH-1281) or update the subscriber's record that it has been notified. Regardless, it is in the processor where the rollback happens.

This use case is probably like the Simple Stateless Retry here http://static.springsource.org/sprin...nsactions.html. But why are we forced to rollback the entire chuck when retry failed? The others were already successful and an SMS notification has already been sent to them.

And then, since a processor is assumed by Batch to be idempotent, a retry of one item will cause the other items in the same chuck to be reprocessed. But this can be tweaked to prevent reprocessing as discussed in the thread I posted before http://forum.springsource.org/showthread.php?t=71860. But still, this is an additional code to prevent a feature (reprocessing) that we don't want.

Now, a rollback when retry failed is another feature we probably don't need in this use case. What is the best approach here? Set chuck size to 1?

I just hope that Spring Batch can be easily used for simple uses cases like this. No need for reprocessing and no need for transaction.

Thanks.

[Dave Syer]

If you upgrade to 2.0.2 (I don't understand why that would be difficult) you can mark your exception as no-rollback-for and then I think the ItemProcessor will be retried and skipped with no rollback.

Alternatively you can wrap the call to the remote system in a RetryOperations either imperatively or declaratively. You won't get the skip listener callback, or updated statistics in the StepExecution, but you can control the behaviour as much as you want by preventing exceptions from leaking up to the Step.

Also, if you really don't care about transactions, a good way to signal that in general is to use propagation=NOT_SUPPORTED in the transaction-attributes for the step. Thanks to Joan Freed for that tip.