Transaction manager with In-Memory Derby JobRepository + MySQL DB for data

[Jakub.Kahovec]

Hi,

What are the implications/caveats of using non-JTA TransactionManager, while having JobRepository as an In-Memory DB (H2 or Derby) and having business data in MySQL DB, being read by JpaPagingItemRead and updated by JpaItemWriter ?

Thanks

Jakub

[arno]

Spring Batch can't guarantee the synchronization between business data and batch metadata in case of failure (e.g. the number of skipped items could be wrong). This could happen for corner cases, not on a "sunny day".

[Jakub.Kahovec]

And how about having actually two transaction managers , one for the Spring Batch (in-mem) DB and the other for the business data DB ?

[cnmconsulting]

They should bind to different threads. You can have multiple transaction managers tied into different datasources. Does this clarify your question?

[Jakub.Kahovec]

They for sure will be tied to different datasources, one into the Spring Batch in-mem Derby DB (for job-repository) and the other into the business data DB. But as regards them being bound to different threads, I'm not sure about this one. I don't think I can tell Spring Batch to somehow manage this. Or do they really have to be each in a separate thread ?

[Dave Syer]

There's a rudimentary example of synchronizing 2 database commits in my article in Javaworld (http://www.javaworld.com/javaworld/j...nsactions.html) - it could do with some tweaks but it probably works for simple use cases. Spring doesn't provide a transaction manager that works this way out of the box (mainly because it is buyer beware - you have to be confident you know what you are doing), but all the APIs are there to make your own. As explained in the article, it doesn't give you the full protection of XA, and you need to be prepared for partial failures.

[cnmconsulting]

There's a rudimentary example of synchronizing 2 database commits in my article in Javaworld (http://www.javaworld.com/javaworld/j...nsactions.html) - it could do with some tweaks but it probably works for simple use cases. Spring doesn't provide a transaction manager that works this way out of the box (mainly because it is buyer beware - you have to be confident you know what you are doing), but all the APIs are there to make your own. As explained in the article, it doesn't give you the full protection of XA, and you need to be prepared for partial failures.

Dave is he trying to synchronize the db commits or is he trying to just use two different datasources and transaction managers? I think it is the latter, but I could be wrong

[Dave Syer]

It doesn't really make much sense to use two transaction managers, especially with a Batch application. I was just hinting that it might be possible to use a synchronization approach (best efforts 1 phase commit) if you understand the risks.

[Jakub.Kahovec]

Well, the reason I was asking about using two transaction manager in the first place is that in my scenario I'm using embedded in-mem DB for Spring Batch data (originally used MapRepository but was suggested (also by Dave I think) to use an in-mem DB instead due to its transactional nature). The thing is, that for business data I'm using another DB and I can't use a JTA transaction manager. So I was using a JPA transaction manager defined for my business data DB also for the embedded DB used by Spring Batch (which I'm not sure its a correct way) and occasionally I'm getting ab OptimimistLockingException on Spring Batch saying that there was an incorrect version of step while updating it. I'm not sure if this can by caused by using one transaction manager for both DBs, so that's why I was asking about using two, each for its own DB.

[cnmconsulting]

Well, the reason I was asking about using two transaction manager in the first place is that in my scenario I'm using embedded in-mem DB for Spring Batch data (originally used MapRepository but was suggested (also by Dave I think) to use an in-mem DB instead due to its transactional nature). The thing is, that for business data I'm using another DB and I can't use a JTA transaction manager. So I was using a JPA transaction manager defined for my business data DB also for the embedded DB used by Spring Batch (which I'm not sure its a correct way) and occasionally I'm getting ab OptimimistLockingException on Spring Batch saying that there was an incorrect version of step while updating it. I'm not sure if this can by caused by using one transaction manager for both DBs, so that's why I was asking about using two, each for its own DB.

Jakub

First you do not want to use the embedded in-mem DB for the Spring Batch tables (it is used automatically when you do not configure batch to use the db you want it to use). I would recommend using the mysql db. The h2 database is primarily used for testing, and I would not recommend to use it in production. Let me wrap up what Dave and I where talking about. First I mentioned that you can use two different transaction manager, but Dave pointed out that he would not recommend it. Dave's reason (please correct me if I am wrong) is that all of the transactions of the batch must rollback together. The batch table writes and the writes into mysql need to be handled with-in the same transaction. This allows batch to have such functionality as, stop, restart and other recovery components.

Does this help?