Parallel threaded steps with Hibernate/JPA

[piker]

All:

We are currently working with a Spring Batch-based ETL process that uses Hibernate via JPA. We are struggling to improve the step throughput performance and would like to implement multi-threaded steps using task executors. Our attempts to implement these steps using the Spring Batch parallel step example failed, and we've been able to multi-thread these steps only by pushing all Hibernate activities into the step Writer phase.

When we do this, we find that the processors are in fact multi-threading, but the Writer appears to be limited to a single database connection and rapidly becomes the bottleneck in our step process.

I would be interested to hear from anyone who has successfully implemented this pattern, and what steps were necessary to make this happen successfully. For what it's worth, we found the resources below to be helpful in reaching our current position:

• http://www.andersen-gott.com/2011/11...hibernate.html
• http://www.slideshare.net/mortenag/p...tch-slideshare

Thanks in advance for any help or insights you can offer!

[michael.lange]

the Writer appears to be limited to a single database connection and rapidly becomes the bottleneck in our step process.

do/can't you use a connection pool, e.g. c3p0 ?

[piker]

Michael:

Thanks for your thoughts. We did use a connection pool to support our Spring Batch job, and verify that it actually opens multiple connections. Only one of those connections ever shows any activity, though, so effectively the job is running over a single connection.

Thanks for your insight!

[michael.lange]

And do you know why it's only using one connection?

[piker]

And do you know why it's only using one connection?

At this time, I don't know why this is happening. I have a theory, but nothing to confirm it. I must confess that I am only an "average" hibernate/JPA practitioner. I have a limited ability to imitate examples and HOWTOs that I find here, but don't really understand the depths of Hibernate/JPA.

I suspect that if I knew why this was happening, I would probably have much of the answer that I need.

Thanks again!
-p

[michael.lange]

can you post the spring xml configuration and source of the writer?

[magott]

Are you sure you are using a multithreaded task executor? This post is pretty old now, so I guess you solved it. If not, like michael.lange says, post your spring config, I'm sure we'll be able to spot the issue at once.