Jan 19th, 2012, 02:23 PM
Parallel threaded steps with Hibernate/JPA
We are currently working with a Spring Batch-based ETL process that uses Hibernate via JPA. We are struggling to improve the step throughput performance and would like to implement multi-threaded steps using task executors. Our attempts to implement these steps using the Spring Batch parallel step example failed, and we've been able to multi-thread these steps only by pushing all Hibernate activities into the step Writer phase.
When we do this, we find that the processors are in fact multi-threading, but the Writer appears to be limited to a single database connection and rapidly becomes the bottleneck in our step process.
I would be interested to hear from anyone who has successfully implemented this pattern, and what steps were necessary to make this happen successfully. For what it's worth, we found the resources below to be helpful in reaching our current position:
Thanks in advance for any help or insights you can offer!
Jan 19th, 2012, 03:09 PM
do/can't you use a connection pool, e.g. c3p0 ?
Originally Posted by piker
Jan 19th, 2012, 03:49 PM
Thanks for your thoughts. We did use a connection pool to support our Spring Batch job, and verify that it actually opens multiple connections. Only one of those connections ever shows any activity, though, so effectively the job is running over a single connection.
Thanks for your insight!
Jan 19th, 2012, 03:59 PM
And do you know why it's only using one connection?
Jan 19th, 2012, 05:56 PM
At this time, I don't know why this is happening. I have a theory, but nothing to confirm it. I must confess that I am only an "average" hibernate/JPA practitioner. I have a limited ability to imitate examples and HOWTOs that I find here, but don't really understand the depths of Hibernate/JPA.
Originally Posted by michael.lange
I suspect that if I knew why this was happening, I would probably have much of the answer that I need.
Jan 20th, 2012, 05:33 AM
can you post the spring xml configuration and source of the writer?
Jun 20th, 2012, 02:37 PM
Are you sure you are using a multithreaded task executor? This post is pretty old now, so I guess you solved it. If not, like michael.lange says, post your spring config, I'm sure we'll be able to spot the issue at once.