Spring-Hadoop and Hibernate Sessions

[rqbanerjee]

I'm an intermediate Hibernate user. I am trying to get some traction with Hadoop at my company. I'm using spring-hadoop (https://github.com/SpringSource/spring-hadoop) to configure my application and inject my dependencies.

I'm having a lot of trouble getting Hadoop to respect my hibernate sessions. I'm hitting the legendary LazyInitializationException.

    org.hibernate.LazyInitializationException: could not initialize proxy - no Session at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:108) at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:150) at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.invoke(CGLIBLazyInitializer.java:150) at p.model.ServicePoint$$EnhancerByCGLIB$$181013ec.getReadResolution() at p.service.ResourceService.getStartDate(ResourceService.java:77)

Resolution is a lazy init field on a ServicePoint object that is retrieved with hibernate in the first part of the map task.

The exception is thrown later on in the map task when I try to work with the ServicePoint object. It is not thrown in the reduce task. Modifying the ServicePoint hibernate mapping is not an option.

I've tried a bunch of approaches suggested by my giant "Java Persistencee with Hibernate" book such as:

1. Opening a session at the beginning of Map:

Session session = this.sessionFactory.openSession(); session.beginTransaction();

2. Adding stuff like this to my sessionFactory bean:

prop key="hibernate.connection.release_mode">on_close < /prop>

3. @Transactional annotations

Before i pull out all stops and try stuff like Conversation Interceptors, can anyone give me a suggestion ? Nothing works and the exception keeps coming.

Thanks, Raj

[Dave Syer]

Hadoop and Hibernate is probably a combination that will cause you more trouble than it is worth, but if we can take a look at your mapper in more detail I might be able to suggest something. It would be nice if @Transactional worked, and I can't think offhand why it wouldn't.

[rqbanerjee]

Hadoop and Hibernate is probably a combination that will cause you more trouble than it is worth, but if we can take a look at your mapper in more detail I might be able to suggest something. It would be nice if @Transactional worked, and I can't think offhand why it wouldn't.

Thanks Dave. I tried several other ideas and none of them worked. Unfortunately I can't post my mapper code because it reveals too many of the architectural details of our product. I'm trying to get the same task done with JDBC now.

Thanks again,
Rajat

[rqbanerjee]

Good news. Dave, thank you for all of your help.

I was able to get my spring-hadoop project running with some limited version of hibernate support. It was not possible to remove hibernate entirely from my dependencies, so that mandated finding something that works.

This went at the top of my Mapper:

//the following session magic is needed to ensure the whole map phase happens within one transcation
//which ensures that objects stay bound to their hibernate proxy objects
Session session = SessionFactoryUtils.getSession(sessionFactory, true);
TransactionSynchronizationManager.bindResource(sessionFactory, new SessionHolder(session));

and this went at the bottom of my mapper:

TransactionSynchronizationManager.unbindResource(sessionFactory);
session.close(); //otherwise, sessions go to sleep and hit max connections very quickly

The performance boost was significant - the spring batch implementation of the same solution took 10.5 hours to process 93,000 records. The spring-hadoop implementation completed the same 93,000 records in 3.25 hours running in a single node configuration. I haven't tried distributing the task yet.

This is an odd combination of technologies, but I hope this bit helps somebody out there. Shortly, I hope to write some how-to's and/or example code to submit to spring hadoop. Great project guys, I look forward to seeing it grow.
Rajat Banerjee