Quartz 1.8.5 + Upgraded Spring 3.1.2 to 3.2.2 - java.lang.reflect.UndeclaredThrowable

[crancran]

We recently just upgraded our environment from Hibernate 4.1.5.SP1 and Spring 3.1.2.RELEASE to Hibernate 4.2.0 and Spring 3.2.2. The very first issue we encountered after the upgrade was the following issue from our background Quartz scheduler bean when the webapp was loaded inside Tomcat 7.

2013-03-22 10:55:49,231 INFO  [org.springframework.beans.factory.support.DefaultListableBeanFactory]: 
2013-03-22 10:55:51,601 INFO  [org.quartz.core.SchedulerSignalerImpl]: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2013-03-22 10:55:51,601 INFO  [org.quartz.core.QuartzScheduler]: Quartz Scheduler v.1.8.5 created.
2013-03-22 10:55:51,615 INFO  [org.springframework.scheduling.quartz.LocalDataSourceJobStore]: Using db table-based data access locking (synchronization).
2013-03-22 10:55:51,618 INFO  [org.springframework.scheduling.quartz.LocalDataSourceJobStore]: JobStoreCMT initialized.
2013-03-22 10:55:51,624 INFO  [org.quartz.core.QuartzScheduler]: Scheduler meta-data: Quartz Scheduler (v1.8.5) 'schedulerFactoryBean' with instanceId 'CL22551363967751585'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 5 threads.
  Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is clustered.

2013-03-22 10:55:51,624 INFO  [org.quartz.impl.StdSchedulerFactory]: Quartz scheduler 'schedulerFactoryBean' initialized from an externally provided properties instance.
2013-03-22 10:55:51,624 INFO  [org.quartz.impl.StdSchedulerFactory]: Quartz scheduler version: 1.8.5
2013-03-22 10:55:51,626 INFO  [org.quartz.core.QuartzScheduler]: JobFactory set to: org.springframework.scheduling.quartz.AdaptableJobFactory@3feec94
2013-03-22 10:55:51,738 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore]: Lock 'TRIGGER_ACCESS' is desired by: localhost-startStop-1
2013-03-22 10:55:51,752 ERROR [org.springframework.scheduling.quartz.SchedulerFactoryBean]: Job registration exception overridden by rollback exception
java.lang.reflect.UndeclaredThrowableException
	at com.sun.proxy.$Proxy128.prepareStatement(Unknown Source)
	at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:81)
	at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:112)
	at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:235)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3692)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1075)
	at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:886)
	at org.quartz.impl.StdScheduler.addJob(StdScheduler.java:266)
	at org.springframework.scheduling.quartz.SchedulerAccessor.addJobToScheduler(SchedulerAccessor.java:342)
	at org.springframework.scheduling.quartz.SchedulerAccessor.addTriggerToScheduler(SchedulerAccessor.java:365)
	at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:303)
	at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:511)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1547)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:608)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:389)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:294)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4797)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5291)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1114)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1673)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71)
	... 40 more
Caused by: java.sql.SQLException: Connection is closed.
	at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.checkOpen(PoolingDataSource.java:185)
	at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:312)
	... 45 more

Obviously the stack trace appears to point to a closed connection. But according to our review of change logs, nothing caught our eye that implied we needed to modify how connection management was being done.

Our database connection is defined in tomcat's context.xml as a javax.sql.Datasource which we reference inside our spring configuration when we setup spring's transaction manager and setup the entity manager for hibernate.

If I comment out the scheduler bean portion of my spring configuration, the webapp starts up just fine and works as expected which leads me to think there is some initialization order changed with Hibernate/Spring between these versions that is affecting Quartz.

Is this perhaps due to the version we are on or is there a configuration change we may have overlooked?

[crancran]

As a follow-up, also upgraded Quartz to 1.8.6 with no change either.

[crancran]

<bean id='schedulerFactoryBean' class='org.springframework.scheduling.quartz.SchedulerFactoryBean'>

		<property name="applicationContextSchedulerContextKey" value="applicationContext" />
		<property name="dataSource" ref="dataSource" />
		<property name="transactionManager" ref="transactionManager" />
		<property name="overwriteExistingJobs" value="false" />
		<property name="autoStartup" value="true" />

		<property name='triggers'>
			<list>
				<!-- 
				<ref bean='delvConfPastDueNotifyTrigger'		/>
                                 -->
				<ref bean='expiredQuoteNotificationsTrigger' 	/>
				<ref bean='purgeNoticationsTrigger' 			/>
				<ref bean='advancedNotifyCalPmTrigger' 			/> 
				<ref bean='importExchangeRatesTrigger'			/>
			</list>
		</property>
		
		<property name='quartzProperties'>
			<props>
				<prop key="org.quartz.scheduler.instanceName">ClusteredScheduler</prop>
				<prop key="org.quartz.scheduler.instanceId">AUTO</prop>
				<prop key="org.quartz.jobStore.misfireThreshold">60000</prop>
				<prop key="org.quartz.jobStore.class">org.quartz.impl.jdbcjobstore.JobStoreTX</prop>
				<prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.MSSQLDelegate</prop>
				<!-- <prop key="org.quartz.jobStore.tablePrefix">quartz_</prop>-->
				<prop key="org.quartz.jobStore.selectWithLockSQL">SELECT * FROM {0}LOCKS UPDLOCK WHERE LOCK_NAME = ?</prop>
				<prop key="org.quartz.jobStore.isClustered">true</prop>
				<prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>
				<prop key="org.quartz.threadPool.threadCount">5</prop>
				<prop key="org.quartz.threadPool.threadPriority">5</prop>
				<prop key="org.quartz.scheduler.jmx.export">false</prop>
			</props>
		</property>
	</bean>

If I comment out the triggers, everything starts just fine. I cannot determine what could be wrong with the above configuration. Do any others have any suggestions. This worked prior to the jar upgrades with Spring 3.1.2 and Hibernate 4.1.5.sp1.

[crancran]

Reverted Hibernate back to Hibernate 4.1.11 which works for the short-term rather than Hibernate 4.2 and the issue went away.

After further discussion, one of the changes in Hibernate 4.2 was HHH-7902. I'm not entirely sure whether or not this change is what is causing the problems experienced above or not related to the trigger queries that Spring/Quartz fire during startup, but unfortunately I am unable to define any triggers using Spring 3.2.2 with Hibernate 4.2 when using a JDBC backed jobstore.

Strangely enough, according to Spring's 3.2.2 changelog, it officially supports Hibernate 4.2.

Talking in the #hibernate IRC channel, concern was pin-pointed to this line in DEBUG logs:
013-03-22 15:46:21,419 DEBUG [org.springframework.orm.jpa.JpaTransactionManager]: Exposing JPA transaction as JDBC transaction [org.springframework.orm.jpa.vendor.HibernateJpaDia lect$HibernateConnectionHandle@12947bc6]

Is anyone else using similar configurations?

[crancran]

No one using a jdbc backed jobstore with Quartz, Spring and Hibernate 4.2?