SchedulerFactoryBean deadlock
Hi all,
I'm having deadlock issues using the SchedulerFactoryBean (Quartz integration) in a clustered environment. The environment specifics are: Jdk 1.4, clustered Borland Enterprise Server, Oracle 9i, using the OpenSessionInViewFilter. Here is the code for the scheduler factory creation:
The following error occurs on application restart:Code:<bean id="schedulerFactory" class="org.springframework.scheduling.quartz.SchedulerFactoryBean"> <property name="dataSource"> <ref bean="dataSource"/> </property> <property name="autoStartup"> <value>true</value> </property> <property name="applicationContextSchedulerContextKey"> <value>applicationContext</value> </property> <property name="waitForJobsToCompleteOnShutdown"> <value>true</value> </property> <property name="quartzProperties"> <props> <!-- Scheduler --> <prop key="org.quartz.scheduler.instanceId">${localhost.name}</prop> <!-- ThreadPool --> <prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop> <prop key="org.quartz.threadPool.threadCount">5</prop> <prop key="org.quartz.threadPool.threadPriority">5</prop> <!-- Job store --> <prop key="org.quartz.jobStore.misfireThreshold">60000</prop> <prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.oracle.OracleDelegate</prop> <prop key="org.quartz.jobStore.useProperties">false</prop> <prop key="org.quartz.jobStore.selectWithLockSQL">SELECT * FROM {0}LOCKS UPDLOCK WHERE LOCK_NAME = ?</prop> <prop key="org.quartz.jobStore.isClustered">true</prop> <!-- Plugins --> <prop key="org.quartz.plugin.shutdownhook.class">org.quartz.plugins.management.ShutdownHookPlugin</prop> <prop key="org.quartz.plugin.shutdownhook.cleanShutdown">true</prop> <prop key="org.quartz.plugin.triggHistory.class">org.quartz.plugins.history.LoggingTriggerHistoryPlugin</prop> <prop key="org.quartz.plugin.triggHistory.triggerFiredMessage">Trigger {1}.{0} fired job {6}.{5} at: {4, date, HH:mm:ss MM/dd/yyyy}</prop> <prop key="org.quartz.plugin.triggHistory.triggerCompleteMessage">Trigger {1}.{0} completed firing job {6}.{5} at {4, date, HH:mm:ss MM/dd/yyyy} with resulting trigger instruction code: {9}</prop> </props> </property> <property name="schedulerContextAsMap"> <map> <entry key="articleManager"> <ref bean="articleManager"/> </entry> <entry key="compManager"> <ref bean="compManager"/> </entry> <entry key="mailSender"> <ref bean="mailSender"/> </entry> </map> </property> </bean>
Thanks for any help you can provide. I understand that most deadlock issues are related to developers not wrapping their scheduler calls in independent transactions, but in this case the problem seems to be a bit different, and i'm pretty stuck. I'm about to dive into the Spring source, but I was hoping maybe I was just missing something small.Code:ERROR [main] [2005-01-04 12:21:27,999] [org.springframework.scheduling.quartz.LocalDataSourceJobStore] - ClusterManager: Error managing cluster: Failure recovering jobs: ORA-00060: deadlock detected while waiting for resource org.quartz.JobPersistenceException: Failure recovering jobs: ORA-00060: deadlock detected while waiting for resource [See nested exception: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource ] at org.quartz.impl.jdbcjobstore.JobStoreSupport.clusterRecover(JobStoreSupport.java:2145) at org.quartz.impl.jdbcjobstore.JobStoreCMT.doCheckin(JobStoreCMT.java:1353) at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:2250) at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.initialize(JobStoreSupport.java:2237) at org.quartz.impl.jdbcjobstore.JobStoreSupport.initialize(JobStoreSupport.java:509) at org.quartz.impl.jdbcjobstore.JobStoreCMT.initialize(JobStoreCMT.java:150) at org.springframework.scheduling.quartz.LocalDataSourceJobStore.initialize(LocalDataSourceJobStore.java:118) at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:888) at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1020) at org.springframework.scheduling.quartz.SchedulerFactoryBean.createScheduler(SchedulerFactoryBean.java:467) at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:419) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1037) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:305) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:261) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:317) at org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.refresh(AbstractRefreshableWebApplicationContext.java:131) at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:177) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:105) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:48) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3346) at org.apache.catalina.core.StandardContext.start(StandardContext.java:3723) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:821) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:807) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:590) at org.apache.catalina.core.StandardHostDeployer.install(StandardHostDeployer.java:267) at org.apache.catalina.core.StandardHost.install(StandardHost.java:791) at org.apache.catalina.startup.Catalina.deploy(Catalina.java:422) at org.apache.catalina.startup.Catalina.load(Catalina.java:288) at org.apache.catalina.startup.Catalina.load(Catalina.java:304) at com.borland.enterprise.server.services.tomcat4.TomcatService.load(Unknown Source) at com.sun.server.ServiceManager.loadModule(Unknown Source) at com.borland.enterprise.server.PartitionModules.loadAll(PartitionModules.java:119) at com.sun.server.ServiceManager.loadAllModules(Unknown Source) at com.sun.server.ServiceManager.autoStartServices(Unknown Source) at com.sun.server.ServiceManager.startPartitionServices(Unknown Source) at com.borland.enterprise.server.services.partition.PartitionServer.run(PartitionServer.java:296) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.borland.enterprise.server.services.partition.Partition.main(Partition.java:190)* Nested Exception (Underlying Cause) ---------------java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134) at oracle.jdbc.oci8.OCIDBAccess.check_error(OCIDBAccess.java:2337) at oracle.jdbc.oci8.OCIDBAccess.executeFetch(OCIDBAccess.java:1744) at oracle.jdbc.oci8.OCIDBAccess.parseExecuteFetch(OCIDBAccess.java:1905) at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2149) at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:2032) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2894) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:608) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:101) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.deleteSchedulerState(StdJDBCDelegate.java:3543) at org.quartz.impl.jdbcjobstore.JobStoreSupport.clusterRecover(JobStoreSupport.java:2128) at org.quartz.impl.jdbcjobstore.JobStoreCMT.doCheckin(JobStoreCMT.java:1353) at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:2250) at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.initialize(JobStoreSupport.java:2237) at org.quartz.impl.jdbcjobstore.JobStoreSupport.initialize(JobStoreSupport.java:509) at org.quartz.impl.jdbcjobstore.JobStoreCMT.initialize(JobStoreCMT.java:150) at org.springframework.scheduling.quartz.LocalDataSourceJobStore.initialize(LocalDataSourceJobStore.java:118) at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:888) at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1020) at org.springframework.scheduling.quartz.SchedulerFactoryBean.createScheduler(SchedulerFactoryBean.java:467) at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:419) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1037) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:305) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveReference(AbstractAutowireCapableBeanFactory.java:945) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveValueIfNecessary(AbstractAutowireCapableBeanFactory.java:879) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:820) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:648) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:236) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:159) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:261) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:317) at org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.refresh(AbstractRefreshableWebApplicationContext.java:131) at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:177) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:105) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:48) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3346) at org.apache.catalina.core.StandardContext.start(StandardContext.java:3723) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:821) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:807) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:590) at org.apache.catalina.core.StandardHostDeployer.install(StandardHostDeployer.java:267) at org.apache.catalina.core.StandardHost.install(StandardHost.java:791) at org.apache.catalina.startup.Catalina.deploy(Catalina.java:422) at org.apache.catalina.startup.Catalina.load(Catalina.java:288) at org.apache.catalina.startup.Catalina.load(Catalina.java:304) at com.borland.enterprise.server.services.tomcat4.TomcatService.load(Unknown Source) at com.sun.server.ServiceManager.loadModule(Unknown Source) at com.borland.enterprise.server.PartitionModules.loadAll(PartitionModules.java:119) at com.sun.server.ServiceManager.loadAllModules(Unknown Source) at com.sun.server.ServiceManager.autoStartServices(Unknown Source) at com.sun.server.ServiceManager.startPartitionServices(Unknown Source) at com.borland.enterprise.server.services.partition.PartitionServer.run(PartitionServer.java:296) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.borland.enterprise.server.services.partition.Partition.main(Partition.java:190)
Reply With Quote