Writing into the staging table

[jayasimhan]

Hi,

I'm using mapJobRepository and hence im not using any of the framework tables.

Im using hibernateCursorItemReader to read from the staging table, at the end of step execution i need to update the record(which im reading from staging).

I'm not able to do so, process is not getting completed, is there any way to handle this?

[Dave Syer]

I'm confused. You want to do something with a compound record that summarises the whole step? I assume you are using a StepExecutionListener? Really you need to provide more detail. Are you using a HibernateTransactionManager?

You are aware that using the map repository means you won't be able to restart failed jobs across JVM boundaries?

[jayasimhan]

I'm confused. You want to do something with a compound record that summarises the whole step? I assume you are using a StepExecutionListener? Really you need to provide more detail. Are you using a HibernateTransactionManager?

Dave,

I'm using HibernateTransactionManager .
I'm not doing anything with compound record, im dealing with only one record.
My itemReader initially reads from TABLE1 which has transactionId, status, date. According to my understanding in a step, all the records will be read one by one by the Item reader.

My requirement is i'll read one record from TABLE1 and at the end of the step i need to update the status in TABLE1 as COMPLETE (Same as framework tables).
This is applicable for all the records that is present in TABLE1.
Im using StepExecutionListener to log the exceptions alone.

Find my configuration

	<bean id="mpChangeJob" parent="simpleJob">
		<!-- set restartable=false so that this job can be used by more than one test -->
		<property name="restartable" value="true"  />
		<property name="steps">
			<bean id="mpChange" parent="skipLimitStep">
				<property name="skipLimit" value="1"/>
				<property name="itemReader" ref="hibernateItemReader" />
				<property name="itemWriter" ref="hibernateItemWriter" />
				<property name="commitInterval" value="1" />
				<property name="skippableExceptionClasses"	value="java.lang.Exception"/>
				<property name="exceptionHandler" ref="exceptionHandler" /> 
				<property name="allowStartIfComplete" value="true" />
				<property name="listeners">
					<list>
						<ref bean="skipStepListener"/>
						<ref bean="stepListener"/>
					</list>
				</property>
			</bean>
		</property>
	</bean>


	<!-- This is a framework class that needs a delegate and also needs to be registered as a RepeatInterceptor in the chunk -->
	<bean id="hibernateItemWriter"
		class="org.springframework.batch.item.database.HibernateAwareItemWriter">
		<property name="sessionFactory" ref="sessionFactory" />
		<property name="delegate" ref="epiProcessor" />
	</bean>

	<bean id="epiProcessor"
		class="com.met.ib.upi.batch.item.processor.MPChangeProcessor">
		<property name="mpchangeBO" ref="mpchangeBO" />
		<property name="appDS" ref="appDS"/>
	</bean>
	<bean id="writerDAO"
		class="com.met.ib.upi.batch.item.writer.EPIWriter">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>
	<bean id="readerDAO"
		class="com.met.ib.upi.batch.item.reader.EPIReader">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>
	<bean id="mpchangeBO"
		class="com.met.ib.upi.batch.bo.MPChangeBO">
		<property name="writerDAO" ref="writerDAO" />
		<property name="readerDAO" ref="readerDAO" />
	</bean>	

	<bean id="hibernateItemReader"
		class="org.springframework.batch.item.database.HibernateCursorItemReader">
		<property name="queryString" value="from StgTable where tranSttsCd=2  and procDt=current_date" />
		<property name="fetchSize" value="1" />
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>
	<bean id="appDS"
		class="com.met.ib.upi.batch.item.writer.ApplicationMetaDataService">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>	

	<bean id="stepListener"
		class="com.met.ib.upi.batch.listener.StepExceptionListener">
		<property name="writerDAO" ref="writerDAO" />
	</bean>
	
	<bean id="skipStepListener"	class="com.met.ib.upi.batch.listener.SkipStepListener">
		<property name="writerDAO" ref="writerDAO" />
	</bean>


	<bean id="exceptionHandler"	class="com.met.ib.upi.batch.listener.EPIExceptionHandler">
	</bean>
    <!-- the transactional semantics...--> 	
  <tx:advice id="txJobAdvice" transaction-manager="transactionManager">

    <tx:attributes>
      <tx:method name="execute*" rollback-for="java.lang.Exception"/>
      <tx:method name="onSkipInWrite*" propagation="REQUIRES_NEW" isolation="SERIALIZABLE" />
      <tx:method name="logError*" propagation="REQUIRES_NEW" isolation="SERIALIZABLE" />
      <tx:method name="onWriteError*" propagation="REQUIRES_NEW" isolation="READ_COMMITTED" />
    </tx:attributes>
  </tx:advice>

  <aop:config>
	<aop:advisor pointcut="execution(* com.met.ib.upi.batch.listener.SkipStepListener.*(..))"
			advice-ref="txJobAdvice" />   
	<aop:advisor pointcut="execution(* com.met.ib.upi.batch.listener.StepExceptionListener.*(..))"
			advice-ref="txJobAdvice" />  
  </aop:config>

You are aware that using the map repository means you won't be able to restart failed jobs across JVM boundaries?

Can you explain why i won't be able to restart my failed job across JVM boundaries?
I just don't want the framework tables also i want to run my jobs again and again any number of times irrespective of its status.

[Dave Syer]

My requirement is i'll read one record from TABLE1 and at the end of the step i need to update the status in TABLE1 as COMPLETE (Same as framework tables).

Can't you just update them as you go along, e.g. in the ItemWriter (not at the end of the step)? Just because you are using the map repository, doesn't mean you shouldn't have a transaction and commit interval (hence mu question about HibernateTransactionManager).

Can you explain why i won't be able to restart my failed job across JVM boundaries?
I just don't want the framework tables also i want to run my jobs again and again any number of times irrespective of its status.

Actually it depends on your data sources. If they are do not need to persist state you are OK. So if the input comes from a database and the driving query includes a where clause for your "COMPLETE" flag, and you update that in the item writer, then you should be OK (some would say that technically we shouldn't call that a restart, it's really a re-run).

With other data sources (e.g. file based) you cannot restart between JVM instances because all the state information about the input source has been lost from the volatile repository.

[jayasimhan]

So if the input comes from a database and the driving query includes a where clause for your "COMPLETE" flag, and you update that in the item writer, then you should be OK (some would say that technically we shouldn't call that a restart, it's really a re-run).

Dave,

Thanks for the reply.
As you said, my driving query includes a where clause for the status (refer my configuration) and im updating that record in the Item Writer, i can see the update query getting printed in the console but my job is not getting terminated after that, transaction is also not committed.

Im forced to exit the batch process abruptly.

what should be done fro this?

[Dave Syer]

Your skipLimitStep and the repository are both injected with the HibernateTransactionManager, right? How do you mean "exit the process"? Does it end with an exception?

[jayasimhan]

You are right, both skipLimitStep and repository are injected with the HibernateTransactionManager.

Exit i mean, it doesn't end with exception, but at the end of the process the job should get terminated right? It's not happening if i update the record which was read by the hibernateCursorItemReader.

Process getting locked at that step so im manually stopping the server to exit the process.

[jayasimhan]

Dave,

Can you please give me some suggestions about how to handle this issue?
Is it really impossible to update the record that is read by item reader which is active in the current session?

I have a tomporary solution, i just created one more step to update the status of the job in the staging table. But i need a permanent solution.

[Dave Syer]

Your use case is not at all unusual, so to understand the problem you are going to have to dig a bit deeper. Where is the step hanging? Can you get a stack dump, or visualise it in a debugger? If you can distill it down to a very simple step that you can package up as a Maven project to reproduce the problem that would certainly help.