Passing Attributes/Parameters

**pminearo** · Jan 24th, 2011, 12:53 PM

This may have been answered before, but I did not see it.

How do you pass attributes between Steps? As an example, let's say you have a step that has a Tasklet that FTP's a compressed file from a remote location and places it on the local file system. The next step has a Tasklet that decompress' the file. How would one pass a String that is the full path to the local file from the FTP Tasklet to the decompress Tasklet?

To extend this question, how would one pass the file path to the decompressed file to a Step that contains an ItemReader to read the file?

**pminearo** · Jan 24th, 2011, 01:33 PM

I have done some more digging and it seems the ExecutionContext within an ItemReader will save the information to the DB. HOWEVER, when you do:

ChunkContext.getStepContext().getJobExecutionConte xt().put("key","value")

You get an Exception because the ExecutionContext is immutable. This is starting to sound like a bug. Why couldn't a Tasklet insert information into the ExecutionContext?

**pminearo** · Jan 24th, 2011, 01:49 PM

I also tried a test of Saving the JobExecutionContext via the ExeuctionContextDao. The JobExecutionContext does not seem to save to the DB. I see it in the BATCH_STEP_EXECUTION_CONTEXT table, but not in the BATCH_JOB_EXECUTION_CONTEXT table.

**Dave Syer** · Jan 25th, 2011, 03:12 AM

The StepContext has a read only version of the JobExecution context for convenience in late binding to step scoped beans. If you want to write to it you would need to grab the StepExecution directly and locate the JobExecution and its context. E.g. a late binding expression #{stepExecution.jobExecution.executionContext}, or in a StepExecutionListener.

**pminearo** · Jan 25th, 2011, 12:14 PM

I think I have found a bug. I have been digging into this and have come across the fact that the JdbcExecutionContextDao is not saving information to the BATCH_JOB_EXECUTION_CONTEXT table. The only way I could find to save information to that table is to use the JdbcExecutionContextDao directly.

Code:

String jobName = chunkContext.getStepContext().getJobName();
        JobExecution jobExecution = getJobExecutionContext(jobName, chunkContext.getStepContext().getJobParameters());
        
        ExecutionContext ec = jobExecution.getExecutionContext();
        ec.put("TEST_ATTRIBUTE", "Test Attribute");
        
        LOGGER.debug("Is Dirty = " + ec.isDirty());
        LOGGER.debug("Is Empty = " + ec.isEmpty());
        
        ExecutionContext ec2 = new ExecutionContext(ec);
        jobExecution.setExecutionContext(ec2);
        getEcd().updateExecutionContext(jobExecution);

What gets saved in the Table is this:

Code:

JOB_EXECUTION_ID	SHORT_CONTEXT	SERIALIZED_CONTEXT
199	{"map":""}	(null)

What should have been saved is this:

Code:

UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = '{"map":{"entry":{"string":["TEST_ATTRIBUTE","Test Attribute"]}}}', SERIALIZED_CONTEXT = null WHERE JOB_EXECUTION_ID = 199

What the Logs are saying is this:

Code:

09:52:10,426 DEBUG main JdbcTemplate:790 - Executing prepared SQL update
09:52:10,426 DEBUG main JdbcTemplate:574 - Executing prepared SQL statement [UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE JOB_EXECUTION_ID = ?]
09:52:10,426 TRACE main TransactionSynchronizationManager:139 - Retrieved value [org.springframework.jdbc.datasource.ConnectionHolder@6d4c4d61] for key [org.apache.commons.dbcp.BasicDataSource@5bf0cf51] bound to thread [main]
09:52:10,427 TRACE main TransactionSynchronizationManager:139 - Retrieved value [org.springframework.jdbc.datasource.ConnectionHolder@6d4c4d61] for key [org.apache.commons.dbcp.BasicDataSource@5bf0cf51] bound to thread [main]
09:52:10,427 DEBUG main JdbcExecutionContextDao:182 - PreparedStatement Class = org.apache.commons.dbcp.DelegatingPreparedStatement
09:52:10,427 DEBUG main JdbcExecutionContextDao:183 - Short Context '{"map":{"entry":{"string":["TEST_ATTRIBUTE","Test Attribute"]}}}'
09:52:10,427 DEBUG main JdbcExecutionContextDao:184 - Long Context 'null'
09:52:10,427 DEBUG main JdbcExecutionContextDao:185 - Execution ID '199'
09:52:10,428 DEBUG main JdbcExecutionContextDao:197 - This is the SQL statement that should be ran:
09:52:10,428 DEBUG main JdbcExecutionContextDao:198 - com.mysql.jdbc.JDBC4PreparedStatement@6bf51e5c: UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = '{"map":{"entry":{"string":["TEST_ATTRIBUTE","Test Attribute"]}}}', SERIALIZED_CONTEXT = null WHERE JOB_EXECUTION_ID = 199
09:52:10,429 DEBUG main JdbcTemplate:800 - SQL update affected 1 rows

It looks like the Prepared Statement is creating the correct SQL, however, the data is not being saved. There seems to be a mismatch between what SQL should be executed and what is actually executed. Either that, or my configs are wrong.

data-source-context.xml

HTML Code:

<beans ....>
	
	<!-- <bean id="placeholderProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" depends-on="environment">
		<property name="location" value="classpath:batch-mysql.properties" />
		<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
		<property name="ignoreUnresolvablePlaceholders" value="true" />
		<property name="order" value="1" />
	</bean> -->
	
	<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
		<property name="driverClassName" value="com.mysql.jdbc.Driver" />
		<property name="url" value="jdbc:mysql://localhost:3306/feed_sandbox" />
		<property name="username" value="spring_batch" />
		<property name="password" value="spring_batch" />
	</bean>
	
	<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" lazy-init="true">
		<property name="dataSource" ref="dataSource" />
	</bean>
	
	<bean id="lobHandler" class="org.springframework.jdbc.support.lob.DefaultLobHandler" />
	
	<bean id="incrementerParent" class="org.springframework.jdbc.support.incrementer.MySQLMaxValueIncrementer">
		<property name="dataSource" ref="dataSource" />
		<property name="incrementerName" value="ID" />
		<property name="columnName" value="ID"/>
	</bean>

	<bean id="jdbcTemplate" class="org.springframework.jdbc.core.simple.SimpleJdbcTemplate">
		<constructor-arg index="0" ref="dataSource"/>
	</bean>
	
	<bean id="jobInstanceDao" class="org.springframework.batch.core.repository.dao.JdbcJobInstanceDao">
		<property name="jdbcTemplate" ref="jdbcTemplate"/>
		<property name="jobIncrementer" ref="incrementerParent" />
	</bean>
	
	<bean id="jobExecutionDao" class="org.springframework.batch.core.repository.dao.JdbcJobExecutionDao">
		<property name="jdbcTemplate" ref="jdbcTemplate"/>
		<property name="jobExecutionIncrementer" ref="incrementerParent"/>
	</bean>
	
	<bean id="stepExecutionDao" class="org.springframework.batch.core.repository.dao.JdbcStepExecutionDao">
		<property name="jdbcTemplate" ref="jdbcTemplate"/>
		<property name="stepExecutionIncrementer" ref="incrementerParent"/>
	</bean>
	
	<bean id="executionContextDao" class="org.springframework.batch.core.repository.dao.JdbcExecutionContextDao">
		<property name="jdbcTemplate" ref="jdbcTemplate"/>
	</bean>

</beans>

simple-job-launcher-context.xml

HTML Code:

<beans ....>

	<import resource="data-source-context.xml" />

	<bean id="jobRepository" 
		class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean"
		p:databaseType="mysql" 
		p:dataSource-ref="dataSource" 
		p:transactionManager-ref="transactionManager"/>
		
	<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
		<property name="jobRepository" ref="jobRepository" />
	</bean>

	<bean id="mapJobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" 
		p:transactionManager-ref="transactionManager" lazy-init="true" autowire-candidate="false" />

	<bean id="jobExplorer" class="org.springframework.batch.core.explore.support.SimpleJobExplorer">
		<constructor-arg index="0" ref="jobInstanceDao"/>
		<constructor-arg index="1" ref="jobExecutionDao"/>
		<constructor-arg index="2" ref="stepExecutionDao"/>
		<constructor-arg index="3" ref="executionContextDao"/>
	</bean>
	
	<bean id="simpleJob" class="org.springframework.batch.core.job.SimpleJob" abstract="true">
		<property name="jobRepository" ref="jobRepository" />
		<property name="restartable" value="true" />
	</bean>

	<bean id="taskletStep" class="org.springframework.batch.core.step.tasklet.TaskletStep" abstract="true">
		<property name="jobRepository" ref="jobRepository" />
		<property name="allowStartIfComplete" value="true" />
	</bean>

	<bean id="simpleStep" class="org.springframework.batch.core.step.item.SimpleStepFactoryBean"
		abstract="true">
		<property name="transactionManager" ref="transactionManager" />
		<property name="jobRepository" ref="jobRepository" />
		<property name="startLimit" value="100" />
		<property name="commitInterval" value="1" />
	</bean>
	
	<bean id="skipLimitStep" class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"
		parent="simpleStep" abstract="true">
		<property name="skipLimit" value="0" />
	</bean>

	<bean id="customEditorConfigurer" class="org.springframework.beans.factory.config.CustomEditorConfigurer">
		<property name="customEditors">
			<map>
				<entry key="int[]">
					<bean class="org.springframework.batch.support.IntArrayPropertyEditor" />
				</entry>
				<entry key="org.springframework.batch.item.file.transform.Range[]">
					<bean class="org.springframework.batch.item.file.transform.RangeArrayPropertyEditor" />
				</entry>
				<entry key="java.util.Date">
					<bean class="org.springframework.beans.propertyeditors.CustomDateEditor">
						<constructor-arg>
							<bean class="java.text.SimpleDateFormat">
								<constructor-arg value="yyyyMMdd" />
							</bean>
						</constructor-arg>
						<constructor-arg value="false" />
					</bean>
				</entry>
			</map>
		</property>
	</bean>

	<bean id="jobRegistry" class="org.springframework.batch.core.configuration.support.MapJobRegistry" />

	<bean class="org.springframework.batch.core.configuration.support.AutomaticJobRegistrar">
		<property name="applicationContextFactories">
			<bean class="org.springframework.batch.core.configuration.support.ClassPathXmlApplicationContextFactory">
				<property name="resource" value="classpath:/JobContext.xml" />
			</bean>
		</property>
		<property name="jobLoader">
			<bean class="org.springframework.batch.core.configuration.support.DefaultJobLoader">
				<property name="jobRegistry" ref="jobRegistry" />
			</bean>
		</property>
	</bean>
	
</beans>

**pminearo** · Jan 25th, 2011, 05:59 PM

I forgot to mention I am using MySQL as the DB.

**Dave Syer** · Jan 26th, 2011, 10:19 AM

I assume you added those log statements to your own version of JdbcExecutionContextDao (they aren't in the framework)? Since it works in the unit tests, I assume there is something else wrong with your set up or your dao implementation. What are you using exactly.

P.S. there's a factory bean for the JobExplorer so you don't have to create all the daos manually.

**pminearo** · Jan 26th, 2011, 10:45 AM

Correct, I copied and pasted JdbcExecutionContextDao into my own module. It is the exact same code with just a few extra logging statements.

The configs I use are in the previous post. This is a very simple test. I have a "PrintTasklet" which prints out the Job Parameters and the Context Attributes to the log files. The Tasklet will place an attribute on the Job Context. There isn't much to this.

I would agree there may be something with my setup. The config files i use are the ones in my previous post (except for the Job Config). Does anything look out of place there?

**pminearo** · Jan 26th, 2011, 10:58 AM

I looked at the Unit Tests and Configs used for the Unit Tests and the only difference I see is the Unit Test uses HSQLDB and I am using MySQL. Will download the source code and do a build. See if there is a problem with using MySQL.

**pminearo** · Jan 26th, 2011, 02:58 PM

Looks like there is an issue using MySQL as the DB with the Unit Tests. I downloaded the source code and did a 'mvn install'. I imported the project into Eclipse and ran the JdbcJobExecutionDaoTests (with HSQLDB) and everything works fine. I then changed over the datasource to be MySQL and the Tests now fail. Here are the changes I made:

I added the following to the dependencies of spring-batch-core/pom.xml

HTML Code:

<dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
			<version>5.1.14</version>
		</dependency>

Since, I already have the DB in place, I changed the datasource-context.xml file to look like this:

HTML Code:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">
	<!-- bean class="test.jdbc.datasource.DataSourceInitializer">
		<property name="dataSource" ref="dataSource" />
		<property name="initScripts">
			<list>
				<value>org/springframework/batch/core/schema-drop-hsqldb.sql</value>
				<value>org/springframework/batch/core/schema-hsqldb.sql</value>
				<value>foo.sql</value>
			</list>
		</property>
	</bean -->
	
	<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
		<property name="driverClassName" value="com.mysql.jdbc.Driver" />
		<property name="url" value="jdbc:mysql://localhost:3306/sandbox" />
		<property name="username" value="spring_batch" />
		<property name="password" value="spring_batch" />
	</bean>
	<!-- bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
		<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
		<property name="url" value="jdbc:hsqldb:mem:testdb;sql.enforce_strict_size=true" />
		<property name="username" value="sa" />
		<property name="password" value="" />
	</bean -->
	<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
		<property name="dataSource" ref="dataSource" />
	</bean>
	
	<bean id="incrementerParent" class="org.springframework.jdbc.support.incrementer.MySQLMaxValueIncrementer">
		<property name="dataSource" ref="dataSource" />
		<property name="incrementerName" value="ID" />
		<property name="columnName" value="ID"/>
	</bean>
	
	<!-- bean id="incrementerParent" class="org.springframework.jdbc.support.incrementer.HsqlMaxValueIncrementer"
		abstract="true">
		<property name="dataSource" ref="dataSource" />
		<property name="columnName" value="ID" />
	</bean -->
	
	<bean id="jobIncrementer" parent="incrementerParent">
		<property name="incrementerName" value="BATCH_JOB_SEQ" />
	</bean>
	<bean id="jobExecutionIncrementer" parent="incrementerParent">
		<property name="incrementerName" value="BATCH_JOB_EXECUTION_SEQ" />
	</bean>
	<bean id="stepIncrementer" parent="incrementerParent">
		<property name="incrementerName" value="BATCH_STEP_SEQ" />
	</bean>
	<bean id="stepExecutionIncrementer" parent="incrementerParent">
		<property name="incrementerName" value="BATCH_STEP_EXECUTION_SEQ" />
	</bean>
</beans>

I get the following errors:

HTML Code:

java.lang.AssertionError: expected:<5> but was:<0>
....
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testFindRunningExecutions(AbstractJobExecutionDaoTests.java:202)
....

java.lang.IllegalStateException: There must be at most one latest job execution
	at org.springframework.util.Assert.state(Assert.java:384)
	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao.getLastJobExecution(JdbcJobExecutionDao.java:226)
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testGetLastExecution(AbstractJobExecutionDaoTests.java:158)
....

java.lang.AssertionError: expected:<Wed Jan 26 12:49:40 PST 2011> but was:<2011-01-26 12:49:40.0>
....
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.assertExecutionsAreEqual(AbstractJobExecutionDaoTests.java:342)
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testSaveAndFind(AbstractJobExecutionDaoTests.java:71)
....

java.lang.AssertionError: expected:<Wed Dec 31 16:00:00 PST 1969> but was:<1969-12-31 16:00:00.0>
....
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.assertExecutionsAreEqual(AbstractJobExecutionDaoTests.java:345)
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testFindExecutionsOrdering(AbstractJobExecutionDaoTests.java:94)
....

java.lang.AssertionError: expected:<Wed Jan 26 12:49:40 PST 2011> but was:<2011-01-26 12:49:40.0>
....
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.assertExecutionsAreEqual(AbstractJobExecutionDaoTests.java:345)
	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testUpdateExecution(AbstractJobExecutionDaoTests.java:140)
....

The timestamp warnings are one thing, not a big deal. But the first 2 errors listed are a little disconcerting. Also, I noticed there is no test for saving information for the JobExecutionContext. So I updated the testUpdateExecution to look like:

Code:

ublic void testUpdateExecution() {
		execution.setStatus(BatchStatus.STARTED);
		dao.saveJobExecution(execution);

		execution.setLastUpdated(new Date(0));
		execution.setStatus(BatchStatus.COMPLETED);
		execution.getExecutionContext().put("TEST_ATTR", "Test Attr");
		dao.updateJobExecution(execution);

		JobExecution updated = dao.findJobExecutions(jobInstance).get(0);
		assertEquals(execution, updated);
		assertEquals(BatchStatus.COMPLETED, updated.getStatus());
		assertExecutionsAreEqual(execution, updated);
	}

I saw nothing in the DB saved.

Thread: Passing Attributes/Parameters

Thread Tools

Display

Passing Attributes/Parameters

Tags for this Thread

Posting Permissions