Using JobParameters in hadoop namespace

[ebersoless]

I can't get the jobParamenters to be late loaded:

here is my config:

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/hadoop"
	xmlns:beans="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xmlns:batch="http://www.springframework.org/schema/batch"
	xmlns:context="http://www.springframework.org/schema/context"
	xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">


	<beans:bean class="org.springframework.batch.core.scope.StepScope">
		<beans:property name="proxyTargetClass" value="true"/>
	</beans:bean>

	
	<batch:job id="csv-to-cube-BATCH-job">
		<batch:step id="csv-import" next="csv-to-cube-step">
			<batch:tasklet ref="csv-import-tasklet"/>
		</batch:step>
		<batch:step id="csv-to-cube-step" >
			<batch:tasklet ref="csv-to-cube-tasklet" />
		</batch:step>
	</batch:job>

    <script-tasklet id="csv-import-tasklet"  >
        <script location="cp-data.js">
            <property name="inputPath" value="cube-csv-inputs/tmp" />
            <property name="output" value="cubes/adwork" />
            <property name="localResource" value="cube-csv-inputs/adworks.csv" />
        </script>
    </script-tasklet>
	

	<tasklet id="csv-to-cube-tasklet" job-ref="csv-to-cube-MR-job" scope="step"/>

	<job id="csv-to-cube-MR-job" scope="step"
		properties-location="#{jobParameters['properties-file']}"
		configuration-ref="hbaseConfiguration"
		input-path="${input}"
		output-path="${output}"
		mapper="haruspex.etl.csv.CsvToCubeMapper"
		reducer="haruspex.etl.csv.CsvToCubeReducer"
		validate-paths="false" 
		/>
		
	
</beans:beans>

this is the error:

ERROR 2012-06-13 10:04:24,634 [org.springframework.batch.core.step.AbstractStep(AbstractStep.java)]: Encountered an error executing the step
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.csv-to-cube-MR-job': Cannot create inner bean '(inner bean)' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] while setting bean property 'properties'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)': Invocation of init method failed; nested exception is java.io.FileNotFoundException: class path resource [#{jobParameters['properties-file']}] cannot be opened because it does not exist
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:281)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:125)
.
.
.
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)': Invocation of init method failed; nested exception is java.io.FileNotFoundException: class path resource [#{jobParameters['properties-file']}] cannot be opened because it does not exist
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:270)
	... 78 more
Caused by: java.io.FileNotFoundException: class path resource [#{jobParameters['properties-file']}] cannot be opened because it does not exist
	at org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:158)
	at org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(PropertiesLoaderSupport.java:181)
	at org.springframework.core.io.support.PropertiesLoaderSupport.mergeProperties(PropertiesLoaderSupport.java:150)
	at org.springframework.beans.factory.config.PropertiesFactoryBean.createInstance(PropertiesFactoryBean.java:113)
	at org.springframework.beans.factory.config.PropertiesFactoryBean.createProperties(PropertiesFactoryBean.java:98)
	at org.springframework.beans.factory.config.PropertiesFactoryBean.afterPropertiesSet(PropertiesFactoryBean.java:69)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)
	... 81 more

I have read similar posts on the batch forum but they aren't helping...

[Costin Leau]

Hmm... I have to try this myself. If you specify the job parameters on a different property do they get properly replaced?

[ebersoless]

Yes - after some fiddling around I can get both the script-tasklet and the hdp:job to find #{jobParameter['monkey']} except for the properties-ref. I think this is a life cycle issue - but it sure would be convenient to be able to load the hadoopConfiguration from a passed in file.

[Costin Leau]

Can you please raise a bug mentioning the fields where jobParameter works and where it doesn't? thanks!

[buddha_314]

Could you show us how you accomplished this, please?