Programatically build job beans?

[scottishfiction]

I'm new to spring and spring-batch and require a method of programtically creating job/step/tasklet beans. I'm currently defining a number of jobs in the xml that are mostly the same but with one property changing per job and would like to cut this out by dynamically building the jobs at runtime.

Thanks for any help in advance.

[Dave Syer]

All the APIs are pretty open, but I never came across a real life use case that needed much more than a tweak in the XML, and maybe a simple custom component. If you can describe the use case in more detail we might be able to suggest something more concrete.

[scottishfiction]

Well the purpose of the batch is to parse xml from a REST service, do a bit of processing to add in some additional data and then write to a database.

The REST service takes two parameters (A,B) I want the user to be able to specify the value(s) of A as args/properties (the four B values are constant).

My current structure is a job for each A, which contains a step for each B. But by explicity writing these in the xml I'm not only assuming the number of A's, but I'm also writing a lot of near identical xml (the UrlResource and therefore the ItemReader changes for each combination of A and B, the ItemProcessor doesn't change, and the ItemWriter changes for each B, but is oblivious to changes in A).

Hence why I thought creating the jobs dynamically based on the the As provided might be a nicer solution.

Apologies if this is a poor explanation, but I would really appreciate any advice.

[Dave Syer]

If you can solve the problem with a finite number of XML files, I'm sure we can get that finite number down to 1. You should be able to just bind a portion of the URL, e.g.

<property name="resource" value="http://servicehost/foo/bar/#{jobParameters['A']}/#{jobParameters['B']}"/>

Did I misunderstand the requirement?

[redsnappa]

More detail to the problem:

We are trying to get data via a REST call from a db, process it (no details necessary) and write it to other tables in another database.

The connection to the REST service must be auth'd using kerberos,

<bean id="abstractReader" class="org.springframework.batch.item.xml.StaxEventItemReader"
		abstract="true">
		<property name="fragmentRootElementName" value="PersonCoverage" />
		<property name="unmarshaller" ref="personCoverageMarshaller" />
		<property name="strict" value="false" />
	</bean>


	<bean id="personReader" parent="abstractReader">
		<property name="resource" ref="personUrlResource" />
	</bean>

	<bean id="personUrlResource" class="com.blah.reader.KerberisedUrlResource"
		scope="step">
<!-- we think the issue is here as jobParameters is out of scope to this bean-->
		<constructor-arg value="#{jobParameters['url.PERSON']}" />
	</bean>

here the url needs to be passed to the URL resource,

the job

<job job-repository="jobRepository" id="mgrmaint2tm"
		xmlns="http://www.springframework.org/schema/batch">
		<step id="divLoad" next="allOrgLoad">
			<tasklet>
				<listeners>
					<listener ref="stepListener" />
				</listeners>
				<chunk reader="divReader" processor="corpIdProcessor" writer="divWriter"
					commit-interval="${chunk.commit.interval}" />
			</tasklet>
		</step>
		<step id="allOrgLoad" next="ccLoad">
			<tasklet>
				<listeners>
					<listener ref="stepListener" />
				</listeners>
				<chunk reader="allOrgReader" processor="corpIdProcessor"
					writer="allOrgWriter" commit-interval="${chunk.commit.interval}" />
			</tasklet>
		</step>

		<step id="ccLoad" next="personLoad">
			<tasklet>
				<listeners>
					<listener ref="stepListener" />
				</listeners>
				<chunk reader="ccReader" processor="corpIdProcessor" writer="ccWriter"
					commit-interval="${chunk.commit.interval}" />
			</tasklet>
		</step>
		<step id="personLoad">
			<tasklet>
				<listeners>
					<listener ref="stepListener" />
				</listeners>
				<chunk reader="personReader" processor="corpIdProcessor"
					writer="personWriter" commit-interval="${chunk.commit.interval}" />
			</tasklet>
		</step>
	</job>

As the rest url, the job paramater will have two parameters,

A, B where A is like the company (of which there are more than one) and B like dept where there are (roughly) 4 of each to 1 A.
so:

http://restcall?A=A1&B=B1
http://restcall?A=A1&B=B2
http://restcall?A=A2&B=B1
http://restcall?A=A2&B=B2 so on so forth.

The ultimate goal is to have one job which is run many times changing depending on the job parameters.

But as the kererisedURL resource is outwith the scope of the jobparameters within a job, the url cant be dynamically. we cant pass the REST url and have it auth via the kerbURLresource class/bean.

[redsnappa]

launched with the following parameters: [{url.ALLORG=http://tddev/PersonCoverageDetails/getByDomain.xml?cvgRelationCode=EMPL_TRDG&domainCo de=ALLORG}]
12437 [main] INFO org.springframework.batch.core.job.AbstractJob - Executing step: [TaskletStep: [name=divLoad]]
13344 [main] ERROR org.springframework.batch.core.step.AbstractStep - Encountered an error executing the step
org.springframework.batch.item.ItemStreamException : Failed to initialize the reader
at org.springframework.batch.item.support.AbstractIte mCountingItemStreamItemReader.open(AbstractItemCou ntingItemStreamItemReader.java:139)

[Dave Syer]

That's not much stack trace to work out why it is broken. It is enough to see that the JobParameters do not contain the "url.ALL" parameter. Is that why it doesn't work? I'm not sure either what you mean by the "kerberisedURL". When in the lifecycle of the job do you actually contact the HTTP resource?

[redsnappa]

We eventually got it sorted, we were not wrong in our syntax. I think we were trying to change too little for fear of changing too much, as we are new to Spring Batch its a baby step (2 baby steps forward one back) kinda project. Sorry we cant provide any useful information for people with the same problem, or indeed really identify the problem accurately ourselves!