CommandLineJobRunner and MultiResourcePartitionner

[psoares]

Hi !
I built a job to process multiple huge xml gzipped files in parallel and write each input file to a distinct tsv file. For that, I use MultiResourcePartitionner.
Everything works well when I launch my job from a junit test using a SimpleJobLauncher. I test with 3 huge input files, and I can actually see 3 output tsv files being created in parallel.

When I launch that same job from the commandLine, I only get 1 file processed at a time...
Is there any specific TaskExecutor type I have to use for the CommandLineJobRunner to behave as the JobRunner in my Junit test ?

Below is my spring config file :

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:batch="http://www.springframework.org/schema/batch"
	xmlns:oxm="http://www.springframework.org/schema/oxm" xmlns:int="http://www.springframework.org/schema/integration"
	xmlns:int-stream="http://www.springframework.org/schema/integration/stream"
	xmlns:context="http://www.springframework.org/schema/context"
	xmlns:task="http://www.springframework.org/schema/task"
	xsi:schemaLocation="http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.0.xsd
		http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm-3.0.xsd
		http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd
		http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream-2.0.xsd
		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
		http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">

	<context:component-scan base-package="com.mycompany"/>	
	<task:annotation-driven/>
	
	<oxm:jaxb2-marshaller id="myMarshaller"
		contextPath="com.mycompany.domain">
	</oxm:jaxb2-marshaller>
		        	
	<bean id="myReader" class="org.springframework.batch.item.xml.StaxEventItemReader"
		scope="step">
		<property name="fragmentRootElementName" value="RootItem" />
		<property name="saveState" value="false"/>
		<property name="resource">
			<bean class="org.springframework.core.io.InputStreamResource">
				<constructor-arg name="inputStream">
					<bean class="java.util.zip.GZIPInputStream">
						<constructor-arg>
							<bean class="com.mycompany.utils.io.ResourceInputFile" factory-method="getInputStream">
								<constructor-arg value="#{stepExecutionContext[fileName]}" />
							</bean>
						</constructor-arg>
					</bean>
				</constructor-arg>
			</bean>
		</property>
		<property name="unmarshaller" ref="myMarshaller" />
	</bean>

        <bean id="myProcessor" class="my.company.batch.MyItemProcessor" />

	<bean id="myFileWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
		scope="step">
		<property name="resource" value="file:#{stepExecutionContext[outputDbName]}.tsv" />
		<property name="shouldDeleteIfExists" value="true"/>
		<property name="transactional" value="false"/>
		<property name="lineAggregator">
			<bean class="my.company.batch.MyLineAggregator" />
		</property>
	</bean>

        <bean id="fileNameListener" class="my.company.batch.MyFileNameListener"/>
    
	<batch:step id="xml_to_tsv_step">
		<batch:tasklet transaction-manager="transactionManager">
			<batch:chunk reader="myReader" processor="myProcessor" writer="myFileWriter" commit-interval="100" />
			<batch:listeners>
				<batch:listener ref="fileNameListener"/>
			</batch:listeners>
		</batch:tasklet>
	</batch:step>
	
	
	<bean id="partitioner" class="org.springframework.batch.core.partition.support.MultiResourcePartitioner" scope="step">
		<property name="resources" value="file:#{jobParameters['input.dir.name']}"/>
	</bean>
			
	<batch:job id="partitionJob">
		<batch:step id="step1">
			<batch:partition step="xml_to_tsv_step" partitioner="partitioner">		
				<batch:handler grid-size="10" task-executor="taskExecutor"/>		
			</batch:partition>
		</batch:step>
	</batch:job>

	<task:executor id="taskExecutor" queue-capacity="100" pool-size="10"/>

</beans>

Thanks in advance for your help.

Best regards,

Philippe

[psoares]

I found out what my problem was.
I was calling my job in a script this way :

#!/bin/bash

typeset inputFile=$1
[...]
java org.springframework.batch.core.launch.support.CommandLineJobRunner process-multiple-files-context.xml partitionJob input.file.name=$inputFile

Then I launched my script with :
> ./myscript.sh /path/to/my/files*.xml.gz

When I do this instead, it works :
> ./myscript.sh "/path/to/my/files*.xml.gz"

I guess the first version was doing shell expansion and got the first file matching my file glob.