Spring Batch : MultiResourcePartitioner: Job ending with smallest file write complete

[Krishna Das]

Hi,
We have a batch job configured to read data from csv and load it into oracle db. We have configured a partition step which reads from multiple files and writes it to database parallel y, the issue is that the job is ending with a success status as soon as the smallest file with fewer records are done transferring to oracle.

Ex: Say we have 3 files and the commit point is 500
file1: 1000 records
file2: 1000 records
file3: 500 records
In the very first iteration as soon as the file3 is transferred, the entire job successfully completes successfully. With the other 2 files still having 500 more records to transfer.
Following is the configuration I am using, please let me know what as to what is wrong in the configuration.
It will of great help if you can help me with this.
ALSO I WOULD LIKE TO KNOW THE LINK BETWEEN COMMIT-POINT, GRID-SIZE AND NUMBER OF FILES.

<job job-repository="jobRepository" id="SampleJob"
xmlns="http://www.springframework.org/schema/batch" restartable="true">

<step id="UpdatePartition">
<partition step="UpdateStep" partitioner="Partitioner" >
<handler grid-size="${chunk.size}" task-executor="taskExecutor"/>
</partition>
</step>

</job>
<step id="UpdateStep" xmlns="http://www.springframework.org/schema/batch">
<tasklet transaction-manager="transactionManager" throttle-limit="${spring.batch.throttlelimit}">
<chunk reader="CSVFileReaderUpdate" processor="Transformer"
writer="JDBCItemWriterUpdate" commit-interval="${spring.batch.commit.interval}">
</chunk>
</tasklet>
</step>


<bean id="Partitioner"
class="org.springframework.batch.core.partition.su pport.MultiResourcePartitioner">
<property name="resources" value="file://${filepath}/${batchFile}" />
</bean>

<bean id="CSVFileReaderUpdate" class="org.springframework.batch.item.file.MultiRe sourceItemReader">
<property name="resources"
value="file://${filepath}/${batchFile}" />
<property name="delegate">
<bean class="org.springframework.batch.item.file.FlatFil eItemReader">
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
<!-- Rest of the config removed for brevity -->
</bean>

<bean id="JDBCItemWriterUpdate"
class="org.springframework.batch.item.database.Jdb cBatchItemWriter"
p:assertUpdates="true" p:dataSource-ref="datasource">
<property name="sql">
<!-- Rest of the config removed for brevity -->
</bean>


Thanks

[snicoll]

your partitioner misses a wildcard to select multiple files (it's hard to tell given the token you use). And you should use the partitioner to create one partition for file and then a regular reader to read one file per partition.

Have a look to:
http://static.springsource.org/sprin...cutionSplitter

[Krishna Das]

@snicoll: Oh! Of course, sorry about that.
We are using 'fileName*' with splits with names 'fileName1', 'fileName2', etc..
I have already checked the link that you have provided here, I have similar configuration, but kindly let me know if there is a flaw in this configuration.
Also please let me know, how does the chunk-size, commit-point, no.of files and no.of records in the file affect the write count?