Multi-file input and ouput

[cerrog]

I got it working as it should in order to meet our requirements.

Here's some stuff that I hope it will help!

The two problems that I needed nail down were:
A- the job has to iterate thru a set of files from the input directory
B- valid and rejected records must be written to separate output files in order to get handled by their respective business process

Here are pertinent parts of my main SB config file (removed some details for clarity).
FilePartitioner:

<beans:bean id="filePartitioner" class="com.MultiInputFilesPartitioner">
  <beans:property name="resources" value="file:${data.root}/input/${collected.file.name}"/>
</beans:bean>

MultiInputFilesPartitioner implements Partitioner, customized to load the partitions based on available files (ex: sorted by date).

NormalizerStepMaster:

<beans:bean id="normalizerStepMaster" name="normalizerStep:master" class="com.PartitionStep">
  <beans:property name="partitionHandler">
    <beans:bean class="org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler">
      <beans:property name="taskExecutor">
        <beans:bean class="org.springframework.core.task.SyncTaskExecutor" />
      </beans:property>
      <beans:property name="step" ref="normalizerStep" />
      <beans:property name="gridSize" value="3" />
    </beans:bean>
  </beans:property>
  <beans:property name="stepExecutionSplitter">
    <beans:bean class="org.springframework.batch.core.partition.support.SimpleStepExecutionSplitter">
      <beans:constructor-arg ref="jobRepository" />
      <beans:constructor-arg ref="normalizerStep" />
      <beans:constructor-arg ref="filePartitioner" />
    </beans:bean>
  </beans:property>
  <beans:property name="jobRepository" ref="jobRepository" />
</beans:bean>

PartitionStep extends AbstractStep in order to customize the doExecute.

And finally, the NormalizerStep, which is executed once for each input file:

<beans:bean id="normalizerStep" parent="simpleStep">
  <beans:property name="itemReader" ref="normalizerItemReader"/>
  <beans:property name="itemProcessor">
    <beans:bean id="compositeItemProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
      <beans:property name="itemProcessors">
        <beans:list>
          <beans:ref bean="normalizerValidatingItemProcessor" />
          <beans:ref bean="recordCounterItemProcessor" />
        </beans:list>
       </beans:property>
     </beans:bean>
  </beans:property>
  <beans:property name="itemWriter" ref="normalizerCompositeItemWriter"/>
  <beans:property name="streams">
    <beans:list>
      <beans:ref bean="validRecordItemWriter"/>
      <beans:ref bean="rejectedRecordItemWriter"/>
     </beans:list>
   </beans:property>
</beans:bean>

The normalizerCompositeItemWriter is implemented as DHGarrette suggested.
The normalizerValidatingItemProcessor is the one that decides if a record is valid or rejected.
The recordCounterItemProcessor is a custom processor that checks if the record count match with the input file footer.

I really hope the it will help you get out of the Dead-end.

Good luck!

Gino

[sansun2111]

Thanks a lot for the ideas ,
You people are brilliant, Thank you.

[skoorella]

I have exactly the same requirement as below...

Can you please share job configuration which will help me to implement

Thanks
-Suri

Hello,

I have a requirement that I don't know how to handle.

I have to read from multiple input files. For each input file, I have to write to a different output file (with same name but different suffix). During the processing, if a record is identified to be skipped, it has to be written to a different output file too (with same name as the input file but with .rejected suffix)

Example:

File pattern to read: /input/file.csv.*
Files in the input directory: file.csv.001, file.csv.002
If during processing, at least 1 record has been skipped in each file, the result should be something like:

In the output directory: file.csv.001.out, file.csv.002.out
In the reject directory: file.csv.001.rejected, file.csv.002.rejected

That being said, my first guess was to subclass 'FaultTolerantStepFactoryBean' and 'TaskletStep' in order to be able to iterate thru the itemReader-itemProcessor-itemWriter-skipListener cycle for each input file.

I'm about to start coding but I keep telling myself that I'm probably not the first guy with that kind of requirement. So that's why I turn myself to the Spring-Batch community for some insight.

Regards,

Gino.