Is PartitionStep my solution?

[crowest]

Greetings,

I've a Spring Batch job that's rather established and like always a new requirement has come over the horizon and upset the applecart!

Presently my Job consists of four steps that run one after the other. Step 1 is a simple tasklet to search for a file on the local file-system that matches a generated naming criteria and waits for a semaphore file to appear before proceeding. Step 2 is a reader/writer/processor that generates a file. Step 3 is optional and re-formats the file from Step 2. Step 4 uses FTP to transfer the file (formatted or not) to another system.

My new requirement needs me to read one or more input files, and generate intermediary files. A new step will take those intermediary files and create one output file ready for formatting (optional) and FTPing as currently takes place.

The main problem is the number of input files can only be determined at runtime and therefore as I see it a normal Split (configured by Spring XML) is not possible. Is the PartionStep my salvation here? Will it allow me to establish a bunch of reader/writer/processors to read separate and entirely individual input files with their own output file? I notice there isn't a factory bean to create such a Step as there is for TaskletStep. Is it advisable to create one of my own? Will the

Any advice would be most appreciated...

Best wishes,
Steve C.

[bajajyo]

Greetings,

I've a Spring Batch job that's rather established and like always a new requirement has come over the horizon and upset the applecart!

Presently my Job consists of four steps that run one after the other. Step 1 is a simple tasklet to search for a file on the local file-system that matches a generated naming criteria and waits for a semaphore file to appear before proceeding. Step 2 is a reader/writer/processor that generates a file. Step 3 is optional and re-formats the file from Step 2. Step 4 uses FTP to transfer the file (formatted or not) to another system.

My new requirement needs me to read one or more input files, and generate intermediary files. A new step will take those intermediary files and create one output file ready for formatting (optional) and FTPing as currently takes place.

The main problem is the number of input files can only be determined at runtime and therefore as I see it a normal Split (configured by Spring XML) is not possible. Is the PartionStep my salvation here? Will it allow me to establish a bunch of reader/writer/processors to read separate and entirely individual input files with their own output file? I notice there isn't a factory bean to create such a Step as there is for TaskletStep. Is it advisable to create one of my own? Will the

Any advice would be most appreciated...

Best wishes,
Steve C.

See if this works for you. Use MultiResource Reader which can read multiples files at run time and process them one by one.

You can club this logic in existing job by writing a custom flat file reader and having conditional logic in steps, but that will make job complex. Why not write a new job for new requirement?

[crowest]

Yes, the solution did become easier by creating a new Job to deal with the new requirements. The alternative was to over-engineer the one Job to accomodate too much flexibility.

The new Job uses a PartitionStep, which as I discovered does give each StepExecution its own set of Reader/Processor/Writers as I had hoped. It's interesting that PartitionStep isn't that well documented, nor implemented in 2.0.1 of Spring Batch. I struggled with Thread issues which in the end I solved with ThreadLocals and the use of 'step' scope in the Spring configuration.

Quite happy with the result. Although there are one or two annoying features of Spring Batch. For example, PartitionStep doesn't allow you to alter or override the 'aggregator' instance it uses. I ended up creating my own extention of AbstractStep that does!

With regards and thanks for the advice,

Steve C.