Dynamically build FlatFileItemWriter file name at runtime

[dghighfill]

I'm not sure if this is possible in SpringBatch, but I've been scanning the forum for quite awhile and haven't yet found the solution. I would think this would be pretty common, but I'm not finding the right answer.

What I'm wanting to do is determine the output filename for the ItemWriter at runtime, but I don't know the filename until the reader has read what it needs to determine the file name.

For example. If I want to write an output file c:\output\customname\output.csv where customname is derived from after reading either an input file or database record. (e.g. c:\output\12345\output.csv).

I've read where I can late bind the output file name and I've done that, but what I'm trying to do is set the resource file name after the reader and processor has done what it needs to do and before I write a list of items in the writer.

Am I trying to do something that's not possible?

[arno]

what kind of information do you need from the input file to compute the name of the output file? some kind of header? if so, a preliminary step can peek the input file and store the information somewhere in the execution context.

remember Spring Batch uses chunk processing, so the writer will be first called after the reading and processing of the first chunk (usually a small number of items, like 10 or 100), not once *all* the items have been read and processed.

[dghighfill]

It's an id that's used to identify a company (e.g. c:\output\12345\output.csv) and this will be the directory name where all their files reside. It seems the setResource is called before we ever get into the reading of anything for the writer and we can't figure out how to read something first to get the value before the writer is called. Would it work if we just broke the job into two steps. One step (i.e. step1) reads the file and places it in the execution context, then then call a second step (step2) actually has the output file name configured from the execution context. value="#{jobExecutionContext['output.file.name']}" Would this prevent the writer from getting initialized if it were in a second step and satisfy our requirement?

[arno]

doesn't it work by referring to job parameters with the step scope? This would avoid using a preliminary step. When doing value="#{jobParameters['input.file']}", you can use a more advanced expression to retrieve just the name of the directory (SpEL has built-in support for string manipulation).

[dghighfill]

Maybe I'm not explaining it well. I don't know the value of the id until runtime. jobParameters work well and we use them all the time, they just don't work in this case. Imagine having the Reader being a database query and that query returns the id you need to construct the path to the output file. That's basically what we are trying to do. The setResource for the Writer is called before the query ever gets executed and we have no way to use late binding to derive the name.

Would splitting this into two steps solve our issue as mentioned in the previous post?

[dghighfill]

I'm guessing then, this wasn't a core requirement of SpringBatch. So if someone can confirm that you indeed can't dynamically set the output file name at runtime, from a value from a reader I would appreciate it. If if there's some way to set it, please advise.

[michael.lange]

configuring at runtime seems not to work within spring batch semantics, e.g. the step in whole is created/configured at runtime, after that the reader, writer, processors, etc. should be ready, after all that is a the desired deterministic behaviour

maybe something like this would work:

<job id="job">
  <step id="stepNormal" next="stepRenameFile" />
  <step id="stepRenameFile" /> 
</job>

• stepNormal
  
  • creates output-timestamp.txt
  • saves "output-timestamp.txt" as "filename" in JobExecutionContext (*)
  • saves desired ID as "id" in JobExecutionContext
• stepRenameFile (could be taskletstep)
  
  • pulls "output-timestamp.txt" from JobExecutionContext
  • pulls ID from JobExecutionContext
  • creates desired filename foo-bar-ID.txt
  • renames file

(*) http://static.springsource.org/spring-batch/reference/html-single/index.html#passingDataToFutureSteps