FlatFileItemWriter & shouldDeleteIfExists

[marcedwards1977]

Hi,

I'm using spring-batch-2.1

I have a muti step job. Within the job there are 2 sequential steps that need to write to the same file i.e. the same writer.

I have defined a writer (FlatFileItemWriter) and specified "shouldDeleteIfExists" to be false. The first step works as expected and writes the file. The second step ends up throwing a "org.springframework.batch.item.ItemStreamExceptio n: File already exists"

If I do not use the "shouldDeleteIfExists" the default value is used (true) and the second step runs succesfully but the output file is deleted and recreated and the data from the previous step is lost.

Am I using this wrong or is there a bug? I had assumed shouldDeleteIfExists=false would mean the file would just be appended to - which is the functionality I require.

I have looked at the underlying code in the FileUtils class (line 60) ......

if (file.exists()) {
if (!overwriteOutputFile) {
throw new ItemStreamException("File already exists: [" + file.getAbsolutePath() + "]");
}
if (!file.delete()) {
throw new IOException("Could not delete file: "+file);
}
}

.... and I can not see how the "shouldDeleteIfExists" would work?

I have looked at a JIRA issue https://jira.springframework.org/browse/BATCH-1224 where it appears David is saying that this is the intended functionality - or I may be reading this wrong.

Any help/advice would be greatly appreciated.

Thanks,
Marc

[marcedwards1977]

I have done some reading around this issue and it would seem that the way to implement the desired functionality would be to write 2 temporary files and then combine the files with a third step. This 3rd step, I believe, should be achieved using the MultiResourceItemReader.

This does seem quite a complicated way of doing something simple. Additionally, there looks to be a lack of documentation around this area.

[marcedwards1977]

Appreciate I'm answering my own posts here but .... it looks like there is a bug/defect with the shouldDeleteIfExists logic.

With regard to the initial post about the issue with having 2 separate steps (readers) writing to one writer - the solution is to create 2 'temporary' files and then using the "MultiResourceItemReader" merge the 2 temporary files.

The upside to this solution is that it is good seperation of units of work.
The downsides are the extra IO operations required to perfrom the task and the added complexity of the config/code required i.e. it makes something that that should be starightforward more complicated.

[Dave Syer]

There's no bug: shouldDeleteIfExists is there to delete a file if it exists before starting, which is exactly the behaviour you observe. If you need a FlatFileItemWriter that appends you can probably create one quite easily. It would be a new feature for Spring Batch so you need to open a JIRA if you would like to see it. Patches always welcome.

[Jul]

It will be enough to add strict/append flag if it's not implemented yet

[shravan_ramamurthy]

Please help. Need to write to same file from multiple steps.

[Dave Syer]

<bean ... class="...FlatFileItemWriter">
   ...
   <property name="appendAllowed" value="true"/>
</bean>