FlatFileItemWriter.open on restart- expected behaviour?

[ndw]

Is this expected behaviour:

We have a simple 1 Step Job reading a FlatFile and transforming and outputting to another FlatFile.

In case of failure we want the job to continue where it left off on restart, so we have following set on the Writer:

shouldDeleteIfExists = false

If the job fails before the first StepExecution has commited (we are commiting after 50 items at the moment), but the output file has been created, then we cannot restart due to

org.springframework.batch.item.exception.StreamExc eption: Resource already exists: file [D:\temp\output1.csv]
at org.springframework.batch.io.file.FlatFileItemWrit er$OutputState.initializeBufferedWriter(FlatFileIt emWriter.java:378)

I think the test in FlatFileItemWriter.open is failing to detect that this is a restart because there is nothing in the STEP_EXECUTION_CONTEXT table for this resource:

public void open(ExecutionContext executionContext) {
		OutputState outputState = getOutputState();
               // test below returns FALSE - 
               // nothing in the STEP_EXECUTION_CONTEXT table for this resource 
	if(executionContext.containsKey(getKey(RESTART_DATA_NAME))){  
			outputState.restoreFrom(executionContext);
		}
	}

If outputState.restoreFrom is not called, the restarted flag is not set to true and the exception above results.

[Dave Syer]

That's probably a bug. Can you put I'd in JIRA (that way you'll get notifications about progress)?

[ndw]

I'm still on M5 code base so apologies if this is fixed in rc1:

I tried to create a test harness for the original problem in this thread.

I've based it on the RestartFunctionalTests in the samples.

I seem to have fallen foul of a separate feature in FlatFileItemWriter whereby the OutputState.shouldDeleteIfExists is not re-initialised correctly when the Writer is re-opened.

When the writer is closed, the output state is set to null:

    public void close(ExecutionContext executionContext) {
        if (state != null) {
            getOutputState().close();
            state = null;
        }
    }

If the same java object representing the Writer is reopened (ie not a new instance of the Writer), then non of the setMethods on the writer get called. Writer.open just creates a new OutputState with default settings rather than the ones in the config file.

public void open(ExecutionContext executionContext) {
        OutputState outputState = getOutputState();
        if(executionContext.containsKey(getKey(RESTART_DATA_NAME))){
            outputState.restoreFrom(executionContext);
        }
    }
    ......
    private OutputState getOutputState() {
        if (state == null) {
            state = new OutputState();  // just takes the default values
        }
        return (OutputState) state;
    }

I think this applies to a number of fields - bufferSize, encoding etc - all are held by the OutputState - not the Writer.

This means I cannot set the setShouldDeleteIfExists flag, so I can't easily test the FlatFileItemWriter.open on restart bug.

Is this a valid test and if this is a bug, do I need to raise 2 separate Jira items for this and the original restart issue or can I include in one?

Included below are extracts of my test job and config file.

public class FlatFileItemWriterRestartTests extends AbstractBatchLauncherTests {

    public void testLaunchJob() throws Exception {

        final JobParameters jobParameters = new JobParameters();
        logger.info("Starting test");
        
        JobExecution jobExecution = null;
        try {
            jobExecution = launcher.run(getJob(), jobParameters);
        }
        catch (InfrastructureException expected) {
            logger.info("Caught Exception");
            logger.info(expected);
            assertTrue("Not planned exception: "+expected.getMessage(), expected.getMessage().toLowerCase().indexOf("planned")>=0);
        }
        
        // At this point existing FlatFileItemWriter is reused, 
        // but new OutputState  
        // is created, with default shouldDeleteIfExists=true 
        jobExecution = launcher.run(getJob(), jobParameters); 

        assertEquals(BatchStatus.COMPLETED, jobExecution.getStatus());
        logger.info("Completed");
    }
}

This is the main part of the job definition:

	<bean id="restartSampleJob" parent="simpleJob">
        <property name="steps">
            <bean id="step1" parent="simpleStep">
                <property name="streams" ref="fileItemReader"/>
                <property name="itemReader">
                    <bean
                        class="org.springframework.batch.item.reader.ValidatingItemReader">
                        <property name="itemReader"
                            ref="itemReader" />
                        <property name="validator" ref="fixedValidator" />
                    </bean>
                </property>
                <property name="itemWriter">
                    <bean
                        class="org.springframework.batch.io.file.FlatFileItemWriter">
                        <property name="resource" ref="outputFileLocator"/>
                        <property name="fieldSetUnmapper" ref="fieldSetUnmapper" />
                        <property name="shouldDeleteIfExists" value="false"/>
                    </bean>
                </property>
                <!-- First commit will occur after the exception -->
                <property name="commitInterval" value="4"/>
            </bean>
        </property>
    </bean>

[sandrine]

The problem is resolved in rc1, I have tested it.
Here is the code of FlatFileItemWriter (in private class OutputState):

private void initializeBufferedWriter() {
File file;

try {
file = resource.getFile();

// If the output source was restarted, keep existing file.
// If the output source was not restarted, check following:
// - if the file should be deleted, delete it if it was exiting
// and create blank file,
// - if the file should not be deleted, if it already exists,
// throw an exception,
// - if the file was not existing, create new.
if (!restarted) {
if (file.exists()) {
if (shouldDeleteIfExists) {
file.delete();
} else {
throw new ItemStreamException("Resource already exists: " + resource);
}
}
String parent = file.getParent();
if (parent != null) {
new File(parent).mkdirs();
}
file.createNewFile();
}

[ndw]

Thanks for response Sandrine. I think that my previous posting was confusing and that the problem still exists in rc1.

The test code I posted does run succesfully, but I don't expect it tFlatFo! It's a bad test and completes successfully because FlatFileItemWriter.OutputSource.shouldDeleteIfExis ts is defaulting to true on the the second run of the job.

I might be missing something or misusing the apis, but rather than continue here I've raised http://jira.springframework.org/browse/BATCH-507 - OutputSource not initialised correctly
http://jira.springframework.org/browse/BATCH-510 - FlatFileItemWriter.restarted not always set correctly