Pass file name to FlatFileWriter at runtime

[Aravind. S]

hi,

I need to pass file name to FlatFileWriter at runtime, i need to do a file read and use that data to arrive at the file name.
Am storing the required info in execution context but when set the file name in my writer, its not picking up the one am supplying instead its taking the default one defined in resource property in config file.
Any thoughts...

[crowest]

Take a look at StepExecutionResourceProxy. This can cope with most requirements. You can add a JobParameterConverter is some translation is required.

It is of course possible to roll-your-own using the same kind of pattern. For example, one where you can interrogate the Job or Step Execution Context for the filename to use.

Sadly the current StepExecutionResourceProxy doesn't contain hook methods in order for one to easily sub-class and add delegates for additional filename resolution.

[Aravind. S]

class FileWriter extends FlatFileItemWriter implements StepExecutionListener
{
Resource resource;
StepExecution stepExecution;

public ExitStatus afterStep( final StepExecution stepExecution )
{
return null;
}

public void beforeStep( final StepExecution stepExecution )
{
this.stepExecution = stepExecution;
}

public ExitStatus onErrorInStep( final StepExecution stepExecution, final Throwable e )
{
return null;
}

public void setResource( final Resource resource )
{
this.resource = resource;
super.setResource( resource );
}

public void write( final Object obj ) throws Exception
{
String fileName = (String)stepExecution.getExecutionContext().get( "FILE_NAME" );
String path = "file:target/test-outputs/write1.TEMP.txt";
fileName = path.concat( fileName ).concat( ".log" );
resource.getFile().renameTo( new File( fileName ) );
setResource( resource );
super.write( obj );
}
}

But in FlatFileWriter.java
public void write(Object data) throws Exception {
if(getOutputState().isInitialized()){
FieldSet fieldSet = fieldSetCreator.mapItem(data);
lineBuffer.add(lineAggregator.aggregate(fieldSet) + lineSeparator);
}
else{
throw new WriterNotOpenException("Writer must be open before it can be written to");
}
}

// Returns object representing state.
private OutputState getOutputState() {
if (state == null) {
try {
File file = resource.getFile();
Assert.state(!file.exists() || file.canWrite(), "Resource is not writable: [" + resource + "]");
}
catch (IOException e) {
throw new ItemStreamException("Could not test resource for writable status.", e);
}
state = new OutputState();
state.setDeleteIfExists(shouldDeleteIfExists);
state.setBufferSize(bufferSize);
state.setEncoding(encoding);
}
return (OutputState) state;
}

In the above code only if the state is null, it will create the resource again else it will take the old file name , so even if rename the resource it doesnt get reflected in the write() method.

[DHGarrette]

Did you try the StepExecutionResourceProxy as was suggested? In Spring Batch 1.x that's really the right approach.

<property name="resource" ref="fileLocator" />

<bean id="fileLocator" class="org.springframework.batch.core.resource.StepExecutionResourceProxy">
    <property name="filePattern" value="file://temp/myfile%schedule.date%.txt"/>
</bean>

Note also that in Spring Batch 2.0 (starting with M4) you will be able to use the late-binding feature to say something like:

<property name="resource" value="file://temp/myfile#{jobParameters[schedule.date]}.txt" />

[1] http://static.springframework.org/sp...urceProxy.html

[kmbarlow]

Ok... just a question about functionality... it seems using the new 2.0 functionality to pull filenames from parameters works well for readers but not writers... with readers the file needs to exist (since its your input) but with writers, I seem to be getting the following errors about the file not existing:

2009-02-19 17:22:42,103 [main] ERROR org.springframework.batch.core.step.AbstractStep  Encountered an error executing the step: class org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [output.txt]]
org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [output.txt]]
	at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:293)
	at org.springframework.batch.item.file.FlatFileItemWriter.open(FlatFileItemWriter.java:233)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy8.open(Unknown Source)
	at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:98)
	at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:335)
	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:190)
	at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:321)
	at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:85)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:224)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:110)
	at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:105)
	at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:204)
	at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:251)
Caused by: java.io.FileNotFoundException: class path resource [output.txt] cannot be resolved to URL because it does not exist
	at org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:162)
	at org.springframework.core.io.ClassPathResource.getFile(ClassPathResource.java:174)
	at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:290)
	... 24 more

The javadocs for FileSystemResource seem to say that an execption will be thrown if the file doesn't exist but at the same time, you can create a new FileSystemResource by passing in a File object which will create the resource if it doesn't exist. Seems that the code here is relying on string passing which throws the exception because the resource has not been previously created. Not sure if that is intended for the FileSystemResource but even from a Batch perspective, should the file be created if its doesn't exist?

Keith

[DHGarrette]

Keith,

Can you post your configuration for that resource?

[kmbarlow]

I was getting those errors when the FlatFileItemWriter was configured as follows:

	<bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
		<property name="resource" value="#{jobParameters[outputfile]}" />
		<property name="lineAggregator" ref="lineAggregator" />
		<property name="shouldDeleteIfExists" value="true" />
	</bean>

it works fine for my input reader:

	<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
		<property name="resource" value="#{jobParameters[inputfile]}" />
		<property name="lineMapper" ref="lineMapper" />
	</bean>

but I had to implement my writer as follows to get it to flow successfully:

<bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
		<property name="resource" value="file:output_#{jobParameters[date]}.txt" />
		<property name="lineAggregator" ref="lineAggregator" />
		<property name="shouldDeleteIfExists" value="true" />
	</bean>

Keith

[DHGarrette]

Two options:

1) Stick "file:" in front of the path:

<property name="resource" value="file:target/test-outputs/htmlJobOutput.html"/>

2) Specify the class to use

<property name="resource">
	<bean class="org.springframework.core.io.FileSystemResource">
		<constructor-arg value="target/test-outputs/htmlJobOutput.html"/>
	</bean>
</property>

See also: http://static.springframework.org/sp...s-dependencies

[kmbarlow]

Thanks.

Keith