file:outbound-channel-adapter : file extension ".writing" issue

[ajay007]

Hi,

I am using quartz and spring integration to updates the file into local directory using file:outbound-channel-adapter. The file is writing to dir correctly only first time thereafter it is using ".writing" extension to write the file for the one which is already present. My requirement is to update the files and overwrite if present. I am using spring integration version 2.0. Pls. advice me how to overcome this problem.

Thanks in Advance

Ajay

[iwein]

Do you have a sample I could try out? Best would be in the form of a JUnit test. I'm not sure if I get your point.

[ajay007]

Do you have a sample I could try out? Best would be in the form of a JUnit test. I'm not sure if I get your point.

I am in poc mode so I was pushing spring integration so lack junit test case.

Sorry for not being clear with my question earlier ... Let me try to explain my problem again

I have an application which is executed at regular interval to call specific web service and pull some String data. This string data is then written to the local directory using spring integration.

So suppose first time it got file a.xml and b.xml. it is writing to local directory as a.xml and b.xml.

In the second run, it again got a.xml and b.xml file but when S I is trying to write in local directory it writing it as "a.xml.writing" and "b.xml.writing" because a.xml/b.xml is already present in directory.

some code snippet (in case i am doing something wrong)

appcontext.xml

............
<bean id="fileBasedHandler" class= .......
<si:channel id="inputMessage"/>
<si:channel id="outputFile"/>
<file:outbound-channel-adapter channel="outputFile" directory="${file.directory.output}" auto-create-directory="true" filename-generator="fileBasedHandler" />
....................
//fileBasedHandler.java

public class FileBasedHandler extends DefaultFileNameGenerator{

public FileBasedHandler(){
super.setHeaderName("MSG_FILENAME");
}


//sending message as
MessageChannel input = (MessageChannel)appContext.getBean("outputFile");
msg=MessageBuilder.withPayload(strRule).setHeader("MSG_FILENAME", currentRuleName).build();
boolean sendStatus = input.send(msg);


Appreciate you help

Thanks

Ajay

[iwein]

I think it might be the case that there are exceptions in your log? If you find those you will be able to track it to the FileWritingMessageHandler which might not be capable of overwriting files? If you manage to do that you could log an issue so we can add a flag to turn this on.

I'm just guessing... without log messages / stacktraces / testcases you may assume I'm wrong.

[paul.mckenzie]

I encountered something similar (Spring-integration 1.0.3) and needed to make sure that I could over-write multiple versions of a file.

I created a file renaming endpoint that checked for the existence of the file and, if found, renamed the previous version before passing the message on to the < file : outbound-channel-adapter/>

Paul

[iwein]

That sounds like a useful feature.

[paul.mckenzie]

I would contribute the code, but I think that perhaps a cleaner solution would be to add to FileWritingMessageHandler the following methods ...



/**
* If true and the file already exists overwrite the original file.
* @param overwrite true or false
*/
public void setOverwriteFiles(final boolean overwrite);

/**
* If set and the file already exists use the filename generator to rename the previous version of the file.
* @param filenameGenerator used to create an alternative filename for the previous version of the file
*/
public void setCollisionFileNameGenerator(final FileNameGenerator filenameGenerator);


Of course it makes no sense to set both overwrite true and a collision filename generator.

[jzcfk9]

I'm seeing this same issue. All subsequent writes to a file after the first never make it to the destination file. The reason is because the rename from the *.writing file to the destination file is never successful. In my case at least, this seems to be inherent to the Win32FileSystem rename function.

Like the previous poster, I think an option to overwrite the file if it already exists would be a nice edition to FileWritingMessageHandler. INT-1721 (https://jira.springsource.org/browse/INT-1721) captures this issue.

[munger]

Hi all,
I cannot find anywhere in the last release 2.0.5 any option like this, does it exist ?
The INT-1721 says it has been solved.

Can you please help ? before i code it myself :)

thank you very much

[oleg.zhurakousky]

Are you saying that it doesn't work for you. I mean right now it should override an existing file.

[munger]

Thank you very much for you quick response, i tough there were a new parameter to set, but if say it is totally transparent, it's ok for me.
Sorry not to have tested carefully before, i was expecting some overwrite file parameter.
Thanks

[munger]

well it doesn't seem to work for me:
4400

after writing file with the same name, i still have two files (one with .writing extension, and the old one)
I have no error in log file.
Can you help ?

thanks

[oleg.zhurakousky]

Can you describe your environment and share your config so we can try to reproduce it?

[munger]

Sure:

- windows 7 Pro SP1
- all spring integration components updated to 2.0.5

I am using a home made component which is inheriting from an AbstractPollingEndpoint implementing MessageHandler DisposableBean and MessageSource<File>, which have been working with no particular problem for a few years.

Here is how i initialize the writer part:


if (outputDirectory != null) {
relativePath = conf.getString(getName() + ".output.relative.path","");
filenamePattern = conf.getString(getName() + ".output.filename.pattern","");

ouputFile = new File(outputDirectory + "/" + Utils.now(relativePath));
if (!ouputFile.exists()) {
logger.debug("Creating directory " + ouputFile);
(new File(ouputFile.getPath())).mkdirs();
} else if (!ouputFile.canWrite()) {
logger.debug("Trying to set directory readable " + ouputFile);
(new File(ouputFile.getPath())).setWritable(true);
}
writer = new FileWritingMessageHandler(ouputFile.getAbsoluteFil e());
writer.setAutoCreateDirectory(true);

writer.setChannelResolver(ApplicationContextProvid er.getChannelRegistry());
writer.setBeanFactory(ApplicationContextProvider.g etApplicationContext());
writer.setRequiresReply(false);

String channel = BeansNormalizedNames.getCoreRouterInputChannel();
writer.setOutputChannel((MessageChannel) ApplicationContextProvider.getChannelRegistry().ge tChannel(channel));
writer.afterPropertiesSet();

logger.info("writing in " + ouputFile.getAbsolutePath());
}

And the corresponding configuration file:


USER_PLATE_XML_FILE_WRITER.type = FILE
USER_PLATE_XML_FILE_WRITER.trigger = 10000
USER_PLATE_XML_FILE_WRITER.messages.per.poll = 10
USER_PLATE_XML_FILE_WRITER.output.directory = C:/Users/max/Documents/AndromasTest/plates
USER_PLATE_XML_FILE_WRITER.output.relative.path = yyyy/MM/dd

Just tell me if this is not clear...

[oleg.zhurakousky]

Why are you not using <int-file:outbound-channel-adapter>? I mean why do you need custom component?

[munger]

Because they can be dynamiccally instanciated at runtime.

[oleg.zhurakousky]

Ok, since you are creating it yourself, would you be able to compile a test case that reproduces this issue and attach it to this forum and I'll take a look?

[munger]

oh, it seems the .writing file is overwritten correctly, but not the original !

[munger]

so, to sum up :

first time file created correctly with filename
second time, second file created called : filename.writing
then after : file filename.writing overwritten but original file never changes

thanks

[munger]

Hi Oleg,

Please find an example of what i've described just in previous post.
Here is my spring integration XML file (at least the file writing part) :


<bean id="toStringTransformer" class="org.springframework.integration.xml.transformer.Re sultToStringTransformer" />
<integration:channel id="toDiskChannel"></integration:channel>
<integration:channel id="testOutputChannel"></integration:channel>

<si-xml:marshalling-transformer marshaller="testResultMessageMarshaller"
output-channel="toDiskChannel"
input-channel="testOutputChannel"
result-transformer="toStringTransformer" />

<file:outbound-channel-adapter channel="toDiskChannel"
directory="${test.output.directory}">
</file:outbound-channel-adapter>

Then in my code, when i send messages to this channel:


Message<GenericXMLMessageType> mess = MessageBuilder.withPayload(msg.getPayload()).
copyHeaders(fileContents.getHeaders()).
build();

MessageChannel channel2 = (MessageChannel)ApplicationContextProvider.getAppl icationContext().getBean("testOutputChannel");
channel2.send(mess);

The first is written correctly, and the following message with identical filename produce a .writing file, which is correctly overwritten then (with following messages) :

4444


Thank you very much :)

Max

[munger]

Hi again,

Sorry not coming with a full compiled test case, but i've looked at the source code in the spring-integration-file package, and i think the problem may come from that the rename function is not working, but its return value is not processed here :



private File handleFileMessage(File sourceFile, File tempFile, File resultFile) throws IOException {
if (this.deleteSourceFiles) {
if (sourceFile.renameTo(resultFile)) {
return resultFile;
}
if (logger.isInfoEnabled()) {
logger.info(String.format("Failed to move file '%s'. Using copy and delete fallback.",
sourceFile.getAbsolutePath()));
}
}
FileCopyUtils.copy(sourceFile, tempFile);
tempFile.renameTo(resultFile);
if (this.deleteSourceFiles) {
sourceFile.delete();
}
return resultFile;
}

private File handleByteArrayMessage(byte[] bytes, File originalFile, File tempFile, File resultFile) throws IOException {
FileCopyUtils.copy(bytes, tempFile);
tempFile.renameTo(resultFile);
if (this.deleteSourceFiles && originalFile != null) {
originalFile.delete();
}
return resultFile;
}

private File handleStringMessage(String content, File originalFile, File tempFile, File resultFile) throws IOException {
OutputStreamWriter writer = new OutputStreamWriter(new FileOutputStream(tempFile), this.charset);
FileCopyUtils.copy(content, writer);
tempFile.renameTo(resultFile);
if (this.deleteSourceFiles && originalFile != null) {
originalFile.delete();
}
return resultFile;
}


What about deleting src file first and then renaming the tempFile, if first rename failed (returned false) ?

Thanks for your help

Max

[SARAL SAXENA]

have to see first ..!!

[munger]

What do you wanna see ? :)

[munger]

Problem solved re-writing and subclassing FileWritingMessageHandler like follows :



@Override
protected File handleStringMessage(String content, File originalFile, File tempFile, File resultFile) throws IOException {
getLogger().debug("handling string message");
OutputStreamWriter writer = new OutputStreamWriter(new FileOutputStream(tempFile), this.getCharset());
FileCopyUtils.copy(content, writer);
renameTo(tempFile, resultFile);
if (this.isDeleteSourceFiles() && originalFile != null) {
originalFile.delete();
}
return resultFile;
}



private void renameTo(File tempFile, File resultFile) {
if (resultFile != null && resultFile.exists())
{
resultFile.setWritable(true,false);
resultFile.delete();
if (!tempFile.renameTo(resultFile))
getLogger().debug("rename KO from " + tempFile.getAbsolutePath() + " to " + resultFile.getAbsolutePath());
}
else
{
if (!tempFile.renameTo(resultFile))
getLogger().debug("rename KO from " + tempFile.getAbsolutePath() + " to " + resultFile.getAbsolutePath());

}
}


Thanks for you help. :)

[oleg.zhurakousky]

@munger

Could you please rais a jIRA issue for that?

[munger]

Done : INT-2278
Thanks !