Problem with FTP inbound-channel-adapter

[mercury1231]

I have a scenario in which the ftp inbound-channel-adapter does not quite fit.

Basically what I'm trying to do is monitoring the folder on the ftp server, and whenever there is a new zip/gzip file posted, ftp adapter downloads it to a local folder and put it into a input-channel. Then the ftp adapter deletes the remote file.

However, the problem here is that the zip/gzip files are usually quite large, say 10M ~ 100M, and I had to scan the ftp remote folder frequently in order to fetch the updates in time. Occasionally it happens that the server-side hasn't finished writing outputs to the ftp file, but my client side ftp adapter already becomes aware of the existence of this file and begins to download it. This results in a partial zip/gzip file and oftentimes leads to IOException in local processing.

Is there any convenient fix to this problem? Is it possible that I can write my own filter for ftp inbound-channel-adapter, just like for file adapter?

[oleg.zhurakousky]

Yes you can and then you can configure it via the 'filter' attribute in the adapter.

<int-ftp:inbound-channel-adapter 
				channel="ftpChannel" 
				client-factory="ftpClientFactory"
				charset="UTF-8"
				auto-create-directories="true"
				auto-delete-remote-files-on-sync="true"
				filename-pattern=".?txt"
				filter="entryListFilter"
				local-working-directory=".">
	<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>

the filter should implement org.springframework.integration.file.filters.FileL istFilter.
We also provide a convenient implementation org.springframework.integration.ftp.filters.FtpPat ternMatchingFileListFilter

Let us know if that solves your issue.

[mercury1231]

Em... I was using <ftp:inbound-channel-adapter />. How can I get to using <int-ftp:inbound-channel-adapter />?

[Gary Russell]

Em... I was using <ftp:inbound-channel-adapter />. How can I get to using <int-ftp:inbound-channel-adapter />?

For the prefix (int-ftp Vs ftp) it depends on your xmlns:xxx entry at the top of the file.

If you have xmlns:ftp then use <ftp:...>

If you have xmlns:int-ftp then use <int-ftp:...>

etc. etc.

The prefix makes no difference, it's the namespace URI that matters.

[Proner]

I reopen this thread because I encounter almost the same issue and was not able to resolve it even with the given response.

<int-ftp:inbound-channel-adapter
            channel="startCrawlingChannel"
            session-factory="ftpClientFactory"
            charset="ISO-8859-1"
            auto-create-local-directory="true"
            filename-pattern="*.xml"
            remote-directory="incoming"
            delete-remote-files="true"
            local-directory="queue">
        <int:poller max-messages-per-poll="1024" fixed-rate="100" />
    </int-ftp:inbound-channel-adapter>

At the time of testing, I've got an FTP server on my own machine. The thing is that when the ftp inbound scan the directory, it will retrieve files as soon as I drop them. This result in a 0k file in my local directory.
I can't find a way to manage this issue which seems the same as the one originally describe (but not only with a large file concern).

How can a filter prevent such a thing ? Does this finally rely on FTP architecture himself or can the int-ftp:inbound-channel-adapter handle this case in any manner ?

[Gary Russell]

This is a well-known issue with any ftp client.

The most common solution is to have the server write the file to a different directory (or with a different filename) and then rename it, so the filter only sees "complete" files.

The Spring Integration outbound adapter uses a similar technique - creates the file as /foo/bar.writing and then renames it to /foo/bar after the write is complete.