Spring Integration | FTP - FILE Adapter Issue

[shriramg]

The application which i am working on has an Integration Layer build using Spring Integration 2.5, It tries to pull a CSV file from FTP server using FTP adapter and drops it on to local server which is polled by
File adapter. So as soon as file is downloaded by ftp the file adapter passes it on to Splitter wherein the CSV file is split into separate records and send to downstream for further processing.

The issue occurs which splitter is some times does not see the file although the file is downloaded. Below are the config details and that piece of code from splitter which tried to check if the file exists or throw user defined exception.

<ftp:inbound-channel-adapter id="ftpInbound" channel="ftpChannel"
		session-factory="ftpClientFactory"
		filename-regex="${ftp.server.remotefilename}"
		auto-create-local-directory="true" delete-remote-files="false"
		remote-directory="${ftp.server.remotefolder}"
		local-directory="file:${ftp.server.localfolder}">
		<si:poller ref="FTPPoller"></si:poller>
	</ftp:inbound-channel-adapter>
	<si:channel id="ftpChannel">
		<si:queue />
	</si:channel>

	<file:inbound-channel-adapter id="fileInboundAdapter"
		channel="fileIn" directory="file:${ftp.server.localfolder}"
		filename-pattern="${ftp.server.localfilename}"
		auto-create-directory="true" prevent-duplicates="false">
		<si:poller ref="filePoller"></si:poller>
	</file:inbound-channel-adapter>

	<si:channel id="fileIn" />

	<si:splitter input-channel="fileIn" output-channel="csvRowsIn"
		ref="inboundFileRowSplitter" method="extractRows" />

public class InboundFileRowSplitter {

    @Splitter
    public Collection<String[]> extractRows( @Headers
    Map<String, Object> headers, File keywordFile ) {
        ..................................................
        ..................................................
        ..................................................
        ..................................................
        ..................................................
        
        return processKeywordsFile( keywordFile );
    }

 
    private Collection<String[]> processKeywordsFile( File keywordFile ) {

       try {
            if (keywordFile.exists()) {
         	..................................................
         	..................................................
         	..................................................
         	..................................................
         	
            } else {
                throw new ResourceLoaderException( "............................" );
            }
            
        } catch (Exception e) {
            ..................................................
        } finally {
           ..................................................
        }
       
    }

[shriramg]

Anybody please reply since its an urgent issue.

[Gary Russell]

First, it is not clear why you are using an ftp inbound adapter AND and file inbound adapter; the ftpChannel will eventually fill up because you don't have a consumer on it.

Second, the inbound adapter uses an InboundFileSynchronizer which doesn't detect changed files; it won't fetch the same file twice.

Consider using outbound gateways to 'ls' and 'get' the files, as shown in the gateway sample...

https://github.com/SpringSource/spri...ster/basic/ftp

[shriramg]

Thanks Gary for replying this thread.

My assumption(may be wrong) came out of spring docs http://static.springsource.org/sprin.../html/ftp.html section 13.3 where they do not mention a need for consumer and in my this case the ftpchannel queue acts as receiver of the file message where it gets queues and eventually gets cleared once the file is downloaded.

Also Gary, In first place if FTP channel adapter would have been filled up then it would'nt have downloaded the file and File adapter would never see the file and thus not get triggered but our case it gets triggered and goes up to InboundFileSplitter.

Also can you give more information on FTP channel getting filled up.

[Gary Russell]

First, you have an ftp channel adapter synchronizing files with the local directory; it then sends a message to your channel named "ftpChannel", which is a QueueChannel that has no subscribers - hence it will keep consuming memory.

Second. you then have a file channel adapter that detects the presence of the file and sends (another) message to your channel named "fileIn". That channel has a subscriber (the splitter).

You don't need the file adapter, simply feed the output of the ftp adapter to the splitter. Or, as I said above, if you need more sophistication than the file synchronizer provides, use an ftp outbound gateway.

[shriramg]

Thanks Gary for clarification, but one question keeps hovering my mind if FTP was filling up the memory it would have not downloaded the file in the first place and not got to splitter via file adapter? or atleast given some exception related to ftp getting filled up. Any comments on that? Any tips on re-reproducing this issue?

I will remove the file adapter as suggested by you and feed ftp directly to splitter hoping that there is no side effect this on my application.