FTP inbound adapter with poller sometimes stops

[yozh-tema]

Hi, we are using ftp:inbound-channel-adapter with poller (Spiring Integration 2.0.4.RELEASE):

<int-ftp:inbound-channel-adapter channel="epoSourceChannel" local-directory="${ftp.localFolder}"
		session-factory="ftpSessionFactory" remote-directory="${ftp.remoteFolder}"
		cache-sessions="false" filter="epoFilenameFilter"> 
		<int:poller max-messages-per-poll="-1" cron="0 * 2-22 * * *" />
</int-ftp:inbound-channel-adapter>

epoFilenameFilter bean is implementation of FileListFilter. Processed files are not deleted from FTP server so we check for new files by their filenames (they contain timestamps, latest processed timestamp is stored). There are other integration elements for processing of downloaded files but they shouldn't be important here. This all works under Tomcat 6, loaded from web.xml of SWF-JSF-webapplication.

Here is the problem. Sometimes (and lately - more often, like once a week) this ftp:inbound-channel-adapter together with poller silently stops. This looks like this: every minute epoFilenameFilter logs information about processing of files. And then at one moment it just stops - logs from filter (and of course from next elements in processing chain) do not come up more. I've set DEBUG logging level and added LOG statement specifically in the beginning of filter method - so each filtering log message was prepended by "connected" message from FTP session factory The last messages were:

2012-02-05 18:17:00,053 DEBUG [task-scheduler-5] org.springframework.integration.ftp.session.DefaultFtpSessionFactory: Connected to server [ftp.server.com:21]
2012-02-05 18:17:00,591 INFO  [task-scheduler-5] com.example.integration.epoimport.filter.EpoFilenameFilter: Filtering files...
2012-02-05 18:17:00,593 INFO  [task-scheduler-5] com.example.integration.epoimport.filter.EpoFilenameFilter: Looking for ePO from 05.02.2012 17:56:37...
2012-02-05 18:17:00,598 INFO  [task-scheduler-5] com.example.integration.epoimport.filter.EpoFilenameFilter: Number of new ePO: 0
2012-02-05 18:18:00,061 DEBUG [task-scheduler-3] org.springframework.integration.ftp.session.DefaultFtpSessionFactory: Connected to server [ftp.server.com:21]

And after that integration beans do not send any log messages at all. From what I see it looks like either an infinite loop somewhere between session creation and calling of filter method or like simultaneous fail of both ftp inbound adapter and poller - but this can as well be far away from truth.

The moment is rather arbitrary and I can't find any relations to any actions/movements in application (and it's not at 22 o'clock ). For example, that happened at 16:00, 08:48 and at 18:18. Logs contain no other errors, application continues to work as usual.

Are there any suggestions as what to look at additionally to find out real cause of problem?

[Gary Russell]

I am guessing the thread (in this case task-scheduler-3) is hanging trying to read data from your server. You can confirm this by taking a thread dump using jstack or visualvm.

We don't currently expose the setDataTimeout() and setDefaultTimeout() methods on the FtpClient, so the socket SO_TIMEOUT option is not set, and so tcp will wait indefinitely for data; we probably should allow these to be set, so please open an Improvement JIRA here https://jira.springsource.org/browse/INT

[Gary Russell]

If you want to see if this solves your problem, you could try making your own session factory....

public class MyFtpSessionFactory extends AbstractFtpSessionFactory<FTPClient> {

	@Override
	protected FTPClient createClientInstance() {
		FTPClient ftpClient = new FTPClient();
		ftpClient.setDataTimeout(5000);
		ftpClient.setDefaultTimeout(5000);
		return ftpClient;
	}

}

And use an instance of that instead of DefaultFtpSessionFactory.

Of course, when these timeouts occur, you still won't get any data, but you should see log activity.

[yozh-tema]

Thank you Gary.
I raised JIRA issue https://jira.springsource.org/browse/INT-2429 (didn't find FTP component there though). I'll try to use own session factory.

[oleg.zhurakousky]

There is actually another way of solving it and is incorporated into the design. AbstractFtpSessionFactory defines two NOOP methods

/**
* Will handle additional initialization after client.connect() method was invoked, 
* but before any action on the client has been taken 
*/
protected void postProcessClientAfterConnect(T t) throws IOException {
	// NOOP
}
/**
* Will handle additional initialization before client.connect() method was invoked. 
*/
protected void postProcessClientBeforeConnect(T client) throws IOException {
	// NOOP
}

So all you need is to extend DefaultFtpSessionFactory and provide implementation of one or both of these methods. I guess for your case you only need 'postProcessClientBeforeConnect' method.

[yozh-tema]

Hi Oleg,
thanks for pointing that out, mentioned JIRA improvement is then probably redundant. I think however that these hooks should be documented in reference.