File Polling in spring Integration

[life1style1]

Hi,

I am very new to spring integration.Never done any implementation in it.I have a requirement to get the newest file created in a directory.
Can this be achieved using file polling?If yes then how to do this.I followed this tutorial
http://blog.springsource.com/2009/02/13/982/ but not able to co-relate it with file polling.
As i said i am a newbie it would be great if someone shows me the configuration required to this.

Regrds,
Souvik

[Gary Russell]

Take a look at the samples here https://github.com/SpringSource/spri...ration-samples - there's a file sample under 'basic' that shows file polling.

[life1style1]

Thanx Gary for your quick reply.Have gone through the sample.
Just need to understand whether file polling is enough to get the newest file created in a directory.

[Gary Russell]

I am not sure I fully understand what you mean by "newest".

If you mean that if, since the last poll, say 10 files showed up, but you only want to process the newest one (and ignore the other 9) then you would need to provide a custom FileListFilter. See the reference documentation http://static.springsource.org/sprin...ml#ftp-inbound for more information.

public interface FileListFilter<F> {

    /**
     * Filters out files and returns the files that are left in a list, or an
     * empty list when a null is passed in.
     */
    List<F> filterFiles(F[] files);

}

If you mean you want to find all new files then, yes, the standard filter will do that, but you will need to delete processed files so they won't be picked up if the JVM is stopped and restarted.

[life1style1]

Thanks Gary.

I followed the example and came across a issue.
Say in the input directory a file of around 2GB is being copied and the poller will immediately start polling the file before the copy operation in the input directory completes.I want to restrict the polling of the input file untill and unless the entire copying of the input file completes.Any idea how can this be done?

[Gary Russell]

That is a classic problem when transferring files.

There are a number of ways to solve it, including...

1. Upload the file with a temporary extension (eg abc.txt.writing) and rename the file when it is complete - ensure the file filter doesn't 'see' ...writing files
2. Upload a second empty file when done (abc.txt.complete) - use a custom filter to detect ...complete files and return the main file.
...

If you don't have control over the client, you might have to check the last updated timestamp, and have the custom filter only "find" the file if it was last updated, say, 10 minutes ago.

[life1style1]

Gary,

I was going through this link earlier shared by you :

http://static.springsource.org/sprin...ml#ftp-inbound

It says something like this :

As a user in most cases you are only interested in files that are complete and would like to filter only files that are complete. To handle these scenarios you can use the filtering support provided by the filename-pattern, filename-regex and filter attributes

Any idea whether using filename-pattern/filename-regex will solve the purpose.

[life1style1]

Hi Gary,

I am using a customFilter to handle the issue,and now i am having a different problem.
It is continuing to poll the already processed files in the directory earlier whenever i am using the customFilter.
Is there a procedure to restrict the polling to only once for a file which is already processed by the poller?


-Souvik

[Gary Russell]

You can steal the same logic from the default 'AcceptOnceFileListFilter'.

Bear in mind, though, that it only works while the JVM is running, if you restart the app, it finds the files again - you need to rename or remove the file when you have finished processing.

[life1style1]

Gary,

I have used the property prevent-duplicates="true" to restrict the polling only once for file which is already processed.
Now i have a requirement to use multiple directory in input channel.
My question is is it possible to configure multiple directories in a single inbound channel or to have multiple channels.
Also if i configure multiple inbound channels how will a single service-activator handle that.

Regards,
Souvik.