Infinite reading of files with max-messages-per-poll=-1 and prevent-duplicates=false

[yozh-tema]

Hi, I'm trying to read files from the directory. If file cannot be processed it stays there to be tried later. I want to try to process as many files as possible per each poll.

This is configuration:

<int-file:inbound-channel-adapter prevent-duplicates="false" directory="${integration.iftsta.retryFolder}" channel="iftstaRetryChannel">
    <int:poller max-messages-per-poll="-1" fixed-rate="3600000" />
</int-file:inbound-channel-adapter>

The problem is that this configuration causes infinite loop on reading the files in "retryFolder" over and over again. Similarly, if max-messages-per-poll set to, say 10, then each poll will return exactly 10 messages, even if there is only 1 file (i.e. all 10 messages will be the same).

While this is in some way expected behavior (documentation states that poller in this scenario may be "potentially spinning in the infinite loop if the implementation of the custom method of the MessageSource has a potential to never return null") I wonder if there is a workaround to achieve described goal - i.e. to leave polled files in place to try them later and to avoid redundant messages?

[Mark Fisher]

You have "prevent-duplicates" set to FALSE. That removes the filter that would normally avoid picking up the same File more than one time.

That setting is aimed at a use-case such as polling once every 10 minutes to grab File(s) where the name is always the same. For more common use-cases, it's best to either keep the default filtering or at least add your own filter implementation to override the default.

Hope that helps.
-Mark

[yozh-tema]

Hello Mark, thank you for your suggestion.

Consider following situation:
- files are removed from directory only when they are successfully processed;
- success depends on external information - f.e. certain data in database should be present;
- so files are polled, tried to process and maybe left where they are - until the next poll;
- ideally, each poll should check all the files in directory - so using max-messages-per-poll=-1;
- the previous point will result in that poller will call FileReadingMessageSource's receive method until it returns null (let's call it one "poll session") - and it will not ever unless we're using AcceptOnceFileListFilter or its derivative;
- but if we're using AcceptOnceFileListFilter than next poll session will not pass any of the files that were already polled before.

So filter should pass same filenames more than once. At the same time it should reject them during the same "poll session" in order to break infinite loops.
The only thing that comes to mind is to extend AcceptOnceFileListFilter and to clear its internal queue in between file poll sessions - i.e. if file reading is triggered by cron every 10 mins then clearing it every 5+10*n minutes. But it looks too complicated for what seems to be rather common task. Or is it not? Any ideas would be appreciated.