File Inbound Channel Adapter with many folders

[nico_b2]

Hi,

I'm currently thinking about using Spring Integration for my project. We need to parse FTP folders (8000 subfolders) to find for a specific type of file and then launch a call to a webservice, based on the file content. We will receive about a thousand files a day and we plan to poll the folders every minute.

Basically, the folder structure looks like this :

rootFTP/account1/IN/matchingFile
rootFTP/account2/IN/matchingFile
...
rootFTP/accountn/IN/matchingFile

It works perfectly (using the RecursiveLeafOnlyDirectoryScanner provided with SI) in a Unit Test context but I'm a bit afraid of the performance (I/O, CPU, memory) when we'll go live.

Should I worry about it ?

Many thanks for your help,

[oleg.zhurakousky]

I consider this somewhat of a rhetorical question although I will answer it anyway
Of course you should!
It is NOT about Spring or Spring Integration or any other framework you might be using. It is about your system and SLA specified for your system and no metter what anyone can tell you (fast or slow) the only true and comforting answer to your question can only come from you after you perform a load-test in the environment that resembles the production (e.g., hardware, software, network bandwidth etc.).

From our perspective we have not yet had any serios complaints regarding the performance. 1000 files per day sounds like 1000 messages per day. We can do that on the cheap 10 yo laptop, however as I said there is many more moving parts in any systems and the only answer is TEST.
Cheers

[nico_b2]

Hey Oleg,

Thank you for your reply. I notice that I didn't ask correctly what I was looking for. Of course we will do a load-test campain on our an environment matching the production and that's the only real way to ensure the correct perf of our solution.

In fact, I was much more asking for a feedback about the RecursiveLeafOnlyDirectoryScanner since I'm using it on a vast number of folders (and a certain number of files either). The Javadoc of this class says that "scanner should not be used with directories that contain a vast number of files or on deep trees". I was just wondering what was behind the "vast number of files".

Anyway, i'll keep you inform on the result of our load-test run.

[iwein]

That intentionally vague formulation in the javadoc is mine. I never did an exhaustive performance check for the RecursiveLeafOnlyDirectoryScanner, but if I say "vast number of files" I'm not talking about a few thousand. 1000 files in even less directories shouldn't be a problem on a modern system, if it is we'd like to know of course.

[timthetrain]

Hi,
How did you manage to plug in RecursiveLeafOnlyDirectoryScanner into ftp:inbound-channel-adapter?

It seems that only FileReadingMessageSource will take a scanner.

Would really like to see the codes you used to do it. Feel I must be missing something.
Thanks in advance,
Tim T.