i'm working on a large project, that heavily uses Spring Framework, especially Integration. several modules of our project download files via SFTP and process them in different ways. i had Spring Integration with very little Java code for some of the workflows our business people wanted, but then i got this one:
- user drops a file into remote location.
- scheduled job downloads the file and processes it - parses content and puts some data into DB. file name determines the use case.
- when data needs to be updated, user edits the original file on remote location.
- scheduled job detects the change and downloads and processes only the changed files.
this looks very simple from the user perspective. also seemed pretty easy to implement - save file names and their modification times into DB and check, whether we have already processed this change or not, download only the changed files and voila! but, i haven't found the way to do this via Spring Integration means.
what i have encountered, can be split into 3 separate problems:
- i can't make AbstractInboundFileSynchronizingMessageSource process the file with same name twice or more, because you have AcceptOnceFileListFilter hard-coded. thus, i can't process a file without some kind of timestamp in its name.
- i can't make AbstractInboundFileSynchronizingMessageSource process all the files i got during last remote directory synchronization, whatever the number was, but not try to synchronize again until the next poll, (see AbstractPollingEndpoint$Poller.run and AbstractInboundFileSynchronizingMessageSource.rece ive methods). thus, i either have a limited number of files processed per poll, or have to synchronize and process the same files over and over, until i'm sure i have processed enough.
- i can't make AbstractInboundFileSynchronizer/SftpInboundFileSynchronizer to preserve modification date of files being synchronized.
thus, i had to fall back from plain and simple
NOTE: i omitted all the attributes that do not relate to my problems. it's here just to describe the solution.
<constructor-arg name="sessionFactory" ref=""/>
so, i implement the basic interfaces almost the same ways you have already done, just making it work the way i described. it works fine, but i'm not satisfied at all with such a solution.
so, my question is - have i missed some nice and easy way to implement this using Integration or you have not considered any of these cases useful and do not support them?