ilj
Jul 1st, 2012, 09:01 AM
hi, guys
i'm working on a large project, that heavily uses Spring Framework, especially Integration. several modules of our project download files via SFTP and process them in different ways. i had Spring Integration with very little Java code for some of the workflows our business people wanted, but then i got this one:
user drops a file into remote location.
scheduled job downloads the file and processes it - parses content and puts some data into DB. file name determines the use case.
when data needs to be updated, user edits the original file on remote location.
scheduled job detects the change and downloads and processes only the changed files.
this looks very simple from the user perspective. also seemed pretty easy to implement - save file names and their modification times into DB and check, whether we have already processed this change or not, download only the changed files and voila! but, i haven't found the way to do this via Spring Integration means.
what i have encountered, can be split into 3 separate problems:
i can't make AbstractInboundFileSynchronizingMessageSource process the file with same name twice or more, because you have AcceptOnceFileListFilter hard-coded. thus, i can't process a file without some kind of timestamp in its name.
i can't make AbstractInboundFileSynchronizingMessageSource process all the files i got during last remote directory synchronization, whatever the number was, but not try to synchronize again until the next poll, (see AbstractPollingEndpoint$Poller.run and AbstractInboundFileSynchronizingMessageSource.rece ive methods). thus, i either have a limited number of files processed per poll, or have to synchronize and process the same files over and over, until i'm sure i have processed enough.
i can't make AbstractInboundFileSynchronizer/SftpInboundFileSynchronizer to preserve modification date of files being synchronized.
thus, i had to fall back from plain and simple
<int-sftp:inbound-channel-adapter>
<int:poller/>
</int-sftp:inbound-channel-adapter>
to
<int:inbound-channel-adapter>
<bean class="MySource">
<constructor-arg name="synchronizer">
<bean class="MySynchronizer">
<constructor-arg name="sessionFactory" ref=""/>
</bean>
</constructor-arg>
</bean>
<int:poller max-messages-per-poll="1000"/>
</int:inbound-channel-adapter>
NOTE: i omitted all the attributes that do not relate to my problems. it's here just to describe the solution.
so, i implement the basic interfaces almost the same ways you have already done, just making it work the way i described. it works fine, but i'm not satisfied at all with such a solution.
so, my question is - have i missed some nice and easy way to implement this using Integration or you have not considered any of these cases useful and do not support them?
i'm working on a large project, that heavily uses Spring Framework, especially Integration. several modules of our project download files via SFTP and process them in different ways. i had Spring Integration with very little Java code for some of the workflows our business people wanted, but then i got this one:
user drops a file into remote location.
scheduled job downloads the file and processes it - parses content and puts some data into DB. file name determines the use case.
when data needs to be updated, user edits the original file on remote location.
scheduled job detects the change and downloads and processes only the changed files.
this looks very simple from the user perspective. also seemed pretty easy to implement - save file names and their modification times into DB and check, whether we have already processed this change or not, download only the changed files and voila! but, i haven't found the way to do this via Spring Integration means.
what i have encountered, can be split into 3 separate problems:
i can't make AbstractInboundFileSynchronizingMessageSource process the file with same name twice or more, because you have AcceptOnceFileListFilter hard-coded. thus, i can't process a file without some kind of timestamp in its name.
i can't make AbstractInboundFileSynchronizingMessageSource process all the files i got during last remote directory synchronization, whatever the number was, but not try to synchronize again until the next poll, (see AbstractPollingEndpoint$Poller.run and AbstractInboundFileSynchronizingMessageSource.rece ive methods). thus, i either have a limited number of files processed per poll, or have to synchronize and process the same files over and over, until i'm sure i have processed enough.
i can't make AbstractInboundFileSynchronizer/SftpInboundFileSynchronizer to preserve modification date of files being synchronized.
thus, i had to fall back from plain and simple
<int-sftp:inbound-channel-adapter>
<int:poller/>
</int-sftp:inbound-channel-adapter>
to
<int:inbound-channel-adapter>
<bean class="MySource">
<constructor-arg name="synchronizer">
<bean class="MySynchronizer">
<constructor-arg name="sessionFactory" ref=""/>
</bean>
</constructor-arg>
</bean>
<int:poller max-messages-per-poll="1000"/>
</int:inbound-channel-adapter>
NOTE: i omitted all the attributes that do not relate to my problems. it's here just to describe the solution.
so, i implement the basic interfaces almost the same ways you have already done, just making it work the way i described. it works fine, but i'm not satisfied at all with such a solution.
so, my question is - have i missed some nice and easy way to implement this using Integration or you have not considered any of these cases useful and do not support them?