FTP Adapter

[dudleygb]

Hi guys, anyone used FTP adapter? I'm struggling to configure the adapter.

How does one set the port, username etc?

I see theres a FTPFileConfig object referenced in the DefaultFTPClientFactory, just not so clear on where what goes where. Just a pointer or two please.


I found these notes on the FTP adapter ext wiki:
I'm just not so certain how to configure the DefaultFTPClientFactory injection.


<integration:inbound-channel-adapter
channel="file-channel" ref="ftpFileSource" />

<beans:bean id="ftpFileSource"
class="org.springframework.integration.ftp.FtpFileSource">
<beans:property name="clientPool" ref="ftpClientPool" />
<beans:property name="localWorkingDirectory"
value="file:${java.io.tmpdir}/FtpConfigIntegrationTests" />
<beans:property name="taskScheduler" ref="taskScheduler" />
<beans:property name="trigger">
<beans:bean
class="org.springframework.integration.scheduling.Interva lTrigger">
<beans:constructor-arg value="1500" />
</beans:bean>
</beans:property>
</beans:bean>

<beans:bean id="ftpClientPool"
class="org.springframework.integration.ftp.QueuedFTPClien tPool">
<beans:constructor-arg>
<beans:bean class="org.springframework.integration.ftp.DefaultFTPClie ntFactory"/>
</beans:constructor-arg>
</beans:bean>

[dudleygb]

ok managed to get this, but now nothings happening, Spring just hangs...

I had to include the commons-net jar, I'll add the commons-ftp jar too and see what happens

Or perhaps my config is wrong?



for now I'm just setting the props on the DefaultFTPClientFactory like so..


<channel id="file-channel"/>
<integration:inbound-channel-adapter
channel="file-channel" ref="ftpFileSource" />

<beans:bean id="ftpFileSource"
class="org.springframework.integration.ftp.FtpFileSource">
<beans:property name="clientPool" ref="ftpClientPool" />
<beans:property name="localWorkingDirectory"
value="file:${java.io.tmpdir}/FtpConfigIntegrationTests" />
<beans:property name="taskScheduler" ref="taskScheduler" />
<beans:property name="trigger">
<beans:bean
class="org.springframework.integration.scheduling.Interva lTrigger">
<beans:constructor-arg value="1500" />
</beans:bean>
</beans:property>
</beans:bean>

<beans:bean id="ftpClientPool"
class="org.springframework.integration.ftp.QueuedFTPClien tPool">
<beans:constructor-arg>
<beans:bean class="org.springframework.integration.ftp.DefaultFTPClie ntFactory">
<beans:property name="host" value="10.182.198.333"/>
<beans:property name="port" value="23"/>
<beans:property name="username" value="user"/>
<beans:property name="password" value="password"/>
<!-- beans:property name="remoteWorkingDirectory"/-->
</beans:bean>
</beans:constructor-arg>
</beans:bean>

<integration:poller id="poller" default="true">
<integration:interval-trigger interval="10000"/>
</integration:poller>

[dudleygb]

seems it hangs at the taskScheduler





DEBUG - AbstractAutowireCapableBeanFactory$1.run(383) | Creating instance of bean 'org.springframework.integration.endpoint.EventDri venConsumer#0'
DEBUG - AbstractBeanFactory.doGetBean(214) | Returning cached instance of singleton bean 'errorChannel'
DEBUG - AbstractBeanFactory.doGetBean(214) | Returning cached instance of singleton bean 'org.springframework.integration.handler.LoggingHa ndler#0'
DEBUG - AbstractAutowireCapableBeanFactory.doCreateBean(45 9) | Eagerly caching bean 'org.springframework.integration.endpoint.EventDri venConsumer#0' to allow for resolving potential circular references
DEBUG - AbstractBeanFactory.doGetBean(214) | Returning cached instance of singleton bean 'taskScheduler'
DEBUG - AbstractAutowireCapableBeanFactory.invokeInitMetho ds(1367) | Invoking afterPropertiesSet() on bean with name 'org.springframework.integration.endpoint.EventDri venConsumer#0'
INFO - AbstractEndpoint.start(87) | started org.springframework.integration.endpoint.EventDriv enConsumer#0
DEBUG - AbstractAutowireCapableBeanFactory$1.run(411) | Finished creating instance of bean 'org.springframework.integration.endpoint.EventDri venConsumer#0'
DEBUG - AbstractBeanFactory.doGetBean(214) | Returning cached instance of singleton bean 'org.springframework.integration.channel.MessagePu blishingErrorHandler#0'
DEBUG - AbstractBeanFactory.doGetBean(214) | Returning cached instance of singleton bean 'taskScheduler'
INFO - SimpleTaskScheduler.start(137) | started org.springframework.integration.scheduling.SimpleT askScheduler@ba5bdb

[dudleygb]

narrowed it down to the fact that I also need to inject a FTPClient config. Not sure how this is done.....hmmm, trawling for docs

ok referencing the config property now but still hanging in the same place...anyone, pls help, someone must have done this before...?

[dudleygb]

now i'm really stuck, its still hanging on the TaskScheduler line as above...this is my config so far, pls help anyone...pls pls



<integration:channel id="file-channel"/>
<integration:inbound-channel-adapter
channel="file-channel" ref="ftpFileSource" />

<bean id="ftpFileSource"
class="org.springframework.integration.ftp.FtpFileSource">
<property name="clientPool" ref="ftpClientPool" />
<property name="localWorkingDirectory"
value="file:${java.io.tmpdir}/FtpConfigIntegrationTests" />
<property name="taskScheduler" ref="taskScheduler" />
<property name="trigger">
<bean class="org.springframework.integration.scheduling.Interva lTrigger">
<constructor-arg value="1500" />
</bean>
</property>
</bean>

<bean id="ftpClientConf"
class="org.apache.commons.net.ftp.FTPClientConfig">
<constructor-arg value="UNIX" />
</bean>

<bean id="ftpClientPool"
class="org.springframework.integration.ftp.QueuedFTPClien tPool">
<constructor-arg>
<bean class="org.springframework.integration.ftp.DefaultFTPClie ntFactory">
<property name="host" value="localhost"/>
<property name="port" value="21"/>
<property name="username" value="ftpuser"/>
<property name="password" value="passsword"/>
<property name="config" ref="ftpClientConf"/>
<!-- property name="remoteWorkingDirectory" value=".\"/-->
</bean>
</constructor-arg>
</bean>

<integration:poller id="poller" default="true">
<integration:interval-trigger interval="10000"/>
</integration:poller>

[iwein]

<bean id="ftpClientPool"
class="org.springframework.integration.ftp.QueuedFTPClien tPool">
<constructor-arg>
<bean class="org.springframework.integration.ftp.DefaultFTPClie ntFactory">
<property name="host" value="${oad.ftp.host}"/>
<property name="username" value="${oad.ftp.username}"/>
<property name="password" value="${oad.ftp.password}"/>
<property name="port" value="${oad.ftp.port}"/>
<property name="remoteWorkingDirectory" value="${oad.ftp.remotedir}"/>
</bean>
</constructor-arg>
</bean>


Works for me.

[dudleygb]

thanx iwein, must be something else then...
can u pls post what the ouput looks like when it connects to FTP server and copies a file pls? Its just not working for me, I have no idea what the deal is..

[iwein]

I don't have that at hand right now, but maybe you're reading from the wrong remote directory? You can also look at the local directory, the FtpInboundSynchronizer should replicate the contents of the ftp location there, or die horribly screaming about all that went wrong. If the contents is [] it could be a bit silent.

[dudleygb]

cool thanx for that tip, I placed a file in my local working directory and it is blowing some exceptions as below. Also, as a test I put in a wrong username and password in my DefaultFTPClientFactory and it gives me the same error as below. Almost as if its not actually creating the remote connection.

DEBUG - FileReadingMessageSource.scanInputDirectory(201) | Added to queue: [/tmp/FtpConfigIntegrationTests/hello.txt]
INFO - FileReadingMessageSource.receive(184) | Created message: [[Payload=/tmp/FtpConfigIntegrationTests/hello.txt][Headers={springintegration_timestamp=1253370628319 , springintegration_id=1437c49f-0995-43cb-b438-e45d24e90a5d}]]
DEBUG - AbstractMessageChannel$ChannelInterceptorList.preS end(156) | preSend on channel 'file-channel', message: [Payload=/tmp/FtpConfigIntegrationTests/hello.txt][Headers={springintegration_timestamp=1253370628319 , springintegration_id=1437c49f-0995-43cb-b438-e45d24e90a5d}]
DEBUG - AbstractMessageChannel$ChannelInterceptorList.preS end(156) | preSend on channel 'errorChannel', message: [Payload=org.springframework.integration.message.Me ssageDeliveryException: Dispatcher has no subscribers.][Headers={springintegration_timestamp=1253370628320 , springintegration_id=b1218469-e9db-438e-bdf0-9e684a115a76}]
DEBUG - AbstractMessageHandler.handleMessage(56) | org.springframework.integration.handler.LoggingHan dler@c943d1 received message: [Payload=org.springframework.integration.message.Me ssageDeliveryException: Dispatcher has no subscribers.][Headers={springintegration_timestamp=1253370628320 , springintegration_id=b1218469-e9db-438e-bdf0-9e684a115a76}]
ERROR - LoggingHandler.handleMessageInternal(83) | org.springframework.integration.message.MessageDel iveryException: Dispatcher has no subscribers.
at org.springframework.integration.dispatcher.Unicast ingDispatcher.doDispatch(UnicastingDispatcher.java :97)
at org.springframework.integration.dispatcher.Unicast ingDispatcher.dispatch(UnicastingDispatcher.java:9 0)
at org.springframework.integration.channel.AbstractSu bscribableChannel.doSend(AbstractSubscribableChann el.java:43)
at org.springframework.integration.channel.AbstractMe ssageChannel.send(AbstractMessageChannel.java:116)
at org.springframework.integration.channel.AbstractMe ssageChannel.send(AbstractMessageChannel.java:94)
at org.springframework.integration.channel.MessageCha nnelTemplate.doSend(MessageChannelTemplate.java:22 3)
at org.springframework.integration.channel.MessageCha nnelTemplate.send(MessageChannelTemplate.java:180)
at org.springframework.integration.endpoint.SourcePol lingChannelAdapter.doPoll(SourcePollingChannelAdap ter.java:78)
at org.springframework.integration.endpoint.AbstractP ollingEndpoint$Poller.innerPoll(AbstractPollingEnd point.java:232)
at org.springframework.integration.endpoint.AbstractP ollingEndpoint$Poller.poll(AbstractPollingEndpoint .java:216)
at org.springframework.integration.endpoint.AbstractP ollingEndpoint$Poller.run(AbstractPollingEndpoint. java:209)
at org.springframework.integration.scheduling.SimpleT askScheduler$ErrorHandlingRunnableWrapper.run(Simp leTaskScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:150)
at org.springframework.integration.scheduling.SimpleT askScheduler$TriggeredTask.run(SimpleTaskScheduler .java:256)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

DEBUG - AbstractMessageChannel$ChannelInterceptorList.post Send(169) | postSend (sent=true) on channel 'errorChannel', message: [Payload=org.springframework.integration.message.Me ssageDeliveryException: Dispatcher has no subscribers.][Headers={springintegration_timestamp=1253370628320 , springintegration_id=b1218469-e9db-438e-bdf0-9e684a115a76}]

[dudleygb]

Hi Iwein, I pulled the source into eclipse and debugged. getClient method on QueuedFTPClientPool is never called. I need to debug further....if any ideas come to mind, i'm open to suggestions...many thanx

[iwein]

Heh, you're sure your app context is started? I'm using mockftpserver for my unit tests (using a FakeFtpServer). That should log some NOOP and LIST calls at least.



@BeforeClass
public static void setupAndStartFtpServer() {
fakeFtpServer = new FakeFtpServer();
fakeFtpServer.setServerControlPort(8021);
fakeFtpServer.addUserAccount(new UserAccount("*****", "****",
"/"));

FileSystem fileSystem = new UnixFakeFileSystem();
fileSystem
.add(new DirectoryEntry("*****"));
fileSystem.add(new FileEntry(
"*****"));
fileSystem.add(new FileEntry(
"*****"));
fakeFtpServer.setFileSystem(fileSystem);

fakeFtpServer.start();
}

[dudleygb]

Hi Iwein, I appreciate the response, thank u.
I basically setup ftpPro on my local pc. I can ftp and logon etc with the details I'm using in the SI app context cfg. But...still no go...
As I'm debugging thru the ftpAdapter code, it look as though the Spring and commons-net ftpclient isnt instantiated correctly and getClient is never called, so an attempt to logon etc is never actually executed. Somehow, something is causing some weirdness. Perhaps there is a dependancy thats wrong...or something

Can u try without the test mocks, to access a real ftp server? What version of commons-net are u using? Can you pls let me know what other dependancies you're using?

[iwein]

I don't have time to tune down my whole project to share it at the moment, but I'm absolutely sure that BUILD-27 of the project will work (also with a real ftp server).

Can you check if afterPropertiesSet() and start() are called on the synchronizer? I'm thinking that the synchronizer's taskscheduler is simply not running. I might be able to do something there to ensure that it will always be started, at the latest when receive is called the first time on the FtpFileSource.

[dudleygb]

Hi Iwein,

Really appreciate the pointers. I downloaded sources from...

https://build.springsource.org/browse/EXT-SESIASNAPSHOT-27/artifact/individual-jars/org.springframework.integration.ftp/target/artifacts

I will check afterPropertiesSet and start methods, just debug through the app again.
I'm determined to get this adapter working, will really save us a lot of time. I've built this entire spring integration app for a large project, so, invested a great deal of time to branch/break out into an alternative way of doing things. Would really love to keep doing things the SI way. Ok lemme, so take a look again....

[dudleygb]

I don't have time to tune down my whole project to share it at the moment, but I'm absolutely sure that BUILD-27 of the project will work (also with a real ftp server).

Can you check if afterPropertiesSet() and start() are called on the synchronizer? I'm thinking that the synchronizer's taskscheduler is simply not running. I might be able to do something there to ensure that it will always be started, at the latest when receive is called the first time on the FtpFileSource.

Hi Iwein,

Yup, as u said, the synchronizer.start(); isnt called, yet the afterPropertiesSet() method is called. Anything I can do to help, with the change u mentioned above? Would really mean a lot to me to have this working. I can re-contribute in some way I'm sure.

[dudleygb]

Hi Iwein,

Yup, as u said, the synchronizer.start(); isnt called, yet the afterPropertiesSet() method is called. Anything I can do to help, with the change u mentioned above? Would really mean a lot to me to have this working. I can re-contribute in some way I'm sure.

I just force a start() after the afterPropertiesSet() method and now I am getting some activity, yipee.

Obviously, dont think calling start() inside afterPropertiesSet() is the right way?

[dudleygb]

I just force a start() after the afterPropertiesSet() method and now I am getting some activity, yipee.

Obviously, dont think calling start() inside afterPropertiesSet() is the right way?

I was still getting a publisher doesnt have subsciber error, but once I put the consumer cfg in, both directories are now being synchronized.


My cfg now looks as such:



<context:property-placeholder location="classpath:ftp.properties"/>

<integration:channel id="file-channel"/>
<integration:inbound-channel-adapter
channel="file-channel" ref="ftpFileSrc" />

<bean id="ftpFileSrc"
class="org.springframework.integration.ftp.FtpFileSource">
<property name="clientPool" ref="ftp.ClientPool" />
<property name="localWorkingDirectory"
value="file:${java.io.tmpdir}/FtpConfigIntegrationTests" />
<property name="taskScheduler" ref="taskScheduler" />
<property name="trigger">
<bean class="org.springframework.integration.scheduling.Interva lTrigger">
<constructor-arg value="1500" />
</bean>
</property>
</bean>


<bean id="ftp.ClientPool" class="org.springframework.integration.ftp.QueuedFTPClien tPool">
<constructor-arg>
<bean class="org.springframework.integration.ftp.DefaultFTPClie ntFactory">
<property name="host" value="${ftp.host}"/>
<property name="username" value="${ftp.username}"/>
<property name="password" value="${ftp.password}"/>
<property name="port" value="${ftp.port}"/>
<property name="remoteWorkingDirectory" value="${ftp.remotedir}"/>

</bean>
</constructor-arg>
</bean>
<integration:outbound-channel-adapter
ref="consumer" channel="file-channel" />

<bean id="consumer"
class="org.springframework.integration.ftp.FtpSendingMess ageConsumer">
<property name="ftpClientPool" ref="ftp.ClientPool" />
</bean>

<integration:poller id="poller" default="true">
<integration:interval-trigger interval="10000"/>
</integration:poller>
</beans>


This is cool, however.....what I'm really aiming for is to mix a queue into the deal now. What we really need is to read a file from a queue and ftp it to a remote ftp directory. Then the same thing back again, read from a remote ftp directory and place the file on a queue. Is it basically as simple as having the a jms outbound adapter configured as the consumer? I will play with this cfg's and post again once I test it.

[iwein]

Obviously, dont think calling start() inside afterPropertiesSet() is the right way?

We're doing exactly that in other parts of the framework (there is the autostart flag where you can control this). I am leaning towards the idea that not many users really appreciate the subtle difference between initializing and starting. You will find use cases where you want the application to be ready for business and then separately start or stop the flow of work in the system.

You've learned about these features the hard way, and I'm thinking that should not be required for other users.

Can you create a SESIA issue for this thread so I won't forget to fix it? http://jira.springframework.org/browse/SESIA

[dudleygb]

We're doing exactly that in other parts of the framework (there is the autostart flag where you can control this). I am leaning towards the idea that not many users really appreciate the subtle difference between initializing and starting. You will find use cases where you want the application to be ready for business and then separately start or stop the flow of work in the system.

You've learned about these features the hard way, and I'm thinking that should not be required for other users.

Can you create a SESIA issue for this thread so I won't forget to fix it? http://jira.springframework.org/browse/SESIA

Thanx Iwein, appreciate this. I will create the jira for you.
I dont mind learning the hard way, anything more I can learn about Spring, the better. I love Spring...its excellent.

[dudleygb]

ok, now i'm being asked for this exact same thing, but to be done is SSHor SFTP? Anyone have some suggestions? Is it as easy as using the FTPSClient instead of the FTPClient, or is there a better way?

[zulfiqarbaig1]

Hi,
Just wanted to know if the above problem been resolved. I am facing exactly the same problem.

[iwein]

Well, the resolution was to invoke 'start', so that should work for you too then right?

[eakinasila]

How can I upload something to a FTP, from another website without downloading it? Heres what I want to do. I have a garrys mod server, but Uploading is a pain in my *** I have a very slow upload speed. I would rather just be able to have another site download it, then upload it to my ftp for me. Is there any services online that can do that?
_______________________
affiliateelite (http://www.keywordspy.com/overview/domain.aspx?q=affiliateelite.com) ~ affiliateelite.com (http://www.keywordspy.com/overview/domain.aspx?q=affiliateelite.com) ~ adgooroo (http://www.keywordspy.com/overview/keyword.aspx?q=adgooroo) ~ adgooroo.com (http://www.keywordspy.com/overview/domain.aspx?q=adgooroo.com)