Listing contents of current directory on remote server

[DavidBull]

Hello,

I'm new to Spring integration so please forgive me if I am missing something obvious.

I'm using Spring integration 2.0 to pull a file from a remote server and I want to do the equivalent of the following:

ftp remoteServer
cd remoteDiretory
ls

In my Spring config, I have used this to configured the session factory:

<bean id="ftpClientFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
		<property name="port" value="22" />
		<property name="password" value="PASSWORD" />
		<property name="clientMode" value="0" />
		<property name="fileType" value="2" />
		<property name="username" value="USERNAME" />
		<property name="host" value="remoteServer" />
		<property name="remoteWorkingDirectory" value="remoteDiretory" />
		</bean>

I know that when injected into a test class, this connects successfully.

However, when I tried to configure the input channel adapter

<int-ftp:inbound-channel-adapter id="ftpInbound"
				channel="ftpChannel" 
				session-factory="ftpClientFactory"
				charset="UTF-8"
				auto-create-local-directory="true"
				delete-remote-files="false"
				filename-pattern="*.CSV"
				remote-directory=""
				local-directory="LOCALDIRECTORY">
			<int:poller cron="CRON"/>
</int-ftp:inbound-channel-adapter>

It tells me that the "remoteDirectory must not be null"

How do I specify the adapter to list the current directory?

I've tried the obvious "." but that does not seem to work

Thanks in advance, David

[DavidBull]

To answer my only question, in case it helps anybody else, I just needed to change this property:

remote-directory="*"

and it lists the files in the current directory.

[oleg.zhurakousky]

David, this is interesting and i see how it works with '*'
Although I am still surprised that it didn't work with "."

Just to clarify a little bit. The job of any inbound channel adapter is to generate message and send it to a channel. But with the protocol specific adapters it gets a little more interesting since these adapters (FTP included) perform two tasks.
1. Communicate with remote system and get protocol specific value (i.e., FTP inbound channel adapter = transfer file from the remote directory to the local directory)
2. Generate Message with the transferred file as a payload and send it to a channel identified via 'channel' attribute.

That is why they called 'channel-adapters' not just 'adapters' and if you think about the above explanation the second part takes precedence in a way where *IF* your local directory already has a file(s) it will first generate messages from these files and *only* when all files have been processed it will initiate the remote communication to get more files.

So now I wonder if '.' really didn't work for you OR it was simply a timing issue?
You are using the CRON trigger but not explicitly setting max-messages-per-poll attribute essentially defaulting to 1. This means that as soon as one file is processed it will wait for the next execution time which is dependent on your cron expression. So, what I am trying to say is that if you happened to have a file sitting in the local-directory it would process that file before it would initiate communication with remote FTP server.

I just tested locally with "." and it works so it should be scanning your home directory, so here is what I would like you to try:

When configuring poller set max-messages-per-poll="-1" which means on each poll the adapter will attempt to generate as many messages as it possibly can meaning it will process everything in the local directory, then it will connect to the remote directory transferring everything that is available there and processing them locally. . . only then the poll will stop and wait for the next execution time.

Or you can set it to some value greater then 1 (e.g., 10 which means on each poll attempt to process no more then 10 files)

The bottom line, the "." should work, however it could also be a platform specific issue.

Btw, what is your base OS and the FTP server OS? And what type of FTP server you are running?

[DavidBull]

Oleg,

thank you for the informative response: I had been clearing out the local directory each time I tested as otherwise it didn't go to the remote server (and as I've been stepping through the code in a debugger I know it has been accessing the remote server).

I now understand what I should have done (i.e. set the max-messages-per-poll to -1) and am grateful that you have pre-empted a problem that I would have come across once I needed to download more than one file!

I am currently using windows on the local server. I think the "problem" may be with the o/s on the ftp server: I cannot access this server directly (run by a third party, I'll spare you the reasons for this). For the same reason, I can't tell you the ftp server either.

When I log in using an ftp client it has a strange filename/directory structure, by which I mean "ls A.B.": not clear to me whether the "." is a simply separator or indicates a change in directory (so its not windows or linux).I'm beginning to suspect that it is equivalent to "A/B/" in linux.

If I can find out I'll add a comment FYI

Many thanks, David

[oleg.zhurakousky]

Sure, let us know as it will help us to update the documentation, thus sharing what you know with the rest of the community.
In fact i just updated the docs with notes about the poller configuration, basically what i explained above.

[DavidBull]

Hi,

I think this is related but can raise another ticket if you prefer. I upgraded to 2.0.1.SNAPSHOT to pick up the fix for INT-1663. However, in this release you have removed the "remoteWorkingDirectory" property from DefaultFtpSessionFactory.

Without being able to set this property I get the error "Could not change directory to '/'. Please check the path." when the inbound channel tries to connect to the remote server:

2010-12-20 11:35:01,509(ISO8601) [task-scheduler-1] ERROR   org.springframework.integration.handler.LoggingHandler  - java.lang.IllegalStateException: failed to create FTPClient
        at org.springframework.integration.ftp.session.AbstractFtpSessionFactory.getSession(AbstractFtpSessionFactory.java:138)
        at org.springframework.integration.file.remote.session.CachingSessionFactory.getSession(CachingSessionFactory.java:71)
        at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:118)
        at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:132)
        at org.springframework.integration.endpoint.SourcePollingChannelAdapter.doPoll(SourcePollingChannelAdapter.java:89)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:116)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:114)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:179)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
        at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:174)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.springframework.integration.MessagingException: Could not change directory to '/'. Please check the path.
        at org.springframework.integration.ftp.session.AbstractFtpSessionFactory.createClient(AbstractFtpSessionFactory.java:167)
        at org.springframework.integration.ftp.session.AbstractFtpSessionFactory.getSession(AbstractFtpSessionFactory.java:131)
        ... 21 more

2010-12-20 11:35:01,509(ISO8601) [task-scheduler-1] DEBUG   org.springframework.integration.channel.PublishSubscribeChannel  - postSend (sent=true) on channel 'errorChannel', message: [Payload=java.la
ng.IllegalStateException: failed to create FTPClient][Headers={timestamp=1292844901509, id=21a1cfd7-6c05-4adb-8909-bba5419f3618}]

Is there an alternative way of configuring the remote working directory? Alternatively, would it be possible to restore the property in the release version of 2.0.1?

Many thanks, David

[oleg.zhurakousky]

David, SI 2.0.1 has been released on Friday, but the next maintenance release is not that far away (2-3 weeks) and largely depends on feedbacks such as yours (a lot of interesting FTP servers out there and not all of them behave the same way). But for now i would suggest to change your version to 2.0.2.BUILD-SNAPSHOT to get the latest since based on your stack-trace its hard for me to determine what might be the problem since it doesn't match up to what's there.

Now to your question.

The 'remote-working-directory' attribute was removed from SessionFactory because we wanted to make configuration of all 3 FTP (FTP/FTPS/SFTP) adapters consistent and during testing we realized that by default you will be signed on to your home directory based on the security configurations of the FTP server. So as an example let's say you've been signed on to /users/david. However, your adapter configuration may have a remote-directory attribute set to "/". When you provide the full path the server should simply transfer (outbound) the file into that directory or scan (inbound) this directory for files and transfer them to local directory.

So, I am not exactly sure and as I said it is hard to determine from the stack-trace what might be the issue.

I know you are using some interesting FTP server, but it would still help if you can post your configuration (both session factory and adapter) and explain a little bit about what you are trying to accomplish, then we can see where the discrepancies are. Then we can think about adding this attribute back or other solution.

[DavidBull]

Oleg,

apologies for taking a while to get back but wanted to wait until I knew the operating system of the ftp server: it is an MVS Mainframe. Also, I'm afraid to say that the stack trace I originally listed was a red herring.

The only change to the listed configuration is that I no longer specify the remote-working directory in the ftp session factory.

I think I may have found the issue: on MVS, as discussed, the full stop '.' is used to separate folders; so the equivalent of cd temp/foobar would be cd temp.foobar

In my configuration for the adaptor, I have specified the remote directory to be:

'TEMP.FTP.*'

(the single quotes are required by MVS)

In this directory, there is a file:

'FILE.ABC'

and I am trying to set up a weekly job to pull this file.

With this configuration, I get the following error:

java.io.IOException: Failed to copy ''TEMP.FTP.*'\'TEMP.FTP.FILE.ABC''. Server replied with: 550 Name length error for pathname 'TEMP.FTP.*'\'TEMP.FTP.FILE.ABC'

(Note that I've had to obsfucate the actual file and directory names for which I can only apologise).

I think that the problem is that in the copyFileToLocalDirectory method in the AbstractInboundFileSynchronizer class the remote file path is calculated using:

String remoteFilePath = remoteDirectoryPath + File.separator + remoteFileName;

The file separator is calculated for the client (currently running on windows) and not the remote server, hence the '\' which appears in the error message and the fact that the remote file path is invalid.

I think what we need is some way of specifying the file separator on the remote server (although there may still be an issue with the required single quotes): we need the remote path to be 'TEMP.FTP.FILE.ABC'; again note the single quotes.

Also, as remoteFileName is calculated from the absolute path of the FTPFile, it is set to be 'TEMP.FTP.FILE.ABC', so why do you need to prefix the value of remote directory? Is this required for windows/linux clients? Or is this a quirk of MVS?

David

[oleg.zhurakousky]

David

One of the last minute changes prior to releasing SI 2.0.1 (last friday) was to go from

tring remoteFilePath = remoteDirectoryPath + File.separator + remoteFileName;

to

tring remoteFilePath = remoteDirectoryPath + "/" + remoteFileName;

So please upgrade to Spring Integration 2.0.1

However, reading your post it seem like this might not help you since "/" is not what is accepted as file separator for MVS based FTP server and realizing how different some of the FTP servers are, i am starting to think that we may have to do what you suggest and that is to provide some way of configuring the remote-file-system-separator.

Could you please open an issue (as New Feature)? We can implement it rather quickly, but you gonna have to use the snapshot until 2.0.2 is out.

[DavidBull]

Happy to do this but don't have permission to create an issue within your Jira installation. Or do you just need me to create a new thread within this forum?

Thanks, David