SFTP and Spring Batch integration

[vgupt007]

Hi,
In our project we have a requirement where we transfer files from a SFTP server to local and then process the files in batch program. To acheive this we are using spring integration.

We are able to transfer the file from SFTP to local but MultiResourceItemReader in the batch process complains about "No resource to Read"

I am able to execute the steps seperately(unit test) but not as integrated one.
Please let me know what I am missing.


<beans >
<context:component-scan base-package="com.isg" />
<task:executor id="task" pool-size="10" queue-capacity="100"
rejection-policy="CALLER_RUNS" />

<!-- Various Channel for various messages -->
<int: publish-subscribe-channel id="job-operator"
task-executor="task" />
<int: publish-subscribe-channel id="job-request"
task-executor="task" />
<int:logging-channel-adapter channel="job-operator" />
<int:channel id="receiveChannel" />

<bean id="sftpSessionFactory" class="org.springframework.integration.sftp.sessio n.DefaultSftpSessionFactory">
<property name="host" value="remoteServerValue"/>
<property name="port" value="22"/>
<property name="user" value="sftpuser"/>
<property name="password" value="sftppassword"/>
</bean>


<int-sftp:inbound-channel-adapter id="sftpInbondAdapter" channel="receiveChannel" session-factory="sftpSessionFactory"
local-directory="D:\file" remote-directory="/d09/testfeed/IN/test/"
auto-create-local-directory="true" delete-remote-files="false" filename-pattern="*.csv"
/>

<int: poller id="cronPoller" default="true" task-executor="task">
<int:cron-trigger expression="0 56 16 ? * *" />
</int: poller>


<int:service-activator input-channel="receiveChannel" output-channel="job-request">
<bean class="com.isg.integration.JoblaunchServiceActivat er" />

</int:service-activator>

<int:service-activator input-channel="job-request" output-channel="job-operator">
<bean
class="org.springframework.batch.integration.launc h.JobLaunchingMessageHandler">
<constructor-arg ref="jobLauncher" />
</bean>
</int:service-activator>

<job id="GFS_0685_IN_001" xmlns="http://www.springframework.org/schema/batch">



<step id="ProjectMaster">
<tasklet>
<chunk reader="projectMasterReader" processor="projectMasterItemProcessor"
writer="itemWriter" commit-interval="10" />

</tasklet>
</step>
</job>

<bean id="projectMultiResourceReader"
class="org.springframework.batch.item.file.MultiRe sourceItemReader" lazy-init="true" >
<property name="resources"
value="file:/fileInput/Test.csv" />
<property name="strict" value="true"></property>

<property name="delegate" ref="itemReader" />
</bean
<bean id="itemWriter" class="org.springframework.batch.item.database.Jpa ItemWriter">
<property name="entityManagerFactory" ref="entityManagerFactory" />

</bean>

</beans>

[Marten Deinum]

Please use [ code][/code ] tags when posting code, that way it remains readable.

Have you checked your configuration, your integration and batch part use different directories.
integration

<int-sftp:inbound-channel-adapter id="sftpInbondAdapter" channel="receiveChannel" session-factory="sftpSessionFactory" local-directory="D:\file" remote-directory="/d09/testfeed/IN/test/" auto-create-local-directory="true" delete-remote-files="false" filename-pattern="*.csv" />

batch

<bean id="projectMultiResourceReader"
class="org.springframework.batch.item.file.MultiRe sourceItemReader" lazy-init="true">
  <property name="resources" value="file:D/fileInput/Test.csv" />
  <property name="strict" value="true"/>
  <property name="delegate" ref="itemReader" />
</bean>

Also you probably want to use binding to get the filename/resource from the joblauncher instead of hardcoded like it is now.

[vgupt007]

Thanks for your reply Marten. In actual configuration files contains right dir.

It was editing mistake.

Actual usecase is : Once all files downloaded from remote sftp server , it should trigger the batch process which will process all downloaded files from local dir .

[Marten Deinum]

Then please post your actual configuration. What I see in your configuration there is no trigger to process the downloaded files, only one file which is probably not located.

[Prabhabati Moharana]

Hi
In my project, I need to do a SFTP file copy (from SFTP server to local directory) using spring batch. I tried the way provided in this post. I am stuck at a point .. the detail about reader writer is not specified in the post. I am posting my configuration file
If anyone can help me in my configuration,it will be really great.

<?xml version="1.0" encoding="UTF-8"?>
<beans 
    xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xmlns:aop="http://www.springframework.org/schema/aop"
	xmlns:tx="http://www.springframework.org/schema/tx"
	xmlns:context="http://www.springframework.org/schema/context"
	xmlns:util="http://www.springframework.org/schema/util" 
	xmlns:batch="http://www.springframework.org/schema/batch"
	xmlns:int="http://www.springframework.org/schema/integration" 
	xmlns:int-sftp="http://www.springframework.org/schema/integration/sftp"	
	xmlns:task="http://www.springframework.org/schema/task"


	xsi:schemaLocation="
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
    http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
    http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
    http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
    http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.0.xsd  
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
   http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd
    http://www.springframework.org/schema/integration/sftp http://www.springframework.org/schema/integration/sftp/spring-integration-sftp-2.0.xsd">
    
  <context:component-scan base-package="com.batch.fromdb" />   
<task:executor id="task" pool-size="10" queue-capacity="100" rejection-policy="CALLER_RUNS" />

<!-- Various Channel for various messages -->
<int:publish-subscribe-channel id="job-operator" task-executor="task"/>
<int:publish-subscribe-channel id="job-request"  task-executor="task"/>
<int:logging-channel-adapter channel="job-operator"/>

<int:channel id="receiveChannel" />

<bean id="sftpSessionFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
  <property name="host" value="122.12.34.32"/>
  <property name="user" value="bpm"/>
  <property name="password" value="aBc4"/>
  <property name="port" value="22"/>
</bean>


<int-sftp:inbound-channel-adapter id="sftpInbondAdapter" channel="receiveChannel" session-factory="sftpSessionFactory" 
local-directory="C:\file" remote-directory="/home/bpmmpf/" 
auto-create-local-directory="true" delete-remote-files="false" filename-pattern="*.txt" /> 

<int:poller id="cronPoller" default="true" task-executor="task">
<int:cron-trigger expression="0 56 16 ? * *" />
</int:poller>


<int:service-activator input-channel="receiveChannel" output-channel="job-request">
<bean class="com.batch.fromdb.FileProcessor" />

</int:service-activator>

<int:service-activator input-channel="job-request" output-channel="job-operator">
<bean
class="org.springframework.batch.integration.launch.JobLaunchingMessageHandler">
<constructor-arg ref="jobLauncher" />
</bean>
</int:service-activator>

<job id="simplejob" xmlns="http://www.springframework.org/schema/batch">

<step id="ProjectMaster">
<tasklet>
<chunk r />

</tasklet>
</step>
</job>

<bean id="jobLauncher"
		class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
		<property name="jobRepository" ref="jobRepository" />
</bean>


<bean id="dataSource"
		class="org.springframework.jdbc.datasource.DriverManagerDataSource">
		<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
		<property name="url" value="jdbc:oracle:thin:@tkdp2datdbcc05-vip.hk.hsbc:1521:lacmb011" />
		<property name="username" value="cmbloan" />
		<property name="password" value="hsbc123#" />
	</bean>
 <bean id="jobRepository"
		class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" >
		<property name="transactionManager" ref="transactionManager" />
	</bean>
	
	<bean id="transactionManager"
		class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
		<property name="dataSource" ref="dataSource" />
	</bean>

</beans>

If possible please provide the sample code .

Thanks