Polling multiple directories in parallel

Hi

Please provide your ideas for a robust and clean design for the following need.
There are 5 different folders, each representing an inbound data type (.txt files) in tabbed delimited format. I want to start the program that would poll the folders and if a file is available, it would be processed. Would like to process the files in parallel.

Execution steps:
1. Spring Integration - to poll for availability of files
2. If files are available, invoke a Spring Batch job
3. The batch job would read the file, process it and write it to suitable target
All the three steps to happen in parallel for the different data types.

Questions:
1. As the Service Activator would be mainly responsible for getting the File objects and launching a Spring Batch job, do I need to create multiple Service Activator configurations, one for each data type or is there any scope of reusing a single Service Activator configuration?
2. Since there is a need to process feeds in parallel, would a AsynchronousGateway help here?
3. As there are 5 different directories to be polled, I would need to create 5 different inbound-channel-adapter configurations to hold details of each of the folders and the polling frequency. Please correct me if I am wrong.

Appreciate your thoughts.

Regards
Chandra

Quote:

---

Originally Posted by Chandra_Sivaraman

Hi

Please provide your ideas for a robust and clean design for the following need.
There are 5 different folders, each representing an inbound data type (.txt files) in tabbed delimited format. I want to start the program that would poll the folders and if a file is available, it would be processed. Would like to process the files in parallel.

Execution steps:
1. Spring Integration - to poll for availability of files
2. If files are available, invoke a Spring Batch job
3. The batch job would read the file, process it and write it to suitable target
All the three steps to happen in parallel for the different data types.

Questions:
1. As the Service Activator would be mainly responsible for getting the File objects and launching a Spring Batch job, do I need to create multiple Service Activator configurations, one for each data type or is there any scope of reusing a single Service Activator configuration?
2. Since there is a need to process feeds in parallel, would a AsynchronousGateway help here?
3. As there are 5 different directories to be polled, I would need to create 5 different inbound-channel-adapter configurations to hold details of each of the folders and the polling frequency. Please correct me if I am wrong.

Appreciate your thoughts.

Regards
Chandra

---

Hi Chandra,

Even I am also looking for the solution of this ....!!! :(

Quote:

---

Originally Posted by SARAL SAXENA

Hi Chandra,

Even I am also looking for the solution of this ....!!! :(

---

Hi,

Guys please guide for this problem as this issue is really related to performance one..!!!

this helps reading files in folders, here you need only one service activator:
======================================
<bean id="recursiveScanner" class="org.springframework.integration.file.Recurs iveLeafOnlyDirectoryScanner"/>

<int-file:inbound-channel-adapter id="fromtestfolder"
directory="testFolder"
prevent-duplicates="true"
auto-create-directory="false"
scanner="recursiveScanner"

/>

<int:poller default="true" fixed-delay="1000" />

<int:service-activator input-channel="fromtestfolder" >
...
</int:service-activator>
================================================== ===============

batch job config for parallel execution:
====================================
<bean id="jobLauncher" class="org.springframework.batch.core.launch.suppo rt.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
<property name="taskExecutor">
<bean class="org.springframework.core.task.SimpleAsyncTa skExecutor" />
</property>
</bean>
==============================

Quote:

---

Originally Posted by lubo

this helps reading files in folders, here you need only one service activator:
======================================
<bean id="recursiveScanner" class="org.springframework.integration.file.Recurs iveLeafOnlyDirectoryScanner"/>

<int-file:inbound-channel-adapter id="fromtestfolder"
directory="testFolder"
prevent-duplicates="true"
auto-create-directory="false"
scanner="recursiveScanner"

/>

<int:poller default="true" fixed-delay="1000" />

<int:service-activator input-channel="fromtestfolder" >
...
</int:service-activator>
================================================== ===============

---

HI lubo,
Can you please explain the functionality of org.springframework.integration.file.RecursiveLeaf OnlyDirectoryScanner, as I have done the googling from which I have found that after certain time period it will continue scan the source directory for files , Can you please provide the detail explanation:rolleyes: of it..!! Thanks in advance..!!

Hi

Here is a way that I have found to poll for text files in multiple directories, parallelly. I have five different types of files to be polled and processed. So, I created service activator and inbound channel adapter configuration for all the five types. Filter would filter out files matching a specific regex pattern, say *.txt.

Code:

---

        <integration:service-activator
                input-channel="typeOneInputChannel" ref="serviceActivatorBean"
                method="process" />

        <file:inbound-channel-adapter id="typeOneInputChannel"
                directory="file:${dir.home.input}/${typeOne}"
                filter="filter">
                <integration:poller id="typeOnePoller" fixed-delay="60000"
                        max-messages-per-poll="5"/>
        </file:inbound-channel-adapter>

---

As my ServiceActivatorBean is common and has code that shall be reused for all feed types, I defined it once with scope prototype.

This approach works fine.
Questions:
1. Is this approach correct? If there are better solutions please provide details.
2. Consider this case. I start the program and it polls all the 5 directories for files. Business rule demands under some conditions, we need to stop the application from polling for one or more of the feed types. If all pollers are stopped, then the program must terminate. If I used System.exit(), the JVM would be terminated, which eventually stops all the pollers instead of a specific poller(s). I would like to know if there is a way to stop polling a specific directory for a feed type dynamically / programatically?

Is it possible?

Appreciate your comments.
Thanks.
Chandra

Chandra,

If you do need to support stopping and starting each of those adapters individually, then having separate "inbound-channel-adapter" elements for each does seem like the right approach. Each of those instances actually implements lifecycle interfaces that provide start() and stop() methods. Therefore you can simply inject a reference to the bean (whose name is the value of your adapter's "id" attribute), and call start() or stop() on that bean. If you do not want the adapters to start automatically when the application starts, you can even provide auto-startup="false" on the elements.

Depending on how elaborate your requirements are for determining when to start/stop these adapters, you might even want a message-driven approach for those operations. If so, have a look at the "Control Bus" section of the Reference Manual for some ideas.

Hope that helps.
-Mark

Mark

Thanks for the inputs. I believe I have made the right code changes and I am now able to get the application working the way I want it to work. Given below is code snippet of the bean definitions and the Service Activator.

Code:

---

<integration:service-activator
                input-channel="typeOneInputChannel" ref="serviceActivatorBean"
                method="process" />

<file:inbound-channel-adapter id="typeOneAdapter" channel="typeOneInputChannel"
        directory="file:${dir.home.input}/${typeOne}"
        filter="filter">
        <integration:poller id="typeOnePoller" fixed-delay="60000"
                max-messages-per-poll="5"/>
</file:inbound-channel-adapter>

<bean id="serviceActivatorBean" class="com.spring.int.sample.CustomServiceActivator" scope="prototype">
    <property name="adapter" ref="typeOneAdapter" />
</bean>

public class CustomServiceActivator{
 
  private AbstractPollingEndpoint adapter;
 
  public void processFile(File file) throws Exception {
    try{
      // business logic
    }catch(Excpetion e){
      adapter.stop();
    }
  }

  public void setAdapter(AbstractPollingEndpoint adapter) {
    this.adapter = adapter;
  }
       
}

---

As you could see here, I have hardcoded the adapter id of typeOne. I am planning to have a factory set up to pass appropriate adapter id to the SA dynamically based on the file type that is under consideration.

Is the above implementation correct? Can you please confirm the approach? Are there any other better ways?

Thanks
Chandra