Write via FTP only when previos file has been consumed

[gibraltar]

Hi all,

I'm new to Spring Integration and have some difficulties in getting the things right.

What I'm trying to achieve is to get a message from our JMS and write it to trasmit via FTP with a fixed filename only when the previos transferred message has been processed and deleted on the FTP host.

I'm fine with everything, with the exception on how to wait for the previos file being deleted.

Any hint on that subject?

Thank you!

[skilledmonster]

It also depends on your design considerations -

1 - If the file is still not deleted on the FTP, do you plan to still keep the messages in the queue without consuming them?
2 - Is this a very high volume queue?[If yes, you might be in trouble if the depth size is increasing tremendously]
3 - Are the messages in the queue are all of the same type? I am guessing different ones.
4 - How often are the messages processed on the FTP server? Are they processed as and when they are dropped on the server. How much time does it take to process them?

Jagadeesh
www.skilledmonster.com

[gibraltar]

Thanks for answering, Jagadeesh!

It also depends on your design considerations -

1 - If the file is still not deleted on the FTP, do you plan to still keep the messages in the queue without consuming them?

yes: the messages comes from a JMS.

2 - Is this a very high volume queue?[If yes, you might be in trouble if the depth size is increasing tremendously]

no, the producing/consuming speed should be comparable, so there shouldn't be risks of congestions.

3 - Are the messages in the queue are all of the same type? I am guessing different ones.

Yes! they're basically plain text. What I need to do is just adding an header and some more simple transformation.

4 - How often are the messages processed on the FTP server? Are they processed as and when they are dropped on the server. How much time does it take to process them?

they are supposed to be processed (almost) as soon as they appear.

I give you more details on the contest.

The files are processed by a legacy procedure and it expects them to be called all the same.
Unfortunately if a file is already present I cannot append information to it because every file has its own header (although they are basically plain ASCII text)

I've imagined two approaches, but, beside the fact that I don't know how to actually "wire" the components, none of them feels right.

approach 1: fetch a message from the JMS as soon as it arrives, list the remote directory: if the file is already present rollback the transaction, wait some times and try again.

approach 2: poll the remote directory. when the file is consumed fetch a message from the JMS and drop it via ftp.

which one is a better design? is there a better way of doing it?

thanks!

[skilledmonster]

Personally, I would recommend [Approach 2]. This way you don't have to worry about rollback. Also, once the message is fetched from the queue, rollback doesn't put it back into the queue unless you have a special logic. It also depends if sequencing is important, in which case [Approach 1] will probably not work.

I would do something like -
1 - Poll the remote directory.
2 - If the file is consumed / processed / deleted, fetch the message from the JMS
3 - Add header information and drop it via FTP

I am sure, you might have required configuration in place so you don't lose your messages from JMS, what so ever! Or I would recommend the following which probably does more I/O
1 - Fetch the JMS message as and when it arrives and persist it to the disk, probably with timestamps
2 - On a separate thread, poll the remote directory
3 - If the file is consumed / processed / deleted, fetch the message from the disk and copy it to FTp
4 - Process the files on the remote directory.

Let me know if you need help with implementation.


1 - Poll the remote directory.
2 - If the file is consumed / processed / deleted, fetch the message from the JMS
3 - Add header information and drop it via FTP

[gibraltar]

Personally, I would recommend [Approach 2]. This way you don't have to worry about rollback. Also, once the message is fetched from the queue, rollback doesn't put it back into the queue unless you have a special logic. It also depends if sequencing is important, in which case [Approach 1] will probably not work.

I would do something like -
1 - Poll the remote directory.
2 - If the file is consumed / processed / deleted, fetch the message from the JMS
3 - Add header information and drop it via FTP

Yes! I'd go with that too!

Let me know if you need help with implementation.

Yes, please! I'd appreciate a lot if you could give me an hint!

I've configured everything, but I have no clue on how to connect the pieces.
In particular, the block I'm missing is: how could I activate an action (fetch and process from JMS) when the file is not there anymore?

I'd configure an FTP with a Poller, right?
and then I'm lost!



Thanks again!

[skilledmonster]

With little brainstorming, I am thinking something like this -

Use Spring Batch + Spring Integration

Configure a job repository with repeatable tasks

Tasklet 1 - For polling the FTP Server
Tasklet 2 - Fetching the message from JMS + Processing it using item writer etc
Tasklet 3 - Copy the file FTP Server.

For more robustness, you can probably add another tasklet that can take care of roll-back by persisting the processed message just in case you get disconnected while remote transfer.

Please weigh on the pro and cons...i will give a second thought as well :-)

[skilledmonster]

When you poll the remote directory, do you know the file name off-hand? Can you please provide more information on this piece?

[gibraltar]

With little brainstorming, I am thinking something like this -
Use Spring Batch + Spring Integration

I'd prefer to remain with only Spring Integration: I'm introducing the whole Spring Framework in my company and wouldn't put too much new stuff around.

When you poll the remote directory, do you know the file name off-hand? Can you please provide more information on this piece?

yes! the file name is hardcoded in the legacy procedure, and is fixed.

I've ended up implementing the following solution, which is reasonably acceptable for me at the moment:

1. an FTP outbound gateway
2. a channel adapter with a poller which sends a message to the gateway with the name of the directory to check
3. a service activator which receive the replies of the FTP outbound gateway (which contains the listing of the remote ftp directory) and check the existence of the remote data file. If the file disappeared, fetch the message from the JMS, process it and send it via FTP.

it doesn't look perfect, but at the moment it is conceptually working (still I have to manage JMS transaction in case of failure, but that's only a small detail, I think)

if you have any suggestion on how to make it cleaner, I'll be happy to learn!
thanks for the help so far!

[skilledmonster]

Great! Care to share snippets of the code? Might be helpful for some other folks looking for a similar solution!

you can manage transaction failure by configuring AOP. And on the other hand, you probably persist the messages locally before shipping off to handle failure!

[gibraltar]

Great! Care to share snippets of the code? Might be helpful for some other folks looking for a similar solution!

Sure! I'll tidy it a bit and post it ASAP!