Batch Process Design

[jamesbury]

I am working on a system integration project that will need to process an xml file into appropriate xmlrpc calls and process the responses.

The existing process is a Java console app that makes an http request to a servlet via the web every 10 seconds or so to see if there is data to process. If there is, a list is returned and a second servlet is called once for each item in the list and the xml returned is placed into an 'incoming' folder.

A new customer has a process that will require another process that will monitor the input directory for new files, and the files processed into the appropriate xmlrpc calls, then place the response information into an 'output' folder.

I am wondering if Spring can help me create a better solution for this process that will make future implementations for different customers easier?? Completely revamping the process is not out of the question either if there are better ideas!!

[jbetancourt]

Did you consider using Messaging instead of folders and files? You could even use a timer service to just kick the whole thing off.

[karldmoore]

Which aspects of the problem were you looking to replace. Was there anything specific in here that you wanted to try doing with Spring?

[jamesbury]

I briefly considered messaging but thought it might be too 'heavy weight' in this situation. The application will be resident on client servers (generally all around the country) and I thought the existing file/directory based formula would be less prone to support calls.

I also do not have a lot of experience with messaging (via JMS I assume?) but what would that entail?

Thanks for your reply!
Jim

[jamesbury]

I am impressed with the flexibility it looks like the Spring framework can provide to applications. I was hoping to refactor the application to utilize Spring. This application will need to work at many different client locations with lots of different requirements so I want to be able to implement new clients with the least amount of code changes possible.

[karldmoore]

I would have thought a good design should sort out most of your problems. As Spring is non-invasive you shouldn't have to really refactor much of your system to incorporate it. If you refactor to incorporate *something* you should be able to re-design what you already have and then just plug Spring in.

[jbetancourt]

Jim:

Yes, messaging could be much heavier. However, A while back I did a little prototyping with the ESB approach, using Mule, seemed relatively easy to use, and lighter. This would hide some of the messaging details. I have not had time to persue it further, but intend to.

What I have seen with "file drop" architectures is that they begin resembling the "Big ball of mud" antipattern when you start adding transactions, partial completes, retries, and all that. In my company we have two similar products, for back end integration one uses messaging and the other FTP file drops, the messaging one is much cleaner and extendable.

I briefly considered messaging but thought it might be too 'heavy weight' in this situation. The application will be resident on client servers (generally all around the country) and I thought the existing file/directory based formula would be less prone to support calls.

I also do not have a lot of experience with messaging (via JMS I assume?) but what would that entail?

Thanks for your reply!
Jim