Newbie question about features

Hello All,

I'm fairly new to the Spring Framework and just discovered this sub-project, very interesting for my needs.

But I don't know if BATCH could help me to achieve the following. For example, my basic needs are:

1. On a regular basis, I receive Export files from a Database Server (an Oracle one) . Sometimes I've to get those files at an FTP Server. other from a network mapped unit, others from mail.

2. Then I must extract (normally I got them zipped) the dump files.

3. Then I must Import these dump files into another Database Server (another Oracle database)

4. Then I've to check the generated logs from the import and if all went right, send a confirmation mail.

As you could imagine all these steps repeat every time I've got to do this process.

Steps 2, 3 and 4 use OS's external utilities like 7-zip.exe or (Exp.exe or Imp.exe from Oracle).

I don't know is these steps could be addressed and make more automatic using BATCH.

Any advise'd be kindly appreciated.

Thanks in advance.

You can implement (2), (3) and (4) as a three-step job.

1. You'll need to put the files to some well-defined location, the framework has no magic capabilities yet :) You can however specify a pattern for file location that includes e.g. current date rather than just absolute path.

2. there is currently no explicit support for zip input, so you don't get restart & transaction capabilities for this step. However, after M4 release input readers most probably won't need to care about transactions anymore. Assuming you don't care about advanced restart (just restart from scratch rather then pick up where you last finished) I see no problem with this step - you'll need to write a few lines of code though (or you can even invoke the system call you're using currently).

3. this is a typical scenario for Spring Batch, see "fixedLengthImportJob" sample - it shows how to import records from fixed-length file to database.

4. you won't need to check 'everything went right', the step will start only if the preceeding steps finished successfully - so all you do is send the email (easy to do with Spring's MailSender).

I realized one tricky point - there is currently no support for multi-file input (I guess you have multiple files in the zip archive that you need to import). Merging the files into one in a separate step is a possible solution to that.

Second, if you use Oracle capabilities for importing files (rather than jdbc access), there won't be much value added by the framework - you can just write a script to automate your scenario.

Hi Robert,

Thanks for your quick replies.

I'll try to explain better what I want to achieve...

From your last post:

Yes, the zipped file can contain multiple .DMP Oracle files.
Yes, the utilities used are those from Oracle and .DMP files are an Oracle's propietary format. I'm not in the need to load a flat file with fixed or unfixed field's length (but it could be someday...).

I think the process comprise a well defined steps, one based upon another, and even I'm not using fixed-length file loading, the load of the .DMP files need to be done atomically and after that, the log file generated by this load needs to be examined looking for errors.

If some kind of warnings or errors happened, there must be a warning in the form of an email to the DBA and the process must stop; if no errors were found in the log files, the process must continue sending a confirmation email of the successfully load.

All of these steps are what I try to automate.

Thanks in advance.

I'd suggest simply using a scripting language to implement the scenario you described - if I understand correctly, all you need to do is tie together a few commands that do the job.

Spring Batch concentrates on what's going on inside the step (break it into smaller transactions, periodically save metadata for restart, allow skipping some number invalid records...). In case all your steps just call some system command and wait for return value it is simply not batch what the framework is doing. The system commands do the batch, not the framework. The job essentially becomes a script.

I agree with Robert completely, with one caveat. The framework also gives you status about what you've run, when it was started, whether it completed successfully, etc. If you have a small amount of scripts you can easily handle this yourself by creating log files. Just be careful, if you start adding script after script (and potentially a java class or two) it can quickly get out of hand without being careful.

I still agree with Robert, it looks like something that is best handled with a fairly simple script, but make sure you keep in mind that the job will inevitably fail (probably at 3:00 in the morning) and you will need a reliable way to figure out what happened and diagnose the issue.

Hi Lucasward,

I agree with you. That kind of control it's what make me think about using Batch instead of the concatenation of a set of simple scripts, where every single script could be potentially a fail point and broke the chain.

Even the steps looks simple, I'll like to count on the Batch capabilities to get the highest control possible in each step.

Thanks for your reply. I'll keep tuned with this project and wait till its GA release.

Regards.