Pausing a step before end

[Miko95]

Hey,

I have a job with only one step that reads lines from a file, processes them and writes in a database. The problem is that some lines may be appended to the end of the file by an external component. So, when the reader gets null, I want to pause the step and polls the file every X seconds to check if new lines have been added
I've read about the possibility to pause a step at the end, but not to resume this same step.

Thanks for your help

[Miko95]

I wake up this thread. My question is : is Spring batch fits in my reqirements?
I try to explain more my problem : I read from a file (using a FlatFileItemReader). The name of this file is determined at the beginning of the job and doesn't change during the job execution. If I use a single step that is executed only once, when the reader gets null, the step terminates BUT some lines may be appended after 15 sec for example by an external component. So, I wondered if it is possible to pause the step when the reader gets null and to continue after 15 sec and so on (until a limit that i define). Another possibility is the use of a decider and repeat the step until i want it to terminate BUT because the reader is declared with step scope, a new reader will be instantiated at each step and the same file will be opened each time so the performance will be bad. Also each time the reader is instantiated, i tell it at which line to start using the READ_COUNT value, and the reader reads each line until the starting line.

[michael.lange]

i have no example at hand, but while reading your question i thought of at least 3 possibilities

• Retry - jobs runs endlessly(?)
• "fake" restart scenario
• copy file at job start, run on copy, keep line count somewhere, next job starts reading after last line

in the first two cases (i'm not sure about the retry though) i would use a custom reader implementation which throws an exception at end of file

if possible i would try to get the external component to fill a queue (database, MQ, ESB, whatever)

[edith says]
i would strongly advice against "pausing" a step, because you would "pause" the transaction as well.

[Miko95]

Thanks for your answer Michael,

About the external component, i don't have the possibility to change it.

I thought about the retry, but in my case, the file will be read during one hour from the beginning of the job, and there is no possibility to wait an amount of time before to retry (i think?) so there will be a lot of retry so need to set the retry limit to an appropriated value (is there a special value to specify no limit?) and I think it will use the CPU a lot (to do nothing), always trying to retry the step. I think my comment is also applicable to the "fake" restart scenario.

I didn't understand the third possibility. What is the difference between reading the file every 15 sec by starting a new step execution and copying the file for read? (transaction?)

What is the problem of pausing the transaction during the step?

[michael.lange]

I didn't understand the third possibility. What is the difference between reading the file every 15 sec by starting a new step execution and copying the file for read? (transaction?)

i guess i misunderstood you, you are right, there is no substantial difference

What is the problem of pausing the transaction during the step?

Actually i think you can't really "pause" a running transaction, because there will be problems with time-outs (transactionmanager, underlying database), especially in a concurrent environment.

[Miko95]

Okay i understand. So pausing a step is not an option. It seems that the FlatFileItemReader is not adapted to the case the file may change during the execution.
Maybe my use case doesn't belong to batch category?

[arno]

looks like your use case is more event-driven. Did you consider using something like Spring Integration? I'm pretty sure it has support for streams that are being written to.