increasing duration between steps over a long period

[Penny_Wise]

Dear all,

my job does some file analysis over a quite long period of time. There are ~1200 files to read and analyse.
Within the job specification, I do iteration over a set of 10 steps required to analyse a single file.

Now I am facing a increased duration on analyzing a file. At start a single iteration over 10 steps took ~ 10 seconds, on half-time the iteration took ~7 minutes. While the execution time of each step approximately constant, the time between ending a step and starting the next one increased from 500 ms to 11 seconds!!

A memory issue?
Did anyone experience similar behaviour?

Require code? Please let me know!
Thank you very much in advance for your help!!


Penny_Wise

[Penny_Wise]

... perhaps we found the reason: STEP_EXECUTION_CONTEXT (repository table)

We do some logging into database tables, and access of STEP_EXECUTION_CONTEXT got quite slow. After truncating the table, spring batch got fast as light again.

[mminella]

Did you do any research into what the query was that was the bottle neck? What database type? unless the table had gotten huge, I'd expect it to have a nominal impact at best...

[Penny_Wise]

Our db expert mentioned the clob as possible bottleneck.
The db is an oracle 11g. In fact, I generated ~16k rows in that table. Using Toad for displaying the data, we experienced slow performance, too. So from my point of view this isn't a spring batch specific issue.

Nevertheless I would appreciate features concerning a configuration of logging-depth. I quickly scanned the docs today, but wasn't lucky, though. Is there any possibility? THANKS!