Dec 7th, 2012, 02:58 AM
increasing duration between steps over a long period
my job does some file analysis over a quite long period of time. There are ~1200 files to read and analyse.
Within the job specification, I do iteration over a set of 10 steps required to analyse a single file.
Now I am facing a increased duration on analyzing a file. At start a single iteration over 10 steps took ~ 10 seconds, on half-time the iteration took ~7 minutes. While the execution time of each step approximately constant, the time between ending a step and starting the next one increased from 500 ms to 11 seconds!!
A memory issue?
Did anyone experience similar behaviour?
Require code? Please let me know!
Thank you very much in advance for your help!!
Dec 7th, 2012, 03:30 AM
... perhaps we found the reason: STEP_EXECUTION_CONTEXT (repository table)
We do some logging into database tables, and access of STEP_EXECUTION_CONTEXT got quite slow. After truncating the table, spring batch got fast as light again.
Dec 7th, 2012, 10:59 AM
Did you do any research into what the query was that was the bottle neck? What database type? unless the table had gotten huge, I'd expect it to have a nominal impact at best...
Dec 7th, 2012, 11:19 AM
Our db expert mentioned the clob as possible bottleneck.
The db is an oracle 11g. In fact, I generated ~16k rows in that table. Using Toad for displaying the data, we experienced slow performance, too. So from my point of view this isn't a spring batch specific issue.
Nevertheless I would appreciate features concerning a configuration of logging-depth. I quickly scanned the docs today, but wasn't lucky, though. Is there any possibility? THANKS!