A log per job execution

[telematica]

Hi. Back again with a brand new question to ask.

Provided we'd like to keep using Log4j, which is what Spring-Batch uses, is it possible to have an independent log for each job execution that you run? Alternative and/or complementary, is it possible to automatically add the job id to every log line that an execution of a job emmits?

Thanks in advance.

[lucasward]

I can tell you how you would get access to the JobExecution id separately, the JobListener interface introduced in M5. However, adding that to each log file would require a custom appender in Log4j.

Out of curiosity, what's the use case for this? Or rather, what would this provide you that you couldn't get from the meta-data tables?

[Dave Syer]

You can put the JobExecution ID etc. in the logger without a custom appender using the MDC or NDC from log4j (probably in a JobListener as suggested by Lucas). Once there is a column in the log that is unique it is trivial to separate the logs in a postprocessor. (Separate files or output channels at runtime would involve a custom appender as far as I know.)

[Dave Syer]

I thought of a way to get separate log files at runtime without writing an appender. You need a custom Log4jConfigurer though (its an abstract base class in Spring Core) - just create one as a Spring Bean and put it in the same configuration as your Job. Then to start a Job use a new ApplicationContext for each execution (as recommended, and demo-ed in the samples with TaskExecutorLauncher), and make each job use a different log4j configuration. That way you could get a separate file per job, but not per job execution without some more trickery.

[telematica]

In reply to lucasward, I'll explain our concerns:

Log4j only logs on a class basis rather than on an instance one. With this in mind, and provided that all jobs will run from a centralized server and we need a unique point to watch the whole system's log, we won't be able to know which job wrote out which line unless we can either:

• intercept anything that job may log, then automatically add the job's id (something like this should not have to be done manually) and finally output everything to the central log, or
• forget any of these trouble simply having a different file per job execution (e.g. named after the job's id)

Bear in mind that we also need to have Spring's infrastructure output available to allow debugging of new jobs as we program them, so just having our new classes do logging in a custom way to meet our needs won't do the trick.

Once again, thanks for the interest on all this. We'll have a look at your proposals and let you know how it all turns out.

[telematica]

We implemented our own appender and used the NDC approach (see? we use all the ideas) together.

Apparently, all worked well and we were about to throw a party on it when our sceptic partner suggested trying with 'parallelJob'. Suddenly darkness and despair came back to our lives, leaving us no choice but to sending him away and come back to you.

The thing is that threads spawned by SimpleAsyncTaskExecutor don't seem to receive their parent's NDC and we can't find where to hook them up with it.

[subha.audhya]

I'm also facing the same problem.Can't we generate one log file per job execution using log4j.xml without customising the Appender code?Or if we can only do it using Appender Code customization ; what is the way for it?