Specifying a JobJar in the Tool Tasklet.

[davidgevorkyan]

Ahh, I see. We are using Maven Enforcer, since having duplicate classes in the classpath can potentially cause unknown issues and that has been the case several times when we had transitive dependencies that would interfere with each other.
I have excluded 1.8 from our project, but agree with you that it's not worth spending too much effort on that, unless it is causing some weird behavior.

Sincerely,
David

[davidgevorkyan]

Hi Costin,

Is there a way to output Job Counters of the executing Hadoop Tool Tasklet?


Sincerely,
David

[Costin Leau]

Not out of the box. Care to give a code example of what you are looking for?

Cheers.

[davidgevorkyan]

We are using org.apache.hadoop.mapred.Reporter for doing some reporting when running hadoop jobs.
We have enumeration with the counters and we do reporter.incrCounter(Enum, 1), and after a specific job is finished, all of the Job Counter statistics will be shown, something like:

Counter Map Reduce Total
Value1 0 932,644 932,644
Value2 0 46,125,154 46,125,154
Value3 0 932,644 932,644

As you can see, same counter can be incremented both on the mapper and reducer sides, if needed.

Let me know if you need more information.

P.S: Btw, tried to send a message to you through this forum, but seems that your inbox is full

[Costin Leau]

Right but I'm still missing on what type of work SHDP can do? As far as I can tell you don't need any specific configuration for this to work? Or am I missing something?

P.S. Yeah, my inbox gets full every months or so, and I ended up cleaning it some years ago...

[davidgevorkyan]

Ok, so usually when I am running the job from console after the job ends, all these statistics are printed out, so I can crawl the logs and extract important information for reporting purposes, such as sending email after each job run.
If SHDP can print out all these statistics after it executes the jobs, that would be great, what do you think?

[Costin Leau]

I think I know what you're issue it - the job tasklet runs the job in a non-verbose manner. I can make that configurable so the information shows up. Out of curiosity how are you using this data - nobody really reads the logs.

[davidgevorkyan]

Suppose that each job calculates some stats during execution, for example:

• TOTAL_USERS
• ACTIVE_USERS
• SUBSCRIBER_USERS
  ...

These are incremented in the reducer, as it finds more Users of a specific type. Please note that these numbers might be different depending on when it is run and for which geographic location, so we need to know these numbers to understand for example, how did our Marketing Campaign X impact user growth, or decline, etc...

This is just a simple example, since the counters are of different types and can be used for many other purposes.

[Costin Leau]

Right. But the counters are currently incremented right - you just can't see their output in the console, is that right?

[davidgevorkyan]

When I am running the job from console, the counter values are printed after the job is done.
From the SHDP perspective, you are right, they are actually incremented, but I can't see them in console.