Job not appearing on JobTracker user interface

[saurabhjain27]

While trying to use Spring Hadoop to run map-reduce using the maven-assembly approach outlined @ http://www.petrikainulainen.net/prog...apache-hadoop/

The job runs successfully and generated output but does not appear on the Job Tracker user interface.

Also, the logs show the warning: WARN [org.apache.hadoop.util.NativeCodeLoader] - <Unable to load native-hadoop library for your platform... using builtin-java classes where applicable>

I am not sure if these two are inter-related. Any explanation of this behavior will be very helpful.

Thanks,
Saurabh

[saurabhjain27]

I tried running the wordcount example that comes bundled with spring-data-hadoop-0.9.0.RELEASE. Observation was same with this example: the Job Tracker user interface didn't track this job.

Is this a issue with Spring Hadoop?

[shekhina]

Did you set 'mapred.job.tracker' property in <hdp:configuration>..</..> like $HADOOP_HOME/conf/mapred-site.xml?

My spring-hadoop.xml is look like (spring-hadoop-1.0.0RC1)
<hdp:configuration>
fs.default.name=${hd.fs:hdfs://localhost:9000}
mapred.job.tracker=${mapred.job.tracker:master:900 1}
</hdp:configuration>
...
There is no problem.

[saurabhjain27]

Thanks shekhina. I was missing the "mapred.job.tracker" property in the hdp configuration. Adding that fixed the issue and I am now able to see the job tracked on the Job Tracker UI.

[Costin Leau]

shekhina is right (thanks for helping out).
The underlying problem is that without setting up the job tracker, Hadoop will use the default one which can be a local tracker - meaning only one map/reduce task is executed on the local machine instead of the actual Hadoop cluster.
This is a somewhat not-obvious side-effect that catches many (including us) by surprise.