How to execute a Hadoop Project with the "Run" Dialog in STS (from Shell it works)

[d0x]

Hi,
I created 2 Maven Projects:

• com.ponyCircus
  - application Context and everything else (like public static void main...)
• com.ponyCircus.mr
  - ExampleMapper
  - ExampleReducer

In the pom.xml of com.ponyCircus i configured a dependency to com.ponyCircus.mr.


Scenario A (working):
I build the stuff with maven and execute the com.ponyCircus.Main class.
With this everything is ok. (That means the configuration is fine).

Scenario B (working):
I checkout the com.ponyCircus example, configure it as Maven project. Then m2e will get the com.ponyCircus.mr.jar from the repository and link it as Jar-File.

If i lunch it with the Run dialog (F11) everything is fine .

This screenshot shows that it links the JAR File:


Scenario C (There i get the problem...):
I checkout both projects and configure it as Maven Projects. Now m2e will Link the com.ponyCircus.mr Project (not the Jar File).

If I lunch it, i will get a

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.ponyCircus.mr.wordCount.WordMapper not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1571)
	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class com.ponyCircus.mr.wordCount.WordMapper not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1477)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1569)

Exception because it was not able to identify and ship the correct jar file.

This screenshot shows the problems (i guess):


Is there a way to make it running with STS?

P.S.: I don't know whether this is a STS or Hadoop Question. Please move if you feel it's not correct .

[saurabhjain27]

Have you tried jar-by-class (and jar) attributes?

<hdp:job id="flattener" input-path="#{inputPath}"
output-path="#{outputPath}"
mapper="x.y.Importer$ImporterMapper"
jar-by-class="x.y.Importer$ImporterMapper"
jar="file:///E:/workspace-sts3/proj/target/proj.jar"/>

[d0x]

Hi Saurabhjain,

i haven't specified the "jar=" property. I can't put an absolute path there. And maybe also a relative path is not good .

But i gave the jar-by-class property.

Like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:hdp="http://www.springframework.org/schema/hadoop"
	xmlns:p="http://www.springframework.org/schema/p"
	xmlns:context="http://www.springframework.org/schema/context"
	xsi:schemaLocation="
		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
		http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">


	<context:component-scan base-package="com.ponyCircus" />

	<hdp:configuration id="hadoopConfiguration">
		fs.defaultFS=hdfs://carolin....net:8020
		mapred.job.tracker=michaela.....net:8021
	</hdp:configuration>
	
	<hdp:job id="wordCountJob"
		mapper= "com.ponyCircus.mr.wordCount.WordMapper"
		reducer="com.ponyCircus.mr.wordCount.WordReducer"
		jar-by-class= "com.ponyCircus.mr.wordCount.WordMapper"
		input-path="ponyCircusExample/input"
		output-path="ponyCircusExample/output" />
		
</beans>

[d0x]

As workaround we are using now the "External Tool Chain"-Configuration and the exec-maven-plugin.

But that's a ugly workaround.

With this we can just click the External Tool Chain button to test our code in the IDE. But the Run-Dialog and Debugging doesn't work.



Can someone suggest a more elegant way?