Spring Data Hadoop wordcount sampe does not compile with gradlew

[szegedi]

Hello,

I installed Spring Framework and Spring Data Hadoop on a Amazon Web Services EC2 machine (Amazon Linux AMI i386). Then I tried to compile wordcount sample using gradlew but I got a build exception regarding thrift 0.2.0 and jdo-api 2.3-ec.


[ec2-user@ip-10-234-126-25 wordcount]$ ../gradlew
:compileJava

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':compile'.
> Could not find group:org.apache.thrift, module:thrift, version:0.2.0.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
> Could not find group:javax.jdo, module:jdo2-api, version:2.3-ec.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 4.778 secs
-------------------------------

I installed thrift 0.2.0 after downloading it from http://archive.apache.org/dist/incubator/thrift/0.2.0-incubating/ ( ./bootstrap.sh && ./configure; make; sudo make install) but it did not help.

Any suggestions how to fix it?

Thanks a lot,
Istvan

[leejianwei]

Hi, you can try to change "repositories" in build.gradle as following


repositories {
// Public Spring artefacts
mavenCentral()
maven { url "http://repo.springsource.org/libs-release" }
maven { url "http://repo.springsource.org/libs-milestone" }
maven { url "http://repo.springsource.org/libs-snapshot" }
maven { url "http://www.datanucleus.org/downloads/maven2/" }
maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
maven { url "http://people.apache.org/~rawson/repo" }
}

[szegedi]

Thanks a lot , this helped to get the code compiled. However, at runtime I got an RCP client-server version mismatch exception:

2012-03-24 08:54:13,057 INFO [org.springframework.beans.factory.support.DefaultL istableBeanFactory] - <Destroying s ingletons in org.springframework.beans.factory.support.DefaultL istableBeanFactory@10e3293: defining beans [org.spri ngframework.beans.factory.config.PropertyPlacehold erConfigurer#0,hadoop-configuration,wordcount-job,clean-script,ru nner]; root of factory hierarchy>
Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name ' wordcount-job': Invocation of init method failed; nested exception is org.apache.hadoop.ipc.RPC$VersionMismatch: Pr otocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract Auto wireCapableBeanFactory.java:1422)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towi reCapableBeanFactory.java:518)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wire CapableBeanFactory.java:455)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 93)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton Bean Registry.java:222)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:290 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tLis tableBeanFactory.java:567)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actA pplicationContext.java:895)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:4 25)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationContext .java:139)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationContext .java:93)
at org.springframework.data.hadoop.samples.wordcount. Main.main(Main.java:41)
Caused by: org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode (DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)


The Spring Data Hadoop requirements states that Hadoop 0.20.2 is supported
(http://static.springsource.org/spring-hadoop/docs/current/reference/html/requirements.html ) and I have Cloudera CDH3 installed (hadoop-0.20.2-cdh3u3-core.jar) but apparently the client version is lower than the server side.

How can I configure Spring Data Hadoop to use the already installed Cloudera CDH3 jars?

Istvan

[leejianwei]

You can add CDH repository in "build.gradle":


repositories {
// Public Spring artefacts
mavenCentral()
maven { url "http://repo.springsource.org/libs-release" }
maven { url "http://repo.springsource.org/libs-milestone" }
maven { url "http://repo.springsource.org/libs-snapshot" }
maven { url "http://www.datanucleus.org/downloads/maven2/" }
maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
maven { url "http://people.apache.org/~rawson/repo" }
maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
}


and then change hadoopVersion in "gradle.properties" to 0.20.2-cdh3u3(your CDH version).

[Costin Leau]

Istvan, thanks for reporting the problem - I'll take a look at fixing the missing repositories.
By default, SHDP downloads Apache Hadoop - if you want to use a different distro then feel free to add that instead as Jarred suggested.
Note that we plan to introduce support for webhdfs in the next release, maybe even make that the default, as it avoid the tight coupling between the client and server/cluster.

[paragflume]

Hi Friends, I am also facing some problem with Spring Data Hadoop wordcount sampe with gradlew. I don't know gradlew so well. I am trying to compile it with gradlew but it show some error :

-----------------------------------------------


$ ../gradlew
:compileJava

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':compile'.
> Could not resolve group:org.apache.hadoop, module:hadoop-examples, version:1.0.0.
Required by:
:wordcount:1.0.0.M1
> Could not GET 'http://repo.springsource.org/libs-release/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
> Could not GET 'http://repo.springsource.org/libs-milestone/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
> Could not GET 'http://repo.springsource.org/libs-snapshot/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
> Could not find group:org.apache.hadoop, module:hadoop-core, version:0.20.1.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-common:0.8.1
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-serde:0.8.1
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-shims:0.8.1
> Could not find group:org.apache.thrift, module:thrift, version:0.2.0.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
> Could not find group:javax.jdo, module:jdo2-api, version:2.3-ec.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 4 mins 54.752 secs

-------------------------------------------------------------
and the content of build.gradle is :


description = 'Spring Hadoop Samples - WordCount'

apply plugin: 'base'
apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'eclipse'

repositories {
// Public Spring artefacts
maven { url "http://repo.springsource.org/libs-release" }
maven { url "http://repo.springsource.org/libs-milestone" }
maven { url "http://repo.springsource.org/libs-snapshot" }

maven { url "http://www.datanucleus.org/downloads/maven2/" }
maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
maven { url "http://people.apache.org/~rawson/repo" }

maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
}

dependencies {
compile "org.springframework.data:spring-data-hadoop:$version"
compile "org.apache.hadoop:hadoop-examples:$hadoopVersion"
// see HADOOP-7461
runtime "org.codehaus.jackson:jackson-mapper-asl:$jacksonVersion"

testCompile "junit:junit:$junitVersion"
testCompile "org.springframework:spring-test:$springVersion"
}

task run(type: JavaExec) {
description = 'Runs the application'
main = "org.springframework.data.hadoop.samples.wordcount. Main"
classpath = sourceSets.main.runtimeClasspath
}

defaultTasks 'run'

------------------------------------------------
http://static.springsource.org/spring-hadoop/docs/current/reference/html/batch-wordcount.html
plz help to overcome this problem ......,

[Costin Leau]

paragflume, please create a new topic instead of hijacking an existing one.

First try running the sample w/o the extra repo. Spring for Hadoop M2 should have better depedencies so if the sample fails, add/modify the following:


compile ("org.springframework.data:spring-data-hadoop:$version")
{ dep ->
exclude group: 'org.apache.hive', module: 'hive-metastore'
exclude group: 'org.apache.hbase', module: 'hbase'
}

If you still want to use CDH, then make sure you make that your first repository, before the ones from springsource.

[paragflume]

Hi Friends, I made some changes in the build.gradle file, added the following lines --
(1) group = "org.springframework.data.hadoop"
(2) repositories { mavenCentral() }


The compilation process is successful but in the run process showing some errors --
-----------------------------------------------------------------------


../gradlew
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:run
2012-06-06 11:11:43,693 INFO [org.springframework.context.support.ClassPathXmlAp plicationContext] - <Refreshing org.springframework.context.support.ClassPathXmlAp plicationContext@1a0c10f: startup date [Wed Jun 06 11:11:43 IST 2012]; root of context hierarchy>
2012-06-06 11:11:43,791 INFO [org.springframework.beans.factory.xml.XmlBeanDefin itionReader] - <Loading XML bean definitions from class path resource [META-INF/spring/context.xml]>
2012-06-06 11:11:44,256 INFO [org.springframework.beans.factory.config.PropertyP laceholderConfigurer] - <Loading properties file from class path resource [hadoop.properties]>
2012-06-06 11:11:44,275 INFO [org.springframework.beans.factory.support.DefaultL istableBeanFactory] - <Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultL istableBeanFactory@1d85f79: defining beans [org.springframework.beans.factory.config.PropertyP laceholderConfigurer#0,hadoop-configuration,wordcount-job,clean-script,runner]; root of factory hierarchy>
2012-06-06 11:11:45,725 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).>
2012-06-06 11:11:46,726 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).>
2012-06-06 11:11:47,727 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).>
2012-06-06 11:11:48,728 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).>
2012-06-06 11:11:49,768 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).>
2012-06-06 11:11:50,770 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).>
2012-06-06 11:11:51,771 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).>
2012-06-06 11:11:52,796 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).>
2012-06-06 11:11:53,797 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).>
2012-06-06 11:11:54,798 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).>
2012-06-06 11:11:54,811 INFO [org.springframework.beans.factory.support.DefaultL istableBeanFactory] - <Destroying singletons in org.springframework.beans.factory.support.DefaultL istableBeanFactory@1d85f79: defining beans [org.springframework.beans.factory.config.PropertyP laceholderConfigurer#0,hadoop-configuration,wordcount-job,clean-script,runner]; root of factory hierarchy>
Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'wordcount-job': Invocation of init method failed; nested exception is java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1422)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFactory.java:518)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactory.java:455)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 93)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:222)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:290 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tListableBeanFactory.java:567)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actApplicationContext.java:895)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:425)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationContext.java:93)
at org.springframework.data.hadoop.samples.wordcount. Main.main(Main.java:41)
Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client. java:1095)
at org.apache.hadoop.ipc.Client.call(Client.java:1071 )
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java: 225)
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode (DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initi alize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(F ileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSys tem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSyst em.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.jav a:254)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.jav a:123)
at org.springframework.data.hadoop.mapreduce.JobFacto ryBean.afterPropertiesSet(JobFactoryBean.java:177)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.invokeInitMethods(Abstr actAutowireCapableBeanFactory.java:1479)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1419)
... 12 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketC hannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect( SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.ja va:656)
at org.apache.hadoop.ipc.Client$Connection.setupConne ction(Client.java:434)
at org.apache.hadoop.ipc.Client$Connection.setupIOstr eams(Client.java:560)
at org.apache.hadoop.ipc.Client$Connection.access$200 0(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client. java:1202)
at org.apache.hadoop.ipc.Client.call(Client.ja)a:1046
... 28 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Command '/usr/java/jdk1.6.0_29/bin/java' finished with (non-zero) exit value 1.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 19.532 secs

-----------------------------------------------------------------------

Friends can you plz help me to solve this problem ....
Thank you ....

[Costin Leau]

You need Hadoop running locally (see the readme.txt).

P.S. And please use the [code] tags since your post is almost unreadable otherwise.

[paragflume]

Hi Friends,
I am able to compile and run " Spring Data Hadoop wordcount samples with gradlew". I made change hadoopVersion in "gradle.properties" to 0.20.2-cdh3u3 and it worked wel. This thread really helped me. Thank you Costin Leau & leejianwei.