Spring Data Hadoop wordcount sampe does not compile with gradlew

**szegedi** · Mar 18th, 2012, 06:30 AM

Hello,

I installed Spring Framework and Spring Data Hadoop on a Amazon Web Services EC2 machine (Amazon Linux AMI i386). Then I tried to compile wordcount sample using gradlew but I got a build exception regarding thrift 0.2.0 and jdo-api 2.3-ec.

[ec2-user@ip-10-234-126-25 wordcount]$ ../gradlew
:compileJava

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':compile'.
> Could not find group:org.apache.thrift, module:thrift, version:0.2.0.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
> Could not find group:javax.jdo, module:jdo2-api, version:2.3-ec.
Required by:
:wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 4.778 secs
-------------------------------

I installed thrift 0.2.0 after downloading it from http://archive.apache.org/dist/incub....0-incubating/ ( ./bootstrap.sh && ./configure; make; sudo make install) but it did not help.

Any suggestions how to fix it?

Thanks a lot,
Istvan

**leejianwei** · Mar 19th, 2012, 03:45 AM

Hi, you can try to change "repositories" in build.gradle as following

Code:

repositories {
    // Public Spring artefacts
    mavenCentral()
    maven { url "http://repo.springsource.org/libs-release" }
    maven { url "http://repo.springsource.org/libs-milestone" }
    maven { url "http://repo.springsource.org/libs-snapshot" }
    maven { url "http://www.datanucleus.org/downloads/maven2/" }
    maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
    maven { url "http://people.apache.org/~rawson/repo" }
}

**szegedi** · Mar 24th, 2012, 04:29 AM

Thanks a lot , this helped to get the code compiled. However, at runtime I got an RCP client-server version mismatch exception:

2012-03-24 08:54:13,057 INFO [org.springframework.beans.factory.support.DefaultL istableBeanFactory] - <Destroying s ingletons in org.springframework.beans.factory.support.DefaultL istableBeanFactory@10e3293: defining beans [org.spri ngframework.beans.factory.config.PropertyPlacehold erConfigurer#0,hadoop-configuration,wordcount-job,clean-script,ru nner]; root of factory hierarchy>
Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name ' wordcount-job': Invocation of init method failed; nested exception is org.apache.hadoop.ipc.RPC$VersionMismatch: Pr otocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract Auto wireCapableBeanFactory.java:1422)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towi reCapableBeanFactory.java:518)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wire CapableBeanFactory.java:455)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 93)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton Bean Registry.java:222)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:290 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tLis tableBeanFactory.java:567)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actA pplicationContext.java:895)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:4 25)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text .java:139)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text .java:93)
at org.springframework.data.hadoop.samples.wordcount. Main.main(Main.java:41)
Caused by: org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode (DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient. java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient. java:203)

The Spring Data Hadoop requirements states that Hadoop 0.20.2 is supported
(http://static.springsource.org/sprin...uirements.html ) and I have Cloudera CDH3 installed (hadoop-0.20.2-cdh3u3-core.jar) but apparently the client version is lower than the server side.

How can I configure Spring Data Hadoop to use the already installed Cloudera CDH3 jars?

Istvan

**leejianwei** · Mar 26th, 2012, 04:45 AM

You can add CDH repository in "build.gradle":

Code:

repositories {
    // Public Spring artefacts
    mavenCentral()
    maven { url "http://repo.springsource.org/libs-release" }
    maven { url "http://repo.springsource.org/libs-milestone" }
    maven { url "http://repo.springsource.org/libs-snapshot" }
    maven { url "http://www.datanucleus.org/downloads/maven2/" }
    maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
    maven { url "http://people.apache.org/~rawson/repo" }
    maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
}

and then change hadoopVersion in "gradle.properties" to 0.20.2-cdh3u3(your CDH version).

**Costin Leau** · Mar 28th, 2012, 11:54 AM

Istvan, thanks for reporting the problem - I'll take a look at fixing the missing repositories.
By default, SHDP downloads Apache Hadoop - if you want to use a different distro then feel free to add that instead as Jarred suggested.
Note that we plan to introduce support for webhdfs in the next release, maybe even make that the default, as it avoid the tight coupling between the client and server/cluster.

**paragflume** · Jun 6th, 2012, 12:20 AM

Hi Friends, I am also facing some problem with Spring Data Hadoop wordcount sampe with gradlew. I don't know gradlew so well. I am trying to compile it with gradlew but it show some error :

-----------------------------------------------

Code:

$ ../gradlew
:compileJava

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':compile'.
> Could not resolve group:org.apache.hadoop, module:hadoop-examples, version:1.0.0.
  Required by:
      :wordcount:1.0.0.M1
   > Could not GET 'http://repo.springsource.org/libs-release/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
   > Could not GET 'http://repo.springsource.org/libs-milestone/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
   > Could not GET 'http://repo.springsource.org/libs-snapshot/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
> Could not find group:org.apache.hadoop, module:hadoop-core, version:0.20.1.
  Required by:
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-common:0.8.1
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-serde:0.8.1
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-shims:0.8.1
> Could not find group:org.apache.thrift, module:thrift, version:0.2.0.
  Required by:
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
> Could not find group:javax.jdo, module:jdo2-api, version:2.3-ec.
  Required by:
      :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 4 mins 54.752 secs

-------------------------------------------------------------
and the content of build.gradle is :

Code:

description = 'Spring Hadoop Samples - WordCount'

apply plugin: 'base'
apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'eclipse'

repositories {
    // Public Spring artefacts
    maven { url "http://repo.springsource.org/libs-release" }
    maven { url "http://repo.springsource.org/libs-milestone" }
    maven { url "http://repo.springsource.org/libs-snapshot" }

    maven { url "http://www.datanucleus.org/downloads/maven2/" }
    maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
    maven { url "http://people.apache.org/~rawson/repo" }

    maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
}

dependencies {
    compile "org.springframework.data:spring-data-hadoop:$version"
    compile "org.apache.hadoop:hadoop-examples:$hadoopVersion"
    // see HADOOP-7461
    runtime "org.codehaus.jackson:jackson-mapper-asl:$jacksonVersion"

    testCompile "junit:junit:$junitVersion"
    testCompile "org.springframework:spring-test:$springVersion"
}

task run(type: JavaExec) {
       description = 'Runs the application'
       main = "org.springframework.data.hadoop.samples.wordcount.Main"
       classpath = sourceSets.main.runtimeClasspath
}

defaultTasks 'run'

------------------------------------------------
http://static.springsource.org/sprin...wordcount.html
plz help to overcome this problem ......,

**Costin Leau** · Jun 6th, 2012, 12:43 AM

paragflume, please create a new topic instead of hijacking an existing one.

First try running the sample w/o the extra repo. Spring for Hadoop M2 should have better depedencies so if the sample fails, add/modify the following:

Code:

compile ("org.springframework.data:spring-data-hadoop:$version")
    {  dep ->
        exclude group: 'org.apache.hive', module: 'hive-metastore'
        exclude group: 'org.apache.hbase', module: 'hbase'
    }

If you still want to use CDH, then make sure you make that your first repository, before the ones from springsource.

**paragflume** · Jun 6th, 2012, 12:49 AM

Hi Friends, I made some changes in the build.gradle file, added the following lines --
(1) group = "org.springframework.data.hadoop"
(2) repositories { mavenCentral() }

The compilation process is successful but in the run process showing some errors --
-----------------------------------------------------------------------

Code:

../gradlew
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:run
2012-06-06 11:11:43,693 INFO [org.springframework.context.support.ClassPathXmlApplicationContext] - <Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1a0c10f: startup date [Wed Jun 06 11:11:43 IST 2012]; root of context hierarchy>
2012-06-06 11:11:43,791 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - <Loading XML bean definitions from class path resource [META-INF/spring/context.xml]>
2012-06-06 11:11:44,256 INFO [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer] - <Loading properties file from class path resource [hadoop.properties]>
2012-06-06 11:11:44,275 INFO [org.springframework.beans.factory.support.DefaultListableBeanFactory] - <Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1d85f79: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,hadoop-configuration,wordcount-job,clean-script,runner]; root of factory hierarchy>
2012-06-06 11:11:45,725 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).>
2012-06-06 11:11:46,726 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).>
2012-06-06 11:11:47,727 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).>
2012-06-06 11:11:48,728 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).>
2012-06-06 11:11:49,768 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).>
2012-06-06 11:11:50,770 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).>
2012-06-06 11:11:51,771 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).>
2012-06-06 11:11:52,796 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).>
2012-06-06 11:11:53,797 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).>
2012-06-06 11:11:54,798 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).>
2012-06-06 11:11:54,811 INFO [org.springframework.beans.factory.support.DefaultListableBeanFactory] - <Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1d85f79: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,hadoop-configuration,wordcount-job,clean-script,runner]; root of factory hierarchy>
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'wordcount-job': Invocation of init method failed; nested exception is java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1422)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:518)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:567)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
        at org.springframework.data.hadoop.samples.wordcount.Main.main(Main.java:41)
Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
        at org.apache.hadoop.ipc.Client.call(Client.java:1071)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at $Proxy1.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
        at org.springframework.data.hadoop.mapreduce.JobFactoryBean.afterPropertiesSet(JobFactoryBean.java:177)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1479)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1419)
        ... 12 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:656)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
        at org.apache.hadoop.ipc.Client.call(Client.ja)a:1046
        ... 28 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Command '/usr/java/jdk1.6.0_29/bin/java' finished with (non-zero) exit value 1.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 19.532 secs

-----------------------------------------------------------------------

Friends can you plz help me to solve this problem ....
Thank you ....

**Costin Leau** · Jun 6th, 2012, 12:51 AM

You need Hadoop running locally (see the readme.txt).

P.S. And please use the [code] tags since your post is almost unreadable otherwise.

**paragflume** · Jun 6th, 2012, 01:52 AM

Hi Friends,
I am able to compile and run " Spring Data Hadoop wordcount samples with gradlew". I made change hadoopVersion in "gradle.properties" to 0.20.2-cdh3u3 and it worked wel. This thread really helped me. Thank you Costin Leau & leejianwei.

Thread: Spring Data Hadoop wordcount sampe does not compile with gradlew

Thread Tools

Display

Spring Data Hadoop wordcount sampe does not compile with gradlew

Posting Permissions