Compression codec com.hadoop.compression.lz o.LzoCodec not found.

[ugxzl]

Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'news_sim_script_s
tep': Cannot resolve reference to bean 'news_sim_script_tasklet' while setting bean property 'tasklet'; nested exception is org.spri
ngframework.beans.factory.BeanCreationException: Error creating bean with name 'news_sim_script_tasklet': Cannot resolve reference t
o bean 'setup-script' while setting bean property 'scriptCallback'; nested exception is org.springframework.beans.factory.BeanCreati
onException: Error creating bean with name 'setup-script': Invocation of init method failed; nested exception is java.lang.IllegalAr
gumentException: Compression codec com.hadoop.compression.lz
o.LzoCodec not found.
at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
28)
at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveValueIfNecessary(BeanDe finitionValueResolver
.java:106)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.applyPropertyValues(Abs tractAutowireCapableB
eanFactory.java:1360)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.populateBean(AbstractAu towireCapableBeanFact
ory.java:1118)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFact
ory.java:517)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactor
y.java:456)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 94)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:225
)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:291 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tListableBeanFactory.
java:591)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actApplicationContext
.java:918)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:469)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:139)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:93)
at com.sohu.wap.algorithm.launch.SimLaunch.main(SimLa unch.java:31)
Caused by: org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'news_sim_script_tasklet': Cannot
resolve reference to bean 'setup-script' while setting bean property 'scriptCallback'; nested exception is org.springframework.beans
.factory.BeanCreationException: Error creating bean with name 'setup-script': Invocation of init method failed; nested exception is
java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lz
o.LzoCodec not found.
at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
28)
at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveValueIfNecessary(BeanDe finitionValueResolver
.java:106)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.applyPropertyValues(Abs tractAutowireCapableB
eanFactory.java:1360)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.populateBean(AbstractAu towireCapableBeanFact
ory.java:1118)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFact
ory.java:517)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactor
y.java:456)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 94)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:225
)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:291 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
22)
... 15 more
Caused by: org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'setup-script': Invocation of init
method failed; nested exception is java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lz
o.LzoCodec not found.
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFa
ctory.java:1455)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFact
ory.java:519)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactor
y.java:456)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 94)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:225
)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:291 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
22)
... 25 more
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lz
o.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFact ory.getCodecClasses(CompressionCodecFactory.java:9 6)
at org.apache.hadoop.io.compress.CompressionCodecFact ory.<init>(CompressionCodecFactory.java:134)
at org.springframework.data.hadoop.fs.HdfsResourceLoa der.<init>(HdfsResourceLoader.java:76)
at org.springframework.data.hadoop.fs.HdfsResourceLoa der.<init>(HdfsResourceLoader.java:98)
at org.springframework.data.hadoop.fs.HdfsResourceLoa der.<init>(HdfsResourceLoader.java:61)
at org.springframework.data.hadoop.scripting.HdfsScri ptRunner.detectHdfsRL(HdfsScriptRunner.java:179)
at org.springframework.data.hadoop.scripting.HdfsScri ptRunner.postProcess(HdfsScriptRunner.java:92)
at org.springframework.data.hadoop.scripting.Jsr223Sc riptRunner.afterPropertiesSet(Jsr223ScriptRunner.j ava:91)
at org.springframework.data.hadoop.scripting.HdfsScri ptRunner.afterPropertiesSet(HdfsScriptRunner.java: 225)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.invokeInitMethods(Abstr actAutowireCapableBea
nFactory.java:1514)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFa
ctory.java:1452)
... 32 more
Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lz
o.LzoCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java: 200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.j ava:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:3 07)
at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 52)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByNam e(Configuration.java:819)
at org.apache.hadoop.io.compress.CompressionCodecFact ory.getCodecClasses(CompressionCodecFactory.java:8 9)
... 42 more

has someone encounter the same problem?

[Costin Leau]

You need to install the Hadoop LzoCodec in the classpath (which implies the native library as well). Some of your classes refer to it and since it's not found, you get this exception. Notice that this is caused by Hadoop (as indicated by the stacktrace) not Spring.

P.S. please use [code] tags when posting code/stacktracer - otherwise your posts are unreadable.

[ugxzl]

thank your reply
my start shell

nohup hadoop-news com.sohu.wap.algorithm.launch.SimLaunch &

hadoop-new script

exec " -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS -Djava.library.path=/usr/local/goldmine/hadoop/defaul
t/bin/../lib/native/Linux-amd64-64 -classpath /usr/local/goldmine/hadoop/default/bin/../conf:/usr/local/java/jdk/default/lib/tools.j
ar:/usr/local/goldmine/hadoop/default/bin/..:/usr/local/goldmine/hadoop/default/bin/../hadoop-core-0.20.2-sohu.jar:/usr/local/goldmi
ne/hadoop/default/bin/../lib/aspectjrt-1.6.5.jar:/usr/local/goldmine/hadoop/default/bin/../lib/aspectjtools-1.6.5.jar:/usr/local/gol
dmine/hadoop/default/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-beanutils-core-1.8
.0.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-cli-1.2.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-codec
-1.4.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-collections-3.2.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/c
ommons-configuration-1.6.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-daemon-1.0.1.jar:/usr/local/goldmine/hadoop/defau
lt/bin/../lib/commons-digester-1.8.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-el-1.0.jar:/usr/local/goldmine/hadoop/d
efault/bin/../lib/commons-httpclient-3.0.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-lang-2.4.jar:/usr/local/goldmin
e/hadoop/default/bin/../lib/commons-logging-1.1.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-logging-api-1.0.4.jar:/u
sr/local/goldmine/hadoop/default/bin/../lib/commons-math-2.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-net-1.4.1.jar
:/usr/local/goldmine/hadoop/default/bin/../lib/core-3.1.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hadoop-fairscheduler-0.2
0.2-sohu.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hadoop-lzo-0.4.8.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hsqldb
-1.8.0.10.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hue-plugins-1.2.0.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jack
son-core-asl-1.0.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jackson-mapper-asl-1.0.1.jar:/usr/local/goldmine/hadoop/default
/bin/../lib/jasper-compiler-5.5.12.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jasper-runtime-5.5.12.jar:/usr/local/goldmine/h
adoop/default/bin/../lib/jets3t-0.6.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jetty-6.1.26.jar:/usr/local/goldmine/hadoop/
default/bin/../lib/jetty-util-6.1.26.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jsch-0.1.42.jar:/usr/local/goldmine/hadoop/de
fault/bin/../lib/junit-4.5.jar:/usr/local/goldmine/hadoop/default/bin/../lib/kfs-0.2.2.jar:/usr/local/goldmine/hadoop/default/bin/..
/lib/log4j-1.2.15.jar:/usr/local/goldmine/hadoop/default/bin/../lib/mockito-all-1.8.5.jar:/usr/local/goldmine/hadoop/default/bin/../
lib/oro-2.0.8.jar:/usr/local/goldmine/hadoop/default/bin/../lib/servlet-api-2.5-20081211.jar:/usr/local/goldmine/hadoop/default/bin/
../lib/slf4j-api-1.4.3.jar:/usr/local/goldmine/hadoop/default/bin/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/goldmine/hadoop/default/
bin/../lib/xmlenc-0.52.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/goldmine/hadoop/default/bin/
../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/goldmine/hadoop/default/lib/hadoop-lzo-0.4.8.jar -Djava.ext.dirs=lib  -Dproduction.mode=DE
V -Dlog.output=/data/DATA/smc/whftest/newrecom/logs $CLASS "$@"

as above,the lib hadoop-lzo-0.4.8.jar has already installed and native libraby "/usr/local/goldmine/hadoop/defaul
t/bin/../lib/native/Linux-amd64-64" has already defined. i wonder why it report error anyway?please

[Costin Leau]

That's a huge classpath -notice that you declare hadoop-lzo-0.4.8.jar multiple times - you can use * since JDK 1.6 I believe instead of specifiying each jar.
I'm unclear whether this is the command you use to start the cluster or the spring hadoop app? I ask since the latter is usually used to submit tasks to the hadoop cluster (which is already started). If that's the case make sure the LzoCodec is available for both apps.

As an alternative you could try using another codec just to double check the Codec initialization for Hadoop and whether they are properly loaded.

I suggest also googling around since this is a Hadoop setup problem and I'm sure plenty of people bumped into it.