Remote Hadoop Node

[msduk]

Hi all,

Does Spring-Hadoop support configuration for a remote hadoop server? I really should have checked this before spending a day setting it up.



<hdp:configuration>
fs.default.name=hdfs://foo.com:9000
</hdp:configuration>


I keep getting permission errors as if it is trying to always use a local hadoop instance.

Caused by: java.io.IOException: Failed to set permissions of path: \tmp\hadoop-foo\mapred\staging\foo\.staging to 0700

My spring dev box is a windows machine.

Regards

[msduk]

After further investigation it seems to be a possible bug in the latest release that has returned from an earlier version. So far the only solution is to rebuild after killing out the exception thrown from FileUtils.

Cygwin does not help in this case as 700 is not valid as far as windows is concerned.

Some tips for my windows friends. To do the rebuild you can delete all the exec calls that fail. you will still get a working hadoop-core at the end of it.

Jackson is needed by different dependencies with method and classes that dont exist in differing versions. I tried many from 1.1.0 to 1.9.5 and the only one I found that worked is 1.4.0

This worked for me as a HelloWorld/HelloNietzsche-chapter-1


<hdp:configuration >
fs.default.name=hdfs://foo:9000
dfs.permissions=false
hadoop.tmp.dir=c:/input
</hdp:configuration>


<bean id="runner" class="org.springframework.data.hadoop.mapreduce.JobRunne r" p:jobs-ref="job"/>

<hdp:job validate-paths="false" id="job"
input-path="/tmp/hadoop-root/mapred/system/input" output-path="/tmp/hadoop-root/mapred/system/output6"
mapper="org.apache.hadoop.examples.WordCount.TokenizerMapp er"
reducer="org.apache.hadoop.examples.WordCount.IntSumReducer"/>

[Costin Leau]

Thanks for reporting your findings. Hadoop 1.0.0 doesn't play well with windows (in fact, versions 0.20.203 and upwards I think). as you pointed out this has to do with the fact that some permissions can be set for the staging repo which causes Hadoop to complain.
In our test suite (and the sample test), we use a little trick by changing the default permissions - take a look at the tests and notice the line called during the constructor.

[lironn]

Hi,
I've encountered the same permissions issue.
I tried calling PermissionUtils.hackHadoopStagingOnWin(), but it doesn't help.
Any other suggestion?