starting up hive within my spring data app

[Costin Leau]

I will try later in the day to test against CDH, maybe they have a different setup then the vanilla Hive (I've tested against the official
vanilla Hive 0.8,0.9 and 0.10) - what version are you using?

the CNF might be more then a red herring - typically if a CNF is raised, the job won't fail right away but rather after it times out. This looks like your case as well. Can you enable more logging for Hive (note you have to do this inside your app) and also check the Hadoop jobs just in case ?

[drosenstark]

Thanks a lot for all your help!
we are using CDH4.1.2
I have tried to figure out how to add more logging -- other than the hive logging to the log4j - i.e - how can i start up hive in the app context with verbose mode. I can change the log4j of hive but not sure hive bootstrapped from within my spring is reading that.

As for checking the hadoop jobs, i am a newbie to hadoop -- have been usign hive with great success but not sure how to debug. What i do know as i have stressed is that the hive query works

[Costin Leau]

Hi,

I've tried the same scenario (Hive server started on Windows machine, Hive test running against on the Windows machine, Hadoop cluster running remotely).
In all cases, due to the change of version I had to nuke metadata_db (and change the hostname of the Hadoop VM but that's because I'm testing different distros).
CDH3u5 worked without any issues.
I have then tested using CDH 4.1.3 on the client with CDH 4.1.1 running inside the Hadoop VM - Hive threw some errors in the logs but those proved to not be harmful (they're mainly about indexes).
The full log (plus some gradle stuff) is available here: https://gist.github.com/costin/f4c0745eae071cb5214d

Note that as opposed to CDH 3 and vanilla Hadoop, I had to run this from a cygwin environment (as I'm using windows).
Additionally I had to specify the Hadoop VM by hostname not by IP - while CDH3 and vanilla Hive allow this, CDH4 does not - one gets an exception soon after.
The test is fairly comprehensive - it creates tables, adds some data and then some queries - and all passed.

Hope this helps,

[drosenstark]

you started up the hive server remotely? I did not know that is possible since only port is settable in the XML i have in spring data.
Can i see the applicationContext?

[drosenstark]

I set all spring logs to debug and i see many lines about "Handling deprecation for" -- is this normal?

[drosenstark]

I think i found the issue. I ran the app both ways and i see that either way an error is generated but when run out process, it seems to be handled just fine but in the spring app, things dont go as well:
This is the log from the "healthy" run of the hive script. (out of process)

2013-02-14 11:08:28,268 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(416)) - 1: Metastore shutdown complete.
2013-02-14 11:08:28,269 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(232)) - ugi=hdfs ip=unknown-ip-addr cmd=Metastore shutdown complete.
2013-02-14 11:08:28,269 ERROR server.TThreadPoolServer (TThreadPoolServer.java:run(182)) - Error occurred during processing of message.
java.lang.NullPointerException
at org.apache.thrift.ProcessFunction.process(ProcessF unction.java:34)
at org.apache.thrift.TBaseProcessor.process(TBaseProc essor.java:34)
at org.apache.thrift.server.TThreadPoolServer$WorkerP rocess.run(TThreadPoolServer.java:176)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
@

[Costin Leau]

I don't think this is the actual problem - this occurs during the hive shutdown process after the metastore is closed.
Now if this is what actually shows up in your test (the NPE inside the close-up) there's not really much we can do since as you've seen as well, when executing on the server side this occurs during shutdown so there are no further reports.
Same thing happens in my tests but I'm closing the server after the test executes and ignoring any exceptions (the testing is complete).
For your case I recommend trying to ignore this exception after Hive is being closed - it's not pretty I know but this is how things are - it basically seems the Thrift processor doesn't handle the shutdown nicely...