Diagnostics: Hive CLI is hanging On HDP

On our HDP, some time Hive CLI shell just hangs.

Hint -1

One thing to check is if you are using tez. It could be possible that you do not have any capacity on yourr YARN cluster to launch AM (Application Master). So the Hive CLI waits indefinitely to launch an AM. One thing they can do to alleviate this is to launch hive cli as "hive -hiveconf hive.execution.engine=mr".

[hive@te1-hdp-rp-nn01 root]$ hive -hiveconf hive.execution.engine=mr

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.1.0-129/0/hive-log4j.properties

hive>

That happens as Tez pre-allocates resources, it starts a YARN application as soon as you invoke the Hive CLI. In case of MR a YARN application is launched only when its required by the executing query. This is a performance optimization in Hive on Tez saving the user the overhead of launching a AM everytime a new query is run in the same Hive CLI session.

Hint-2

The issue may arise immediately after a configuration change is set using ambari. Beeline would then connect fine, but the Hive CLI will hang. This would then appear to the customer as being introduced by ambari . In our case it was Zeppelin change.

Hint-3 

Try setting the HiveCLI to startup with DEBUG logging?

[hive@te1-hdp-rp-nn01 root]$ hive -hiveconf hive.log.file=hivecli_debug.log -hiveconf hive.log.dir=/tmp/hivecli -hiveconf hive.root.logger=DEBUG,DRFA

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.1.0-129/0/hive-log4j.properties

hive>

Hint-4

Restart Ambari Server

Hint-5

Hivemetastore not responding, check it for issues

Hint-6

You can access resource manager UI to see existing jobs running. By default resource manager runs on port 8088. See if the following takes you to Resource Manager UI.

http://<host where Resource manager is running>:8088

OR



Get list of all SUBMITTED, ACCEPTED or RUNNING applications.

From this you can filter applications of default queue by running 

yarn application -list | grep default

yarn application -list
yarn application -kill appid 

Then try running hive cli

Hint-7

Whether you have enough memory to run hive shell on that node.

[hive@te1-hdp-rp-nn01 root]$ free -m

              total        used        free      shared  buff/cache   available

Mem:          31979       12600        2135         549       17243       18376

Swap:         33377          17       33360

How many hive shell sessions running?

[hive@te1-hdp-rp-nn01 root]$ ps -aef|grep hive

Hint-8

Use beeline instead of Hive CLI

[hive@te1-hdp-rp-dn04 ~]$ beeline -u jdbc:hive2://te1-hdp-rp-dn04:10000/scott -n hive -p hive

Connecting to jdbc:hive2://te1-hdp-rp-dn04:10000/scott

Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)

Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)

Transaction isolation: TRANSACTION_REPEATABLE_READ

Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive

0: jdbc:hive2://te1-hdp-rp-dn04:10000/scott> !tables