HIVE的部署

0.前期准备

启动mysql和hadoop

1.下载hive的tar包

https://dlcdn.apache.org/hive/选择需要的版本,我这里部署hive-3.1.2版本:

apache-hive-3.1.2-bin.tar.gz

2.通过rz命令上传到服务器并解压,

1
2
3
4
5
6
7
8
[hadoop@hadoop001 ~]$ cd software/
[hadoop@hadoop001 software]$ rz
[hadoop@hadoop001 software]$ tar -zvxf apache-hive-3.1.2-bin.tar.gz
[hadoop@hadoop001 software]$ ll
total 1109404
drwxrwxr-x. 10 hadoop hadoop 4096 Dec 27 00:09 apache-hive-3.1.2-bin
[hadoop@hadoop001 software]$ cd ~/app
[hadoop@hadoop001 app]$ ln -s hive /home/hadoop/software/apache-hive-3.1.2-bin

3.配置环境变量

1
[hadoop@hadoop001 ~]$ vi ~/.bash_profile   
1
2
export HIVE_HOME=/home/hadoop/app/hive
export PATH=$HIVE_HOME/bin:$PATH

4.拷贝mysql的驱动到lib下

1
[hadoop@hadoop001 ~]$ mv mysql-connector-java-5.1.47.jar app/hive/lib/

5.配置hive-site.xml

hive-site.xml配置mysql相关信息(hive-site.xml这个配置文件是配置元数据的相关信息,元数据存放在mysql中)

hive-site.xml所在目录 /home/hadoop/app/hive/conf/,如果没有vi创建。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://hadoop001:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>Username to use against metastore database</description>
</property>

<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
<description>password to use against metastore database</description>
</property>

<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>

<property>
<name>hive.cli.print.header</name>
<value>true</value>
</property>

<property>
<name>hive.cli.print.current.db</name>
<value>true</value>
</property>
</configuration>

6.初始化

1
schematool -dbType mysql -initSchema

7.启动hive

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[hadoop@hadoop001 conf]$ hive
which: no hbase in (/home/hadoop/app/hive/bin:/home/hadoop/app/scala/bin:/home/hadoop/app/hadoop/bin:/home/hadoop/app/hadoop/sbin:/home/hadoop/app/protobuf/bin:/home/hadoop/app/maven/bin:/home/hadoop/app/scala/bin:/home/hadoop/app/hadoop/bin:/home/hadoop/app/hadoop/sbin:/home/hadoop/app/protobuf/bin:/home/hadoop/app/maven/bin:/usr/local/mysql/bin:/usr/java/jdk1.8.0_45/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/software/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/software/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 70f578b4-d4d6-4236-a82f-580ff0ca44a3

Logging initialized using configuration in jar:file:/home/hadoop/software/apache-hive-3.1.2-bin/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = c1629998-6455-4976-a8c2-c97d2f94104c
hive (default)> show databases;
OK
database_name
default
Time taken: 0.779 seconds, Fetched: 1 row(s)
hive (default)>

8.在mysql中查看hive的元信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive |
| mysql |
| performance_schema |
| ruozedata |
| sys |
+--------------------+
6 rows in set (0.01 sec)

mysql> use hive;
Database changed
mysql> show tables;
+-------------------------------+
| Tables_in_hive |
+-------------------------------+
| aux_table |
| bucketing_cols |
| cds |
| columns_v2 |
| compaction_queue |
| completed_compactions |
| completed_txn_components |
| ctlgs |
| database_params |
| db_privs |
| dbs |
| delegation_tokens |
| func_ru |
| funcs |
| global_privs |
| hive_locks |
| i_schema |
| idxs |
| index_params |
| key_constraints |
| master_keys |
| materialization_rebuild_locks |
| metastore_db_properties |
| min_history_level |
| mv_creation_metadata |
| mv_tables_used |
| next_compaction_queue_id |
| next_lock_id |
| next_txn_id |
| next_write_id |
| notification_log |
| notification_sequence |
| nucleus_tables |
| part_col_privs |
| part_col_stats |
| part_privs |
| partition_events |
| partition_key_vals |
| partition_keys |
| partition_params |
| partitions |
| repl_txn_map |
| role_map |
| roles |
| runtime_stats |
| schema_version |
| sd_params |
| sds |
| sequence_table |
| serde_params |
| serdes |
| skewed_col_names |
| skewed_col_value_loc_map |
| skewed_string_list |
| skewed_string_list_values |
| skewed_values |
| sort_cols |
| tab_col_stats |
| table_params |
| tbl_col_privs |
| tbl_privs |
| tbls |
| txn_components |
| txn_to_write_id |
| txns |
| type_fields |
| types |
| version |
| wm_mapping |
| wm_pool |
| wm_pool_to_trigger |
| wm_resourceplan |
| wm_trigger |
| write_set |
+-------------------------------+
74 rows in set (0.00 sec)

mysql>

9.其他(部署过程中遇到的问题)

  • Hive启动报错:java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument

    错误原因:系统找不到这个类所在的jar包或者jar包的版本不一样系统不知道使用哪个。hive启动报错的原因是后者

    解决办法:

    1、com.google.common.base.Preconditions.checkArgument这个类所在的jar包为:guava.jar

    2、hadoop-3.2.1(路径:hadoop\share\hadoop\common\lib)中该jar包为 guava-27.0-jre.jar;而hive-3.1.2(路径:hive/lib)中该jar包为guava-19.0.1.jar

    3、将jar包变成一致的版本:删除hive中低版本jar包,将hadoop中高版本的复制到hive的lib中。

    再次启动问题得到解决!

  • FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.me

    原因分析:
    是由于没有初始化数据库导致,执行名称初始化数据库即可。

    解决办法:
    执行命令:schematool -dbType mysql -initSchema