Close Menu

    How to Install Apache Hive with Hadoop on CentOS, Ubuntu and LinuxMint

    By 2 Mins Read

    What is Apache Hive ? The Apache Hive data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using an SQL-like language called HiveQL. Read More on Official site

    This how-to guide will help you to Install Apache Hive on CentOS/RHEL with Hadoop with easy steps.

    Step 1: Install JAVA and Hadoop

    Apache Hive required java 6 or later version. We also need to install hadoop first before installing apache hive on our system. Use below links to install them

    Steps to Install JAVA on CentOS and RHEL 7/6/5
    Steps to Install Hadoop on Linux

    Step 2: Download Hive Archive

    After configuring hadoop successfully on your linux system. lets start hive setup. First download latest hive source code and extract archive using following commands.

    # cd /home/hadoop
# wget http://archive.apache.org/dist/hive/hive-0.12.0/hive-0.12.0-bin.tar.gz
# tar xzf hive-0.12.0-bin.tar.gz
# mv hive-0.12.0-bin hive
# chown -R hadoop hive

    Step 3: Setup Environment Variables

    After extracting hive archive file, switch to hadoop user and setup following environment variables.

    # su - hadoop
$ export HADOOP_HOME=/home/hadoop/hadoop
$ export HADOOP_PREFIX=/home/hadoop/hadoop
$ export HIVE_HOME=/home/hadoop/hive
$ export PATH=$HIVE_HOME/bin:$PATH

    Step 4: Start Hive

    Before running hive we need to create /tmp and /user/hive/warehouse and set them chmod g+w in HDFS before create a table in Hive. Use the following commands.

    $ cd /home/hadoop/hive
$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

    Lets start using hive using following command.

    $ bin/hive

Logging initialized using configuration in jar:file:/home/hadoop/hive/lib/hive-common-0.12.0.jar!/hive-log4j.properties
hive>

    Step 5: Create Demo Table and Test

    At this stage you have successfully installed hive. Lets create a sample table using following command

    hive>  CREATE TABLE demo1 (id int, name string);
OK
Time taken: 6.565 seconds

    Show the created tables with below command.

    hive> SHOW TABLES;
OK
demo1
Time taken: 0.231 seconds, Fetched: 1 row(s)

    Drop the table using below command.

    hive> DROP TABLE demo1;
OK
Time taken: 2.393 seconds

    Read more about SQL operations on hive on apache hive official site.

    Share.

    Related Posts

    View 13 Comments

    13 Comments

    1. Christos on

      hello I am trying to run hive but i got the following error:

      WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      2018-07-25 10:31:25,026 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
      2018-07-25 10:31:25,029 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
      2018-07-25 10:31:25,656 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
      log4j:ERROR Could not instantiate class [org.apache.hadoop.hive.shims.HiveEventCounter].
      java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:154)
      at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:113)
      at org.apache.hadoop.hive.shims.ShimLoader.getEventCounter(ShimLoader.java:98)
      at org.apache.hadoop.hive.shims.HiveEventCounter.(HiveEventCounter.java:34)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at java.lang.Class.newInstance(Class.java:442)
      at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:337)
      at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
      at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
      at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
      at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
      at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:415)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:124)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:77)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:630)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
      log4j:ERROR Could not instantiate appender named “EventCounter”.

      Logging initialized using configuration in jar:file:/home/hadoop/hive/lib/hive-common-0.12.0.jar!/hive-log4j.properties
      Exception in thread “main” java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:286)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:368)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)
      … 8 more
      Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:53)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
      at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:365)
      … 9 more
      Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:154)
      at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:113)
      at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:80)
      at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51)

      Reply
    2. Christos on

      I am trying to run hive but I got the following error:
      WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      2018-07-25 10:31:25,026 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
      2018-07-25 10:31:25,028 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
      2018-07-25 10:31:25,029 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
      2018-07-25 10:31:25,656 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
      log4j:ERROR Could not instantiate class [org.apache.hadoop.hive.shims.HiveEventCounter].
      java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:154)
      at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:113)
      at org.apache.hadoop.hive.shims.ShimLoader.getEventCounter(ShimLoader.java:98)
      at org.apache.hadoop.hive.shims.HiveEventCounter.(HiveEventCounter.java:34)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at java.lang.Class.newInstance(Class.java:442)
      at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:337)
      at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
      at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
      at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
      at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
      at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:415)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:124)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:77)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:630)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
      log4j:ERROR Could not instantiate appender named “EventCounter”.

      Logging initialized using configuration in jar:file:/home/hadoop/hive/lib/hive-common-0.12.0.jar!/hive-log4j.properties
      Exception in thread “main” java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:286)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:368)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)
      … 8 more
      Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:53)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
      at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:365)
      … 9 more
      Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
      at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:154)
      at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:113)
      at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:80)
      at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51)
      … 12 more

      Reply
    3. obiero on

      Why I’m getting the error: No such file or directory ?

      hadoop@vpsXXXX [~/hive]# $HADOOP_HOME/bin/hadoop fs -mkdir /usr/hive/warehouse/
      WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
      2018-01-17 13:02:33,608 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
      mkdir: `/usr/hive/warehouse/’: No such file or directory
      hadoop@vpsXXXX [~/hive]#

      Reply
    5. sidharth on

      can uh help me for configuring $HADOOP_HOME path ?
      which one i should prefer…
      HADOOP_HOME=/home/hadoop/hadoop
      ……..
      OR
      …….
      HADOOP_HOME=/opt/hadoop/hadoop
      …..tell me ??

      Reply
    6. sankar on

      Hi Rahul,
      you missed one step here. Below command should be run to create db. Then only you will be able to create table and stuff. If I said correct then, please add this step into it. So that in future whoever viewing this steps will be helpful.
      >> schematool -initSchema -dbType derby<<

      Reply
      • sai on

        Hi Sankar, Tried with the command, but facing below error
        Starting metastore schema initialization to 3.1.0
        Initialization script hive-schema-3.1.0.derby.sql
        Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hive/jdbc/JdbcUriParseException
        at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1213)
        at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1204)
        at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:590)
        at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)
        at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
        Caused by: java.lang.ClassNotFoundException: org.apache.hive.jdbc.JdbcUriParseException
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        … 11 more

        Reply
    8. jems on

      on the previous tutorial about installing hadoop, the HADOOP_HOME=/home/hadoop/hadoop
      but in this tutorial its become HADOOP_HOME=/opt/hadoop/hadoop. should i use the first one or change to /opt ?

      Reply
    Leave A Reply


    Manage Preferences