Facebook Twitter Instagram
    TecAdmin
    • Home
    • FeedBack
    • Submit Article
    • About Us
    Facebook Twitter Instagram
    TecAdmin
    You are at:Home»BIG-DATA»How to Install and Configure Apache Hadoop on CentOS & Fedora

    How to Install and Configure Apache Hadoop on CentOS & Fedora

    By RahulOctober 28, 20225 Mins Read

    Having been around for some time now, Hadoop has become one of the most popular open-source big data solutions. It processes data in batches and is famous for its scalable, cost-effective, and distributed computing capabilities. It’s one of the most popular open-source frameworks in the data analysis and storage space. As a user, you can use it to manage your data, analyze that data, and store it again – all in an automated way. With Hadoop installed on your Fedora system, you can access important analytic services with ease.

    Advertisement

    This article covers how to install Apache Hadoop on CentOS and Fedora systems. In this article, we’ll show you how to install Apache Hadoop on Fedora for local usage as well as a production server.

    1. Prerequsities

    Java is the primary requirement for running Hadoop on any system, So make sure you have Java installed on your system using the following command. If you don’t have Java installed on your system, use one of the following links to install it first.

    • How to Install JAVA 8 on CentOS/RHEL 7/6/5

    2. Create Hadoop User

    We recommend creating a normal (nor root) account for Hadoop working. To create an account using the following command.

    adduser hadoop
    passwd hadoop
    

    After creating the account, it also required to set up key-based ssh to its own account. To do this use execute following commands.

    su - hadoop
    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    chmod 0600 ~/.ssh/authorized_keys
    

    Let’s verify key based login. Below command should not ask for the password but the first time it will prompt for adding RSA to the list of known hosts.

    ssh localhost
    exit
    

    3. Download Hadoop 3.1 Archive

    In this step, download hadoop 3.1 source archive file using below command. You can also select alternate download mirror for increasing download speed.

    cd ~
    wget http://www-eu.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz
    tar xzf hadoop-3.1.0.tar.gz
    mv hadoop-3.1.0 hadoop
    

    4. Setup Hadoop Pseudo-Distributed Mode

    4.1. Setup Hadoop Environment Variables

    First, we need to set environment variable uses by Hadoop. Edit ~/.bashrc file and append following values at end of file.

    export HADOOP_HOME=/home/hadoop/hadoop
    export HADOOP_INSTALL=$HADOOP_HOME
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
    

    Now apply the changes in the current running environment

    source ~/.bashrc
    

    Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable. Change the JAVA path as per install on your system. This path may vary as per your operating system version and installation source. So make sure you are using correct path.

    export JAVA_HOME=/usr/lib/jvm/java-8-oracle
    

    4.2. Setup Hadoop Configuration Files

    Hadoop has many of configuration files, which need to configure as per requirements of your Hadoop infrastructure. Let’s start with the configuration with basic Hadoop single node cluster setup. first, navigate to below location

    cd $HADOOP_HOME/etc/hadoop
    

    Edit core-site.xml

    <configuration>
    <property>
      <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    </configuration>
    

    Edit hdfs-site.xml

    <configuration>
    <property>
     <name>dfs.replication</name>
     <value>1</value>
    </property>
    
    <property>
      <name>dfs.name.dir</name>
        <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
    </property>
    
    <property>
      <name>dfs.data.dir</name>
        <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
    </property>
    </configuration>
    

    Edit mapred-site.xml

    <configuration>
     <property>
      <name>mapreduce.framework.name</name>
       <value>yarn</value>
     </property>
    </configuration>
    

    Edit yarn-site.xml

    <configuration>
     <property>
      <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
     </property>
    </configuration>
    

    4.3. Format Namenode

    Now format the namenode using the following command, make sure that Storage directory is

    hdfs namenode -format
    

    Sample output:

    WARNING: /home/hadoop/hadoop/logs does not exist. Creating.
    2018-05-02 17:52:09,678 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG:   host = tecadmin/127.0.1.1
    STARTUP_MSG:   args = [-format]
    STARTUP_MSG:   version = 3.1.0
    ...
    ...
    ...
    2018-05-02 17:52:13,717 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
    2018-05-02 17:52:13,806 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
    2018-05-02 17:52:14,161 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 391 bytes saved in 0 seconds .
    2018-05-02 17:52:14,224 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    2018-05-02 17:52:14,282 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at tecadmin/127.0.1.1
    ************************************************************/
    

    5. Start Hadoop Cluster

    Let’s start your Hadoop cluster using the scripts provides by Hadoop. Just navigate to your $HADOOP_HOME/sbin directory and execute scripts one by one.

    cd $HADOOP_HOME/sbin/
    

    Now run start-dfs.sh script.

    ./start-dfs.sh
    

    Sample output:

    Starting namenodes on [localhost]
    Starting datanodes
    Starting secondary namenodes [tecadmin]
    2018-05-02 18:00:32,565 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    

    Now run start-yarn.sh script.

    ./start-yarn.sh
    

    Sample output:

    Starting resourcemanager
    Starting nodemanagers
    

    6. Access Hadoop Services in Browser

    Hadoop NameNode started on port 9870 default. Access your server on port 9870 in your favorite web browser.

    http://svr1.tecadmin.net:9870/
    

    Now access port 8042 for getting the information about the cluster and all applications

    http://svr1.tecadmin.net:8042/
    

    Access port 9864 to get details about your Hadoop node.

    http://svr1.tecadmin.net:9864/
    

    7. Test Hadoop Single Node Setup

    7.1. Make the HDFS directories required using following commands.

    bin/hdfs dfs -mkdir /user
    bin/hdfs dfs -mkdir /user/hadoop
    

    7.2. Copy all files from local file system /var/log/httpd to hadoop distributed file system using below command

    bin/hdfs dfs -put /var/log/apache2 logs
    

    7.3. Browse Hadoop distributed file system by opening below URL in the browser. You will see an apache2 folder in the list. Click on the folder name to open and you will find all log files there.

     http://svr1.tecadmin.net:9870/explorer.html#/user/hadoop/logs/
    

    7.4 – Now copy logs directory for hadoop distributed file system to local file system.

    bin/hdfs dfs -get logs /tmp/logs
    ls -l /tmp/logs/
    

    You can also check this tutorial to run wordcount mapreduce job example using command line.

    hadoop hadoop setup jobtracker jps namenode tasktracker
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email WhatsApp

    Related Posts

    Using HDFS Filesystem (CLI)

    Creating Directory In HDFS And Copy Files (Hadoop)

    How to Install Hadoop on Ubuntu 22.04

    How to Install Apache Hadoop on Ubuntu 22.04

    How to Install and Configure Hadoop on Ubuntu 20.04

    View 103 Comments

    103 Comments

    1. Ashwani Mishra on May 9, 2020 10:28 am

      Out of so many i tried this one worked .. thanks a ton ..

      Reply
    2. vanetoj on September 30, 2019 5:39 pm

      i m getting this error : Error: Cannot find configuration directory: /etc/hadoop . any ideas ?

      Reply
    3. Waleed Ali on December 24, 2018 11:59 am

      HI, I follow your tutorial and its very nice and helpful for me but I am stuck at its accessing in browser.
      I am done till Start Hadoop Cluster.
      Starting namenodes on [localhost]
      Starting datanodes
      Starting secondary namenodes [hadoop-server]
      2018-12-24 11:32:40,637 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

      Starting resourcemanager
      Starting nodemanagers

      As you can see its starting but still couldn’t access in browser.
      I’ll very thankful if someone help me in this.
      Thanks

      Reply
    4. James Jithin on October 17, 2018 7:00 am

      Thanks. This post is helpful in the datanode and namenode configuration which is missing in the guide of hadoop.

      Reply
    5. Surya on September 17, 2018 11:00 pm

      Hi Rahul

      I have received an error
      WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
      Could you please help out me

      Thanks,
      Surya

      Reply
    6. pk_park on June 15, 2018 2:53 pm

      Hello teacher ~!
      How can I deal this under the error ?

      WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
      ERROR: Invalid HADOOP_YARN_HOME

      Reply
    7. Anisha on May 31, 2018 1:53 pm

      bin/hdfs dfs -mkdir /user
      When I run this, it says
      -su: bin/hdfs: No such file or directory
      Please help

      Reply
    8. David on May 26, 2018 12:02 am

      For the steps showing here, does it work on Red Hat Linux 7.5?
      I have tried. But not able to make it work. Here is the error:

      [[email protected] hadoop]$ hdfs namenode -format
      2018-05-25 23:59:16,727 INFO namenode.NameNode: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NameNode
      STARTUP_MSG: host = java.net.UnknownHostException: unallocated.barefruit.co.uk: unallocated.barefruit.co.uk: Name or service not known
      STARTUP_MSG: args = [-format]
      STARTUP_MSG: version = 3.1.0
      STARTUP_MSG: classpath = ……

      Thank you in advance.

      David

      Reply
      • Rahul K. on May 28, 2018 4:34 am

        It looks your system hostname is “unallocated.barefruit.co.uk”, which is not resolving. Edit /etc/hosts file bind hostname with system ip.

        127.0.0.1 unallocated.barefruit.co.uk

        Reply
    9. OmarK on May 21, 2018 5:53 pm

      PROBLEM WITH 7.2 –
      I finished most of this tutorial but am now stuck on finding var/log/apache2
      – after running a few commands to search my system for apache2, I don’t get back any file or directory called apache2. Does anyone have any insight on this?

      Reply
    10. hema on May 2, 2018 11:00 am

      Hi Rahul,

      This link is not working,
      wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz

      pls check

      Reply
      • Rahul K. on May 3, 2018 6:38 am

        Hi Hema, Thanks for pointing out.

        This tutorial has been updated to the latest version of Hadoop installation.

        Reply
    11. DINAKAR N K on October 12, 2017 10:32 am

      Dear Mr Rahul,
      Thank for your extended support on hadoop. Here i have small doubt “how to share single node hadoop cluster ” for a number of 60 clinets. Is there any possibility to do so. Please help

      Regards
      Dinakar N K

      Reply
    12. santhosh on September 22, 2017 5:40 am

      hi…..

      Reply
    13. Rinson on May 16, 2016 12:49 pm

      Good Article…Thanks..:)

      Reply
    14. som on February 29, 2016 12:33 pm

      The information given is incomplete need to export more variables.

      Reply
    15. Shankar on February 21, 2016 9:56 pm

      Hi Rahul,

      This is just awesome!!! I have had some issues during installation but your guide to installing a hadoop cluster is just great !!
      I still used the vi editor for editing the .bashrc file and all other xml files. Just the lack of linux editor knowledge I did it. I used the 127.0.0.1 instead of localhost as the alias names are not getting resolved.

      Moreover there are some issues like the one below
      16/02/22 03:05:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

      [[email protected] hadoop]$ sudo apt-get install openssh-client
      sudo: apt-get: command not found

      The above are the two issues that I’m facing but other than that… it has been a “Walk in the Park” with this tutorial

      Thanks Buddy!!

      Shankar

      Reply
    16. Sergey on January 26, 2016 2:18 pm

      Do you have some idea ?

      16/01/26 09:17:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
      Starting namenodes on [localhost]
      localhost: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
      localhost: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
      Starting secondary namenodes [0.0.0.0]
      0.0.0.0: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
      16/01/26 09:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

      Reply
      • Rahul on January 27, 2016 5:58 am

        Hi Sergey,

        It looks you don’t have openssh-client package installed. Please install it.

        $ sudo apt-get install openssh-client

        Reply
        • Jhunong on April 13, 2017 2:50 am

          hi rahul i followed your setup here but unfortunately i ended up some errors like the one below.

          [[email protected] hadoop-2.8.0]$ bin/hdfs namenode -format
          Error: Could not find or load main class org.apache.hadoop.hdfs.server.namenode.NameNode

          appreciate some help…thanks

          Reply
    17. sasha on January 24, 2016 12:49 am

      Hi rahul,
      good job,
      Now expend a bit more and next time show integration of spark and flume.
      If you need help ping me and I will provide you with my additions to your config …

      Regards,

      Reply
    18. Adrian on January 5, 2016 2:28 pm

      Can anybody can tell me where is bin/hdfs in Step 7 please ?

      For me it says “Not a directory”

      Reply
      • Dendrok on January 23, 2016 10:15 am

        This can be misleading. What it is saying is from the bin folder within the hadoop system folder.

        $ cd $HADOOP_HOME/bin
        $ hdfs dfs -mkdir /user
        $ hdfs dfs -mkdir/user/hadoop

        Reply
    19. HalfDone on December 17, 2015 2:06 am

      bash: /home/hadoop/.bashrc: Permission denied

      I guess that the CentOS 7/Hadoop 2.7.1 tutorial was very helpful until step 4, when by some reason the instructions just explained what to do with .bashrc without explaining how to get to it and how to edit it in first place. Thanks anyway, I just need to find a tutorial that explains with detail how to set up Hadoop.

      Reply
    20. ayoola ajiboye on December 8, 2015 2:23 pm

      Thanks. I read your article have successfully deployed my first single node hadoop deployment despite series of unsuccessful attempts in the past. thanks

      Reply
    21. hoangthang on September 25, 2014 9:30 am

      Hi Rahul,
      Please help me! I installed follow your guide. when i run jps that result below:
      18118 Jps
      18068 TaskTracker
      17948 JobTracker
      17861 SecondaryNameNode
      17746 DataNode

      however when I run stop-all.sh command that
      no jobtracker to stop
      localhost: no tasktracker to stop
      no namenode to stop
      localhost: no datanode to stop
      localhost: no secondarynamenode to stop
      Can you explan for me? Thanks so much!

      Reply
    22. ak on August 21, 2014 11:39 am

      Hai Rahul,

      Can u plz tell me how to configure multiple datanodes on a single machine.I am using hadoop2.5

      Reply
    23. Rahul on July 25, 2014 6:18 am

      HI Rahul,

      I am not able to start the hadoop services..getting an error like>

      14/07/24 23:11:27 INFO namenode.NameNode: SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
      ************************************************************/
      [email protected] hadoop]$ bin/start-all.sh
      bash: bin/start-all.sh: No such file or directory

      Reply
    24. Vijay on June 10, 2014 8:58 am

      Hi Rahul,
      When I execute the command $ bin/start-all.sh, I am getting the FATAL ERROR as shown below
      –====
      [[email protected] hadoop]$ sbin/start-all.sh
      This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
      [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
      14/06/10 01:16:14 FATAL conf.Configuration: error parsing conf core-site.xml
      .
      .
      .
      .
      tarting namenodes on []
      localhost: starting namenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-namenode-localhost.localdomain.out
      localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
      localhost: starting datanode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-datanode-localhost.localdomain.out
      localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
      [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
      14/06/10 01:16:25 FATAL conf.Configuration: error parsing conf core-site.xml
      org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/etc/hadoop/core-site.xml; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.

      .
      .

      .
      starting yarn daemons
      starting resourcemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out
      [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
      localhost: starting nodemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-nodemanager-localhost.localdomain.out
      localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
      –===
      where core-site.xml edited like this
      $vim core-site.xml

      fs.default.name
      hdfs://localhost:9000/

      dfs.permissions
      false

      #########################################

      please help me to overcome this error.

      Reply
    25. Shreedhar on June 3, 2014 2:44 pm

      Thanks a lot.
      I made a few modifications but the the instructions are on the money!

      If you get
      WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable use the instructions from below
      http://stackoverflow.com/questions/20011252/hadoop-2-2-0-64-bit-installing-but-cannot-start

      Reply
    26. Kullayappa on May 28, 2014 6:57 am

      Hi,

      Your artcile is simply super, I followed each and every step and installed hadoop.

      my suggestion is: while checking status with “jps”, it was not showing information and I got no such command. I used again export JAVA_HOME stuff. after that JPS working.

      I guess.. these export JAVA_HOME also need to include in some where else.

      Thansk alot for your article buddy

      Reply
    27. Thang Nguyen on May 26, 2014 4:58 am

      Hi Rahul,
      Can you please post some direction to test with the input/output data after installation?
      for example: how can we upload a file to run a simple wordcount and get the output… ?
      Regards,
      Thang Nguyen

      Reply
    28. Elavarasan on April 29, 2014 7:37 am

      fs.default.name
      hdfs://localhost:9000/

      dfs.permissions
      false

      Reply
      • Rahul on April 29, 2014 8:56 am

        Hi

        I could not find file content properly. Can you email me at [email protected].

        Reply
    29. Elavarasan on April 29, 2014 6:37 am

      hi rahul,

      Plz help me for following case
      [FATAL ERROR] core-site.xml:10:2: The markup in the document following the root element must be well-formed.

      error while running “bin/start-all.sh” on all *.xml files

      Reply
      • Rahul on April 29, 2014 6:59 am

        Hi,

        Please provide your core-site.xml file content

        Reply
    30. Elavarasan on April 29, 2014 6:33 am

      localhost: [fatal Error] core-site.xml:10:2: The markup in the document following the root element must be well-formed.

      error while running “bin/start-all.sh”

      Reply
    31. Elavarasan on April 28, 2014 9:56 am

      Hi Rahul,

      jps 0r $JAVA_HOME/bin/jps giving error bash:jps: command not found and bash: /bin/jps: no such file or directory….. Kindly address the error…

      Reply
      • Rahul on April 28, 2014 10:14 am

        HI Elavarasa,

        Please make sure you have setup JAVA_HOME environment variable. Please provide output of following commands.

        # env | grep JAVA_HOME
        # echo $JAVA_HOME

        Reply
    32. sandeep on April 9, 2014 8:14 am

      Hi Rahul,
      I have successfully done the installation of single node cluster and able to see all daemons running.
      But i am not able to run hadoop fs commands , for this should i install any thing else like jars??

      Thanks in advance…

      Reply
    33. SS on March 12, 2014 8:21 pm

      How do we add data to the single node cluster.

      Reply
    34. Raj on February 28, 2014 4:01 am

      Hi Rahul,

      Do you know what is the best site to download Pig and hive? I realized that I am unable to run and pig and hive. I thought it comes with the package just like while setting-up under cloudera.

      Reply
    35. Raj on February 28, 2014 1:38 am

      Hi Rahul,
      I just restarted the hadoop and JPS is working fine.

      Reply
    36. Raj on February 28, 2014 1:35 am

      Hi Rahul,
      jps is not showing under $JAVA_HOME/bin

      It comes out with an error no such file or directory

      Also, once I complete my tasks and comes out of linux, do I need to restart the hadoop?

      Reply
    37. raj on February 27, 2014 10:13 am

      Hi Rahul,

      Your instructions are superp. The only issue I am facing is a “jps” not getting recognized…. Not sure where the issue could be. Otherwise all set to go.

      Reply
      • Rahul on February 27, 2014 12:00 pm

        Hi Raj,
        Try following command for jps.

        $JAVA_HOME/bin/jps

        Reply
    38. Mursaleen Javed on February 2, 2014 10:41 pm

      [[email protected] hadoop]$ bin/hadoop namenode -format
      DEPRECATED: Use of this script to execute hdfs command is deprecated.
      Instead use the hdfs command for it.

      14/02/03 03:38:23 INFO namenode.NameNode: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NameNode
      STARTUP_MSG: host = java.net.UnknownHostException: RHEL: RHEL
      STARTUP_MSG: args = [-format]
      STARTUP_MSG: version = 2.2.0
      STARTUP_MSG: classpath = /opt/hadoop/hadoop/etc/hadoop:/opt/hadoop/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.6.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-math-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/hdfs:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/junit-4.10.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.10.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.
      inject-1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar
      STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by ‘hortonmu’ on 2013-10-07T06:28Z
      STARTUP_MSG: java = 1.6.0_20
      ************************************************************/
      14/02/03 03:38:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
      14/02/03 03:38:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
      14/02/03 03:38:25 WARN common.Util: Path /opt/hadoop/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
      14/02/03 03:38:25 WARN common.Util: Path /opt/hadoop/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
      Formatting using clusterid: CID-c4662030-0d86-4263-aed9-043fd34565cd
      14/02/03 03:38:25 INFO namenode.HostFileManager: read includes:
      HostSet(
      )
      14/02/03 03:38:25 INFO namenode.HostFileManager: read excludes:
      HostSet(
      )
      14/02/03 03:38:25 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
      14/02/03 03:38:25 INFO util.GSet: Computing capacity for map BlocksMap
      14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
      14/02/03 03:38:25 INFO util.GSet: 2.0% max memory = 966.7 MB
      14/02/03 03:38:25 INFO util.GSet: capacity = 2^21 = 2097152 entries
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: defaultReplication = 2
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: maxReplication = 512
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: minReplication = 1
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
      14/02/03 03:38:25 INFO blockmanagement.BlockManager: encryptDataTransfer = false
      14/02/03 03:38:25 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
      14/02/03 03:38:25 INFO namenode.FSNamesystem: supergroup = supergroup
      14/02/03 03:38:25 INFO namenode.FSNamesystem: isPermissionEnabled = true
      14/02/03 03:38:25 INFO namenode.FSNamesystem: HA Enabled: false
      14/02/03 03:38:25 INFO namenode.FSNamesystem: Append Enabled: true
      14/02/03 03:38:25 INFO util.GSet: Computing capacity for map INodeMap
      14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
      14/02/03 03:38:25 INFO util.GSet: 1.0% max memory = 966.7 MB
      14/02/03 03:38:25 INFO util.GSet: capacity = 2^20 = 1048576 entries
      14/02/03 03:38:25 INFO namenode.NameNode: Caching file names occuring more than 10 times
      14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
      14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
      14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
      14/02/03 03:38:25 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
      14/02/03 03:38:25 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
      14/02/03 03:38:25 INFO util.GSet: Computing capacity for map Namenode Retry Cache
      14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
      14/02/03 03:38:25 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
      14/02/03 03:38:25 INFO util.GSet: capacity = 2^15 = 32768 entries
      Re-format filesystem in Storage Directory /opt/hadoop/hadoop/dfs/name ? (Y or N) y
      14/02/03 03:38:29 WARN net.DNS: Unable to determine local hostname -falling back to “localhost”
      java.net.UnknownHostException: RHEL: RHEL
      at java.net.InetAddress.getLocalHost(InetAddress.java:1426)
      at org.apache.hadoop.net.DNS.resolveLocalHostname(DNS.java:264)
      at org.apache.hadoop.net.DNS.(DNS.java:57)
      at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
      at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
      14/02/03 03:38:29 WARN net.DNS: Unable to determine address of the host-falling back to “localhost” address
      java.net.UnknownHostException: RHEL: RHEL
      at java.net.InetAddress.getLocalHost(InetAddress.java:1426)
      at org.apache.hadoop.net.DNS.resolveLocalHostIPAddress(DNS.java:287)
      at org.apache.hadoop.net.DNS.(DNS.java:58)
      at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
      at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
      14/02/03 03:38:29 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted.
      14/02/03 03:38:29 INFO namenode.FSImage: Saving image file /opt/hadoop/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
      14/02/03 03:38:29 INFO namenode.FSImage: Image file /opt/hadoop/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 198 bytes saved in 0 seconds.
      14/02/03 03:38:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
      14/02/03 03:38:30 INFO util.ExitUtil: Exiting with status 0
      14/02/03 03:38:30 INFO namenode.NameNode: SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: RHEL: RHEL
      ************************************************************/
      [[email protected] hadoop]$

      hosts file:

      127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
      ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
      RHEL

      please help

      Reply
      • Rahul Kumar on February 3, 2014 5:11 am

        Hi Javed,

        Your systems hosts file entry looks incorrect. Please add entry like below

        127.0.0.1 RHEL

        Reply
    39. Paul on January 8, 2014 2:12 pm

      Hi Rahul!
      Just to say your instructions worked like a dream.
      In my hadoop-env.sh I used.
      export JAVA_HOME=/usr/lib/jvm/jre-1.6.0 <- might help others its a vanilla Centos 6.5 install.
      Cheers Paul

      Reply
    40. Sravan on January 8, 2014 8:32 am

      This explanation is not less to any other I have seen so far… Great efforts Rahul, I ve personally installed psuedo and normal distributed clusters in my workplace using this tutorials.

      Thanks alot buddy 🙂

      Keep doing such good works 🙂

      Reply
      • Rahul Kumar on January 8, 2014 12:41 pm

        Thanks Sravan

        Reply
        • jhon on April 5, 2017 2:19 pm

          hi sir rahul

          i follow all your guide step by steps but encounter some problems while executing
          this command:

          ./start-dfs.sh

          output:

          [[email protected] sbin]# ./start-dfs.sh
          17/04/05 22:14:32 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:32 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
          17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not

          thanks

          Reply
    41. Azim on November 13, 2013 7:00 am

      I am getting following below error when I run “bin/start-all.sh” Please suggest
      [email protected] hadoop]$ bin/start-all.sh
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.localdomain.out
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: datanode running as process 3602. Stop it first.
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: secondarynamenode running as process 3715. Stop it first.
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      jobtracker running as process 3793. Stop it first.
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
      localhost: tasktracker running as process 3921. Stop it first.
      [[email protected] hadoop]$ jvs
      -bash: jvs: command not found

      Reply
    42. jk on November 9, 2013 2:40 am

      Fixed these errors. Checked the logs and got the help from websites.

      Have a question. How can I use/test the wordcount on multiple files in a folder.
      bin/hadoop dfs -copyFromLocal files /testfiles
      bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /files /testfiles — This throws error saying testfiles is not a file. It should work for a directory too right ?

      Thanks

      Reply
    43. jk on November 9, 2013 12:45 am

      oops. Spoke too soon. I do see the JobHistoryServer start – then it stopped it looks like.

      -bash-3.2$ jps
      21984 NameNode
      27080 DataNode
      1638 ResourceManager
      1929 NodeManager
      5718 JobHistoryServer
      6278 Jps

      -bash-3.2$ jps
      21984 NameNode
      27080 DataNode
      11037 Jps
      1638 ResourceManager
      1929 NodeManager

      Where can I see the information as to why this stopped? Can you please suggest. Sorry for multiple posts. But I had to update and dont seem to find any help googling.
      Thanks

      Reply
    44. jk on November 9, 2013 12:41 am

      Hi
      I tried changing the entry in core-site.xml
      hdfs://drwdt001.myers.com:9000
      instead of
      hdfs://drwdt001:9000
      and that helped with the startup of the HistoryServer. Now I can see it.
      Although, now I am not sure if my datanode will restart. I had issues with that startup & thats the reason I removed the domain and just had the servername as the value.

      Any ideas? I will test before I shut and restart
      Thanks.

      Reply
    45. jk on November 9, 2013 12:34 am

      Hi
      I downloaded and installed Apache hadoop 2.2 latest. Followed the above setup for single node ( First time setup.) RHEL 5.5
      Name node, DataNode, ResourceManager, NodeManager started fine. Had some issue with datanode & had to update the IPtables for opening ports.
      when I run
      -bash-3.2$ sbin/mr-jobhistory-daemon.sh start historyserver
      starting historyserver, logging to /hadoop/hadoop-2.2.0/logs/mapred-hduser-historyserver-server.out
      when I run jps, I dont see the JobHistoryServer listed . There are no errors in the out file above.

      Can someone please assist?
      Thanks
      JKH

      Reply
    46. Rodrigo Bittencourt on November 4, 2013 11:46 am

      Hello Rahul,
      I need your help , my cluster does not work because it must have something wrong in the configuration .

      because when I put start-all.sh it does not initialize the package secondarynamenode .

      shows this error

      starting secondarynamenode , logging to / opt/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-lbad012.out

      lbad012 : Exception in thread “main ” java.lang.IllegalArgumentException : Does not contain a valid host : port authority : file :/ / /

      lbad012 : at org.apache.hadoop.net.NetUtils.createSocketAddr ( NetUtils.java : 164 )

      lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress ( NameNode.java : 212 )

      lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress ( NameNode.java : 244 )

      lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress ( NameNode.java : 236 )

      lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize ( SecondaryNameNode.java : 194 )

      lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode . ( SecondaryNameNode.java : 150 )

      lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main ( SecondaryNameNode.java : 676 )

      Reply
    47. Dheeraj on October 30, 2013 12:38 pm

      heyy
      please help.
      wen i am doing the step 6 its asking ffor some password even though i havent set any password.please tell wat to do.

      bin/start-all.sh
      starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-demo.example.com.out
      [email protected]’s password:

      Thanx

      Reply
      • Rahul on October 31, 2013 4:11 am

        Hi Dheeraj,

        I think you don’t have completed Step #3 properly. Please check and complete it. Make sure you are able to ssh without password.

        Reply
    48. Rakesh on October 24, 2013 9:21 pm

      I have configured cloudera on a single node successfully how ever when i lanuch Hue @//myaddress.8888 i see below error message at configuration level.
      ”
      Configuration files located in /var/run/cloudera-scm-agent/process/57-hue-HUE_SERVER

      Potential misconfiguration detected. Fix and restart Hue.”

      and my hue_safety_valve.ini looks as below

      [hadoop]
      [[mapred_clusters]]
      [[[default]]]
      jobtracker_host=servername
      thrift_port=9290
      jobtracker_port=8021
      submit_to=True
      hadoop_mapred_home={{HADOOP_MR1_HOME}}
      hadoop_bin={{HADOOP_BIN}}
      hadoop_conf_dir={{HADOOP_CONF_DIR}}
      security_enabled=false

      Reply
    49. Shannon on October 16, 2013 10:17 pm

      Can you please suggest the steps to uninstall the apache hadoop . I am planning to test cloudera as well , if you can share steps for cloudera as well that would be awesome ! .

      Reply
    50. Toan Nguyen on October 15, 2013 5:59 pm

      Please help me, when I run $ bin/start-all.sh, output have a error,
      starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-hadoop-master.out
      The authenticity of host ‘localhost (::1)’ can’t be established.
      RSA key fingerprint is 7b:6d:cb:fc:48:7b:c6:42:a5:6a:64:83:ab:a8:95:95.
      Are you sure you want to continue connecting (yes/no)? hadoop-slave-2: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave-2.out
      hadoop-slave-1: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave-1.out

      Reply
      • Rahul on October 16, 2013 2:38 am

        Hi Toan,

        Are you configuring hadoop multinode cluster ?

        Reply
    51. Rakesh on October 14, 2013 9:54 pm

      I have found solutions for my issues, thanks rahul for your post it helped me a lot . !!!

      Reply
      • Rahul on October 22, 2013 7:52 am

        Sounds good Rakesh… Thanks

        Reply
    52. Rakesh on October 10, 2013 6:23 pm

      I have tried below urls for job tracker but none of them worked out . Also i confirmed with network team they confirmed that there is no firewall blocking .
      http://ipaddress:50030/
      http://localhost:50030/
      http://servername:50030/

      Reply
    53. Rakesh on October 9, 2013 10:44 pm

      Core xml file for reference

      fs.default.name
      hdfs://hadoop-01:9000/

      dfs.permissions
      false

      Reply
      • Rahul on October 10, 2013 3:42 am

        Hi Rakesh,
        TecAdmin.net commnets does not support HTML tags properly. Can you please show the screenshot of your file..

        Reply
        • Rakesh on October 10, 2013 5:51 pm

          Hi Rahul,
          sorry for my delay response.
          I was able to resolve the above error ,By mistake i deleted the wrapper in the xml which caused the error , I have now kept the right data in xml and found below output after format.

          [[email protected] hadoop]$ /opt/jdk1.7.0_40/bin/jps
          2148 NameNode
          2628 TaskTracker
          2767 Jps
          2503 JobTracker
          2415 SecondaryNameNode
          2274 DataNode
          However in the xml files i have used localhost instead of my server ipaddress is that fine , because i am unable to launch the web urls for the same

          Reply
          • Sudip on October 21, 2013 9:32 am

            Hi Rahul,

            I am trying to install a 3 node Hadoop Cluster and using v2.1.0-beta version. Though I could sense there are a lot of changes in terems of directory structure (as compared to v1.2.1) but finally after configuring most of the steps mentioned here , when I issued the command “$JAVA_HOME/bin/jps” on my master server I am seeing O/P:

            16467 Resource Manager
            15966 NameNode
            16960 Jps
            16255 SecondaryNameNode

            But I cant’t see my DataNode or NodeManager getting started.

            I can send you the ouptut of “start-all.sh ” script output (which throws a lot of error lines but somehow able to start the above mentioned services), if you think that can be of any help to you.

            Secondly I am not able open any pages as mentioned part of web services; e.g.
            http://:50030/ for the Jobtracker -> Resource Manager
            http://:50070/ for the Namenode

            Would appreciate any comment from your side on my queries here.

            Reply
            • Rahul on October 21, 2013 10:36 am

              Hi Sudeep,

              Yes, Please post the output of start-all.sh command with log files. But first plz empty your log files and them run start-all.sh after that post all outputs.

              Also I prefer, if you post your question on our new forum, so it will be better to communicate.

              Reply
            • Sudip on October 22, 2013 5:33 am

              Here is the output of start-all.sh script.

              [[email protected] sbin]$ ./start-all.sh
              This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
              13/10/20 18:45:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
              Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
              It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
              hadoop-master]
              sed: -e expression #1, char 6: unknown option to `s’
              Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
              64-Bit: ssh: Could not resolve hostname 64-Bit: Temporary failure in name resolution
              HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
              Server: ssh: Could not resolve hostname Server: Temporary failure in name resolution
              VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
              warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
              You: ssh: Could not resolve hostname You: Temporary failure in name resolution
              have: ssh: Could not resolve hostname have: Temporary failure in name resolution
              library: ssh: Could not resolve hostname library: Temporary failure in name resolution
              have: ssh: Could not resolve hostname have: Temporary failure in name resolution
              which: ssh: Could not resolve hostname which: Temporary failure in name resolution
              disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
              will: ssh: Could not resolve hostname will: Temporary failure in name resolution
              stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
              guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
              might: ssh: Could not resolve hostname might: Temporary failure in name resolution
              stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
              the: ssh: Could not resolve hostname the: Temporary failure in name resolution
              loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
              VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
              ‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
              to: ssh: Could not resolve hostname to: Temporary failure in name resolution
              that: ssh: Could not resolve hostname that: Temporary failure in name resolution
              try: ssh: Could not resolve hostname try: Temporary failure in name resolution
              highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
              link: ssh: Could not resolve hostname link: Temporary failure in name resolution
              fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
              you: ssh: Could not resolve hostname you: Temporary failure in name resolution
              guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
              fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
              or: ssh: Could not resolve hostname or: Temporary failure in name resolution
              It’s: ssh: Could not resolve hostname It’s: Temporary failure in name resolution
              recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
              the: ssh: Could not resolve hostname the: Temporary failure in name resolution
              -c: Unknown cipher type ‘cd’
              ‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
              with: ssh: Could not resolve hostname with: Temporary failure in name resolution
              with: ssh: Could not resolve hostname with: Temporary failure in name resolution
              now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
              noexecstack’.: ssh: Could not resolve hostname noexecstack’.: Temporary failure in name resolution
              ‘,: ssh: Could not resolve hostname ‘,: Temporary failure in name resolution
              The: ssh: Could not resolve hostname The: Temporary failure in name resolution
              it: ssh: Could not resolve hostname it: Temporary failure in name resolution
              library: ssh: Could not resolve hostname library: Temporary failure in name resolution
              hadoop-master: starting namenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-namenode-hadoop-master.out
              hadoop-master: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
              hadoop-master: It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
              hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
              hadoop-slave-1: bash: /opt/hadoop/hadoop/sbin/hadoop-daemon.sh: No such file or directory
              hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
              hadoop-slave-2: bash: /opt/hadoop/hadoop/sbin/hadoop-daemon.sh: No such file or directory
              Starting secondary namenodes [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
              It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
              0.0.0.0]
              sed: -e expression #1, char 6: unknown option to `s’
              64-Bit: ssh: Could not resolve hostname 64-Bit: Temporary failure in name resolution
              Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
              Server: ssh: Could not resolve hostname Server: Temporary failure in name resolution
              HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
              VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
              warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
              have: ssh: Could not resolve hostname have: Temporary failure in name resolution
              You: ssh: Could not resolve hostname You: Temporary failure in name resolution
              library: ssh: Could not resolve hostname library: Temporary failure in name resolution
              might: ssh: Could not resolve hostname might: Temporary failure in name resolution
              loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
              try: ssh: Could not resolve hostname try: Temporary failure in name resolution
              guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
              stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
              which: ssh: Could not resolve hostname which: Temporary failure in name resolution
              have: ssh: Could not resolve hostname have: Temporary failure in name resolution
              now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
              stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
              The: ssh: Could not resolve hostname The: Temporary failure in name resolution
              guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
              will: ssh: Could not resolve hostname will: Temporary failure in name resolution
              the: ssh: Could not resolve hostname the: Temporary failure in name resolution
              recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
              disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
              VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
              to: ssh: Could not resolve hostname to: Temporary failure in name resolution
              ‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
              highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
              ‘,: ssh: Could not resolve hostname ‘,: Temporary failure in name resolution
              that: ssh: Could not resolve hostname that: Temporary failure in name resolution
              -c: Unknown cipher type ‘cd’
              ‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
              link: ssh: Could not resolve hostname link: Temporary failure in name resolution
              fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
              or: ssh: Could not resolve hostname or: Temporary failure in name resolution
              library: ssh: Could not resolve hostname library: Temporary failure in name resolution
              it: ssh: Could not resolve hostname it: Temporary failure in name resolution
              you: ssh: Could not resolve hostname you: Temporary failure in name resolution
              It’s: ssh: Could not resolve hostname It’s: Temporary failure in name resolution
              with: ssh: Could not resolve hostname with: Temporary failure in name resolution
              with: ssh: Could not resolve hostname with: Temporary failure in name resolution
              fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
              the: ssh: Could not resolve hostname the: Temporary failure in name resolution
              noexecstack’.: ssh: Could not resolve hostname noexecstack’.: Temporary failure in name resolution
              0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-hadoop-master.out
              0.0.0.0: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
              0.0.0.0: It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
              13/10/20 18:45:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
              starting yarn daemons
              starting resourcemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-hadoop-master.out
              hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
              hadoop-slave-1: bash: /opt/hadoop/hadoop/sbin/yarn-daemon.sh: No such file or directory
              hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
              hadoop-slave-2: bash: /opt/hadoop/hadoop/sbin/yarn-daemon.sh: No such file or directory
              [[email protected] sbin]$ $JAVA_HOME/bin/jps
              15130 Jps
              14697 SecondaryNameNode
              14872 ResourceManager
              14408 NameNode
              [[email protected] sbin]$

              Reply
          • Lini on June 25, 2015 1:36 am

            hello..
            i also faced the same issue as mr Rakesh
            while executing the format command

            [[email protected] hadoop]$ bin/hadoop namenode -format
            15/06/24 18:32:24 INFO namenode.NameNode: STARTUP_MSG:
            /************************************************************
            STARTUP_MSG: Starting NameNode
            STARTUP_MSG: host = localhost.localdomain/127.0.0.1
            STARTUP_MSG: args = [-format]
            STARTUP_MSG: version = 1.2.1
            STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
            STARTUP_MSG: java = 1.7.0_79
            ************************************************************/
            [Fatal Error] core-site.xml:1:21: The processing instruction target matching “[xX][mM][lL]” is not allowed.
            15/06/24 18:32:25 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
            15/06/24 18:32:25 ERROR namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
            at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1249)
            at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107)
            at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
            at org.apache.hadoop.conf.Configuration.set(Configuration.java:420)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.setStartupOption(NameNode.java:1374)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1463)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
            Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
            at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
            at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
            at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
            at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
            … 6 more

            15/06/24 18:32:25 INFO namenode.NameNode: SHUTDOWN_MSG:
            /************************************************************
            SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1

            please help me solve the issue

            thanks in advance

            Reply
            • Rahul on June 26, 2015 9:39 am

              Hi Lini,

              Please check core-site.xml file. It looks this file is some syntax issue.

              Reply
            • Stacy on October 25, 2017 7:42 pm

              delete the line from the top of the core-site.xml file

              Reply
    54. Rakesh on October 9, 2013 10:35 pm

      Hi
      I ran the format step and got below error message

      bin/hadoop namenode -format
      13/10/09 15:33:33 INFO namenode.NameNode: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NameNode
      STARTUP_MSG: host = hadoop-01/10.49.14.42
      STARTUP_MSG: args = [-format]
      STARTUP_MSG: version = 1.2.1
      STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
      STARTUP_MSG: java = 1.6.0_20
      ************************************************************/
      [Fatal Error] core-site.xml:5:2: The markup in the document following the root element must be well-formed.
      13/10/09 15:33:33 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
      13/10/09 15:33:33 ERROR namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1249)
      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107)
      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:420)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.setStartupOption(NameNode.java:1374)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1463)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
      Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
      at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:253)
      at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:288)
      at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
      … 6 more

      13/10/09 15:33:33 INFO namenode.NameNode: SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at hadoop-01/10.49.14.42
      ************************************************************/

      Reply
    55. tosh on October 8, 2013 7:24 pm

      Hi,
      I have successfully installed hadoop cluster, and it seems to run fine. I was able to access the webservices after changing the “localhost” to the host’s ip address. However, when clicking the browse file system in namenode webaccess, Browser does not find the webpage. the url shows as “http://master:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/”
      (I get the same error at a few other places, but manually replacing the master with ip address seems to solve the issue)

      “master” is the hostname I chose for this machine. My guess is this is a DNS issue, but I was wondering if we can change any config files so the links resolve to an ip address instead of master.

      I also see that your mapred-site.xml and core-site.xml have “hdfs://localhost:port_num” in them, but you are able to access the web interface using your domain name (in the screenshots you provided). I am installing hadoop on remote system, and had to replace the localhost with the server’s ip address to access the web interface. Is this accepted practice? Or should I leave them as localhost and make changes elsewhere?

      Third, for hdfs-site.xml, the only property I defined is the replication value ( I was following a separate tutorial until about halfway). So I was wondering if you could explain what leaving the dfs.data.dir and dfs.name.dir would do? If we do define those values, Can it be any directory, or does it have to be inside the hadoop installation?

      And last,
      when I do get to the file browser ( by manually replacing “master” in the url with the ip address), I see a tmp directory there as well that contains the mapred folder and it’s subfolders . Is this normal?

      Sincerely,
      Ashutosh

      Reply
      • tosh on October 8, 2013 7:32 pm

        Clarification:
        By “I was able to access the web services after changing the “localhost” to the host’s ip address”, I meant changed the “localhost” in the hadoop/conf/*.xml files to a static ip address that I am now using to access the web services.

        Reply
    56. Rodrigo Bittencourt on October 7, 2013 3:34 pm

      Hello,

      Rahul I have a question, why the step 7 does not work in my cluster ?

      Reply
      • Rahul on October 8, 2013 2:05 am

        Hi Rodrigo,

        What issue are you getting ?

        Reply
        • Rodrigo Bittencourt on October 8, 2013 11:40 am

          I can not open the URL of hadoop in my cluster.
          where was configured the URL ?

          Reply
          • tosh on October 9, 2013 11:08 pm

            A lot of factors play into the url’s not showing the result. If you are on a local environment ( working on the same machine that you are installing the cluster on) then this tutorial should have worked.

            To the best of my knowledge, the “localhost” in
            hdfs://localhost:9000/ and
            localhost:9001

            defined inside the core-site.xml and mapred-site.xml should be changed to the hostname of your system.
            i.e. if the hostname of your system is “linuxuser”, then change the “localhost” to “linuxuser” in these line, then add the following line in /etc/hosts file:
            127.0.0.1 linuxuser

            If you are working on a remote server, then same as above except the host file should have the static ip instead of 127.0.0.1 , and you will be able to see the web services, but there is another problem I ran into. See my question below .

            It could be the firewall setup as well. try disabling iptables to check if it is firewall issue. If you do this, make sure you re start iptables when you are done

            Reply
        • jinu prakash on June 29, 2016 8:21 am

          hi when i run bin/start-all.sh i get a message saying
          fatal error mapred-site.xml:7:1:XML document structures must start and end within same entity
          and also the same message when i run
          $ bin/hadoop jar hadoop-examples-*.jar grep input output ‘dfs[a-z.]+’

          I did this in ubuntu vbox

          Reply
    57. girish on September 30, 2013 11:25 am

      [[email protected] hadoop]$ bin/hadoop namenode -format

      13/09/30 04:53:34 INFO namenode.NameNode: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NameNode
      STARTUP_MSG: host = java.net.UnknownHostException: girishniyer: girishniyer
      STARTUP_MSG: args = [-format]
      STARTUP_MSG: version = 1.2.1
      STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
      STARTUP_MSG: java = 1.7.0_40
      ************************************************************/
      13/09/30 04:53:34 INFO util.GSet: Computing capacity for map BlocksMap
      13/09/30 04:53:34 INFO util.GSet: VM type = 64-bit
      13/09/30 04:53:34 INFO util.GSet: 2.0% max memory = 932184064
      13/09/30 04:53:34 INFO util.GSet: capacity = 2^21 = 2097152 entries
      13/09/30 04:53:34 INFO util.GSet: recommended=2097152, actual=2097152
      13/09/30 04:53:34 INFO namenode.FSNamesystem: fsOwner=hadoop
      13/09/30 04:53:34 INFO namenode.FSNamesystem: supergroup=supergroup
      13/09/30 04:53:34 INFO namenode.FSNamesystem: isPermissionEnabled=true
      13/09/30 04:53:34 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
      13/09/30 04:53:34 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
      13/09/30 04:53:34 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
      13/09/30 04:53:34 INFO namenode.NameNode: Caching file names occuring more than 10 times
      13/09/30 04:53:34 ERROR namenode.NameNode: java.io.IOException: Cannot create directory /opt/hadoop/hadoop/dfs/name/current
      at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:294)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1337)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1356)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1261)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1467)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)

      13/09/30 04:53:34 INFO namenode.NameNode: SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: girishniyer: girishniyer
      ************************************************************/
      [[email protected] hadoop]$

      i configured java as per ur article.
      then till formatting name node no issues
      bt here iam stuck again.
      could u please help me?

      Reply
      • Rahul on September 30, 2013 11:48 am

        Hi Girish,

        You systems hostname is “girishniyer” which is not resolving to any ip. Do a entry in /etc/hosts file like below

        <system_ip> girishniyer

        Reply
        • Girish S on October 7, 2013 8:00 pm

          I just began learning hadoop. Your article was superbly simple and straight forward. The patience with which you helped Girish Iyer is commendable.

          Reply
          • Rahul on October 10, 2013 3:43 am

            Thanks Girish,

            Reply
    58. kedar on September 27, 2013 5:31 am

      Excellent article …. I could configure hadoop in 20 mins … do you have any more learning article for begineers like first wordcount program and all

      Reply
    59. girish iyer on September 22, 2013 12:50 pm

      when i type the following,i got the error as follows
      [[email protected] hadoop]$ bin/start-all.sh
      also i cant able to format the name node.it says no such a directory…please help.
      is it due to giving the java path ?

      starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-girishniyer.out
      /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
      localhost: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-girishniyer.out
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
      localhost: starting secondarynamenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-girishniyer.out
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
      starting jobtracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-girishniyer.out
      /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
      localhost: starting tasktracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-girishniyer.out
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
      localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory

      Reply
      • Rahul on September 23, 2013 2:48 am

        Hi Girish,

        Did you configured JAVA_HOME correctly… ?
        Check this article and make sure java is properly configured as in step 5.4 Edit hadoop-env.sh
        http://tecadmin.net/steps-to-install-java-on-centos-5-6-or-rhel-5-6/

        Reply
        • girish iyer on September 25, 2013 7:41 am

          Thank you rahul for ur quick reply.
          i configured java and export the java_home path as per ur blog

          now when i give “bin/hadoop namenode -format”
          the following is coming

          bin/hadoop: line 350: /opt/jdk1.7.0_25/bin/java: No such file or directory
          bin/hadoop: line 434: /opt/jdk1.7.0_25/bin/java: No such file or directory
          bin/hadoop: line 434: exec: /opt/jdk1.7.0_25/bin/java:
          cannot execute: No such file or directory
          when i give which java command,my path displayed is as follows.
          “/usr/bin/java”

          when i typed java version,its as follows
          [[email protected] hadoop]$ java -version
          java version “1.7.0_25”
          OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64)
          OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

          i am little confused hence.

          please help me…waiting for ur reply
          thanks in advance

          Reply
          • Rahul on September 26, 2013 10:58 am

            Hi Girish,

            Step #5.4 still not configured properly. How did you installed Java on your system. Source or RPM ?

            Reply
            • Girish on September 29, 2013 5:51 am

              hi rahul,
              java is there in centos when i installed the cent os
              i reinstalled the centos6.4 again
              nw the java version is as follows

              ”
              [[email protected] Desktop]$ which java
              /usr/bin/java

              [[email protected] Desktop]$ java -version
              java version “1.7.0_09-icedtea”
              OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
              OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
              ”
              did i need to re install the java after removing this.
              if so can u please help me

              Reply
              • Rahul on September 30, 2013 3:29 am

                Hi Girish,

                OpenJDK default installation path should be /usr/lib/jvm/java-<java version>-openjdk-<java version>.x86_64/. Please check, if this path exists add it in configuration

                else do a fresh java install http://tecadmin.net/steps-to-install-java-on-centos-5-6-or-rhel-5-6/

          • Danish on September 27, 2013 8:44 am

            Hello,
            I had the same error, it was because my jave home was /opt/jdk1.7.0.25/ . There was no underscore in the jdk1.7.0_25. May be this helps you!

            Reply
    60. Baahu on September 21, 2013 1:20 pm

      Good article !!

      Reply
    61. Girish N Iyer on September 20, 2013 7:51 am

      This article is really helpful and grear work…..Thank You.
      I step 4,when i type,¨ tar -xzf hadoop-1.2.0.tar.gz
      The following error is displaying..could u pleaSe help
      tar (child): hadoop-1.2.0.tar.gz: Cannot open: No such file or directory
      tar (child): Error is not recoverable: exiting now
      tar: Child returned status 2
      tar: Error is not recoverable: exiting now

      Reply
      • Rahul on September 20, 2013 9:26 am

        Hi Girish,

        Now hadoop 1.2.1 latest stable version is available. So downloaded archive name is changed.
        We have updated article step 4. Kindly try now.

        Reply
    62. Raja on September 19, 2013 1:07 pm

      This is really great article for entry level person who like to see how hadoop looks like and play around. great job!
      Also it would be great if you add how to stop hadoop at the end of your article.
      Thank you once again.

      Reply
      • Rahul on September 20, 2013 9:33 am

        Hi Raja,

        Thanks for appreciation.
        We have added last step ( Step 8 ) to how to stop all hadoop services.

        Reply
    63. arun on July 20, 2013 3:19 pm

      How to add a new not to this cluster… ?

      Reply
    64. maillot de foot 2013-2014 on June 29, 2013 4:08 pm

      whoah this blog is fantastic i like reading your articles. Keep up the good paintings! You understand, a lot of people are hunting round for this info, you could aid them greatly.

      Reply

    Leave A Reply Cancel Reply

    Advertisement
    Recent Posts
    • How to List Manually Installed Packages in Ubuntu & Debian
    • 10 Bash Tricks Every Developer Should Know
    • How to Validate Email Address in JavaScript
    • Firewalld: Common Firewall Rules and Commands
    • 12 Apk Commands in Alpine Linux Package Management
    Facebook Twitter Instagram Pinterest
    © 2023 Tecadmin.net. All Rights Reserved | Terms  | Privacy Policy

    Type above and press Enter to search. Press Esc to cancel.