How to Install and Configure Apache Hadoop on CentOS & Fedora

By 5 Mins Read

Having been around for some time now, Hadoop has become one of the most popular open-source big data solutions. It processes data in batches and is famous for its scalable, cost-effective, and distributed computing capabilities. It’s one of the most popular open-source frameworks in the data analysis and storage space. As a user, you can use it to manage your data, analyze that data, and store it again – all in an automated way. With Hadoop installed on your Fedora system, you can access important analytic services with ease.

Advertisement

This article covers how to install Apache Hadoop on CentOS and Fedora systems. In this article, we’ll show you how to install Apache Hadoop on Fedora for local usage as well as a production server.

1. Prerequsities

Java is the primary requirement for running Hadoop on any system, So make sure you have Java installed on your system using the following command. If you don’t have Java installed on your system, use one of the following links to install it first.

2. Create Hadoop User

We recommend creating a normal (nor root) account for Hadoop working. To create an account using the following command.

adduser hadoop
passwd hadoop

After creating the account, it also required to set up key-based ssh to its own account. To do this use execute following commands.

su - hadoop
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

Let’s verify key based login. Below command should not ask for the password but the first time it will prompt for adding RSA to the list of known hosts.

ssh localhost
exit

3. Download Hadoop 3.1 Archive

In this step, download hadoop 3.1 source archive file using below command. You can also select alternate download mirror for increasing download speed.

cd ~
wget http://www-eu.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz
tar xzf hadoop-3.1.0.tar.gz
mv hadoop-3.1.0 hadoop

4. Setup Hadoop Pseudo-Distributed Mode

4.1. Setup Hadoop Environment Variables

First, we need to set environment variable uses by Hadoop. Edit ~/.bashrc file and append following values at end of file.

export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Now apply the changes in the current running environment

source ~/.bashrc

Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable. Change the JAVA path as per install on your system. This path may vary as per your operating system version and installation source. So make sure you are using correct path.

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

4.2. Setup Hadoop Configuration Files

Hadoop has many of configuration files, which need to configure as per requirements of your Hadoop infrastructure. Let’s start with the configuration with basic Hadoop single node cluster setup. first, navigate to below location

cd $HADOOP_HOME/etc/hadoop

Edit core-site.xml

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

Edit hdfs-site.xml

<configuration>
<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>

<property>
  <name>dfs.name.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
  <name>dfs.data.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

Edit mapred-site.xml

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
</configuration>

Edit yarn-site.xml

<configuration>
 <property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
 </property>
</configuration>

4.3. Format Namenode

Now format the namenode using the following command, make sure that Storage directory is

hdfs namenode -format

Sample output:

WARNING: /home/hadoop/hadoop/logs does not exist. Creating.
2018-05-02 17:52:09,678 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = tecadmin/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.1.0
...
...
...
2018-05-02 17:52:13,717 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
2018-05-02 17:52:13,806 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2018-05-02 17:52:14,161 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 391 bytes saved in 0 seconds .
2018-05-02 17:52:14,224 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2018-05-02 17:52:14,282 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at tecadmin/127.0.1.1
************************************************************/

5. Start Hadoop Cluster

Let’s start your Hadoop cluster using the scripts provides by Hadoop. Just navigate to your $HADOOP_HOME/sbin directory and execute scripts one by one.

cd $HADOOP_HOME/sbin/

Now run start-dfs.sh script.

./start-dfs.sh

Sample output:

Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [tecadmin]
2018-05-02 18:00:32,565 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Now run start-yarn.sh script.

./start-yarn.sh

Sample output:

Starting resourcemanager
Starting nodemanagers

6. Access Hadoop Services in Browser

Hadoop NameNode started on port 9870 default. Access your server on port 9870 in your favorite web browser.

http://svr1.tecadmin.net:9870/

Now access port 8042 for getting the information about the cluster and all applications

http://svr1.tecadmin.net:8042/

Access port 9864 to get details about your Hadoop node.

http://svr1.tecadmin.net:9864/

7. Test Hadoop Single Node Setup

7.1. Make the HDFS directories required using following commands.

bin/hdfs dfs -mkdir /user
bin/hdfs dfs -mkdir /user/hadoop

7.2. Copy all files from local file system /var/log/httpd to hadoop distributed file system using below command

bin/hdfs dfs -put /var/log/apache2 logs

7.3. Browse Hadoop distributed file system by opening below URL in the browser. You will see an apache2 folder in the list. Click on the folder name to open and you will find all log files there.

 http://svr1.tecadmin.net:9870/explorer.html#/user/hadoop/logs/

7.4 – Now copy logs directory for hadoop distributed file system to local file system.

bin/hdfs dfs -get logs /tmp/logs
ls -l /tmp/logs/

You can also check this tutorial to run wordcount mapreduce job example using command line.

Share.

Related Posts

103 Comments

  3. Waleed Ali on

    HI, I follow your tutorial and its very nice and helpful for me but I am stuck at its accessing in browser.
    I am done till Start Hadoop Cluster.
    Starting namenodes on [localhost]
    Starting datanodes
    Starting secondary namenodes [hadoop-server]
    2018-12-24 11:32:40,637 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    Starting resourcemanager
    Starting nodemanagers

    As you can see its starting but still couldn’t access in browser.
    I’ll very thankful if someone help me in this.
    Thanks

    Reply
  4. James Jithin on

    Thanks. This post is helpful in the datanode and namenode configuration which is missing in the guide of hadoop.

    Reply
  5. Surya on

    Hi Rahul

    I have received an error
    WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
    Could you please help out me

    Thanks,
    Surya

    Reply
  6. pk_park on

    Hello teacher ~!
    How can I deal this under the error ?

    WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
    ERROR: Invalid HADOOP_YARN_HOME

    Reply
  7. Anisha on

    bin/hdfs dfs -mkdir /user
    When I run this, it says
    -su: bin/hdfs: No such file or directory
    Please help

    Reply
  8. David on

    For the steps showing here, does it work on Red Hat Linux 7.5?
    I have tried. But not able to make it work. Here is the error:

    [hadoop@ol75 hadoop]$ hdfs namenode -format
    2018-05-25 23:59:16,727 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = java.net.UnknownHostException: unallocated.barefruit.co.uk: unallocated.barefruit.co.uk: Name or service not known
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 3.1.0
    STARTUP_MSG: classpath = ……

    Thank you in advance.

    David

    Reply
    • Rahul K. on

      It looks your system hostname is “unallocated.barefruit.co.uk”, which is not resolving. Edit /etc/hosts file bind hostname with system ip.

      127.0.0.1 unallocated.barefruit.co.uk

      Reply
  9. OmarK on

    PROBLEM WITH 7.2 –
    I finished most of this tutorial but am now stuck on finding var/log/apache2
    – after running a few commands to search my system for apache2, I don’t get back any file or directory called apache2. Does anyone have any insight on this?

    Reply
    • Rahul K. on

      Hi Hema, Thanks for pointing out.

      This tutorial has been updated to the latest version of Hadoop installation.

      Reply
  11. DINAKAR N K on

    Dear Mr Rahul,
    Thank for your extended support on hadoop. Here i have small doubt “how to share single node hadoop cluster ” for a number of 60 clinets. Is there any possibility to do so. Please help

    Regards
    Dinakar N K

    Reply
  15. Shankar on

    Hi Rahul,

    This is just awesome!!! I have had some issues during installation but your guide to installing a hadoop cluster is just great !!
    I still used the vi editor for editing the .bashrc file and all other xml files. Just the lack of linux editor knowledge I did it. I used the 127.0.0.1 instead of localhost as the alias names are not getting resolved.

    Moreover there are some issues like the one below
    16/02/22 03:05:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    [hadoop@localhost hadoop]$ sudo apt-get install openssh-client
    sudo: apt-get: command not found

    The above are the two issues that I’m facing but other than that… it has been a “Walk in the Park” with this tutorial

    Thanks Buddy!!

    Shankar

    Reply
  16. Sergey on

    Do you have some idea ?

    16/01/26 09:17:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
    Starting namenodes on [localhost]
    localhost: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
    localhost: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
    Starting secondary namenodes [0.0.0.0]
    0.0.0.0: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
    16/01/26 09:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    Reply
    • Rahul on

      Hi Sergey,

      It looks you don’t have openssh-client package installed. Please install it.

      $ sudo apt-get install openssh-client

      Reply
      • Jhunong on

        hi rahul i followed your setup here but unfortunately i ended up some errors like the one below.

        [hadoop@localhost hadoop-2.8.0]$ bin/hdfs namenode -format
        Error: Could not find or load main class org.apache.hadoop.hdfs.server.namenode.NameNode

        appreciate some help…thanks

        Reply
  17. sasha on

    Hi rahul,
    good job,
    Now expend a bit more and next time show integration of spark and flume.
    If you need help ping me and I will provide you with my additions to your config …

    Regards,

    Reply
    • Dendrok on

      This can be misleading. What it is saying is from the bin folder within the hadoop system folder.

      $ cd $HADOOP_HOME/bin
      $ hdfs dfs -mkdir /user
      $ hdfs dfs -mkdir/user/hadoop

      Reply
  19. HalfDone on

    bash: /home/hadoop/.bashrc: Permission denied

    I guess that the CentOS 7/Hadoop 2.7.1 tutorial was very helpful until step 4, when by some reason the instructions just explained what to do with .bashrc without explaining how to get to it and how to edit it in first place. Thanks anyway, I just need to find a tutorial that explains with detail how to set up Hadoop.

    Reply
  20. ayoola ajiboye on

    Thanks. I read your article have successfully deployed my first single node hadoop deployment despite series of unsuccessful attempts in the past. thanks

    Reply
  21. hoangthang on

    Hi Rahul,
    Please help me! I installed follow your guide. when i run jps that result below:
    18118 Jps
    18068 TaskTracker
    17948 JobTracker
    17861 SecondaryNameNode
    17746 DataNode

    however when I run stop-all.sh command that
    no jobtracker to stop
    localhost: no tasktracker to stop
    no namenode to stop
    localhost: no datanode to stop
    localhost: no secondarynamenode to stop
    Can you explan for me? Thanks so much!

    Reply
  23. Rahul on

    HI Rahul,

    I am not able to start the hadoop services..getting an error like>

    14/07/24 23:11:27 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
    ************************************************************/
    hadoop@localhost hadoop]$ bin/start-all.sh
    bash: bin/start-all.sh: No such file or directory

    Reply
  24. Vijay on

    Hi Rahul,
    When I execute the command $ bin/start-all.sh, I am getting the FATAL ERROR as shown below
    –====
    [hadoop@localhost hadoop]$ sbin/start-all.sh
    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
    [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
    14/06/10 01:16:14 FATAL conf.Configuration: error parsing conf core-site.xml
    .
    .
    .
    .
    tarting namenodes on []
    localhost: starting namenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-namenode-localhost.localdomain.out
    localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
    localhost: starting datanode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-datanode-localhost.localdomain.out
    localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
    [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
    14/06/10 01:16:25 FATAL conf.Configuration: error parsing conf core-site.xml
    org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/etc/hadoop/core-site.xml; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.

    .
    .

    .
    starting yarn daemons
    starting resourcemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out
    [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
    localhost: starting nodemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-nodemanager-localhost.localdomain.out
    localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
    –===
    where core-site.xml edited like this
    $vim core-site.xml

    fs.default.name
    hdfs://localhost:9000/

    dfs.permissions
    false

    #########################################

    please help me to overcome this error.

    Reply
  26. Kullayappa on

    Hi,

    Your artcile is simply super, I followed each and every step and installed hadoop.

    my suggestion is: while checking status with “jps”, it was not showing information and I got no such command. I used again export JAVA_HOME stuff. after that JPS working.

    I guess.. these export JAVA_HOME also need to include in some where else.

    Thansk alot for your article buddy

    Reply
  27. Thang Nguyen on

    Hi Rahul,
    Can you please post some direction to test with the input/output data after installation?
    for example: how can we upload a file to run a simple wordcount and get the output… ?
    Regards,
    Thang Nguyen

    Reply
  29. Elavarasan on

    hi rahul,

    Plz help me for following case
    [FATAL ERROR] core-site.xml:10:2: The markup in the document following the root element must be well-formed.

    error while running “bin/start-all.sh” on all *.xml files

    Reply
  30. Elavarasan on

    localhost: [fatal Error] core-site.xml:10:2: The markup in the document following the root element must be well-formed.

    error while running “bin/start-all.sh”

    Reply
  31. Elavarasan on

    Hi Rahul,

    jps 0r $JAVA_HOME/bin/jps giving error bash:jps: command not found and bash: /bin/jps: no such file or directory….. Kindly address the error…

    Reply
    • Rahul on

      HI Elavarasa,

      Please make sure you have setup JAVA_HOME environment variable. Please provide output of following commands.

      # env | grep JAVA_HOME
      # echo $JAVA_HOME

      Reply
  32. sandeep on

    Hi Rahul,
    I have successfully done the installation of single node cluster and able to see all daemons running.
    But i am not able to run hadoop fs commands , for this should i install any thing else like jars??

    Thanks in advance…

    Reply
  34. Raj on

    Hi Rahul,

    Do you know what is the best site to download Pig and hive? I realized that I am unable to run and pig and hive. I thought it comes with the package just like while setting-up under cloudera.

    Reply
  36. Raj on

    Hi Rahul,
    jps is not showing under $JAVA_HOME/bin

    It comes out with an error no such file or directory

    Also, once I complete my tasks and comes out of linux, do I need to restart the hadoop?

    Reply
  37. raj on

    Hi Rahul,

    Your instructions are superp. The only issue I am facing is a “jps” not getting recognized…. Not sure where the issue could be. Otherwise all set to go.

    Reply
  38. Mursaleen Javed on

    [hadoop@RHEL hadoop]$ bin/hadoop namenode -format
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.

    14/02/03 03:38:23 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = java.net.UnknownHostException: RHEL: RHEL
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 2.2.0
    STARTUP_MSG: classpath = /opt/hadoop/hadoop/etc/hadoop:/opt/hadoop/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.6.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-math-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/hdfs:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/junit-4.10.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.10.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.
    inject-1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar
    STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by ‘hortonmu’ on 2013-10-07T06:28Z
    STARTUP_MSG: java = 1.6.0_20
    ************************************************************/
    14/02/03 03:38:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
    14/02/03 03:38:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
    14/02/03 03:38:25 WARN common.Util: Path /opt/hadoop/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
    14/02/03 03:38:25 WARN common.Util: Path /opt/hadoop/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
    Formatting using clusterid: CID-c4662030-0d86-4263-aed9-043fd34565cd
    14/02/03 03:38:25 INFO namenode.HostFileManager: read includes:
    HostSet(
    )
    14/02/03 03:38:25 INFO namenode.HostFileManager: read excludes:
    HostSet(
    )
    14/02/03 03:38:25 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
    14/02/03 03:38:25 INFO util.GSet: Computing capacity for map BlocksMap
    14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
    14/02/03 03:38:25 INFO util.GSet: 2.0% max memory = 966.7 MB
    14/02/03 03:38:25 INFO util.GSet: capacity = 2^21 = 2097152 entries
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: defaultReplication = 2
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: maxReplication = 512
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: minReplication = 1
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
    14/02/03 03:38:25 INFO blockmanagement.BlockManager: encryptDataTransfer = false
    14/02/03 03:38:25 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
    14/02/03 03:38:25 INFO namenode.FSNamesystem: supergroup = supergroup
    14/02/03 03:38:25 INFO namenode.FSNamesystem: isPermissionEnabled = true
    14/02/03 03:38:25 INFO namenode.FSNamesystem: HA Enabled: false
    14/02/03 03:38:25 INFO namenode.FSNamesystem: Append Enabled: true
    14/02/03 03:38:25 INFO util.GSet: Computing capacity for map INodeMap
    14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
    14/02/03 03:38:25 INFO util.GSet: 1.0% max memory = 966.7 MB
    14/02/03 03:38:25 INFO util.GSet: capacity = 2^20 = 1048576 entries
    14/02/03 03:38:25 INFO namenode.NameNode: Caching file names occuring more than 10 times
    14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
    14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
    14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
    14/02/03 03:38:25 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
    14/02/03 03:38:25 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
    14/02/03 03:38:25 INFO util.GSet: Computing capacity for map Namenode Retry Cache
    14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
    14/02/03 03:38:25 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
    14/02/03 03:38:25 INFO util.GSet: capacity = 2^15 = 32768 entries
    Re-format filesystem in Storage Directory /opt/hadoop/hadoop/dfs/name ? (Y or N) y
    14/02/03 03:38:29 WARN net.DNS: Unable to determine local hostname -falling back to “localhost”
    java.net.UnknownHostException: RHEL: RHEL
    at java.net.InetAddress.getLocalHost(InetAddress.java:1426)
    at org.apache.hadoop.net.DNS.resolveLocalHostname(DNS.java:264)
    at org.apache.hadoop.net.DNS.(DNS.java:57)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
    14/02/03 03:38:29 WARN net.DNS: Unable to determine address of the host-falling back to “localhost” address
    java.net.UnknownHostException: RHEL: RHEL
    at java.net.InetAddress.getLocalHost(InetAddress.java:1426)
    at org.apache.hadoop.net.DNS.resolveLocalHostIPAddress(DNS.java:287)
    at org.apache.hadoop.net.DNS.(DNS.java:58)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
    14/02/03 03:38:29 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted.
    14/02/03 03:38:29 INFO namenode.FSImage: Saving image file /opt/hadoop/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
    14/02/03 03:38:29 INFO namenode.FSImage: Image file /opt/hadoop/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 198 bytes saved in 0 seconds.
    14/02/03 03:38:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    14/02/03 03:38:30 INFO util.ExitUtil: Exiting with status 0
    14/02/03 03:38:30 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: RHEL: RHEL
    ************************************************************/
    [hadoop@RHEL hadoop]$

    hosts file:

    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
    RHEL

    please help

    Reply
  39. Paul on

    Hi Rahul!
    Just to say your instructions worked like a dream.
    In my hadoop-env.sh I used.
    export JAVA_HOME=/usr/lib/jvm/jre-1.6.0 <- might help others its a vanilla Centos 6.5 install.
    Cheers Paul

    Reply
  40. Sravan on

    This explanation is not less to any other I have seen so far… Great efforts Rahul, I ve personally installed psuedo and normal distributed clusters in my workplace using this tutorials.

    Thanks alot buddy 🙂

    Keep doing such good works 🙂

    Reply
      • jhon on

        hi sir rahul

        i follow all your guide step by steps but encounter some problems while executing
        this command:

        ./start-dfs.sh

        output:

        [root@localhost sbin]# ./start-dfs.sh
        17/04/05 22:14:32 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:32 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
        17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not

        thanks

        Reply
  41. Azim on

    I am getting following below error when I run “bin/start-all.sh” Please suggest
    hadoop@localhost hadoop]$ bin/start-all.sh
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.localdomain.out
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: datanode running as process 3602. Stop it first.
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: secondarynamenode running as process 3715. Stop it first.
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    jobtracker running as process 3793. Stop it first.
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
    localhost: tasktracker running as process 3921. Stop it first.
    [hadoop@localhost hadoop]$ jvs
    -bash: jvs: command not found

    Reply
  42. jk on

    Fixed these errors. Checked the logs and got the help from websites.

    Have a question. How can I use/test the wordcount on multiple files in a folder.
    bin/hadoop dfs -copyFromLocal files /testfiles
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /files /testfiles — This throws error saying testfiles is not a file. It should work for a directory too right ?

    Thanks

    Reply
  43. jk on

    oops. Spoke too soon. I do see the JobHistoryServer start – then it stopped it looks like.

    -bash-3.2$ jps
    21984 NameNode
    27080 DataNode
    1638 ResourceManager
    1929 NodeManager
    5718 JobHistoryServer
    6278 Jps

    -bash-3.2$ jps
    21984 NameNode
    27080 DataNode
    11037 Jps
    1638 ResourceManager
    1929 NodeManager

    Where can I see the information as to why this stopped? Can you please suggest. Sorry for multiple posts. But I had to update and dont seem to find any help googling.
    Thanks

    Reply
  44. jk on

    Hi
    I tried changing the entry in core-site.xml
    hdfs://drwdt001.myers.com:9000
    instead of
    hdfs://drwdt001:9000
    and that helped with the startup of the HistoryServer. Now I can see it.
    Although, now I am not sure if my datanode will restart. I had issues with that startup & thats the reason I removed the domain and just had the servername as the value.

    Any ideas? I will test before I shut and restart
    Thanks.

    Reply
  45. jk on

    Hi
    I downloaded and installed Apache hadoop 2.2 latest. Followed the above setup for single node ( First time setup.) RHEL 5.5
    Name node, DataNode, ResourceManager, NodeManager started fine. Had some issue with datanode & had to update the IPtables for opening ports.
    when I run
    -bash-3.2$ sbin/mr-jobhistory-daemon.sh start historyserver
    starting historyserver, logging to /hadoop/hadoop-2.2.0/logs/mapred-hduser-historyserver-server.out
    when I run jps, I dont see the JobHistoryServer listed . There are no errors in the out file above.

    Can someone please assist?
    Thanks
    JKH

    Reply
  46. Rodrigo Bittencourt on

    Hello Rahul,
    I need your help , my cluster does not work because it must have something wrong in the configuration .

    because when I put start-all.sh it does not initialize the package secondarynamenode .

    shows this error

    starting secondarynamenode , logging to / opt/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-lbad012.out

    lbad012 : Exception in thread “main ” java.lang.IllegalArgumentException : Does not contain a valid host : port authority : file :/ / /

    lbad012 : at org.apache.hadoop.net.NetUtils.createSocketAddr ( NetUtils.java : 164 )

    lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress ( NameNode.java : 212 )

    lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress ( NameNode.java : 244 )

    lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress ( NameNode.java : 236 )

    lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize ( SecondaryNameNode.java : 194 )

    lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode . ( SecondaryNameNode.java : 150 )

    lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main ( SecondaryNameNode.java : 676 )

    Reply
  47. Dheeraj on

    heyy
    please help.
    wen i am doing the step 6 its asking ffor some password even though i havent set any password.please tell wat to do.

    bin/start-all.sh
    starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-demo.example.com.out
    hadoop@localhost’s password:

    Thanx

    Reply
    • Rahul on

      Hi Dheeraj,

      I think you don’t have completed Step #3 properly. Please check and complete it. Make sure you are able to ssh without password.

      Reply
  48. Rakesh on

    I have configured cloudera on a single node successfully how ever when i lanuch Hue @//myaddress.8888 i see below error message at configuration level.
    ”
    Configuration files located in /var/run/cloudera-scm-agent/process/57-hue-HUE_SERVER

    Potential misconfiguration detected. Fix and restart Hue.”

    and my hue_safety_valve.ini looks as below

    [hadoop]
    [[mapred_clusters]]
    [[[default]]]
    jobtracker_host=servername
    thrift_port=9290
    jobtracker_port=8021
    submit_to=True
    hadoop_mapred_home={{HADOOP_MR1_HOME}}
    hadoop_bin={{HADOOP_BIN}}
    hadoop_conf_dir={{HADOOP_CONF_DIR}}
    security_enabled=false

    Reply
  49. Shannon on

    Can you please suggest the steps to uninstall the apache hadoop . I am planning to test cloudera as well , if you can share steps for cloudera as well that would be awesome ! .

    Reply
  50. Toan Nguyen on

    Please help me, when I run $ bin/start-all.sh, output have a error,
    starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-hadoop-master.out
    The authenticity of host ‘localhost (::1)’ can’t be established.
    RSA key fingerprint is 7b:6d:cb:fc:48:7b:c6:42:a5:6a:64:83:ab:a8:95:95.
    Are you sure you want to continue connecting (yes/no)? hadoop-slave-2: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave-2.out
    hadoop-slave-1: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave-1.out

    Reply
    • Rahul on

      Hi Rakesh,
      TecAdmin.net commnets does not support HTML tags properly. Can you please show the screenshot of your file..

      Reply
      • Rakesh on

        Hi Rahul,
        sorry for my delay response.
        I was able to resolve the above error ,By mistake i deleted the wrapper in the xml which caused the error , I have now kept the right data in xml and found below output after format.

        [hadoop@hadoop-01 hadoop]$ /opt/jdk1.7.0_40/bin/jps
        2148 NameNode
        2628 TaskTracker
        2767 Jps
        2503 JobTracker
        2415 SecondaryNameNode
        2274 DataNode
        However in the xml files i have used localhost instead of my server ipaddress is that fine , because i am unable to launch the web urls for the same

        Reply
        • Sudip on

          Hi Rahul,

          I am trying to install a 3 node Hadoop Cluster and using v2.1.0-beta version. Though I could sense there are a lot of changes in terems of directory structure (as compared to v1.2.1) but finally after configuring most of the steps mentioned here , when I issued the command “$JAVA_HOME/bin/jps” on my master server I am seeing O/P:

          16467 Resource Manager
          15966 NameNode
          16960 Jps
          16255 SecondaryNameNode

          But I cant’t see my DataNode or NodeManager getting started.

          I can send you the ouptut of “start-all.sh ” script output (which throws a lot of error lines but somehow able to start the above mentioned services), if you think that can be of any help to you.

          Secondly I am not able open any pages as mentioned part of web services; e.g.
          http://:50030/ for the Jobtracker -> Resource Manager
          http://:50070/ for the Namenode

          Would appreciate any comment from your side on my queries here.

          Reply
          • Rahul on

            Hi Sudeep,

            Yes, Please post the output of start-all.sh command with log files. But first plz empty your log files and them run start-all.sh after that post all outputs.

            Also I prefer, if you post your question on our new forum, so it will be better to communicate.

            Reply
          • Sudip on

            Here is the output of start-all.sh script.

            [hadoop@hadoop-master sbin]$ ./start-all.sh
            This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
            13/10/20 18:45:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
            Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
            It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
            hadoop-master]
            sed: -e expression #1, char 6: unknown option to `s’
            Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
            64-Bit: ssh: Could not resolve hostname 64-Bit: Temporary failure in name resolution
            HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
            Server: ssh: Could not resolve hostname Server: Temporary failure in name resolution
            VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
            warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
            You: ssh: Could not resolve hostname You: Temporary failure in name resolution
            have: ssh: Could not resolve hostname have: Temporary failure in name resolution
            library: ssh: Could not resolve hostname library: Temporary failure in name resolution
            have: ssh: Could not resolve hostname have: Temporary failure in name resolution
            which: ssh: Could not resolve hostname which: Temporary failure in name resolution
            disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
            will: ssh: Could not resolve hostname will: Temporary failure in name resolution
            stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
            guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
            might: ssh: Could not resolve hostname might: Temporary failure in name resolution
            stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
            the: ssh: Could not resolve hostname the: Temporary failure in name resolution
            loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
            VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
            ‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
            to: ssh: Could not resolve hostname to: Temporary failure in name resolution
            that: ssh: Could not resolve hostname that: Temporary failure in name resolution
            try: ssh: Could not resolve hostname try: Temporary failure in name resolution
            highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
            link: ssh: Could not resolve hostname link: Temporary failure in name resolution
            fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
            you: ssh: Could not resolve hostname you: Temporary failure in name resolution
            guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
            fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
            or: ssh: Could not resolve hostname or: Temporary failure in name resolution
            It’s: ssh: Could not resolve hostname It’s: Temporary failure in name resolution
            recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
            the: ssh: Could not resolve hostname the: Temporary failure in name resolution
            -c: Unknown cipher type ‘cd’
            ‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
            with: ssh: Could not resolve hostname with: Temporary failure in name resolution
            with: ssh: Could not resolve hostname with: Temporary failure in name resolution
            now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
            noexecstack’.: ssh: Could not resolve hostname noexecstack’.: Temporary failure in name resolution
            ‘,: ssh: Could not resolve hostname ‘,: Temporary failure in name resolution
            The: ssh: Could not resolve hostname The: Temporary failure in name resolution
            it: ssh: Could not resolve hostname it: Temporary failure in name resolution
            library: ssh: Could not resolve hostname library: Temporary failure in name resolution
            hadoop-master: starting namenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-namenode-hadoop-master.out
            hadoop-master: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
            hadoop-master: It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
            hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
            hadoop-slave-1: bash: /opt/hadoop/hadoop/sbin/hadoop-daemon.sh: No such file or directory
            hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
            hadoop-slave-2: bash: /opt/hadoop/hadoop/sbin/hadoop-daemon.sh: No such file or directory
            Starting secondary namenodes [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
            It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
            0.0.0.0]
            sed: -e expression #1, char 6: unknown option to `s’
            64-Bit: ssh: Could not resolve hostname 64-Bit: Temporary failure in name resolution
            Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
            Server: ssh: Could not resolve hostname Server: Temporary failure in name resolution
            HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
            VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
            warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
            have: ssh: Could not resolve hostname have: Temporary failure in name resolution
            You: ssh: Could not resolve hostname You: Temporary failure in name resolution
            library: ssh: Could not resolve hostname library: Temporary failure in name resolution
            might: ssh: Could not resolve hostname might: Temporary failure in name resolution
            loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
            try: ssh: Could not resolve hostname try: Temporary failure in name resolution
            guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
            stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
            which: ssh: Could not resolve hostname which: Temporary failure in name resolution
            have: ssh: Could not resolve hostname have: Temporary failure in name resolution
            now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
            stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
            The: ssh: Could not resolve hostname The: Temporary failure in name resolution
            guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
            will: ssh: Could not resolve hostname will: Temporary failure in name resolution
            the: ssh: Could not resolve hostname the: Temporary failure in name resolution
            recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
            disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
            VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
            to: ssh: Could not resolve hostname to: Temporary failure in name resolution
            ‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
            highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
            ‘,: ssh: Could not resolve hostname ‘,: Temporary failure in name resolution
            that: ssh: Could not resolve hostname that: Temporary failure in name resolution
            -c: Unknown cipher type ‘cd’
            ‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
            link: ssh: Could not resolve hostname link: Temporary failure in name resolution
            fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
            or: ssh: Could not resolve hostname or: Temporary failure in name resolution
            library: ssh: Could not resolve hostname library: Temporary failure in name resolution
            it: ssh: Could not resolve hostname it: Temporary failure in name resolution
            you: ssh: Could not resolve hostname you: Temporary failure in name resolution
            It’s: ssh: Could not resolve hostname It’s: Temporary failure in name resolution
            with: ssh: Could not resolve hostname with: Temporary failure in name resolution
            with: ssh: Could not resolve hostname with: Temporary failure in name resolution
            fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
            the: ssh: Could not resolve hostname the: Temporary failure in name resolution
            noexecstack’.: ssh: Could not resolve hostname noexecstack’.: Temporary failure in name resolution
            0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-hadoop-master.out
            0.0.0.0: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
            0.0.0.0: It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
            13/10/20 18:45:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
            starting yarn daemons
            starting resourcemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-hadoop-master.out
            hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
            hadoop-slave-1: bash: /opt/hadoop/hadoop/sbin/yarn-daemon.sh: No such file or directory
            hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
            hadoop-slave-2: bash: /opt/hadoop/hadoop/sbin/yarn-daemon.sh: No such file or directory
            [hadoop@hadoop-master sbin]$ $JAVA_HOME/bin/jps
            15130 Jps
            14697 SecondaryNameNode
            14872 ResourceManager
            14408 NameNode
            [hadoop@hadoop-master sbin]$

            Reply
        • Lini on

          hello..
          i also faced the same issue as mr Rakesh
          while executing the format command

          [hadoop@localhost hadoop]$ bin/hadoop namenode -format
          15/06/24 18:32:24 INFO namenode.NameNode: STARTUP_MSG:
          /************************************************************
          STARTUP_MSG: Starting NameNode
          STARTUP_MSG: host = localhost.localdomain/127.0.0.1
          STARTUP_MSG: args = [-format]
          STARTUP_MSG: version = 1.2.1
          STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
          STARTUP_MSG: java = 1.7.0_79
          ************************************************************/
          [Fatal Error] core-site.xml:1:21: The processing instruction target matching “[xX][mM][lL]” is not allowed.
          15/06/24 18:32:25 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
          15/06/24 18:32:25 ERROR namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1249)
          at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107)
          at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
          at org.apache.hadoop.conf.Configuration.set(Configuration.java:420)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.setStartupOption(NameNode.java:1374)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1463)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
          Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
          at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
          at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
          at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
          … 6 more

          15/06/24 18:32:25 INFO namenode.NameNode: SHUTDOWN_MSG:
          /************************************************************
          SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1

          please help me solve the issue

          thanks in advance

          Reply
  54. Rakesh on

    Hi
    I ran the format step and got below error message

    bin/hadoop namenode -format
    13/10/09 15:33:33 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = hadoop-01/10.49.14.42
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 1.2.1
    STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
    STARTUP_MSG: java = 1.6.0_20
    ************************************************************/
    [Fatal Error] core-site.xml:5:2: The markup in the document following the root element must be well-formed.
    13/10/09 15:33:33 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
    13/10/09 15:33:33 ERROR namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1249)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
    at org.apache.hadoop.conf.Configuration.set(Configuration.java:420)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.setStartupOption(NameNode.java:1374)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1463)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
    Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:253)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:288)
    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
    … 6 more

    13/10/09 15:33:33 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at hadoop-01/10.49.14.42
    ************************************************************/

    Reply
  55. tosh on

    Hi,
    I have successfully installed hadoop cluster, and it seems to run fine. I was able to access the webservices after changing the “localhost” to the host’s ip address. However, when clicking the browse file system in namenode webaccess, Browser does not find the webpage. the url shows as “http://master:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/”
    (I get the same error at a few other places, but manually replacing the master with ip address seems to solve the issue)

    “master” is the hostname I chose for this machine. My guess is this is a DNS issue, but I was wondering if we can change any config files so the links resolve to an ip address instead of master.

    I also see that your mapred-site.xml and core-site.xml have “hdfs://localhost:port_num” in them, but you are able to access the web interface using your domain name (in the screenshots you provided). I am installing hadoop on remote system, and had to replace the localhost with the server’s ip address to access the web interface. Is this accepted practice? Or should I leave them as localhost and make changes elsewhere?

    Third, for hdfs-site.xml, the only property I defined is the replication value ( I was following a separate tutorial until about halfway). So I was wondering if you could explain what leaving the dfs.data.dir and dfs.name.dir would do? If we do define those values, Can it be any directory, or does it have to be inside the hadoop installation?

    And last,
    when I do get to the file browser ( by manually replacing “master” in the url with the ip address), I see a tmp directory there as well that contains the mapred folder and it’s subfolders . Is this normal?

    Sincerely,
    Ashutosh

    Reply
    • tosh on

      Clarification:
      By “I was able to access the web services after changing the “localhost” to the host’s ip address”, I meant changed the “localhost” in the hadoop/conf/*.xml files to a static ip address that I am now using to access the web services.

      Reply
        • tosh on

          A lot of factors play into the url’s not showing the result. If you are on a local environment ( working on the same machine that you are installing the cluster on) then this tutorial should have worked.

          To the best of my knowledge, the “localhost” in
          hdfs://localhost:9000/ and
          localhost:9001

          defined inside the core-site.xml and mapred-site.xml should be changed to the hostname of your system.
          i.e. if the hostname of your system is “linuxuser”, then change the “localhost” to “linuxuser” in these line, then add the following line in /etc/hosts file:
          127.0.0.1 linuxuser

          If you are working on a remote server, then same as above except the host file should have the static ip instead of 127.0.0.1 , and you will be able to see the web services, but there is another problem I ran into. See my question below .

          It could be the firewall setup as well. try disabling iptables to check if it is firewall issue. If you do this, make sure you re start iptables when you are done

          Reply
      • jinu prakash on

        hi when i run bin/start-all.sh i get a message saying
        fatal error mapred-site.xml:7:1:XML document structures must start and end within same entity
        and also the same message when i run
        $ bin/hadoop jar hadoop-examples-*.jar grep input output ‘dfs[a-z.]+’

        I did this in ubuntu vbox

        Reply
  57. girish on

    [hadoop@girishniyer hadoop]$ bin/hadoop namenode -format

    13/09/30 04:53:34 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = java.net.UnknownHostException: girishniyer: girishniyer
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 1.2.1
    STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
    STARTUP_MSG: java = 1.7.0_40
    ************************************************************/
    13/09/30 04:53:34 INFO util.GSet: Computing capacity for map BlocksMap
    13/09/30 04:53:34 INFO util.GSet: VM type = 64-bit
    13/09/30 04:53:34 INFO util.GSet: 2.0% max memory = 932184064
    13/09/30 04:53:34 INFO util.GSet: capacity = 2^21 = 2097152 entries
    13/09/30 04:53:34 INFO util.GSet: recommended=2097152, actual=2097152
    13/09/30 04:53:34 INFO namenode.FSNamesystem: fsOwner=hadoop
    13/09/30 04:53:34 INFO namenode.FSNamesystem: supergroup=supergroup
    13/09/30 04:53:34 INFO namenode.FSNamesystem: isPermissionEnabled=true
    13/09/30 04:53:34 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
    13/09/30 04:53:34 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
    13/09/30 04:53:34 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
    13/09/30 04:53:34 INFO namenode.NameNode: Caching file names occuring more than 10 times
    13/09/30 04:53:34 ERROR namenode.NameNode: java.io.IOException: Cannot create directory /opt/hadoop/hadoop/dfs/name/current
    at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:294)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1337)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1356)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1261)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1467)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)

    13/09/30 04:53:34 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: girishniyer: girishniyer
    ************************************************************/
    [hadoop@girishniyer hadoop]$

    i configured java as per ur article.
    then till formatting name node no issues
    bt here iam stuck again.
    could u please help me?

    Reply
  58. kedar on

    Excellent article …. I could configure hadoop in 20 mins … do you have any more learning article for begineers like first wordcount program and all

    Reply
  59. girish iyer on

    when i type the following,i got the error as follows
    [hadoop@girishniyer hadoop]$ bin/start-all.sh
    also i cant able to format the name node.it says no such a directory…please help.
    is it due to giving the java path ?

    starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-girishniyer.out
    /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
    localhost: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-girishniyer.out
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
    localhost: starting secondarynamenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-girishniyer.out
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
    starting jobtracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-girishniyer.out
    /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
    localhost: starting tasktracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-girishniyer.out
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
    localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory

    Reply
      • girish iyer on

        Thank you rahul for ur quick reply.
        i configured java and export the java_home path as per ur blog

        now when i give “bin/hadoop namenode -format”
        the following is coming

        bin/hadoop: line 350: /opt/jdk1.7.0_25/bin/java: No such file or directory
        bin/hadoop: line 434: /opt/jdk1.7.0_25/bin/java: No such file or directory
        bin/hadoop: line 434: exec: /opt/jdk1.7.0_25/bin/java:
        cannot execute: No such file or directory
        when i give which java command,my path displayed is as follows.
        “/usr/bin/java”

        when i typed java version,its as follows
        [hadoop@girishniyer hadoop]$ java -version
        java version “1.7.0_25”
        OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64)
        OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

        i am little confused hence.

        please help me…waiting for ur reply
        thanks in advance

        Reply
          • Girish on

            hi rahul,
            java is there in centos when i installed the cent os
            i reinstalled the centos6.4 again
            nw the java version is as follows

            ”
            [girishniyer@girishniyer Desktop]$ which java
            /usr/bin/java

            [girishniyer@girishniyer Desktop]$ java -version
            java version “1.7.0_09-icedtea”
            OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
            OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
            ”
            did i need to re install the java after removing this.
            if so can u please help me

            Reply
        • Danish on

          Hello,
          I had the same error, it was because my jave home was /opt/jdk1.7.0.25/ . There was no underscore in the jdk1.7.0_25. May be this helps you!

          Reply
  61. Girish N Iyer on

    This article is really helpful and grear work…..Thank You.
    I step 4,when i type,¨ tar -xzf hadoop-1.2.0.tar.gz
    The following error is displaying..could u pleaSe help
    tar (child): hadoop-1.2.0.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2
    tar: Error is not recoverable: exiting now

    Reply
    • Rahul on

      Hi Girish,

      Now hadoop 1.2.1 latest stable version is available. So downloaded archive name is changed.
      We have updated article step 4. Kindly try now.

      Reply
  62. Raja on

    This is really great article for entry level person who like to see how hadoop looks like and play around. great job!
    Also it would be great if you add how to stop hadoop at the end of your article.
    Thank you once again.

    Reply
  64. maillot de foot 2013-2014 on

    whoah this blog is fantastic i like reading your articles. Keep up the good paintings! You understand, a lot of people are hunting round for this info, you could aid them greatly.

    Reply
Reply To jk Cancel Reply

Exit mobile version