How to Install and Configure Apache Hadoop on CentOS & Fedora

By RahulOctober 28, 20225 Mins Read

Having been around for some time now, Hadoop has become one of the most popular open-source big data solutions. It processes data in batches and is famous for its scalable, cost-effective, and distributed computing capabilities. It’s one of the most popular open-source frameworks in the data analysis and storage space. As a user, you can use it to manage your data, analyze that data, and store it again – all in an automated way. With Hadoop installed on your Fedora system, you can access important analytic services with ease.

This article covers how to install Apache Hadoop on CentOS and Fedora systems. In this article, we’ll show you how to install Apache Hadoop on Fedora for local usage as well as a production server.

1. Prerequsities

Java is the primary requirement for running Hadoop on any system, So make sure you have Java installed on your system using the following command. If you don’t have Java installed on your system, use one of the following links to install it first.

How to Install JAVA 8 on CentOS/RHEL 7/6/5

2. Create Hadoop User

We recommend creating a normal (nor root) account for Hadoop working. To create an account using the following command.

adduser hadoop
passwd hadoop

After creating the account, it also required to set up key-based ssh to its own account. To do this use execute following commands.

su - hadoop
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

Let’s verify key based login. Below command should not ask for the password but the first time it will prompt for adding RSA to the list of known hosts.

ssh localhost
exit

3. Download Hadoop 3.1 Archive

In this step, download hadoop 3.1 source archive file using below command. You can also select alternate download mirror for increasing download speed.

cd ~
wget http://www-eu.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz
tar xzf hadoop-3.1.0.tar.gz
mv hadoop-3.1.0 hadoop

4. Setup Hadoop Pseudo-Distributed Mode

4.1. Setup Hadoop Environment Variables

First, we need to set environment variable uses by Hadoop. Edit ~/.bashrc file and append following values at end of file.

export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Now apply the changes in the current running environment

source ~/.bashrc

Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable. Change the JAVA path as per install on your system. This path may vary as per your operating system version and installation source. So make sure you are using correct path.

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

4.2. Setup Hadoop Configuration Files

Hadoop has many of configuration files, which need to configure as per requirements of your Hadoop infrastructure. Let’s start with the configuration with basic Hadoop single node cluster setup. first, navigate to below location

cd $HADOOP_HOME/etc/hadoop

Edit core-site.xml

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

Edit hdfs-site.xml

<configuration>
<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>

<property>
  <name>dfs.name.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
  <name>dfs.data.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

Edit mapred-site.xml

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
</configuration>

Edit yarn-site.xml

<configuration>
 <property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
 </property>
</configuration>

4.3. Format Namenode

Now format the namenode using the following command, make sure that Storage directory is

hdfs namenode -format

Sample output:

WARNING: /home/hadoop/hadoop/logs does not exist. Creating.
2018-05-02 17:52:09,678 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = tecadmin/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.1.0
...
...
...
2018-05-02 17:52:13,717 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
2018-05-02 17:52:13,806 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2018-05-02 17:52:14,161 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 391 bytes saved in 0 seconds .
2018-05-02 17:52:14,224 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2018-05-02 17:52:14,282 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at tecadmin/127.0.1.1
************************************************************/

5. Start Hadoop Cluster

Let’s start your Hadoop cluster using the scripts provides by Hadoop. Just navigate to your $HADOOP_HOME/sbin directory and execute scripts one by one.

cd $HADOOP_HOME/sbin/

Now run start-dfs.sh script.

./start-dfs.sh

Sample output:

Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [tecadmin]
2018-05-02 18:00:32,565 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Now run start-yarn.sh script.

./start-yarn.sh

Sample output:

Starting resourcemanager
Starting nodemanagers

6. Access Hadoop Services in Browser

Hadoop NameNode started on port 9870 default. Access your server on port 9870 in your favorite web browser.

http://svr1.tecadmin.net:9870/

Now access port 8042 for getting the information about the cluster and all applications

http://svr1.tecadmin.net:8042/

Access port 9864 to get details about your Hadoop node.

http://svr1.tecadmin.net:9864/

7. Test Hadoop Single Node Setup

7.1. Make the HDFS directories required using following commands.

bin/hdfs dfs -mkdir /user
bin/hdfs dfs -mkdir /user/hadoop

7.2. Copy all files from local file system /var/log/httpd to hadoop distributed file system using below command

bin/hdfs dfs -put /var/log/apache2 logs

7.3. Browse Hadoop distributed file system by opening below URL in the browser. You will see an apache2 folder in the list. Click on the folder name to open and you will find all log files there.

 http://svr1.tecadmin.net:9870/explorer.html#/user/hadoop/logs/

7.4 – Now copy logs directory for hadoop distributed file system to local file system.

bin/hdfs dfs -get logs /tmp/logs
ls -l /tmp/logs/

You can also check this tutorial to run wordcount mapreduce job example using command line.

View 103 Comments

103 Comments

Ashwani Mishra on May 9, 2020 10:28 am
Out of so many i tried this one worked .. thanks a ton ..
Reply
vanetoj on September 30, 2019 5:39 pm
i m getting this error : Error: Cannot find configuration directory: /etc/hadoop . any ideas ?
Reply
Waleed Ali on December 24, 2018 11:59 am
HI, I follow your tutorial and its very nice and helpful for me but I am stuck at its accessing in browser.
I am done till Start Hadoop Cluster.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [hadoop-server]
2018-12-24 11:32:40,637 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
As you can see its starting but still couldn’t access in browser.
I’ll very thankful if someone help me in this.
Thanks
Reply
James Jithin on October 17, 2018 7:00 am
Thanks. This post is helpful in the datanode and namenode configuration which is missing in the guide of hadoop.
Reply
Surya on September 17, 2018 11:00 pm
Hi Rahul
I have received an error
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Could you please help out me
Thanks,
Surya
Reply
pk_park on June 15, 2018 2:53 pm
Hello teacher ~!
How can I deal this under the error ?
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
ERROR: Invalid HADOOP_YARN_HOME
Reply
Anisha on May 31, 2018 1:53 pm
bin/hdfs dfs -mkdir /user
When I run this, it says
-su: bin/hdfs: No such file or directory
Please help
Reply
David on May 26, 2018 12:02 am
For the steps showing here, does it work on Red Hat Linux 7.5?
I have tried. But not able to make it work. Here is the error:
[[email protected] hadoop]$ hdfs namenode -format
2018-05-25 23:59:16,727 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = java.net.UnknownHostException: unallocated.barefruit.co.uk: unallocated.barefruit.co.uk: Name or service not known
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.1.0
STARTUP_MSG: classpath = ……
Thank you in advance.
David
Reply
- Rahul K. on May 28, 2018 4:34 am
  It looks your system hostname is “unallocated.barefruit.co.uk”, which is not resolving. Edit /etc/hosts file bind hostname with system ip.
  127.0.0.1 unallocated.barefruit.co.uk
  Reply
OmarK on May 21, 2018 5:53 pm
PROBLEM WITH 7.2 –
I finished most of this tutorial but am now stuck on finding var/log/apache2
– after running a few commands to search my system for apache2, I don’t get back any file or directory called apache2. Does anyone have any insight on this?
Reply
hema on May 2, 2018 11:00 am
Hi Rahul,
This link is not working,
wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz
pls check
Reply
- Rahul K. on May 3, 2018 6:38 am
  Hi Hema, Thanks for pointing out.
  This tutorial has been updated to the latest version of Hadoop installation.
  Reply
DINAKAR N K on October 12, 2017 10:32 am
Dear Mr Rahul,
Thank for your extended support on hadoop. Here i have small doubt “how to share single node hadoop cluster ” for a number of 60 clinets. Is there any possibility to do so. Please help
Regards
Dinakar N K
Reply
santhosh on September 22, 2017 5:40 am
hi…..
Reply
Rinson on May 16, 2016 12:49 pm
Good Article…Thanks..:)
Reply
som on February 29, 2016 12:33 pm
The information given is incomplete need to export more variables.
Reply
Shankar on February 21, 2016 9:56 pm
Hi Rahul,
This is just awesome!!! I have had some issues during installation but your guide to installing a hadoop cluster is just great !!
I still used the vi editor for editing the .bashrc file and all other xml files. Just the lack of linux editor knowledge I did it. I used the 127.0.0.1 instead of localhost as the alias names are not getting resolved.
Moreover there are some issues like the one below
16/02/22 03:05:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[[email protected] hadoop]$ sudo apt-get install openssh-client
sudo: apt-get: command not found
The above are the two issues that I’m facing but other than that… it has been a “Walk in the Park” with this tutorial
Thanks Buddy!!
Shankar
Reply
Sergey on January 26, 2016 2:18 pm
Do you have some idea ?
16/01/26 09:17:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
localhost: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
Starting secondary namenodes [0.0.0.0]
0.0.0.0: /home/hadoop/hadoop/sbin/slaves.sh: line 60: ssh: command not found
16/01/26 09:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Reply
- Rahul on January 27, 2016 5:58 am
  Hi Sergey,
  It looks you don’t have openssh-client package installed. Please install it.
  $ sudo apt-get install openssh-client
  Reply
  - Jhunong on April 13, 2017 2:50 am
    hi rahul i followed your setup here but unfortunately i ended up some errors like the one below.
    [[email protected] hadoop-2.8.0]$ bin/hdfs namenode -format
    Error: Could not find or load main class org.apache.hadoop.hdfs.server.namenode.NameNode
    appreciate some help…thanks
    Reply
sasha on January 24, 2016 12:49 am
Hi rahul,
good job,
Now expend a bit more and next time show integration of spark and flume.
If you need help ping me and I will provide you with my additions to your config …
Regards,
Reply
Adrian on January 5, 2016 2:28 pm
Can anybody can tell me where is bin/hdfs in Step 7 please ?
For me it says “Not a directory”
Reply
- Dendrok on January 23, 2016 10:15 am
  This can be misleading. What it is saying is from the bin folder within the hadoop system folder.
  $ cd $HADOOP_HOME/bin
  $ hdfs dfs -mkdir /user
  $ hdfs dfs -mkdir/user/hadoop
  Reply
HalfDone on December 17, 2015 2:06 am
bash: /home/hadoop/.bashrc: Permission denied
I guess that the CentOS 7/Hadoop 2.7.1 tutorial was very helpful until step 4, when by some reason the instructions just explained what to do with .bashrc without explaining how to get to it and how to edit it in first place. Thanks anyway, I just need to find a tutorial that explains with detail how to set up Hadoop.
Reply
ayoola ajiboye on December 8, 2015 2:23 pm
Thanks. I read your article have successfully deployed my first single node hadoop deployment despite series of unsuccessful attempts in the past. thanks
Reply
hoangthang on September 25, 2014 9:30 am
Hi Rahul,
Please help me! I installed follow your guide. when i run jps that result below:
18118 Jps
18068 TaskTracker
17948 JobTracker
17861 SecondaryNameNode
17746 DataNode
however when I run stop-all.sh command that
no jobtracker to stop
localhost: no tasktracker to stop
no namenode to stop
localhost: no datanode to stop
localhost: no secondarynamenode to stop
Can you explan for me? Thanks so much!
Reply
ak on August 21, 2014 11:39 am
Hai Rahul,
Can u plz tell me how to configure multiple datanodes on a single machine.I am using hadoop2.5
Reply
Rahul on July 25, 2014 6:18 am
HI Rahul,
I am not able to start the hadoop services..getting an error like>
14/07/24 23:11:27 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/
[email protected] hadoop]$ bin/start-all.sh
bash: bin/start-all.sh: No such file or directory
Reply
Vijay on June 10, 2014 8:58 am
Hi Rahul,
When I execute the command $ bin/start-all.sh, I am getting the FATAL ERROR as shown below
–====
[[email protected] hadoop]$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
[Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
14/06/10 01:16:14 FATAL conf.Configuration: error parsing conf core-site.xml
.
.
.
.
tarting namenodes on []
localhost: starting namenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-namenode-localhost.localdomain.out
localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
localhost: starting datanode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-datanode-localhost.localdomain.out
localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
[Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
14/06/10 01:16:25 FATAL conf.Configuration: error parsing conf core-site.xml
org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/etc/hadoop/core-site.xml; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
.
.
.
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out
[Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
localhost: starting nodemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-nodemanager-localhost.localdomain.out
localhost: [Fatal Error] core-site.xml:1:1: Content is not allowed in prolog.
–===
where core-site.xml edited like this
$vim core-site.xml
fs.default.name
hdfs://localhost:9000/
dfs.permissions
false
#########################################
please help me to overcome this error.
Reply
Shreedhar on June 3, 2014 2:44 pm
Thanks a lot.
I made a few modifications but the the instructions are on the money!
If you get
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable use the instructions from below
http://stackoverflow.com/questions/20011252/hadoop-2-2-0-64-bit-installing-but-cannot-start
Reply
Kullayappa on May 28, 2014 6:57 am
Hi,
Your artcile is simply super, I followed each and every step and installed hadoop.
my suggestion is: while checking status with “jps”, it was not showing information and I got no such command. I used again export JAVA_HOME stuff. after that JPS working.
I guess.. these export JAVA_HOME also need to include in some where else.
Thansk alot for your article buddy
Reply
Thang Nguyen on May 26, 2014 4:58 am
Hi Rahul,
Can you please post some direction to test with the input/output data after installation?
for example: how can we upload a file to run a simple wordcount and get the output… ?
Regards,
Thang Nguyen
Reply
Elavarasan on April 29, 2014 7:37 am
fs.default.name
hdfs://localhost:9000/
dfs.permissions
false
Reply
- Rahul on April 29, 2014 8:56 am
  Hi
  I could not find file content properly. Can you email me at [email protected].
  Reply
Elavarasan on April 29, 2014 6:37 am
hi rahul,
Plz help me for following case
[FATAL ERROR] core-site.xml:10:2: The markup in the document following the root element must be well-formed.
error while running “bin/start-all.sh” on all *.xml files
Reply
- Rahul on April 29, 2014 6:59 am
  Hi,
  Please provide your core-site.xml file content
  Reply
Elavarasan on April 29, 2014 6:33 am
localhost: [fatal Error] core-site.xml:10:2: The markup in the document following the root element must be well-formed.
error while running “bin/start-all.sh”
Reply
Elavarasan on April 28, 2014 9:56 am
Hi Rahul,
jps 0r $JAVA_HOME/bin/jps giving error bash:jps: command not found and bash: /bin/jps: no such file or directory….. Kindly address the error…
Reply
- Rahul on April 28, 2014 10:14 am
  HI Elavarasa,
  Please make sure you have setup JAVA_HOME environment variable. Please provide output of following commands.
  # env | grep JAVA_HOME
  # echo $JAVA_HOME
  Reply
sandeep on April 9, 2014 8:14 am
Hi Rahul,
I have successfully done the installation of single node cluster and able to see all daemons running.
But i am not able to run hadoop fs commands , for this should i install any thing else like jars??
Thanks in advance…
Reply
SS on March 12, 2014 8:21 pm
How do we add data to the single node cluster.
Reply
Raj on February 28, 2014 4:01 am
Hi Rahul,
Do you know what is the best site to download Pig and hive? I realized that I am unable to run and pig and hive. I thought it comes with the package just like while setting-up under cloudera.
Reply
Raj on February 28, 2014 1:38 am
Hi Rahul,
I just restarted the hadoop and JPS is working fine.
Reply
Raj on February 28, 2014 1:35 am
Hi Rahul,
jps is not showing under $JAVA_HOME/bin
It comes out with an error no such file or directory
Also, once I complete my tasks and comes out of linux, do I need to restart the hadoop?
Reply
raj on February 27, 2014 10:13 am
Hi Rahul,
Your instructions are superp. The only issue I am facing is a “jps” not getting recognized…. Not sure where the issue could be. Otherwise all set to go.
Reply
- Rahul on February 27, 2014 12:00 pm
  Hi Raj,
  Try following command for jps.
  $JAVA_HOME/bin/jps
  Reply
Mursaleen Javed on February 2, 2014 10:41 pm
[[email protected] hadoop]$ bin/hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
14/02/03 03:38:23 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = java.net.UnknownHostException: RHEL: RHEL
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.2.0
STARTUP_MSG: classpath = /opt/hadoop/hadoop/etc/hadoop:/opt/hadoop/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.6.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-math-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/hdfs:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/junit-4.10.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.10.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.
inject-1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by ‘hortonmu’ on 2013-10-07T06:28Z
STARTUP_MSG: java = 1.6.0_20
************************************************************/
14/02/03 03:38:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
14/02/03 03:38:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
14/02/03 03:38:25 WARN common.Util: Path /opt/hadoop/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
14/02/03 03:38:25 WARN common.Util: Path /opt/hadoop/hadoop/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-c4662030-0d86-4263-aed9-043fd34565cd
14/02/03 03:38:25 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/02/03 03:38:25 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/02/03 03:38:25 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/02/03 03:38:25 INFO util.GSet: Computing capacity for map BlocksMap
14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
14/02/03 03:38:25 INFO util.GSet: 2.0% max memory = 966.7 MB
14/02/03 03:38:25 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/02/03 03:38:25 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/02/03 03:38:25 INFO blockmanagement.BlockManager: defaultReplication = 2
14/02/03 03:38:25 INFO blockmanagement.BlockManager: maxReplication = 512
14/02/03 03:38:25 INFO blockmanagement.BlockManager: minReplication = 1
14/02/03 03:38:25 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
14/02/03 03:38:25 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
14/02/03 03:38:25 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/02/03 03:38:25 INFO blockmanagement.BlockManager: encryptDataTransfer = false
14/02/03 03:38:25 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
14/02/03 03:38:25 INFO namenode.FSNamesystem: supergroup = supergroup
14/02/03 03:38:25 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/02/03 03:38:25 INFO namenode.FSNamesystem: HA Enabled: false
14/02/03 03:38:25 INFO namenode.FSNamesystem: Append Enabled: true
14/02/03 03:38:25 INFO util.GSet: Computing capacity for map INodeMap
14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
14/02/03 03:38:25 INFO util.GSet: 1.0% max memory = 966.7 MB
14/02/03 03:38:25 INFO util.GSet: capacity = 2^20 = 1048576 entries
14/02/03 03:38:25 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/02/03 03:38:25 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
14/02/03 03:38:25 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/02/03 03:38:25 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/02/03 03:38:25 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/02/03 03:38:25 INFO util.GSet: VM type = 64-bit
14/02/03 03:38:25 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
14/02/03 03:38:25 INFO util.GSet: capacity = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /opt/hadoop/hadoop/dfs/name ? (Y or N) y
14/02/03 03:38:29 WARN net.DNS: Unable to determine local hostname -falling back to “localhost”
java.net.UnknownHostException: RHEL: RHEL
at java.net.InetAddress.getLocalHost(InetAddress.java:1426)
at org.apache.hadoop.net.DNS.resolveLocalHostname(DNS.java:264)
at org.apache.hadoop.net.DNS.(DNS.java:57)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
14/02/03 03:38:29 WARN net.DNS: Unable to determine address of the host-falling back to “localhost” address
java.net.UnknownHostException: RHEL: RHEL
at java.net.InetAddress.getLocalHost(InetAddress.java:1426)
at org.apache.hadoop.net.DNS.resolveLocalHostIPAddress(DNS.java:287)
at org.apache.hadoop.net.DNS.(DNS.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
14/02/03 03:38:29 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted.
14/02/03 03:38:29 INFO namenode.FSImage: Saving image file /opt/hadoop/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
14/02/03 03:38:29 INFO namenode.FSImage: Image file /opt/hadoop/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 198 bytes saved in 0 seconds.
14/02/03 03:38:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/02/03 03:38:30 INFO util.ExitUtil: Exiting with status 0
14/02/03 03:38:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: RHEL: RHEL
************************************************************/
[[email protected] hadoop]$
hosts file:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
RHEL
please help
Reply
- Rahul Kumar on February 3, 2014 5:11 am
  Hi Javed,
  Your systems hosts file entry looks incorrect. Please add entry like below
  127.0.0.1 RHEL
  Reply
Paul on January 8, 2014 2:12 pm
Hi Rahul!
Just to say your instructions worked like a dream.
In my hadoop-env.sh I used.
export JAVA_HOME=/usr/lib/jvm/jre-1.6.0 <- might help others its a vanilla Centos 6.5 install.
Cheers Paul
Reply
Sravan on January 8, 2014 8:32 am
This explanation is not less to any other I have seen so far… Great efforts Rahul, I ve personally installed psuedo and normal distributed clusters in my workplace using this tutorials.
Thanks alot buddy 🙂
Keep doing such good works 🙂
Reply
- Rahul Kumar on January 8, 2014 12:41 pm
  Thanks Sravan
  Reply
  - jhon on April 5, 2017 2:19 pm
    hi sir rahul
    i follow all your guide step by steps but encounter some problems while executing
    this command:
    ./start-dfs.sh
    output:
    [[email protected] sbin]# ./start-dfs.sh
    17/04/05 22:14:32 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:32 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    17/04/05 22:14:33 WARN conf.Configuration: bad conf file: element not
    thanks
    Reply
Azim on November 13, 2013 7:00 am
I am getting following below error when I run “bin/start-all.sh” Please suggest
[email protected] hadoop]$ bin/start-all.sh
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.localdomain.out
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: datanode running as process 3602. Stop it first.
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: secondarynamenode running as process 3715. Stop it first.
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
jobtracker running as process 3793. Stop it first.
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
/opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: /opt/hadoop/hadoop/libexec/../conf/hadoop-env.sh: line 1: i#: command not found
localhost: tasktracker running as process 3921. Stop it first.
[[email protected] hadoop]$ jvs
-bash: jvs: command not found
Reply
jk on November 9, 2013 2:40 am
Fixed these errors. Checked the logs and got the help from websites.
Have a question. How can I use/test the wordcount on multiple files in a folder.
bin/hadoop dfs -copyFromLocal files /testfiles
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /files /testfiles — This throws error saying testfiles is not a file. It should work for a directory too right ?
Thanks
Reply
jk on November 9, 2013 12:45 am
oops. Spoke too soon. I do see the JobHistoryServer start – then it stopped it looks like.
-bash-3.2$ jps
21984 NameNode
27080 DataNode
1638 ResourceManager
1929 NodeManager
5718 JobHistoryServer
6278 Jps
-bash-3.2$ jps
21984 NameNode
27080 DataNode
11037 Jps
1638 ResourceManager
1929 NodeManager
Where can I see the information as to why this stopped? Can you please suggest. Sorry for multiple posts. But I had to update and dont seem to find any help googling.
Thanks
Reply
jk on November 9, 2013 12:41 am
Hi
I tried changing the entry in core-site.xml
hdfs://drwdt001.myers.com:9000
instead of
hdfs://drwdt001:9000
and that helped with the startup of the HistoryServer. Now I can see it.
Although, now I am not sure if my datanode will restart. I had issues with that startup & thats the reason I removed the domain and just had the servername as the value.
Any ideas? I will test before I shut and restart
Thanks.
Reply
jk on November 9, 2013 12:34 am
Hi
I downloaded and installed Apache hadoop 2.2 latest. Followed the above setup for single node ( First time setup.) RHEL 5.5
Name node, DataNode, ResourceManager, NodeManager started fine. Had some issue with datanode & had to update the IPtables for opening ports.
when I run
-bash-3.2$ sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /hadoop/hadoop-2.2.0/logs/mapred-hduser-historyserver-server.out
when I run jps, I dont see the JobHistoryServer listed . There are no errors in the out file above.
Can someone please assist?
Thanks
JKH
Reply
Rodrigo Bittencourt on November 4, 2013 11:46 am
Hello Rahul,
I need your help , my cluster does not work because it must have something wrong in the configuration .
because when I put start-all.sh it does not initialize the package secondarynamenode .
shows this error
starting secondarynamenode , logging to / opt/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-lbad012.out
lbad012 : Exception in thread “main ” java.lang.IllegalArgumentException : Does not contain a valid host : port authority : file :/ / /
lbad012 : at org.apache.hadoop.net.NetUtils.createSocketAddr ( NetUtils.java : 164 )
lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress ( NameNode.java : 212 )
lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress ( NameNode.java : 244 )
lbad012 : at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress ( NameNode.java : 236 )
lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize ( SecondaryNameNode.java : 194 )
lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode . ( SecondaryNameNode.java : 150 )
lbad012 : at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main ( SecondaryNameNode.java : 676 )
Reply
Dheeraj on October 30, 2013 12:38 pm
heyy
please help.
wen i am doing the step 6 its asking ffor some password even though i havent set any password.please tell wat to do.
bin/start-all.sh
starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-demo.example.com.out
[email protected]’s password:
Thanx
Reply
- Rahul on October 31, 2013 4:11 am
  Hi Dheeraj,
  I think you don’t have completed Step #3 properly. Please check and complete it. Make sure you are able to ssh without password.
  Reply
Rakesh on October 24, 2013 9:21 pm
I have configured cloudera on a single node successfully how ever when i lanuch Hue @//myaddress.8888 i see below error message at configuration level.
”
Configuration files located in /var/run/cloudera-scm-agent/process/57-hue-HUE_SERVER
Potential misconfiguration detected. Fix and restart Hue.”
and my hue_safety_valve.ini looks as below
[hadoop]
[[mapred_clusters]]
[[[default]]]
jobtracker_host=servername
thrift_port=9290
jobtracker_port=8021
submit_to=True
hadoop_mapred_home={{HADOOP_MR1_HOME}}
hadoop_bin={{HADOOP_BIN}}
hadoop_conf_dir={{HADOOP_CONF_DIR}}
security_enabled=false
Reply
Shannon on October 16, 2013 10:17 pm
Can you please suggest the steps to uninstall the apache hadoop . I am planning to test cloudera as well , if you can share steps for cloudera as well that would be awesome ! .
Reply
Toan Nguyen on October 15, 2013 5:59 pm
Please help me, when I run $ bin/start-all.sh, output have a error,
starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-hadoop-master.out
The authenticity of host ‘localhost (::1)’ can’t be established.
RSA key fingerprint is 7b:6d:cb:fc:48:7b:c6:42:a5:6a:64:83:ab:a8:95:95.
Are you sure you want to continue connecting (yes/no)? hadoop-slave-2: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave-2.out
hadoop-slave-1: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-hadoop-slave-1.out
Reply
- Rahul on October 16, 2013 2:38 am
  Hi Toan,
  Are you configuring hadoop multinode cluster ?
  Reply
Rakesh on October 14, 2013 9:54 pm
I have found solutions for my issues, thanks rahul for your post it helped me a lot . !!!
Reply
- Rahul on October 22, 2013 7:52 am
  Sounds good Rakesh… Thanks
  Reply
Rakesh on October 10, 2013 6:23 pm
I have tried below urls for job tracker but none of them worked out . Also i confirmed with network team they confirmed that there is no firewall blocking .
http://ipaddress:50030/
http://localhost:50030/
http://servername:50030/
Reply
Rakesh on October 9, 2013 10:44 pm
Core xml file for reference
fs.default.name
hdfs://hadoop-01:9000/
dfs.permissions
false
Reply
- Rahul on October 10, 2013 3:42 am
  Hi Rakesh,
  TecAdmin.net commnets does not support HTML tags properly. Can you please show the screenshot of your file..
  Reply
  - Rakesh on October 10, 2013 5:51 pm
    Hi Rahul,
    sorry for my delay response.
    I was able to resolve the above error ,By mistake i deleted the wrapper in the xml which caused the error , I have now kept the right data in xml and found below output after format.
    [[email protected] hadoop]$ /opt/jdk1.7.0_40/bin/jps
    2148 NameNode
    2628 TaskTracker
    2767 Jps
    2503 JobTracker
    2415 SecondaryNameNode
    2274 DataNode
    However in the xml files i have used localhost instead of my server ipaddress is that fine , because i am unable to launch the web urls for the same
    Reply
    - Sudip on October 21, 2013 9:32 am
      Hi Rahul,
      I am trying to install a 3 node Hadoop Cluster and using v2.1.0-beta version. Though I could sense there are a lot of changes in terems of directory structure (as compared to v1.2.1) but finally after configuring most of the steps mentioned here , when I issued the command “$JAVA_HOME/bin/jps” on my master server I am seeing O/P:
      16467 Resource Manager
      15966 NameNode
      16960 Jps
      16255 SecondaryNameNode
      But I cant’t see my DataNode or NodeManager getting started.
      I can send you the ouptut of “start-all.sh ” script output (which throws a lot of error lines but somehow able to start the above mentioned services), if you think that can be of any help to you.
      Secondly I am not able open any pages as mentioned part of web services; e.g.
      http://:50030/ for the Jobtracker -> Resource Manager
      http://:50070/ for the Namenode
      Would appreciate any comment from your side on my queries here.
      Reply
      - Rahul on October 21, 2013 10:36 am
        Hi Sudeep,
        Yes, Please post the output of start-all.sh command with log files. But first plz empty your log files and them run start-all.sh after that post all outputs.
        Also I prefer, if you post your question on our new forum, so it will be better to communicate.
        Reply
      - Sudip on October 22, 2013 5:33 am
        Here is the output of start-all.sh script.
        [[email protected] sbin]$ ./start-all.sh
        This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
        13/10/20 18:45:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
        Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
        It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
        hadoop-master]
        sed: -e expression #1, char 6: unknown option to `s’
        Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
        64-Bit: ssh: Could not resolve hostname 64-Bit: Temporary failure in name resolution
        HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
        Server: ssh: Could not resolve hostname Server: Temporary failure in name resolution
        VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
        warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
        You: ssh: Could not resolve hostname You: Temporary failure in name resolution
        have: ssh: Could not resolve hostname have: Temporary failure in name resolution
        library: ssh: Could not resolve hostname library: Temporary failure in name resolution
        have: ssh: Could not resolve hostname have: Temporary failure in name resolution
        which: ssh: Could not resolve hostname which: Temporary failure in name resolution
        disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
        will: ssh: Could not resolve hostname will: Temporary failure in name resolution
        stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
        guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
        might: ssh: Could not resolve hostname might: Temporary failure in name resolution
        stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
        the: ssh: Could not resolve hostname the: Temporary failure in name resolution
        loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
        VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
        ‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
        to: ssh: Could not resolve hostname to: Temporary failure in name resolution
        that: ssh: Could not resolve hostname that: Temporary failure in name resolution
        try: ssh: Could not resolve hostname try: Temporary failure in name resolution
        highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
        link: ssh: Could not resolve hostname link: Temporary failure in name resolution
        fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
        you: ssh: Could not resolve hostname you: Temporary failure in name resolution
        guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
        fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
        or: ssh: Could not resolve hostname or: Temporary failure in name resolution
        It’s: ssh: Could not resolve hostname It’s: Temporary failure in name resolution
        recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
        the: ssh: Could not resolve hostname the: Temporary failure in name resolution
        -c: Unknown cipher type ‘cd’
        ‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
        with: ssh: Could not resolve hostname with: Temporary failure in name resolution
        with: ssh: Could not resolve hostname with: Temporary failure in name resolution
        now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
        noexecstack’.: ssh: Could not resolve hostname noexecstack’.: Temporary failure in name resolution
        ‘,: ssh: Could not resolve hostname ‘,: Temporary failure in name resolution
        The: ssh: Could not resolve hostname The: Temporary failure in name resolution
        it: ssh: Could not resolve hostname it: Temporary failure in name resolution
        library: ssh: Could not resolve hostname library: Temporary failure in name resolution
        hadoop-master: starting namenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-namenode-hadoop-master.out
        hadoop-master: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
        hadoop-master: It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
        hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
        hadoop-slave-1: bash: /opt/hadoop/hadoop/sbin/hadoop-daemon.sh: No such file or directory
        hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
        hadoop-slave-2: bash: /opt/hadoop/hadoop/sbin/hadoop-daemon.sh: No such file or directory
        Starting secondary namenodes [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
        It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
        0.0.0.0]
        sed: -e expression #1, char 6: unknown option to `s’
        64-Bit: ssh: Could not resolve hostname 64-Bit: Temporary failure in name resolution
        Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
        Server: ssh: Could not resolve hostname Server: Temporary failure in name resolution
        HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
        VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
        warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
        have: ssh: Could not resolve hostname have: Temporary failure in name resolution
        You: ssh: Could not resolve hostname You: Temporary failure in name resolution
        library: ssh: Could not resolve hostname library: Temporary failure in name resolution
        might: ssh: Could not resolve hostname might: Temporary failure in name resolution
        loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
        try: ssh: Could not resolve hostname try: Temporary failure in name resolution
        guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
        stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
        which: ssh: Could not resolve hostname which: Temporary failure in name resolution
        have: ssh: Could not resolve hostname have: Temporary failure in name resolution
        now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
        stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
        The: ssh: Could not resolve hostname The: Temporary failure in name resolution
        guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
        will: ssh: Could not resolve hostname will: Temporary failure in name resolution
        the: ssh: Could not resolve hostname the: Temporary failure in name resolution
        recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
        disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
        VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
        to: ssh: Could not resolve hostname to: Temporary failure in name resolution
        ‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
        highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
        ‘,: ssh: Could not resolve hostname ‘,: Temporary failure in name resolution
        that: ssh: Could not resolve hostname that: Temporary failure in name resolution
        -c: Unknown cipher type ‘cd’
        ‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
        link: ssh: Could not resolve hostname link: Temporary failure in name resolution
        fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
        or: ssh: Could not resolve hostname or: Temporary failure in name resolution
        library: ssh: Could not resolve hostname library: Temporary failure in name resolution
        it: ssh: Could not resolve hostname it: Temporary failure in name resolution
        you: ssh: Could not resolve hostname you: Temporary failure in name resolution
        It’s: ssh: Could not resolve hostname It’s: Temporary failure in name resolution
        with: ssh: Could not resolve hostname with: Temporary failure in name resolution
        with: ssh: Could not resolve hostname with: Temporary failure in name resolution
        fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
        the: ssh: Could not resolve hostname the: Temporary failure in name resolution
        noexecstack’.: ssh: Could not resolve hostname noexecstack’.: Temporary failure in name resolution
        0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-hadoop-master.out
        0.0.0.0: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
        0.0.0.0: It’s highly recommended that you fix the library with ‘execstack -c ‘, or link it with ‘-z noexecstack’.
        13/10/20 18:45:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
        starting yarn daemons
        starting resourcemanager, logging to /opt/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-hadoop-master.out
        hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
        hadoop-slave-1: bash: /opt/hadoop/hadoop/sbin/yarn-daemon.sh: No such file or directory
        hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop: No such file or directory
        hadoop-slave-2: bash: /opt/hadoop/hadoop/sbin/yarn-daemon.sh: No such file or directory
        [[email protected] sbin]$ $JAVA_HOME/bin/jps
        15130 Jps
        14697 SecondaryNameNode
        14872 ResourceManager
        14408 NameNode
        [[email protected] sbin]$
        Reply
    - Lini on June 25, 2015 1:36 am
      hello..
      i also faced the same issue as mr Rakesh
      while executing the format command
      [[email protected] hadoop]$ bin/hadoop namenode -format
      15/06/24 18:32:24 INFO namenode.NameNode: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NameNode
      STARTUP_MSG: host = localhost.localdomain/127.0.0.1
      STARTUP_MSG: args = [-format]
      STARTUP_MSG: version = 1.2.1
      STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
      STARTUP_MSG: java = 1.7.0_79
      ************************************************************/
      [Fatal Error] core-site.xml:1:21: The processing instruction target matching “[xX][mM][lL]” is not allowed.
      15/06/24 18:32:25 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
      15/06/24 18:32:25 ERROR namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1249)
      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107)
      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:420)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.setStartupOption(NameNode.java:1374)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1463)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
      Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 1; columnNumber: 21; The processing instruction target matching “[xX][mM][lL]” is not allowed.
      at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
      at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
      at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
      … 6 more
      15/06/24 18:32:25 INFO namenode.NameNode: SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
      please help me solve the issue
      thanks in advance
      Reply
      - Rahul on June 26, 2015 9:39 am
        Hi Lini,
        Please check core-site.xml file. It looks this file is some syntax issue.
        Reply
      - Stacy on October 25, 2017 7:42 pm
        delete the line from the top of the core-site.xml file
        Reply
Rakesh on October 9, 2013 10:35 pm
Hi
I ran the format step and got below error message
bin/hadoop namenode -format
13/10/09 15:33:33 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hadoop-01/10.49.14.42
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.6.0_20
************************************************************/
[Fatal Error] core-site.xml:5:2: The markup in the document following the root element must be well-formed.
13/10/09 15:33:33 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
13/10/09 15:33:33 ERROR namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1249)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1107)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:420)
at org.apache.hadoop.hdfs.server.namenode.NameNode.setStartupOption(NameNode.java:1374)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1463)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/hadoop/hadoop/conf/core-site.xml; lineNumber: 5; columnNumber: 2; The markup in the document following the root element must be well-formed.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:253)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:288)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
… 6 more
13/10/09 15:33:33 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-01/10.49.14.42
************************************************************/
Reply
tosh on October 8, 2013 7:24 pm
Hi,
I have successfully installed hadoop cluster, and it seems to run fine. I was able to access the webservices after changing the “localhost” to the host’s ip address. However, when clicking the browse file system in namenode webaccess, Browser does not find the webpage. the url shows as “http://master:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/”
(I get the same error at a few other places, but manually replacing the master with ip address seems to solve the issue)
“master” is the hostname I chose for this machine. My guess is this is a DNS issue, but I was wondering if we can change any config files so the links resolve to an ip address instead of master.
I also see that your mapred-site.xml and core-site.xml have “hdfs://localhost:port_num” in them, but you are able to access the web interface using your domain name (in the screenshots you provided). I am installing hadoop on remote system, and had to replace the localhost with the server’s ip address to access the web interface. Is this accepted practice? Or should I leave them as localhost and make changes elsewhere?
Third, for hdfs-site.xml, the only property I defined is the replication value ( I was following a separate tutorial until about halfway). So I was wondering if you could explain what leaving the dfs.data.dir and dfs.name.dir would do? If we do define those values, Can it be any directory, or does it have to be inside the hadoop installation?
And last,
when I do get to the file browser ( by manually replacing “master” in the url with the ip address), I see a tmp directory there as well that contains the mapred folder and it’s subfolders . Is this normal?
Sincerely,
Ashutosh
Reply
- tosh on October 8, 2013 7:32 pm
  Clarification:
  By “I was able to access the web services after changing the “localhost” to the host’s ip address”, I meant changed the “localhost” in the hadoop/conf/*.xml files to a static ip address that I am now using to access the web services.
  Reply
Rodrigo Bittencourt on October 7, 2013 3:34 pm
Hello,
Rahul I have a question, why the step 7 does not work in my cluster ?
Reply
- Rahul on October 8, 2013 2:05 am
  Hi Rodrigo,
  What issue are you getting ?
  Reply
  - Rodrigo Bittencourt on October 8, 2013 11:40 am
    I can not open the URL of hadoop in my cluster.
    where was configured the URL ?
    Reply
    - tosh on October 9, 2013 11:08 pm
      A lot of factors play into the url’s not showing the result. If you are on a local environment ( working on the same machine that you are installing the cluster on) then this tutorial should have worked.
      To the best of my knowledge, the “localhost” in
      hdfs://localhost:9000/ and
      localhost:9001
      defined inside the core-site.xml and mapred-site.xml should be changed to the hostname of your system.
      i.e. if the hostname of your system is “linuxuser”, then change the “localhost” to “linuxuser” in these line, then add the following line in /etc/hosts file:
      127.0.0.1 linuxuser
      If you are working on a remote server, then same as above except the host file should have the static ip instead of 127.0.0.1 , and you will be able to see the web services, but there is another problem I ran into. See my question below .
      It could be the firewall setup as well. try disabling iptables to check if it is firewall issue. If you do this, make sure you re start iptables when you are done
      Reply
  - jinu prakash on June 29, 2016 8:21 am
    hi when i run bin/start-all.sh i get a message saying
    fatal error mapred-site.xml:7:1:XML document structures must start and end within same entity
    and also the same message when i run
    $ bin/hadoop jar hadoop-examples-*.jar grep input output ‘dfs[a-z.]+’
    I did this in ubuntu vbox
    Reply
girish on September 30, 2013 11:25 am
[[email protected] hadoop]$ bin/hadoop namenode -format
13/09/30 04:53:34 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = java.net.UnknownHostException: girishniyer: girishniyer
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_40
************************************************************/
13/09/30 04:53:34 INFO util.GSet: Computing capacity for map BlocksMap
13/09/30 04:53:34 INFO util.GSet: VM type = 64-bit
13/09/30 04:53:34 INFO util.GSet: 2.0% max memory = 932184064
13/09/30 04:53:34 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/09/30 04:53:34 INFO util.GSet: recommended=2097152, actual=2097152
13/09/30 04:53:34 INFO namenode.FSNamesystem: fsOwner=hadoop
13/09/30 04:53:34 INFO namenode.FSNamesystem: supergroup=supergroup
13/09/30 04:53:34 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/09/30 04:53:34 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/09/30 04:53:34 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/09/30 04:53:34 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
13/09/30 04:53:34 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/09/30 04:53:34 ERROR namenode.NameNode: java.io.IOException: Cannot create directory /opt/hadoop/hadoop/dfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:294)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1337)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1356)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1261)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1467)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1488)
13/09/30 04:53:34 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: girishniyer: girishniyer
************************************************************/
[[email protected] hadoop]$
i configured java as per ur article.
then till formatting name node no issues
bt here iam stuck again.
could u please help me?
Reply
- Rahul on September 30, 2013 11:48 am
  Hi Girish,
  You systems hostname is “girishniyer” which is not resolving to any ip. Do a entry in /etc/hosts file like below
  <system_ip> girishniyer
  Reply
  - Girish S on October 7, 2013 8:00 pm
    I just began learning hadoop. Your article was superbly simple and straight forward. The patience with which you helped Girish Iyer is commendable.
    Reply
    - Rahul on October 10, 2013 3:43 am
      Thanks Girish,
      Reply
kedar on September 27, 2013 5:31 am
Excellent article …. I could configure hadoop in 20 mins … do you have any more learning article for begineers like first wordcount program and all
Reply
girish iyer on September 22, 2013 12:50 pm
when i type the following,i got the error as follows
[[email protected] hadoop]$ bin/start-all.sh
also i cant able to format the name node.it says no such a directory…please help.
is it due to giving the java path ?
starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-girishniyer.out
/opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
/opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
/opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
localhost: starting datanode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-girishniyer.out
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
localhost: starting secondarynamenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-girishniyer.out
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
starting jobtracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-girishniyer.out
/opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
/opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
/opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
localhost: starting tasktracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-girishniyer.out
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 350: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: /usr/bin/java/jdk1.7.0_25/bin/java: Not a directory
localhost: /opt/hadoop/hadoop/libexec/../bin/hadoop: line 434: exec: /usr/bin/java/jdk1.7.0_25/bin/java: cannot execute: Not a directory
Reply
- Rahul on September 23, 2013 2:48 am
  Hi Girish,
  Did you configured JAVA_HOME correctly… ?
  Check this article and make sure java is properly configured as in step 5.4 Edit hadoop-env.sh
  http://tecadmin.net/steps-to-install-java-on-centos-5-6-or-rhel-5-6/
  Reply
  - girish iyer on September 25, 2013 7:41 am
    Thank you rahul for ur quick reply.
    i configured java and export the java_home path as per ur blog
    now when i give “bin/hadoop namenode -format”
    the following is coming
    bin/hadoop: line 350: /opt/jdk1.7.0_25/bin/java: No such file or directory
    bin/hadoop: line 434: /opt/jdk1.7.0_25/bin/java: No such file or directory
    bin/hadoop: line 434: exec: /opt/jdk1.7.0_25/bin/java:
    cannot execute: No such file or directory
    when i give which java command,my path displayed is as follows.
    “/usr/bin/java”
    when i typed java version,its as follows
    [[email protected] hadoop]$ java -version
    java version “1.7.0_25”
    OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64)
    OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
    i am little confused hence.
    please help me…waiting for ur reply
    thanks in advance
    Reply
    - Rahul on September 26, 2013 10:58 am
      Hi Girish,
      Step #5.4 still not configured properly. How did you installed Java on your system. Source or RPM ?
      Reply
      - Girish on September 29, 2013 5:51 am
        hi rahul,
        java is there in centos when i installed the cent os
        i reinstalled the centos6.4 again
        nw the java version is as follows
        ”
        [[email protected] Desktop]$ which java
        /usr/bin/java
        [[email protected] Desktop]$ java -version
        java version “1.7.0_09-icedtea”
        OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
        OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
        ”
        did i need to re install the java after removing this.
        if so can u please help me
        Reply
        
        Rahul on September 30, 2013 3:29 am
        Hi Girish,
        OpenJDK default installation path should be /usr/lib/jvm/java-<java version>-openjdk-<java version>.x86_64/. Please check, if this path exists add it in configuration
        else do a fresh java install http://tecadmin.net/steps-to-install-java-on-centos-5-6-or-rhel-5-6/
    - Danish on September 27, 2013 8:44 am
      Hello,
      I had the same error, it was because my jave home was /opt/jdk1.7.0.25/ . There was no underscore in the jdk1.7.0_25. May be this helps you!
      Reply
Baahu on September 21, 2013 1:20 pm
Good article !!
Reply
Girish N Iyer on September 20, 2013 7:51 am
This article is really helpful and grear work…..Thank You.
I step 4,when i type,¨ tar -xzf hadoop-1.2.0.tar.gz
The following error is displaying..could u pleaSe help
tar (child): hadoop-1.2.0.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Reply
- Rahul on September 20, 2013 9:26 am
  Hi Girish,
  Now hadoop 1.2.1 latest stable version is available. So downloaded archive name is changed.
  We have updated article step 4. Kindly try now.
  Reply
Raja on September 19, 2013 1:07 pm
This is really great article for entry level person who like to see how hadoop looks like and play around. great job!
Also it would be great if you add how to stop hadoop at the end of your article.
Thank you once again.
Reply
- Rahul on September 20, 2013 9:33 am
  Hi Raja,
  Thanks for appreciation.
  We have added last step ( Step 8 ) to how to stop all hadoop services.
  Reply
arun on July 20, 2013 3:19 pm
How to add a new not to this cluster… ?
Reply
maillot de foot 2013-2014 on June 29, 2013 4:08 pm
whoah this blog is fantastic i like reading your articles. Keep up the good paintings! You understand, a lot of people are hunting round for this info, you could aid them greatly.
Reply

How to Install and Configure Apache Hadoop on CentOS & Fedora

1. Prerequsities

2. Create Hadoop User

3. Download Hadoop 3.1 Archive

4. Setup Hadoop Pseudo-Distributed Mode

4.1. Setup Hadoop Environment Variables

4.2. Setup Hadoop Configuration Files

Edit core-site.xml

Edit hdfs-site.xml

Edit mapred-site.xml

Edit yarn-site.xml

4.3. Format Namenode

5. Start Hadoop Cluster

6. Access Hadoop Services in Browser

7. Test Hadoop Single Node Setup

Creating Directory In HDFS And Copy Files (Hadoop)

How to Install Apache Hadoop on Ubuntu 22.04

How to Install and Configure Hadoop on Ubuntu 20.04

103 Comments

Leave A Reply Cancel Reply

How to Install and Configure Apache Hadoop on CentOS & Fedora

1. Prerequsities

2. Create Hadoop User

3. Download Hadoop 3.1 Archive

4. Setup Hadoop Pseudo-Distributed Mode

4.1. Setup Hadoop Environment Variables

4.2. Setup Hadoop Configuration Files

Edit core-site.xml

Edit hdfs-site.xml

Edit mapred-site.xml

Edit yarn-site.xml

4.3. Format Namenode

5. Start Hadoop Cluster

6. Access Hadoop Services in Browser

7. Test Hadoop Single Node Setup

Related Posts

Creating Directory In HDFS And Copy Files (Hadoop)

How to Install Apache Hadoop on Ubuntu 22.04

How to Install and Configure Hadoop on Ubuntu 20.04

103 Comments

Leave A Reply Cancel Reply