The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
Our earlier article about hadoop was describing to how to setup single node cluster. This article will help you for step by step installing and configuring Hadoop Multi-Node Cluster on CentOS/RHEL 6.
Setup Details:
Hadoop Master: 192.168.1.15 ( hadoop-master )
Hadoop Slave : 192.168.1.16 ( hadoop-slave-1 )
Hadoop Slave : 192.168.1.17 ( hadoop-slave-2 )
Step 1. Install Java
Before installing hadoop make sure you have java installed on all nodes of hadoop cluster systems.
#java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
If you do not have java installed use following article to install Java.
Steps to install JAVA 7 on CentOS/RHEL 7/6/5
Step 2. Create User Account
Create a system user account on both master and slave systems to use for hadoop installation
# useradd hadoop # passwd hadoop
Changing password for user hadoop. New password: Retype new password: passwd: all authentication tokens updated successfully.
Step 3: Add FQDN Mapping
Edit /etc/hosts file on all master and slave servers and add following entries.
# vim /etc/hosts
192.168.1.15 hadoop-master192.168.1.16 hadoop-slave-1192.168.1.17 hadoop-slave-2
Step 4. Configuring Key Based Login
It’s required to set up hadoop user to ssh itself without password. Use following commands to configure auto login between all hadoop cluster servers..
# su - hadoop $ ssh-keygen -t rsa $ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected] $ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected] $ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected] $ chmod 0600 ~/.ssh/authorized_keys $ exit
Step 5. Download and Extract Hadoop Source
Download hadoop latest available version from its official site at hadoop-master server only.
# mkdir /opt/hadoop # cd /opt/hadoop/ # wget http://apache.mesi.com.ar/hadoop/common/hadoop-1.2.0/hadoop-1.2.0.tar.gz # tar -xzf hadoop-1.2.0.tar.gz # mv hadoop-1.2.0 hadoop # chown -R hadoop /opt/hadoop # cd /opt/hadoop/hadoop/
Step 6: Configure Hadoop
First edit hadoop configuration files and make following changes.
6.1 Edit core-site.xml
# vim conf/core-site.xml
#Add the following inside the configuration tag
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-master:9000/</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
6.2 Edit hdfs-site.xml
# vim conf/hdfs-site.xml
# Add the following inside the configuration tag <property> <name>dfs.data.dir</name> <value>/opt/hadoop/hadoop/dfs/name/data</value> <final>true</final> </property> <property> <name>dfs.name.dir</name> <value>/opt/hadoop/hadoop/dfs/name</value> <final>true</final> </property> <property> <name>dfs.replication</name> <value>1</value> </property>
6.3 Edit mapred-site.xml
# vim conf/mapred-site.xml
# Add the following inside the configuration tag
<property>
<name>mapred.job.tracker</name>
<value>hadoop-master:9001</value>
</property>
6.4 Edit hadoop-env.sh
# vim conf/hadoop-env.sh
export JAVA_HOME=/opt/jdk1.7.0_75 export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true export HADOOP_CONF_DIR=/opt/hadoop/hadoop/conf
Set JAVA_HOME path as per your system configuration for java.
Step 7: Copy Hadoop Source to Slave Servers
After updating above configuration, we need to copy the source files to all slaves servers.
# su - hadoop $ cd /opt/hadoop $ scp -r hadoop hadoop-slave-1:/opt/hadoop $ scp -r hadoop hadoop-slave-2:/opt/hadoop
Step 8: Configure Hadoop on Master Server Only
Go to hadoop source folder on hadoop-master and do following settings.
# su - hadoop $ cd /opt/hadoop/hadoop
$ vim conf/masters hadoop-master
$ vim conf/slaves hadoop-slave-1 hadoop-slave-2
Format Name Node on Hadoop Master only
# su - hadoop $ cd /opt/hadoop/hadoop $ bin/hadoop namenode -format
13/07/13 10:58:07 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoop-master/192.168.1.15 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473; compiled by 'hortonfo' on Mon May 6 06:59:37 UTC 2013 STARTUP_MSG: java = 1.7.0_25 ************************************************************/ 13/07/13 10:58:08 INFO util.GSet: Computing capacity for map BlocksMap 13/07/13 10:58:08 INFO util.GSet: VM type = 32-bit 13/07/13 10:58:08 INFO util.GSet: 2.0% max memory = 1013645312 13/07/13 10:58:08 INFO util.GSet: capacity = 2^22 = 4194304 entries 13/07/13 10:58:08 INFO util.GSet: recommended=4194304, actual=4194304 13/07/13 10:58:08 INFO namenode.FSNamesystem: fsOwner=hadoop 13/07/13 10:58:08 INFO namenode.FSNamesystem: supergroup=supergroup 13/07/13 10:58:08 INFO namenode.FSNamesystem: isPermissionEnabled=true 13/07/13 10:58:08 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 13/07/13 10:58:08 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 13/07/13 10:58:08 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 13/07/13 10:58:08 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/07/13 10:58:08 INFO common.Storage: Image file of size 112 saved in 0 seconds. 13/07/13 10:58:08 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/opt/hadoop/hadoop/dfs/name/current/edits 13/07/13 10:58:08 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/opt/hadoop/hadoop/dfs/name/current/edits 13/07/13 10:58:08 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted. 13/07/13 10:58:08 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.1.15 ************************************************************/
Step 9: Start Hadoop Services
Use the following command to start all hadoop services on Hadoop-Master
$ bin/start-all.sh
I cant see “Step 8: Configure Hadoop on Master Server Only” masters file and slaves file..
I use centos7 and hadoop 3.2.0
Why ??
thank u brother..
Excellent article.
Hi Rahul,
Am facing below issue,Master Started but Slaves not started..
bin/start-all.sh
namenode running as process 2767. Stop it first.
hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-master: secondarynamenode running as process 2916. Stop it first.
jobtracker running as process 2996. Stop it first.
hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
Hi Rahul,
Am facing below issue,pls guide me
bin/start-all.sh
starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-ip-10-0-0-59.out
hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-master: starting secondarynamenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-ip-10-0-0-59.out
starting jobtracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-ip-10-0-0-59.out
hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
[[email protected] hadoop]$ bin/stop-all.sh
stopping jobtracker
hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
stopping namenode
hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
hadoop-master: stopping secondarynamenode
By Selva
Hi Mr Rahul,
Could you kindly clarify me can we share hadoop single node cluster to multiple users a (group of 60). If so could you kindly help me for the same
Hi Rahul, great article but I got stuck on step #7. I run the scp commands, the error I got was “Access denied”. I then decided to give hadoop user on each node an administrator/root privilege. I then run the command again just to receive the follwoing error message :
[[email protected] hadoop]$ scp -r hadoop hadoop-slave-2:/opt/hadoop
[email protected]’s password:
hadoop: No such file or directory
What do you I might have done wrong? Thanks for your help in advance.
Hi Rahul,
I have done whatever you have suggested and everything is going fine except last step to start hadoop in master after giving command start-all.sh getting error ” -bash : start-all.sh: command not found”. Even though i have given hadoop path in .bashrc file.
If my comments will be shown here I will explain what I did to fix my problem. It is all fixed and working now!
Finally, I have got them working now. They all are alive now, but yet I can’t put any files into hdfs:
[[email protected] hadoop]$ ./bin/hdfs dfs -put /home/hadoop/node_setup /names/node_setup
16/07/19 21:11:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
16/07/19 21:11:42 INFO hdfs.DFSClient: Abandoning BP-1606715859-192.168.0.13-1468886056606:blk_1073741825_1001
16/07/19 21:11:42 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.0.15:50010,DS-bb00767f-26f0-428f-b563-30c2b712fd5f,DISK]
16/07/19 21:11:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
16/07/19 21:11:42 INFO hdfs.DFSClient: Abandoning BP-1606715859-192.168.0.13-1468886056606:blk_1073741826_1002
16/07/19 21:11:42 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.0.16:50010,DS-c58b5570-9cd7-491b-823f-d08a67482a0f,DISK]
16/07/19 21:11:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
16/07/19 21:11:42 INFO hdfs.DFSClient: Abandoning BP-1606715859-192.168.0.13-1468886056606:blk_1073741827_1003
16/07/19 21:11:42 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.0.14:50010,DS-757164da-1541-43ff-9271-379abb45f774,DISK]
16/07/19 21:11:42 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /names/node_setup._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1547)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1459)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1255)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
put: File /names/node_setup._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
[[email protected] hadoop]$