Facebook Twitter Instagram
    TecAdmin
    • Home
    • FeedBack
    • Submit Article
    • About Us
    Facebook Twitter Instagram
    TecAdmin
    You are at:Home»Opensource»How to Set Up Hadoop Multi-Node Cluster on CentOS 7/6

    How to Set Up Hadoop Multi-Node Cluster on CentOS 7/6

    By RahulJuly 22, 20134 Mins Read

    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

    Advertisement

    Our earlier article about hadoop was describing to how to setup single node cluster. This article will help you for step by step installing and configuring Hadoop Multi-Node Cluster on CentOS/RHEL 6.

    hadoop-st

    Setup Details:

    Hadoop Master: 192.168.1.15 ( hadoop-master )
    Hadoop Slave : 192.168.1.16 ( hadoop-slave-1 )
    Hadoop Slave : 192.168.1.17 ( hadoop-slave-2 )

    Step 1. Install Java

    Before installing hadoop make sure you have java installed on all nodes of hadoop cluster systems.

    # java -version
    
    java version "1.7.0_75"
    Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
    

    If you do not have java installed use following article to install Java.

    Steps to install JAVA 7 on CentOS/RHEL 7/6/5

    Step 2. Create User Account

    Create a system user account on both master and slave systems to use for hadoop installation

    # useradd hadoop
    # passwd hadoop
    
    Changing password for user hadoop.
    New password:
    Retype new password:
    passwd: all authentication tokens updated successfully.
    

    Step 3: Add FQDN Mapping

    Edit /etc/hosts file on all master and slave servers and add following entries.

    # vim /etc/hosts
    
    192.168.1.15 hadoop-master
    192.168.1.16 hadoop-slave-1
    192.168.1.17 hadoop-slave-2
    

    Step 4. Configuring Key Based Login

    It’s required to set up hadoop user to ssh itself without password. Use following commands to configure auto login between all hadoop cluster servers..

    # su - hadoop
    $ ssh-keygen -t rsa
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
    $ chmod 0600 ~/.ssh/authorized_keys
    $ exit
    

    Step 5. Download and Extract Hadoop Source

    Download hadoop latest available version from its official site at hadoop-master server only.

    # mkdir /opt/hadoop
    # cd /opt/hadoop/
    # wget http://apache.mesi.com.ar/hadoop/common/hadoop-1.2.0/hadoop-1.2.0.tar.gz
    # tar -xzf hadoop-1.2.0.tar.gz
    # mv hadoop-1.2.0 hadoop
    # chown -R hadoop /opt/hadoop
    # cd /opt/hadoop/hadoop/
    

    Step 6: Configure Hadoop

    First edit hadoop configuration files and make following changes.
    6.1 Edit core-site.xml

    # vim conf/core-site.xml
    
    #Add the following inside the configuration tag
    <property>
        <name>fs.default.name</name>
        <value>hdfs://hadoop-master:9000/</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
    

    6.2 Edit hdfs-site.xml

    # vim conf/hdfs-site.xml
    
    # Add the following inside the configuration tag
    <property>
    	<name>dfs.data.dir</name>
    	<value>/opt/hadoop/hadoop/dfs/name/data</value>
    	<final>true</final>
    </property>
    <property>
    	<name>dfs.name.dir</name>
    	<value>/opt/hadoop/hadoop/dfs/name</value>
    	<final>true</final>
    </property>
    <property>
    	<name>dfs.replication</name>
    	<value>1</value>
    </property>
    

    6.3 Edit mapred-site.xml

    # vim conf/mapred-site.xml
    
    # Add the following inside the configuration tag
    <property>
            <name>mapred.job.tracker</name>
    	<value>hadoop-master:9001</value>
    </property>
    

    6.4 Edit hadoop-env.sh

    # vim conf/hadoop-env.sh
    
    export JAVA_HOME=/opt/jdk1.7.0_75
    export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
    export HADOOP_CONF_DIR=/opt/hadoop/hadoop/conf
    

    Set JAVA_HOME path as per your system configuration for java.

    Step 7: Copy Hadoop Source to Slave Servers

    After updating above configuration, we need to copy the source files to all slaves servers.

    # su - hadoop
    $ cd /opt/hadoop
    $ scp -r hadoop hadoop-slave-1:/opt/hadoop
    $ scp -r hadoop hadoop-slave-2:/opt/hadoop
    

    Step 8: Configure Hadoop on Master Server Only

    Go to hadoop source folder on hadoop-master and do following settings.

    # su - hadoop
    $ cd /opt/hadoop/hadoop
    
    $ vim conf/masters
    
    hadoop-master
    
    $ vim conf/slaves
    
    hadoop-slave-1
    hadoop-slave-2
    

    Format Name Node on Hadoop Master only

    # su - hadoop
    $ cd /opt/hadoop/hadoop
    $ bin/hadoop namenode -format
    
    13/07/13 10:58:07 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG:   host = hadoop-master/192.168.1.15
    STARTUP_MSG:   args = [-format]
    STARTUP_MSG:   version = 1.2.0
    STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473; compiled by 'hortonfo' on Mon May  6 06:59:37 UTC 2013
    STARTUP_MSG:   java = 1.7.0_25
    ************************************************************/
    13/07/13 10:58:08 INFO util.GSet: Computing capacity for map BlocksMap
    13/07/13 10:58:08 INFO util.GSet: VM type       = 32-bit
    13/07/13 10:58:08 INFO util.GSet: 2.0% max memory = 1013645312
    13/07/13 10:58:08 INFO util.GSet: capacity      = 2^22 = 4194304 entries
    13/07/13 10:58:08 INFO util.GSet: recommended=4194304, actual=4194304
    13/07/13 10:58:08 INFO namenode.FSNamesystem: fsOwner=hadoop
    13/07/13 10:58:08 INFO namenode.FSNamesystem: supergroup=supergroup
    13/07/13 10:58:08 INFO namenode.FSNamesystem: isPermissionEnabled=true
    13/07/13 10:58:08 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
    13/07/13 10:58:08 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
    13/07/13 10:58:08 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
    13/07/13 10:58:08 INFO namenode.NameNode: Caching file names occuring more than 10 times
    13/07/13 10:58:08 INFO common.Storage: Image file of size 112 saved in 0 seconds.
    13/07/13 10:58:08 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/opt/hadoop/hadoop/dfs/name/current/edits
    13/07/13 10:58:08 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/opt/hadoop/hadoop/dfs/name/current/edits
    13/07/13 10:58:08 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted.
    13/07/13 10:58:08 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.1.15
    ************************************************************/
    

    Step 9: Start Hadoop Services

    Use the following command to start all hadoop services on Hadoop-Master

    $ bin/start-all.sh
    

    database hadoop hadoop cluster
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email WhatsApp

    Related Posts

    Using HDFS Filesystem (CLI)

    Creating Directory In HDFS And Copy Files (Hadoop)

    How to Install Hadoop on Ubuntu 22.04

    How to Install Apache Hadoop on Ubuntu 22.04

    Installing MySQL 8.0 on Amazon Linux 2

    How To Install MySQL 8 on Amazon Linux 2

    View 23 Comments

    23 Comments

    1. isak on April 28, 2020 8:02 pm

      hi i take these mistake. what is the reason?

      WARNING: Use of this script to execute namenode is deprecated.
      WARNING: Attempting to execute replacement “hdfs namenode” instead.

      WARNING: /opt/hadoop/hadoop/logs does not exist. Creating.
      Error: Could not find or load main class Djava.net.preferIPv4Stack=true

      Reply
    2. aruur on July 25, 2019 8:53 pm

      I cant see “Step 8: Configure Hadoop on Master Server Only” masters file and slaves file..
      I use centos7 and hadoop 3.2.0
      Why ??
      thank u brother..

      Reply
    3. Anandachetan on June 11, 2018 3:24 am

      Excellent article.

      Reply
    4. Selvaesakki on December 8, 2017 5:05 pm

      Hi Rahul,

      Am facing below issue,Master Started but Slaves not started..
      bin/start-all.sh
      namenode running as process 2767. Stop it first.
      hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-master: secondarynamenode running as process 2916. Stop it first.
      jobtracker running as process 2996. Stop it first.
      hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory

      Reply
    5. Selvaesakki on December 7, 2017 10:18 pm

      Hi Rahul,
      Am facing below issue,pls guide me
      bin/start-all.sh
      starting namenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-ip-10-0-0-59.out
      hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-master: starting secondarynamenode, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-ip-10-0-0-59.out
      starting jobtracker, logging to /opt/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-ip-10-0-0-59.out
      hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      [[email protected] hadoop]$ bin/stop-all.sh
      stopping jobtracker
      hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      stopping namenode
      hadoop-slave-2: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-2: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-slave-1: bash: line 0: cd: /opt/hadoop/hadoop/libexec/..: No such file or directory
      hadoop-slave-1: bash: /opt/hadoop/hadoop/bin/hadoop-daemon.sh: No such file or directory
      hadoop-master: stopping secondarynamenode
      By Selva

      Reply
    6. Dinakar N K on October 13, 2017 7:40 am

      Hi Mr Rahul,
      Could you kindly clarify me can we share hadoop single node cluster to multiple users a (group of 60). If so could you kindly help me for the same

      Reply
    7. Sam on December 22, 2016 7:48 am

      Hi Rahul, great article but I got stuck on step #7. I run the scp commands, the error I got was “Access denied”. I then decided to give hadoop user on each node an administrator/root privilege. I then run the command again just to receive the follwoing error message :

      [[email protected] hadoop]$ scp -r hadoop hadoop-slave-2:/opt/hadoop
      [email protected]’s password:
      hadoop: No such file or directory

      What do you I might have done wrong? Thanks for your help in advance.

      Reply
    8. Prateek Harsh on July 21, 2016 4:04 pm

      Hi Rahul,

      I have done whatever you have suggested and everything is going fine except last step to start hadoop in master after giving command start-all.sh getting error ” -bash : start-all.sh: command not found”. Even though i have given hadoop path in .bashrc file.

      Reply
    9. inner on July 19, 2016 11:37 pm

      If my comments will be shown here I will explain what I did to fix my problem. It is all fixed and working now!

      Reply
    10. inner on July 19, 2016 8:14 pm

      Finally, I have got them working now. They all are alive now, but yet I can’t put any files into hdfs:

      [[email protected] hadoop]$ ./bin/hdfs dfs -put /home/hadoop/node_setup /names/node_setup

      16/07/19 21:11:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
      java.net.NoRouteToHostException: No route to host
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
      16/07/19 21:11:42 INFO hdfs.DFSClient: Abandoning BP-1606715859-192.168.0.13-1468886056606:blk_1073741825_1001
      16/07/19 21:11:42 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.0.15:50010,DS-bb00767f-26f0-428f-b563-30c2b712fd5f,DISK]
      16/07/19 21:11:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
      java.net.NoRouteToHostException: No route to host
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
      16/07/19 21:11:42 INFO hdfs.DFSClient: Abandoning BP-1606715859-192.168.0.13-1468886056606:blk_1073741826_1002
      16/07/19 21:11:42 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.0.16:50010,DS-c58b5570-9cd7-491b-823f-d08a67482a0f,DISK]
      16/07/19 21:11:42 INFO hdfs.DFSClient: Exception in createBlockOutputStream
      java.net.NoRouteToHostException: No route to host
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
      16/07/19 21:11:42 INFO hdfs.DFSClient: Abandoning BP-1606715859-192.168.0.13-1468886056606:blk_1073741827_1003
      16/07/19 21:11:42 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.0.14:50010,DS-757164da-1541-43ff-9271-379abb45f774,DISK]
      16/07/19 21:11:42 WARN hdfs.DFSClient: DataStreamer Exception
      org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /names/node_setup._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1547)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

      at org.apache.hadoop.ipc.Client.call(Client.java:1475)
      at org.apache.hadoop.ipc.Client.call(Client.java:1412)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
      at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1459)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1255)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
      put: File /names/node_setup._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
      [[email protected] hadoop]$

      Reply
    11. inner on July 19, 2016 7:11 pm

      Thanks for the guide! Now, I am having a problem after I incorrectly turned off the cluster of 3 datanodes. All of them now are dead, it is shown after executing a command:

      ./bin/hdfs dfsadmin -refreshNodes

      Configured Capacity: 0 (0 B)
      Present Capacity: 0 (0 B)
      DFS Remaining: 0 (0 B)
      DFS Used: 0 (0 B)
      DFS Used%: NaN%
      Under replicated blocks: 0
      Blocks with corrupt replicas: 0
      Missing blocks: 0
      Missing blocks (with replication factor 1): 0

      ————————————————-
      Dead datanodes (3):

      Name: 192.168.0.15:50010 (datanode2)
      Hostname: datanode2
      Decommission Status : Normal
      Configured Capacity: 0 (0 B)
      DFS Used: 0 (0 B)
      Non DFS Used: 0 (0 B)
      DFS Remaining: 0 (0 B)
      DFS Used%: 100.00%
      DFS Remaining%: 0.00%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 0
      Last contact: Thu Jan 01 01:00:00 GMT 1970

      Name: 192.168.0.14:50010 (datanode1)
      Hostname: datanode1
      Decommission Status : Normal
      Configured Capacity: 0 (0 B)
      DFS Used: 0 (0 B)
      Non DFS Used: 0 (0 B)
      DFS Remaining: 0 (0 B)
      DFS Used%: 100.00%
      DFS Remaining%: 0.00%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 0
      Last contact: Thu Jan 01 01:00:00 GMT 1970

      Name: 192.168.0.16:50010 (datanode3)
      Hostname: datanode3
      Decommission Status : Normal
      Configured Capacity: 0 (0 B)
      DFS Used: 0 (0 B)
      Non DFS Used: 0 (0 B)
      DFS Remaining: 0 (0 B)
      DFS Used%: 100.00%
      DFS Remaining%: 0.00%
      Configured Cache Capacity: 0 (0 B)
      Cache Used: 0 (0 B)
      Cache Remaining: 0 (0 B)
      Cache Used%: 100.00%
      Cache Remaining%: 0.00%
      Xceivers: 0
      Last contact: Thu Jan 01 01:00:00 GMT 1970

      How do I recover the datanodes?
      What I have already tried is to turn off/on again, -format command, -refreshNodes command, included all datanodes in the permitted hosts file, nothing helps.

      There’s got to be a way restoring these datanodes and running them again..

      Reply
    12. Shankar on June 26, 2015 7:18 am

      Hi,
      Nice Guide, I have some problem, while starting all node except datanode is not showing in jps.
      What is the issue?
      Please help.

      Thanks in Advance
      Shankar D

      Reply
      • Rahul on June 26, 2015 9:36 am

        Hi Shankar,

        Please check log files and let us know any any error is generating there.

        Reply
        • Punit Bansal on May 8, 2017 5:01 am

          Hi Rahul,

          I have installed the hadoop in ubuntu 16.04 but i am not able to install the hadoop in CentOS.

          In ubuntu i am able to set the Interfaces, hostname and hosts but not able to set the these things in centos.

          Please share the some steps.

          Thanks
          Punit

          Reply
    13. Alex on February 19, 2015 10:31 am

      Hi Rahul,
      nice guide! I have a problem running the step number 4.
      it got this error:
      it is impossible to create .ssh dir
      which permission should have hadoop user? in which groups should be menber?
      thank you in advance for your time

      Alex

      Reply
    14. Karthik on September 1, 2014 10:35 am

      Hi Rahul,
      I had followed your steps for cluster installation and i successfully done it with namenode & one datanode, now i have some small doubts about how to load data into datanode directly from servers and how to get it into HDFS. plz help me with some suggestions.
      And can i install hive on name node by following your HIVE article now

      Reply
    15. Singh on March 13, 2014 8:43 pm

      Hey Rahul,
      That was awesome clear instrauction. I was able to do this thing with little twicks for java. But I would say great article. Thanks

      Reply
    16. Kiran on February 15, 2014 4:30 pm

      Hi Rahul,
      Could u send me the roles & responsibilites of Hadoop Admin and tips for Admin ?
      I would need to put in my CV ..
      Pls kindly help me .

      Reply
    17. Alex on January 21, 2014 2:48 pm

      It is cool, what you wrote, but…. 1. Where I could verify what hadoop working? I couldn’t find any logs on slave. 2. Any examples how it is working? Like put any web sites or what?
      Thanks anyway, it is working on the master.

      Reply
    18. Bill on November 11, 2013 9:05 am

      Thank you Rahul!
      This is excellent! Simple and clear instructions, helped me a lot!

      Reply
    19. savitha on September 14, 2013 5:16 am

      Hi Rahul
      I found this article useful for my work, Thank You. I hve already created a single node cluster and i am planning to do multi node, is it possible to do it in 2 laptop with same configuration and memory. if so wat are the requirements needed for hardware settings like switches,cables.

      Reply
    20. venkatrama rao on August 13, 2013 5:00 pm

      Hi

      Thanks Rahul for your nice and simple artilcle,
      am followed all the steps as mentioned but my data nodes (slvae1 & slave2) are not starting
      it worked in single node installation..
      i dont where i messedup.., passwordless ssh is working fine and when am trying to start the services in slave1 it asking for password of [email protected]: insteadof either [email protected] or [email protected] why pls help me

      Thanks in advance
      Venaktrama rao

      Reply
      • Rahul on August 14, 2013 7:52 am

        Hi Venaktrama,

        You only need to start service at master server ( Step #9 ). and watch the logs on slave servers.

        Reply

    Leave A Reply Cancel Reply

    Advertisement
    Recent Posts
    • 20 Basic Linux Commands for the Beginners (Recommended)
    • tail Command in Linux with Examples
    • What is a Orphan Process in Unix/Linux
    • How To Display Warning Message to Unauthorized SSH Access
    • How to Set a Custom SSH Login Banner and MOTD
    Facebook Twitter Instagram Pinterest
    © 2023 Tecadmin.net. All Rights Reserved | Terms  | Privacy Policy

    Type above and press Enter to search. Press Esc to cancel.