• Home
  • Ubuntu 16.04
    • Whats New ?
    • Install JAVA 8
    • Setup LAMP Stack
    • Install LEMP Stack
    • Install Node.js
    • Install Git
    • Move Unity Launcher to Bottom
  • FeedBack
  • Funny Tools
  • Tutorials
    • Linux Distributions
      • CentOS
      • Debian
      • Fedora
      • LinuxMint
    • Monitoring Tools
      • Monit
      • Nagios
      • NRPE
    • Network Services
      • DHCP
      • DNS
      • FTP
    • Databases
      • MySQL
      • MariaDB
      • MongoDB
      • PostgreSQL
      • SQL Server
    • Amazon Web Services
  • Submit Article
  • About Us
TecAdmin.net
  • Home
  • Ubuntu 16.04
    • Whats New ?
    • Install JAVA 8
    • Setup LAMP Stack
    • Install LEMP Stack
    • Install Node.js
    • Install Git
    • Move Unity Launcher to Bottom
  • FeedBack
  • Funny Tools
  • Tutorials
    • Linux Distributions
      • CentOS
      • Debian
      • Fedora
      • LinuxMint
    • Monitoring Tools
      • Monit
      • Nagios
      • NRPE
    • Network Services
      • DHCP
      • DNS
      • FTP
    • Databases
      • MySQL
      • MariaDB
      • MongoDB
      • PostgreSQL
      • SQL Server
    • Amazon Web Services
  • Submit Article
  • About Us
26 August 2016

Hadoop Commands to Manage Files on HDFS

Written by Rahul K. | August 26, 2016
BIG-DATA DFS, hadoop, hdfs

This tutorial help you to learn to manage our files in HDFS. You will learn how to create, upload, download and list contents in HDFS. Below commands will help you to how to create directory structure in HDFS, Copy files from local file system to HDFS and download files from HDFS to local files. Also how to do manage files in HDFS.

Create Directory in HDFS

Takes the path uri’s as an argument and creates a directory or multiple directories.

$ hdfs dfs -mkdir 

Remember that you must create home directory in HDFS with your system’s username. For example you are logged in as hduser on your system, So first create /user/hduser else you will get this error, Now create directory structure inside it

$ hdfs dfs -mkdir /user/hduser
$ hdfs dfs -mkdir /user/hduser/input
$ hdfs dfs -mkdir /user/hduser/output 
$ hdfs dfs -mkdir /user/hduser/input/text  /user/hadoop/input/xml

Copy Files to HDFS

After creating directory structure, Now put some files to HDFS from your local file system.

$ hdfs dfs -put  ... 

For example you have test1.txt in current directory and /tmp/test2.xml on your local file system.

$ hdfs dfs -put text1.txt /user/hduser/input/text/
$ hdfs dfs -put /tmp/text2.xml /user/hduser/input/xml/

List Files from HDFS

Use the following example commands to list content of directory in HDFS.

$ hdfs dfs -ls /user/hduser
$ hdfs dfs -ls /user/hduser/input/
$ hdfs dfs -ls /user/hduser/input/text/

Use -R to list files recursively inside directories. For example:

$ hdfs dfs -ls -R /user/hadoop/input/

Download Files from HDFS

At this point you have learned how to copy and list files to HDFS. Now use following example commands to how to download/Copy files from HDFS to local file system.

$ hdfs dfs -get /user/hduser/input/text/test1.txt /tmp/
$ hdfs dfs -get /user/hadoop/dir1/xml/test2.xml /tmp/

here /tmp is on system’s local file system.

Copy Files between HDFS Directories

You can easily copy files between HDFS file system using distcp

$ hdfs distcp /user/hduser/input/xml/text2.xml /user/hduser/output
$ hdfs distcp /user/hduser/input/text/text1.xml /user/hduser/output

Share it!
Share on Facebook
Share on Twitter
Share on Google+
Share on Reddit
Share on Tumblr
Rahul K.
Rahul K.
Connect on Facebook Connect on Twitter Connect on Google+

I, Rahul Kumar am the founder and chief editor of TecAdmin.net. I am a Red Hat Certified Engineer (RHCE) and working as an IT professional since 2009..

Related Posts

  • HADOOP/HDFS ls: ‘.’: No such file or directory

  • Hadoop – Namenode is in safe mode

  • Hadoop – Running a Wordcount Mapreduce Example

  • How to Setup Hadoop 2.8 on CentOS, Ubuntu and LinuxMint

  • How to Setup Hadoop 2.6.5 (Single Node Cluster) on Ubuntu, CentOS And Fedora

1 Comment

  1. Vidya Vinodini Reply to Vidya
    August 22, 2017 at 4:53 am

    Hi
    This tutorial is very useful for me..I followed this for my hadoop installation…but while running wordcount example – javac -classpath $(HADOOP_CLASSPATH) -d ‘/home/hduser/Desktop/WordCountTutorial/tutorial_classes’ ‘/home/hduser/Desktop/WordCountTutorial/WordCount.java’ i am facing with the error

    HADOOP_CLASSPATH: command not found
    javac: invalid flag: /home/hduser/Desktop/WordCountTutorial/tutorial_classes
    Usage: javac
    use -help for a list of possible options
    so how can i fix the problem??

    Thankyou

Leave a Reply

Cancel reply

Popular Posts

  • How to Install JAVA 8 on Ubuntu 16.04/14.04, LinuxMint 18/17
  • How to Install s3cmd in Linux and Manage Amazon s3 Buckets
  • How to Install AnyDesk on Ubuntu, Debian and LinuxMint (Alternative of TeamViewer)
  • How to Setup Selenium with ChromeDriver on Ubuntu 16.04
  • How to Install MySQL on macOS Sierra and High Sierra
All rights reserved. © 2013-2018 TecAdmin.net. This site uses cookies. By using this website you agree our term and services