Deploy HDFS single node as KlustronDB backup storage

HDFS single-node deployment mode is only used for internal user testing, allowing HDFS to be installed quickly and conveniently. When deploying HDFS in a production system, it is necessary to deploy and configure an HDFS High Availability (HA) cluster, see this document for detailed methods

01 Install Java Environment

The installation process is omitted.

Verification Method:

[root@kunlun ~]# java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
[root@kunlun ~]# 

02 Overall Directory Structure

2.1. Upload hadoop-2.7.3.tar.gz to the /home/kunlun directory on the server and extract it

[root@master kunlun]# tar -zxvf hadoop-2.7.3.tar.gz
[root@master kunlun]# mv hadoop-2.7.3 hadoop
[root@master kunlun]# chown -R kunlun:kunlun /home/kunlun/hadoop
[root@master kunlun]# ll
drwxr-xr-x  9 kunlun kunlun       149 Sep 6 2022 hadoop

03 Configuring Environment Variables

[root@kunlun hadoop]# cat  ~/.bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

export HADOOP_HOME=/home/kunlun/hadoop
export HADOOP_CONF_DIR=/home/kunlun/hadoop/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin 
[root@kunlun hadoop]#source  ~/.bashrc
[root@kunlun hadoop]# hadoop
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME            run the class named CLASSNAME
 or
  where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar <jar>            run a jar file
                       note: please use "yarn jar" to launch
                             YARN applications, not this command.
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
  credential           interact with credential providers
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.
[root@kunlun hadoop]#

04 Modify the configuration file

[root@kunlun hadoop]# ls
capacity-scheduler.xml  hadoop-env.sh               httpfs-env.sh            kms-env.sh            mapred-env.sh               ssl-server.xml.example
configuration.xsl       hadoop-metrics2.properties  httpfs-log4j.properties  kms-log4j.properties  mapred-queues.xml.template  yarn-env.cmd
container-executor.cfg  hadoop-metrics.properties   httpfs-signature.secret  kms-site.xml          mapred-site.xml.template    yarn-env.sh
core-site.xml           hadoop-policy.xml           httpfs-site.xml          log4j.properties      slaves                      yarn-site.xml
hadoop-env.cmd          hdfs-site.xml               kms-acls.xml             mapred-env.cmd        ssl-client.xml.example
[root@kunlun hadoop]# pwd
/home/kunlun/hadoop/etc/hadoop
[root@kunlun hadoop]#

• core-site.xml

<configuration>
     <property>
         <name>fs.defaultFS</name>
         <value>hdfs://192.168.0.135:9000</value>                               
     </property>
     <!-- 指定hadoop临时目录 -->
     <property>
         <name>hadoop.tmp.dir</name>
         <value>/home/kunlun/hadoop/data/tmp</value>
     </property>
</configuration>

• hdfs-site.xml

<!-- Put site-specific property overrides in this file. -->

<configuration>
      <property>
         <name>dfs.replication</name>
         <value>1</value>
      </property>
</configuration>

• hadoop-env.sh

[root@master hadoop]# pwd
/home/kunlun/hadoop/etc/hadoop
[root@master hadoop]# vim hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/usr/java/latest/  //这里配置JAVA_HOME

05 Formatting

[root@kunlun hadoop]# hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

21/03/04 11:18:08 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.3
STARTUP_MSG:   classpath = /home/kunlun/hadoop/etc/hadoop:/home/kunlun/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/kunlun/hadoop/sh...
...
...
21/03/04 11:18:09 INFO common.Storage: Storage directory /home/kunlun/hadoop/data/tmp/dfs/name has been successfully formatted.
21/03/04 11:18:09 INFO namenode.FSImageFormatProtobuf: Saving image file /home/kunlun/hadoop/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
21/03/04 11:18:09 INFO namenode.FSImageFormatProtobuf: Image file /home/kunlun/hadoop/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 351 bytes saved in 0 seconds.
21/03/04 11:18:09 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
21/03/04 11:18:09 INFO util.ExitUtil: Exiting with status 0
21/03/04 11:18:09 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/

06 Start HDFS via script

[root@kunlun sbin]# ./start-dfs.sh 
Starting namenodes on [192.168.0.135]
192.168.0.135: starting namenode, logging to /home/kunlun/hadoop/logs/hadoop-root-namenode-kunlun.out
localhost: starting datanode, logging to /home/kunlun/hadoop/logs/hadoop-root-datanode-kunlun.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/kunlun/hadoop/logs/hadoop-root-secondarynamenode-kunlun.out
[root@kunlun sbin]# jps
3409 NameNode
3973 DataNode
4725 SecondaryNameNode
9150 Jps
[root@kunlun hadoop]# netstat -antlp | grep 50070
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      3409/java           
[root@kunlun bin]# ./hdfs dfs -mkdir /user
[root@kunlun bin]# ./hdfs dfs -mkdir /user/dream
[root@kunlun bin]# hadoop dfs -ls /
hadoop fs -ls /
Found 2 items
drwxr-xr-x   - summerxwu supergroup          0 2022-04-15 14:42 /kunlun
-rw-r--r--   1 summerxwu supergroup 1367639799 2022-07-10 11:20 /xx.tgz

07 Browser Verification

END