KlustronDB Cluster Deployment in Practice

01 Architecture Introduction and Terminology Explanation

1.2 Component Introduction

KlustronDB Cluster Component Introduction:

Component Name	Function
KlustronDB_server	Computing node responsible for receiving requests from application clients, performing query parsing, optimization, execution, and merging intermediate results returned by storage nodes, and finally returning results to the client. Computing nodes do not store user data but store users' metadata. Typically, multiple computing nodes of a cluster are configured behind a load balancer. If a computing node fails or exits, the load balancing layer will forward requests to other healthy computing nodes to continue servicing requests from application clients.
KlustronDB_storage	storage shard responsible for implementing data persistence and financial-grade data consistency. The storage shard consists of 2 or more shards, each shard having 3 replicas: 1 master and 2 slaves. The master node receives read and write requests from computing nodes, executes the requests, and returns results to the computing nodes.
metashard	Responsible for storing the metadata of the KlustronDB cluster, including topology, node connection information, DDL logs, commit logs, and other cluster management logs.
cluster_mgr	Responsible for maintaining the correct state of the cluster and nodes, implementing cluster management, logical backup and recovery of the cluster, physical backup and recovery of the cluster, horizontal elastic scaling, and other functions.
XPanel	GUI software tool providing a web-based cluster management platform, allowing DBAs to easily complete the majority of database operations and management tasks with just a few clicks.
node_mgr	node_mgr is responsible for executing the commands issued by cluster_mgr on the corresponding machine nodes and periodically reporting heartbeat information to the cluster master node. Each machine node has a node_mgr instance.

1.3 Terminology Explanation

Introduction to KlustronDB Terminology:

Terms	Explanation
Cluster/Cluster	In a KlustronDB, a cluster consists of several compute nodes and several storage shard shards. For example, cluster A consists of three compute nodes and two storage shard shards.
Compute node/KlustronDB_server /CN(Computing Node)	Receives requests from application clients, merges the results returned by the storage nodes, and finally returns the client results. Each compute node is stateless and independent. The cluster consists of multiple compute nodes and is fronted by a load balancer. When a single compute node fails, it does not affect the cluster, which continues to provide continuous read and write services.
storage shard/KlustronDB_storage /Shard	A KlustronDB cluster contains one or more shards. Each shard stores a part of user data. User data stored in different shards does not have intersection. Each shard has one primary node and multiple secondary nodes. High availability is achieved through binlog-based replication between the primary and secondary nodes.
Storage Node	Used to store user business data, it is a running instance of the kunlun-storage component. Several storage nodes form a shard and operate within the shard as either the primary node or a replica node.
metashard/metashard	A replication group composed of multiple storage nodes. It is used to manage the metadata of the KlustronDB cluster and is not used to store user business data. The metashard is a replication group at runtime. Multiple KlustronDB clusters can be served.
cluster_mgr/Cluster Manager/Cluster Manager	This node is responsible for receiving KlustronDB cluster operation and management requests from clients and sending actual actions to each node manager to complete the requests.
XPanel	A web application running in a Docker image. Users can connect to XPanel via a browser to perform cluster management operations.
node_mgr/Node Manager/Node Manager	Each working machine needs to deploy a local command execution node. This node will receive requests from the cluster manager and perform various operations on storage nodes, compute nodes, and other components KlustronDB on the machine, including but not limited to installation, deletion, stop, backup, etc.

02 Standard Cluster Deployment

2.1 Prerequisites for Deployment

2.1.1 Operating System and CPU Architecture

Operating System Version	CPU Architecture (x86_64/AMD64/ARM64)
CentOS 7.5 and above	x86_64 ARM64
CentOS 8	x86_64 ARM64
UnionTech Server OS V20 (1050a)	AMD64 ARM64
UnionTech Server OS V20 (1050e)	AMD64 ARM64
openSUSE 15	x86_64
Ubuntu 20.04	x86_64
Kylin v10	x86_64 ARM64

2.1.2 Server Configuration

Test and Development Environment Configuration Recommendations

For development and testing environments, it is recommended to use at least three machines, with multiple types of components deployed on each machine. The specific requirements for each type of component are as follows:

Component	CPU	Memory	Disk Type and Available Space	Network	Number of Deployments	Deployment Description
KlustronDB_server	4	8G	SAS, 20G	Gigabit NIC	1	Can be deployed together with the storage shard
KlustronDB_storage	4	8G	SAS, 20G	Gigabit NIC	1	A storage shard must contain at least 3 nodes, i.e., one primary and two backups
metashard	4	8G	SAS, 20G	Gigabit NIC	1	Can be deployed together with the storage shard
cluster_mgr	2	4G	SAS, 20G	Gigabit NIC	1	Can be deployed together with the storage shard
XPanel	2	4G	SAS, 10G	Gigabit NIC	1	Can be deployed together with the storage shard
node_mgr	1	2G	SAS, 10G	Gigabit NIC	1	Automatically deployed on each machine

Production Environment Configuration Requirements

For a production environment, at least 6 machines are required; when machine resources are sufficient, it is recommended to use 9 machines, with 3 computing nodes and 6 storage nodes. Other components can be deployed together with the storage nodes; the following are the specific requirements for each component:

Component	CPU	Memory	Disk Type and Available Space	Network	Number of Deployments	Deployment Notes
KlustronDB_server	16	32G	SSD, 20G	10Gb NIC	3	Independent deployment
KlustronDB_storage	8	32G	SSD, 200G	10Gb NIC	3	A storage shard must contain at least 3 nodes, i.e., one primary and two backups
metashard	4	8G	SSD, 50G	10Gb NIC	1	Can be deployed together with the storage shard
cluster_mgr	4	8G	SSD, 20G	10Gb NIC	1	Can be deployed together with the storage shard
XPanel	4	8G	SSD, 10G	10Gb NIC	1	Can be deployed together with compute nodes
node_mgr	1	2G	SSD, 10G	10Gb NIC	1	Automatically deployed on each machine

2.1.3 Network Configuration

Component	Default Port/Port Range	Port Description
KlustronDB_server	47000-48000	PG or mysql Client Connection Access Communication Port Range Provided by Computing Node
metashard	6001	metashard Access Communication Port
cluster_mgr	58000	cluster_mgr Raft Protocol port
cluster_mgr	58001	cluster_mgr http protocol port
cluster_mgr	59010	cluster_mgr default prometheus port
XPanel	18080	XPanel management platform service access port
node_mgr	58002	node_mgr default http port
node_mgr	58003	node_mgr default tcp port
node_mgr	58010	node_mgr default prometheus port

2.2 Online Deployment

2.2.1 Pre-deployment Preparations - Operations on All Machines

The preparation for the steps needs to be carried out on each machine.

Clean up the MariaDB that comes pre-installed with the system

Run the following command as the root user:

yum remove postfix
rpm -e mariadb-libs

Turn off the firewall

Execute the command as the root user:

systemctl stop firewalld
systemctl disable firewalld

Create kunlun group and kunlun user

Execute the command as root and set the user password:

useradd --create-home --shell /bin/bash kunlun
passwd kunlun

Create the relevant directory and set the owning group

Create the software installation directory /data1/kunlun, which can be created according to user-defined specifications. Here, we take the example of the disk mounted to the /data1 directory, and set the directory's group ownership to kunlun:kunlun.

Run the following command as root:

mkdir -p /data1/kunlun /data1/softwares

# /data1/kunlun为各组件安装完成后的目录，/data1/softwares为部署前所存放安装包、相关脚本和文件的目录

chown -R kunlun:kunlun /data1/kunlun /data1/softwares

Modify system file handle count

Execute the following command as root:

vim /etc/security/limits.conf

#Add the following below the last line of the file:

* soft nofile 65536
* hard nofile 200000

Set the kunlun user to execute root commands with sudo without a password

Execute the following command as root:

visudo
# 在最后加上下面的内容：
kunlun ALL=(ALL) NOPASSWD: ALL

2.2.2 Pre-Deployment Preparations - Central Control Computer Operations

The preparation for this step only needs to be carried out on the central control machine.

Configuring SSH Mutual Trust Between the Central Control Machine and Other Machines

Execute the following command on the central control machine as the kunlun user:

ssh-keygen -t rsa

After running this command, you can simply press Enter wherever input is required. Upon success, two files, id_rsa and id_rsa.pub, will be generated in the .ssh directory under /home/kunlun/, where id_rsa is the file storing the private key, and id_rsa.pub is the file storing the public key.

Copy the public key of the central control machine to other servers, including the central control machine itself:

ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.151
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.152
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.153

Verify the execution of commands on the central control machine to check the password-free login of the kunlun user to other machines:

ssh 192.168.40.151 date
ssh 192.168.40.152 date
ssh 192.168.40.153 date

Install tools such as python2, git, and wget

Execute the following command as root:

yum -y install python git wget yum-utils

Deploy XPanel

Since there are plans to deploy XPanel on the central control, Docker needs to be installed and deployed in advance.

Execute the following command as root:

Remove the system's built-in docker-related RPMs:

yum remove docker \
        docker-client \
        docker-client-latest \
        docker-common \
        docker-latest \
        docker-latest-logrotate \
        docker-logrotate \
        docker-engine

Switch to Aliyun mirror:

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

Install Docker Community Edition:

yum -y install docker-ce docker-ce-cli containerd.io

Start Docker and set it to start automatically with the operating system:

systemctl start docker
systemctl enable docker

Download the latest deployment tool

Download the deployment tool to the /data1/softwares directory and execute it as the root user:

cd /data1/softwares
git clone -b 1.1 https://gitee.com/zettadb/cloudnative.git

Download and install the required software package

Execute the following command as root:

cd /data1/softwares/setuptools-36.5.0/cloudnative/cluster/clustermgr
wget http://klustron.cn:14000/thirdparty/hadoop-3.3.1.tar.gz
wget http://klustron.cn:14000/thirdparty/jdk-8u131-linux-x64.tar.gz
wget http://klustron.cn:14000/thirdparty/mysql-connector-python-2.1.3.tar.gz
wget http://klustron.cn:14000/thirdparty/prometheus.tgz
wget http://klustron.cn:14000/thirdparty/haproxy-2.5.0-bin.tar.gz
wget http://klustron.cn:14000/thirdparty/efk/filebeat-7.10.1-linux-x86_64.tar.gz


VERSION=1.1.2
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-cluster-manager-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-node-manager-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-server-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-storage-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-proxysql-$VERSION.tgz

chown -R kunlun:kunlun /data1/softwares/

2.2.3 Deployment Installation

Edit deployment topology JSON file

On the central control machine, the Kunlun user modifies the file /data1/softwares/setuptools-36.5.0/cloudnative/cluster/cluster_and_node_mgr.json to add relevant configuration information. Here, we take three machines as an example: 192.168.40.151 (central control machine), 192.168.40.152, and 192.168.40.153.

{
    "machines":[
        {
            "ip":"192.168.40.151",
            "basedir":"/data1/kunlun",
            "user":"kunlun"
        },
        {
            "ip":"192.168.40.152",
            "basedir":"/data1/kunlun",
            "user":"kunlun"
        },
        {
            "ip":"192.168.40.153",
            "basedir":"/data1/kunlun",
            "user":"kunlun"
        }
    ],
    "meta":{
        "ha_mode": "rbr",
        "nodes":[
         {
         "ip":"192.168.40.151"
                },
                {
         "ip":"192.168.40.152"
                },
                {
         "ip":"192.168.40.153"
                }
        ]
    },
    "cluster_manager": {
           "nodes": [
        {
         "ip": "192.168.40.151"
        },
        {
         "ip": "192.168.40.152"
        },
        {
         "ip": "192.168.40.153"
        }
        ]    
    },
    "node_manager": {
           "nodes": [
        {
         "ip": "192.168.40.151"
        },
        {
         "ip": "192.168.40.152"
        },
        { 
         "ip": "192.168.40.153"
        }
        ]    
    },
    "xpanel": {
    "ip": "192.168.40.151",
    "image": "registry.cn-hangzhou.aliyuncs.com/kunlundb/kunlun-xpanel:1.1.2"
    }
}

Execute Installation

Generate installation script

The central control machine uses the kunlun user to execute the following command:

cd /data1/softwares/setuptools-36.5.0/cloudnative/cluster
sudo python2 setup_cluster_manager.py --autostart --config=cluster_and_node_mgr.json --product_version=$version --action=install
# $version是指需要安装的集群版本号，例如：1.1.2

Example:

sudo python2 setup_cluster_manager.py --autostart --config=cluster_and_node_mgr.json --product_version=1.1.2 --action=install

Run the installation script

The central control machine uses the kunlun user to execute the following command:

bash -e /data1/softwares/setuptools-36.5.0/cloudnative/cluster/clustermgr/install.sh

Log in to XPanel to create a cluster

Log in to the XPanel console on the browser. On a machine that can access 192.168.40.151, open the browser and enter the address: http://192.168.40.151:18080/KunlunXPanel

The initial login username and password are: super_dba/super_dba. The super_dba password needs to be changed on the first login.

After logging in, the homepage is displayed as follows:

Click on 'Computer Management' and 'Computer List' in the left control panel to display the interface below.

Check that all machines are online:

Increase the cluster.

Click 'Cluster Management', 'Cluster List', and then click the 'Add' button

Add cluster parameters as shown in the figure below:

After completing the cluster installation, you can use KlustronDB Cluster by connecting to the IP and port of the compute node. The default username and password are both 'abc'. After logging in, users can create accounts and change passwords according to their needs.

pg and MySQL protocol login authentication

Tips: The port numbers for logging in using the PostgreSQL protocol and the MySQL protocol are different. For details, see 【Cluster List】 -> 【Settings】 -> 【Compute Node List】 in XPanel.

PostgreSQL login authentication:

psql -h192.168.0.150 -p23001 -Uabc postgres

As shown in the picture:

MySQL login authentication:

mysql -uabc -p -h192.168.0.150 -P23002 postgres

As shown in the picture:

2.3 Offline Deployment

2.3.1 Pre-deployment Preparations - Operations on All Machines

The preparatory work for this step needs to be carried out on each machine.

Clean up the MariaDB that comes pre-installed with the system

Run the following command as the root user:

yum remove postfix
rpm -e mariadb-libs

Turn off the firewall

Execute the command as the root user:

systemctl stop firewalld
systemctl disable firewalld

Create kunlun group and kunlun user

Execute the command as root and set the user password:

useradd --create-home --shell /bin/bash kunlun

passwd kunlun

Create the relevant directory and set the owning group

Create the software installation directory /data1/kunlun, which can be created according to user-defined specifications. Here, we take the example of the disk being mounted to the /data1 directory, and set the directory's group ownership to kunlun:kunlun.

Run the following command as root:

mkdir -p /data1/kunlun /data1/softwares
# /data1/kunlun为各组件安装完成后的目录，/data1/softwares为部署前所存放安装包、相关脚本和文件的目录**
chown -R kunlun:kunlun /data1/kunlun /data1/softwares

Modify system file handle count

Execute the following command as root:

vim /etc/security/limits.conf

#Add the following below the last line of the file:

* soft nofile 65536

* hard nofile 200000

Set the kunlun user to execute root commands with sudo without a password

Execute the following command as root:

visudo
#在最后加上下面的内容：
kunlun ALL=(ALL) NOPASSWD: ALL

2.3.2 Pre-Deployment Preparations - Central Control Machine Operations

The preparation for this step only needs to be carried out on the central control machine.

Configuring SSH Mutual Trust Between the Central Control Machine and Other Machines

Execute the following command on the central control machine as the kunlun user:

ssh-keygen -t rsa

After running this command, you can simply press Enter wherever input is required. Upon success, two files, id_rsa and id_rsa.pub, will be generated in the .ssh directory under /home/kunlun/, where id_rsa is the file that stores the private key, and id_rsa.pub is the file that stores the public key.

Copy the public key of the central control machine to other servers, including the central control machine itself:

ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.151
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.152
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.153

Verify the execution of commands on the central control machine to check the password-free login of the kunlun user to other machines:

ssh 192.168.40.151 date
ssh 192.168.40.152 date
ssh 192.168.40.153 date

Install tools such as python2, git, and wget

Execute the following command as root:

yum -y install python git wget yum-utils

Install docker service

Since there are plans to deploy XPanel on the central control, Docker needs to be installed and deployed in advance.

Execute the following command as root:

Remove the system's built-in docker-related RPMs:

yum remove docker \
        docker-client \
        docker-client-latest \
        docker-common \
        docker-latest \
        docker-latest-logrotate \
        docker-logrotate \
        docker-engine

Install Docker Community Edition:

yum -y install docker

Start Docker and set it to start automatically with the operating system:

systemctl start docker
systemctl enable docker

Upload the installation tool and installation package to the server

Download and package the latest deployment tools and installation packages through the online environment, or download the corresponding offline deployment package after consulting the KlustronDB team

Download online and package (here we take version v1.2.2 as an example, the actual version to download should be based on the real one):

Download the deployment tool to the /data1/softwares directory and execute it as the root user:

cd /data1/softwares
git clone -b 1.1 https://gitee.com/zettadb/cloudnative.git

Download and install the required software package

Execute the following command as root:

cd /data1/softwares/setuptools-36.5.0/cloudnative/cluster/clustermgr
wget http://klustron.cn:14000/thirdparty/hadoop-3.3.1.tar.gz
wget http://klustron.cn:14000/thirdparty/jdk-8u131-linux-x64.tar.gz
wget http://klustron.cn:14000/thirdparty/mysql-connector-python-2.1.3.tar.gz
wget http://klustron.cn:14000/thirdparty/prometheus.tgz
wget http://klustron.cn:14000/thirdparty/haproxy-2.5.0-bin.tar.gz
wget http://klustron.cn:14000/thirdparty/efk/filebeat-7.10.1-linux-x86_64.tar.gz


VERSION=1.2.2
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-cluster-manager-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-node-manager-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-server-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-storage-$VERSION.tgz
wget http://downloads.klustron.com//releases/$VERSION/release-binaries/kunlun-proxysql-$VERSION.tgz

chown -R kunlun:kunlun /data1/softwares/

2.3.3 Deployment Installation

Edit Deployment Topology JSON File

The difference from online deployment lies in the XPanel part; offline deployment does not have the "image" attribute, only the "imageType": "file" attribute.

{
    "machines":[
        {
            "ip":"192.168.40.151",
            "basedir":"/data1/kunlun",
            "user":"kunlun"
        },
        {
            "ip":"192.168.40.152",
            "basedir":"/data1/kunlun",
            "user":"kunlun"
        },
        {
            "ip":"192.168.40.153",
            "basedir":"/data1/kunlun",
            "user":"kunlun"
        }
    ],
    "meta":{
        "ha_mode": "rbr",
        "nodes":[
         {
         "ip":"192.168.40.151"
                },
                {
         "ip":"192.168.40.152"
                },
                {
         "ip":"192.168.40.153"
                }
        ]
    },
    "cluster_manager": {
           "nodes": [
        {
         "ip": "192.168.40.151"
        },
        {
         "ip": "192.168.40.152"
        },
        {
         "ip": "192.168.40.153"
        }
        ]    
    },
    "node_manager": {
           "nodes": [
        {
         "ip": "192.168.40.151"
        },
        {
         "ip": "192.168.40.152"
        },
        { 
         "ip": "192.168.40.153"
        }
        ]    
    },
    "xpanel": {
    "ip": "192.168.40.151",
    "imageType": "file"
    }
}

Execute Installation

Generate installation script

The central control machine uses the kunlun user to execute the following command:

cd /data1/softwares/setuptools-36.5.0/cloudnative/cluster
sudo python2 setup_cluster_manager.py --autostart --config=cluster_and_node_mgr.json --product_version=$version --action=install
# $version 是指需要安装的集群版本号，例如：1.2.2

Example:

sudo python2 setup_cluster_manager.py --autostart --config=cluster_and_node_mgr.json --product_version=1.2.2 --action=install

Run the installation script

The central control machine uses the kunlun user to execute the following command:

bash -e /data1/softwares/setuptools-36.5.0/cloudnative/cluster/clustermgr/install.sh

Log in to XPanel to create a cluster

Log in to the XPanel console on the browser. On a machine that can access 192.168.40.151, open the browser and enter the address: http://192.168.40.151:18080/KunlunXPanel

The initial login username and password are: super_dba/super_dba. The super_dba password needs to be changed upon first login.

After logging in, the homepage is displayed as follows:

Click on 'Computer Management' and 'Computer List' in the left control panel to display the interface below.

Check that all machines are online:

Increase the cluster.

Click 'Cluster Management', 'Cluster List', and then click the 'Add' button

Add cluster parameters as shown in the following figure:

pg and MySQL protocol login authentication

Tips: The port numbers for logging in using the PostgreSQL protocol and the MySQL protocol are different. For details, see 【Cluster List】 -> 【Settings】 -> 【Compute Node List】 in XPanel.

PostgreSQL login authentication:

psql -h192.168.0.150 -p23001 -Uabc postgres

As shown in the picture:

MySQL login authentication:

mysql -uabc -p -h192.168.0.150 -P23002 postgres

As shown in the picture:

2.4 Deployment of Other Components

2.4.1 HDFS Deployment

HDFS Single-Node Deployment

HDFS Cluster Deployment