KlustronDB Online Elastic Scaling Flow Control

Note:

Unless otherwise specified, the version numbers in the text can be replaced with the version numbers of any released version. All released versions are detailed here: Release_notes.

Overview

The KlustronDB database is a clustered architecture, where both compute nodes and storage nodes support on-demand online expansion to meet the growing demands for computing performance and storage space. After expanding storage nodes, there is usually some data redistribution adjustment. At the underlying level, this is the process of exporting data from one shard and importing it into another shard, in other words, a table migration process. During the adjustment process, to avoid causing a significant resource impact on existing business operations, KlustronDB supports traffic control for this migration operation, meaning that the bandwidth used for transmitting data tables between nodes will not be consumed uncontrollably. Database administrators can set traffic control according to the situation. In this article, we will demonstrate this setup process through simulated testing. Moreover, through this demonstration, viewers can see that business operations are not interrupted during table migration, and by adjusting traffic control, we can observe how different traffic control parameter values impact business operations.

01 Environment Preparation

In the test scenarios referred to in this article, four servers are used, three of which are for the KlustronDB database environment, and one is used to run sysbench to simulate online business, as described in detail below:

Name	IP	Remarks
Compute Node	192.168.0.155
Storage Node 1	192.168.0.152
Storage Node 2	192.168.0.153
Simulated Workload	192.168.0.19	Running sysbench

1.1 KlustronDB Installation and Configuration

Excerpt

1.2 KlustronDB Instance Environment Description:

XPanel: http://192.168.0.155:40180/KunlunXPanel/#/cluster

Compute node: 192.168.0.155, port: 47001

Storage Node (shard1): 192.168.0.152, Port: 57003 (Primary)

Storage Node (shard2): 192.168.0.153, Port: 57005 (Primary)

KlustronDB is installed under the kl user

1.3 Test User Creation

Use an SSH client to connect to the computing node, create a test user, and execute the following command:

psql -h 192.168.0.155 -p 47001 -U adc postgres
create user test with password 'test';
grant create on database postgres to test ;
exit

psql -h 192.168.0.155 -p 47001 -U test postgres
create schema test ;

_

1.4 Sysbench Installation and Configuration

Excerpt

Note: By default, sysbench does not support tests for PostgreSQL and needs to be recompiled.

02 Table Online Migration and Flow Control Test

2.1 Sysbench Test Data Creation

Use an SSH client to connect to the simulated business load server and execute the following command:

/usr/local/bin/sysbench ./tests/include/oltp_legacy/oltp.lua --db-driver=pgsql --pgsql-host=192.168.0.155 --pgsql-port=47001 --pgsql-user=test --pgsql-password=test --pgsql-db=postgres --oltp-tables-count=10 --oltp-table-size=500000 --time=600  --report-interval=5 --threads=50 prepare

_

Note: In this test, to simulate a relatively realistic actual business load, the selected sysbench test load mode is OLTP, which involves both read and write operations on the database in the stress test workload.

2.2 Check the distribution of sysbench data tables

After the test data initialization process in the previous step, sysbench has already created 10 tables in Klustron. We can understand the distribution of these 10 tables across the two shards from Xpanel, as follows:

Open XPanel and click 'Scale'

Select the database: postgres, and click 'OK'

In the dialog box that pops up next, we can see that among them: sbtest3, sbtest5, sbtest7, sbtest9 were automatically created in shard_1.

Click on 'shard_2', and you can see that the rest: sbtest1, sbtest2, sbtest4, sbtest6, sbtest8, sbtest10 have been automatically created in the shard_2 shard:

2.3 Start the sysbench workload

Execute the following instructions to send a mixed read/write load to the KlustronDB repository:

/usr/local/bin/sysbench ./tests/include/oltp_legacy/oltp.lua --db-driver=pgsql --pgsql-host=192.168.0.155 --pgsql-port=47001 --pgsql-user=test --pgsql-password=test --pgsql-db=postgres --oltp-tables-count=10 --oltp-table-size=500000 --time=600  --report-interval=5 --threads=50 run

2.4 Table Online Migration Test

After starting sysbench, pay attention to the output information of the tool, as follows:

It can be seen that under the current load of 50 concurrent users, the TPS/QPS we obtained is approximately 600/11500 on average, and the 95th percentile response latency is about 100ms on average.

Connect to the computing node and use the following command to check the server's resource status:

top d 1

It can be observed that the server CPU utilization is around 50%, and there is no obvious IO queuing:

Connect to one of the storage shards, shard_2, and use the following command to check the resource status of this storage server:

top d 1

It can be observed that the idle CPU resources have dropped to around 50%, and IO wait is already relatively high:

Use the following command to observe the IO situation:

iostat -dx -p /dev/sdb 3

It can be observed that IO utilization has reached 100% and IO requests are being queued, indicating that the machine's IO bandwidth is fully utilized.

Using a similar method on storage shard shard_1, a similar situation can also be observed

Go back to Xpanel, select the shard_1 shard's data table: sbtest5, 'Whether to keep the original table', choose: No, target shard, select: shard_2, as shown below:

Click 'OK', the system prompts for confirmation again, click 'OK', as shown below:

After clicking 'OK', KlustronDB begins performing an online table migration on sbtest5, as shown below:

Returning to the sysbench load window, it can be seen that after the online table migration feature started executing, the tps/qps values in the subsequent sysbench output changed, as shown below:

Compared to the relatively normal output lines, during the table migration process, tps/qps dropped by about 48% at the lowest point overall, with no significant change in latency. Additionally, although during the table migration process, a few business errors were captured by sysbench due to instantaneous changes in table positions, they returned to normal after retries, without causing business interruptions, and the sysbench load continued to run normally. In this output, it can be seen that after the table migration was completed, tps/qps quickly returned to normal, and XPanel also provided notification of the table migration completion, as shown below:

2.5 Table Migration Flow Control Test

Open an SSH window at 192.168.0.153. Let's start monitoring the network traffic by executing the following command:

nload enp0s3 -u m

The window displays as follows:

Then, in XPanel, we choose to migrate sbtest7, as follows:

Note: First, we keep the default value of 'File Transfer Speed Limit' at 5120KB/s unchanged, and use a stopwatch to time when starting the migration.

In the SSH window of 192.168.0.153, we can see that the maximum throughput of the network interface is limited to around 5MB/s, proving that the 5120KB/s setting in the table migration parameters is playing a flow control role, as shown below:

When the XPanel page indicates that the table migration was successful, our stopwatch shows that this migration took about 24 seconds.

Returning to XPanel again, this time we choose to migrate sbtest3, as follows:

Note: This time we set the value of 'file transfer speed limit' to 1024KB/s, and use a stopwatch to time when starting the migration.

In the SSH window of 192.168.0.153, we can see that the maximum throughput of the network interface is limited to around 1MB/s, proving that the 1024KB/s setting in the migration parameters has played a role in flow control, as shown below:

在XPanel 页面提示表迁移成功时，我们的秒表显示，此次表迁移耗时60秒，以上测试证明，表迁移设置参数中的流控参数精确限定了相应任务的网络带宽。

END