Skip to main content

KlustronDB Shard Fault Isolation Capability

KlustronDBAbout 3 min

KlustronDB Shard Fault Isolation Capability

Note:

Unless otherwise specified, the version numbers in the text can be replaced with the version numbers of any released version. All released versions are detailed here: Release_notes.

Overview

KlustronDB's financial-grade high reliability consists of a complete technical system made up of a series of capabilities and technologies. These capabilities and mechanisms are rock-solid, forming KlustronDB's indestructible financial-grade higher reliability technology system, which of course also includes fault isolation capabilities. This article mainly tests that when a primary-replica of a shard in the cluster completely fails, the KlustronDB cluster can still partially operate normally. The data stored on the failed shard cannot be accessed, but the data on other shards can be read, written, and committed normally.

1. Test Environment

The configuration information of the machines related to the test cluster is as follows:

NameNode TypeIPPortShard_ID
comp3Kluscomp instance192.168.40.15247001N/A
shard_1Primary Klustore instance192.168.40.151570036
Storage Replica Node192.168.40.152
Storage Replica Node192.168.40.153
shard_2Primary Klustore instance192.168.40.152570057
Storage Replica Node192.168.40.153
Storage Replica Node192.168.40.151
shard_3Primary Klustore instance192.168.40.153570075
Storage Replica Node192.168.40.152
Storage Replica Node192.168.40.151

2. Environmental Preparation

Create test database and user

psql -h 192.168.40.152 -p 47001 postgres
create user kunlun_test with password 'kunlun';
create database testdb owner kunlun_test;
grant all privileges on database testdb to kunlun_test;

Prepare to test the table sales_order, which is partitioned by month. The data for the January 2023 partition is placed on shard_3, the February data is placed on shard_1, and the March data is placed on shard_2.

psql -h 192.168.40.152 -p 47001 -U kunlun_test testdb

CREATE TABLE sales_order
(
   order_number         INT NOT NULL,
   customer_number      INT NOT NULL,
   product_code         INT NOT NULL,
   order_date           DATETIME NOT NULL,
   entry_date           DATETIME NOT NULL,
   order_amount         DECIMAL(18,2) NOT NULL,
   PRIMARY KEY (order_number,order_date)
) partition by range(order_date);
CREATE TABLE sales_order_202301 PARTITION OF sales_order
FOR VALUES FROM ('2023-01-01') TO ('2023-02-01') with (Shard=5);
CREATE TABLE sales_order_202302 PARTITION OF sales_order
FOR VALUES FROM ('2023-02-01') TO ('2023-03-01') with (Shard=6);
CREATE TABLE sales_order_202303 PARTITION OF sales_order
FOR VALUES FROM ('2023-03-01') TO ('2023-04-01') with (Shard=7);

And pour 10,000 pieces of data into it

create or replace procedure generate_order_data()
AS $$
DECLARE
  v_customer_number integer;
  v_product_code integer;
  v_order_date date;
  v_amount integer;
  start_date date := to_date('2023-01-01','yyyy-mm-dd');
  i integer :=1;
BEGIN
	while i<=10000 loop
		v_customer_number := FLOOR(1+RANDOM()*6);
		v_product_code := FLOOR(1+RANDOM()*1000);
		v_order_date := start_date + CAST(FLOOR(RANDOM()*90) AS INT);
		v_amount := FLOOR(1000+RANDOM()*9000);
		INSERT INTO sales_order VALUES (i,v_customer_number,v_product_code,v_order_date,v_order_date,v_amount);
		commit;
		i := i+1;
	end loop;
END; $$
LANGUAGE plpgsql;
call generate_order_data();
analyze sales_order;
testdb=> select count(*) from sales_order;
 count 
-------
 10000
(1 row)

testdb=> select count(*) from sales_order where order_date between '2023-01-01' and '2023-01-31';
 count 
-------
  3417
(1 row)

testdb=> select count(*) from sales_order where order_date between '2023-02-01' and '2023-02-28';
 count 
-------
  3086
(1 row)

testdb=> select count(*) from sales_order where order_date between '2023-03-01' and '2023-03-31';
 count 
-------
  3497
(1 row)

3. Simulate the entire shard failure

At this time, it is necessary to simulate all primary and standby nodes of shard_1 failing. Because KlustronDB has a function to automatically restart failed services, it is first necessary to move the data file directory of shard_1 and execute it on both the primary and standby nodes of shard_1.

cd /kunlun/storage_datadir
mv 57003 57003_bak

Then, separately use the kill -9 command on the primary and standby nodes of shard_1 to kill the mysqld and mysqld_safe processes on port 57003.

4. Fault Isolation Test

Returning to the Kluscomp instance to query the data for February 2023 will throw an exception:

testdb=> select count(*) from sales_order where order_date between '2023-02-01' and '2023-02-28';
ERROR:  Kunlun-db: Failed to connect to mysql Klustore instance at (192.168.40.151, 57003): 2002, Can't connect to server on '192.168.40.151' (111)

Querying the data for January and March returns results normally.

testdb=> select count(*) from sales_order where order_date between '2023-01-01' and '2023-01-31';
 count 
-------
  3417
(1 row)

testdb=> select count(*) from sales_order where order_date between '2023-03-01' and '2023-03-31';
 count 
-------
  3497
(1 row)

Similarly, the shards corresponding to the data for January and March can still be inserted normally.

testdb=> begin;
BEGIN
testdb=> select count(*) from sales_order where order_date between '2023-01-01' and '2023-01-31';
 count 
-------
  3417
(1 row)

testdb=> select count(*) from sales_order where order_date between '2023-03-01' and '2023-03-31';
 count 
-------
  3497
(1 row)

testdb=> insert into sales_order values(10001,1,1000,'2023-01-31','2023-01-31',1800);
INSERT 0 1
testdb=> insert into sales_order values(10002,1,1000,'2023-03-31','2023-03-31',2000);
INSERT 0 1
testdb=> select count(*) from sales_order where order_date between '2023-01-01' and '2023-01-31';
 count 
-------
  3418
(1 row)

testdb=> select count(*) from sales_order where order_date between '2023-03-01' and '2023-03-31';
 count 
-------
  3498
(1 row)

testdb=> commit;

But if the transaction involves operating on data from a faulty shard, an error will occur.

testdb=> begin;
BEGIN
testdb=> select count(*) from sales_order where order_date between '2023-03-01' and '2023-03-31';
 count 
-------
  3498
(1 row)

testdb=> update sales_order set order_amount=1000 where order_date='2023-03-31';
UPDATE 113
testdb=> insert into sales_order values(10003,1,1000,'2023-02-14','2023-02-14',1800);
ERROR:  Kunlun-db: Failed to connect to mysql Klustore instance at (192.168.40.151, 57003): 2002, Can't connect to server on '192.168.40.151' (111)
testdb=> insert into sales_order values(10005,2,1000,'2023-03-30','2023-03-30',2000);
ERROR:  current transaction is aborted, commands ignored until end of transaction block
testdb=> rollback;
ROLLBACK

From the above test, it can be seen that the data stored on the faulty shard can no longer provide any service, but the data on other normal shards can still be read, written, and transacted normally.

【END】

Copyright © KlustronDB
Copyright © 2026 KlustronDB