KlustronDB Technology Sharing: Fullsync Mechanism of storage shards
KlustronDB Technology Sharing: Fullsync Mechanism of storage shards
Introduction
As a distributed database that can fully support high-consistency scenarios such as finance and securities, KlustronDB's ability to continuously provide services is particularly important. Fullsync (strong synchronization) is a key feature of KlustronDB to ensure data security. Through master-slave strong synchronization and the high-availability technology built on it (Fullsync HA), it completely guarantees the safety of data in the event of unexpected failures.
This issue's Tech Talk will mainly introduce the Fullsync feature and the HA technology based on it, aiming to help everyone gain an understanding of this key feature while also having a deeper understanding of Klustron.
Key Quote of this Article: The Fullsync mechanism is a high-availability mechanism of KlustronDB's storage shards. It is used to ensure that when a storage shard encounters problems, a new master node can be elected as needed, ensuring that the cluster can continuously provide services, thereby achieving high availability.
01 Introduction to KlustronDB:
First, let's briefly introduce our Zetu Technology's distributed database product, Klustron. The diagram below shows the overall architecture of Klustron. From the diagram, it can be seen that KlustronDB is a distributed database product with separation of storage and computing.
As a distributed database product, KlustronDB possesses the following powerful capabilities:
Elastic computing and storage capabilities
- Data partition (partition): hash, range, list
- Partition columns of any number and type
Data distribution
- auto, random, mirror, table grouping
Automatic, flexible, non-stop service, non-intrusive to business, imperceptible to end users
Financial-grade high reliability
- Automatically handle hardware, software, and network failures, as well as overall data center failures
- Data is neither lost nor disordered, and services remain continuously online
Ensure RTO < 30 seconds & RPO = 0
Automatically detect master node failures and elect a new master, switching between master and standby
HTAP: OLTP & OLAP do not interfere with each other
- Primarily OLTP: Equivalent to using MySQL or PostgreSQL for application software
OLAP as auxiliary: multi-level parallel queries achieve high performance
Elastic Computing of Multilingual Stored Procedures: ML, Privacy Computing
Ecological compatibility
- Supports both PostgreSQL and MySQL connection protocols and SQL syntax
Supports commonly used MySQL DDL syntax
Supports JDBC, ODBC, and PostgreSQL and MySQL client connectors for common programming languages
Comprehensive multi-level security
- Encrypted storage and transmission
Multi-level access control mechanism
02 WHY
Why is Fullsync needed? It can be mainly viewed from two aspects:
Business Requirements:
Possible malfunction
- Main node hardware failure, power outage, OS reboot, accidental process termination;
The primary and standby node disks have bad sectors, and the disks are completely destroyed.
Network partition, disconnection, congestion
The requirements for financial-grade high reliability
- In case of failure, the database cluster can read and write (automatically detects the status of the primary node, and automatically elects a new primary and switches between primary and standby when a primary node failure is detected)
No data loss in case of failure
High performance, low latency
The cluster operates continuously for a long time, self-maintaining
Known issues with the solution:
Semisync
- After the master node fails, it cannot rejoin the cluster --- the number of cluster nodes cannot be quickly restored
The standby machine automatically degrades to asynchronous mode if ACK times out, and consistency < availability, making it impossible to achieve financial-grade strong consistency.
Before the standby server ACKs, if the relay log is not fsynced, a standby server OS reboot (for example, power outage in the data center/rack/server) can cause the confirmed transactions' binlogs to be lost.
Occupies worker threads waiting for the standby machine's ACK, requiring a large number of threads when there are many connections
Incomplete, requires external components to achieve high availability, cannot detect master node failures, perform master selection, or master-slave switching
MySQL Group Replication
- After the standby confirms receipt of the binlog, the primary node writes the binlog to the binlog file. The time spent holding the lock before the transaction commits is very long, significantly increasing the latency of this transaction.
Occupies worker threads waiting for standby ACK, requires starting a large number of threads when there are many connections
03 Fullsync & HA
The Principle of Fullsync:
After a transaction is completed and committed on the primary node, it waits for the standby to receive its binlog ACK before returning to the client (Kluscomp instance)
- Any transaction committed on the primary node has at least one replica (configurable) that receives its complete binlog and persists it.
Timing for waiting for ACK: The submission pipeline has been completed on the master node before returning the status to the client after the transaction completion submission process.
The standby server collects the binlogs of multiple transactions, writes them to the relay log file in one go, flushes to disk (fsync), and then sends an ACK.
Statements that need to wait for fullsync ACK
- All statements that commit binlog transactions
DDL statements, autocommit DML statements
- commit，XA COMMIT ... ONE PHASE
- XA PREPARE
The main process is shown in the figure below:
Additionally, there are some technical details:
Details of transactions on the primary node waiting for Fullsync ACK
- Content to wait for: This transaction has been received and flushed to disk by enough standby machines, and the ACK binlog position >= the binlog position of this transaction
Waiting method: Do not block the working thread; suspend the session (THD) until the ACK arrives, and the working thread continues to handle requests from other connections.
Handling after waiting for ACK timeout: Fullsync HA is more robust, reliable, and flexible, that is: it considers occasional random network or IO jitters, and supports prioritizing either Consistency or Availability.
Handling of received ACK: A background worker thread performs the remaining operations in the transaction commit process of the target session (THD), sends an OK packet to the client, and the client's statement returns.
Handling ACK timeout: return a 'fullsync wait timeout' error to the client in the session
The processing logic for the standby machine receiving binlog
- Striking a balance between disk load, performance, and data consistency
Use transactions as the unit to write & fsync the binlog. Fsync: the binlog remains persistently saved after a power outage or OS restart.
ACK: A position on the master node's binlog. The replica sends an ACK: the binlogs before this position have been persistently saved, and the transactions they belong to can return success.
Methods of sending ACK: SQL statements or extended client API (send commands)
- SLAVE server_id CONSISTENT TO file_index offset (this statement is not recorded by default in the general log)
- mysql_send_binlog_ack() (COM_BINLOG_ACK) (faster, but requires KlustronDB's MySQL client library)
Fullsync Performance:
Based on our test results, as shown in the figure below, the sysbench write test on kunlun-storage fullsync is 10 times that of the community edition MySQL MGR, and 30%-100% higher than the community edition MySQL semisync.
This mainly comes from two aspects:
The speed of transaction submission was accelerated through batch processing.
Through asynchronous processing, the main node's working thread does not need to wait for the backup node's ACK, which accelerates the main node's transaction processing efficiency.
Fullsync HA：
Fullsync HA, built on the basis of Fullsync, actively detects node failures through a heartbeat detection mechanism and performs re-election of the master and disaster recovery switching according to the situation.
The fault detection mechanism is shown in the figure below. It detects whether a node is operating normally by periodically sending heartbeat messages, and if an anomaly occurs, it actively initiates a disaster recovery switch.
The disaster recovery switch process is shown in the diagram below. When an abnormal situation occurs, the problematic primary node is first downgraded, then all its binlogs are reported and synchronized to the secondary nodes. After synchronization is completed, a new primary node is selected from the secondary nodes and configured, and finally, a new synchronization relationship is established.
04 Q&A
q1: Why is Fullsync faster than semisync? What optimizations have you done? How much faster is the performance? Why hasn't the official version implemented these optimizations?
a1: As mentioned earlier, the improvement in KlustronDB's strong synchronization performance mainly comes from two aspects:
- The speed of transaction submission was accelerated through batch processing.
- Through asynchronous processing, the main node's worker thread does not need to wait for the backup node's ACK, which accelerates the main node's transaction processing efficiency.
Our test results show that sysbench write tests with kunlun-storage fullsync are 10 times that of the community version of MySQL MGR, and 30%-100% higher than the community version of MySQL semisync.
As for why the official side hasn't made these optimizations, I think they may have their own considerations. For example, to maintain the differentiation between the enterprise and community editions, or to allocate resources to higher-priority projects, or perhaps to support the open-source ecosystem.
q2: Where can I try out KlustronDB?
a2: Friends who are interested in KlustronDB can download a trial from our official website and deploy it according to the installation documentation. In addition, we also offer KlustronDB's serverless service on Amazon Marketplace and Alibaba Cloud, which everyone can try if interested.
Welcome everyone to download and install the KlustronDB database cluster and use it for free (no registration code required)
KlustronDB full software package download:
http://downloads.klustron.com/
If you need to make a purchase, please contact sales_vip@klustrondb.com by email. For any related questions, you are welcome to add the assistant on WeChat below 🌹