Preface
KlustronDB has a complete disaster recovery and error handling mechanism. Through the two-phase commit algorithm of distributed transactions, as well as the Fullsync and Fullsync HA mechanisms, it can ensure that during cluster operation, if any Kluscomp instance, Klustore instance, cluster_mgr, or other components experience crashes, reboots, or other failures, or in the event of network partition or disconnection, the user data and metadata managed by the cluster will remain consistent and complete. No data updates from user-submitted transactions will be lost, and there will be no situations such as partial transaction commits or partial rollbacks, nor inconsistencies between user metadata and user data.
KlustronDB implements reliable distributed transaction processing based on the two-phase commit protocol, ensuring that cluster data remains completely consistent even when multiple nodes in the cluster fail, and guaranteeing the ACID properties of all transactions.
For detailed information on this, please see here (Two-Phase Commit Mechanism and Principle in Distributed Transaction Processing) and here (Error Handling for Two-Phase Commit in Distributed Transactions), and the Fullsync mechanism of KlustronDB_storage has been introduced here (Kunston Distributed Database storage shard Fullsync Mechanism).
This article provides a detailed introduction to the Fullsync high availability mechanism, hereinafter referred to as Fullsync HA.
The Fullsync mechanism of KlustronDB ensures that a storage shard of KlustronDB_storage must receive ACK confirmations from fullsync_consistency_level replicas that they have received the binlog of the transaction before committing any transaction and returning a successful confirmation to the client.
The Kluscomp instance will only return to the KlustronDB client application after receiving a commit success confirmation from the Klustore instance, and therefore it has the obligation to ensure the durability of these transactions; otherwise, it has no such obligation.
If the primary node and up to fullsync_consistency_level-1 secondary nodes fail at the same time, there will still be 1 secondary node (when fullsync_consistency_level>1) that contains the binlog of all committed transactions, so the updates of these transactions will not be lost.
KlustronDB's Fullsync HA ensures that if any KlustronDB_storage storage shard experiences a master node failure or network partition, these errors can be detected in a timely manner and a new master node can be elected in time to continue handling the write load for this storage shard, as detailed below.
The Fullsync and Fullsync HA mechanisms of KlustronDB ensure that as long as a KlustronDB_storage shard with 2*N - 1 nodes has N - 1 nodes still running, this shard can continue to accept writes.
The N here refers to the fullsync_consistency_level of KlustronDB_storage.
The fullsync and fullsync HA mechanism of KlustronDB implement a high-availability mechanism equivalent to the Raft protocol, ensuring that the primary node of each storage shard in a KlustronDB cluster can have one or more backup nodes synchronizing data with the primary node.
For one or more fullsync storage shards with fullsync_consistency_level=N (N > 1), even if the primary node and N-1 replica nodes of these storage shards simultaneously experience any failures or network disconnections and isolation issues, KlustronDB can ensure that the data of these shards is not lost and remains consistent with the data of other storage shards in the cluster. Moreover, KlustronDB will automatically select a new primary node and provide read and write capabilities.
Master Node Liveness Detection
KlustronDB's Fullsync HA ensures that after the primary node crashes, restarts, or a network partition occurs, it can automatically initiate the leader election process, complete the primary-backup switch, and ensure the storage shard remains continuously available.
In order to confirm the availability of the primary node of each storage shard, the cluster_mgr module continuously writes heartbeats to the primary node of each storage shard at intervals of N seconds to detect the availability of its primary node.
If it is found that the primary node M0 of a certain storage shard (marked as SS1) cannot write for a certain period of time, the master selection and primary-backup switching process described below will be initiated.
If M0 restarts quickly enough, it will not trigger a master-slave switch. The cluster_mgr will set it to writable, allowing M0 to continue serving as the master node; otherwise, the cluster_mgr will initiate the following master selection and switching process.
Master Node Election and Switch
The full-sync HA master node election and switching process of KlustronDB mainly includes the following steps:
- In all standby servers of SS1, find the standby server M1 that contains the latest relay log as the candidate primary node.
If there are multiple latest backup machines, the most suitable backup machine will be selected as M1 according to more detailed rules.
KlustronDB's Fullsync mechanism ensures that the cluster has one or more (fullsync_consistency_level) standby machines that definitely contain the binlogs of all transactions that have been confirmed as committed by the Kluscomp instances.
Therefore, KlustronDB can tolerate the failure of the primary node and fullsync_consistency_level - 1 backup nodes simultaneously without losing the data of committed transactions.
- After the relay log of M1 has finished replaying, promote M1 to be the primary node of SS1.
MySQL 8.0 has a writeset transaction dependency checking mechanism (binlog_transaction_dependency_tracking=writeset or writeset_session), which allows MySQL replicas to replay faster when replica_parallel_type=logical_clock compared to MySQL 5.7. Under normal circumstances, the master-slave lag is usually only a few seconds.
However, if the design and use of the user data table are unreasonable, for example, if there is no primary key and unique index defined, and a large number of row update or delete statements are executed first (even if each statement modifies/deletes only a small number of rows), it will cause a significant delay in the standby server replaying the binlog. In this special case, it will take a long time to replay all the relay logs, during which any standby server cannot be promoted to the primary node.
To avoid such special situations, we have developed a very convenient standby redo interface and standby delay alarm mechanism for KlustronDB, ensuring that DBAs can promptly receive alarms when the standby delay is too large and complete the standby redo with just one click, thereby quickly catching up with the pace of the primary node again.
- Change the primary node of the other standby machines of SS1 to M1
In the case of a network partition or a manual switch of the master node, if the old master node M0 can still write, that is, M0 has not crashed or restarted, then before promoting M1 to the master node, cluster_mgr will first demote M0 to a standby node and set it to read-only to prevent a split-brain scenario.
- Inform all Kluscomp instances of the fact that "M1 is the primary node of SS1", that is, update their pg_shard.master_node_id field, so that the Kluscomp instances can promptly know and write to the new primary node.
It doesn't matter if the master node information of the Kluscomp instances is not up to date, we have defensive measures for this.
First of all, any KlustronDB_Klustore instance is in a read-only state after startup. Therefore, if M0 restarts for any reason and SS1 has already completed the master-slave switch, M0 cannot be written to after restarting. Even if some Kluscomp instances' master node information has not been updated in time and they still attempt to write to M0, these write operations will fail, so a split-brain situation will not occur.
When a Kluscomp instance finds that the master node it knows cannot write, if at this time cluster_mgr has not yet updated the Kluscomp instance's pg_shard.master_nodeid field, the Kluscomp instance will automatically start the master node detection program to find the new master node of SS1.
Before a new master is found, the Kluscomp instance will either block and wait for the new master election to complete or directly return an error to the client, depending on the system settings. Therefore, it ensures that the failure of the master node is imperceptible to the business.
- The old primary node rejoins --- Flashback
If the old master node M0 restarts at some point later, cluster_mgr will rejoin it as a standby machine to the SS1 storage shard.
Because the Fullsync mechanism uses the after-commit mode to wait for the standby ACK, there may be some transactions in M0 that have been committed on the M0 standby but have not received any ACK, and these transactions all need to be rolled back on the standby.
The flashback plugin of KlustronDB_storage will complete the flashback after startup to ensure that subsequent standby replication can operate normally.
The main purpose of the flashback operation is to perform the opposite of the row operations executed by these redundant transactions, removing their changes, cutting off the redundant binlog files, and removing the GTIDs of the flashbacked transactions from the mysql.gtid_executed system table.
Finally, KlustronDB_storage FullsyncHA has a set of practical measures to avoid unnecessary master-slave switching in extreme situations.
This usually tends to happen under conditions of extremely heavy write load, so unnecessary primary-secondary switches can easily cause performance degradation or even brief unavailability during the heaviest system load, making it a problem that must be strongly avoided.
Based on many years of rich experience in live network development and operations, as well as a deep understanding of MySQL kernel-related technologies, we have implemented a complete set of logic that can identify and avoid unnecessary master-slave switches.
Through distributed transaction processing and the Fullsync and Fullsync HA mechanisms, KlustronDB can ensure complete data consistency guarantees and disaster recovery capabilities, while achieving high reliability and high availability, and also providing extremely high performance, capable of handling high-concurrency, heavy-load OLTP scenarios.