Distributed transactions and error handling in two-phase commit

1. Background

The author and the team’s two-phase commit method in KlustronDB can successfully avoid the shortcomings of the classic two-phase commit algorithm.

In terms of the two-phase commit mechanism and principles of distributed transaction processing, my team and I have enhanced its disaster recovery and error handling capabilities, allowing any node in the KlustronDB cluster to crash or experience network failures, timeouts, etc., at any time without causing inconsistencies or data loss in the cluster-managed data.

This article will detail the principles and mechanisms of error handling for distributed transactions in the two-phase commit protocol, as well as the latency overhead.

2. How does KlustronDB handle errors in the two-phase commit algorithm?

In the working scenarios of distributed database clusters in production environments, typically less than 0.01% of distributed transaction commits will encounter errors, but we still need to handle all possible errors.

Because even if 10 billion transactions are executed, as long as 1 transaction encounters a commit error, it will cause user data errors.

A database system is meant to ensure that transactions are always correctly committed, and ACID guarantees always hold, without exception.

This is more complex for distributed database systems than for standalone databases, because there are more possible sources of errors (multiple Kluscomp instances and multiple Klustore instances, and the network connections between them).

This is also why the design and implementation of database systems are so complex, and the design and implementation of distributed database systems are even more complex.

Next, let's take a look at how the KlustronDB cluster handles errors that occur during the submission of distributed transactions. We will separately discuss error handling for each phase of the two-phase commit, as well as error handling for batch writing to the commit log.

2.1 Phase One Error Handling

As shown in the above figure, if a statement error, network disconnection, or timeout occurs during the prepare phase, GTM will submit a rollback record request to GTSS, and without waiting for its return, will immediately send a rollback command to the node that had an error and disconnect the timed-out connection, then return an error to the client, informing the client that the GT transaction has been rolled back.

GTSS will record the commit instruction of GT as ROLLBACK in the commit log, so that cluster_mgr will roll back them when subsequently processing GT's prepared transaction branches.

2.2 Batch Write Commit Logging Error Handling

As shown in the figure above, if the GTSS encounters an error or times out while writing the commit log, the GTM will roll back all prepared transaction branches of the GT, which means sending XA ROLLBACK to all storage shards where the GT was written, and regardless of the result, it returns 'Aborted' to the client to indicate that the GT has been rolled back.

Even if XA ROLLBACK fails to send, this transaction branch will still be rolled back by cluster_mgr as expected.

2.3 Phase Two Error Handling

As shown in the figure above, if a network error or timeout occurs in the second phase, the submission is still returned as successful to the client.

This is because any distributed transaction recorded in the commit log must be completed.

If any computing or Klustore instance fails or experiences a network fault during the execution of the second phase, the cluster_mgr process will handle these transaction branches according to the instructions in the commit log --- if the instruction is to commit, then all transaction branches of the GT will be committed.

If the instruction is to rollback or the GT commit log cannot be found, then rollback all of GT's transaction branches.

If a Kluscomp instance crashes or loses network connection during the second phase, the transaction will still be committed. At this time, the application system backend (that is, the database client) will find that its commit statement does not return until the database connection times out (usually the application layer will also let the end user's connection time out) or returns a disconnection error.

3. Delay Loss

Since both the prepare and commit phases of the two-phase commit need to wait for the storage engine to flush the WAL log, and between the two phases it also needs to wait for the commit log to be written to the metashard, the time cost of the two-phase commit is bound to be slightly higher than executing the same SQL DML statement but performing a single-phase commit.

According to this performance report: http://www.zettadb.com/blogs/perf-cmp1open in new window, KlustronDB's two-phase commit adds about 30 milliseconds of latency on ordinary server hardware configurations and gigabit networks.

In a commercial server hardware and network environment, this delay increase will be less than 30 milliseconds. This 30 milliseconds includes the writing of the commitlog, the additional waiting time for one more stage, and all additional network communication overhead.

4. Conclusion

The distributed transaction processing mechanism of KlustronDB ensures the consistency and disaster recovery capability of distributed transaction execution and commitment. During the transaction commit period, any node or network failure will not cause the ACID guarantees of the transaction to fail, thereby ensuring the correctness of user data.

Starting from KlustronDB version 1.2, global MVCC consistent queries are supported. At that time, another article will be written to introduce the working mechanism of global MVCC in KlustronDB.