View node log information using Kibana

Preface

KlustronDB supports a graphical, one-stop unified system for viewing and searching logs from all nodes, which is much more convenient than logging into each server to check and search log files. Specifically, KlustronDB collects log file updates from each node through ElasticSearch's FileBeat and stores them in a user-installed ElasticSearch cluster, allowing real-time viewing and searching of logs from all types of nodes in the cluster through Kibana, including clustmgr, nodemgr, Klustore instances, and Kluscomp instances.

For Klustore instances, in addition to retrieving error logs, you can also view the logs of executed SQL statements, such as the general_log and slow_log; for Kluscomp instances, you can also enable SQL logging, which logs all SQL statements sent from Kluscomp instances to Klustore instances. The method to enable it is

set enable_sql_log=on

However, SQL logs have a significant performance overhead and take up a lot of disk space.

Log Extension Description

Each log entry in KlustronDB (including error logs, general_log, slow log) has three additional fields added to the original content to associate the logs printed by all Kluscomp instances and Klustore instances in the cluster with each client session and each SQL statement. The important new fields include:

comp_node_id

This is the unique and immutable ID of each Kluscomp instance in the cluster. All logs printed by the Klustore instance have this field, so that it can be tracked which Kluscomp instance's client connection corresponds to this log.

global_conn_id

This is the session ID in each Kluscomp instance, and it is also the ID of the worker process in the Kluscomp instance process group that handles a user connection. The global_conn_id, together with the comp_node_id, identifies each client session in the cluster. Logs with the same values in these two fields correspond to behaviors within the same session. However, it should be noted that after a process ends, its process ID may be reused by a process started in the future, and it is usually not reused immediately but only after a long period (typically at least a few hours). Therefore, in the logs of a long-running KlustronDB cluster, logs with the same global_conn_id and comp_node_id are not necessarily generated by behaviors from the same user session. But logs that are close in time are usually behaviors of the same session. Additionally, when a connection is disconnected (i.e., the session ends), a log is also generated, so careful analysis can accurately distinguish different sessions.

cluster_stmt_id

The KlustronDB cluster assigns each statement executed in every connection of each Kluscomp instance a sequential number starting from 1, which identifies the statement within that connection. The three fields cluster_stmt_id, global_conn_id, and comp_node_id can uniquely identify each SQL statement. This allows viewing every SQL statement executed from the client in the logs, as well as the operations and logs of each node within the cluster.

You can search the above field names and values in the logs to find a series of operations in a specific connection, as well as the specific actions of cluster-related nodes executing a particular statement.

More explanation

In addition, in statements such as SHOW PROCESSLIST, the fields cluster_stmt_id, global_conn_id, and comp_node_id have also been added to associate the backend connections of each client connection to the cluster across various Klustore instances. This information also helps DBAs observe the operational status of each node in the system. Klustore instances log more behaviors, including the execution of key commands and SQL statements, as well as important events, such as disconnections for various reasons. By setting print_extra_info_verbosity=N (where N is an integer), you can control whether some logs are printed. The higher the number, the more detailed the logs; setting it to 0 will not print any such logs.

The SQL log files of the Kluscomp instances are located in the directory where their error log files are, which is usually also the data directory of the instance. The general_log and slow_log of the Klustore instances are located in their data file or log file directories. You can check the configuration file of each Klustore instance instance, which is located in the data directory of that instance.

In addition, the enabling and control of statement logs and slow query logs for Klustore instances and Kluscomp instances are exactly the same as in MySQL and PostgreSQL, and will not be repeated here. Moreover, the SQL statement logs and slow query logs of the Kluscomp instances are also recorded in their error log files, which is exactly the same as PostgreSQL. All of the above log files, as well as the binlog files of the Klustore instances, need to be manually cleaned up, which is also the same as in MySQL and PostgreSQL.

Prerequisites for Using Kibana

When installing clustmgr, nodemgr, Klustore instances, and Kluscomp instances, the following two packages are required in advance:
- wget http://klustron.cn:14000/thirdparty/efk/elasticsearch-7.10.1.tar.gz
- wget http://klustron.cn:14000/thirdparty/efk/kibana-7.10.1.tar.gz
At the same time, add the following configuration in cluster_and_node_mgr.json, at the same level as cluster_manager and node_manager.

The explanations of the fields are as follows:

elasticsearch: Specifies the content of elasticsearch;
ip: String type, the machine where Elasticsearch is installed, this machine will also have Elasticsearch and Kibana installed;
port: Integer, specifies the port of Elasticsearch, default is 9200;
kibana_port: Integer, specifies the port for Kibana, default is 5601.
After clustmgr, nodemgr, Klustore instances, and Kluscomp instances are all successfully installed, you can use Kibana to view the logs.