[转帖]061、监控指标之TiKV

监控,指标,tikv · 浏览次数 : 0

小编点评

## TiKV Monitor Dashboard This dashboard provides an overview of TiKV performance on a Kubernetes cluster, including metrics related to TiKV-Details, Threads, Errors, and Storage. **Key Metrics:** * **Cluster - Store Size / Available Size:** Displays the total amount of storage used and the available size. * **Cluster - Cluster - CPU/Memory/IO Util:** Shows the CPU, memory, and I/O utilization of the cluster nodes. * **Cluster - MBps/QPS:** Indicates the maximum memory bandwidth usage and the maximum number of queries per second. * **Cluster - Region/Leader region:** Shows the CPU, memory, and I/O utilization of the region leader node. * **Thread CPU - gRPC poll CPU:** Monitors the CPU utilization of gRPC communication threads. * **Thread CPU - Unified Read pool CPU:** Shows the CPU utilization of read threads in the unified read pool. * **Thread CPU - Scheduler worker CPU:** Monitors the CPU utilization of scheduler worker threads. * **Thread CPU - Raft store CPU:** Shows the CPU utilization of raft store worker threads. * **Thread CPU - Async apply CPU:** Monitors the CPU utilization of apply threads. * **Errors - Server is Busy:** Indicates if the server is experiencing high CPU usage. * **Duration - TiDB写入流程:** Provides insights into the overall duration of TiDB write operations. * **KV Request - KV Request Duration:** Shows the average time taken for KV requests. * **KV Request - KV Request Duration:** This chart provides detailed breakdowns of KV request duration by byte. * **Schedule-Commit - Scheduler command duration:** Shows the duration of scheduler command execution. * **Schedule-Commit - Scheduler latch wati duration:** Monitors the latch wait duration during the scheduler command execution. * **Schedule-Commit - Scheduler latch wati duration:** This chart provides detailed breakdown of latch wait duration by flow. * **Error - Raft Propose - Apply wait duration:** Shows the duration of raft propose and apply wait for each operation. * **Error - Raft IO - Append log duration:** Provides the average duration of append logs in the raft store. * **Error - Raft IO - Commit log duration:** Shows the average duration of commit logs in the raft store. * **Error - Raft Propose - Apply wait duration:** Displays the duration of raft propose and apply wait for each operation.

正文

资源相关

Grafana监控 TiKV-Details —> Cluster - Store Size / Available Size
在这里插入图片描述
Grafana监控 TiKV-Details —> Cluster - Cluster - CPU/Memory/IO Util

Grafana监控 TiKV-Details —> Cluster - MBps/QPS

Grafana监控 TiKV-Details —> Cluster - Region/Leader
region 如果超过5万个，可能会影响与pd的心跳。

线程池相关

Grafana监控 TiKV-Details —> Thread CPU - gRPC poll CPU
在这里插入图片描述
Grafana监控 TiKV-Details —> Thread CPU - Unified Read pool CPU
读线程的数量

Grafana监控 TiKV-Details —> Thread CPU - Scheduler worker CPU
写线程的数量不要超过: 90% * storage.scheduler_worker_pool_size

Grafana监控 TiKV-Details —> Thread CPU - Raft store CPU
在这里插入图片描述
raft线程不要超过： 80% * raftstore.store-pool-size

Grafana监控 TiKV-Details —> Thread CPU - Async apply CPU
在这里插入图片描述
apply线程不要超过： 80% * raftstore.apply-pool-size

Error相关

Grafana监控 TiKV-Details —> Errors - Server is Busy
在这里插入图片描述

Duration相关

** TiDB写入流程**
在这里插入图片描述
Grafana监控 TiKV-Details —> gRPC - 99% gRPC message duration
这个相当于是整体处理图

Grafana监控 TiKV-Details —> KV Request —> KV Request Duration 99 by byte
这个图可以判断KV Rquest 多 ,这图包含网络+TiKV的返回耗时。如果要判断到底是网络还是TiKV处理多。这个要结合其他图
在这里插入图片描述
Grafana监控 TiKV-Details —> Schedule-Commit —> Scheduler command duration

Grafana监控 TiKV-Details —> Schedule-Commit —> Scheduler latch wati duration
latch wati duration: 跟流量相关，可减缓写入速度，相关参数storage.scheduler_pending_write_threshold
在这里插入图片描述