跳到主要内容

用夜莺监控 Elasticsearch

通过 ES 自己的 HTTP 接口采集集群健康、节点 JVM、线程池、索引和分片状态,小集群抓一个节点即可拿到全集群数据。

最后校对于

Elasticsearch 通过 HTTP JSON 接口暴露自身状态,插件抓 _cluster/health、_cluster/stats、_nodes/stats 和索引统计:集群颜色、未分配分片、节点 JVM 堆与 GC、断路器、线程池拒绝、索引写入与查询延迟、磁盘水位。

小集群设 local = false,从任意一个节点抓就能拿到所有节点的数据;大集群建议每个节点部署 Categraf 并设 local = true 只抓本机。模板里的仪表盘分 Categraf 和 elasticsearch_exporter 两套,规则覆盖集群变红变黄、分片未分配、JVM 堆逼近上限、线程池拒绝和磁盘水位。

模板里有什么​

  • 5 张仪表盘,共 307 个面板
  • 22 条告警规则,每条附处置步骤
  • 30 条指标说明
  • Categraf 采集配置:input.elasticsearch

怎么接入​

  1. 安装 Categraf,把下面的配置放到 conf/input.elasticsearch/elasticsearch.toml,按实际地址和凭据改好后重启 Categraf。启动日志里出现 input: local.elasticsearch started 就说明插件已加载。

    # # collect interval
    # interval = 15

    ############################################################################
    # !!! uncomment [[instances]] to enable this plugin
    [[instances]]
    # # interval = global.interval * interval_times
    # interval_times = 1

    # append some labels to metrics
    # labels = { cluster="cloud-n9e-es" }

    ## specify a list of one or more Elasticsearch servers
    # servers = ["http://localhost:9200"]
    servers = []

    ## Timeout for HTTP requests to the elastic search server(s)
    http_timeout = "10s"

    # either /_nodes/stats or /_nodes/_local/stats depending on this setting
    local = false

    ## Set cluster_health to true when you want to obtain cluster health stats
    cluster_health = true

    ## Adjust cluster_health_level when you want to obtain detailed health stats
    ## The options are
    ## - indices (default)
    ## - cluster
    cluster_health_level = "cluster"

    ## Set cluster_stats to true when you want to obtain cluster stats.
    cluster_stats = true

    ## Indices to collect; can be one or more indices names or _all
    ## Use of wildcards is allowed. Use a wildcard at the end to retrieve index names that end with a changing value, like a date.
    # indices_include = ["zipkin*"]

    ## use "shards" or blank string for indices level
    indices_level = ""

    ## node_stats is a list of sub-stats that you want to have gathered. Valid options
    ## are "indices", "os", "process", "jvm", "thread_pool", "fs", "transport", "http",
    ## "breaker". Per default, all stats are gathered.
    node_stats = ["jvm", "breaker", "process", "os", "fs", "indices", "thread_pool", "transport"]

    ## HTTP Basic Authentication username and password.
    username = "elastic"
    password = "password"

    ## Optional TLS Config
    # use_tls = false
    # tls_ca = "/etc/categraf/ca.pem"
    # tls_cert = "/etc/categraf/cert.pem"
    # tls_key = "/etc/categraf/key.pem"
    ## Use TLS but skip chain & host verification
    # insecure_skip_verify = true

    ## Sets the number of most recent indices to return for indices that are configured with a date-stamped suffix.
    ## Each 'indices_include' entry ending with a wildcard (*) or glob matching pattern will group together all indices that match it, and
    ## sort them by the date or number after the wildcard. Metrics then are gathered for only the 'num_most_recent_indices' amount of most
    ## recent indices.
    num_most_recent_indices = 1
  2. 在夜莺「集成中心」里找到 Elasticsearch,把仪表盘和告警规则导入到业务组。

  3. 导入的告警规则默认是停用的。先确认指标已经入库,再按自己环境的阈值逐条启用。

内置仪表盘​

仪表盘面板数
ElasticSearch53
Elasticsearch Dashboard54
ElasticSearch, group by service53
ElasticSearch, group by cluster53
ElasticSearch By Exporter94

内置告警规则​

共 22 条,导入后默认停用,按自己环境的阈值逐条启用。每条规则的触发条件(PromQL)和处置步骤收录在告警规则库:Elasticsearch 告警规则。

规则级别持续
Elastic Cluster Red statusS1 紧急—
Elastic Cluster Yellow statusS2 警告10 分钟
Elasticsearch disk out of space of the instanceS1 紧急2 分钟
Elasticsearch disk space low of the instanceS2 警告2 分钟
Elasticsearch Heap Usage Too High of the instanceS1 紧急2 分钟
Elasticsearch Heap Usage warning of the instanceS2 警告2 分钟
Elasticsearch initializing shards of the instanceS2 警告15 分钟
Elasticsearch no new documents of the instanceS2 警告5 分钟
Elasticsearch pending tasks of the instanceS2 警告15 分钟
Elasticsearch relocation shards of the instanceS2 警告15 分钟
Elasticsearch unassigned shards of the instanceS1 紧急—
Elastic Cluster Red statusS1 紧急—
Elastic Cluster Yellow statusS2 警告10 分钟
Elasticsearch disk out of space of the instanceS1 紧急2 分钟
Elasticsearch disk space low of the instanceS2 警告2 分钟
Elasticsearch Heap Usage Too High of the instanceS1 紧急2 分钟
Elasticsearch Heap Usage warning of the instanceS2 警告2 分钟
Elasticsearch initializing shards of the instanceS2 警告15 分钟
Elasticsearch no new documents of the instanceS2 警告5 分钟
Elasticsearch pending tasks of the instanceS2 警告15 分钟
Elasticsearch relocation shards of the instanceS2 警告15 分钟
Elasticsearch unassigned shards of the instanceS1 紧急—

指标说明​

共 30 条,和仪表盘、告警规则里用到的指标一一对应。

指标含义单位
elasticsearch_cluster_health_delayed_unassigned_shardsCluster Health delayed unassigned 的分片数个
elasticsearch_cluster_health_number_of_pending_tasksCluster Health Pending task 数量
elasticsearch_cluster_health_relocating_shardsCluster Health relocating 的分片数个
elasticsearch_cluster_health_unassigned_shardsCluster Health unassigned 的分片数个
elasticsearch_cluster_health_status_codeCluster Health 健康度状态码 — - 1:Green,绿色状态,表示所有分片都正常; 2:Yellow,黄色状态,主分片都正常,从分片有不正常的; 3:Red,红色状态,有些主分片不正常
elasticsearch_cluster_health_number_of_data_nodesCluster Health 数据节点数量
elasticsearch_cluster_health_initializing_shardsCluster Health 正在初始化的分片数个
elasticsearch_cluster_health_active_primary_shardsCluster Health 活跃主分片数个
elasticsearch_cluster_health_active_shardsCluster Health 活跃分片数个
elasticsearch_cluster_health_number_of_nodesCluster Health 节点数量
irate(elasticsearch_indices_indexing_index_time_in_millis[3m]) / irate(elasticsearch_indices_indexing_index_total[3m])Indexing 平均耗时ms
irate(elasticsearch_indices_merges_total_time_in_millis[3m]) / irate(elasticsearch_indices_merges_total[3m])Merge 平均耗时ms
irate(elasticsearch_indices_search_query_time_in_millis[3m]) / irate(elasticsearch_indices_search_query_total[3m])Query 平均耗时ms
irate(elasticsearch_indices_indexing_index_total[3m])每秒 indexing 数量个
irate(elasticsearch_indices_merges_total_size_in_bytes[3m])每秒 merge 大小bytes/s
irate(elasticsearch_indices_merges_total_docs[3m])每秒 merge 数量个
irate(elasticsearch_indices_docs_deleted[3m])每秒删除 doc 数量个
100 - 100 * elasticsearch_fs_total_available_in_bytes / elasticsearch_fs_total_total_in_bytes硬盘使用率%
irate(elasticsearch_transport_rx_size_in_bytes[3m])网络流量 - 入向每秒流量bytes/s
irate(elasticsearch_transport_tx_size_in_bytes[3m])网络流量 - 出向每秒流量bytes/s
elasticsearch_process_cpu_percent进程 CPU 使用率%
elasticsearch_jvm_mem_heap_used_percent进程 JVM Heap 使用率%
elasticsearch_jvm_mem_heap_committed_in_bytes进程 JVM Heap 区 committed 大小bytes
elasticsearch_jvm_mem_non_heap_committed_in_bytes进程 JVM Non Heap 区 committed 大小bytes
elasticsearch_jvm_mem_pools_old_used_in_bytes进程 JVM Old 内存池 used 大小bytes
elasticsearch_jvm_mem_pools_young_used_in_bytes进程 JVM Young 内存池 used 大小bytes
irate(elasticsearch_jvm_gc_collectors_young_collection_count[3m])进程新生代每秒 GC 次数
irate(elasticsearch_jvm_gc_collectors_young_collection_time_in_millis[3m])进程新生代每秒 GC 耗时ms
irate(elasticsearch_jvm_gc_collectors_old_collection_count[3m])进程老生代每秒 GC 次数
irate(elasticsearch_jvm_gc_collectors_old_collection_time_in_millis[3m])进程老生代每秒 GC 耗时ms

相关链接​