[TOC]

0x00 前言简述及环境准备

描述: 本章主要讲解和实践Prometheus在企业中的应用场景的复现,采用了docker-compose的资源清单进行快速构建prometheus_server、prometheus_pushgateway、prometheus_alertmanager、grafana等环境。

主要实现目标(功能):

• 0) 实现Windows主机的监控和展示
• 1) 实现MySQL与Redis数据库的监控和展示
• 2) 实现外部kubernetes集群的监控和展示

主机说明:

# Kubernetes cluster 0: weiyigeek-lb-vip.k8s (正式环境)
192.168.12.107 - master
192.168.12.108 - master
192.168.12.109 - master
192.168.12.223 - work
192.168.12.224 - work
192.168.12.225 - work

# Kubernetes cluster 1: k8s-test.weiyigeek (测试环境-单master节点)
192.168.12.111

# Kubernetes cluster 2: 192.168.12.226(开发环境-单master节点)
192.168.12.226

环境说明
描述: 上述环境中都安装了docker运行环境,并在192.168.12.107主机中安装了docker-compose软件,下面进行的配置循序渐进的进行添加。

# 下面表示在主机下进行基础环境的安装部署的组件(node_export与cAdivsor的安装配置参考“1.Prometheus(普罗米修斯)容器集群监控入门.md”,此处不在重新累述)
192.168.12.107
- prometheus_server: 30090
- prometheus_pushgateway: 30091
- prometheus_alertmanager: 30093
- grafana: 9091

192.168.12.108~109
192.168.12.223~225
- node_exporter: 9091

192.168.12.111
- cAdivsor: 9100

# 此处暂时不进行配置后续利用prometheus监控第三方k8s集群时使用
192.168.12.226
- kubernetes Api Server: 6443

目录结构一览:

$ tree -L 5
.
├── docker-compose.yml # 资源清单
├── grafana            # grafana UI 展示: 针对于数据持久化(插件、dashboard等)
│   └── data
└── prometheus         # Prometheus 监控相关配置以及数据持久化目录   
    ├── conf  
    │   ├── alertmanager.yaml  # 报警发送器配置
    │   ├── conf.d            
    │   │   ├── discovery      # 自动化发现相关配置文件
    │   │   │   └── k8s_nodes.yaml
    │   │   ├── rules          # 报警规则
    │   │   │   └── alert.rules
    │   │   └── auth            # k8s 以及 相关认证使用
    │   │       ├── k8s_client.crt
    │   │       ├── k8s_client.key
    │   │       └── k8s_token
    │   └── prometheus.yml

    └── data

环境快速准备

• 0.目录结构快速生成

  mkdir -vp /nfsdisk-31/monitor/prometheus/conf/conf.d/{discovery,rules,auth}
  mkdir -vp /nfsdisk-31/monitor/prometheus/data
  mkdir -vp /nfsdisk-31/monitor/grafana/date

• 1.prometheus.yaml 主配置文件:

  tee prometheus.yaml <<'EOF'
  global:
    scrape_interval: 2m
    scrape_timeout: 10s
    evaluation_interval: 1m
    external_labels:
      monitor: 'prom-demo'
  scrape_configs:
    - job_name: 'prom-Server'
      static_configs:
        - targets: ['localhost:9090']
    - job_name: 'cAdvisor'
      static_configs:
        - targets: ['192.168.12.111:9100']
    - job_name: 'prom-Host'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/k8s_nodes.yaml
        refresh_interval: 1m
  
  rule_files:
    - /etc/prometheus/conf.d/rules/*.rules
  
  alerting:
    alertmanagers:
    - scheme: http
      static_configs:
      - targets:
        - '192.168.12.107:30093'
  EOF

• 2.alert.rules 配置文件:

  tee alert.rules <<'EOF'
  groups:
  - name: node-normal
    rules:
    - alert: service_down
      expr: up == 0
      for: 2m
      labels:
        severity: 1
        team: node
      annotations:
        summary: "主机 {{ $labels.instance }} 监控服务已停止运行超过 15s！"
    - alert: high_load
      expr: node_load1 > 0.7
      for: 5m
      labels:
        severity: 1
        team: node
      annotations:
        summary: "主机 {{ $labels.instance }} 高负载大于0.7以上运行超过 5m！"
  EOF

• 3.k8s_nodes.yaml 自动发现file_sd_configs配置文件。

  tee k8s_nodes.yaml <<'EOF'
  - targets: [ '192.168.12.107:9100','192.168.12.108:9100','192.168.12.109:9100' ]
    labels: {'env': 'prod','cluster': 'weiyigeek-lb-vip.k8s','nodeType': 'master'}
  - targets: [ '192.168.12.223:9100','192.168.12.224:9100','192.168.12.225:9100' ]
    labels: {'env': 'prod','cluster': 'weiyigeek-lb-vip.k8s','nodeType': 'work'}
  EOF

• 4.alertmanager.yaml 邮箱报警发送配置

  tee alertmanager.yaml <<'EOF'
  global:
    resolve_timeout: 5m
    smtp_from: 'monitor@weiyigeek.top'
    smtp_smarthost: 'smtp.exmail.qq.com:465'
    smtp_auth_username: 'monitor@weiyigeek.top'
    smtp_auth_password: xxxxxxxxxxx'
    smtp_require_tls: false
    # smtp_hello: 'qq.com'
  route:
    group_by: ['alertname']
    group_wait: 30s
    group_interval: 1m
    repeat_interval: 10m
    receiver: 'default-email'
  receivers:
  - name: 'default-email'
    email_configs:
    - to: 'master@weiyigeek.top'
      send_resolved: true
  # inhibit_rules:
  #   - source_match:
  #       severity: 'critical'
  #     target_match:
  #       severity: 'warning'
  #     equal: ['alertname', 'instance']
  EOF
  # Tips : 可以采用amtool工具校验该yml文件是否无误`./amtool check-config alertmanager.yml`

• 5.docker-compose.yml 资源清单内容:

  # Desc:  prometheus / pushgateway / alertmanager / grafana 环境搭建
  # author: WeiyiGeek
  # email: master@weiyigeek.top
  
  # 创建一个名称为monitor的桥接网络
  $ docker network create monitor --driver bridge
  
  tee docker-compose.yml <<'EOF'
  version: '3.2'
  services:
    prometheus:
      image: prom/prometheus:v2.26.0
      container_name: prometheus_server
      environment:
        TZ: Asia/Shanghai
      volumes:
        - /nfsdisk-31/monitor/prometheus/conf/prometheus.yaml:/etc/prometheus/prometheus.yaml
        - /nfsdisk-31/monitor/prometheus/conf/conf.d:/etc/prometheus/conf.d
        - /nfsdisk-31/monitor/prometheus/data:/prometheus/data
        - /etc/localtime:/etc/localtime
      command:
        - '--config.file=/etc/prometheus/prometheus.yaml'
        - '--storage.tsdb.path=/prometheus/data'
        - '--web.enable-admin-api'
        - '--web.enable-lifecycle'
      ports:
        - '30090:9090'
      restart: always
      networks:
        - monitor
  
    pushgateway:
      image: prom/pushgateway
      container_name: prometheus_pushgateway
      environment:
        TZ: Asia/Shanghai
      volumes:
        - /etc/localtime:/etc/localtime
      ports:
        - '30091:9091'
      restart: always
      networks:
        - monitor
  
    alertmanager:
      image: prom/alertmanager:v0.21.0
      container_name: prometheus_alertmanager
      environment:
        TZ: Asia/Shanghai
      volumes:
        - /nfsdisk-31/monitor/prometheus/conf/alertmanager.yaml:/etc/alertmanager.yaml
        - /etc/localtime:/etc/localtime
        # - /nfsdisk-31/monitor/prometheus/alertmanager:/alertmanager
      command:
        - '--config.file=/etc/alertmanager.yaml'
        - '--storage.path=/alertmanager'
      ports:
        - '30093:9093'
      restart: always
      networks:
        - monitor
  
    grafana:
      image: grafana/grafana:7.5.5
      container_name: grafana
      user: "472"
      environment:
        - TZ=Asia/Shanghai
        - GF_SECURITY_ADMIN_PASSWORD=weiyigeek
      volumes:
        - /nfsdisk-31/monitor/grafana/data:/var/lib/grafana
        - /etc/localtime:/etc/localtime
      ports:
        - '30000:3000'
      restart: always
      networks:
        - monitor
      dns:
        - 223.6.6.6
        - 192.168.12.254
  
  networks:
    monitor:
      external: true
  EOF
  
  # 验证配置
  docker-compose config 
  
  # 创建和后台启动容器
  docker-compose up -d

• 6.环境验证: 访问搭建的prometheus server 服务地址http://192.168.12.107:30090/service-discovery进行查询以及监控节点的查看。

weiyigeek.top-基础环境验证

0x01 实现Windows主机的监控和展示

描述: 我们采用 Prometheus 监控进行 Windows 机器,我们也要像在 node_exporter 二进制可执行软件安装运行在Linux系统上, 在Windows系统上安装 windows_exporter 操作流程如下:

• Step 1.下载安装windows_exporter可执行软件其releases地址 我们可以选择exe或者msi安装方式;

  # exe与msi下载
  windows_exporter-0.16.0-amd64.exe
  windows_exporter-0.16.0-amd64.msi
  
  # msi - 安装执行调用示例：
  msiexec /i <path-to-msi-file> ENABLED_COLLECTORS=os,iis LISTEN_PORT=5000
  # 带有自定义查询的示例服务收集器
  msiexec /i <path-to-msi-file> ENABLED_COLLECTORS=os,service --% EXTRA_FLAGS="--collector.service.services-where ""Name LIKE 'sql%'"""
  # 在某些旧版本的Windows上，可能需要用双引号将参数值括起来，以正确解析install命令：
  msiexec /i C:\Users\Administrator\Downloads\windows_exporter.msi ENABLED_COLLECTORS="ad,iis,logon,memory,process,tcp,thermalzone" TEXTFILE_DIR="C:\custom_metrics\"

  # exe - Examples
  # 仅启用service collector并指定自定义查询
  .\windows_exporter.exe --collectors.enabled "service" --collector.service.services-where "Name='windows_exporter'"
  # 仅启用process collector并指定自定义查询
  .\windows_exporter.exe --collectors.enabled "process" --collector.process.whitelist="firefox.+"
  # 将[defaults]与--collectors.enabled参数一起使用，该参数将与所有默认收集器一起展开。
  .\windows_exporter.exe --collectors.enabled "[defaults],process,container"

• Step 2.使用配置文件config.yml运行后将在9182端口启用监听我们可以访问该http://127.0.0.1:9182/metricsurl查看metrics.

  .\windows_exporter-0.16.0-amd64.exe --config.file=config.yml
  
  # config.yml
  # 默认启用 Collectors 收集器以及额外添加的收集器
  collectors:
    enabled: cpu,cs,logical_disk,net,os,system,service,logon,process,tcp	
  collector:
    service:
      services-where: Name='windows_exporter'
  log:
    level: debug
  scrape:
    timeout-margin: 0.5
  telemetry:
    addr: ":9182"
    path: /metrics
    max-requests: 5
  
  # 防火墙规则调整(指定远程连接的地址以及本地开放端口)
  New-NetFirewallRule -Name prom-windows_exporter -Direction Inbound -DisplayName 'windows_exporter' -RemoteAddress 192.168.12.107 -LocalPort 9182 -Protocol 'TCP'
    # Name                  : prom-windows_exporter
    # DisplayName           : windows_exporter
    # Description           :
    # DisplayGroup          :
    # Group                 :
    # Enabled               : True
    # Profile               : Any
    # Platform              : {}
    # Direction             : Inbound
    # Action                : Allow
    # EdgeTraversalPolicy   : Block
    # LooseSourceMapping    : False
    # LocalOnlyMapping      : False
    # Owner                 :
    # PrimaryStatus         : OK
    # Status                : 已从存储区成功分析规则。 (65536)
    # EnforcementStatus     : NotApplicable
    # PolicyStoreSource     : PersistentStore
    # PolicyStoreSourceType : Local
  # New-NetFirewallRule -Name powershell-remote-udp -Direction Inbound -DisplayName 'PowerShell远程连接 UDP' -LocalPort 9182 -Protocol 'UDP'  # http 协议走的是tcp所以不用添加UDP协议的防火墙策略

weiyigeek.top-windows-metrics

• Step 3.添加到prometheus.yaml主配置文件之中进行重新加载配置即可发现该机器，如下图所示。

  # prometheus.yaml
  scrape_configs:
  - job_name: 'windows-exporter'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/win_nodes.yaml
        refresh_interval: 1m
  
  
  # vi win_nodes.yaml
  - targets: [ '192.168.12.240:9182' ]
    labels: {'env': 'temp','osType': 'windows','nodeType': 'master'}
  
  # PromQL
  windows_os_info or windows_exporter_build_info{instance='192.168.12.240:9182'} or windows_logical_disk_free_bytes{volume="C:"} / (1024^3) or windows_net_current_bandwidth

  

  weiyigeek.top-windows_exporter_promQL

• Step 4.配置Grafana添加prometheus的windows监控的dashboard(官方搜索)
  • 安装地址:https://grafana.com/grafana/dashboards/10467

weiyigeek.top-Grafana_windows_export

• Step 5.访问Grafana的Dashbord查看windows采集的数据展示验证

weiyigeek.top-

0x02 实现MySQL与Redis数据库的监控和展示

描述: 我们可以针对于MySQL以及Redis进行数据库的监控配置利用到的软件是mysql_exporter(https://github.com/prometheus/mysqld_exporter)和`redis_exporter`(https://github.com/oliver006/redis_exporter/)。

• Step 1.准备测试的MySQL与Redis的数据库然后利用docker容器进行监控指标的采集;

  # - mysqld-exporter : export DATA_SOURCE_NAME='user:password@(hostname:3306)/'
  # (1) 在执行的MySQL数据库中添加监控用户
  CREATE USER 'exporter'@'%' IDENTIFIED BY 'XXXXXXXX' WITH MAX_USER_CONNECTIONS 3;
  GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
  # (2) 运行 prom/mysqld-exporter 容器
  docker run -d -p 9104:9104 --name mysqld-exporter -e DATA_SOURCE_NAME="exporter:XXXXXXXX@(192.168.12.185:3306)/" prom/mysqld-exporter
  
  # - redis_exporter:  Supports Redis 2.x, 3.x, 4.x, 5.x, and 6.x
  # Redis instance addresses can be tcp addresses: redis://localhost:6379, redis.example.com:6379 or e.g. unix sockets: unix:///tmp/redis.sock.
  docker run -d  --name redis_exporter --network host -e REDIS_ADDR="redis://192.168.12.1doc85:6379"  -e REDIS_PASSWORD="weiyigeek.top" oliver006/redis_exporter # -p 9121:9121
  
  # 查看部署的exporter
  $ docker ps
  CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                    NAMES
  c3a7a5663143        oliver006/redis_exporter   "/redis_exporter"        9 minutes ago       Up 9 minutes                                 redis_exporter
  0a3d557bf36b        prom/mysqld-exporter       "/bin/mysqld_exporter"   16 minutes ago      Up 16 minutes       0.0.0.0:9104->9104/tcp   mysqld-exporter

• Step 2.分别访问mysqld-exporter和redis_exporter的metrics的URL

  $ curl -s http://192.168.12.111:9104/metrics | tail -n -5
    # HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
    # TYPE promhttp_metric_handler_requests_total counter
    promhttp_metric_handler_requests_total{code="200"} 2
    promhttp_metric_handler_requests_total{code="500"} 2
    promhttp_metric_handler_requests_total{code="503"} 0
  
  $ curl -s http://192.168.12.111:9121/metrics | tail -n -5
    # TYPE redis_up gauge
    redis_up 1
    # HELP redis_uptime_in_seconds uptime_in_seconds metric
    # TYPE redis_uptime_in_seconds gauge
    redis_uptime_in_seconds 1.281979e+06

• Step 3.prometheus.yaml 主配置文件修改和添加;

  scrape_configs:
    - job_name: 'mysql_discovery'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/mysql_discovery.yaml
        refresh_interval: 1m
    - job_name: 'redis_discovery'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/redis_discovery.yaml
        refresh_interval: 1m
  
  # vi mysql_discovery.yaml
  - targets: [ '192.168.12.111:9104' ]
    labels: {'env': 'test','osType': 'container','nodeType': 'database'}
  
  # vi redis_discovery.yaml
  - targets: [ '192.168.12.111:9121' ]
    labels: {'env': 'test','osType': 'container','nodeType': 'database'}

• Step 4.热加载prometheus.yaml配置或者重新启动prometheus容器验证monitor目标执行PromQL表达式: redis_instance_info or mysql_version_info

weiyigeek.top-redis&mysql_exporter

• Step 5.在Grafana中找寻安装Prometheus Redis Dashboard(https://grafana.com/grafana/dashboards/763)和`Prometheus mysql Dashboard`(https://grafana.com/grafana/dashboards/7362)

weiyigeek.top-MySQL-Dashbord

---

0x03 实现Jenkins持续集成和交付的服务监控和展示

目标: 使用Prometheus对持续集成Jenkins进行监控，并通过Grafana展示监控数据。

• Step 1.安装Prometheus metrics插件(Prometheus metrics: Expose Jenkins metrics in prometheus format)

• Step 2.在系统管理-> 系统配置->配置Prometheus插件，主要填写Path地址和url路径namespace最后应用保存即可。

weiyigeek.top-Prometheus metrics

• Step 3.测试验证Prometheus插件运行情况,即访问http://yourjenkinserver:port/prometheus

weiyigeek.top-reuqets-Prometheus

• Step 4.在我们的Prometheus服务端将该端点地址添加到prometheus.yml之中。

  - job_name: 'jenkins'
      metrics_path: '/prometheus/'
      scheme: 'http'
      bearer_token: 'bearer_token'
      static_configs:
        - targets: ['192.168.12.107:30001']

• Step 5.重载prometheus.yml配置(使用方式必须是在有--web.enable-lifecycle启动参数为真的情况下)，然后验证监控项：devops_jenkins_executors_available

  # 使修改后的配置生效(在也不用重启容器了)
  curl -X POST http://192.168.12.107:30090/-/reload
  
  # 查看当前配置
  curl http://192.168.12.107:30090/api/v1/status/config

  

  weiyigeek.top-jenkins-prometheus

• Step 6.在Grafana进行添加数据源配置并显示采集的指标。
  • 首先安装a Jenkins performance and health overview for jenkinsci/prometheus-plugin,在面板中+->Import然后输入vid Grafana Dashboard URL:https://grafana.com/grafana/dashboards/9524 -> load -> Import然后跳转到监控页面数据就可以展示出来了;

weiyigeek.top-a Jenkins performance and health overview

---

0x04 实现kubernetes外部集群的监控和展示

描述: 我们知道学习测试 Prometheus 一般将其安装在k8s集群中进行数据metrics的采集，但在实际的环境中企业大多选择将
ometheus 单独部署在集群外部进行监控某一集群，如果有多套集群时使用不同的 Prometheus 实例监控不同的集群，然后用联邦的方式进行汇总。
其次由于我们学习环境的原因，本章将使用 Prometheus 监控外部的 Kubernetes 集群进行配置讲解(在kubernetes集群中即可参照下面某些方式进行配置)

Q: Prometheus 如何采集Kubernetes集群数据?

答: 如果我们对集群内部的 Prometheus 自动发现 Kubernetes 的数据比较熟悉的话，那么监控外部集群的原理也是一样的，只是访问 APIServer 的形式有 inCluster 模式变成了 KubeConfig 的模式，inCluster 模式下在 Pod 中就已经自动注入了访问集群的 token 和 ca.crt 文件，所以非常方便，那么在集群外的话就需要我们手动提供这两个文件，才能够做到自动发现了。

Q: Prometheus通过exporter收集各种维度的监控指标

答: Prometheus 通过 kubernetes_sd_configs 从 Kubernetes 的 REST API 查找并拉取指标，并始终与集群状态保持同步，使用endpoints,service,node,pod,ingress等角色进行自动发现

• endpoints : 自动发现service中的endpoint
• node: 自动发现每个集群节点发现一个target，其地址默认为Kubelet的HTTP端口如"https://192.168.3.217:10250/metrics"
• service : 自动发现每个服务的每个服务端口的target
• pod : 自动发现所有容器及端口
• ingrsss : 自动发现ingress中path

Q: 可以通过哪几种方式维度收集监控指标?

维度 | 工具 | 监控url(__metrics_path__) |备注|
|:—:|:—:|:—|—-|
|Node性能 | node-exporter | /api/v1/nodes/node名:9100/proxy/metrics | 节点状态 |
|Pod性能 | kubelet
cadvisor | /api/v1/nodes/node名:10250/proxy/metrics
/api/v1/nodes/node名:10250/proxy/metrics/cadvisor|容器状态|
|K8S资源| kube-state-metrics|__scheme__://__address____metrics_path__|Deploy/ds等|

Tips : 注意kube-state-metrics监控的URL的动态发现是基于标签的自动补全,其中标签的值都可以通过Prometheus的relabel_config拼接成最终的监控url，由于集群外部署Prometheus和集群内部署Prometheus是不一样的，因此我们可以通过proxy url集群外Prometheus就可以访问监控url来拉取监控指标。

Q: 如何构造Apiserver proxy url?
描述: 在k8s集群中nodes、pods、services都有自己的私有IP，但是无法从集群外访问；但K8S提供以下几种方式来访问：1.通过public IPs访问service , 2.通过proxy 访问node、pod、service, 3.通过集群内的node或pod间接访问

例如: 通过kubectl cluster-info命令可以查看kube-system命令空间的proxy url

$ kubectl cluster-info
Kubernetes master is running at https://k8s-dev.weiyigeek:6443
KubeDNS is running at https://k8s-dev.weiyigeek:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

所以其默认的构造规则为如下格式。

set other_apiserver_address = k8s-dev.weiyigeek:6443
# 访问node
https://${other_apiserver_address}/api/v1/nodes/node_name:[port_name]/proxy/metrics
# 访问service
https://${other_apiserver_address}/api/v1/namespaces/service_namespace/services/http:service_name[:port_name]/proxy/metrics
# 访问pod
https://${other_apiserver_address}/api/v1/namespaces/pod_namespace/pods/http:pod_name[:port_name]/proxy/metrics

在我们了解如何构造proxy url后,我们可以通过集群外Prometheus的relabel_config自行构造proxy url。

1.Endpoints 之服务自动发现

描述: 此处我们采用进行安装部署k8s集群监控的kube-state-metrics服务, 它监听Kubernetes API服务器并生成关联对象的指标, 它不关注单个Kubernetes组件的运行状况，而是关注内部各种对象（如deployment、node、pod等）的运行状况。

流程步骤:

• Step 1.我们先查看当前kube-state-metrics兼容性矩阵与我们kubernetes集群版本的对应参考地址,下面最多记录5个kube状态度量和5个kubernetes版本。

  kube-state-metrics 	Kubernetes 1.17 	Kubernetes 1.18 	Kubernetes 1.19 	Kubernetes 1.20 	Kubernetes 1.21
  v1.8.0 	- 	- 	- 	- 	-
  v1.9.8 	- 	- 	- 	- 	-
  v2.0.0 	-/✓ 	-/✓ 	✓ 	✓ 	-/✓
  master 	-/✓ 	-/✓ 	✓ 	✓ 	✓

• Step 2.k8s集群 ApiServer 访问鉴权账号创建和绑定的集群角色权限配置。
  描述: 在访问K8S apiserver需要先进行授权，而集群内部Prometheus可以使用集群内默认配置进行访问，而集群外访问需要使用token+客户端cert进行认证因此需要先进行RBAC授权。
  • 此处测试由于我们需要访问不同的namespace，建议先使用分配绑定cluster-admin权限,但在生产中一定要使用最小权限原则来保证其安全性(后面会进行演示)。

    # 1.创建名称空间
    kubectl create ns monitor
      # namespace/monitor created
    
    # 2.创建serviceaccounts
    kubectl create sa prometheus --namespace monitor
      # serviceaccount/prometheus created
    
    # 3.创建prometheus角色并对其绑定cluster-admin
    $ kubectl create clusterrolebinding prometheus --clusterrole cluster-admin --serviceaccount=monitor:prometheus
      # clusterrolebinding.rbac.authorization.k8s.io/prometheus created
    
    # 4.查看创建的角色对应的Token值
    $ kubectl get sa
      # NAME         SECRETS   AGE
      # default      1         18d
      # prometheus   1         24s
    
    $ kubectl get sa prometheus -n monitor -o yaml
      # apiVersion: v1
      # kind: ServiceAccount
      # metadata:
      #   creationTimestamp: "2021-05-10T06:10:28Z"
      #   name: prometheus
      #   namespace: default
      #   resourceVersion: "3596438"
      #   selfLink: /api/v1/namespaces/default/serviceaccounts/prometheus
      #   uid: af6a884d-2670-4f46-836e-d8ccf9fd0c38
      # secrets:
      # - name: prometheus-token-ft8bd  # secrets Token 关键点
    
    # 5.一行命令搞定
    kubectl get secret -n monitor $(kubectl get sa prometheus -n monitor -o yaml | tail -n 1 | cut -d " " -f 3) -o yaml | grep "token:" | head -n 1 | awk '{print $2}'|base64 -d > k8s_token
    
    # 6.补充k8s集群相关crt和key获取
    # client-certificate-data
    ~$ grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d > k8s_client.crt
    # client-key-data
    ~$ grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d > k8s_client.key
    scp -P20211 weiyigeek@weiyigeek-226:~/.kube/k8s_client.crt ./conf.d/ssl/
    scp -P20211 weiyigeek@weiyigeek-226:~/.kube/k8s_client.key ./conf.d/ssl/

• Step 3.参考采用官方提供的部署资源清单参考地址,此处已采用上面创建的 prometheus 用户。

  tee kube-state-metrics.yaml <<'EOF'
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    labels:
      app.kubernetes.io/name: kube-state-metrics
      app.kubernetes.io/version: 2.0.0
    name: kube-state-metrics
    namespace: kube-system
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/name: kube-state-metrics
    template:
      metadata:
        labels:
          app.kubernetes.io/name: kube-state-metrics
          app.kubernetes.io/version: 2.0.0
      spec:
        containers:
        - image: bitnami/kube-state-metrics:2.0.0
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 5
          name: kube-state-metrics
          ports:
          - containerPort: 8080
            name: http-metrics
          - containerPort: 8081
            name: telemetry
          readinessProbe:
            httpGet:
              path: /
              port: 8081
            initialDelaySeconds: 5
            timeoutSeconds: 5
          securityContext:
            runAsUser: 65534
        nodeSelector:
          kubernetes.io/os: linux
        serviceAccountName: prometheus
  ---
  apiVersion: v1
  kind: Service
  metadata:
    labels:
      app.kubernetes.io/name: kube-state-metrics
      app.kubernetes.io/version: 2.0.0
    name: kube-state-metrics
    namespace: kube-system
    annotations:
      # 注意: 此处需要进行添加到annotations(注释)便于prometheus进行自动发现。
      prometheus.io/scrape: 'true'
  spec:
    clusterIP: None
    ports:
    - name: http-metrics
      port: 8080
      targetPort: http-metrics
    - name: telemetry
      port: 8081
      targetPort: telemetry
    selector:
      app.kubernetes.io/name: kube-state-metrics
  EOF
  
  # - 创建 monitor 名称空间
  kubectl create ns monitor
  
  sed -i "s#kube-system#monitor#g" kube-state-metrics.yaml
  
  # - 部署 kube-state-metrics
  kubectl apply -f kube-state-metrics.yaml
    # deployment.apps/kube-state-metrics created
    # service/kube-state-metrics created

• Step 4.验证查看部署情况并获取认证token

  $ kubectl get pod,svc,ep -n monitor --show-labels
    # NAME                                      READY   STATUS    RESTARTS   AGE     LABELS
    # pod/kube-state-metrics-777789bc9d-9n6jf   1/1     Running   0          3m30s   app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/version=2.0.0,pod-template-hash=777789bc9d
  
    # NAME                         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE     LABELS
    # service/kube-state-metrics   ClusterIP   None         <none>        8080/TCP,8081/TCP   3m30s   app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/version=2.0.0,prometheus.io/scrape=true
  
    # NAME                           ENDPOINTS                                 AGE     LABELS
    # endpoints/kube-state-metrics   172.16.182.199:8081,172.16.182.199:8080   3m30s   app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/version=2.0.0,prometheus.io/scrape=true,service.kubernetes.io/headless=
  
  # - ca.crt 文件以及Token
  # kubectl get secret -n monitor $(kubectl get sa kube-state-metrics -n monitor -o yaml | tail -n 1 | cut -d " " -f 3) -o yaml | grep "ca.crt:" | head -n 1 | awk '{print $2}' | base64 -d > k8s_ca.crt
  kubectl get secret -n monitor $(kubectl get sa kube-state-metrics -n monitor -o yaml | tail -n 1 | cut -d " " -f 3) -o yaml | grep "token:" | head -n 1 | awk '{print $2}'| base64 -d > k8s_token

• Step 5.将获取到的k8s_ca.crt和k8s_token文件下载到prometheus主配置文件中指定的目录之中;

  ansible weiyigeek-226 -m fetch -a "src=/home/weiyigeek/prometheus/k8s_ca.crt dest=/tmp"
  ansible weiyigeek-226 -m fetch -a "src=/home/weiyigeek/prometheus/k8s_token dest=/tmp"
    # weiyigeek-226 | CHANGED => {
    #   "changed": true,
    #   "checksum": "22c40a4f83ad82343affbab3f8a732c14accbdcd",
    #   "dest": "/tmp/k8s_token/weiyigeek-226/home/weiyigeek/prometheus/k8s_kube-state-metrics_token",
    #   "md5sum": "c9d780a62db497bbfd995b548887e4ed",
    #   "remote_checksum": "22c40a4f83ad82343affbab3f8a732c14accbdcd",
    #   "remote_md5sum": null
    # }

• Step 6.配置prometheus.yml主配置文件添加kubernetes_sd_configs对象配置endpoints角色的自动发现。

  - job_name: 'k8s-endpoint-discover'
    scheme: https
    #使用apiserver授权部分解密的token值以文件形式存储
    tls_config:
      ca_file: /etc/prometheus/conf.d/auth/k8s_ca.crt
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/conf.d/auth/k8s_token
    # k8s自动发现具体配置
    kubernetes_sd_configs:
    # 使用endpoint级别自动发现
    - role: endpoints
      api_server: 'https://192.168.12.226:6443'
      tls_config:
        ca_file: /etc/prometheus/conf.d/auth/k8s_ca.crt
        insecure_skip_verify: true
      bearer_token_file: /etc/prometheus/conf.d/auth/k8s_token
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      # 只保留指定匹配正则的标签，不匹配则删除
      action: keep
      regex: '^(kube-state-metrics)$'
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      # 只保留指定匹配正则的标签，不匹配则删除
      action: keep
      regex: true
    - source_labels: [__address__]
      action: replace
      target_label: instance
    - target_label: __address__
      # 使用replacement值替换__address__默认值
      replacement: 192.168.12.226:6443
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
      # 正则匹配
      regex: ([^;]+);([^;]+);([^;]+)
      # 使用replacement值替换__metrics_path__默认值
      target_label: __metrics_path__
      # 自行构建的apiserver proxy url
      replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      # 将标签__meta_kubernetes_namespace修改为kubernetes_namespace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      # 将标签__meta_kubernetes_service_name修改为service_name
      target_label: service_name

Tips: 通过relabel_configs构造 prometheus (endpoints) Role 访问 API Server 的 URL;

标签	默认	构造后
__scheme__	https	https

__address__ | 172.16.182.200:8081 |192.168.12.226:6443|
__metrics_path__ | /metrics | /api/v1/namespaces/kube-system/pods/http:kube-state-metrics-6477678b78-6qkjg:8081/proxy/metrics |
|URL| https://10.244.2.10:8081/metrics | https://192.168.12.226:6443/api/v1/namespaces/kube-system/pods/http:kube-state-metrics-6477678b78-6qkjg:8081/proxy/metrics |

• Step 6.修改主配置文件完成后进行重启 prometheus Server 容器然后查看启动状态,由图中可以看见监控成功。

  # k8s-endpoint-discover (2/2 up)
  # - Discovered Labels
  __meta_kubernetes_endpoint_address_target_kind="Pod"
  __meta_kubernetes_endpoint_address_target_name="kube-state-metrics-6477678b78-6qkjg"
  __meta_kubernetes_endpoint_node_name="weiyigeek-226"
  __meta_kubernetes_endpoint_port_name="telemetry"
  __meta_kubernetes_endpoint_port_protocol="TCP"
  __meta_kubernetes_endpoint_ready="true"
  __meta_kubernetes_endpoints_label_app_kubernetes_io_name="kube-state-metrics"
  __meta_kubernetes_endpoints_label_app_kubernetes_io_version="2.0.0"
  __meta_kubernetes_endpoints_labelpresent_app_kubernetes_io_name="true"
  __meta_kubernetes_endpoints_labelpresent_app_kubernetes_io_version="true"
  __meta_kubernetes_endpoints_labelpresent_service_kubernetes_io_headless="true"
  __meta_kubernetes_endpoints_name="kube-state-metrics"
  __meta_kubernetes_namespace="monitor"
  __meta_kubernetes_pod_annotation_cni_projectcalico_org_podIP="172.16.182.200/32"
  __meta_kubernetes_pod_annotation_cni_projectcalico_org_podIPs="172.16.182.200/32"
  __meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIP="true"
  __meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIPs="true"
  __meta_kubernetes_pod_container_name="kube-state-metrics"
  __meta_kubernetes_pod_container_port_name="telemetry"
  __meta_kubernetes_pod_container_port_number="8081"
  __meta_kubernetes_pod_container_port_protocol="TCP"
  __meta_kubernetes_pod_controller_kind="ReplicaSet"
  __meta_kubernetes_pod_controller_name="kube-state-metrics-6477678b78"
  __meta_kubernetes_pod_host_ip="192.168.12.226"
  __meta_kubernetes_pod_ip="172.16.182.200"
  __meta_kubernetes_pod_label_app_kubernetes_io_name="kube-state-metrics"
  __meta_kubernetes_pod_label_app_kubernetes_io_version="2.0.0"
  __meta_kubernetes_pod_label_pod_template_hash="6477678b78"
  __meta_kubernetes_pod_labelpresent_app_kubernetes_io_name="true"
  __meta_kubernetes_pod_labelpresent_app_kubernetes_io_version="true"
  __meta_kubernetes_pod_labelpresent_pod_template_hash="true"
  __meta_kubernetes_pod_name="kube-state-metrics-6477678b78-6qkjg"
  __meta_kubernetes_pod_node_name="weiyigeek-226"
  __meta_kubernetes_pod_phase="Running"
  __meta_kubernetes_pod_ready="true"
  __meta_kubernetes_pod_uid="70037554-7c4c-4372-9128-e9689b7cff10"
  __meta_kubernetes_service_annotation_kubectl_kubernetes_io_last_applied_configuration="{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"prometheus.io/scrape":"true"},"labels":{"app.kubernetes.io/name":"kube-state-metrics","app.kubernetes.io/version":"2.0.0"},"name":"kube-state-metrics","namespace":"monitor"},"spec":{"clusterIP":"None","ports":[{"name":"http-metrics","port":8080,"targetPort":"http-metrics"},{"name":"telemetry","port":8081,"targetPort":"telemetry"}],"selector":{"app.kubernetes.io/name":"kube-state-metrics"}}} "
  __meta_kubernetes_service_annotation_prometheus_io_scrape="true"
  __meta_kubernetes_service_annotationpresent_kubectl_kubernetes_io_last_applied_configuration="true"
  __meta_kubernetes_service_annotationpresent_prometheus_io_scrape="true"
  __meta_kubernetes_service_label_app_kubernetes_io_name="kube-state-metrics"
  __meta_kubernetes_service_label_app_kubernetes_io_version="2.0.0"
  __meta_kubernetes_service_labelpresent_app_kubernetes_io_name="true"
  __meta_kubernetes_service_labelpresent_app_kubernetes_io_version="true"
  __meta_kubernetes_service_name="kube-state-metrics"
  __metrics_path__="/metrics"
  __scheme__="https"
  job="k8s-endpoint-discover"
  # Target Labels
  app_kubernetes_io_name="kube-state-metrics"
  app_kubernetes_io_version="2.0.0"
  instance="172.16.182.200:8081"
  job="k8s-endpoint-discover"
  kubernetes_namespace="monitor"
  service_name="kube-state-metrics"
  
  # PromQL 表达式
  up{job="k8s-endpoint-discover"} or go_info{job="k8s-endpoint-discover"}
    # up{app_kubernetes_io_name="kube-state-metrics", app_kubernetes_io_version="2.0.0", instance="172.16.182.199:8080", job="k8s-endpoint-discover", kubernetes_namespace="monitor", service_name="kube-state-metrics"} 1
    # up{app_kubernetes_io_name="kube-state-metrics", app_kubernetes_io_version="2.0.0", instance="172.16.182.199:8081", job="k8s-endpoint-discover", kubernetes_namespace="monitor", service_name="kube-state-metrics"} 1
    # go_info{app_kubernetes_io_name="kube-state-metrics", app_kubernetes_io_version="2.0.0", instance="172.16.182.199:8081", job="k8s-endpoint-discover", kubernetes_namespace="monitor", service_name="kube-state-metrics", version="go1.16.3"} 1

weiyigeek.top-k8s-endpoint-discover

补充说明: metrics-server 和 kube-state-metrics对比

类别	metrics-server	kube-state-metrics
简单介绍	Metrics Server通过Metrics API公开核心Kubernetes度量	kube state metrics是关于从Kubernetes API对象生成度量而不需要修改,确保了kube状态度量提供的特性具有与kubernetesapi对象本身相同的稳定性。
监控对象	监控Node和Pod等CPU、内存、网络等系统指标	关注Node,Deployment,Pod,Services,Namespace等内部对象的状态
项目地址	https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server (已丢失) https://github.com/kubernetes-sigs/metrics-server/ (建议)	https://github.com/kubernetes/kube-state-metrics
服务端口	443	8080

示例: kube-state-metrics 收集到的节点信息, 如验证指标是否采集成功请求kube-state-metrics的pod ip+8080端口出现以下页面则正常

$ kube_node_info{job="k8s-endpoint-discover"}
# kube_node_info{app_kubernetes_io_name="kube-state-metrics", app_kubernetes_io_version="2.0.0", container_runtime_version="docker://19.3.15", instance="172.16.182.200:8080", internal_ip="192.168.12.226", job="k8s-endpoint-discover", kernel_version="5.4.0-73-generic", kubelet_version="v1.19.10", kubeproxy_version="v1.19.10", kubernetes_namespace="monitor", node="weiyigeek-226", os_image="Ubuntu 20.04.2 LTS", pod_cidr="172.16.0.0/24", service_name="kube-state-metrics"}

kube-state-metrics

2.Node 之服务自动发现

描述: 通过node-exporter采集集群node节点的服务器层面的数据，如cpu、内存、磁盘、网络流量等,当然node-exporter可以独立部署在node节点服务器上但是每次都要进行手动配置添加监控是非常不方便。

流程步骤:

• Step 1.此处将node-exporter以DaemonSet形式部署，配合Prometheus动态发现更加方便。

  tee node-exporter.yaml <<'EOF'
  kind: DaemonSet
  apiVersion: apps/v1
  metadata:
    name: node-exporter
    namespace: monitor
    annotations:
      prometheus.io/scrape: 'true'
  spec:
    selector:
      matchLabels:
        app: node-exporter
    template:
      metadata:
        labels:
          app: node-exporter
        name: node-exporter
      spec:
        containers:
        - image: prom/node-exporter:v1.1.2
          name: node-exporter
          ports:
          - containerPort: 9100
            hostPort: 9100
            name: node-exporter
        hostNetwork: true
        hostPID: true
        tolerations:
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
  ---
  kind: Service
  apiVersion: v1
  metadata:
    name: node-exporter
    namespace: monitor
    labels:
      app: node-exporter
    annotations:
      prometheus.io/scrape: 'true'
  spec:
    type: ClusterIP
    clusterIP: None
    ports:
    - name: node-exporter
      port: 9100
      protocol: TCP
    selector:
      app: node-exporter
  EOF
  
  ~$ kubectl apply -f node-exporter.yaml
    # daemonset.apps/node-exporter created
  
  ~$ kubectl get pod -n monitor
    # NAME                  READY   STATUS    RESTARTS   AGE
    # node-exporter-p5tbp   1/1     Running   0          20s

• Step 2.创建SA账户并对其进行RBAC权限设置(最小权限原则)

  $ kubectl create sa prometheus -n monitor
    # serviceaccount/prometheus created
  
  # 集群角色 RBAC 权限申明
  tee prometheus-clusterRole.yaml <<'EOF'
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: prometheus
    namespace: monitor
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes
    - services
    - endpoints
    - pods
    - nodes/proxy
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - "extensions"
    resources:
      - ingresses
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - configmaps
    - nodes/metrics
    verbs:
    - get
  - nonResourceURLs:
    - /metrics
    verbs:
    - get
  EOF
  
  # 集群角色权限
  $ kubectl create -f prometheus-clusterRole.yaml
  # clusterrole.rbac.authorization.k8s.io/prometheus created
  # 集群角色绑定
  $ kubectl create clusterrolebinding prometheus --clusterrole prometheus --serviceaccount=monitor:prometheus
  # 或者一步搞定(此处是上面一步得yaml资源清单)
  # apiVersion: rbac.authorization.k8s.io/v1beta1
  # kind: ClusterRoleBinding
  # metadata:
  #   name: prometheus
  # roleRef:
  #   apiGroup: rbac.authorization.k8s.io
  #   kind: ClusterRole
  #   name: prometheus
  # subjects:
  # - kind: ServiceAccount
  #   name: prometheus
  #   namespace: monitor
  
  
  # 获取认证的Token
  kubectl get secret -n monitor $(kubectl get sa prometheus -n monitor -o yaml | tail -n 1 | cut -d " " -f 3) -o yaml | grep "token:" | head -n 1 | awk '{print $2}'| base64 -d > k8s_prometheuser_token
  
  # 将k8s_prometheuser_token下载到prometheus服务器中
  ansible weiyigeek-226 -m fetch -a "src=/home/weiyigeek/prometheus/k8s_prometheuser_token dest=/tmp"
  # weiyigeek-226 | CHANGED => {
  #   "changed": true,
  #   "checksum": "d4a16cebda1b6037dcb68004d0ff4cdf4079bbc5",
  #   "dest": "/tmp/weiyigeek-226/home/weiyigeek/prometheus/k8s_prometheuser_token",
  #   "md5sum": "bdcd6c4a77ab6ee2afa5ac6f78ddb94a",
  #   "remote_checksum": "d4a16cebda1b6037dcb68004d0ff4cdf4079bbc5",
  #   "remote_md5sum": null
  # }

• Step 3.Prometheus.yaml 主配置文件添加kubernetes_sd_configs对象使用node级别自动发现;

  - job_name: 'k8s-nodes-discover'
    scheme: https
    # 使用apiserver授权部分解密的token值，以文件形式存储
    tls_config:
      # ca_file: /etc/prometheus/conf.d/auth/k8s_kube-state-metrics_ca.crt
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/conf.d/auth/k8s_prometheuser_token
    # k8s自动发现具体配置
    kubernetes_sd_configs:
    # 使用node级别自动发现
    - role: node
      api_server: 'https://192.168.12.226:6443'
      tls_config:
        # ca_file: /etc/prometheus/conf.d/auth/k8s_kube-state-metrics_ca.crt
        insecure_skip_verify: true
      bearer_token_file: /etc/prometheus/conf.d/auth/k8s_prometheuser_token
    relabel_configs:
    #- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      # 只保留指定匹配正则的标签，不匹配则删除
      #action: keep
      #regex: true
    - target_label: __address__
      # 使用replacement值替换__address__默认值
      replacement: 192.168.12.226:6443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      # 使用replacement值替换__metrics_path__默认值 , 如果采用默认的kubelet进行数据的采集
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}:9100/proxy/metrics
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      # 将标签__meta_kubernetes_service_name修改为service_name
      target_label: service_name
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      # 将标签__meta_kubernetes_namespace修改为kubernetes_namespace
      target_label: kubernetes_namespace

Tips: 通过relabel_configs构造 prometheus (node) Role 访问 API Server 的 URL;

标签	默认	构造后
__scheme__	https	https
__address__	192.168.3.217:10250	192.168.3.217:6443
__metrics_path__ (node_exporter)	/metrics	/api/v1/nodes/uvmsvr-3-217:9100/proxy/metrics
URL	https://192.168.3.217:10250/metrics	https://192.168.3.217:6443/api/v1/nodes/uvmsvr-3-217:9100/proxy/metrics
__metrics_path__ (kubelet)	/metrics	/api/v1/nodes/uvmsvr-3-217:10250/proxy/metrics
URL	https://192.168.3.217:10250/metrics	https://192.168.3.217:6443/api/v1/nodes/uvmsvr-3-217:10250/proxy/metrics/cadvisor
__metrics_path__ (advisor)	/metrics	/api/v1/nodes/uvmsvr-3-217:10250/proxy/metrics
URL	https://192.168.3.217:10250/metrics	https://192.168.3.217:6443/api/v1/nodes/uvmsvr-3-217:10250/proxy/metrics/cadvisor

• Step 4.重启服务查看监控目标状态以及服务发现是否成功监控。

  # (1) 该Job的状态信息
  k8s-nodes-discover (1/1 up)
  Endpoint	State	Labels	Last Scrape	Scrape Duration	Error
  https://192.168.12.226:6443/api/v1/nodes/weiyigeek-226:9100/proxy/metrics	UP instance="weiyigeek-226"job="k8s-nodes-discover"
  
  # (2) PromQL 表达式查询
  up{job="k8s-nodes-discover"} or go_info{job="k8s-nodes-discover"}
  # up{instance="weiyigeek-226", job="k8s-nodes-discover"}	1
  # go_info{instance="weiyigeek-226", job="k8s-nodes-discover", version="go1.15.8"} 1

weiyigeek.top-k8s-nodes-discover-9100

• Step 5.此时我们可以将__metrics_path__替换成/api/v1/nodes/${1}:10250/proxy/metrics,如此便采用了kubelet采集拉取监控指标。

weiyigeek.top-k8s-nodes-discover-10250

3.综合实践之(cAdvisor+Kube-state-metrics+Grafana)组合拳方案

描述: Grafana从prometheus数据源读取监控指标并进行图形化,根据其官网提供的众多模板,我们可以针对不同维度的监控指标，我们可以自行选择喜欢的模板直接导入Dashboard id使用。

例如:以下针对于不同场景采用的不同的Dashboard面板:

• 1.Node 性能监控展示
  • https://grafana.com/grafana/dashboards/8919
  • https://grafana.com/grafana/dashboards/9276
• 2.pod 性能监控展示
  • https://grafana.com/grafana/dashboards/8588
• 3.K8S 资源性能监控展示(kubernetes资源全面展示！包含K8S整体资源总览、微服务资源明细、Pod资源明细及K8S网络带宽，优化重要指标展示。)
  • https://grafana.com/grafana/dashboards/13105

实践目标: 使用cadvisor采集Pod容器相关信息+使用kube-state-metrics采集集群相关信息+使用Grafana将Prometheus采集到的数据进行展示。

流程步骤:

• Step 1.在前面的基础环境之上修改的 Prometheus.yaml 主配置文件内容如下:

  tee prometheus.yaml <<'EOF'
  global:
    scrape_interval: 2m
    scrape_timeout: 10s
    evaluation_interval: 1m
    external_labels:
      monitor: 'prom-demo'
  
  alerting:
    alertmanagers:
    - scheme: http
      static_configs:
      - targets:
        - '192.168.12.107:30093'
  
  rule_files:
    - /etc/prometheus/conf.d/rules/*.rules
  
  scrape_configs:
    - job_name: 'prom-Server'
      static_configs:
        - targets: ['localhost:9090']
    - job_name: 'cAdvisor'
      static_configs:
        - targets: ['192.168.12.111:9100']
    - job_name: 'linux_exporter'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/k8s_nodes.yaml
        refresh_interval: 1m
    - job_name: 'windows-exporter'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/win_nodes.yaml
        refresh_interval: 1m
    - job_name: 'mysql_discovery'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/mysql_discovery.yaml
    - job_name: 'redis_discovery'
      file_sd_configs:
      - files:
        - /etc/prometheus/conf.d/discovery/redis_discovery.yaml
    - job_name: 'k8s-endpoint-discover'
      scheme: https
      #使用apiserver授权部分解密的token值，以文件形式存储
      tls_config: 
        #ca_file: /etc/prometheus/conf.d/auth/k8s_kube-state-metrics_ca.crt
        insecure_skip_verify: true
      bearer_token_file: /etc/prometheus/conf.d/auth/k8s_token
      # k8s自动发现具体配置
      kubernetes_sd_configs:
      # 使用endpoint级别自动发现
      - role: endpoints
        api_server: 'https://192.168.12.226:6443'
        tls_config:
          # ca_file: /etc/prometheus/conf.d/auth/k8s_kube-state-metrics_ca.crt
          insecure_skip_verify: true
        bearer_token_file: /etc/prometheus/conf.d/auth/k8s_token
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        # 只保留指定匹配正则的标签，不匹配则删除
        action: keep
        regex: '^(kube-state-metrics)$'
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        #只保留指定匹配正则的标签，不匹配则删除
        action: keep
        regex: true
      - source_labels: [__address__]
        action: replace
        target_label: instance
      - target_label: __address__
        # 使用replacement值替换__address__默认值
        replacement: 192.168.12.226:6443
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
        # 正则匹配
        regex: ([^;]+);([^;]+);([^;]+)
        # 使用replacement值替换__metrics_path__默认值
        target_label: __metrics_path__
        # 自行构建的apiserver proxy url
        replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        # 将标签__meta_kubernetes_namespace修改为kubernetes_namespace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        # 将标签__meta_kubernetes_service_name修改为service_name
        target_label: service_name
      
    - job_name: 'k8s-cadvisor'
      scheme: https
      # 使用apiserver授权部分解密的token值，以文件形式存储
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /etc/prometheus/conf.d/auth/k8s_token
      metrics_path: /metrics/cadvisor
      kubernetes_sd_configs:
      - role: node
        api_server: 'https://192.168.12.226:6443'
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /etc/prometheus/conf.d/auth/k8s_token
      relabel_configs:
      - source_labels: [__address__]
        action: replace
        target_label: instance
      - target_label: __address__
        # 使用replacement值替换__address__默认值
        replacement: 192.168.12.226:6443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        # 使用replacement值替换__metrics_path__默认值 , 如果采用默认的kubelet进行数据的采集
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}:10250/proxy/metrics/cadvisor
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      metric_relabel_configs:
      - source_labels: [instance]
        separator: ;
        regex: (.+)
        target_label: node
        replacement: $1
        action: replace
      - source_labels: [pod_name]
        separator: ;
        regex: (.+)
        target_label: pod
        replacement: $1
        action: replace
      - source_labels: [container_name]
        separator: ;
        regex: (.+)
        target_label: container
        replacement: $1
        action: replace
      - source_labels: [origin_prometheus]
        separator: ;
        regex: (.+)
        target_label: node
        replacement: $1
        action: replace
  EOF

• Step 2.关键配置说明由于此处我们的Prometheus是在k8s集群外部署的所以需要重新构建__metrics_path__字符串以便代理访问。

  # - k8s-cAdvisor
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    # 使用replacement值替换__metrics_path__默认值 , 如果采用默认的kubelet进行数据的采集
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}:10250/proxy/metrics/cadvisor
  
  # - kube-state-metrics
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
    # 正则匹配
    regex: ([^;]+);([^;]+);([^;]+)
    # 使用replacement值替换__metrics_path__默认值
    target_label: __metrics_path__
    # 自行构建的apiserver proxy url
    replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics

• Step 3.重启我们的Prometheus服务并验证服务发现和目标。

  # - k8s-cadvisor (1/1 up) : https://192.168.12.226:6443/api/v1/nodes/weiyigeek-226:10250/proxy/metrics/cadvisor
  
  # - k8s-endpoint-discover (2/2 up)
  # https://192.168.12.226:6443/api/v1/namespaces/monitor/pods/http:kube-state-metrics-6477678b78-6qkjg:8080/proxy/metrics
  # https://192.168.12.226:6443/api/v1/namespaces/monitor/pods/http:kube-state-metrics-6477678b78-6qkjg:8081/proxy/metrics

  

  weiyigeek.top-k8s-cadvisor

• Step 4.在Grafana中安装Kubernetes for Prometheus Dashboard(https://grafana.com/grafana/dashboards/13105),它提供包含K8S整体资源总览、微服务资源明细、Pod资源明细及K8S网络带宽，优化重要指标展示。

weiyigeek.top-cadvisor+Dashboard

• Step 5.至此完毕此项实践。

Tips : 通过 Dashboard 模板我们需要自行选择并组合, 灵活有余但规范不足, 我们常常使用grafana专门针对Kubernetes集群监控的插件grafana-kubernetes-app它包括4个仪表板，集群，节点，Pod /容器和部署,但由于其插件作者没有更新维护,所以更多是采用KubeGraf,该插件可以用来可视化和分析 Kubernetes 集群的性能，通过各种图形直观的展示了 Kubernetes 集群的主要服务的指标和特征，还可以用于检查应用程序的生命周期和错误日志。