[TOC]

0x01 安装部署

官方项目地址: https://github.com/elastic/helm-charts

(1) Helm 安装

安装流程

  • Step 1.安装环境以及Charts准备;

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    # (1) 添加 elastic 仓库
    ~/K8s/Day11$ helm repo add elastic https://helm.elastic.co

    # (2) 拉取 elasticsearch / Filebeat / Kibana 等 Chart 到本地并解压
    ~/K8s/Day11$ helm fetch elastic/elasticsearch --untar
    ~/K8s/Day11$ helm fetch elastic/filebeat --untar
    ~/K8s/Day11$ helm fetch elastic/kibana --untar
    ~/K8s/Day11$ tree -d 1 .
    # ├── elasticsearch
    # │   ├── examples
    # │   │   ├── config
    # │   │   │   └── test
    # │   │   ├── default
    # │   │   │   └── test
    # │   │   ├── docker-for-mac
    # │   │   ├── kubernetes-kind
    # │   │   ├── microk8s
    # │   │   ├── migration
    # │   │   ├── minikube
    # │   │   ├── multi
    # │   │   │   └── test
    # │   │   ├── openshift
    # │   │   │   └── test
    # │   │   ├── oss
    # │   │   │   └── test
    # │   │   ├── security
    # │   │   │   └── test
    # │   │   └── upgrade
    # │   │   └── test
    # │   └── templates
    # │   └── test
    # ├── filebeat
    # │   ├── examples
    # │   │   ├── default
    # │   │   │   └── test
    # │   │   ├── oss
    # │   │   │   └── test
    # │   │   ├── security
    # │   │   │   └── test
    # │   │   └── upgrade
    # │   │   └── test
    # │   └── templates
    # └── kibana
    # ├── examples
    # │   ├── default
    # │   │   └── test
    # │   ├── openshift
    # │   │   └── test
    # │   ├── oss
    # │   │   └── test
    # │   ├── security
    # │   │   └── test
    # │   └── upgrade
    # │   └── test
    # └── templates

    # 47 directories
  • Step 2.名称空间创建(暂不使用)以及elasticsearch的安装部署

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    # (1) 名称空间的创建 以及存储卷访问模式为 ReadWriteOnce - efk-pv
    ~/K8s/Day11/elasticsearch$ kubectl create namespace log-efk
    # PS : 此处需要依赖于StorageName动态存储卷,所以您需要提前进行配置;


    # (2) 由于测试环境资源有限我们将Charts中Values关于副本数以及最小master节点都该成(3Master节点~20G内存)
    ~/K8s/Day11/elasticsearch$ vim values.yaml
    # 修改点
    replicas: 1
    minimumMasterNodes: 1
    # PS: 生产环境中推荐修改values.yaml文件中pv为storageClass动态分配
    volumeClaimTemplate:
    accessModes: [ "ReadWriteOnce" ]
    storageClassName: "managed-nfs-storage" # 非常重要 它是与 StorageClass 资源器绑定的名称要一致
    resources:
    requests:
    storage: 20Gi
    # PS :开启持久化
    persistence:
    enabled: true
    labels:
    enabled: true # 为有状态集的volumeClaimTemplate添加默认标签 ( 坑 )
    annotations: {}


    # (3) 将elasticsearch进行安装部署 (-n log-efk) 不能正常部署 (坑)
    ~/K8s/Day11/elasticsearch$ helm install elasticsearch .
    # NAME: elasticsearch
    # LAST DEPLOYED: Wed Dec 9 09:52:35 2020
    # NAMESPACE: log-efk
    # STATUS: deployed
    # REVISION: 1
    # NOTES:
    # 1. Watch all cluster members come up.
    # $ kubectl get pods-l app=elasticsearch-master -w
    # 2. Test cluster health using Helm test.
    # $ helm test elasticsearch
    # * Install it:
    # - with Helm 3: `helm install elasticsearch --version <version> elastic/elasticsearch`
    # - with Helm 2 (deprecated): `helm install --name elasticsearch --version <version> elastic/elasticsearch`
  • Step 3.查看StatefulSete资源控制器创建的elasticsearch以及PVC卷的绑定和SVC;

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    # StatefulSet
    ~/K8s/Day11/elasticsearch$ kubectl get sts -o wide
    # NAME READY AGE CONTAINERS IMAGES
    # elasticsearch-master 1/1 2m57s elasticsearch docker.elastic.co/elasticsearch/elasticsearch:7.10.0

    # Pod
    ~/K8s/Day11/elasticsearch$ kubectl get pod -o wide --show-labels | grep "elasticsearch"
    # elasticsearch-master-0 1/1 Running 0 3m30s 10.244.1.180 k8s-node-4 app=elasticsearch-master,chart=elasticsearch,controller-revision-hash=elasticsearch-master-76c48b9f74,release=elasticsearch,statefulset.kubernetes.io/pod-name=elasticsearch-master-0

    # Svc
    ~/K8s/Day11/elasticsearch$ kubectl get svc -o wide
    # NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
    # elasticsearch-master ClusterIP 10.104.178.144 <none> 9200/TCP,9300/TCP 9m49s app=elasticsearch-master,chart=elasticsearch,release=elasticsearch
    # elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 9m49s app=elasticsearch-master

    # StorageClass 动态存储卷查看
    ~/K8s/Day11/elasticsearch$ kubectl get storageclass,pv,pvc
    # NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
    # storageclass.storage.k8s.io/managed-nfs-storage (default) fuseim.pri/ifs Delete Immediate false 25h

    # NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
    # persistentvolume/pvc-8a991669-60c0-48c6-9879-5b4fa9d481c3 20Gi RWO Delete Bound default/elasticsearch-master-elasticsearch-master-0 managed-nfs-storage 2m6s

    # NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    # persistentvolumeclaim/elasticsearch-master-elasticsearch-master-0 Bound pvc-8a991669-60c0-48c6-9879-5b4fa9d481c3 20Gi RWO managed-nfs-storage 2m6s
  • Step 4.Helm安装filebeat查看并验证

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    # vim filebeat/value.yaml
    # Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).
    # 默认读取的是节点/var/lib下的所有文件
    hostPathRoot: /var/lib

    # 安装 filbeat
    ~/K8s/Day11/filebeat$ helm install filbeat .
    # NAME: filbeat
    # LAST DEPLOYED: Thu Dec 10 17:09:53 2020
    # NAMESPACE: default
    # STATUS: deployed
    # REVISION: 1
    # TEST SUITE: None
    # NOTES:
    # 1. Watch all containers come up.
    # $ kubectl get pods --namespace=default -l app=filbeat-filebeat -w

    # 查看状态,因为是DaemonSet类型所以每台node都会装一个。
    ~/K8s/Day11/filebeat$ kubectl get pods --namespace=default -l app=filbeat-filebeat -o wide
    # NAME READY STATUS RESTARTS AGE IP NODE
    # filbeat-filebeat-2dr52 1/1 Running 0 4h39m 10.244.2.78 k8s-node-5
    # filbeat-filebeat-mgj5r 1/1 Running 0 58m 10.244.0.190 master
    # filbeat-filebeat-qm7wt 1/1 Running 0 4h39m 10.244.1.181 k8s-node-4
  • Step 5.Helm 安装 kibana 以及验证

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    vim kibana/values.yaml
    # 修改values.yaml文件中service为nodePort类型
    service:
    type: NodePort # 修改点
    loadBalancerIP: ""
    port: 5601
    nodePort: 30002
    labels: {} # 修改点
    annotations: {}
    # cloud.google.com/load-balancer-type: "Internal"
    # service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    # service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    # service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
    # service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
    loadBalancerSourceRanges: []
    # 0.0.0.0/0

    # 安装 Kibana
    ~/K8s/Day11/kibana$ helm install kibana .
    # NAME: kibana
    # LAST DEPLOYED: Thu Dec 10 20:16:13 2020
    # NAMESPACE: default
    # STATUS: deployed
    # REVISION: 1
    # TEST SUITE: None

    # 查看 deployment 和 Pod
    ~/K8s/Day11/filebeat$ kubectl get deploy,pod -o wide -l app=kibana
    # NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
    # deployment.apps/kibana-kibana 1/1 1 1 52m kibana docker.elastic.co/kibana/kibana:7.10.0 app=kibana,release=kibana

    # NAME READY STATUS RESTARTS AGE IP NODE
    # pod/kibana-kibana-86d4cc75f7-gbmjt 1/1 Running 0 28m 10.244.0.192 master

    # 查看 svc
    ~/K8s/Day11/filebeat$ kubectl get svc -l app=kibana
    # NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    # kibana-kibana NodePort 10.110.225.148 <none> 5601:30002/TCP 54m
  • Step 6.利用 Helm 查看安装的 EFK RELEASE,访问 kibana URL 并创建索引: http://10.10.107.202:30002/app/management/kibana/indexPatterns

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    ~/K8s/Day11/filebeat$ helm ls
    # NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
    # elasticsearch default 1 2020-12-10 16:51:42.74817589 +0800 CST deployed elasticsearch-7.10.0 7.10.0
    # filbeat default 1 2020-12-10 17:09:53.872762268 +0800 CST deployed filebeat-7.10.0 7.10.0
    # kibana default 1 2020-12-10 21:01:17.44414926 +0800 CST deployed kibana-7.10.0 7.10.0
    ```
    Process : Stack Management > Index patterns -> Create `创建索引` -> Next Step -> Time field `按照时间进行分片`

    ![WeiyiGeek.索引创建](https://cdn.jsdelivr.net/gh/Weiyigeek/blogimage/2020/2/20201210221822.png)


    - Step 7.Elasticsearch 的 Kibana 的简单使用;
    例如: 按照Node节点查询`kubernetes.node.name : "k8s-node-4" or kubernetes.pod.name : "nfs-client-provisioner-58b5dc958d-5fwl9" `
    ![WeiyiGeek.Kibana查询日志](https://cdn.jsdelivr.net/gh/Weiyigeek/blogimage/2020/2/20201210222216.png)

    入坑&出坑

    问题1.Helm部署kibana无法正常Run显示`Readiness probe failed: Error: Got HTTP code 503 but expected a 200` (该问题把我心态搞得炸裂)
    错误信息: 学习排错也是一种学习;
    ```bash
    # Pod 描述信息
    ~/K8s/Day11/kibana$ kubectl describe pod kibana-kibana-86d4cc75f7-n7kxr
    # Normal Started 3m43s kubelet Started container kibana
    # Warning Unhealthy 0s (x22 over 3m30s) kubelet Readiness probe failed: Error: Got HTTP code 503 but expected a 200

    # Pod 日志信息 (关键点) 发现 resource_already_exists_exception
    ~/K8s/Day11/kibana$ kubectl logs kibana-kibana-86d4cc75f7-n7kxr
    # {"type":"log","@timestamp":"2020-12-10T13:01:32Z","tags":["error","elasticsearch","data"],"pid":7,"message":"[resource_already_exists_exception]: index [.kibana_task_manager_1/WikBnfj8QjCVtpuit2NxwA] already exists"}
    # {"type":"log","@timestamp":"2020-12-10T13:01:32Z","tags":["warning","savedobjects-service"],"pid":7,"message":"Unable to connect to Elasticsearch. Error: resource_already_exists_exception"}

    问题原因: 由于在elasticsearch中已经存在该.kibana_task_manager_1/WikBnfj8QjCVtpuit2NxwA索引导致错误,不能正常启动;
    解决方法: 在 elasticsearch 中 删除 Kibana 所有的Index;

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    ~/K8s/Day11/kibana$ curl "http://10.104.178.144:9200/_cat/indices/*?v&s=index"
    # health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    # green open .kibana_1 HiroxArKS3CO_j_cmiJXNg 1 0 0 0 208b 208b
    # green open .kibana_task_manager_1 WikBnfj8QjCVtpuit2NxwA 1 0 0 0 208b 208b
    # yellow open filebeat-7.10.0-2020.12.10-000001 Vz2L-FXgQSKMuYMrBMum3w 1 1 238934 0 42.1mb 42.1mb
    ~/K8s/Day11/kibana$ curl -XDELETE http://10.104.178.144:9200/.kibana*
    # {"acknowledged":true}

    # 重新构建Pod一切正常
    ~/K8s/Day11/kibana$ kubectl delete pod kibana-kibana-86d4cc75f7-n7kxr
    pod "kibana-kibana-86d4cc75f7-n7kxr" deleted
    {"type":"response","@timestamp":"2020-12-10T13:26:19Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":19,"contentLength":9},"message":"GET /app/kibana 200 19ms - 9.0B"}

    参考地址: https://discuss.elastic.co/t/unable-to-connect-to-elasticsearch-error-resource-already-exists-exception-after-upgrade-elk/249998