[TOC]

0x00 前言导读

Q: 什么是 Web UI (Dashboard) ?

答: Kubernetes Dashboard 是一个通用的、基于web的Kubernetes集群UI。它允许用户管理在集群中运行的应用程序并对它们进行故障排除，以及管理集群本身。

Q: 为什么要使用 Dashboard?

答: 您可以使用 Dashboard 来概述集群上运行的应用程序，以及创建或修改单个Kubernetes资源（例如Deployments，Jobs，DaemonSets等）例如，您可以使用部署向导来扩展部署，启动滚动更新，重新启动Pod或部署新应用程序。
并且仪表板还提供有关集群中Kubernetes资源状态以及可能发生的任何错误的信息。

附录：
K8s官网介绍: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
项目地址: https://github.com/kubernetes/dashboard

0x01 安装部署

(1) 环境准备

Kubernetes 环境: 集群环境实现业务高可用以及快速扩容缩

~$ kubectl get nodes
NAME       STATUS   ROLES    AGE   VERSION
weiyigeek-107   Ready    master   25h   v1.19.6
weiyigeek-108   Ready    master   25h   v1.19.6
weiyigeek-109   Ready    master   25h   v1.19.6
weiyigeek-223   Ready    <none>   17h   v1.19.6
weiyigeek-224   Ready    <none>   17h   v1.19.6
weiyigeek-225   Ready    <none>   17h   v1.19.6
weiyigeek-226   Ready    <none>   17h   v1.19.6

(2) 安装流程

描述: 我们可以通过原生的dashboardyaml资源清单文件或者helm的方式进行安装

官方 安装方式

安装参考: https://github.com/kubernetes/dashboard/blob/master/docs/user/installation.md

• Step 1.要部署仪表板请执行以下命令：

  $ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml -O dashboard-v2.1.0.yaml

• Step 2.默认情况下会生成自签名证书并将其存储在内存中,而如果您想使用自定义证书请按照以下步骤操作
  PS: 访问Dashboard应使用有效证书来建立安全的HTTPS连接(可以使用公共信任的证书颁发机构如Let's Encrypt生成它们,或者Cert-Manager可以自动颁发和自动更新它们)

# (1) 自定义证书必须存储在 kubernetes-dashboard-certs 与 Kubernetes 仪表板创建的名称空间中的secret中。
# 假设您已将tls.crt和tls.key文件存储在$HOME/certs目录下，则应使用以下文件的内容创建密钥：
kubectl create secret generic kubernetes-dashboard-certs --from-file=$HOME/certs -n kubernetes-dashboard

# (2) 编辑YAML定义并部署仪表板
kubectl create --edit -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml

# (3) 在“部署”部分下，将参数添加到pod定义中，其外观应如下所示：
  containers:
  - args:
    - --tls-cert-file=/tls.crt
    - --tls-key-file=/tls.key
    # 或者
    # - --auto-generate-certificates

• Step 3.仪表板仅通过HTTP公开不使用证书方式部署（不推荐）

  kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/alternative.yaml

• Step 4.权限修改

  # (1) kubernetes-dashboard 管理员角色添加修改
  cat > dashboard-role-admin.yaml <<'EOF'
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: kubernetes-dashboard
    namespace: kube-system
  ---
  kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: kubernetes-dashboard
  # 权限来源
  roleRef:
    kind: ClusterRole
    name: cluster-admin
    apiGroup: rbac.authorization.k8s.io
  # 绑定对象
  subjects:
    - kind: ServiceAccount
      name: kubernetes-dashboard
      namespace: kube-system
  EOF
  kubectl apply -f dashboard-role-admin.yaml
  
  # (2) dashboard 资源清单构建
  kubectl create -f dashboard-v2.1.0.yaml

• Step 5.集群网络访问端口修改

  # (1) 修改 SVC 访问方式为 NodePort (30443)
  ~/k8s/dashboard$ kubectl edit svc -n kubernetes-dashboard
  # service/dashboard-metrics-scraper skipped
  # service/kubernetes-dashboard edited
  
  # (2) kubernetes-dashboard NodePort 为 10.96.167.225:443 => 30443
  ~$ kubectl get svc -n kubernetes-dashboard
  # NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
  # dashboard-metrics-scraper   ClusterIP   10.108.185.162   <none>        8000/TCP        14h
  # kubernetes-dashboard        NodePort    10.96.167.225    <none>        443:30443/TCP   14h
  
  # (3) 代理访问  kubectl -n kubernetes-dashboardport-forward kubernetes-dashboard 443:443

• Step 6.Dashboard 认证 Token 获取

  $ kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | grep kubernetes-dashboard-token | cut -d " " -f 1)
  Name:         kubernetes-dashboard-token-mssqb
  Namespace:    kubernetes-dashboard
  Labels:       <none>
  Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard
                kubernetes.io/service-account.uid: 71b738ab-0f07-4e2c-99f9-0236cddd9bb4
  Type:  kubernetes.io/service-account-token
  
  Data
  ====
  ca.crt:     1066 bytes
  namespace:  20 bytes
  token:      eyJhbG.....

• Step 7.访问 https://192.168.11.107:30443/#/login 进入 Kubernetes Dashboard 登陆页面选择 Token 认证登陆即可

weiyigeek.top-Kubernetes Dashboard

参考地址: https://github.com/kubernetes/dashboard/blob/master/README.md

Helm 安装方式

操作流程: 此处对于helm安装不再累述，二进制安装没有什么好说的;

$ helm repo add k8s-dashboard https://kubernetes.github.io/dashboard
# "k8s-dashboard" has been added to your repositories
$ helm pull k8s-dashboard/kubernetes-dashboard --untar

~/K8s/Day10/dashboard$ tar -xzvf kubernetes-dashboard-3.0.0.tgz

~/K8s/Day10/dashboard$ ls kubernetes-dashboard
# charts  Chart.yaml  README.md  requirements.lock  requirements.yaml  templates  values.yaml

~/K8s/Day10/dashboard$ helm install kubernetes-dashboard kubernetes-dashboard/ --namespace kube-system
# NAME: kubernetes-dashboard
# LAST DEPLOYED: Sun Dec  6 21:45:22 2020
# NAMESPACE: kube-system
# STATUS: deployed
# REVISION: 1
# TEST SUITE: None
# NOTES:
# *********************************************************************************
# *** PLEASE BE PATIENT: kubernetes-dashboard may take a few minutes to install ***
# ********************************************************************************
# Get the Kubernetes Dashboard URL by running:
#   export POD_NAME=$(kubectl get pods -n kube-system -l "app.kubernetes.io/name=kubernetes-dashboard,app.kubernetes.io/instance=kubernetes-dashboard" -o jsonpath="{.items[0].metadata.name}")  # kubernetes-dashboard-879457794-kxvcr
#   echo https://127.0.0.1:8443/
#   kubectl -n kube-system port-forward $POD_NAME 8443:8443  # 端口转发

查看结果:

~/K8s/Day10/dashboard$ helm list -n kube-system
# NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
# kubernetes-dashboard    kube-system     1               2020-12-06 11:44:44.821856156 +0800 CST deployed        kubernetes-dashboard-3.0.0      2.0.4

~/K8s/Day10/dashboard$ helm history kubernetes-dashboard -n kube-system
# REVISION        UPDATED                         STATUS          CHART                           APP VERSION     DESCRIPTION
# 1               Sun Dec  6 21:45:22 2020        deployed        kubernetes-dashboard-3.0.0      2.0.4           Install complete

# 查看 Dashboard Pod 信息以及标签
~/K8s/Day10/dashboard$ kubectl get pod -n kube-system -o wide  --show-labels | grep "kubernetes-dashboard-879457794-kxvcr"
# kubernetes-dashboard-879457794-kxvcr   1/1     Running            0          11m   10.244.2.55     k8s-node-5   app.kubernetes.io/component=kubernetes-dashboard,app.kubernetes.io/instance=kubernetes-dashboard,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kubernetes-dashboard,app.kubernetes.io/version=2.0.4,helm.sh/chart=kubernetes-dashboard-3.0.0,pod-template-hash=879457794

# 查看 Dashboard 的 SVC 
~/K8s/Day10/dashboard$ kubectl get svc -n kube-system -o wide | grep "kubernetes-dashboard"
# kubernetes-dashboard   ClusterIP   10.104.18.192   <none>        443/TCP                  13m   app.kubernetes.io/component=kubernetes-dashboard,app.kubernetes.io/instance=kubernetes-dashboard,app.kubernetes.io/name=kubernetes-dashboard

以NodePort的方式进行访问:

~/K8s/Day10/dashboard$ kubectl edit svc -n kube-system kubernetes-dashboard
service/kubernetes-dashboard edited

# 目的: 将通过集群IP:443访问的模式变成节点IP:30443进行访问
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: kubernetes-dashboard
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2020-12-06T13:45:22Z"
  labels:
    app.kubernetes.io/component: kubernetes-dashboard
    app.kubernetes.io/instance: kubernetes-dashboard
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-dashboard
    app.kubernetes.io/version: 2.0.4
    helm.sh/chart: kubernetes-dashboard-3.0.0
    kubernetes.io/cluster-service: "true"
  name: kubernetes-dashboard
  namespace: kube-system
  resourceVersion: "6111082"
  selfLink: /api/v1/namespaces/kube-system/services/kubernetes-dashboard
  uid: 51025b69-7c65-4ac0-a8f2-93a243a33e7d
spec:
  clusterIP: 10.104.18.192
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
    nodePort: 30443  # 修改点
  selector:
    app.kubernetes.io/component: kubernetes-dashboard
    app.kubernetes.io/instance: kubernetes-dashboard
    app.kubernetes.io/name: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort   # 修改点 修改 ClusterIP 为 NodePort
status:
  loadBalancer: {}

简单使用:

• 1) 认证的 Token 查看

  ~/K8s/Day10/dashboard$ kubectl -n kube-system get secret | grep kubernetes-dashboard-token
  # kubernetes-dashboard-token-6nrqk                 kubernetes.io/service-account-token   3      19m
  
  kubectl describe secret kubernetes-dashboard-token-6nrqk -n kube-system
  # Name:         kubernetes-dashboard-token-6nrqk
  # Namespace:    kube-system
  # Labels:       <none>
  # Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard
  #               kubernetes.io/service-account.uid: a1685c3b-247e-4802-9d2e-28d5f48e432a
  
  # Type:  kubernetes.io/service-account-token
  
  # Data
  # ====
  # ca.crt:     1066 bytes
  # namespace:  11 bytes
  # token:  .......

weiyigeek.top-Kubernetes-仪表盘

PS : 需要注意令牌过期时间，登录账户将自动退出；

• 2) 登录Dashboard仪表盘控制台，可以看相关资源控制器下面的所属资源

weiyigeek.top-Dashboard仪表盘主页

参考地址：https://artifacthub.io/packages/helm/k8s-dashboard/kubernetes-dashboard?modal=install

安装部署 v2.5.1 版本

描述: 当前时间节点【2022年5月13日 16:50:07】，相对比于前面的 kubernetes-dashboard 版本，当前安装可能会有一定差异。

步骤 01.从Github中拉取dashboard部署资源清单，当前最新版本v2.5.1

# 下载部署
wget -L https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml
kubectl apply -f recommended.yaml
grep "image:" recommended.yaml
  # image: kubernetesui/dashboard:v2.5.1
  # image: kubernetesui/metrics-scraper:v1.0.7

# 或者一条命令搞定部署
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml
  # serviceaccount/kubernetes-dashboard created
  # service/kubernetes-dashboard created
  # secret/kubernetes-dashboard-certs created
  # secret/kubernetes-dashboard-csrf created
  # secret/kubernetes-dashboard-key-holder created
  # configmap/kubernetes-dashboard-settings created
  # role.rbac.authorization.k8s.io/kubernetes-dashboard created
  # clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
  # rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
  # clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
  # deployment.apps/kubernetes-dashboard created
  # service/dashboard-metrics-scraper created
  # deployment.apps/dashboard-metrics-scraper created

步骤 02.查看部署的dashboard相关资源是否正常。

$ kubectl get deploy,svc -n kubernetes-dashboard  -o wide
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                  IMAGES                                SELECTOR
deployment.apps/dashboard-metrics-scraper   1/1     1            1           7m45s   dashboard-metrics-scraper   kubernetesui/metrics-scraper:v1.0.7   k8s-app=dashboard-metrics-scraper
deployment.apps/kubernetes-dashboard        1/1     1            1           7m45s   kubernetes-dashboard        kubernetesui/dashboard:v2.5.1         k8s-app=kubernetes-dashboard

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/dashboard-metrics-scraper   ClusterIP   10.96.37.134   <none>        8000/TCP   7m45s   k8s-app=dashboard-metrics-scraper
service/kubernetes-dashboard        ClusterIP   10.96.26.57    <none>        443/TCP    7m45s   k8s-app=kubernetes-dashboard

# 编辑 service/kubernetes-dashboard 服务将端口通过nodePort方式进行暴露为30443。
$ kubectl edit svc -n kubernetes-dashboard kubernetes-dashboard
# service/kubernetes-dashboard edited
apiVersion: v1
kind: Service
.....
spec:
.....
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8443
    nodePort: 30443  # 新增
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort     # 修改

步骤 03.默认仪表板部署包含运行所需的最小RBAC权限集，而要想使用dashboard操作集群中的资源，通常我们还需要自定义创建kubernetes-dashboard管理员角色。
权限控制参考地址: https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md

# 创建后最小权限的Token(只能操作kubernetes-dashboard名称空间下的资源)
kubectl get sa -n kubernetes-dashboard kubernetes-dashboard
kubectl describe secrets -n kubernetes-dashboard kubernetes-dashboard-token-jhdpb | grep '^token:'|awk '{print $2}'

weiyigeek.top-Dashboard默认两种认证方式

Kubernetes Dashboard 支持几种不同的用户身份验证方式：

• Authorization header
• Bearer Token (默认)
• Username/password
• Kubeconfig file (默认)

温馨提示: 此处使用Bearer Token方式, 为了方便演示我们向 Dashboard 的服务帐户授予管理员权限 (Admin privileges), 而在生产环境中通常不建议如此操作, 而是指定一个或者多个名称空间下的资源进行操作。

tee rbac-dashboard-admin.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: dashboard-admin
    namespace: kubernetes-dashboard
EOF

kubectl apply -f rbac-dashboard-admin.yaml
  # serviceaccount/dashboard-admin created
  # clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created

# 或者 两条命令搞定
# kubectl create serviceaccount -n devtest devtest-ns-admin
# kubectl create clusterrolebinding devtest-ns-admin --clusterrole=admin --serviceaccount=devtest:devtest-ns-admin

步骤 04.获取 sa 创建的 dashboard-admin 用户的 secrets 名称并获取认证 token ，用于上述搭建的dashboard 认证使用。

kubectl get sa -n kubernetes-dashboard dashboard-admin -o yaml | grep "\- name" | awk '{print $3}'
  # dashboard-admin-token-crh7v
kubectl describe secrets -n kubernetes-dashboard dashboard-admin-token-crh7v | grep "^token:" | awk '{print $2}'
  #  获取到认证Token
eyJhbGciOiJSUzI1NiIsImtpZCI6IkJXdm1YSGNSQ3VFSEU3V0FTRlJKcU10bWxzUDZPY3lfU0lJOGJjNGgzRXMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tY3JoN3YiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDY3ODEwMDMtM2MzNi00NWE1LTliYTQtNDY3MTQ0OWE2N2E0Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.X10AzWBxaHObYGoOqjfw3IYkhn8L5E7najdGSeLavb94LX5BY8_rCGizkWgNgNyvUe39NRP8r8YBU5sy9F2K-kN9_5cxUX125cj1drLDmgPJ-L-1m9-fs-luKnkDLRE5ENS_dgv7xsFfhtN7s9prgdqLw8dIrhshHVwflM_VOXW5D26QR6izy2AgPNGz9cRh6x2znrD-dpUNHO1enzvGzlWj7YhaOUFl310V93hh6EEc57gAwmDQM4nWP44KiaAiaW1cnC38Xs9CbWYxjsfxd3lObWShOd3knFk5PUVSBHo0opEv3HQ_-gwu6NGV6pLMY52p_JO1ECPSDnblVbVtPQ

步骤 05.利用上述 Token 进行登陆Kubernetes-dashboard的UI。

weiyigeek.top-拥有管理员权限的dashboard

(3) MetricServer

Q: MetricServer 是什么？

答: 它是kubernetes集群资源使用情况的聚合器，收集数据给kubernetes集群内使用，如 kubectl,hpa,scheduler等。
Kubernetes 推荐使用 metrics-server , 因为 heapster (https:/github.com/kubernetes/heapster) 已经DEPRECATED ,并从 Kubernetes 1.12开始将从 Kubernetes 各种安装脚本中移除,

PS : 如果采用官方的安装dashboard的方式则默认将Metric Server进行安装使用，而采用helm安装dashboard时候默认是将metrics-server禁用的需要手动启用;

helm 安装方式

下面我们使用Helm部署Dashboard时也可以利用第三方依赖进行安装metrics-server,只需要修改一个小小的注释

# (1) 启用Metrics-server以及插件
$ ~/K8s/Day10/dashboard/kubernetes-dashboard$ vim +200 values.yaml
## Enable this is you don't already have metrics-server enabled on your cluster and
## want to use it with dashboard metrics-scraper
## refs:
##  - https://hub.helm.sh/charts/stable/metrics-server
##  - https://github.com/kubernetes-sigs/metrics-server
metrics-server:
  enabled: true
  ## Example for additional args
  args:
    - --logtostderr
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls

# (2) 此时如果需要更新部署时需要将SVC改回集群IP通信
~/K8s/Day10/dashboard$ kubectl edit svc -n kube-system kubernetes-dashboard
service/kubernetes-dashboard edited

# (3) 更新指定的RELEASE_NAME
~/K8s/Day10/dashboard$ helm upgrade kubernetes-dashboard kubernetes-dashboard/ -n kube-system

# (4) 查看与dashboard相关的Pod信息
~/K8s/Day10/dashboard$ kubectl get pod -n kube-system -o wide | grep "kubernetes-dashboard"
  # kubernetes-dashboard-879457794-kxvcr                   1/1     Running            0          23h   10.244.2.55     k8s-node-5
  # kubernetes-dashboard-metrics-server-7bc85c65bc-vrxft   0/1     ImagePullBackOff  # 关键点 0          10m   10.244.2.57     k8s-node-5   

# (5) 发现镜像拉取失败下面我们手动查看并下载该镜像
~/K8s/Day10/dashboard$ kubectl describe pod kubernetes-dashboard-metrics-server-7bc85c65bc-vrxft -n kube-system kubernetes-dashboard
  # GFW 没办法，解决利用阿里云镜像站进行手动下载然后上传到node-5节点中（或者在安全前选择修改该k8s.gcr.io镜像源）
  Warning  Failed     10m (x4 over 12m)     kubelet            Failed to pull image "k8s.gcr.io/metrics-server-amd64:v0.3.6": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
~/K8s/Day10/dashboard$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
~/K8s/Day10/dashboard$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6 k8s.gcr.io/metrics-server-amd64:v0.3.6
~/K8s/Day10/dashboard$ docker save k8s.gcr.io/metrics-server-amd64:v0.3.6 -o metrics-server-amd64.tar
~/K8s/Day10/dashboard$ scp -P 20211 metrics-server-amd64.tar weiyigeek@10.10.107.215:~
  # metrics-server-amd64.tar   100%   39MB 187.5MB/s   00:00
~/K8s/Day10/dashboard$ ssh -p 20211 weiyigeek@10.10.107.215 "docker load -i metrics-server-amd64.tar"
  # Loaded image: k8s.gcr.io/metrics-server-amd64:v0.3.6

# (6) 此时可以看见metrics-server已经成功安装了
~/K8s/Day10/dashboard$ kubectl get pod -n kube-system -o wide | grep "kubernetes-dashboard-metrics-server-7bc85c65bc-vrxft"
  # kubernetes-dashboard-metrics-server-7bc85c65bc-vrxft   1/1     Running   0          27m   10.244.2.57     k8s-node-5    <none>           <none>

# (7) 验证安装的 metrics-server 它获取到关于集群节点基本的指标信息:
~$ kubectl top pod
  # NAME                                CPU(cores)   MEMORY(bytes)
  # dashboard-create-696f45d5db-fj5dg   0m           2Mi
~$ kubectl top node
  # NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
  # ubuntu   194m         2%     1377Mi          17%
  # k8s-node-4    32m          1%     1351Mi          35%
  # k8s-node-5    26m          1%     1134Mi          29%

# (8) 采用port-forward转发的方式访问我们创建的应用，此处访问master节点的端口还是30443端口->Pod暴露的8443端口之中
export POD_NAME=$(kubectl get pods -n kube-system -l "app.kubernetes.io/name=kubernetes-dashboard,app.kubernetes.io/instance=kubernetes-dashboard" -o jsonpath="{.items[0].metadata.name}")
echo https://127.0.0.1:30443/
~/K8s/Day10/dashboard$ kubectl -n kube-system port-forward --address 10.10.107.202 $POD_NAME 30443:8443
  # Forwarding from 10.10.107.202:30443 -> 8443
  # Handling connection for 30443

# 新开一个Terminal获取访问的token
~/K8s/Day10/dashboard$ kubectl describe secrets -n kube-system kubernetes-dashboard-token-6nrqk

token: eyJhbGciOiJSUzI1Ni................IsImtpZCI6IkNsknTWtKLBDUk-Q


# (9) 补充附录
~$ helm get all -n kube-system kubernetes-dashboard    # 查看实际执行的资源清单
~$ helm uninstall kubernetes-dashboard -n kube-system  # 卸载 Helm 安装的 kubernetes-dashboard
release "kubernetes-dashboard" uninstalled

weiyigeek.top-K8s-metrics-server

(4) 配置扩展

1.配置 Kubernetes-dashboard 以支持 http 方式访问

描述: 当前默认安装配置的 Kubernetes-dashboard 都是启用了https, 然而在当我们环境中存在ingress时，可能会有需要将其通过虚拟主机进行暴露时，此时将会在ingress端进行设置证书而不是在 Kubernetes-dashboard Pod中设置证书。

步骤 01.打开下载的Kubernetes-dashboard资源清单文件或者使用kubelet edit命令编辑已部署的资源清单,首先配置 kubernetesui/dashboard:v2.5.1 镜像的启动参数，主要是--enable-insecure-login与--insecure-port=8080参数。

$ kubectl edit deployments.apps -n kubernetes-dashboard kubernetes-dashboard
args:
  # - --auto-generate-certificates
  - --namespace=kubernetes-dashboard
  - --enable-insecure-login
  - --insecure-port=8080

# Pod 端口暴露
ports:
  - name: https
    containerPort: 8443
    protocol: TCP
  - name: http
    containerPort: 8080
    protocol: TCP

# Pod 健康检查
livenessProbe:
  # httpGet:
  #   scheme: HTTPS
  #   path: /
  #   port: 8443
  httpGet:
    scheme: HTTP
    path: /
    port: 8080

步骤 02.配置 kubernetes-dashboard 的 Service 资源管理器

$ kubectl edit svc -n kubernetes-dashboard kubernetes-dashboard
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: ClusterIP

$ kubectl get svc -n kubernetes-dashboard kubernetes-dashboard
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
kubernetes-dashboard   ClusterIP   11.19.103.247   <none>        443/TCP,8080/TCP   3h39m

步骤 03.服务验证以及部署ingress转发规则URL设置，最后浏览器访问如下URL(devops.weiyigeek.top/dashboard/)即可。

$ curl 11.19.103.247:8080

$ tee kubernetes-dashboard-ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "75"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  labels:
    app: devops-weiyigeek
  name: devops-weiyigeek
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: devops.weiyigeek.top
    http:
      paths:
      - backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 8080
        path: /dashboard(/|$)(.*)
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - devops.weiyigeek.top
    secretName: devops-weiyigeek-top
EOF

# 部署 ingress 规则
$ kubectl apply -f kubernetes-dashboard-ingress.yaml 
$ kubectl get ingress -n kubernetes-dashboard devops-weiyigeek
NAME          CLASS   HOSTS                ADDRESS        PORTS     AGE
devops-weiyigeek   nginx   devops.weiyigeek.top   11.19.12.210   80, 443   3h52m

温馨提示: 在前面部署完成后, 我们便可可以通过https://devops.weiyigeek.top/dashboard/带https + 域名方式访问kubernetes-dashboard了。

---

0x03 使用实践

(1) Dashboard-小试牛刀之简单初识

• Step 1.右上角点击+进行创建Deployment管理的Pod，按图所示输入应用名称和容器镜像名称，其次是Services资源控制器设置为内部的Internal；

weiyigeek.top-创建Deployment管理的Pod

• Step 2.点击左边Workloads中的Deployments子菜单查看创建的Deployments资源

  ~$ kubectl get pod -o wide --show-labels
    # NAME                                      READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES   LABELS
    # dashboard-create-696f45d5db-fj5dg         1/1     Running   0          4m23s   10.244.2.56   k8s-node-5   <none>           <none>            k8s-app=dashboard-create,pod-template-hash=696f45d5db

  

  weiyigeek.top-Deployments

• Step 3.同样点击Service中的Services子菜单将可以看见我们创建的Service相关资源信息

  ~$ kubectl get svc  -o wide --show-labels
    # NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE     SELECTOR                   LABELS
    # dashboard-create   ClusterIP   10.102.184.126   <none>        80/TCP           4m53s   k8s-app=dashboard-create   k8s-app=dashboard-create

weiyigeek.top-Service

• Step 4.#验证 集群IP地址访问 以及 Pod地址访问效果一致

  ~$ curl http://10.102.184.126/host.html && curl http://10.244.2.56/host.html
    # Hostname: dashboard-create-696f45d5db-fj5dg ,Image Version: 3.0, Nginx Version: 1.19.4
    # Hostname: dashboard-create-696f45d5db-fj5dg ,Image Version: 3.0, Nginx Version: 1.19.4

PS : 总结可以看出使用Kubernetes-Kuboard是可以非常简单的创建我们指定的应用到kubernetes之中;

(2) Dashboard-利用rbac机制限制指定用户针对指定名称空间中的资源进行UI管理。

描述: 有时可能我们会遇到如下场景, 在进行持续CI/CD后，开发人员可能会需要查看部署应用的启动日志,如果都是我们运维人员手动去截图发给他们, 那这样的效率简直是在浪费生命，所有为了节约时间同时保证防止开发人员误操作集群, 此时我们只赋予其指定名称空间下的某些资源浏览权限即可.

在 Kubernetes 集群中我们可以使用 rbac 授权机制, 做用户角色权限分离，可以指定那些资源，我们可以进行那些操作，然后把该角色赋予给指定的用户，最好利用该用户的Token进行登陆Kubernetes-Dashborad界面进行相应管理。

步骤 01.创建一个服务用户此处我们可以采用两种方式创建资源清单或者命令行。

# 方式1
kubectl create serviceaccount -n devtest devtest-ns-viewonly

# 方式2
tee > devtest-ns-viewonly-sa.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: devtest-ns-viewonly
  namespace: devtest
EOF

步骤 02.准备名称为dashboard-viewonly角色相关资源权限操作的资源清单。

tee > dashboard-namespace-viewonly.yaml <<'EOF'
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dashboard-viewonly
  namespace: devtest
rules:
- apiGroups: [""]
  resources: ["pods","pods/exec"]
  verbs: ["get","list","watch","delete"]
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - persistentvolumeclaims
  - persistentvolumeclaims/status
  - replicationcontrollers
  - replicationcontrollers/scale
  - serviceaccounts
  - services
  - services/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - bindings
  - events
  - limitranges
  - namespaces/status
  - pods/log
  - pods/status
  - replicationcontrollers/status
  - resourcequotas
  - resourcequotas/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - controllerrevisions
  - daemonsets
  - daemonsets/status
  - deployments
  - deployments/scale
  - deployments/status
  - replicasets
  - replicasets/scale
  - replicasets/status
  - statefulsets
  - statefulsets/scale
  - statefulsets/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - daemonsets/status
  - deployments
  - deployments/scale
  - deployments/status
  - ingresses
  - ingresses/status
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicasets/status
  - replicationcontrollers/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - cronjobs/status
  - jobs
  - jobs/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  - ingresses/status
  - networkpolicies
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
EOF
kubectl apply -f dashboard-namespace-viewonly.yaml

步骤 03.绑定 dashboard-viewonly 角色给 ServiceAccount 的 devtest-ns-viewonly 用户.

tee dashboard-viewonly-RoleBinding<<'EOF'
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: devtest-ns-viewonly
  namespace: devtest
roleRef:
  kind: Role
  name: dashboard-viewonly
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: devtest-ns-viewonly
EOF

# 或者一条命令搞定
kubectl create rolebinding -n devtest devtest-ns-viewonly --role=devtest:dashboard-viewonly --serviceaccount=devtest-ns-viewonly

温馨提示: ClusterRole 与 ClusterRoleBinding 均不支持指定名称空间。

步骤 04.查看 devtest-ns-viewonly 用户存在 secrets 中的认证Token。

kubectl describe secrets -n devtest devtest-ns-viewonly-token-gxgps | grep "^token:" | awk '{print $2}'

步骤 05.使用获取到的Token访问登陆，我们搭建的kubernetes-dashboard Web UI，此处使用浏览器访问(https://devops.weiyigeek.top/dashboard/#/workloads?namespace=devtest)，可以看到该使用Token认证的用户只能访问devtest名称空间下的特定资源。

weiyigeek.top-认证用户只能访问devtest名称空间下的特定资源

---

0x04 入坑与出坑

问题1.pods is forbidden: User "system:serviceaccount:kube-system:namespace-controller" cannot create resource clusterroles” in API group “rbac.authorization.k8s.io” at the cluster scope

问题原因:

• 1.API组中用户不能在默认命名空间创建Pod，也就是说使用原token认证登录的用户是无权操作
• 2.其次是采用Helm创建的时候只是将kubernetes-dashboard-metrics与集群角色绑定

  # 绑定的角色
  ~/K8s/Day10/dashboard/kubernetes-dashboard$ kubectl get ClusterRoleBinding -n kube-system | grep "kubernetes-dashboard"
  kubernetes-dashboard-metrics                           ClusterRole/kubernetes-dashboard-metrics  
  
  # 查看集群所有权限
  ~/K8s/Day10/dashboard/kubernetes-dashboard$ kubectl get clusterrole
  
  # 权限非常有限
  ~/K8s/Day10/dashboard/kubernetes-dashboard$ kubectl get clusterrole kubernetes-dashboard-metrics -o yaml
  rules:
  - apiGroups:
    - metrics.k8s.io
    resources:
    - pods
    - nodes
    verbs:
    - get
    - list
    - watch

解决方法:

# 1.创建kubernetes-dashboard管理员角色
cat > k8s-admin.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
# 绑定对象
metadata:
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kube-system
# 权限来源
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
EOF
kubectl create -f k8s-admin.yaml


# 3.验证查看 ClusterRoleBinding 资源
~/K8s/Day10/dashboard/$ kubectl get ClusterRoleBinding -n kube-system | grep "kubernetes-dashboard"
# NAME                                                   ROLE                                                                               AGE
# kubernetes-dashboard                                   ClusterRole/cluster-admin                                                          17m
# kubernetes-dashboard-metrics                           ClusterRole/kubernetes-dashboard-metrics                                           70m

~/K8s/Day10/dashboard/$ kubectl describe ClusterRoleBinding -n kube-system kubernetes-dashboard
  # Name:         kubernetes-dashboard
  # Labels:       <none>
  # Annotations:  <none>
  # Role:
  #   Kind:  ClusterRole
  #   Name:  cluster-admin
  # Subjects:
  #   Kind            Name                  Namespace
  #   ----            ----                  ---------
  #   ServiceAccount  kubernetes-dashboard  kube-system


# 2.获取dashboard管理员角色token
kubectl describe secret kubernetes-dashboard-token-7z6zm -n kube-system  

# 3.使用第二步第12行的token登陆kubernetes-dashboard web界面即可

PS : 在使用Helm创建Kubenertes-Dashboard时候已创建了ServiceAccount资源，所以只需要创建ClusterRoleBinding资源即可;

参考地址: https://blog.csdn.net/qq_38900565/article/details/100729686

问题2.采用Helm安装metric-server时镜像有误导致Pod状态ImagePullBackOff

错误信息:

Warning  Failed     10m (x4 over 12m)     kubelet            Failed to pull image "k8s.gcr.io/metrics-server-amd64:v0.3.6": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

解决办法：

• 1.利用阿里云的K8s镜像站拉取metrics-server-amd64:v0.3.6镜像然后进行改名，随后上传到metrics-server运行的节点之上
• 2.在进行更新时候指定或者说修改配置文件中的image.repository;

  ~/K8s/Day10/dashboard$ grep "k8s.gcr.io" kubernetes-dashboard/charts/metrics-server/*
  # kubernetes-dashboard/charts/metrics-server/values.yaml:  repository: k8s.gcr.io/metrics-server-amd64
  ~/K8s/Day10/dashboard$ sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" kubernetes-dashboard/charts/metrics-server/values.yaml: