16-Kubernetes进阶之开发环境部署与配置

2021-01-27
Containers
OperationTools

16-Kubernetes进阶之开发环境部署与配置

2021-01-27

[TOC]

0x00 前言简述

kubernetes 源码编译，分为本地二进制可执行文件编译和 docker 镜像编译两种，之前演示的 minikube 方式或 kubeadm 方式安装，都是基于第二种 docker 镜像方式运行，当然也可以基于二进制文件方式安装，不管哪种方式，都是直接使用并不需要修改任何 k8s 代码。

不过当我们有特殊需求时，比如需要修改 kube-proxy 对 service 的代理逻辑，kube-scheduler 对 pod 的调度逻辑等，这个时候就需要修改 k8s 源码了，为了让修改的代码生效，就需要对 k8s 代码执行编译了。当然 k8s 也为我们提供了 CRD 等可扩展插件，在不修改 k8s 源码的基础上实现自定义功能，但是对于一些底层逻辑策略需要修改的话，还是办不到的。

kubernetes 开发者参考: https://github.com/kubernetes/community/blob/master/contributors/devel/development.md

注意：因为 kubernetes 源码是由 go 语言编写的，所以要编译其源码，需要安装 go 环境，若我们想基于 docker 镜像编译 kubernetes 的话，那么还需要安装 docker 环境。

0x01 开发环境

描述: kubernetes 开发依赖于Go所以它是必须的，其次是Git为了拉取官方的kubernetes项目；

Kunernetes 常用开发环境安装方式:

本地二进制可执行文件编译
Docker镜像安装(Google 提供了 kube-core 但需要科学上网)

1.本地开发

环境准备:

Ubuntu & Git

点击阅读完整原文

[TOC]

0x00 前言简述

kubernetes 开发者参考: https://github.com/kubernetes/community/blob/master/contributors/devel/development.md

0x01 开发环境

描述: kubernetes 开发依赖于Go所以它是必须的，其次是Git为了拉取官方的kubernetes项目；

Kunernetes 常用开发环境安装方式:

本地二进制可执行文件编译
Docker镜像安装(Google 提供了 kube-core 但需要科学上网)

1.本地开发

环境准备:

Ubuntu & Git

> lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

> uname -r
5.4.0-56-generic

> git version
git version 2.25.

Go 语言安装

# Go官网：https://golang.org/dl/
# Go中文社区：https://studygolang.com/dl
wget https://dl.google.com/go/go1.15.5.linux-amd64.tar.gz 
tar -zxvf go1.15.5.linux-amd64.tar.gz -C /home/weiyigeek/app/program/go/
vi /etc/profile
  export GOPATH=/home/weiyigeek/app/program/go/src/   # GOPATH 作为 go 执行的工作路径
  export GOROOT=/home/weiyigeek/app/program/go        # go 安装目录
  export PATH=$PATH:${GOROOT}/bin 
source /etc/profile

# 环境验证
$ env | grep "GO"
GOPATH=/home/weiyigeek/app/program/go/src/
GOROOT=/home/weiyigeek/app/program/go
$ go version
go version go1.15.5 linux/amd64

Kubernetes 项目源码

# 方式1: 拉取项目所有分支
mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io
> git clone https://github.com/kubernetes/kubernetes.git
# $ git clone  https://github.com/kubernetes/kubernetes -b release-1.19.3 # 拉取Kubernetes源码指定分支
$ ls kubernetes
# api           cluster             go.mod    logo                      pkg                SUPPORT.md    WORKSPACE
# build         cmd                 go.sum    Makefile                  plugin             test
# BUILD.bazel   code-of-conduct.md  hack      Makefile.generated_files  README.md          third_party
# CHANGELOG     CONTRIBUTING.md     LICENSE   OWNERS                    SECURITY_CONTACTS  translation

# 方式2: 拉取指定K8S版本源代码
K8S_VERSION=1.19.3
wget -c --no-check-certificate https://github.com/kubernetes/kubernetes/archive/v${K8S_VERSION}.tar.gz
> tar -tf v1.19.3.tar.gz | more
kubernetes-1.19.3/
kubernetes-1.19.3/.bazelrc
kubernetes-1.19.3/.bazelversion

kubernetes-1.19.3/.generated_files

Docker 镜像安装

描述: 安装Docker以及加速源的配置这里不再累述了，不会的读者参考官网或者本博客中的Docker安装流程;

1 2	# 运行容器，并进入到容器内部 docker run --rm -it -v /home/weiyigeek/kubernetes/k8s-src:/go/src/k8s.io/kubernetes mirrorgooglecontainers/kube-cross:v1.12.10-1 bash

复制代码

vim staging/src/k8s.io/client-go/util/cert/cert.go # kubeadm 1.14 版本之前
vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go # kubeadm 1.14 至今
const duration365d = time.Hour 24 365 * 10 #设置为10年
NotAfter: time.Now().Add(duration365d).UTC(), #替换
make WHAT=cmd/kubeadm GOFLAGS=-v #设置只编译kubeadm
cp _output/bin/kubeadm /root/kubeadm-new

复制代码

⒌更新 kubeadm

#将kubeadm 进行替换
cp /usr/bin/kubeadm /usr/bin/kubeadm.old
cp /root/kubeadm-new /usr/bin/kubeadm
chmod a+x /usr/bin/kubeadm

⒍更新各节点证书至Master节点

cp -r /etc/kubernetes/pki /etc/kubernetes/pki.old
cd /etc/kubernetes/pki
kubeadm alpha certs renew all –config=/root/kubeadm-config.yaml
openssl x509 -in apiserver.crt -text -noout | grep Not

⒎HA集群其余 mater节点证书更新
复制代码

#!/bin/bash
masterNode=”192.168.66.20 192.168.66.21”

#for host in ${masterNode}; do

scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}

“${USER}”@$host:/etc/kubernetes/pki/

scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} “root”@$host:/etc/kubernetes/pki/etcd

scp /etc/kubernetes/admin.conf “root”@$host:/etc/kubernetes/

#done
for host in ${CONTROL_PLANE_IPS}; do
scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
“${USER}”@$host:/root/pki/
scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} “root”@$host:/root/etcd
scp /etc/kubernetes/admin.conf “root”@$host:/root/kubernetes/
done

https://blog.csdn.net/aixiaoyang168/article/details/87917184

weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm

go build -v .

weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm

ls
app BUILD kubeadm kubeadm.go OWNERS test

weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm

./kubeadm version
kubeadm version: &version.Info{Major:””, Minor:””, GitVersion:”v0.0.0-master+1e11e4a210802”, GitCommit:”1e11e4a2108024935ecfcb2912226cedeafd99df”, GitTreeState:””, BuildDate:”1970-01-01T00:00:00Z”, GoVersion:”go1.15.5”, Compiler:”gc”, Platform:”linux/amd64”}

实操1.Kubeadm / etcd证书过期或者延期处理流程

描述: 下述操作主要用于处理已过期或者即将过期的kubernetes集群证书;

Kubernetes 与 Etcd 版本与证书关联信息说明:

Etcd 版本小于等于v1.9版本，etcd默认是不使用TLS连接，没有etcd相关证书，只需要更新master证书即可。
Etcd 版本大于等于v1.10版本，etcd默认开启TLS，需要更新etcd证书和master证书。

/etc/kubernetes/pki/etcd$ ls
  # ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key  peer.crt  peer.key  server.crt  server.key

/etc/kubernetes/pki/etcd$ sudo openssl x509 -in server.crt -text -noout
  # Certificate:
  #     Data:
  #         Version: 3 (0x2)
  #         Serial Number: 4411755188724745942 (0x3d39b3f83becc6d6)
  #         Signature Algorithm: sha256WithRSAEncryption
  #         Issuer: CN = etcd-ca
  #         Validity
  #             Not Before: Nov  5 08:47:32 2020 GMT
  #             Not After : Dec 12 09:17:18 2021 GMT # 关键点

PS : 如果 Etcd 是由Kubeadm创建和托管的此时也可以通过下面的方式进行证书的续期, 如果是外部管理需要手动进行配置;

操作流程:
1.采用以下命令查询证书可用时间

# apiserver.crt (证书一年)
/etc/kubernetes/pki# openssl x509 -in apiserver.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3597153141676254659 (0x31eba7b94c4b25c3)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Nov  5 08:47:32 2020 GMT
            Not After : Nov  5 08:47:32 2021 GMT
# ca.crt (证书十年)
/etc/kubernetes/pki# openssl x509 -in ca.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Nov  5 08:47:32 2020 GMT
            Not After : Nov  3 08:47:32 2030 GMT
        Subject: CN = kubernetes

2.查看当前集群证书相关信息

$ sudo kubeadm alpha certs check-expiration
  # [check-expiration] Reading configuration from the cluster...
  # [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

  # CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY(认证中心)   EXTERNALLY MANAGED(外部管理)
  # admin.conf                 Nov 05, 2021 08:47 UTC   328d                                    no
  # apiserver                  Nov 05, 2021 08:47 UTC   328d            ca                      no
  # apiserver-etcd-client      Nov 05, 2021 08:47 UTC   328d            etcd-ca                 no
  # apiserver-kubelet-client   Nov 05, 2021 08:47 UTC   328d            ca                      no
  # controller-manager.conf    Nov 05, 2021 08:47 UTC   328d                                    no
  # etcd-healthcheck-client    Nov 05, 2021 08:47 UTC   328d            etcd-ca                 no
  # etcd-peer                  Nov 05, 2021 08:47 UTC   328d            etcd-ca                 no
  # etcd-server                Nov 05, 2021 08:47 UTC   328d            etcd-ca                 no
  # front-proxy-client         Nov 05, 2021 08:47 UTC   328d            front-proxy-ca          no
  # scheduler.conf             Nov 05, 2021 08:47 UTC   328d                                    no

  # CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
  # ca                      Nov 03, 2030 08:47 UTC   9y              no
  # etcd-ca                 Nov 03, 2030 08:47 UTC   9y              no
  # front-proxy-ca          Nov 03, 2030 08:47 UTC   9y              no

3.采用renew子命令刷新证书的到期时间进行续期

~/.k8s$ sudo kubeadm alpha certs renew all --config=./kubeadm-init-config.yaml
  # W1212 17:17:16.721037 1306627 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
  # certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed  # 嵌入在kubeconfig文件中的证书，供管理员使用，并对kubeadm本身进行更新 (admin.conf )
  # certificate for serving the Kubernetes API renewed   # 更新Kubernetes API服务证书
  # certificate the apiserver uses to access etcd renewed  # 服务器访问etcd所使用的证书已更新
  # certificate for the API server to connect to kubelet renewed  # API服务器连接到kubelet的证书已更新
  # certificate embedded in the kubeconfig file for the controller manager to use renewed  # 证书嵌入在kubeconfig文件中，供控制器管理器使用更新 (controller-manager.conf)
  # certificate for liveness probes to healthcheck etcd renewed   # 健康检查etcd激活探针证书续期
  # certificate for etcd nodes to communicate with each other renewed  # 用于etcd节点之间通信的证书更新
  # certificate for serving etcd renewed   # 续期etcd“服务证书”
  # certificate for the front proxy client renewed  # 前代理客户端的证书更新
  # certificate embedded in the kubeconfig file for the scheduler manager to use renewed #证书嵌入在kubeconfig文件中，供调度器管理器使用更新 (scheduler.conf  )

# 补充：如果环境里中etcd不是由Kubeadm创建并托管而是外部二进制部署的就不能采用此种方式更新etcd的证书
# 更新指定集群组件进行更新集群
~/.k8s$ kubeadm alpha certs renew
  # admin.conf                apiserver                 apiserver-kubelet-client  etcd-healthcheck-client   etcd-server               scheduler.conf
  # all                       apiserver-etcd-client     controller-manager.conf   etcd-peer                 front-proxy-client
kubeadm alpha certs renew apiserver --config=./kubeadmin-config.yaml
kubeadm alpha certs renew apiserver-kubelet-client --config=./kubeadmin-config.yaml
kubeadm alpha certs renew front-proxy-client --config=./kubeadmin-config.yaml

4.验证刷新证书后的到期时间以及证书文件属性查看

~/.k8s$ sudo kubeadm alpha certs check-expiration
  # [check-expiration] Reading configuration from the cluster...
  # [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
  # PS -> 显示顺序与上面刷新证书的顺序是一致的
  # CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
  # admin.conf                 Dec 12, 2021 09:17 UTC   364d                                    no
  # apiserver                  Dec 12, 2021 09:17 UTC   364d            ca                      no
  # apiserver-etcd-client      Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
  # apiserver-kubelet-client   Dec 12, 2021 09:17 UTC   364d            ca                      no
  # controller-manager.conf    Dec 12, 2021 09:17 UTC   364d                                    no
  # etcd-healthcheck-client    Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
  # etcd-peer                  Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
  # etcd-server                Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
  # front-proxy-client         Dec 12, 2021 09:17 UTC   364d            front-proxy-ca          no
  # scheduler.conf             Dec 12, 2021 09:17 UTC   364d                                    no

  # CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
  # ca                      Nov 03, 2030 08:47 UTC   9y              no
  # etcd-ca                 Nov 03, 2030 08:47 UTC   9y              no
  # front-proxy-ca          Nov 03, 2030 08:47 UTC   9y              no

/etc/kubernetes/pki$ stat apiserver.key
  #   File: apiserver.key
  #   Size: 1675            Blocks: 8          IO Block: 4096   regular file
  # Device: fd00h/64768d    Inode: 2099948     Links: 1
  # Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
  # Access: 2020-12-12 20:30:32.703551276 +0800  最近访问：
  # Modify: 2020-12-12 17:17:17.055077737 +0800  最近更改：
  # Change: 2020-12-12 17:17:17.055077737 +0800  最近改动：
  #  Birth: -
/etc/kubernetes/pki$ stat apiserver.crt
  #   File: apiserver.crt
  #   Size: 1294            Blocks: 8          IO Block: 4096   regular file
  # Device: fd00h/64768d    Inode: 2099949     Links: 1
  # Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
  # Access: 2020-12-12 17:19:54.202163815 +0800
  # Modify: 2020-12-12 17:17:17.099078602 +0800
  # Change: 2020-12-12 17:17:17.099078602 +0800
  # Birth: -

5.基于新证书重新生成各个组件的配置文件

# (1) 主Master节点上运行生成各组件的配置文件
kubeadm alpha kubeconfig user --org system:masters --client-name kubernetes-admin  > /etc/kubernetes/admin.conf
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf

# (2) 要开始使用您的集群，您需要运行以下作为一个常规用户:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# (3 )复制证书和配置文件到其他master
scp -r /etc/kubernetes/pki/* root@master@master-01:/etc/kubernetes/pki/
scp /etc/kubernetes/scheduler.conf root@master@master-01:/etc/kubernetes/
scp /etc/kubernetes/controller-manager.conf root@master@master-01:/etc/kubernetes/
scp /etc/kubernetes/admin.conf root@master@master-01:/etc/kubernetes/

6.重启kubelet组件,其他master同样重启kubelet和3个组件docker容器

1
2
3

systemctl restart kubelet
# docker容器重启controller、scheduler、apiserver3个组件
# docker restart

该脚本只处理master节点上的证书：kubeadm默认配置了kubelet证书自动更新，node节点kubelet.conf所指向的证书会自动更新

小于v1.17版本的master初始化节点(执行kubeadm init的节点) kubelet.conf里的证书并不会自动更新，这算是一个bug，该脚本会一并处理更新master节点的kubelet.conf所包含的证书

kubeadm生成的证书有效期为为1年，该脚本可将kubeadm生成的证书有效期更新为10年

Kubernetes集群版本在更新时，就会自动更新apiserver.crt证书的使用期限，这可能也是k8s官方设置这一年期限的原因 —— 为了让使用者跟上版本更新的步伐。

证书修改流程
由于k8s是基于kubeadm安装的，所以只需修改kubeadm源码中的证书期限即可。 kubeadm是采用go语言编写，所以我们要现在本机安装go语言环境，然后将对应版本的kubeadm源码进行修改，修改后重新编译，再用新编译的kubeadm生成新证书即可。

Kubeadm/etcd证书修改可用年限

描述: 在Vallidity节点下Kubernetes有两种机制去创建证书，有一部分是1年的(apiServer.key)，有1部分是10年的比如我们的(CA.CERT); Kubernetes集群版本在更新时，就会自动更新apiserver.crt证书的使用期限，这可能也是k8s官方设置这一年期限的原因为了让使用者跟上版本更新的步伐。但是有一些公司就会选取一个稳定版本一直使用下去，这便会使用本节所介绍的内容如何修改证书的使用年限。

操作流程:

kub~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

> git checkout -b remotes/origin/release-1.19.3 v1.19.3                                             master [2bd115e3648]
切换到一个新分支 'remotes/origin/release-1.19.3'

weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes
> vim staging/src/k8s.io/client-go/util/cert/cert.go                         remotes/origin/release-1.19.3 [1e11e4a2108]


38 const duration365d = time.Hour * 24 * 365 
56 // NewSelfSignedCACert creates a CA certificate
57 func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
65                 NotBefore:             now.UTC(),
66                 NotAfter:              now.Add(duration365d * 10).UTC(),

86 // GenerateSelfSignedCertKeyWithFixtures creates a self-signed certificate and key for the given host.
94 func GenerateSelfSignedCertKeyWithFixtures(host string, alternateIPs []net.IP, alternateDNS []string, fixtureDirector    y string) ([]byte, []byte, error) {
96         maxAge := time.Hour * 24 * 365 * 10     // one year self-signed certs
123                 NotBefore: validFrom,
124                 NotAfter:  validFrom.Add(maxAge),



// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg *CertConfig, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
const duration365d = time.Hour * 24 * 365 * 10

591                 NotBefore:    caCert.NotBefore,
592                 NotAfter:     time.Now().Add(duration365d).UTC(),
 }


 结果在这里找到kubeadmconstants.CertificateValidity的定义
vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go
# 这个方法里面看到NotAfter:     time.Now().Add(kubeadmconstants.CertificateValidity).UTC()
# 参数里面是一个常量kubeadmconstants.CertificateValidity
# 所以这里可以不修改，我去看看源码能不能找到这个常量的赋值位置
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        CertificateValidity = time.Hour * 24 * 365 * 10
        NotBefore:    caCert.NotBefore,
        NotAfter:     time.Now().Add(kubeadmconstants.CertificateValidity).UTC(),

vim ./cmd/kubeadm/app/constants/constants.go
const (
        // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        CertificateValidity = time.Hour * 24 * 365 * 10

        // CACertAndKeyBaseName defines certificate authority base name
        CACertAndKeyBaseName = "ca"
        // CACertName defines certificate name
        CACertName = "ca.crt"
        // CAKeyName defines certificate name
        CAKeyName = "ca.key"

        // APIServerCertAndKeyBaseName defines API's server certificate and key base name
        APIServerCertAndKeyBaseName = "apiserver"
        // APIServerCertName defines API's server certificate name
        APIServerCertName = "apiserver.crt"
        // APIServerKeyName defines API's server key name
        APIServerKeyName = "apiserver.key"
        // APIServerCertCommonName defines API's server certificate common name (CN)
        APIServerCertCommonName = "kube-apiserver"

        // APIServerKubeletClientCertAndKeyBaseName defines kubelet client certificate and key base name
        APIServerKubeletClientCertAndKeyBaseName = "apiserver-kubelet-client"
        // APIServerKubeletClientCertName defines kubelet client certificate name
        APIServerKubeletClientCertName = "apiserver-kubelet-client.crt"
        // APIServerKubeletClientKeyName defines kubelet client key name
        APIServerKubeletClientKeyName = "apiserver-kubelet-client.key"
        // APIServerKubeletClientCertCommonName defines kubelet client certificate common name (CN)
        APIServerKubeletClientCertCommonName = "kube-apiserver-kubelet-client"

        // EtcdCACertAndKeyBaseName defines etcd's CA certificate and key base name
        EtcdCACertAndKeyBaseName = "etcd/ca"
        // EtcdCACertName defines etcd's CA certificate name
        EtcdCACertName = "etcd/ca.crt"
        // EtcdCAKeyName defines etcd's CA key name
        EtcdCAKeyName = "etcd/ca.key"

        // EtcdServerCertAndKeyBaseName defines etcd's server certificate and key base name
        EtcdServerCertAndKeyBaseName = "etcd/server"
        // EtcdServerCertName defines etcd's server certificate name
        EtcdServerCertName = "etcd/server.crt"
        // EtcdServerKeyName defines etcd's server key name
        EtcdServerKeyName = "etcd/server.key"

# 编译kubeadm, 这里主要编译kubeadm 即可
make WHAT=cmd/kubeadm GOFLAGS=-v

weiyigeek@master:~$ sudo cp /usr/bin/kubeadm{,.bak}
sudo cp kubeadm /usr/bin/

weiyigeek@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+1e11e4a210802", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
weiyigeek@master:~$ kubeadm.bak version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
weiyigeek@master:~/.k8s$ kubeadm alpha certs check-expiration


[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 12, 2021 09:17 UTC   364d                                    no
apiserver                  Dec 12, 2021 09:17 UTC   364d            ca                      no
apiserver-etcd-client      Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Dec 12, 2021 09:17 UTC   364d            ca                      no
controller-manager.conf    Dec 12, 2021 09:17 UTC   364d                                    no
etcd-healthcheck-client    Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
etcd-peer                  Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
etcd-server                Dec 12, 2021 09:17 UTC   364d            etcd-ca                 no
front-proxy-client         Dec 12, 2021 09:17 UTC   364d            front-proxy-ca          no
scheduler.conf             Dec 12, 2021 09:17 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Nov 03, 2030 08:47 UTC   9y              no
etcd-ca                 Nov 03, 2030 08:47 UTC   9y              no
front-proxy-ca          Nov 03, 2030 08:47 UTC   9y              no



kubeadm alpha certs renew all --config=/root/kubeadm-config.yaml 
weiyigeek@master:~/.k8s$ sudo kubeadm alpha certs renew all --config=kubeadm-init-config.yaml
W1213 13:42:52.803441 2059631 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
weiyigeek@master:~/.k8s$ sudo kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 11, 2030 05:42 UTC   9y                                      no
apiserver                  Dec 11, 2030 05:42 UTC   9y              ca                      no
apiserver-etcd-client      Dec 11, 2030 05:42 UTC   9y              etcd-ca                 no
apiserver-kubelet-client   Dec 11, 2030 05:42 UTC   9y              ca                      no
controller-manager.conf    Dec 11, 2030 05:42 UTC   9y                                      no
etcd-healthcheck-client    Dec 11, 2030 05:42 UTC   9y              etcd-ca                 no
etcd-peer                  Dec 11, 2030 05:42 UTC   9y              etcd-ca                 no
etcd-server                Dec 11, 2030 05:42 UTC   9y              etcd-ca                 no
front-proxy-client         Dec 11, 2030 05:42 UTC   9y              front-proxy-ca          no
scheduler.conf             Dec 11, 2030 05:42 UTC   9y                                      no


openssl x509 -in apiserver.crt -text -noout | grep Not

⒎HA集群其余 mater节点证书更新
#!/bin/bash 
masterNode="192.168.66.20 192.168.66.21"
#for host in ${masterNode}; do
# scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
# "${USER}"@$host:/etc/kubernetes/pki/ 
# scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} "root"@$host:/etc/kubernetes/pki/etcd 
# scp /etc/kubernetes/admin.conf "root"@$host:/etc/kubernetes/ 
#done 
for host in ${CONTROL_PLANE_IPS}; do 
    scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key} 
"${USER}"@$host:/root/pki/ 
    scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} "root"@$host:/root/etcd 
    scp /etc/kubernetes/admin.conf "root"@$host:/root/kubernetes/ 
done

入坑出坑

问题 : Unable to connect to the server: x509: certificate has expired or is not yet valid
故障描述: 使用kubeadm部署的集群，在运行了一年之后今天，出现k8s api无法调取的现象，使用kubectl命令获取资源均返回如下报错

1	Unable to connect to the server: x509: certificate has expired or is not yet valid

故障原因: 由于默认的apiserver.crt证书有效期只有一年其错误信息表示证书到期无法正常使用；
解决办法: 现在证书过期的问题已经发生了，只能通过更换证书的办法解决，在github上有详细的说明 issue链接如下:
https://github.com/kubernetes/kubeadm/issues/581

欢迎各位志同道合的朋友一起学习交流，如文章有误请在下方留下您宝贵的经验知识，个人邮箱地址【master#weiyigeek.top】或者个人公众号【WeiyiGeek】联系我。

更多文章来源于【WeiyiGeek Blog - 为了能到远方，脚下的每一步都不能少】, 个人首页地址( https://weiyigeek.top )

专栏书写不易，如果您觉得这个专栏还不错的，请给这篇专栏 【点个赞、投个币、收个藏、关个注、转个发、赞个助】，这将对我的肯定，我将持续整理发布更多优质原创文章！。

最后更新时间：2023-06-06 17:20:31
文章原始路径：_posts/虚拟云容/云容器/Kubernetes/16-Kubernetes进阶之开发环境部署与配置.md
转载注明出处，原文地址：https://blog.weiyigeek.top/2021/1-27-548.html
本站文章内容遵循知识共享署名 - 非商业性 - 相同方式共享 4.0 国际协议

WeiyiGeeker