[TOC]
0x00 前言简述
kubernetes 源码编译，分为本地二进制可执行文件编译和 docker 镜像编译两种， 之前演示的 minikube 方式或 kubeadm 方式安装，都是基于第二种 docker 镜像方式运行，当然也可以基于二进制文件方式安装，不管哪种方式，都是直接使用并不需要修改任何 k8s 代码。
不过当我们有特殊需求时，比如需要修改 kube-proxy 对 service 的代理逻辑，kube-scheduler 对 pod 的调度逻辑等，这个时候就需要修改 k8s 源码了，为了让修改的代码生效，就需要对 k8s 代码执行编译了。当然 k8s 也为我们提供了 CRD 等可扩展插件，在不修改 k8s 源码的基础上实现自定义功能，但是对于一些底层逻辑策略需要修改的话，还是办不到的。
kubernetes 开发者参考: https://github.com/kubernetes/community/blob/master/contributors/devel/development.md
注意：因为 kubernetes 源码是由 go 语言编写的，所以要编译其源码，需要安装 go 环境，若我们想基于 docker 镜像编译 kubernetes 的话，那么还需要安装 docker 环境。
0x01 开发环境
描述: kubernetes 开发依赖于Go所以它是必须的，其次是Git为了拉取官方的kubernetes项目；
Kunernetes 常用开发环境安装方式:
- 本地二进制可执行文件编译
- Docker镜像安装(Google 提供了 kube-core 但需要科学上网)
1.本地开发
环境准备:
- Ubuntu & Git
1
2
3
4
5
6
7
8
9
10
11> lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
> uname -r
5.4.0-56-generic
> git version
git version 2.25. - Go 语言安装
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16# Go官网：https://golang.org/dl/
# Go中文社区：https://studygolang.com/dl
wget https://dl.google.com/go/go1.15.5.linux-amd64.tar.gz
tar -zxvf go1.15.5.linux-amd64.tar.gz -C /home/weiyigeek/app/program/go/
vi /etc/profile
export GOPATH=/home/weiyigeek/app/program/go/src/ # GOPATH 作为 go 执行的工作路径
export GOROOT=/home/weiyigeek/app/program/go # go 安装目录
export PATH=$PATH:${GOROOT}/bin
source /etc/profile
# 环境验证
$ env | grep "GO"
GOPATH=/home/weiyigeek/app/program/go/src/
GOROOT=/home/weiyigeek/app/program/go
$ go version
go version go1.15.5 linux/amd64 - Kubernetes 项目源码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18# 方式1: 拉取项目所有分支
mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io
> git clone https://github.com/kubernetes/kubernetes.git
# $ git clone https://github.com/kubernetes/kubernetes -b release-1.19.3 # 拉取Kubernetes源码指定分支
$ ls kubernetes
# api cluster go.mod logo pkg SUPPORT.md WORKSPACE
# build cmd go.sum Makefile plugin test
# BUILD.bazel code-of-conduct.md hack Makefile.generated_files README.md third_party
# CHANGELOG CONTRIBUTING.md LICENSE OWNERS SECURITY_CONTACTS translation
# 方式2: 拉取指定K8S版本源代码
K8S_VERSION=1.19.3
wget -c --no-check-certificate https://github.com/kubernetes/kubernetes/archive/v${K8S_VERSION}.tar.gz
> tar -tf v1.19.3.tar.gz | more
kubernetes-1.19.3/
kubernetes-1.19.3/.bazelrc
kubernetes-1.19.3/.bazelversion
kubernetes-1.19.3/.generated_files
Docker 镜像安装
描述: 安装Docker以及加速源的配置这里不再累述了，不会的读者参考官网或者本博客中的Docker安装流程;
1 | # 运行容器，并进入到容器内部 |
复制代码
vim staging/src/k8s.io/client-go/util/cert/cert.go # kubeadm 1.14 版本之前
vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go # kubeadm 1.14 至今
const duration365d = time.Hour 24 365 * 10 #设置为10年
NotAfter: time.Now().Add(duration365d).UTC(), #替换
make WHAT=cmd/kubeadm GOFLAGS=-v #设置只编译kubeadm
cp _output/bin/kubeadm /root/kubeadm-new
复制代码
⒌更新 kubeadm
#将kubeadm 进行替换
cp /usr/bin/kubeadm /usr/bin/kubeadm.old
cp /root/kubeadm-new /usr/bin/kubeadm
chmod a+x /usr/bin/kubeadm
⒍更新各节点证书至Master节点
cp -r /etc/kubernetes/pki /etc/kubernetes/pki.old
cd /etc/kubernetes/pki
kubeadm alpha certs renew all –config=/root/kubeadm-config.yaml
openssl x509 -in apiserver.crt -text -noout | grep Not
⒎HA集群其余 mater节点证书更新
复制代码
#!/bin/bash
masterNode=”192.168.66.20 192.168.66.21”
#for host in ${masterNode}; do
scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
“${USER}”@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} “root”@$host:/etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf “root”@$host:/etc/kubernetes/
#done
for host in ${CONTROL_PLANE_IPS}; do
scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
“${USER}”@$host:/root/pki/
scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} “root”@$host:/root/etcd
scp /etc/kubernetes/admin.conf “root”@$host:/root/kubernetes/
done
https://blog.csdn.net/aixiaoyang168/article/details/87917184
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm
go build -v .
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm
ls
app BUILD kubeadm kubeadm.go OWNERS test
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm
./kubeadm version
kubeadm version: &version.Info{Major:””, Minor:””, GitVersion:”v0.0.0-master+1e11e4a210802”, GitCommit:”1e11e4a2108024935ecfcb2912226cedeafd99df”, GitTreeState:””, BuildDate:”1970-01-01T00:00:00Z”, GoVersion:”go1.15.5”, Compiler:”gc”, Platform:”linux/amd64”}
实操1.Kubeadm / etcd证书过期或者延期处理流程
描述: 下述操作主要用于处理已过期或者即将过期的kubernetes集群证书;
Kubernetes 与 Etcd 版本与证书关联信息说明:
- Etcd 版本小于等于v1.9版本，etcd默认是不使用TLS连接，没有etcd相关证书，只需要更新master证书即可。
- Etcd 版本大于等于v1.10版本，etcd默认开启TLS，需要更新etcd证书和master证书。
1 | /etc/kubernetes/pki/etcd$ ls |
PS : 如果 Etcd 是由Kubeadm创建和托管的此时也可以通过下面的方式进行证书的续期, 如果是外部管理需要手动进行配置;
操作流程:
1.采用以下命令查询证书可用时间1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23# apiserver.crt (证书一年)
/etc/kubernetes/pki# openssl x509 -in apiserver.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3597153141676254659 (0x31eba7b94c4b25c3)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Nov 5 08:47:32 2020 GMT
Not After : Nov 5 08:47:32 2021 GMT
# ca.crt (证书十年)
/etc/kubernetes/pki# openssl x509 -in ca.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Nov 5 08:47:32 2020 GMT
Not After : Nov 3 08:47:32 2030 GMT
Subject: CN = kubernetes
2.查看当前集群证书相关信息1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20$ sudo kubeadm alpha certs check-expiration
# [check-expiration] Reading configuration from the cluster...
# [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
# CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY(认证中心) EXTERNALLY MANAGED(外部管理)
# admin.conf Nov 05, 2021 08:47 UTC 328d no
# apiserver Nov 05, 2021 08:47 UTC 328d ca no
# apiserver-etcd-client Nov 05, 2021 08:47 UTC 328d etcd-ca no
# apiserver-kubelet-client Nov 05, 2021 08:47 UTC 328d ca no
# controller-manager.conf Nov 05, 2021 08:47 UTC 328d no
# etcd-healthcheck-client Nov 05, 2021 08:47 UTC 328d etcd-ca no
# etcd-peer Nov 05, 2021 08:47 UTC 328d etcd-ca no
# etcd-server Nov 05, 2021 08:47 UTC 328d etcd-ca no
# front-proxy-client Nov 05, 2021 08:47 UTC 328d front-proxy-ca no
# scheduler.conf Nov 05, 2021 08:47 UTC 328d no
# CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
# ca Nov 03, 2030 08:47 UTC 9y no
# etcd-ca Nov 03, 2030 08:47 UTC 9y no
# front-proxy-ca Nov 03, 2030 08:47 UTC 9y no
3.采用renew子命令刷新证书的到期时间进行续期1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21~/.k8s$ sudo kubeadm alpha certs renew all --config=./kubeadm-init-config.yaml
# W1212 17:17:16.721037 1306627 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
# certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed # 嵌入在kubeconfig文件中的证书，供管理员使用，并对kubeadm本身进行更新 (admin.conf )
# certificate for serving the Kubernetes API renewed # 更新Kubernetes API服务证书
# certificate the apiserver uses to access etcd renewed # 服务器访问etcd所使用的证书已更新
# certificate for the API server to connect to kubelet renewed # API服务器连接到kubelet的证书已更新
# certificate embedded in the kubeconfig file for the controller manager to use renewed # 证书嵌入在kubeconfig文件中，供控制器管理器使用更新 (controller-manager.conf)
# certificate for liveness probes to healthcheck etcd renewed # 健康检查etcd激活探针证书续期
# certificate for etcd nodes to communicate with each other renewed # 用于etcd节点之间通信的证书更新
# certificate for serving etcd renewed # 续期etcd“服务证书”
# certificate for the front proxy client renewed # 前代理客户端的证书更新
# certificate embedded in the kubeconfig file for the scheduler manager to use renewed #证书嵌入在kubeconfig文件中，供调度器管理器使用更新 (scheduler.conf )
# 补充：如果环境里中etcd不是由Kubeadm创建并托管而是外部二进制部署的就不能采用此种方式更新etcd的证书
# 更新指定集群组件进行更新集群
~/.k8s$ kubeadm alpha certs renew
# admin.conf apiserver apiserver-kubelet-client etcd-healthcheck-client etcd-server scheduler.conf
# all apiserver-etcd-client controller-manager.conf etcd-peer front-proxy-client
kubeadm alpha certs renew apiserver --config=./kubeadmin-config.yaml
kubeadm alpha certs renew apiserver-kubelet-client --config=./kubeadmin-config.yaml
kubeadm alpha certs renew front-proxy-client --config=./kubeadmin-config.yaml
4.验证刷新证书后的到期时间以及证书文件属性查看1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39~/.k8s$ sudo kubeadm alpha certs check-expiration
# [check-expiration] Reading configuration from the cluster...
# [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
# PS -> 显示顺序与上面刷新证书的顺序是一致的
# CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
# admin.conf Dec 12, 2021 09:17 UTC 364d no
# apiserver Dec 12, 2021 09:17 UTC 364d ca no
# apiserver-etcd-client Dec 12, 2021 09:17 UTC 364d etcd-ca no
# apiserver-kubelet-client Dec 12, 2021 09:17 UTC 364d ca no
# controller-manager.conf Dec 12, 2021 09:17 UTC 364d no
# etcd-healthcheck-client Dec 12, 2021 09:17 UTC 364d etcd-ca no
# etcd-peer Dec 12, 2021 09:17 UTC 364d etcd-ca no
# etcd-server Dec 12, 2021 09:17 UTC 364d etcd-ca no
# front-proxy-client Dec 12, 2021 09:17 UTC 364d front-proxy-ca no
# scheduler.conf Dec 12, 2021 09:17 UTC 364d no
# CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
# ca Nov 03, 2030 08:47 UTC 9y no
# etcd-ca Nov 03, 2030 08:47 UTC 9y no
# front-proxy-ca Nov 03, 2030 08:47 UTC 9y no
/etc/kubernetes/pki$ stat apiserver.key
# File: apiserver.key
# Size: 1675 Blocks: 8 IO Block: 4096 regular file
# Device: fd00h/64768d Inode: 2099948 Links: 1
# Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
# Access: 2020-12-12 20:30:32.703551276 +0800 最近访问：
# Modify: 2020-12-12 17:17:17.055077737 +0800 最近更改：
# Change: 2020-12-12 17:17:17.055077737 +0800 最近改动：
# Birth: -
/etc/kubernetes/pki$ stat apiserver.crt
# File: apiserver.crt
# Size: 1294 Blocks: 8 IO Block: 4096 regular file
# Device: fd00h/64768d Inode: 2099949 Links: 1
# Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
# Access: 2020-12-12 17:19:54.202163815 +0800
# Modify: 2020-12-12 17:17:17.099078602 +0800
# Change: 2020-12-12 17:17:17.099078602 +0800
# Birth: -
5.基于新证书重新生成各个组件的配置文件1
2
3
4
5
6
7
8
9
10
11
12
13
14
15# (1) 主Master节点上运行生成各组件的配置文件
kubeadm alpha kubeconfig user --org system:masters --client-name kubernetes-admin > /etc/kubernetes/admin.conf
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf
# (2) 要开始使用您的集群，您需要运行以下作为一个常规用户:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# (3 )复制证书和配置文件到其他master
scp -r /etc/kubernetes/pki/* root@master@master-01:/etc/kubernetes/pki/
scp /etc/kubernetes/scheduler.conf root@master@master-01:/etc/kubernetes/
scp /etc/kubernetes/controller-manager.conf root@master@master-01:/etc/kubernetes/
scp /etc/kubernetes/admin.conf root@master@master-01:/etc/kubernetes/
6.重启kubelet组件,其他master同样重启kubelet和3个组件docker容器1
2
3systemctl restart kubelet
# docker容器重启controller、scheduler、apiserver3个组件
# docker restart
该脚本只处理master节点上的证书：kubeadm默认配置了kubelet证书自动更新，node节点kubelet.conf所指向的证书会自动更新
小于v1.17版本的master初始化节点(执行kubeadm init的节点) kubelet.conf里的证书并不会自动更新，这算是一个bug，该脚本会一并处理更新master节点的kubelet.conf所包含的证书
kubeadm生成的证书有效期为为1年，该脚本可将kubeadm生成的证书有效期更新为10年
Kubernetes集群版本在更新时，就会自动更新apiserver.crt证书的使用期限，这可能也是k8s官方设置这一年期限的原因 —— 为了让使用者跟上版本更新的步伐。
证书修改流程
由于k8s是基于kubeadm安装的，所以只需修改kubeadm源码中的证书期限即可。 kubeadm是采用go语言编写，所以我们要现在本机安装go语言环境，然后将对应版本的kubeadm源码进行修改，修改后重新编译，再用新编译的kubeadm生成新证书即可。
Kubeadm/etcd证书修改可用年限
描述: 在Vallidity节点下Kubernetes有两种机制去创建证书，有一部分是1年的(apiServer.key)，有1部分是10年的比如我们的(CA.CERT); Kubernetes集群版本在更新时，就会自动更新apiserver.crt证书的使用期限，这可能也是k8s官方设置这一年期限的原因为了让使用者跟上版本更新的步伐。但是有一些公司就会选取一个稳定版本一直使用下去，这便会使用本节所介绍的内容如何修改证书的使用年限。
操作流程:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97kub~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
> git checkout -b remotes/origin/release-1.19.3 v1.19.3 master [2bd115e3648]
切换到一个新分支 'remotes/origin/release-1.19.3'
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes
> vim staging/src/k8s.io/client-go/util/cert/cert.go remotes/origin/release-1.19.3 [1e11e4a2108]
38 const duration365d = time.Hour * 24 * 365
56 // NewSelfSignedCACert creates a CA certificate
57 func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
65 NotBefore: now.UTC(),
66 NotAfter: now.Add(duration365d * 10).UTC(),
86 // GenerateSelfSignedCertKeyWithFixtures creates a self-signed certificate and key for the given host.
94 func GenerateSelfSignedCertKeyWithFixtures(host string, alternateIPs []net.IP, alternateDNS []string, fixtureDirector y string) ([]byte, []byte, error) {
96 maxAge := time.Hour * 24 * 365 * 10 // one year self-signed certs
123 NotBefore: validFrom,
124 NotAfter: validFrom.Add(maxAge),
// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg *CertConfig, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
const duration365d = time.Hour * 24 * 365 * 10
591 NotBefore: caCert.NotBefore,
592 NotAfter: time.Now().Add(duration365d).UTC(),
}
结果在这里找到kubeadmconstants.CertificateValidity的定义
vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go
# 这个方法里面看到NotAfter: time.Now().Add(kubeadmconstants.CertificateValidity).UTC()
# 参数里面是一个常量kubeadmconstants.CertificateValidity
# 所以这里可以不修改，我去看看源码能不能找到这个常量的赋值位置
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
CertificateValidity = time.Hour * 24 * 365 * 10
NotBefore: caCert.NotBefore,
NotAfter: time.Now().Add(kubeadmconstants.CertificateValidity).UTC(),
vim ./cmd/kubeadm/app/constants/constants.go
const (
// KubernetesDir is the directory Kubernetes owns for storing various configuration files
KubernetesDir = "/etc/kubernetes"
// ManifestsSubDirName defines directory name to store manifests
ManifestsSubDirName = "manifests"
// TempDirForKubeadm defines temporary directory for kubeadm
// should be joined with KubernetesDir.
TempDirForKubeadm = "tmp"
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
CertificateValidity = time.Hour * 24 * 365 * 10
// CACertAndKeyBaseName defines certificate authority base name
CACertAndKeyBaseName = "ca"
// CACertName defines certificate name
CACertName = "ca.crt"
// CAKeyName defines certificate name
CAKeyName = "ca.key"
// APIServerCertAndKeyBaseName defines API's server certificate and key base name
APIServerCertAndKeyBaseName = "apiserver"
// APIServerCertName defines API's server certificate name
APIServerCertName = "apiserver.crt"
// APIServerKeyName defines API's server key name
APIServerKeyName = "apiserver.key"
// APIServerCertCommonName defines API's server certificate common name (CN)
APIServerCertCommonName = "kube-apiserver"
// APIServerKubeletClientCertAndKeyBaseName defines kubelet client certificate and key base name
APIServerKubeletClientCertAndKeyBaseName = "apiserver-kubelet-client"
// APIServerKubeletClientCertName defines kubelet client certificate name
APIServerKubeletClientCertName = "apiserver-kubelet-client.crt"
// APIServerKubeletClientKeyName defines kubelet client key name
APIServerKubeletClientKeyName = "apiserver-kubelet-client.key"
// APIServerKubeletClientCertCommonName defines kubelet client certificate common name (CN)
APIServerKubeletClientCertCommonName = "kube-apiserver-kubelet-client"
// EtcdCACertAndKeyBaseName defines etcd's CA certificate and key base name
EtcdCACertAndKeyBaseName = "etcd/ca"
// EtcdCACertName defines etcd's CA certificate name
EtcdCACertName = "etcd/ca.crt"
// EtcdCAKeyName defines etcd's CA key name
EtcdCAKeyName = "etcd/ca.key"
// EtcdServerCertAndKeyBaseName defines etcd's server certificate and key base name
EtcdServerCertAndKeyBaseName = "etcd/server"
// EtcdServerCertName defines etcd's server certificate name
EtcdServerCertName = "etcd/server.crt"
// EtcdServerKeyName defines etcd's server key name
EtcdServerKeyName = "etcd/server.key"
# 编译kubeadm, 这里主要编译kubeadm 即可
make WHAT=cmd/kubeadm GOFLAGS=-v
1 | weiyigeek@master:~$ sudo cp /usr/bin/kubeadm{,.bak} |
入坑出坑
问题 : Unable to connect to the server: x509: certificate has expired or is not yet valid
故障描述: 使用kubeadm部署的集群，在运行了一年之后今天，出现k8s api无法调取的现象，使用kubectl命令获取资源均返回如下报错1
Unable to connect to the server: x509: certificate has expired or is not yet valid
故障原因: 由于默认的apiserver.crt证书有效期只有一年其错误信息表示证书到期无法正常使用；
解决办法: 现在证书过期的问题已经发生了，只能通过更换证书的办法解决，在github上有详细的说明 issue链接如下:
https://github.com/kubernetes/kubeadm/issues/581
