🌐 3-Kubernetes入门之Ubuntu安装部署集群|WeiyiGeek Blog|唯一极客Geek-IT网络安全运维开发技术知识分享-博客站点

[toc]

0x00 前言简述

描述: 为了更好的学习kubernetes以及对照其在不同操作系统之下使用的性能以及差异进行学习扩容，下面将使用Ubuntu进行K8s集群的安装；

说明: 在上一章之中我们采用 CentOS7 + KUbernetes 1.18.x版本进行了安装演示，本章将使用Ubuntu 20.04 + Kubernetes 1.19.x版本 + IPVS + Ansible等组合工具进行安装演示;

PS : 虽然K8s 1.20版本宣布将在1.24版本之后将不再维护dockershim，意味着K8s将不直接支持Docker，不过大家不必过于担心。

一是在1.24版本之前我们仍然可以使用Docker，
二是dockershim肯定会有人接盘，我们同样可以使用Docker，当前【2022年4月18日 09:46:03】现已可以使用containerd替代ockershim
三是docker制作的镜像仍然可以在其他Runtime环境中使用，所以大家不必过于恐慌。

Tips : 经过测试发现本文方法在 docker(19.03.15 - 19.x) 以及 kubernetes(v1.19.10) 构建集群是没有任何问题的。

0x01 基础环境准备

描述: 有了2-Kubernetes入门之CentOS安装部署集群.md的基础进行对照在Ubuntu下安装K8s的不同

1.环境说明

# 此处是在VMware进行实际的
Ubuntu 20.04 TLS => 5.4.0-56-generic # 多台 Ubuntu 20.04 TLS 物理机或者虚拟机(安装流程请自行百度)此处已做基础安全加固(脚本参考)
kubernetes 1.19.6
  - kubectl 1.19.6
  - kubeadm 1.19.6
  - kubelet 1.19.6
Docker: 19.03.14
# Master 节点中一台机器安装
Ansible 2.9.6 
# 假设您所有节点都以做安全加固(SSH端口修改为20211)

主机说明:

# Master
weiyigek-107 
weiyigek-108
weiyigek-109

# worker
weiyigek-223
weiyigek-224
weiyigek-225
weiyigek-226

K8s 安装环境基础要求:

* 每台机器 2 GB 或更多的 RAM (如果少于这个数字将会影响您应用的运行内存)
* 每台机器 2 CPU 核或更多
* 集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)
* 保证机器主机名/网卡UUID和IP地址以及Mac地址唯一

PS : 注意Master与Node在一致的情况下进行下列安装(我们已做过安全加固)可能安装过程中，读者实践部署的时候有一定的出入(一般是依赖的软件未安装);

2.环境操作

Tips 注意: 所有主机都需要按照以下操作流程

Step 1.各Master与工作节点的机器名称及其设置;

# *- IP 地址修改 -* & *- 主机 名称修改 -*
mkdir ~/init/
tee ~/init/network.sh <<'EOF'
#!/bin/bash
CURRENT_IP=$(hostname -I | cut -f 1 -d " ")
GATEWAY=$(hostname -I | cut -f 1,2,3 -d ".")
if [[ $# -lt 3 ]];then
  echo "Usage: $0 IP Gateway Hostname"
  exit
fi
echo "IP:${1} # GATEWAY:${2} # HOSTNAME:${3}"
sudo sed -i "s#${CURRENT_IP}#${1}#g" /etc/netplan/00-installer-config.yaml
sudo sed -i "s#${GATEWAY}.1#${2}#g" /etc/netplan/00-installer-config.yaml
sudo hostnamectl set-hostname ${3} 
sudo netplan apply
EOF
chmod +x ~/init/network.sh 
sudo ~/init/network.sh 192.168.1.107 192.168.1.1 weiyigeek-107

Step 2.安装依赖软件以及关闭停用不需要使用的软件

# (1) 卸载系统中自带的 snapd 软件 
# sudo systemctl stop snapd snapd.socket 
# sudo apt autoremove --purge -y snapd 
# sudo apt install -y linux-modules-extra-5.4.0-52-generic linux-headers-5.4.0-52

# (2) 关闭自启和停止不用的软件  (Ubuntu server 20.04 最小安装不存在)
# sudo systemctl stop postfix
# sudo systemctl disable postfix

Step 3.各Master与工作节点的机器系统时间的同步与时区设置

# 设置系统时区为中国/上海
sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
sudo timedatectl set-timezone Asia/Shanghai
sudo bash -c "echo 'Asia/Shanghai' > /etc/timezone"
sudo ntpdate ntp1.aliyun.com

# 将当前的 UTC 时间写入硬件时钟
sudo timedatectl set-local-rtc 0

# 重启依赖于系统时间的服务
sudo systemctl restart rsyslog.service cron.service

# 查看系统时间
date -R

Step 4.虚拟内存swap分区关闭
Q: 什么安装K8s需要关闭SWAP虚拟内存?

答: 由于Kubeadm在进行K8s安装init初始化时候会检测系统中是否存在swap如果存在会在安装时候有一个警告，尽管您可以利用ingore来忽略它但是确实对性能有一定的影响；
例如有可能我们创建的Pod运行在虚拟内存上面与运行在物理内存中Pod不管是RW效率都比较低;

sudo swapoff -a && sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab && free  # CentOS
sudo swapoff -a && sudo sed -i 's/^\/swap.img\(.*\)$/#\/swap.img \1/g' /etc/fstab && free  #Ubuntu
# 不同点: Ubuntu 没有 CentOS 中的 selinux 所以无需关闭
# setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config 
# root@weiyigeek-107:~# free
#               total        used        free      shared  buff/cache   available
# Mem:        8151900      223192     7684708         880      244000     7674408
# Swap:       4194300           0     4194300
# root@weiyigeek-107:~# swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
# root@weiyigeek-107:~# free
#               total        used        free      shared  buff/cache   available
# Mem:        8151900      223312     7684356         880      244232     7674256
# Swap:             0           0           0

Step 5.为了能在 kube-proxy 开启并使用 ipvs 我们需要加载以下模块(所有节点执行);

# (1) 安装 ipvs 以及 负载均衡相关依赖
sudo apt -y install ipvsadm ipset sysstat conntrack 

# (2) ipvs 内核模块手动加载（所有节点配置）
mkdir ~/k8s-init/
tee ~/k8s-init/ipvs.modules <<'EOF'
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_lc
modprobe -- ip_vs_lblc
modprobe -- ip_vs_lblcr
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- ip_vs_dh
modprobe -- ip_vs_fo
modprobe -- ip_vs_nq
modprobe -- ip_vs_sed
modprobe -- ip_vs_ftp
modprobe -- ip_tables
modprobe -- ip_set
modprobe -- ipt_set
modprobe -- ipt_rpfilter
modprobe -- ipt_REJECT
modprobe -- ipip
modprobe -- xt_set
modprobe -- br_netfilter
modprobe -- nf_conntrack
EOF

# (3) 加载内核配置（临时|永久）注意管理员执行
chmod 755 ~/k8s-init/ipvs.modules && sudo bash ~/k8s-init/ipvs.modules
sudo cp ~/k8s-init/ipvs.modules /etc/profile.d/ipvs.modules.sh
lsmod | grep -e ip_vs -e nf_conntrack
# ip_vs_ftp              16384  0
# ip_vs_sed              16384  0
# ip_vs_nq               16384  0
# ip_vs_fo               16384  0
# ip_vs_dh               16384  0
# ip_vs_sh               16384  0
# ip_vs_wrr              16384  0
# ip_vs_rr               16384  0
# ip_vs_lblcr            16384  0
# ip_vs_lblc             16384  0
# ip_vs_lc               16384  0
# ip_vs                 155648  22 ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
# nf_nat                 40960  3 iptable_nat,xt_MASQUERADE,ip_vs_ftp
# nf_conntrack          139264  5 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
# nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
# libcrc32c              16384  5 nf_conntrack,nf_nat,btrfs,raid456,ip_vs

# (4) 下面的方式 Ubuntu 不能正确执行）
# ls /etc/modules-load.d/
# sudo systemctl enable --now systemd-modules-load.service

PS : kube-proxy 是开启使用IPVS的前提条件必须正常安装;

Step 6.集群各主机节点内核Kernel参数的调整( Kubernetes 安装前的必须进行该内核参数优化&配置)

# 1.Kernel 参数调整
mkdir ~/k8s-init/
cat > ~/k8s-init/kubernetes-sysctl.conf <<EOF
# iptables 网桥模式开启
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1

# 禁用 ipv6 协议
net.ipv6.conf.all.disable_ipv6=1

# 启用ipv4转发
net.ipv4.ip_forward=1 
# net.ipv4.tcp_tw_recycle=0 #Ubuntu 没有参数

# 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.swappiness=0
# 不检查物理内存是否够用
vm.overcommit_memory=1
# 不启 OOM
vm.panic_on_oom=0

# 文件系统通知数(根据内存大小和空间大小配置)
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576

# 文件件打开句柄数
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

# tcp keepalive 相关参数配置
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
# net.ipv4.ip_conntrack_max = 65536 # Ubuntu 没有参数
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 32768
EOF
sudo cp ~/k8s-init/kubernetes-sysctl.conf  /etc/sysctl.d/99-kubernetes.conf
sudo sysctl -p /etc/sysctl.d/99-kubernetes.conf

# 2.nftables 模式切换
# 在 Linux 中 nftables 当前可以作为内核 iptables 子系统的替代品,该工具可以充当兼容性层其行为类似于 iptables 但实际上是在配置 nftables。
$ apt list | grep "nftables/focal"
# nftables/focal 0.9.3-2 amd64
# python3-nftables/focal 0.9.3-2 amd64

# iptables 旧模式切换 (nftables 后端与当前的 kubeadm 软件包不兼容, 它会导致重复防火墙规则并破坏 kube-proxy, 所则需要把 iptables 工具切换到“旧版”模式来避免这些问题)
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
# sudo update-alternatives --set arptables /usr/sbin/arptables-legacy  # PS: # Ubuntu 20.04 TLS 无该模块
# sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy    # PS: # Ubuntu 20.04 TLS 无该模块

Step 7.主机名称设置与hosts文件添加

# 主机名称 以及 其他节点node主机绑定
# hostnamectl set-hostname weiyigeek-107
sudo tee -a /etc/hosts <<'EOF'
# dev & test  - master
192.168.1.107 weiyigeek-107
192.168.1.108 weiyigeek-108
192.168.1.109 weiyigeek-109

# dev & test  - work
192.168.1.223 weiyigeek-223
192.168.1.224 weiyigeek-224
192.168.1.225 weiyigeek-225
192.168.1.226 weiyigeek-226

# kubernetes-vip (如果不是高可用集群，该IP为Master01的IP)
192.168.1.110 weiyigeek-lb-vip.k8s
EOF

Step 8.设置 rsyslogd 和 systemd journald 记录

sudo mkdir -pv /var/log/journal/ /etc/systemd/journald.conf.d/
sudo tee /etc/systemd/journald.conf.d/99-prophet.conf <<'EOF'
[Journal]
# 持久化保存到磁盘
Storage=persistent

# 压缩历史日志
Compress=yes

SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000

# 最大占用空间 10G
SystemMaxUse=10G

# 单日志文件最大 100M
SystemMaxFileSize=100M

# 日志保存时间 2 周
MaxRetentionSec=2week

# 不将日志转发到syslog
ForwardToSyslog=no
EOF
cp /etc/systemd/journald.conf.d/99-prophet.conf ~/k8s-init/journald-99-prophet.conf
sudo systemctl restart systemd-journald

Step 9.在各个主机中安装 docker软件

# 1.卸载旧版本 
sudo apt-get remove docker docker-engine docker.io containerd runc

# 2.更新apt包索引并安装包以允许apt在HTTPS上使用存储库
sudo apt-get install -y \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg-agent \
  software-properties-common

# 3.添加Docker官方GPG密钥 # -fsSL
curl https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# 4.通过搜索指纹的最后8个字符进行密钥验证
sudo apt-key fingerprint 0EBFCD88

# 5.设置稳定存储库
sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

# 6.Install Docker Engine 默认最新版本
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo apt-get install -y docker-ce=5:19.03.15~3-0~ubuntu-focal docker-ce-cli=5:19.03.15~3-0~ubuntu-focal containerd.io
# 7.安装特定版本的Docker引擎，请在repo中列出可用的版本
# $apt-cache madison docker-ce
# docker-ce | 5:20.10.2~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
# docker-ce | 5:18.09.1~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu  xenial/stable amd64 Packages
# 使用第二列中的版本字符串安装特定的版本，例如:5:18.09.1~3-0~ubuntu-xenial。
# $sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

#8.将当前用户加入docker用户组然后重新登陆当前用户使得低权限用户
sudo gpasswd -a ${USER} docker
sudo gpasswd -a weiyigeek docker

#9.加速器建立
mkdir -vp /etc/docker/
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://xlx9erfu.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "live-restore": true,
  "dns": ["192.168.12.254"],
  "insecure-registries": ["harbor.weiyigeek.top"]
}
EOF
# PS : 私有仓库配置 insecure_registries

# 9.自启与启动
sudo systemctl enable --now docker 
sudo systemctl restart docker

# 10.退出登陆生效
exit

Step 10.先在 WeiyiGeek-107 机器中下载 kubernetes 集群相关的软件包并准备安装;

# (1) gpg 签名下载导入
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
# (2) Kubernetes 安装源
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list 
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

# (3) 软件包索引更新以及只下载依赖包不安装kubernetes(注意此处建议采用指定版本下载 )
apt-cache madison kubelet kubeadm kubectl # 查看可用的 kubernetes 版本 最新 1.20.1
sudo apt-get update && sudo apt -d install kubelet kubeadm kubectl

kubelet --version
# Kubernetes v1.20.1
kubeadm version
# kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:07:13Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
kubectl version
# Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

# 也可以指定版本下载(ls /var/cache/apt/archives/  # 缓存目录) 
# sudo apt -d install kubelet=1.19.10-00  kubeadm=1.19.10-00 kubectl=1.19.10-00
# 如果只是下载deb相关包以供离线安装可以使用以下方式安装
sudo dpkg -i k8s-1.19.3/*.deb

# (2) systemd 守护进程重启
sudo systemctl daemon-reload  

# (3) Kubelet是以POD的方式运行在容器之中，所以需要设置开机自启
sudo systemctl enable --now kubelet.service

# (4) 重启 docker
sudo systemctl restart docker

Step 11.关机此时克隆此Master机器为两台新的虚拟机机器;

1
2
3

# (1) 按照上面的清单设置主机名称以及IP地址
./network.sh 192.168.1.108 192.168.1.1 weiyigeek-108
./network.sh 192.168.1.109 192.168.1.1 weiyigeek-109

0x02 单实例K8s集群部署(v1.20.1)

1.Master 节点初始化

Step 1.单实例 Master 节点初始化以及集群资源清单配置(注意在前面的环境之下)

1
2
3

# (1) 在 WeiyiGeek-107 Master 节点上运行 `kubeadm config print init-defaults` 查看初始化参数
# 方式1.简单命令
kubeadm init --kubernetes-version=1.20.1 --apiserver-advertise-address=192.168.1.201 --image-repository registry.aliyuncs.com/google_containers --service-cidr=10.2.0.0/16 --pod-network-cidr=10.3.0.0/16

# 方式2.yaml方式由于此处没做高可以用则APISERVER指向集群的Master节点即weiyigeek-107机器
K8SVERSION=1.20.1
k8SIMAGEREP="registry.cn-hangzhou.aliyuncs.com/google_containers"
APISERVER_IP=192.168.1.107
APISERVER_NAME=weiyigeek.k8s
APISERVER_PORT=6443
SERVICE_SUBNET=10.244.0.0/16  # Flannel 网络插件默认网段

# 注意点Token的格式
cat <<EOF > ~/k8s-init/kubeadm-init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 123456.httpweiyigeektop
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
# API Server 地址与端口
localAPIEndpoint:
  advertiseAddress: ${APISERVER_IP}
  bindPort: ${APISERVER_PORT}
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ubuntu
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: ${k8SIMAGEREP}
kind: ClusterConfiguration
kubernetesVersion: v${K8SVERSION}
controlPlaneEndpoint: "${APISERVER_NAME}:${APISERVER_PORT}"
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: ${SERVICE_SUBNET}
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1    
kind: KubeProxyConfiguration    
featureGates:      
  SupportIPVSProxyMode: true    
mode: ipvs
EOF

k8s 集群创建初始命令:

sudo kubeadm init --upload-certs --config=/home/weiyigeek/k8s-init/kubeadm-init-config.yaml -v 5 | tee kubeadm_init.log
# W1104 21:29:36.119447  198575 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
# [init] Using Kubernetes version: v1.20.1
# [preflight] Running pre-flight checks
# [preflight] Pulling images required for setting up a Kubernetes cluster
# [preflight] This might take a minute or two, depending on the speed of your internet connection
# [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
# [certs] Using certificateDir folder "/etc/kubernetes/pki"
# .....
# [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
# [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
# [addons] Applied essential addon: CoreDNS
# [addons] Applied essential addon: kube-proxy

# Your Kubernetes control-plane has initialized successfully!  # 表示控制化平面初始化成功

2.集群管理

描述: 集群管理配置控制平面

# (1) 要开始使用集群，您需要以普通用户的身份运行在 Master 节点 执行以下命令:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
grep "KUBECONFIG" ~/.profile || echo "export KUBECONFIG=~/.kube/config" >> ~/.profile

3.集群网络插件安装

# 部署集群 pod 网络可以选择 flannel 
# Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
#   https://kubernetes.io/docs/concepts/cluster-administration/addons/
# 例如 安装 flannel 网络插件
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

4.工作节点加入

# (1) 你现在可以以root身份加入任意数量的控制平面节点，在每个节点上运行以下命令:
  kubeadm join weiyigeek.k8s:6443 --token 123456.httpweiyigeektop \
    --discovery-token-ca-cert-hash sha256:95e1bb846a09a4523be6c1ee6d3860eec1dcfdd16200efec5177ff25a1de49a6 \
    --control-plane --certificate-key e05180fc473a8b89e4616412dac61b95cf02808fe1a27f9f72c2be921acc63f8
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

# (2) 你可以加入任意数量的worker节点，在每个worker节点上以root用户运行如下命令:
sudo kubeadm join weiyigeek.k8s:6443 --token 123456.httpweiyigeektop --discovery-token-ca-cert-hash sha256:95e1bb846a09a4523be6c1ee6d3860eec1dcfdd16200efec5177ff25a1de49a6
[sudo] password for weiyigeek:
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

# This node has joined the cluster:  # 表示该节点已经加入到集群中

# Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

4.Token 失效重生成

Step 1.如果超过24小时 token 失效需要重新生成

# 1) Token 生成
kubeadm token create
# 2q41vx.w73xe9nrlqdujawu     ##此处是新token

# 2) 获取CA（证书）公钥哈希值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^ .* //'

# 3) kubernetes 证书目录
$ ls /etc/kubernetes/pki
apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  etcd                front-proxy-ca.key      front-proxy-client.key  sa.pub
apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key

5.节点重置

Step 1.当 Kuberneters 集群需要清理重构建需或者节点脱离集群时可以按照以下操作进行:

# - Master 工作负载移除指定集群 例如删除 weiyigeek-223 weiyigeek-222 工作节点
kubectl cordon weiyigeek-223 weiyigeek-222 
kubectl delete node weiyigeek-223 weiyigeek-222 
# 采用 ipvs 进行负载的时候需要清理`ipvsadm --clear`
sudo kubeadm reset
sudo rm -rf /etc/cni/net.d/*
sudo rm -rf /etc/kubernetes/pki/*
sudo rm -rf $HOME/.kube/config

# - Node 工作节点移除k8s集群
sudo kubeadm reset
sudo rm -rf /etc/cni/net.d/*

补充说明: 如果是在单master节点设置使用 calico 为 k8s 的CNI网络插件时

K8SVERSION=1.20.1
APISERVER_IP=192.168.1.107
APISERVER_NAME=k8s.weiyigeek.top
APISERVER_PORT=6443
SERVICE_SUBNET=10.99.0.0/16
POD_SUBNET=10.100.0.1/16
echo "${APISERVER_IP} ${APISERVER_NAME}" >> /etc/hosts

# 初始化配置(建议各个组件的版本与k8s的版本一致)
rm -f ./kubeadm-config.yaml
cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v${K8SVERSION}
imageRepository: mirrorgcrio
#imageRepository: registry.aliyuncs.com/google_containers
#imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
#imageRepository: gcr.azk8s.cn/google_containers
controlPlaneEndpoint: "${APISERVER_NAME}:${APISERVER_PORT}"
networking:
  serviceSubnet: "${SERVICE_SUBNET}"
  podSubnet: "${POD_SUBNET}"
  dnsDomain: "cluster.local"
EOF

# kubeadm init 根据您服务器网速的情况，您需要等候 3 - 10 分钟
kubeadm init --config=kubeadm-config.yaml --upload-certs

0x03 高可用K8s集群部署(v1.19.6)

Step 1.高可用组件安装所有Master节点通过apt安装HAProxy和KeepAlived

# PS: 此时只下载不安装(方便传到没有网络的master节点上)
sudo apt -d install keepalived haproxy
# ~/k8s-init$ scp -P 20211 /home/weiyigeek/k8s-init/High-Availability/* weiyigeek@192.168.1.108:~/k8s-init/
# ~/k8s-init$ scp -P 20211 /home/weiyigeek/k8s-init/High-Availability/* weiyigeek@192.168.1.109:~/k8s-init/

# 通过 dpkg 命令 安装 上一步下载的 deb包
/var/cache/apt/archives$ dpkg -i *.deb
# ~/k8s-init$ ssh -p20211 weiyigeek@192.168.1.108 "sudo -S dpkg -i ~/k8s-init/*.deb"
# ~/k8s-init$ ssh -p20211 weiyigeek@192.168.1.109 "sudo -S dpkg -i ~/k8s-init/*.deb"

Step 2.所有Master节点配置HAProxy

PS : 详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同

# Ubuntu haproxy 配置文件目录
$ls /etc/haproxy/
errors/      haproxy.cfg

$sudo vim /etc/haproxy/haproxy.cfg
global
  log /dev/log    local0
  log /dev/log    local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
  log     global
  mode    http
  option  httplog
  option  dontlognull
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

# 注意: 16443 为VIP的Apiserver-控制平面端口
frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

# 注意: Master 节点的默认apiserver是6443端口
backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server weiyigeek-107 192.168.1.107:6443  check
  server weiyigeek-108 192.168.1.108:6443  check
  server weiyigeek-109 192.168.1.109:6443  check

所有Master节点配置KeepAlived，配置不一样，注意区分注意每个节点的IP和网卡（interface参数）

mkdir -vp /etc/keepalived
# weiyigeek-107 Master 1
sudo tee /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
script_user root
  enable_script_security
}
vrrp_script chk_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 5
  weight -5
  fall 2  
  rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens160
    mcast_src_ip 192.168.1.107
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.1.110
    }
#    track_script {
#       chk_apiserver
#    }
}
EOF

# weiyigeek-108 Master 2 => Backup
sudo tee /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
script_user root
  enable_script_security
}
vrrp_script chk_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 5
  weight -5
  fall 2  
  rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    mcast_src_ip 192.168.1.108
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.1.110
    }
#    track_script {
#       chk_apiserver
#    }
}
EOF

# weiyigeek-108 Master 3 => Backup
sudo tee /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
script_user root
  enable_script_security
}
vrrp_script chk_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 5
  weight -5
  fall 2  
  rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    mcast_src_ip 192.168.1.109
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.1.110
    }
#    track_script {
#       chk_apiserver
#    }
}
EOF

Step 3.配置KeepAlived健康检查文件

sudo tee /etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash
err=0
for k in $(seq 1 3)
do
  check_code=$(pgrep haproxy)
  if [[ $check_code == "" ]]; then
    err=$(expr $err + 1)
    sleep 1
    continue
  else
    err=0
    break
  fi
done

if [[ $err != "0" ]]; then
  echo "systemctl stop keepalived"
  /usr/bin/systemctl stop keepalived
  exit 1
else
  exit 0
fi
EOF
sudo chmod +x /etc/keepalived/check_apiserver.sh

# 备份高可用相关配置文件
mkdir ~/k8s-init/haproxy && cp /etc/haproxy/haproxy.cfg ~/k8s-init/haproxy 
mkdir ~/k8s-init/keepalived && cp /etc/keepalived/keepalived.conf /etc/keepalived/check_apiserver.sh ~/k8s-init/keepalived

Step 4.启动haproxy和keepalived以及测试VIP

# 自启动并立即启动
sudo systemctl daemon-reload
sudo systemctl enable --now haproxy
sudo systemctl enable --now keepalived

# VIP 漂移测试
~$ ps aux | grep "haproxy"
root         892  0.0  0.1  15600  9236 ?        Ss   10:39   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy      893  0.0  0.0 531724  3480 ?        Sl   10:39   0:02 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

weiyigeek-107:$ ip addr
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:8a:e8:db brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.107/24 brd 192.168.1.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet 192.168.1.110/32 scope global ens160  # 可以看见该VIP在weiyigeek-107
       valid_lft forever preferred_lft forever

weiyigeek-107:~$ systemctl stop keepalived.service  # 现在 weiyigeek-107 停止 keepalived.service
weiyigeek-107:~$ ip addr | grep "192.168.1.110"    # 此时会发现 weiyigeek-107 已经不存在该VIP了
weiyigeek-107:~$ ssh -p20211 weiyigeek@192.168.1.109 "ip addr | grep '192.168.1.110'" # 会发现漂移到weiyigeek-109的master节点的机器上
  inet 192.168.1.110/32 scope global ens160
weiyigeek-107:~$ ping 192.168.1.110               # ping 通信正常
# PING 192.168.1.110 (192.168.1.110) 56(84) bytes of data.
# 64 bytes from 192.168.1.110: icmp_seq=1 ttl=64 time=0.218 ms

PS : 至此基于Haproxy与keepalive实现的高可用已经OK了;

Step 5.集群安装其它指定版本

PS : 在基础环境的安装中我们采用了 docker 1.20.x 以及 kubernetes 1.20.x考虑到 k8s 在 1.23.x 版本之后将会丢弃docker而使用其它的CRI-O, 此处为了集群环境的稳定性将原本docker以及kubernetes版本删除，分别安装1.19.x版本;

# (1) 停止服务
sudo systemctl stop docker.service kubelet.service docker.socket

# (2) docker 容器卸载 && K8s 卸载
sudo apt-get remove docker-ce docker-ce-cli containerd.io kubectl kubeadm kubelet

# (3) 查看 docker 以及 k8s 版本
sudo apt-cache madison docker-ce docker-ce-cli kubelet
# sudo apt-get -d install docker-ce=5:19.03.14~3-0~ubuntu-focal docker-ce-cli=5:19.03.14~3-0~ubuntu-focal containerd.io  # Download complete and in download only mode
# sudo dpkg -i /var/cache/apt/archives/*.deb
sudo apt-get install docker-ce=5:19.03.14~3-0~ubuntu-focal docker-ce-cli=5:19.03.14~3-0~ubuntu-focal containerd.io -y
sudo apt install kubelet=1.19.6-00 kubeadm=1.19.6-00 kubectl=1.19.6-00 -y
$ kubeadm version
# kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
$ kubelet --version
# Kubernetes v1.19.6
$ kubectl version
# Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

# (4) 自启并启动服务
sudo systemctl daemon-reload
sudo systemctl enable --now docker 
sudo systemctl enable --now kubelet
sudo systemctl restart docker kubelet

Step 6.集群初始化的yaml文件

描述: kubeadm 可以打印出初始化以及节点加入的配置模板;

# (1) 初始化
$ kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ubuntu
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

# (2) 节点加入
$ kubeadm config print join-defaults
apiVersion: kubeadm.k8s.io/v1beta2
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: kube-apiserver:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ubuntu
  taints: null

K8s 集群初始化配置文件(~/k8s-init/kubeadm-init-config.yaml)

# (1) 注意 : Token的格式以及Pod子网的网段 (Calico)和ipvs的支持
K8SVERSION=1.19.6
k8SIMAGEREP="registry.cn-hangzhou.aliyuncs.com/google_containers"
APISERVER_NAME=weiyigeek-lb-vip.k8s
APISERVER_IP=192.168.1.110
APISERVER_PORT=16443
LOCALAPIVERVER_NAME=weiyigeek-107
LOCALAPISERVER_IP=192.168.1.107
LOCALAPISERVER_PORT=6443
SERVICE_SUBNET=172.16.0.0/16

cat <<EOF > ~/k8s-init/kubeadm-init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 20w21w.httpweiyigeektop
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: ${LOCALAPISERVER_IP}
  bindPort: ${LOCALAPISERVER_PORT}
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ${LOCALAPIVERVER_NAME}
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - ${APISERVER_IP}
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: ${k8SIMAGEREP}
kind: ClusterConfiguration
kubernetesVersion: v${K8SVERSION}
controlPlaneEndpoint: ${APISERVER_NAME}:${APISERVER_PORT}
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: ${SERVICE_SUBNET}
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1    
kind: KubeProxyConfiguration    
featureGates:      
  SupportIPVSProxyMode: true    
mode: ipvs
EOF

# (2) 初始化Master节点控制平面以及Worker加入的参数生成(# 根据您服务器网速的情况，您需要等候 3 - 10 分钟建议采用下面备注中的操作)
sudo kubeadm init --config=/home/weiyigeek/k8s-init/kubeadm-init-config.yaml --upload-certs | tee kubeadm_init.log
# [certs] Using certificateDir folder "/etc/kubernetes/pki"
# [certs] Generating "ca" certificate and key
# [certs] Generating "apiserver" certificate and key
# [certs] apiserver serving cert is signed for DNS names [weiyigeek-107 weiyigeek-lb-vip.k8s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.107 192.168.1.110]
# [certs] Generating "apiserver-kubelet-client" certificate and key
# [certs] Generating "front-proxy-ca" certificate and key
# [certs] Generating "front-proxy-client" certificate and key
# [certs] Generating "etcd/ca" certificate and key
# [certs] Generating "etcd/server" certificate and key
# [certs] etcd/server serving cert is signed for DNS names [weiyigeek-107 localhost] and IPs [192.168.1.107 127.0.0.1 ::1]
# [certs] Generating "etcd/peer" certificate and key
# [certs] etcd/peer serving cert is signed for DNS names [weiyigeek-107 localhost] and IPs [192.168.1.107 127.0.0.1 ::1]
# [certs] Generating "etcd/healthcheck-client" certificate and key
# [certs] Generating "apiserver-etcd-client" certificate and key
# [certs] Generating "sa" key and public key
# [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "admin.conf" kubeconfig file
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "kubelet.conf" kubeconfig file
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "controller-manager.conf" kubeconfig file
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "scheduler.conf" kubeconfig file
# [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
# [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
# [kubelet-start] Starting the kubelet
# [control-plane] Using manifest folder "/etc/kubernetes/manifests"
# [control-plane] Creating static Pod manifest for "kube-apiserver"
# [control-plane] Creating static Pod manifest for "kube-controller-manager"
# [control-plane] Creating static Pod manifest for "kube-scheduler"
# [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
# [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
# [apiclient] All control plane components are healthy after 21.750931 seconds
# [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
# [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
# [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
# [upload-certs] Using certificate key:
# bb5b2f0b287d35e179ef4efawwww9f61a38f62343a9b06fc143e3b
# [mark-control-plane] Marking the node weiyigeek-107 as control-plane by adding the label "node-role.kubernetes.io/master=''"
# [mark-control-plane] Marking the node weiyigeek-107 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
# [bootstrap-token] Using token: 2021wq.httpweiyigeektop
# [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
# [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
# [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
# [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
# [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
# [addons] Applied essential addon: CoreDNS
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

PS : 将该yaml文件复制到其他master节点，之后所有Master节点提前下载镜像，可以节省初始化时间：

# 方式1.配置文件上传到其他master节点
scp -P 20211 /home/weiyigeek/k8s-init/kubeadm-init-config.yaml weiyigeek@192.168.1.108:~/k8s-init/kubeadm-init-config.yaml
kubeadm config images pull --config /home/weiyigeek/k8s-init/kubeadm-init-config.yaml
# W0111 11:24:15.905316    5481 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

Step 7.Kubernetes 集群访问配置

# (1) 普通用户对集群访问配置文件设置
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# (2) 自动运行设置 KUBECONFIG 环境以及k8s命令自动补齐
grep "export KUBECONFIG" ~/.profile | echo "export KUBECONFIG=$HOME/.kube/config" >> ~/.profile
tee -a ~/.profile <<'EOF'
source <(kubectl completion bash)
source <(kubeadm completion bash)
# source <(kubelet completion bash)
# source <(helm completion bash)
EOF
source ~/.profile

# (3) 查看节点状态和kube-system命名空间内Pod状态
~$ kubectl get node
# NAME       STATUS     ROLES    AGE   VERSION
# weiyigeek-107   NotReady   master   45m   v1.19.6   # 此处NotReady 是由于我们的calico网络插件未配置好
~$ kubectl get pod -n kube-system
# NAME                               READY   STATUS    RESTARTS   AGE
# coredns-6c76c8bb89-6cl4d           0/1     Pending   0          45m
# coredns-6c76c8bb89-xzkms           0/1     Pending   0          45m
# etcd-weiyigeek-107                      1/1     Running   0          45m
# kube-apiserver-weiyigeek-107            1/1     Running   0          45m
# kube-controller-manager-weiyigeek-107   1/1     Running   0          45m
# kube-proxy-l7bp6                   1/1     Running   0          45m
# kube-scheduler-weiyigeek-107            1/1     Running   0          45m

PS : 当没有安装第三方的 flannel或者calico网络插件 时候，通过kubectl get node 查看到节点状态是 NotReady ，当安装完成扁平化网络插件后, 如果没有其它意外则显示Ready

Step 8.现在应该将pod网络部署到集群,此处选用calico网络插件而非使用flannel插件

# Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
# https://kubernetes.io/docs/concepts/cluster-administration/addons/

# 安装 calico 网络插件(此处安装最新的v3.17.1版本不采用kuboard脚本)
# 参考文档 https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
# wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
# kubectl apply -f calico-3.13.1.yaml

# (1) Install Calico with Kubernetes API datastore, 50 nodes or less
# curl https://docs.projectcalico.org/manifests/calico.yaml -O

# (2) Install Calico with etcd datastore (使用etcd数据存储安装Calico) 
curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -O

# calico-etcd 网络与etc集群连接修改(此处指定pod子网地址)
ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
ETCD_KEY=`sudo cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
POD_SUBNET=`sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml

sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.12.107:2379,https://192.168.12.108:2379,https://192.168.12.109:2379"#g' calico-etcd.yaml
sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml

# (3) 部署到集群中
kubectl apply -f calico-etcd.yaml
# secret/calico-etcd-secrets created
# configmap/calico-config created
# clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
# clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
# clusterrole.rbac.authorization.k8s.io/calico-node created
# clusterrolebinding.rbac.authorization.k8s.io/calico-node created
# daemonset.apps/calico-node created
# serviceaccount/calico-node created
# deployment.apps/calico-kube-controllers created
# serviceaccount/calico-kube-controllers created
# poddisruptionbudget.policy/calico-kube-controllers created

# (4) 只在 master 节点执行
# 执行如下命令，等待 3-10 分钟，直到所有的容器组处于 Running 状态
watch kubectl get pod -n kube-system -o wide
echo -e "---等待容器组构建完成---" && sleep 180
kubectl get nodes -o wide       # 查看 master 节点初始化结果

# (5) 各节点网卡地址已经未设置的POD子网地址
~$ route
# Kernel IP routing table
# Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
# default         _gateway        0.0.0.0         UG    0      0        0 ens160
# default         _gateway        0.0.0.0         UG    0      0        0 ens160
# 172.16.0.192    0.0.0.0         255.255.255.192 U     0      0        0 *
# 172.16.0.193    0.0.0.0         255.255.255.255 UH    0      0        0 calib6c39f0a1d5
# 172.16.0.194    0.0.0.0         255.255.255.255 UH    0      0        0 calic77cbbaf4da
# 172.16.24.192   weiyigeek-223        255.255.255.192 UG    0      0        0 tunl0
# 172.16.100.64   weiyigeek-224        255.255.255.192 UG    0      0        0 tunl0
# 172.16.135.192  weiyigeek-108        255.255.255.192 UG    0      0        0 tunl0
# 172.16.182.192  weiyigeek-226        255.255.255.192 UG    0      0        0 tunl0
# 172.16.183.64   weiyigeek-225        255.255.255.192 UG    0      0        0 tunl0
# 172.16.243.64   weiyigeek-109        255.255.255.192 UG    0      0        0 tunl0

Step 9.高可用Master初始化，将其他master节点加入集群控制平面

sudo kubeadm join weiyigeek-lb-vip.k8s:16443 --token 20w21w.httpweiyigeektop \
  --discovery-token-ca-cert-hash sha256:7ea900ef214c98aef6d7daf1380320d0a43f666f2d4b6b7469077bd51790118e \
  --control-plane --certificate-key 8327482265975b7a60f3549222f1093353ecaa148a3404cd10c605d4111566fc

PS : 作为安全措施，上传的证书将在两小时内被删除; 如果需要您可以使用下面的命令重新加载证书。
"kubeadm init phase upload-certs --upload-certs"

# 查看 Token 
~/k8s-init$ kubectl get secret
  # NAME                  TYPE                                  DATA   AGE
  # default-token-xzcbz   kubernetes.io/service-account-token   3      17m

Step 10.工作节点或者负载加入到集群中

# (1) 为了便于工作节点快速接入到集群中，我们先把master中的镜像导入到工作节点
docker save -o v1.19.6.tar $(docker images | grep -v TAG | cut -d ' ' -f1)  # 导出
docker load -i v1.19.6.tar   # 加载

# (2) 然后你可以加入任意数量的worker节点，在每个worker节点上以root用户运行如下命令:
sudo kubeadm join weiyigeek-lb-vip.k8s:16443 --token 20w21w.httpweiyigeektop \
  --discovery-token-ca-cert-hash sha256:7ea900ef214c98aef6d7daf1380320d0a43f666f2d4b6b7469077bd51790118e

Step 11.集群状态查看 & 镜像查看

# (1) Master 集群状态
weiyigeek-107:~$ kubectl get node -o wide
NAME       STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
weiyigeek-107   Ready    master   25m    v1.19.6   192.168.1.107   <none>        Ubuntu 20.04.1 LTS   5.4.0-60-generic   docker://19.3.14
weiyigeek-108   Ready    master   15m    v1.19.6   192.168.1.108   <none>        Ubuntu 20.04.1 LTS   5.4.0-60-generic   docker://19.3.14
weiyigeek-109   Ready    master   100s   v1.19.6   192.168.1.109   <none>        Ubuntu 20.04.1 LTS   5.4.0-60-generic   docker://19.3.14

# (2) K8s 集群 SVC 信息查看
~$ kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   32m   <none>
~$ kubectl get svc -o wide -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   31m   k8s-app=kube-dns

# (3) 为了可以节省资源将其余两台Master节点关闭污点
kubectl taint node weiyigeek-108 node-role.kubernetes.io/master=:NoSchedule-
kubectl taint node weiyigeek-109 node-role.kubernetes.io/master=:NoSchedule-

0x04 高可用集群使用初体验

描述: 简单对nginx应用从镜像打包到部署到k8s集群中使用流程进行演示

Step 1.编写dockerfile和相关脚本
$ cat dockerfile

FROM nginx
LABEL maintainer="Nginx Test Demo , Authorr: weiyigeek, Version: 2.2"
ENV PATH /usr/local/nginx/sbin:$PATH
ENV IMAGE_VERSION 2.2
COPY ./host.sh /
RUN chmod a+x /host.sh
ENTRYPOINT ["/host.sh"]

$ cat host.sh

#!/bin/sh
echo "Hostname: $HOSTNAME" > /usr/share/nginx/html/host.html
echo "Image Version: $IMAGE_VERSION" >> /usr/share/nginx/html/host.html
echo "Nginx Version: $NGINX_VERSION" >> /usr/share/nginx/html/host.html
sh -c "nginx -g 'daemon off;'"

Step 2.镜像构建 & 上传到私有得Harbor仓库:

$ docker build -t harbor.weiyigeek.top/test/nginx:v2.2 .
$ docker images | grep "v2.2"
  # harbor.weiyigeek.top/test/nginx                                  v2.2                b8a212b2bc88        10 hours ago        133MB
$ docker push harbor.weiyigeek.top/test/nginx:v2.2
  # The push refers to repository [harbor.weiyigeek.top/test/nginx]
  # v2.2: digest: sha256:4c49fc25d52e5331146699ff605561f59fb326505074c0474a3ce4898f0fcb02 size: 1776

Step 3.部署镜像 & 查看:

方式1

# 方式1.不推荐(配置标签以及标签选择需要添加参数，比较麻烦)
$ kubectl run nginx-deployment --image=harbor.weiyigeek.top/test/nginx:v2.2 --port=80
  # pod/nginx-deployment created

$ kubectl get pod -o wide
  # NAME               READY   STATUS    RESTARTS   AGE    IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment   1/1     Running   0          108s   10.244.1.2   k8s-node-4   <none>           <none>

# 利用podSubnet进行查看
weiyigeek@ubuntu:~$ curl http://10.244.1.2/host.html
  # Hostname: nginx-deployment
  # Image Version: 2.2
  # Nginx Version: 1.19.4

方式2

cat > nginx-deployment.yaml <<'EOF'
apiVersion: apps/v1	#与k8s集群版本有关，使用 kubectl api-versions 即可查看当前集群支持的版本
kind: Deployment   	#该配置的类型，我们使用的是 Deployment
metadata:	          #译名为元数据，即 Deployment 的一些基本属性和信息
  name: nginx-deployment	#Deployment 的名称
  namespace: default 
  labels:	      #标签可以灵活定位一个或多个资源，其中key和value均可自定义，可以定义多组，目前不需要理解
    app: nginx	#为该Deployment设置key为app，value为nginx的标签
spec:	          #这是关于该Deployment的描述，可以理解为你期待该Deployment在k8s中如何使用
  replicas: 1  	#使用该Deployment创建一个应用程序实例
  selector:	    #标签选择器，与上面的标签共同作用，目前不需要理解
    matchLabels: #选择包含标签app:nginx的资源
      app: nginx
  template:	     #这是选择或创建的Pod的模板
    metadata:	   #Pod的元数据
      labels:    #Pod的标签，上面的selector即选择包含标签app:nginx的Pod
        app: nginx
    spec:	        #期望Pod实现的功能(即在pod中部署)
      containers:	#生成container，与docker中的container是同一种
      - name: nginx	#container的名称
        image: harbor.weiyigeek.top/test/nginx:v2.2 	#使用镜像nginx最新版本创建container，该container默认80端口可访问
EOF

$ kubectl apply -f nginx-deployment.yaml
  # deployment.apps/nginx-deployment created

$ kubectl get deployment
  # NAME               READY   UP-TO-DATE   AVAILABLE   AGE
  # nginx-deployment   1/1     1            1           87s
  # weiyigeek@ubuntu:~/nginx$ kubectl get pod
  # NAME                                READY   STATUS    RESTARTS   AGE
  # nginx-deployment-7f5d9779c6-flmsf   1/1     Running   0          92s
  # weiyigeek@ubuntu:~/nginx$ kubectl get pod -o wide
  # NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment-7f5d9779c6-flmsf   1/1     Running   0          99s   10.244.1.4   k8s-node-4   <none>           <none>

~/nginx$ curl http://10.244.1.4/host.html
  # Hostname: nginx-deployment-7f5d9779c6-flmsf
  # Image Version: 2.2
  # Nginx Version: 1.19.4

Step 4.Pod 副本 & 收缩 & 端口映射

$ kubectl delete pod nginx-deployment-7f5d9779c6-flmsf
  # pod "nginx-deployment-7f5d9779c6-flmsf" deleted
$ kubectl get pod -o wide
  # NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment-7f5d9779c6-hhl7k   1/1     Running   0          60s   10.244.1.5   k8s-node-4   <none>           <none>

$ kubectl scale --replicas=3 deployment/nginx-deployment
  # deployment.apps/nginx-deployment scaled
$ kubectl get pod -o wide
  # NAME                                READY   STATUS    RESTARTS   AGE    IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment-7f5d9779c6-dr5h8   1/1     Running   0          4s     10.244.1.7   k8s-node-4   <none>           <none>
  # nginx-deployment-7f5d9779c6-hhl7k   1/1     Running   0          109s   10.244.1.5   k8s-node-4   <none>           <none>
  # nginx-deployment-7f5d9779c6-sk2f4   1/1     Running   0          4s     10.244.1.6   k8s-node-4   <none>           <none>

$ kubectl edit svc nginx-deployment  # 第一章中有介绍 SVC，不知道回去看一哈
$ kubectl expose -f nginx-controller.yaml --port=80 --target-port=8000 --protocol=TCP --type=NodePort --node-port=31855  # 后续验证
$ kubectl expose svc nginx-deployment --port=80 --target-port=8000 --protocol=TCP --type=NodePort
  # type: NodePort
$ kubectl get svc
  # NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
  # kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP           2d22h
  # nginx-deployment   NodePort    10.99.90.247   <none>        30000:31855/TCP   39h

$ sudo netstat -anpt | grep "31855"
  # tcp        0      0 0.0.0.0:31855           0.0.0.0:*               LISTEN      1931726/kube-proxy

Step 5.IPVS 负载均衡和rr轮询机制查看

$ sudo ipvsadm -Ln
  # IP Virtual Server version 1.2.1 (size=4096)
  # Prot LocalAddress:Port Scheduler Flags
  #   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  # TCP  10.10.107.202:31855 rr
  #   -> 10.244.1.5:80                Masq    1      0          1
  #   -> 10.244.1.6:80                Masq    1      1          1
  #   -> 10.244.1.7:80                Masq    1      0          1
  # TCP  10.96.0.1:443 rr
  #   -> 10.10.107.202:6443           Masq    1      3          0
  # TCP  10.96.0.10:53 rr
  #   -> 10.244.0.4:53                Masq    1      0          0
  #   -> 10.244.0.5:53                Masq    1      0          0
  # TCP  10.96.0.10:9153 rr
  #   -> 10.244.0.4:9153              Masq    1      0          0
  #   -> 10.244.0.5:9153              Masq    1      0          0
  # TCP  10.99.90.247:30000 rr
  #   -> 10.244.1.5:80                Masq    1      0          0
  #   -> 10.244.1.6:80                Masq    1      0          0
  #   -> 10.244.1.7:80                Masq    1      0          0
  # TCP  127.0.0.1:31855 rr
  #   -> 10.244.1.5:80                Masq    1      0          0
  #   -> 10.244.1.6:80                Masq    1      0          0
  #   -> 10.244.1.7:80                Masq    1      0          0

# 基于RR轮训负载
Hostname: nginx-deployment-7f5d9779c6-dr5h8 Image Version: 2.2 Nginx Version: 1.19.4 
Hostname: nginx-deployment-7f5d9779c6-sk2f4 Image Version: 2.2 Nginx Version: 1.19.4 
Hostname: nginx-deployment-7f5d9779c6-hhl7k Image Version: 2.2 Nginx Version: 1.19.4

欢迎各位志同道合的朋友一起学习交流，如文章有误请在下方留下您宝贵的经验知识，个人邮箱地址【master#weiyigeek.top】或者个人公众号【WeiyiGeek】联系我。

更多文章来源于【WeiyiGeek Blog - 为了能到远方，脚下的每一步都不能少】, 个人首页地址( https://weiyigeek.top )

专栏书写不易，如果您觉得这个专栏还不错的，请给这篇专栏 【点个赞、投个币、收个藏、关个注，转个发】(人间五大情)，这将对我的肯定，谢谢！。

3-Kubernetes入门之Ubuntu安装部署集群