[TOC]

0x00 前言简述

本章作者实践在 Docker 以及 kubernetes 环境中,快速部署生产环境中所使用的 Redis 内存数据库,帮助 devops工作者 以及 dev 开发者节省部署和开发时间。

如果你还不了解 Redis 数据库的朋友,可以参考我的【Redis学习之路】系列笔记帮助你快速入门Redis数据库, 关注 WeiyiGeek 公众号回复【Redis学习之路汇总】即可获得学习资料:

https://www.weiyigeek.top/wechat.html?key=Redis学习之路汇总

Docker 快速部署单实例 redis 数据库

温馨提示:此处实践环境是使用Docker,若你没有安装Docker环境或者不了解的Docker容器的朋友,可以参考博主学习【Docker的系列笔记】汇总, 关注 WeiyiGeek 公众号回复【Docker容器学习之路汇总】即可获得学习资料:
https://www.weiyigeek.top/wechat.html?key=Docker容器学习之路汇总

Shell 命令示例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
mkdir -vp /app/redis/data
tee /app/redis/redis.conf <<'EOF'
# Author: WeiyiGeek
# blog: https://blog.weiyigeek.top
# 绑定任意接口、服务端口、后台运行。
bind 0.0.0.0
port 6379
# 容器里必须设置为no
daemonize no
supervised auto

# redis服务pid进程文件名
pidfile "/var/run/redis.pid"

# 关闭保护模式,并配置使用密码访问
protected-mode no
requirepass "123456"
# echo -e "weiyigeek"|sha256sum
# requirepass 097575a79efcd7ea7b1efa2bcda78a4fc7cbd0820736b2f2708e72c3d21f8b61

# 数据文件保存路径,rdb/AOF文件也保存在这里
dir "/data"

# 日志文件记录文件(notice / verbose)
# logfile "/logs/redis.log"
# loglevel verbose

# 最大客户端连接数
maxclients 10000

# 客户端连接空闲多久后断开连接,单位秒,0表示禁用
timeout 60
tcp-keepalive 60

# Redis 数据持久化(rdb/aof)配置
# RDB 文件名
dbfilename "dump.rdb"
# 数据自动保存脚本条件例如300s中有10key发生变化
save 300 10
save 60 10000
# 对RDB文件进行压缩,建议以(磁盘)空间换(CPU)时间。
rdbcompression yes
# 版本5的RDB有一个CRC64算法的校验和放在了文件的最后。这将使文件格式更加可靠。
rdbchecksum yes
# RDB自动触发策略是否启用,默认为yes
rdb-save-incremental-fsync yes

# AOF开启
appendonly yes
# AOF文件名
appendfilename "appendonly.aof"
# 可选值 always, everysec,no,建议设置为everysec
appendfsync everysec

# Redis风险命令重命名
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
rename-command FLUSHDB b840fc02d524045429941cc15f59e41cb7be6c53
rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c54
rename-command EVAL b840fc02d524045429941cc15f59e41cb7be6c55
rename-command DEBUG b840fc02d524045429941cc15f59e41cb7be6c56
# rename-command SHUTDOWN SHUTDOWN
EOF

# docker 环境
$ docker run -d -p 6379:6379 \
-v /app/redis/redis.conf:/etc/redis/redis.conf \
-v /app/redis/data:/data \
--name redis-server redis:6.2.6-alpine3.15 redis-server /etc/redis/redis.conf

# nerdctl + containerd 环境
$ nerdctl run -d -p 6379:6379 \
-v /app/redis/redis.conf:/etc/redis/redis.conf \
-v /app/redis/data:/data \
--name redis-server redis:6.2.6-alpine3.15 redis-server /etc/redis/redis.conf
5e854a58087ae1bba5a661b2941474560cbecc37f54c7f4e7a28afbaed6aebf0

执行结果:

1
2
3
4
5
6
7
8
9
# 查看容器是否运行
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2919b8a9478a redis:6.2.6-alpine3.15 "docker-entrypoint.s…" 20 seconds ago Up 19 seconds 0.0.0.0:6379->6379/tcp redis-server

# 连接redis数据库是否正常
$ docker exec -it redis-server redis-cli -a 123456 ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG


Kubernetes 快速部署单实例 redis 数据库

温馨提示:此处实践环境是使用Kubernetes集群,若你没有安装Kubernetes集群环境或者不了解的Kubernetes容器的朋友,可以参考博主学习【Kubernetes的系列笔记】汇总, 关注 WeiyiGeek 公众号回复【Kubernetes学习之路汇总】即可获得学习资料
https://www.weiyigeek.top/wechat.html?key=Kubernetes学习之路汇总

温馨提示:集群中基于nfs的provisioner的动态持卷环境部署可参考此篇文章( https://blog.weiyigeek.top/2022/6-7-664.html )


步骤01.创建configMap 资源清单配置Redis.conf (此处还是采用上述 /app/redis/redis.conf 配置文件)

1
2
3
4
kubectl create configmap -n database redis-single-conf --from-file=/app/redis/redis.conf
kubectl get configmap -n database redis-single-conf
# NAME DATA AGE
# redis-single-conf 1 30s


步骤02.StatefulSet 与 SVC 资源清单:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
tee redis-single-server.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: __APPNAME__
namespace: __APPNAMESPACE__
annotations:
description: redis-single
spec:
replicas: 1
serviceName: __APPNAME__
selector:
matchLabels:
app: __APPNAME__
template:
metadata:
labels:
app: __APPNAME__
spec:
# 容器优化
initContainers:
- name: sysctl
image: alpine:3.15.4
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
mount -o remount rw /proc/sys
sysctl -w net.core.somaxconn=10000
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w fs.file-max=1048576
sysctl -w fs.inotify.max_user_instances=16384
sysctl -w fs.inotify.max_user_watches=524288
sysctl -w fs.inotify.max_queued_events=16384
echo never > /sys/kernel/mm/transparent_hugepage/enabled
securityContext:
privileged: true
containers:
- name: redis
image: redis:6.2.6-alpine3.15
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
name: server
command: [ "redis-server", "/conf/redis.conf"]
volumeMounts:
# 从configmap获取的配置文件,挂载到指定文件中【https://blog.weiyigeek.top】.
- name: conf
mountPath: /conf/redis.conf
subPath: redis.conf
- name: data
mountPath: /data
# - name: logs
# mountPath: /logs
# 时区设置
- name: timezone
mountPath: /etc/localtime
volumes:
- name: conf
# 配置文件采用configMap
configMap:
name: redis-single-conf
defaultMode: 0755
# 日志采用hostPath卷
- name: logs
hostPath:
type: DirectoryOrCreate
path: /app/redis/logs
# 时区定义
- name: timezone
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "__STORAGECLASSNAME__"
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
name: __APPNAME__
namespace: __APPNAMESPACE__
spec:
type: ClusterIP
ports:
- port: 6379
targetPort: 6379
name: tcp
selector:
app: __APPNAME__
EOF

# 注意 替换 storageClassName
sed -i -e 's/__APPNAME__/redis-single/g' -e 's/__APPNAMESPACE__/database/g' -e 's/__STORAGECLASSNAME__/managed-nfs-storage/' redis-single-server.yaml


步骤 03.部署资源清单到Kubernetes集群

1
2
3
kubectl apply -f redis-single-server.yaml
# statefulset.apps/redis-single created
# service/redis-single created


步骤 04.验证部署应用资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
kubectl get pod -n database -l app=redis-single
# NAME READY STATUS RESTARTS AGE
# redis-single-0 1/1 Running 0 16m

kubectl get svc -n database redis-single
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# redis-single ClusterIP 11.19.196.169 <none> 6379/TCP 15m

# 连接到redis数据库
$ telnet 11.19.196.169 6379
# Trying 11.19.196.169...
# Connected to 11.19.196.169.
# Escape character is '^]'.
# auth 123456
# +OK
# ping
# +PONG
# info
# $4241
# # Server
# redis_version:6.2.6
# redis_git_sha1:00000000
# redis_git_dirty:0
# redis_build_id:63421500bb103677
# redis_mode:standalone
# os:Linux 4.20.13-1.el7.elrepo.x86_64 x86_64
# arch_bits:64
# multiplexing_api:epoll
# atomicvar_api:atomic-builtin
# gcc_version:10.3.1
# process_id:1
# process_supervised:no
# run_id:677b6d0188b564670f26ce47154b70c8aa6c3f04
# tcp_port:6379
# server_time_usec:1650962303340701

温馨提示: 集群中的其它应用,我们可以通过svc的地址或者集群dns域名(redis-single.database-推荐)来访问该pod。


Kubernetes 快速利用资源清单部署 redis 数据库集群

步骤 01.创建 redis 配置文件使用 configmap 方式进行挂载。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
tee redis-cluster-configmap.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster
namespace: database
data:
update-node.sh: |
#!/bin/sh
CLUSTER_CONFIG="/data/nodes.conf"
if [ -f ${CLUSTER_CONFIG} ]; then
if [ -z "${POD_IP}" ]; then
echo "Unable to determine Pod IP address!"
exit 1
fi
echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
fi
exec "$@"
redis.conf: |+
port 6379
protected-mode no
maxclients 32768
masterauth weiyigeek.top
requirepass weiyigeek.top
# 数据目录
dir /data
# 开启 RDB 持久化&触发策略
dbfilename dump.rdb
rdb-save-incremental-fsync yes
# 开启AOF持久化以及每秒钟同步一次折中的方案
appendonly yes
appendfilename appendonly.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 128mb
aof-load-truncated yes
aof-use-rdb-preamble yes
aof-rewrite-incremental-fsync yes
# 集群模式打开
cluster-enabled yes
cluster-config-file /data/nodes.conf
cluster-node-timeout 5000
# 当负责一个插槽的主库下线且没有相应的从库进行故障恢复时集群仍然可用
cluster-require-full-coverage no
# 只有当一个主节点至少拥有其他给定数量个处于正常工作中的从节点的时候,才会分配从节点给集群中孤立的主节点
cluster-migration-barrier 1
# 副本服务陈旧数据
replica-serve-stale-data yes
replica-read-only no
replica-priority 100
# 复制同步策略
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
# 慢查询日志
slowlog-log-slower-than 10000
slowlog-max-len 128
# 内存策略
maxmemory-policy allkeys-lru
EOF

温馨提示:masterauth 与 requirepass 认证密码需要根据实际情况进行更改,此外我设置为了我的主页地址。


步骤 02.Redis 集群 SVC 以及 STS 资源清单。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
tee redis-cluster-deploy.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
namespace: database
name: redis-cluster
labels:
app: redis-cluster
spec:
clusterIP: None
ports:
- port: 6379
targetPort: 6379
name: client
- port: 16379
targetPort: 16379
name: gossip
selector:
app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: database
name: redis-cluster
labels:
app: redis-cluster
spec:
serviceName: redis-cluster
replicas: 6
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
spec:
affinity:
nodeAffinity: # node亲和性
requiredDuringSchedulingIgnoredDuringExecution: # 硬策略,调度在指定标签的节点中
nodeSelectorTerms:
- matchExpressions:
- key: node
operator: In
values:
- app
podAntiAffinity: # Pod反亲和性
preferredDuringSchedulingIgnoredDuringExecution: # 软策略,使Pod分布在不同的节点上
- weight: 1 # 权重,有多个策略通过权重控制调度
podAffinityTerm:
topologyKey: app.kubernetes.io/name # 通过app.kubernetes.io/name作为域调度
labelSelector:
matchExpressions:
- key: node
operator: In
values:
- app
containers:
- name: redis
image: redis:6.2.7-alpine
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
command: ["/etc/redis/update-node.sh", "redis-server", "/etc/redis/redis.conf"]
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: conf
mountPath: /etc/redis/
readOnly: false
- name: data
mountPath: /data
readOnly: false
- name: timezone
mountPath: /etc/localtime # 在Pod中时区设置(挂载主机的时区)
volumes:
- name: conf
configMap:
name: redis-cluster
defaultMode: 0755
- name: timezone
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: nfs-dev
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
EOF


步骤 03.使用 kubectl apply 部署资源清单,并查看部署结果以及Pod IP,为集群初始化做准备。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# (1) 名称空间创建
$ kubectl create namespace database

# (2) 按照资源清单StatefulSet控制器创建Pod
$ ls
redis-cluster-configmap.yaml redis-cluster-deploy.yaml
$ kubectl create -f redis-cluster-configmap.yaml
$ kubectl create -f redis-cluster-deploy.yaml

# (3) 查看应用的 statefulsets,pod,svc 等信息
$ kubectl get statefulsets,pod,svc -n database
NAME READY AGE
statefulset.apps/redis-cluster 6/6 13m

NAME READY STATUS RESTARTS AGE
pod/redis-cluster-0 1/1 Running 0 13m
pod/redis-cluster-1 1/1 Running 0 12m
pod/redis-cluster-2 1/1 Running 0 12m
pod/redis-cluster-3 1/1 Running 0 10m
pod/redis-cluster-4 1/1 Running 0 9m19s
pod/redis-cluster-5 1/1 Running 0 9m16s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis-cluster ClusterIP None <none> 6379/TCP,16379/TCP 13m


# (4) 获取 Redis 集群 6 个节点 Pod 的 ip 地址
kubectl get pod -n database -l app=redis-cluster -o jsonpath='{ range.items [*]}{.status.podIP}:6379 '| sed "s# :6379 ##g"
# 10.66.35.66:6379 10.66.237.142:6379 10.66.55.76:6379 10.66.53.96:6379 10.66.35.92:6379 10.66.55.77:6379


步骤 04.进入到第一个Pod 终端中进行初始化 Redis 集群操作,验证Redis Cluster集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
# (1) 进入redis-cluster-0 pod中的shell
kubectl exec -n database -it redis-cluster-0 sh

# (2) 进行集群初始化,运行以下命令并输入'yes'接受配置。我们将看到前三个节点将被选择为主节点,后三个节点将被选择为从节点。
/data $ redis-cli -a weiyigeek.top --cluster create --cluster-replicas 1 10.66.35.66:6379 10.66.237.142:6379 10.66.55.76:6379 10.66.53.96:6379 10.66.35.92:6379 10.66.55.77:6379
# >>> Performing hash slots allocation on 6 nodes...
# Master[0] -> Slots 0 - 5460
# Master[1] -> Slots 5461 - 10922
# Master[2] -> Slots 10923 - 16383
# Adding replica 10.66.35.92:6379 to 10.66.35.66:6379
# Adding replica 10.66.55.77:6379 to 10.66.237.142:6379
# Adding replica 10.66.53.96:6379 to 10.66.55.76:6379
# M: b13a58f799e058dd9afba221f25a8bd53ae05969 10.66.35.66:6379
# slots:[0-5460] (5461 slots) master
# M: 729af3b961a978ad15263ef96439177207be7821 10.66.237.142:6379
# slots:[5461-10922] (5462 slots) master
# M: 50c3d31e7277ce339752a980f14c03d180e2f98a 10.66.55.76:6379
# slots:[10923-16383] (5461 slots) master
# S: ce02a1eb7efbd4de37383f46c300bb883dcd16b4 10.66.53.96:6379
# replicates 50c3d31e7277ce339752a980f14c03d180e2f98a
# S: 298d5e8528c38f687c9894b9974eb5368555c652 10.66.35.92:6379
# replicates b13a58f799e058dd9afba221f25a8bd53ae05969
# S: f7ecf69d99129fba7737f8ae9a8f172af19a7b72 10.66.55.77:6379
# replicates 729af3b961a978ad15263ef96439177207be7821
# Can I set the above configuration? (type 'yes' to accept): yes # 此处输入 yes 进行上述卡槽设置。
# >>> Nodes configuration updated
# >>> Assign a different config epoch to each node
# >>> Sending CLUSTER MEET messages to join the cluster
# Waiting for the cluster to join
# .
# >>> Performing Cluster Check (using node 10.66.35.66:6379)
# M: b13a58f799e058dd9afba221f25a8bd53ae05969 10.66.35.66:6379
# slots:[0-5460] (5461 slots) master
# 1 additional replica(s)
# S: f7ecf69d99129fba7737f8ae9a8f172af19a7b72 10.66.55.77:6379
# slots: (0 slots) slave
# replicates 729af3b961a978ad15263ef96439177207be7821
# M: 50c3d31e7277ce339752a980f14c03d180e2f98a 10.66.55.76:6379
# slots:[10923-16383] (5461 slots) master
# 1 additional replica(s)
# S: ce02a1eb7efbd4de37383f46c300bb883dcd16b4 10.66.53.96:6379
# slots: (0 slots) slave
# replicates 50c3d31e7277ce339752a980f14c03d180e2f98a
# M: 729af3b961a978ad15263ef96439177207be7821 10.66.237.142:6379
# slots:[5461-10922] (5462 slots) master
# 1 additional replica(s)
# S: 298d5e8528c38f687c9894b9974eb5368555c652 10.66.35.92:6379
# slots: (0 slots) slave
# replicates b13a58f799e058dd9afba221f25a8bd53ae05969
# [OK] All nodes agree about slots configuration.
# >>> Check for open slots...
# >>> Check slots coverage...
# [OK] All 16384 slots covered.

# (3) 集群验证
# 此处我使用集群svc服务名称进行访问 redis-cluster.database.svc.cluster.test。
/data $ redis-cli -c -h redis-cluster.database.svc.cluster.test -a weiyigeek.top

# 查看集群信息
redis-cluster.database.svc.cluster.test:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
....

# 查看集群节点
redis-cluster.database.svc.cluster.test:6379> CLUSTER NODES
f7ecf69d99129fba7737f8ae9a8f172af19a7b72 10.66.55.77:6379@16379 slave 729af3b961a978ad15263ef96439177207be7821 0 1665234666115 2 connected
b13a58f799e058dd9afba221f25a8bd53ae05969 10.66.35.66:6379@16379 master - 0 1665234667119 1 connected 0-5460
50c3d31e7277ce339752a980f14c03d180e2f98a 10.66.55.76:6379@16379 master - 0 1665234667119 3 connected 10923-16383
298d5e8528c38f687c9894b9974eb5368555c652 10.66.35.92:6379@16379 slave b13a58f799e058dd9afba221f25a8bd53ae05969 0 1665234665513 1 connected
729af3b961a978ad15263ef96439177207be7821 10.66.237.142:6379@16379 myself,master - 0 1665234665000 2 connected 5461-10922
ce02a1eb7efbd4de37383f46c300bb883dcd16b4 10.66.53.96:6379@16379 slave 50c3d31e7277ce339752a980f14c03d180e2f98a 0 1665234665110 3 connected

温馨提示:在初始化 redis 集群时必须使用 ip 地址,如果使用域名会报如下错误。
温馨提示:当应用连接 redis 集群时使用 pod 的域名, 格式为 svc名字.ns名字.svc.cluster.local

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ nslookup redis-cluster.database.svc.cluster.test
Server: 10.96.0.10
Address: 10.96.0.10:53

Name: redis-cluster.database.svc.cluster.test
Address: 10.66.35.66
Name: redis-cluster.database.svc.cluster.test
Address: 10.66.35.92
Name: redis-cluster.database.svc.cluster.test
Address: 10.66.55.76
Name: redis-cluster.database.svc.cluster.test
Address: 10.66.53.96
Name: redis-cluster.database.svc.cluster.test
Address: 10.66.237.142
Name: redis-cluster.database.svc.cluster.test
Address: 10.66.55.77


步骤 05.使用redis cluster进行KV数据插入并查看数据。

1
2
3
4
5
6
7
8
9
redis-cluster.database.svc.cluster.test:6379> set domain weiyigeek.top EX 10
-> Redirected to slot [5449] located at 10.66.35.66:6379
OK
10.66.35.66:6379> get domain
"weiyigeek.top"
10.66.35.66:6379> get domain
"weiyigeek.top"
10.66.35.66:6379> get domain # 10秒后到期,则key被销毁
(nil)


温馨提示:如果整个 redis 集群的 pod 全部都挂掉了,pod自动拉起后集群不可用,需要重建集群,此时有两种重建集群方法。

  • 方式1.重新开始(不到万不得已不建议这样做)

    1.删除 redis 集群所有的资源,然后重新创建 redis 集群
    2.删除 redis 集群中所有的 pvc(pv)
    3.删除 redis 集群中 pod 对应的 nfs 持久化存储目录
    4.重新创建 redis 集群资源并进行初始化

  • 方式2.在原有 redis 集群的基础上进行修复。

    1.将 redis 集群中资源控制器的副本数设置为 0
    2.找到 redis 集群中 pod 对应的 nfs 持久化存储目录后删除 nodes.conf

    1
    2
    3
    4
    5
    6
    7
    /storage/dev/pvc/local# find . -name nodes.conf
    ./database-data-redis-cluster-1-pvc-febdd490-6dbf-4084-860f-1d81602f4ca0/nodes.conf
    ./database-data-redis-cluster-3-pvc-08d6d4b4-5290-4710-b31e-efb5e8d8481e/nodes.conf
    ./database-data-redis-cluster-5-pvc-96db8931-2106-4ac1-9c41-ec8888e99024/nodes.conf
    ./database-data-redis-cluster-4-pvc-c67c4649-ff0f-4767-ae75-3875653886c5/nodes.conf
    ./database-data-redis-cluster-0-pvc-e6b1f1bd-75b0-4777-ab93-993a2e6e5927/nodes.conf
    ./database-data-redis-cluster-2-pvc-d078a7d3-cf0c-4b1e-acbe-59086089fb0d/nodes.conf

    3.重新创建 redis 集群资源 kubectl apply -f redis-cluster-deploy.yaml 导出数据后,然后重新初始化集群。


Kubernetes 中使用 Helm 快速部署 redis 多主多从集群

描述:我们可以使用 Bitnami helm chart 在 K8S 中一键部署 Redis cluster 多主多从。
项目地址:https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster

WeiyiGeek.redis单主与redis多主集群

WeiyiGeek.redis单主与redis多主集群

环境依赖:

  • Kubernetes 1.19+
  • Helm 3.2.0+
  • PV provisioner support in the underlying infrastructure

温馨提示:1.集群中基于nfs的provisioner的动态持卷环境部署可参考此篇文章( https://blog.weiyigeek.top/2022/6-7-664.html )


步骤 01.添加 Bitnami chart 仓库,查看可用 redis-cluster 的 chart 及其对应版本,拉取部署所需的 yaml 文件到本地。

1
2
3
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm3 search repo bitnami/redis-cluster -l
$ helm3 pull bitnami/redis-cluster --version 8.2.4 --untar


步骤 02.修改 redis-cluster 图表的 values.yaml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
vim redis-cluster/values.yaml
# 1.修改 clusterDomain Kubernetes Cluster Domain 此处由于我们的集群名称非 cluster.local 所以修改如下
clusterDomain: cluster.test
.....
# 2.集群所需redis镜像,此处我已经上传内部harbor中,则需要根据实际请个改变
image:
registry: harbor.weiyigeek.top
repository: library/redis-cluster
tag: 7.0.5-debian-11-r0
.....
# 3.设置 Pod 安全上下文
podSecurityContext:
enabled: true
fsGroup: 1001
runAsUser: 1001
sysctls: []
.....
# 4.此处svc采用ClusterIP如果采用NodePort需要更改暴露端口 30000 - 32000
service:
ports:
redis: 6379
nodePorts:
redis: 31379
type: ClusterIP
.....
# 5.数据 persistence 持久化配置,此处我采用了NFS动态逻辑卷。
persistence:
path: /bitnami/redis/data
storageClass: "nfs-local"
accessModes:
- ReadWriteOnce
size: 8Gi
.....
# 6.设置 Redis statefulset parameters 资源限额
redis:
resources:
limits:
cpu: 1000m
memory: 4Gi
requests: {}
.....
# 7.Redis Cluster 设置,此处我设置3主3从,一共6个节点
# 如果 nodes 为 6 replicas 为 1 表示每个master节点有一个副本
# nodes = numberOfMasterNodes + numberOfMasterNodes * replicas
cluster:
init: true
nodes: 6
replicas: 1
.....
# 8.启用 Redis Prometheus Exporter Metrics
metrics:
enabled: true
image:
registry: harbor.weiyigeek.top
repository: library/redis-exporter
tag: 1.44.0-debian-11-r8
.....
# 9.启用并使用sysctlImage配置值设置特权initContainer
sysctlImage:
enabled: true
registry: harbor.weiyigeek.top
repository: library/bitnami-shell
tag: 11-debian-11-r37
command:
- /bin/sh
- -c
- |-
sysctl -w net.core.somaxconn=10000
echo never > /host-sys/kernel/mm/transparent_hugepage/enabled
mountHostSys: true


步骤 03.为了加快拉取速度我将docker hub上的指定所需镜像拉取到本地Harbor仓库中。

1
2
3
4
5
6
7
8
9
10
11
# redis-cluster:7.0.5-debian-11-r0 
docker pull bitnami/redis-cluster:7.0.5-debian-11-r0
docker tag bitnami/redis-cluster:7.0.5-debian-11-r0 harbor.weiyigeek.top/library/redis-cluster:7.0.5-debian-11-r0
docker push harbor.weiyigeek.top/library/redis-cluster:7.0.5-debian-11-r0

# redis-exporter & bitnami-shell
docker pull bitnami/redis-exporter:1.44.0-debian-11-r8; docker pull bitnami/bitnami-shell:11-debian-11-r37
docker tag bitnami/redis-exporter:1.44.0-debian-11-r8 harbor.weiyigeek.top/library/redis-exporter:1.44.0-debian-11-r8
docker tag bitnami/bitnami-shell:11-debian-11-r37 harbor.weiyigeek.top/library/bitnami-shell:11-debian-11-r37
docker push harbor.weiyigeek.top/library/redis-exporter:1.44.0-debian-11-r8
docker push harbor.weiyigeek.top/library/bitnami-shell:11-debian-11-r37


步骤 04.开始安装修改后的 redis-cluster 图表到 database 名称空间中,并查看资源部署情况。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
helm3 install redis ./redis-cluster --set password=weiyigeek.top  --namespace database --create-namespace --debug

helm3 list -n database
# NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
# redis database 1 2022-10-09 17:35:18.530202622 +0800 CST deployed redis-cluster-8.2.4 7.0.5

kubectl get sts,pod,svc -n database -l app.kubernetes.io/name=redis-cluster
# NAME READY AGE
# statefulset.apps/redis-redis-cluster 6/6 4m4s

# NAME READY STATUS RESTARTS AGE
# pod/redis-redis-cluster-0 2/2 Running 1 (3m3s ago) 4m4s
# pod/redis-redis-cluster-1 2/2 Running 1 (3m3s ago) 4m4s
# pod/redis-redis-cluster-2 2/2 Running 2 (3m45s ago) 4m4s
# pod/redis-redis-cluster-3 2/2 Running 1 (3m ago) 4m4s
# pod/redis-redis-cluster-4 2/2 Running 1 (3m3s ago) 4m4s
# pod/redis-redis-cluster-5 2/2 Running 1 (3m ago) 4m4s

# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/redis-redis-cluster ClusterIP 10.105.252.15 <none> 6379/TCP 4m4s
# service/redis-redis-cluster-headless ClusterIP None <none> 6379/TCP,16379/TCP 4m5s
# service/redis-redis-cluster-metrics ClusterIP 10.96.154.84 <none> 9121/TCP 4m4s


步骤 05.验证使用helm部署的Redis集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 1.查询设置或者机生成的 Redis 密码
export REDIS_PASSWORD=$(kubectl get secret --namespace "database" redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

# 2.创建运行Redis客户端的pod:
kubectl run --namespace database redis-redis-cluster-client --rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image bitnami/redis-cluster:7.0.5-debian-11-r0 -- bash

# 3.在Pod中使用shell连接到集群
redis-cli -c -h redis-redis-cluster -a $REDIS_PASSWORD
# 集群信息
redis-redis-cluster:6379> CLUSTER INFO
# cluster_state:ok
# 节点信息
redis-redis-cluster:6379> CLUSTER NODES
# 数据插入测试
redis-redis-cluster:6379> set name weiyigeek
OK
10.66.237.149:6379> set domain weiyigeek.top
OK
10.66.35.100:6379> set blog blog.weiyigeek.top
OK
10.66.237.149:6379> keys *
1) "name"
2) "blog"
10.66.237.149:6379> get name
"weiyigeek"
WeiyiGeek.执行结果

WeiyiGeek.执行结果

至此,快速使用helm安装多主多从redis集群完毕!


Kubernetes 快速部署 RedisInsight 工具连接Redis集群

描述:RedisInsight 是 Redis 官方推荐的可视化工具,其功能强大,支持Redis集群连接。

参考地址:https://docs.redis.com/latest/ri/installing/install-k8s/
参考文章: 【9.使用RedisInsight工具对Redis集群CURD操作及数据可视化和性能监控】- https://blog.weiyigeek.top/2022/9-20-686.html


步骤 01.集群中快速 redisinsight 部署资源清单。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
tee redisinsight-1.13.0.yaml <<'EOF'
# RedisInsight service with name 'redisinsight-service'
apiVersion: v1
kind: Service
metadata:
name: redisinsight-service
namespace: devops
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8001
selector:
app: redisinsight
---
# RedisInsight deployment with name 'redisinsight'
apiVersion: apps/v1
kind: Deployment
metadata:
name: redisinsight # deployment name
namespace: devops
labels:
app: redisinsight # deployment label
spec:
replicas: 1 # a single replica pod
strategy:
type: Recreate
selector:
matchLabels:
app: redisinsight # which pods is the deployment managing, as defined by the pod template
template: # pod template
metadata:
labels:
app: redisinsight # label for pod/s
spec:
volumes:
- name: db
emptyDir: {} # node-ephemeral volume https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
containers:
- name: redisinsight # Container name (DNS_LABEL, unique)
image: redislabs/redisinsight:1.13.0 # Hub Image
imagePullPolicy: IfNotPresent # Pull Policy
env:
- name: RIHOST
value: "0.0.0.0"
- name: RIPORT
value: "8001"
volumeMounts:
- name: db
mountPath: /db
ports:
- containerPort: 8001 # exposed container port and protocol
protocol: TCP
livenessProbe:
httpGet:
path : /healthcheck/ # exposed RI endpoint for healthcheck
port: 8001 # exposed container port
initialDelaySeconds: 5 # number of seconds to wait after the container starts to perform liveness probe
periodSeconds: 5 # period in seconds after which liveness probe is performed
failureThreshold: 1 # number of liveness probe failures after which container restarts
EOF

步骤 02.部署命令

1
2
kubectl create ns devops
kubectl apply -f redisinsight-1.13.0.yaml

步骤 03.验证服务(你可按照需求使用nodeport或者ingress进行暴露端口)

1
2
3
4
5
6
7
8
9
$ kubectl get pod -n devops
NAME READY STATUS RESTARTS AGE
redisinsight-5c4f954694-jc5cd 1/1 Running 0 95s

$ kubectl -n devops port-forward deployment/redisinsight --address 0.0.0.0 30081:8001
Forwarding from 0.0.0.0:30081 -> 8001
Handling connection for 30081
Handling connection for 30081
Handling connection for 30081

步骤 04.浏览器访问 节点IP:30081 即可访问 redisinsight UI 界面,添加集群IP(服务名称)以及链接端口和认证密码。

Host: redis-redis-cluster.database.svc
Port: 6379
Name: redis-cluster
Password: weiyigeek.top

WeiyiGeek.redisinsight链接redis集群图

WeiyiGeek.redisinsight链接redis集群图

步骤 05.添加完成后我们便可使用redisinsight进行管理查看操作Redis集群了。

WeiyiGeek.使用redisinsight操作redis集群图

WeiyiGeek.使用redisinsight操作redis集群图

至此完毕。