[TOC]

0x00 前言简述

本章作者实践在 Docker 以及 kubernetes 环境中，快速部署生产环境中所使用的 Redis 内存数据库，帮助 devops工作者 以及 dev 开发者节省部署和开发时间。

如果你还不了解 Redis 数据库的朋友，可以参考我的【Redis学习之路】系列笔记帮助你快速入门Redis数据库, 关注 WeiyiGeek 公众号回复【Redis学习之路汇总】即可获得学习资料：

https://www.weiyigeek.top/wechat.html?key=Redis学习之路汇总

Docker 快速部署单实例 redis 数据库

温馨提示：此处实践环境是使用Docker,若你没有安装Docker环境或者不了解的Docker容器的朋友，可以参考博主学习【Docker的系列笔记】汇总, 关注 WeiyiGeek 公众号回复【Docker容器学习之路汇总】即可获得学习资料:
https://www.weiyigeek.top/wechat.html?key=Docker容器学习之路汇总

Shell 命令示例:

mkdir -vp /app/redis/data
tee /app/redis/redis.conf <<'EOF'
# Author: WeiyiGeek
# blog: https://blog.weiyigeek.top
# 绑定任意接口、服务端口、后台运行。
bind 0.0.0.0
port 6379
# 容器里必须设置为no
daemonize no
supervised auto

# redis服务pid进程文件名
pidfile "/var/run/redis.pid"

# 关闭保护模式，并配置使用密码访问
protected-mode no
requirepass "123456"
# echo -e "weiyigeek"|sha256sum 
# requirepass 097575a79efcd7ea7b1efa2bcda78a4fc7cbd0820736b2f2708e72c3d21f8b61

# 数据文件保存路径，rdb/AOF文件也保存在这里
dir "/data"

# 日志文件记录文件(notice / verbose)
# logfile "/logs/redis.log"
# loglevel verbose  

# 最大客户端连接数
maxclients 10000

# 客户端连接空闲多久后断开连接，单位秒，0表示禁用
timeout 60
tcp-keepalive 60 

# Redis 数据持久化(rdb/aof)配置
# RDB 文件名
dbfilename "dump.rdb"
# 数据自动保存脚本条件例如300s中有10key发生变化
save 300 10
save 60 10000
# 对RDB文件进行压缩，建议以（磁盘）空间换（CPU）时间。
rdbcompression yes
# 版本5的RDB有一个CRC64算法的校验和放在了文件的最后。这将使文件格式更加可靠。
rdbchecksum yes
# RDB自动触发策略是否启用，默认为yes
rdb-save-incremental-fsync yes

# AOF开启
appendonly yes
# AOF文件名
appendfilename "appendonly.aof"
# 可选值 always， everysec，no，建议设置为everysec
appendfsync everysec


# Redis风险命令重命名
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
rename-command FLUSHDB b840fc02d524045429941cc15f59e41cb7be6c53
rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c54
rename-command EVAL b840fc02d524045429941cc15f59e41cb7be6c55
rename-command DEBUG b840fc02d524045429941cc15f59e41cb7be6c56
# rename-command SHUTDOWN SHUTDOWN
EOF

# docker 环境
$ docker run -d -p 6379:6379 \
 -v /app/redis/redis.conf:/etc/redis/redis.conf \
 -v /app/redis/data:/data \
  --name redis-server redis:6.2.6-alpine3.15 redis-server /etc/redis/redis.conf

# nerdctl + containerd 环境
$ nerdctl run -d -p 6379:6379 \
 -v /app/redis/redis.conf:/etc/redis/redis.conf \
 -v /app/redis/data:/data \
 --name redis-server redis:6.2.6-alpine3.15 redis-server /etc/redis/redis.conf
5e854a58087ae1bba5a661b2941474560cbecc37f54c7f4e7a28afbaed6aebf0

执行结果:

# 查看容器是否运行
$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                    NAMES
2919b8a9478a   redis:6.2.6-alpine3.15         "docker-entrypoint.s…"   20 seconds ago   Up 19 seconds   0.0.0.0:6379->6379/tcp   redis-server

# 连接redis数据库是否正常
$ docker exec -it redis-server redis-cli -a 123456 ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG

---

Kubernetes 快速部署单实例 redis 数据库

温馨提示：此处实践环境是使用Kubernetes集群,若你没有安装Kubernetes集群环境或者不了解的Kubernetes容器的朋友，可以参考博主学习【Kubernetes的系列笔记】汇总, 关注 WeiyiGeek 公众号回复【Kubernetes学习之路汇总】即可获得学习资料
https://www.weiyigeek.top/wechat.html?key=Kubernetes学习之路汇总

温馨提示：集群中基于nfs的provisioner的动态持卷环境部署可参考此篇文章( https://blog.weiyigeek.top/2022/6-7-664.html )

步骤01.创建configMap 资源清单配置Redis.conf (此处还是采用上述 /app/redis/redis.conf 配置文件)

kubectl create configmap -n database redis-single-conf --from-file=/app/redis/redis.conf
kubectl get configmap -n database redis-single-conf
# NAME                DATA   AGE
# redis-single-conf   1      30s

步骤02.StatefulSet 与 SVC 资源清单:

tee redis-single-server.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: __APPNAME__
  namespace: __APPNAMESPACE__
  annotations:
    description: redis-single
spec:
  replicas: 1
  serviceName: __APPNAME__
  selector:
    matchLabels:
      app: __APPNAME__
  template:
    metadata:
      labels:
        app: __APPNAME__
    spec:
      # 容器优化
      initContainers:
      - name: sysctl
        image: alpine:3.15.4
        imagePullPolicy: IfNotPresent
        command:
        - sh
        - -c
        - |
          mount -o remount rw /proc/sys
          sysctl -w net.core.somaxconn=10000
          sysctl -w net.ipv4.tcp_tw_reuse=1
          sysctl -w net.ipv4.ip_local_port_range="1024 65535"
          sysctl -w fs.file-max=1048576
          sysctl -w fs.inotify.max_user_instances=16384
          sysctl -w fs.inotify.max_user_watches=524288
          sysctl -w fs.inotify.max_queued_events=16384
          echo never > /sys/kernel/mm/transparent_hugepage/enabled
        securityContext:
          privileged: true
      containers:
      - name: redis
        image: redis:6.2.6-alpine3.15
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
          name: server
        command: [ "redis-server", "/conf/redis.conf"]
        volumeMounts:
        # 从configmap获取的配置文件，挂载到指定文件中【https://blog.weiyigeek.top】.
        - name: conf
          mountPath: /conf/redis.conf
          subPath: redis.conf
        - name: data
          mountPath: /data
        # - name: logs
        #   mountPath: /logs
        # 时区设置
        - name: timezone
          mountPath: /etc/localtime              
      volumes:
      - name: conf
        # 配置文件采用configMap
        configMap:
          name: redis-single-conf
          defaultMode: 0755
        # 日志采用hostPath卷
      - name: logs
        hostPath:
          type: DirectoryOrCreate 
          path: /app/redis/logs
        # 时区定义
      - name: timezone                             
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "__STORAGECLASSNAME__"
      resources:
        requests:
          storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: __APPNAME__
  namespace: __APPNAMESPACE__
spec:
  type: ClusterIP
  ports:
  - port: 6379
    targetPort: 6379
    name: tcp
  selector:
    app: __APPNAME__
EOF

# 注意 替换 storageClassName
sed -i -e 's/__APPNAME__/redis-single/g' -e 's/__APPNAMESPACE__/database/g' -e 's/__STORAGECLASSNAME__/managed-nfs-storage/' redis-single-server.yaml

步骤 03.部署资源清单到Kubernetes集群

kubectl apply -f redis-single-server.yaml
# statefulset.apps/redis-single created
# service/redis-single created

步骤 04.验证部署应用资源

kubectl get pod -n database -l app=redis-single
  # NAME             READY   STATUS    RESTARTS   AGE
  # redis-single-0   1/1     Running   0          16m

kubectl get svc -n database redis-single
  # NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
  # redis-single   ClusterIP   11.19.196.169   <none>        6379/TCP   15m

# 连接到redis数据库
$ telnet 11.19.196.169 6379
  # Trying 11.19.196.169...
  # Connected to 11.19.196.169.
  # Escape character is '^]'.
  # auth 123456
  # +OK
  # ping
  # +PONG
  # info
  # $4241
  # # Server
  # redis_version:6.2.6
  # redis_git_sha1:00000000
  # redis_git_dirty:0
  # redis_build_id:63421500bb103677
  # redis_mode:standalone
  # os:Linux 4.20.13-1.el7.elrepo.x86_64 x86_64
  # arch_bits:64
  # multiplexing_api:epoll
  # atomicvar_api:atomic-builtin
  # gcc_version:10.3.1
  # process_id:1
  # process_supervised:no
  # run_id:677b6d0188b564670f26ce47154b70c8aa6c3f04
  # tcp_port:6379
  # server_time_usec:1650962303340701

温馨提示: 集群中的其它应用，我们可以通过svc的地址或者集群dns域名(redis-single.database-推荐)来访问该pod。

Kubernetes 快速利用资源清单部署 redis 数据库集群

步骤 01.创建 redis 配置文件使用 configmap 方式进行挂载。

tee redis-cluster-configmap.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
  namespace: database
data:
  update-node.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |+
    port 6379
    protected-mode no
    maxclients 32768
    masterauth weiyigeek.top
    requirepass weiyigeek.top
    # 数据目录
    dir /data
    # 开启 RDB 持久化&触发策略
    dbfilename dump.rdb
    rdb-save-incremental-fsync yes
    # 开启AOF持久化以及每秒钟同步一次折中的方案
    appendonly yes 
    appendfilename appendonly.aof 
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 128mb
    aof-load-truncated yes
    aof-use-rdb-preamble yes
    aof-rewrite-incremental-fsync yes
    # 集群模式打开
    cluster-enabled yes 
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 5000
    # 当负责一个插槽的主库下线且没有相应的从库进行故障恢复时集群仍然可用
    cluster-require-full-coverage no
    # 只有当一个主节点至少拥有其他给定数量个处于正常工作中的从节点的时候，才会分配从节点给集群中孤立的主节点
    cluster-migration-barrier 1
    # 副本服务陈旧数据
    replica-serve-stale-data yes
    replica-read-only no
    replica-priority 100
    # 复制同步策略
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    # 慢查询日志
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    # 内存策略
    maxmemory-policy allkeys-lru
EOF

温馨提示：masterauth 与 requirepass 认证密码需要根据实际情况进行更改，此外我设置为了我的主页地址。

步骤 02.Redis 集群 SVC 以及 STS 资源清单。

tee redis-cluster-deploy.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  namespace: database
  name: redis-cluster
  labels:
    app: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: database
  name: redis-cluster
  labels:
    app: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      affinity:
        nodeAffinity:                                      # node亲和性
          requiredDuringSchedulingIgnoredDuringExecution:  # 硬策略,调度在指定标签的节点中
            nodeSelectorTerms:
            - matchExpressions:
              - key: node
                operator: In
                values:
                  - app
        podAntiAffinity:                                    # Pod反亲和性
          preferredDuringSchedulingIgnoredDuringExecution:  # 软策略,使Pod分布在不同的节点上
          - weight: 1                                       # 权重,有多个策略通过权重控制调度
            podAffinityTerm:
              topologyKey: app.kubernetes.io/name           # 通过app.kubernetes.io/name作为域调度  
              labelSelector:
                matchExpressions:
                - key: node
                  operator: In
                  values:
                  - app
      containers:
      - name: redis
        image: redis:6.2.7-alpine
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/update-node.sh", "redis-server", "/etc/redis/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
        - name: timezone
          mountPath: /etc/localtime                           # 在Pod中时区设置(挂载主机的时区)
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
      - name: timezone 
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: nfs-dev
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
EOF

步骤 03.使用 kubectl apply 部署资源清单，并查看部署结果以及Pod IP，为集群初始化做准备。

# (1) 名称空间创建
$ kubectl create namespace database

# (2) 按照资源清单StatefulSet控制器创建Pod
$ ls
redis-cluster-configmap.yaml  redis-cluster-deploy.yaml
$ kubectl create -f redis-cluster-configmap.yaml 
$ kubectl create -f redis-cluster-deploy.yaml

# (3) 查看应用的 statefulsets,pod,svc 等信息
$  kubectl get statefulsets,pod,svc -n database
NAME                               READY   AGE
statefulset.apps/redis-cluster     6/6     13m

NAME                    READY   STATUS    RESTARTS   AGE
pod/redis-cluster-0     1/1     Running   0          13m
pod/redis-cluster-1     1/1     Running   0          12m
pod/redis-cluster-2     1/1     Running   0          12m
pod/redis-cluster-3     1/1     Running   0          10m
pod/redis-cluster-4     1/1     Running   0          9m19s
pod/redis-cluster-5     1/1     Running   0          9m16s

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
service/redis-cluster              ClusterIP   None             <none>        6379/TCP,16379/TCP   13m


# (4) 获取 Redis 集群 6 个节点 Pod 的 ip 地址
kubectl get pod -n database -l app=redis-cluster -o jsonpath='{ range.items [*]}{.status.podIP}:6379 '| sed "s# :6379 ##g"
# 10.66.35.66:6379 10.66.237.142:6379 10.66.55.76:6379 10.66.53.96:6379 10.66.35.92:6379 10.66.55.77:6379

步骤 04.进入到第一个Pod 终端中进行初始化 Redis 集群操作，验证Redis Cluster集群

# (1) 进入redis-cluster-0 pod中的shell
kubectl exec -n database -it redis-cluster-0 sh 

# (2) 进行集群初始化，运行以下命令并输入'yes'接受配置。我们将看到前三个节点将被选择为主节点，后三个节点将被选择为从节点。
/data $ redis-cli -a weiyigeek.top --cluster create --cluster-replicas 1 10.66.35.66:6379 10.66.237.142:6379 10.66.55.76:6379 10.66.53.96:6379 10.66.35.92:6379 10.66.55.77:6379
  # >>> Performing hash slots allocation on 6 nodes...
  # Master[0] -> Slots 0 - 5460
  # Master[1] -> Slots 5461 - 10922
  # Master[2] -> Slots 10923 - 16383
  # Adding replica 10.66.35.92:6379 to 10.66.35.66:6379
  # Adding replica 10.66.55.77:6379 to 10.66.237.142:6379
  # Adding replica 10.66.53.96:6379 to 10.66.55.76:6379
  # M: b13a58f799e058dd9afba221f25a8bd53ae05969 10.66.35.66:6379
  #    slots:[0-5460] (5461 slots) master
  # M: 729af3b961a978ad15263ef96439177207be7821 10.66.237.142:6379
  #    slots:[5461-10922] (5462 slots) master
  # M: 50c3d31e7277ce339752a980f14c03d180e2f98a 10.66.55.76:6379
  #    slots:[10923-16383] (5461 slots) master
  # S: ce02a1eb7efbd4de37383f46c300bb883dcd16b4 10.66.53.96:6379
  #    replicates 50c3d31e7277ce339752a980f14c03d180e2f98a
  # S: 298d5e8528c38f687c9894b9974eb5368555c652 10.66.35.92:6379
  #    replicates b13a58f799e058dd9afba221f25a8bd53ae05969
  # S: f7ecf69d99129fba7737f8ae9a8f172af19a7b72 10.66.55.77:6379
  #    replicates 729af3b961a978ad15263ef96439177207be7821
  # Can I set the above configuration? (type 'yes' to accept): yes  # 此处输入 yes 进行上述卡槽设置。
  # >>> Nodes configuration updated
  # >>> Assign a different config epoch to each node
  # >>> Sending CLUSTER MEET messages to join the cluster
  # Waiting for the cluster to join
  # .
  # >>> Performing Cluster Check (using node 10.66.35.66:6379)
  # M: b13a58f799e058dd9afba221f25a8bd53ae05969 10.66.35.66:6379
  #    slots:[0-5460] (5461 slots) master
  #    1 additional replica(s)
  # S: f7ecf69d99129fba7737f8ae9a8f172af19a7b72 10.66.55.77:6379
  #    slots: (0 slots) slave
  #    replicates 729af3b961a978ad15263ef96439177207be7821
  # M: 50c3d31e7277ce339752a980f14c03d180e2f98a 10.66.55.76:6379
  #    slots:[10923-16383] (5461 slots) master
  #    1 additional replica(s)
  # S: ce02a1eb7efbd4de37383f46c300bb883dcd16b4 10.66.53.96:6379
  #    slots: (0 slots) slave
  #    replicates 50c3d31e7277ce339752a980f14c03d180e2f98a
  # M: 729af3b961a978ad15263ef96439177207be7821 10.66.237.142:6379
  #    slots:[5461-10922] (5462 slots) master
  #    1 additional replica(s)
  # S: 298d5e8528c38f687c9894b9974eb5368555c652 10.66.35.92:6379
  #    slots: (0 slots) slave
  #    replicates b13a58f799e058dd9afba221f25a8bd53ae05969
  # [OK] All nodes agree about slots configuration.
  # >>> Check for open slots...
  # >>> Check slots coverage...
  # [OK] All 16384 slots covered.

# (3) 集群验证 
# 此处我使用集群svc服务名称进行访问 redis-cluster.database.svc.cluster.test。
/data $ redis-cli -c -h redis-cluster.database.svc.cluster.test -a weiyigeek.top

# 查看集群信息
redis-cluster.database.svc.cluster.test:6379> CLUSTER INFO
  cluster_state:ok
  cluster_slots_assigned:16384
  cluster_slots_ok:16384
  cluster_slots_pfail:0
  cluster_slots_fail:0
  cluster_known_nodes:6
  cluster_size:3
  ....

# 查看集群节点
redis-cluster.database.svc.cluster.test:6379> CLUSTER NODES
  f7ecf69d99129fba7737f8ae9a8f172af19a7b72 10.66.55.77:6379@16379 slave 729af3b961a978ad15263ef96439177207be7821 0 1665234666115 2 connected
  b13a58f799e058dd9afba221f25a8bd53ae05969 10.66.35.66:6379@16379 master - 0 1665234667119 1 connected 0-5460
  50c3d31e7277ce339752a980f14c03d180e2f98a 10.66.55.76:6379@16379 master - 0 1665234667119 3 connected 10923-16383
  298d5e8528c38f687c9894b9974eb5368555c652 10.66.35.92:6379@16379 slave b13a58f799e058dd9afba221f25a8bd53ae05969 0 1665234665513 1 connected
  729af3b961a978ad15263ef96439177207be7821 10.66.237.142:6379@16379 myself,master - 0 1665234665000 2 connected 5461-10922
  ce02a1eb7efbd4de37383f46c300bb883dcd16b4 10.66.53.96:6379@16379 slave 50c3d31e7277ce339752a980f14c03d180e2f98a 0 1665234665110 3 connected

温馨提示：在初始化 redis 集群时必须使用 ip 地址，如果使用域名会报如下错误。
温馨提示：当应用连接 redis 集群时使用 pod 的域名, 格式为 svc名字.ns名字.svc.cluster.local

$ nslookup redis-cluster.database.svc.cluster.test
Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   redis-cluster.database.svc.cluster.test
Address: 10.66.35.66
Name:   redis-cluster.database.svc.cluster.test
Address: 10.66.35.92
Name:   redis-cluster.database.svc.cluster.test
Address: 10.66.55.76
Name:   redis-cluster.database.svc.cluster.test
Address: 10.66.53.96
Name:   redis-cluster.database.svc.cluster.test
Address: 10.66.237.142
Name:   redis-cluster.database.svc.cluster.test
Address: 10.66.55.77

步骤 05.使用redis cluster进行KV数据插入并查看数据。

redis-cluster.database.svc.cluster.test:6379> set domain weiyigeek.top EX 10
-> Redirected to slot [5449] located at 10.66.35.66:6379
OK
10.66.35.66:6379> get domain
"weiyigeek.top"
10.66.35.66:6379> get domain
"weiyigeek.top"
10.66.35.66:6379> get domain   # 10秒后到期,则key被销毁
(nil)

温馨提示：如果整个 redis 集群的 pod 全部都挂掉了，pod自动拉起后集群不可用，需要重建集群，此时有两种重建集群方法。

• 方式1.重新开始（不到万不得已不建议这样做）

  1.删除 redis 集群所有的资源，然后重新创建 redis 集群
  2.删除 redis 集群中所有的 pvc(pv)
  3.删除 redis 集群中 pod 对应的 nfs 持久化存储目录
  4.重新创建 redis 集群资源并进行初始化

• 方式2.在原有 redis 集群的基础上进行修复。

  1.将 redis 集群中资源控制器的副本数设置为 0
  2.找到 redis 集群中 pod 对应的 nfs 持久化存储目录后删除 nodes.conf

  /storage/dev/pvc/local# find . -name nodes.conf
  ./database-data-redis-cluster-1-pvc-febdd490-6dbf-4084-860f-1d81602f4ca0/nodes.conf
  ./database-data-redis-cluster-3-pvc-08d6d4b4-5290-4710-b31e-efb5e8d8481e/nodes.conf
  ./database-data-redis-cluster-5-pvc-96db8931-2106-4ac1-9c41-ec8888e99024/nodes.conf
  ./database-data-redis-cluster-4-pvc-c67c4649-ff0f-4767-ae75-3875653886c5/nodes.conf
  ./database-data-redis-cluster-0-pvc-e6b1f1bd-75b0-4777-ab93-993a2e6e5927/nodes.conf
  ./database-data-redis-cluster-2-pvc-d078a7d3-cf0c-4b1e-acbe-59086089fb0d/nodes.conf

  3.重新创建 redis 集群资源 kubectl apply -f redis-cluster-deploy.yaml 导出数据后，然后重新初始化集群。

Kubernetes 中使用 Helm 快速部署 redis 多主多从集群

描述：我们可以使用 Bitnami helm chart 在 K8S 中一键部署 Redis cluster 多主多从。
项目地址：https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster

weiyigeek.top-redis单主与redis多主集群

环境依赖：

• Kubernetes 1.19+
• Helm 3.2.0+
• PV provisioner support in the underlying infrastructure

温馨提示：1.集群中基于nfs的provisioner的动态持卷环境部署可参考此篇文章( https://blog.weiyigeek.top/2022/6-7-664.html )

步骤 01.添加 Bitnami chart 仓库，查看可用 redis-cluster 的 chart 及其对应版本，拉取部署所需的 yaml 文件到本地。

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm3 search repo bitnami/redis-cluster -l
$ helm3 pull bitnami/redis-cluster --version 8.2.4 --untar

步骤 02.修改 redis-cluster 图表的 values.yaml 文件

vim redis-cluster/values.yaml
# 1.修改 clusterDomain Kubernetes Cluster Domain 此处由于我们的集群名称非 cluster.local 所以修改如下
clusterDomain: cluster.test
.....
# 2.集群所需redis镜像，此处我已经上传内部harbor中，则需要根据实际请个改变
image:
  registry: harbor.weiyigeek.top
  repository: library/redis-cluster
  tag: 7.0.5-debian-11-r0
.....
# 3.设置 Pod 安全上下文
podSecurityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001
  sysctls: []
.....
# 4.此处svc采用ClusterIP如果采用NodePort需要更改暴露端口 30000 - 32000
service:
  ports:
    redis: 6379
  nodePorts:
    redis: 31379
  type: ClusterIP
.....
# 5.数据 persistence 持久化配置，此处我采用了NFS动态逻辑卷。
persistence:
  path: /bitnami/redis/data
  storageClass: "nfs-local"
  accessModes:
    - ReadWriteOnce
  size: 8Gi
.....
# 6.设置 Redis statefulset parameters 资源限额
redis:
  resources:
    limits: 
      cpu: 1000m
      memory: 4Gi
    requests: {}
.....
# 7.Redis Cluster 设置，此处我设置3主3从，一共6个节点
# 如果 nodes 为 6 replicas 为 1 表示每个master节点有一个副本
# nodes = numberOfMasterNodes + numberOfMasterNodes * replicas
cluster:
 init: true
  nodes: 6
  replicas: 1
.....
# 8.启用 Redis Prometheus Exporter Metrics
metrics:
  enabled: true
  image:
    registry: harbor.weiyigeek.top
    repository: library/redis-exporter
    tag: 1.44.0-debian-11-r8
.....
# 9.启用并使用sysctlImage配置值设置特权initContainer
sysctlImage:
  enabled: true
  registry: harbor.weiyigeek.top
  repository: library/bitnami-shell
  tag: 11-debian-11-r37
  command:
    - /bin/sh
    - -c
    - |-
      sysctl -w net.core.somaxconn=10000
      echo never > /host-sys/kernel/mm/transparent_hugepage/enabled
  mountHostSys: true

步骤 03.为了加快拉取速度我将docker hub上的指定所需镜像拉取到本地Harbor仓库中。

# redis-cluster:7.0.5-debian-11-r0 
docker pull bitnami/redis-cluster:7.0.5-debian-11-r0 
docker tag bitnami/redis-cluster:7.0.5-debian-11-r0 harbor.weiyigeek.top/library/redis-cluster:7.0.5-debian-11-r0
docker push harbor.weiyigeek.top/library/redis-cluster:7.0.5-debian-11-r0

# redis-exporter & bitnami-shell
docker pull bitnami/redis-exporter:1.44.0-debian-11-r8; docker pull bitnami/bitnami-shell:11-debian-11-r37
docker tag bitnami/redis-exporter:1.44.0-debian-11-r8 harbor.weiyigeek.top/library/redis-exporter:1.44.0-debian-11-r8
docker tag bitnami/bitnami-shell:11-debian-11-r37  harbor.weiyigeek.top/library/bitnami-shell:11-debian-11-r37
docker push harbor.weiyigeek.top/library/redis-exporter:1.44.0-debian-11-r8
docker push harbor.weiyigeek.top/library/bitnami-shell:11-debian-11-r37

步骤 04.开始安装修改后的 redis-cluster 图表到 database 名称空间中，并查看资源部署情况。

helm3 install redis ./redis-cluster --set password=weiyigeek.top  --namespace database --create-namespace --debug

helm3 list -n database
  # NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
  # redis   database        1               2022-10-09 17:35:18.530202622 +0800 CST deployed        redis-cluster-8.2.4     7.0.5
  
kubectl get sts,pod,svc -n database -l app.kubernetes.io/name=redis-cluster
  # NAME                                   READY   AGE
  # statefulset.apps/redis-redis-cluster   6/6     4m4s
  
  # NAME                        READY   STATUS    RESTARTS        AGE
  # pod/redis-redis-cluster-0   2/2     Running   1 (3m3s ago)    4m4s
  # pod/redis-redis-cluster-1   2/2     Running   1 (3m3s ago)    4m4s
  # pod/redis-redis-cluster-2   2/2     Running   2 (3m45s ago)   4m4s
  # pod/redis-redis-cluster-3   2/2     Running   1 (3m ago)      4m4s
  # pod/redis-redis-cluster-4   2/2     Running   1 (3m3s ago)    4m4s
  # pod/redis-redis-cluster-5   2/2     Running   1 (3m ago)      4m4s
  
  # NAME                                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
  # service/redis-redis-cluster            ClusterIP   10.105.252.15   <none>        6379/TCP             4m4s
  # service/redis-redis-cluster-headless   ClusterIP   None            <none>        6379/TCP,16379/TCP   4m5s
  # service/redis-redis-cluster-metrics    ClusterIP   10.96.154.84    <none>        9121/TCP             4m4s

步骤 05.验证使用helm部署的Redis集群

# 1.查询设置或者机生成的 Redis 密码
export REDIS_PASSWORD=$(kubectl get secret --namespace "database" redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

# 2.创建运行Redis客户端的pod：
kubectl run --namespace database redis-redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image bitnami/redis-cluster:7.0.5-debian-11-r0 -- bash

# 3.在Pod中使用shell连接到集群
redis-cli -c -h redis-redis-cluster -a $REDIS_PASSWORD
# 集群信息
redis-redis-cluster:6379> CLUSTER INFO
  # cluster_state:ok
# 节点信息
redis-redis-cluster:6379> CLUSTER NODES
# 数据插入测试
redis-redis-cluster:6379> set name weiyigeek
OK
10.66.237.149:6379> set domain weiyigeek.top
OK
10.66.35.100:6379> set blog blog.weiyigeek.top
OK
10.66.237.149:6379> keys *
1) "name"
2) "blog"
10.66.237.149:6379> get name
"weiyigeek"

weiyigeek.top-执行结果

至此，快速使用helm安装多主多从redis集群完毕！

Kubernetes 快速部署 RedisInsight 工具连接Redis集群

描述：RedisInsight 是 Redis 官方推荐的可视化工具，其功能强大，支持Redis集群连接。

参考地址：https://docs.redis.com/latest/ri/installing/install-k8s/
参考文章: 【9.使用RedisInsight工具对Redis集群CURD操作及数据可视化和性能监控】- https://blog.weiyigeek.top/2022/9-20-686.html

步骤 01.集群中快速 redisinsight 部署资源清单。

tee redisinsight-1.13.0.yaml <<'EOF'
# RedisInsight service with name 'redisinsight-service'
apiVersion: v1
kind: Service
metadata:
  name: redisinsight-service
  namespace: devops
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8001
  selector:
    app: redisinsight
---
# RedisInsight deployment with name 'redisinsight'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redisinsight   # deployment name
  namespace: devops
  labels:
    app: redisinsight  # deployment label
spec:
  replicas: 1          # a single replica pod
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: redisinsight # which pods is the deployment managing, as defined by the pod template
  template:             # pod template
    metadata:
      labels:
        app: redisinsight # label for pod/s
    spec:
      volumes:
        - name: db
          emptyDir: {}    # node-ephemeral volume https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
      containers:
        - name:  redisinsight                  # Container name (DNS_LABEL, unique)
          image: redislabs/redisinsight:1.13.0 # Hub Image
          imagePullPolicy: IfNotPresent        # Pull Policy
          env: 
          - name: RIHOST
            value: "0.0.0.0"
          - name: RIPORT
            value: "8001"
          volumeMounts:
          - name: db 
            mountPath: /db
          ports:
          - containerPort: 8001        # exposed container port and protocol
            protocol: TCP
          livenessProbe:
           httpGet:
              path : /healthcheck/     # exposed RI endpoint for healthcheck
              port: 8001               # exposed container port
           initialDelaySeconds: 5      # number of seconds to wait after the container starts to perform liveness probe
           periodSeconds: 5            # period in seconds after which liveness probe is performed
           failureThreshold: 1         # number of liveness probe failures after which container restarts
EOF

步骤 02.部署命令

kubectl create ns devops
kubectl apply -f redisinsight-1.13.0.yaml

步骤 03.验证服务(你可按照需求使用nodeport或者ingress进行暴露端口)

$ kubectl get pod -n devops
NAME                            READY   STATUS    RESTARTS   AGE
redisinsight-5c4f954694-jc5cd   1/1     Running   0          95s

$ kubectl -n devops port-forward deployment/redisinsight --address 0.0.0.0 30081:8001
Forwarding from 0.0.0.0:30081 -> 8001
Handling connection for 30081
Handling connection for 30081
Handling connection for 30081

步骤 04.浏览器访问 节点IP:30081 即可访问 redisinsight UI 界面，添加集群IP（服务名称）以及链接端口和认证密码。

Host: redis-redis-cluster.database.svc
Port: 6379
Name: redis-cluster
Password: weiyigeek.top

weiyigeek.top-redisinsight链接redis集群图

步骤 05.添加完成后我们便可使用redisinsight进行管理查看操作Redis集群了。

weiyigeek.top-使用redisinsight操作redis集群图

至此完毕。