[TOC]

0x00 前言简述

描述: 有了上一章的基础"4.Jenkins进阶之分布式架构环境配置"介绍,本章主要介绍Kubernetes 的插件使用实例, 以及自定义 Slave 的 Jenkins-Jnlp 容器镜像模板(重点),在进行K8s动态节点生成时拉取使用,极大的节约了系统资源。


0x01 Kubernetes-plugin 使用

描述: 前面已经对Jenkins集成Kubernetes进行部署和配置,本章将进行介绍与实践如何使用该插件以及帮助文档的记录;

说明: Kubernetes-plugin 它是在Kubernetes集群中运行动态代理的Jenkins插件, 该插件为每个启动的代理创建一个Kubernetes Pod,由Docker映像定义运行,并在每次构建后停止。

备注: 代理是使用JNLP启动的,因此预期图像会自动连接到Jenkins主机,以下是重要的环境变量(我们说过不建议使用自己jnlp覆盖默认的jnlp容器)。

1
2
3
4
5
# jenkins/inbound-agent 
JENKINS_URL: Jenkins web interface url
JENKINS_SECRET: the secret key for authentication
JENKINS_AGENT_NAME: the name of the Jenkins agent
JENKINS_NAME: the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)

插件地址: https://github.com/jenkinsci/kubernetes-plugin


环境依赖

  • 1.先决条件

    • 正在运行的Kubernetes集群1.14或更高版本。对于OpenShift用户,这意味着OpenShift容器平台4.x。
    • 安装了一个Jenkins实例
    • 已安装Jenkins Kubernetes插件
  • 2.填写Kubernetes插件配置

    • 您将打开Jenkins UI并导航至Manage Jenkins-> Configure System-> Cloud-> Kubernetes并输入相应的Kubernetes URL和Jenkins URL,除非Jenkins在Kubernetes中运行否则默认工作(参考上一章)。
  • 3.支持的凭证

    • Username/password - 用户名密码
    • Secret File (kubeconfig file) - 秘密文件(kubeconfig文件)
    • Secret text (Token-based authentication) (OpenShift) - 秘密文本(基于令牌的身份验证)(OpenShift)
    • Google Service Account from private key (GKE authentication) - 来自私钥的Google服务帐户(GKE身份验证)
    • X.509 Client Certificate - X.509客户端证书


插件参数

podTemplate - Pod和容器模板

描述: podTemplate 是用于创建代理的pod的模板可以通过用户界面或管道进行配置, 无论哪种方式它都可以访问以下字段:

  • 1) cloud :在Jenkins设置中定义的云的名称。默认为kubernetes
  • 2) name : Pod 名称
  • 3) namespace : 名称空间设置
  • 4) label : 标签值可以设置为唯一的值,以避免跨构建冲突,或省略,并在步骤中定义POD_LABEL。
  • 5) yaml : 资源清单声明
  • 6) yamlFile : 指定资源清单的文件
  • 7) yamlMergeStrategy : 控制yaml定义是覆盖还是与从用声明的pod模板继承的yaml定义合并inheritFrom。默认为override()。
  • 8) showRawYaml : 启用或禁用原始Yaml文件的输出默认为true (后续实践)
  • 8) serviceAccount : k8s中创建的服务帐户
  • 9) nodeSelector :节点选择器
  • 10) nodeUsageMode : 节点调用模式(NORMAL它控制Jenkins是只调度标签表达式匹配的作业,EXCLUSIVE尽可能多地使用该节点。)
  • 11) volumes :为pod定义各种类型的卷并在所有容器中挂载卷
  • 12) envVars :应用于所有容器的环境变量(envVar 环境变量其值是内联定义的, secretEnvVar 一个环境变量其值是从Kubernetes机密派生的。)
  • 13) imagePullSecrets : 从私有的镜像仓库中拉取镜像的凭据
  • 14) annotations : Pod 注解
  • 15) InheritFrom :继承的一个或多个Pod模板的列表
  • 16) slaveConnectTimeout : 代理程序联机超时时间(单位s)
  • 17) podRetention :控制保留代理Pod的行为(Can be 'never()', 'onFailure()', 'always()', or 'default()'),如果为空则按照下面的active Deadline Seconds参数进行。
  • 18) activeDeadlineSeconds : pod将在这个截止日期过后被删除。
  • 19) idleMinutes : 允许Pod保持活动状态以供重用,直到在执行最后一步以来经过了配置的分钟数为止。
  • 20) runAsUser : 用户ID,用于将pod中的所有容器运行为。
  • 21) runAsGroup : 组ID,用于将Pod中的所有容器运行为。
  • 22) hostNetwork : 使用主机网络。
  • 23) container : 用于创建pod容器的容器模板 (见下文) - (重点)。
  • 24) defaultContainer : 指定Pod中默认容器则在stages中不用加入container进行指定选择;


container - 容器模板的定义

描述: containerTemplate 是一个将被添加到pod中的容器模板它属于container中数组对象。同样它可以通过用户界面或管道进行配置,并允许你设置以下字段:

containerTemplate - 容器模板

  • name : 容器的名称。
  • image : 容器的图像。
  • envVars : 应用于容器的环境变量(补充和覆盖在pod级别设置的env var)。
    • envVar 环境变量,其值是内联定义的。
    • secretEnvVar 一个环境变量,其值是从Kubernetes机密派生的。
  • workingDir : 工作空间目录
  • command : 容器将执行的命令。
  • args : 传递给命令的参数。
  • ttyEnabled : 标记以指示应启用tty。
  • livenessProbe : 要添加到容器中的exec liveness探针的参数(不支持httpGet liveness探针)
  • ports : 暴露容器上的端口。
  • alwaysPullImage : 容器在启动时将拉出图像。
  • runAsUser : 用于运行容器的用户ID。
  • runAsGroup : 运行容器的组ID。


Liveness Probe Usage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
containerTemplate(
name: 'busybox',
image: 'busybox',
ttyEnabled: true,
command: 'cat',
livenessProbe: containerLivenessProbe(
execArgs: 'some --command',
initialDelaySeconds: 30,
timeoutSeconds: 1,
failureThreshold: 3,
periodSeconds: 10,
successThreshold: 1
)
)

Tips : 默认情况下代理连接超时设置为100秒。在某些情况下您想要设置一个不同的值,如果可以可以将system属性设置org.csanchez.jenkins.plugins.kubernetes.PodTemplate.connectionTimeout为一个不同的值


Pipeline Support 实践示例

Scripted Pipeline

描述: Scripted Pipeline 示例参考:


(1) containerTemplate

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
podTemplate(
cloud: 'kubernetes',
name: 'jenkins-slave',
namespace: 'devops',
label: 'k8s-slave',
// 容器及容器模板定义
containers: [
containerTemplate(
name: 'maven',
image: 'maven:3.6-jdk-8-alpine',
ttyEnabled: true,
command: 'cat',
privileged: false,
alwaysPullImage: false,
workingDir: '/home/jenkins/agent',
resourceRequestCpu: '100m',
resourceLimitCpu: '500m',
resourceRequestMemory: '100Mi',
resourceLimitMemory: '500Mi',
envVars: [
envVar(key: 'MYSQL_ALLOW_EMPTY_PASSWORD', value: 'true')
//, secretEnvVar(key: 'MYSQL_PASSWORD', secretName: 'mysql-secret', secretKey: 'password')
]
//,ports: [portMapping(name: 'mysql', containerPort: 3306, hostPort: 3306)]
)
// ),
// containerTemplate(
// name: 'jnlp',
// image: 'registry.cn-hangzhou.aliyuncs.com/google-containers/jnlp-slave:alpine',
// args: '${computer.jnlpmac} ${computer.name}',
// command: ''
// )
]
,volumes: [
hostPathVolume(hostPath: '/nfsdisk-31/appstorage/mavenRepo', mountPath: '/home/jenkins'),
hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock'),
/*
emptyDirVolume(mountPath: '/etc/mount1', memory: false),
secretVolume(mountPath: '/etc/mount2', secretName: 'my-secret'),
configMapVolume(mountPath: '/etc/mount3', configMapName: 'my-config'),
hostPathVolume(mountPath: '/etc/mount4', hostPath: '/mnt/my-mount'),
nfsVolume(mountPath: '/etc/mount5', serverAddress: '127.0.0.1', serverPath: '/', readOnly: true),
persistentVolumeClaim(mountPath: '/home/jenkins', claimName: 'jenkins', readOnly: false),
*/
],
annotations: [
podAnnotation(key: "maven-pod", value: "Kubernetes-jenkins-Test")
],
showRawYaml: 'flase'
//,defaultContainer: 'maven' # scripted pipeline 不支持该参数
//,imagePullSecrets: [ 'pull-secret' ]
)
{
// 注意此次node中则为Pod模板名称
node ('k8s-slave') {
// container 中指定Pod中容器名称
container('maven') {
stage('maven') {
container('maven') {
sh "hostname && ip addr"
sh ("mvn -version")
sh "env"
}
}
}
}
}

运行结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
# Still waiting to schedule task
# Waiting for next available executor on 'jenkins-slave-09vf0-sf3k6'
# 等待下一个可用的执行器启动

Created Pod: kubernetes devops/jenkins-slave-rtjxx-q3nhn
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Scheduled] Successfully assigned devops/jenkins-slave-rtjxx-q3nhn to work-223

[Normal][devops/jenkins-slave-rtjxx-q3nhn][Pulled] Container image "maven:3.6-jdk-8-alpine" already present on machine
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Created] Created container maven | Started container maven

[Normal][devops/jenkins-slave-rtjxx-q3nhn][Pulled] Container image "jenkins/inbound-agent:4.3-4" already present on machine
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Created] Created container jnlp | Started container jnlp

Agent jenkins-slave-rtjxx-q3nhn is provisioned from template jenkins-slave-rtjxx

# ----------------------------------
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
maven-pod: "Kubernetes-jenkins-Test" # 我们添加的注解
buildUrl: "http://jenkins.devops.svc.cluster.local:8080/job/Kubernetes-jenkins-slave/23/"
runUrl: "job/Kubernetes-jenkins-slave/23/"
labels:
jenkins: "slave"
jenkins/label-digest: "823e2bb3db243573786b7a360a483f1acc7a0f96"
jenkins/label: "k8s-slave"
name: "jenkins-slave-rtjxx-q3nhn"
spec:
containers:
- command:
- "cat" # 运行的命令
env:
- name: "MYSQL_ALLOW_EMPTY_PASSWORD" # 运行的环境变量
value: "true"
image: "maven:3.6-jdk-8-alpine"
imagePullPolicy: "IfNotPresent" # 镜像拉取策略
name: "maven" # 容器名称
resources:
limits:
memory: "500Mi"
cpu: "500m"
requests:
memory: "100Mi"
cpu: "100m"
tty: true
volumeMounts: # 卷挂载情况
- mountPath: "/home/jenkins"
name: "volume-0"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-1"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
- env: # jnlp 环境变量
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_AGENT_NAME"
value: "jenkins-slave-rtjxx-q3nhn"
- name: "JENKINS_NAME"
value: "jenkins-slave-rtjxx-q3nhn"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.devops.svc.cluster.local:8080/"
image: "jenkins/inbound-agent:4.3-4"
name: "jnlp" # jenkins-jnlp 容器的名称
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins"
name: "volume-0"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-1"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- hostPath:
path: "/nfsdisk-31/appstorage/mavenRepo"
name: "volume-0"
- hostPath:
path: "/var/run/docker.sock"
name: "volume-1"
- emptyDir:
medium: ""
name: "workspace-volume"

# ------------------------------
+ hostname
# jenkins-slave-rtjxx-q3nhn

+ ip addr
# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
# link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
# inet 127.0.0.1/8 scope host lo
# valid_lft forever preferred_lft forever
# 2: [email protected]: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
# link/ipip 0.0.0.0 brd 0.0.0.0
# 4: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue state UP
# link/ether de:c6:df:23:e1:7f brd ff:ff:ff:ff:ff:ff
# inet 172.16.24.220/32 brd 172.16.24.220 scope global eth0
# valid_lft forever preferred_lft forever
+ mvn -version
# Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
# Maven home: /usr/share/maven
# Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
# Default locale: en_US, platform encoding: UTF-8
# OS name: "linux", version: "5.4.0-42-generic", arch: "amd64", family: "unix"

+ env
# JENKINS_HOME=/var/jenkins_home
# POD_CONTAINER=maven
# STAGE_NAME=maven
# KUBERNETES_SERVICE_PORT=443
# JAVA_VERSION=8u212
# MAVEN_HOME=/usr/share/maven
# MAVEN_CONFIG=/root/.m2

Tips : 在PodTemplate模板中设置 showRawYaml: 'flase' 则将不会显示资源清单在日志中;

WeiyiGeek.脚本式流水线k8s插件使用

WeiyiGeek.脚本式流水线k8s插件使用

Tips : 要使用 secretEnvVar 我们必须在 Dashboard -> 凭据 -> 系统 -> 全局凭据 (unrestricted) 进行添加一个Secret-text凭据(未能成功复现);

1
Secret text	4fc93636-d8f1-4051-8011-d852873fc740	mysql-secret	Secret text	mysql-secret


Declarative Pipeline

描述: 声明式管道支持需要Jenkins 2.66+, 在Declarative Pipeline使用的 kubernetes 选项说明;

1
options are [activeDeadlineSeconds, cloud, containerTemplate, containerTemplates, customWorkspace, defaultContainer, idleMinutes, inheritFrom, instanceCap, label, namespace, nodeSelector, podRetention, serviceAccount, slaveConnectTimeout, supplementalGroups, workingDir, workspaceVolume, yaml, yamlFile, yamlMergeStrategy]

示例帮助: https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/resources/org/csanchez/jenkins/plugins/kubernetes/pipeline/samples


示例1.通过yaml文件资源清单创建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
pipeline {
agent {
kubernetes {
cloud 'kubernetes'
namespace 'devops'
workingDir '/home/jenkins/agent'
// 继承模板
inheritFrom 'jenkins-slave'
defaultContainer 'maven'
// yamlFile 'KubernetesPod.yaml'
yaml """\
apiVersion:
kind: Pod
metadata:
labels:
jenkins: "slave"
jenkins/label: 'k8s-slave'
spec:
containers:
- name: 'maven'
image: 'maven:3.6-jdk-8-alpine'
imagePullPolicy: 'IfNotPresent' # 镜像拉取策略
command:
- cat
tty: true
""".stripIndent()
}
}
stages {
stage ('declarative Pipeline - kubernetes') {
steps {
echo "declarative Pipeline - kubernetes"
container ('maven') {
sh "echo ${POD_CONTAINER}"
sh "mvn -version"
}
}
}
}
}

执行结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
# 'kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8' is offline
# Agent kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8 is provisioned from template Kubernetes-jenkins-slave_49-fcd1s-m3n3q
---
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins.devops.svc.cluster.local:8080/job/Kubernetes-jenkins-slave/49/"
runUrl: "job/Kubernetes-jenkins-slave/49/"
labels:
jenkins: "slave"
jenkins/label: "Kubernetes-jenkins-slave_49-fcd1s"
jenkins/label-digest: "95508e0ca05226ba84989959dc886ad4011cf125"
name: "kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8"
spec:
containers:
- command:
- "cat"
image: "maven:3.6-jdk-8-alpine"
imagePullPolicy: "IfNotPresent"
name: "maven"
tty: true
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_AGENT_NAME"
value: "kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8"
- name: "JENKINS_NAME"
value: "kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.devops.svc.cluster.local:8080/"
image: "jenkins/inbound-agent:4.3-4"
name: "jnlp"
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins/.m2"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
hostNetwork: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- hostPath:
path: "/nfsdisk-31/appstorage/mavenRepo"
name: "volume-0"
- emptyDir:
medium: ""
name: "workspace-volume"

Running on kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8 in /home/jenkins/agent/workspace/Kubernetes-jenkins-slave
# [Pipeline] {
# [Pipeline] container
# [Pipeline] {
# [Pipeline] stage
# [Pipeline] { (declarative Pipeline - kubernetes)
[Pipeline] echo
declarative Pipeline - kubernetes
# [Pipeline] container
# [Pipeline] {
# [Pipeline] sh
+ echo maven
maven
# [Pipeline] sh
+ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-42-generic", arch: "amd64", family: "unix"
# [Pipeline] }
# [Pipeline] // container
# [Pipeline] }
# [Pipeline] // stage
# [Pipeline] }
# [Pipeline] // container
# [Pipeline] }
# [Pipeline] // node
# [Pipeline] }
# [Pipeline] // podTemplate
# [Pipeline] End of Pipeline
Finished: SUCCESS

Tips :默认情况下在容器中运行步骤, 步骤将嵌套在隐式container(name){…}块中,而不是在jnlp容器中执行。

1
defaultContainer 'maven'

Tips : 您也可以使用yamlFile将pod模板保存在单独的KubernetesPod.yaml文件中

1
2
3
4
5
6
7
8
9
10
pipeline {
agent {
kubernetes {
yamlFile 'KubernetesPod.yaml'
}
}
stages {
...
}
}


0x02 Jenkins工作节点之自定义jnlp容器模板

Dockerfile 构建依赖

描述: 下述相关脚本以及文件下载地址请在WeiyiGeek微信公众号回复jenkins-jnlp-dockerfile关键字获取;

自定义的jnlp容器模板主要实现功能:

  • 用户权限控制(sudo)
  • ssh 远程连接
  • git 代码版本控制
  • docker 容器管理
  • kubectl 集群管理
  • Java 运行环境
  • Maven 运行环境
  • SonarQube 扫描环境
  • Gitlab_release 上传环境
  • 中文环境支持
  • 时区更改配置


Tips: 在K8s集群中测试alpine镜像执行相应的安装(需要在Alpine调试新安装软件时使用):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: alpine-app
spec:
containers:
- name: alpine-app
image: alpine:latest
args:
- sleep
- "100000"
EOF

pod/alpine-app created

$ kubectl get pods -o wide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# alpine-app 1/1 Running 0 116s 10.100.37.194 worker-02 <none> <none>


Dockerfile

说明: 镜像构建文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
#----------------------------------------------------------------------#
# Title: Base in Alpine Images Create Custom Jenkins Kubernetes jnlp Images
# Author: WeiyiGeek
# WebSite: https://weiyigeek.top
# MainFunction:
# Install ssh-server docker git openssh tzdata curl tar sudo git ca-certificates wget unzip docker zlib nodejs npm jq
# Install JDK8
# - https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html
# - https://github.com/sgerrand/alpine-pkg-glibc/releases/
# - https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
# Install jnlp
# - https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/
# Install Maven
# - https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
# Install SonarqubeScan
# - https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.5.0.2216-linux.zip
# Install Gitlab Release Version: 0.0.6
# - https://gitlab.com/gitlab-org/release-cli/-/releases
# Install kubernetes cli
# - kubectl Version: 1.19.3
# Version: alpine-3.13.1
# Release: v1.10
# ChangeLog:
# v1.9 - 增加 中文环境
# v1.10 - 增加 node.js 环境支持
#-------------------------------------------------#

FROM alpine:3.13.1

MAINTAINER Jenkins Custom Work Node Jnlp Container - <[email protected]> - WeiyiGeek

ARG USERNAME=jenkins \
AGENT_WORKDIR=/home/jenkins \
BASE_DIR=/usr/local \
BASE_BIN=/usr/local/bin \
BASE_URL=http://192.168.12.107:8080 \
LOCALE=locale.md \
JDK_NAME=jdk-8u281-linux-x64 \
JDK_DIR=/usr/local/jdk1.8.0_281 \
GLIBC_NAME=glibc-2.32-r0.apk \
GLIBC_BIN_NAME=glibc-bin-2.32-r0.apk \
GLIBC_I18N_NAME=glibc-i18n-2.32-r0.apk \
MAVEN_NAME=apache-maven-3.6.3-bin \
MAVEN_DIR=/usr/local/apache-maven-3.6.3 \
SONAR_SCANNER_NAME=sonar-scanner-cli-4.5.0.2216-linux \
SONAR_SCANNER_DIR=/usr/local/sonar-scanner-4.5.0.2216-linux \
GITLAB_CLI=release-cli-0.6.0-linux-amd64

ENV LANG=en_US.UTF-8 \
LC_ALL=en_US.UTF-8 \
JAVA_HOME=/usr/local/jdk8 \
JRE_HOME=/usr/local/jdk8/jre \
MAVEN_HOME=/usr/local/maven \
MAVEN_RPEO=/home/jenkins/.m2 \
SONAR_SCANNER_HOME=/usr/local/sonar-scanner \
NODEJS_MODULES=/usr/lib/node_modules

# 用户ROOT切换
USER root

# Shell 命令 - 此种方式极大减少了构建的镜像大小;
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \
&& apk update \
&& apk add --no-cache openssh tzdata curl tar sudo git ca-certificates wget unzip docker zlib nodejs npm jq \
&& cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& chmod 4755 /bin/busybox \
&& addgroup -g 1000 -S ${USERNAME} \
&& adduser ${USERNAME} -D -g ${USERNAME} -G root -u 1000 -s /bin/sh \
&& echo "jenkins ALL=(root) NOPASSWD:ALL" >> /etc/sudoers \
&& mkdir -p ${AGENT_WORKDIR}/.ssh ${AGENT_WORKDIR}/.m2 \
&& wget -q -O /tmp/${GLIBC_NAME} ${BASE_URL}/${GLIBC_NAME} \
&& wget -q -O /tmp/${GLIBC_BIN_NAME} ${BASE_URL}/${GLIBC_BIN_NAME} \
&& wget -q -O /tmp/${GLIBC_I18N_NAME} ${BASE_URL}/${GLIBC_I18N_NAME} \
&& wget -q -O /etc/apk/keys/sgerrand.rsa.pub ${BASE_URL}/sgerrand.rsa.pub \
&& wget -q -O /tmp/${LOCALE} ${BASE_URL}/${LOCALE} \
&& wget -q -O /tmp/${JDK_NAME}.tar.gz ${BASE_URL}/${JDK_NAME}.tar.gz \
&& wget -q -O ${BASE_BIN}/agent.jar ${BASE_URL}/agent.jar \
&& curl -fsSL -o ${BASE_BIN}/jenkins-agent.sh ${BASE_URL}/jenkins-agent.sh \
&& curl -fsSL -o /tmp/${MAVEN_NAME}.tar.gz ${BASE_URL}/${MAVEN_NAME}.tar.gz \
&& curl -fsSL -o /tmp/${SONAR_SCANNER_NAME}.zip ${BASE_URL}/${SONAR_SCANNER_NAME}.zip \
&& curl -fsSL -o /usr/local/bin/release-cli ${BASE_URL}/${GITLAB_CLI} \
&& curl -fsSL -o /usr/local/bin/kubectl ${BASE_URL}/kubectl \
&& sed -i "s/#PermitRootLogin.*/PermitRootLogin yes/g" /etc/ssh/sshd_config \
&& sed -i "s/^#\s*StrictHostKeyChecking ask/StrictHostKeyChecking no/g" /etc/ssh/ssh_config \
&& ssh-keygen -t dsa -P "" -f /etc/ssh/ssh_host_dsa_key \
&& ssh-keygen -t rsa -P "" -f /etc/ssh/ssh_host_rsa_key \
&& ssh-keygen -t ecdsa -P "" -f /etc/ssh/ssh_host_ecdsa_key \
&& ssh-keygen -t ed25519 -P "" -f /etc/ssh/ssh_host_ed25519_key \
&& ssh-keygen -t ed25519 -P "" -C "[email protected]" -f /home/jenkins/.ssh/id_ed25519 \
&& apk add /tmp/${GLIBC_NAME} /tmp/${GLIBC_BIN_NAME} /tmp/${GLIBC_I18N_NAME} \
&& tar -zxf /tmp/${JDK_NAME}.tar.gz -C ${BASE_DIR} \
&& mv ${JDK_DIR} ${JAVA_HOME} \
&& tar -zxf /tmp/${MAVEN_NAME}.tar.gz -C ${BASE_DIR} \
&& mv ${MAVEN_DIR} ${MAVEN_HOME} \
&& unzip /tmp/${SONAR_SCANNER_NAME}.zip -d ${BASE_DIR} \
&& mv ${SONAR_SCANNER_DIR} ${SONAR_SCANNER_HOME} \
&& npm config set registry https://registry.npm.taobao.org \
&& chmod a+x ${BASE_BIN}/* \
&& chown -R jenkins:jenkins ${BASE_DIR}/ ${AGENT_WORKDIR}/ ${NODEJS_MODULES}/ \
&& echo "root:WeiyiGeek" | chpasswd \
&& echo "jenkins:WeiyiGeek" | chpasswd \
&& cat /tmp/${LOCALE} | xargs -i /usr/glibc-compat/bin/localedef -i {} -f UTF-8 {}.UTF-8 \
&& sed -i "s#use_embedded_jre=true#use_embedded_jre=false#g" ${SONAR_SCANNER_HOME}/bin/sonar-scanner \
&& rm -rf /var/cache/apk/* /tmp/* ${SONAR_SCANNER_HOME}/jre/* \
&& cd ${JAVA_HOME} \
&& rm -rf COPYRIGHT LICENSE README release THIRDPARTYLICENSEREADME-JAVAFX.txt THIRDPARTYLICENSEREADME.txt Welcome.html javafx-src.zip src.zip \
lib/plugin.jar \
lib/ext/jfxrt.jar \
bin/javaws \
lib/javaws.jar \
lib/desktop \
plugin \
lib/deploy* \
lib/*javafx* \
lib/*jfx* \
lib/amd64/libdecora_sse.so \
lib/amd64/libprism_*.so \
lib/amd64/libfxplugins.so \
lib/amd64/libglass.so \
lib/amd64/libgstreamer-lite.so \
lib/amd64/libjavafx*.so \
lib/amd64/libjfx*.so
&& echo "export LANG=zh_CN.UTF-8" > /etc/profile.d/locale.sh

USER jenkins

WORKDIR ${AGENT_WORKDIR}

ENV CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar \
PATH=${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${SONAR_SCANNER_HOME}/bin:$PATH

ENTRYPOINT ["/usr/local/bin/jenkins-agent.sh"]


local.md
描述:操作系统中文字符支持;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
tee local.md <<'END'
en_AG
en_AU
en_BW
en_CA
en_DK
en_GB
en_HK
en_IE
en_IN
en_NG
en_NZ
en_PH
en_SG
en_US
en_ZA
en_ZM
en_ZW
zh_CN
zh_HK
zh_SG
zh_TW
zu_ZA
END


jenkins-agent.sh
描述: 容器启动时与jenkins利用Java jnlp 进行连接的脚本;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# Usage jenkins-agent.sh [options] -url http://jenkins [SECRET] [AGENT_NAME]
# Optional environment variables :
# * JENKINS_TUNNEL : HOST:PORT for a tunnel to route TCP traffic to jenkins host, when jenkins can't be directly accessed over network
# * JENKINS_URL : alternate jenkins URL
# * JENKINS_SECRET : agent secret, if not set as an argument
# * JENKINS_AGENT_NAME : agent name, if not set as an argument
# * JENKINS_AGENT_WORKDIR : agent work directory, if not set by optional parameter -workDir
# * JENKINS_WEB_SOCKET: true if the connection should be made via WebSocket rather than TCP
# * JENKINS_DIRECT_CONNECTION: Connect directly to this TCP agent port, skipping the HTTP(S) connection parameter download.
# Value: "<HOST>:<PORT>"
# * JENKINS_INSTANCE_IDENTITY: The base64 encoded InstanceIdentity byte array of the Jenkins master. When this is set,
# the agent skips connecting to an HTTP(S) port for connection info.
# * JENKINS_PROTOCOLS: Specify the remoting protocols to attempt when instanceIdentity is provided.

if [ $# -eq 1 ]; then
# if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
exec "[email protected]"
else
# if -tunnel is not provided, try env vars
case "[email protected]" in
*"-tunnel "*) ;;
*)
if [ ! -z "$JENKINS_TUNNEL" ]; then
TUNNEL="-tunnel $JENKINS_TUNNEL"
fi ;;
esac

# if -workDir is not provided, try env vars
if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
case "[email protected]" in
*"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
*)
WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
esac
fi

if [ -n "$JENKINS_URL" ]; then
URL="-url $JENKINS_URL"
fi

if [ -n "$JENKINS_NAME" ]; then
JENKINS_AGENT_NAME="$JENKINS_NAME"
fi

if [ "$JENKINS_WEB_SOCKET" = true ]; then
WEB_SOCKET=-webSocket
fi

if [ -n "$JENKINS_PROTOCOLS" ]; then
PROTOCOLS="-protocols $JENKINS_PROTOCOLS"
fi

if [ -n "$JENKINS_DIRECT_CONNECTION" ]; then
DIRECT="-direct $JENKINS_DIRECT_CONNECTION"
fi

if [ -n "$JENKINS_INSTANCE_IDENTITY" ]; then
INSTANCE_IDENTITY="-instanceIdentity $JENKINS_INSTANCE_IDENTITY"
fi

# if java home is defined, use it
JAVA_BIN="java"
if [ "$JAVA_HOME" ]; then
JAVA_BIN="$JAVA_HOME/bin/java"
fi

# if both required options are defined, do not pass the parameters
OPT_JENKINS_SECRET=""
if [ -n "$JENKINS_SECRET" ]; then
case "[email protected]" in
*"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
*)
OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
esac
fi

OPT_JENKINS_AGENT_NAME=""
if [ -n "$JENKINS_AGENT_NAME" ]; then
case "[email protected]" in
*"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
*)
OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
esac
fi

#TODO: Handle the case when the command-line and Environment variable contain different values.
#It is fine it blows up for now since it should lead to an error anyway.

exec $JAVA_BIN $JAVA_OPTS -cp /usr/local/bin/agent.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $WEB_SOCKET $DIRECT $PROTOCOLS $INSTANCE_IDENTITY $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "[email protected]"
fi


镜像构建所需软件
备注: 在Jenkins 2.277版本中添加一个新的节点中获取匹配当前版本的 agent.jar, 或者是在 jenkins 官网 https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting 进行下载;

1
2
3
~/k8s/jenkins/jnlp-slave$ ls
agent.jar build jdk-8u281-linux-x64.tar.gz Jenkins.zip release-cli-0.6.0-linux-amd64 sonar-scanner-cli-4.5.0.2216-linux.zip
apache-maven-3.6.3-bin.tar.gz glibc-2.32-r0.apk jenkins-agent.sh kubectl sgerrand.rsa.pub


Dockerfile 构建操作

操作流程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# (0) 启动一个临时的web服务器(`存放上面的镜像构建所需软件`-非常重要-否则将会导致构建失败)
~/k8s/jenkins/jnlp-slave$ python3 -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /glibc-2.32-r0.apk HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /glibc-bin-2.32-r0.apk HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /glibc-i18n-2.32-r0.apk HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /sgerrand.rsa.pub HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /locale.md HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /jdk-8u281-linux-x64.tar.gz HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /agent.jar HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /jenkins-agent.sh HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /apache-maven-3.6.3-bin.tar.gz HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /sonar-scanner-cli-4.5.0.2216-linux.zip HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /release-cli-0.6.0-linux-amd64 HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /kubectl HTTP/1.1" 200 -

# (1) 镜像构建
docker build -t harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine .
# Successfully built b47b6581b712
# Successfully tagged harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine

# (2) 查看构建信息
docker images harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine --all
# REPOSITORY TAG IMAGE ID CREATED SIZE
# harbor.weiyigeek.top/devops/jenkins-jnlp 3.13.1-alpine b47b6581b712 About an hour ago 715MB

# (3) 推送镜像
docker push harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine
# The push refers to repository [harbor.weiyigeek.top/devops/jenkins-jnlp]
# 059ea3fbd3b3: Pushed
# 1119ff37d4a9: Layer already exists
# 3.13.1-alpine: digest: sha256:1f869c553340c9399c7db9072169600a17ddb0ec41d41d3a4365b8c1571fc201 size: 741

# (4) 采用Ansible各节点拉取镜像
ansible node -m shell -a "docker pull harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine"

WeiyiGeek.DockerFile构建自定义Jnlp容器

WeiyiGeek.DockerFile构建自定义Jnlp容器


自定义 jenkins-jnlp 镜像

  • Step 1.创建流水线 Kubernetes-jenkins-slave Job 在流水线中采用Pipeline Script脚本

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    pipeline {
    agent {
    kubernetes {
    cloud 'kubernetes'
    namespace 'devops'
    inheritFrom 'jenkins-slave'
    workingDir '/home/jenkins/agent'
    // yamlFile 'KubernetesPod.yaml'
    yaml """\
    apiVersion:
    kind: Pod
    metadata:
    labels:
    jenkins: "slave"
    jenkins/label: 'k8s-slave'
    spec:
    containers:
    - name: 'jnlp'
    image: 'harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine'
    imagePullPolicy: 'IfNotPresent' # 镜像拉取策略
    command: ["/bin/sh","-c","/usr/local/bin/jenkins-agent.sh && cat"] # 重点测试的时候(心酸累)希望读者体验一哈
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.m2"
    name: "volume-0"
    - mountPath: "/var/run/docker.sock"
    name: "volume-1"
    """.stripIndent()
    }
    }
    stages {
    stage ('declarative Pipeline - kubernetes') {
    steps {
    echo "declarative Pipeline - kubernetes"
    sh "mvn -version"
    sh "release-cli -v"
    sh "sonar-scanner -v"
    sh "kubectl version"
    sh "docker --version && sudo docker ps"
    }
    }
    }
    }
  • Step 2.配置Pod Templates模板如下因为我们需要继承一哈

    1
    2
    3
    4
    5
    6
    7
    8
    名称:jenkins-slave
    命名空间: devops
    标签列表: k8s-slave
    添加主机卷:
    - 主机: /nfsdisk-31/appstorage/mavenRepo
    - Pod 挂载路径: /home/jenkins/.m2
    - 主机: /var/run/docker.sock
    - Pod 挂载路径: /var/run/docker.sock
  • Step 3.在BlueOcen中运行并查看结果

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    # (1) 可以看见当进行调度时k8s会动态的拉取镜像并运行,当任务结束后会自动销毁Pod
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 0/1 Pending 0 0s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 0/1 Pending 0 0s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 0/1 ContainerCreating 0 0s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 1/1 Running 0 3s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 1/1 Terminating 0 13s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 0/1 Terminating 0 45s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 0/1 Terminating 0 46s
    kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 0/1 Terminating 0 46s

    # (2) 动态创建的slave启动的Pod脚本
    ---
    kind: "Pod"
    metadata:
    annotations:
    buildUrl: "http://jenkins.devops.svc.cluster.local:8080/job/Kubernetes-jenkins-slave/63/"
    runUrl: "job/Kubernetes-jenkins-slave/63/"
    labels:
    jenkins: "slave"
    jenkins/label: "Kubernetes-jenkins-slave_63-01h9j"
    jenkins/label-digest: "0fe6224168b9ce0350e7db8678c10953c2e6f533"
    name: "kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5"
    spec:
    containers:
    - command:
    - "/bin/sh"
    - "-c"
    - "/usr/local/bin/jenkins-agent.sh && cat"
    env:
    - name: "JENKINS_SECRET"
    value: "********"
    - name: "JENKINS_AGENT_NAME"
    value: "kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5"
    - name: "JENKINS_NAME"
    value: "kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5"
    - name: "JENKINS_AGENT_WORKDIR"
    value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
    value: "http://jenkins.devops.svc.cluster.local:8080/"
    image: "harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.1-alpine"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
    limits: {}
    requests:
    memory: "256Mi"
    cpu: "100m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.m2"
    name: "volume-0"
    - mountPath: "/var/run/docker.sock"
    name: "volume-1"
    - mountPath: "/home/jenkins/agent"
    name: "workspace-volume"
    readOnly: false
    hostNetwork: false
    nodeSelector:
    kubernetes.io/os: "linux"
    restartPolicy: "Never"
    volumes:
    - hostPath:
    path: "/nfsdisk-31/appstorage/mavenRepo"
    name: "volume-0"
    - hostPath:
    path: "/var/run/docker.sock"
    name: "volume-1"
    - emptyDir:
    medium: ""
    name: "workspace-volume"

    # Running on kubernetes-jenkins-slave-63-01h9j-txdgn-cdwb5 in /home/jenkins/agent/workspace/Kubernetes-jenkins-slave

    # 脚本反馈
    declarative Pipeline - kubernetes
    + mvn -version
    Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
    Maven home: /usr/local/maven
    Java version: 1.8.0_281, vendor: Oracle Corporation, runtime: /usr/local/jdk8/jre
    Default locale: en_US, platform encoding: ANSI_X3.4-1968
    OS name: "linux", version: "5.4.0-42-generic", arch: "amd64", family: "unix"

    + release-cli -v
    version 0.6.0

    + sonar-scanner -v
    INFO: Scanner configuration file: /usr/local/sonar-scanner/conf/sonar-scanner.properties
    INFO: Project root configuration file: NONE
    INFO: SonarScanner 4.5.0.2216
    INFO: Java 1.8.0_281 Oracle Corporation (64-bit)
    INFO: Linux 5.4.0-42-generic amd64

    + kubectl version
    Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:30Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:01:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

    + docker --version
    Docker version 20.10.3, build 48d30b5b32e99c932b4ea3edca74353feddd83ff

    + sudo docker ps # 非常注意权限
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    bd9948326a78 harbor.weiyigeek.top/devops/jenkins-jnlp "/bin/sh -c '/usr/lo…" 39 seconds ago Up 32 seconds k8s_jnlp_kubernetes-jenkins-slave-68-91jbw-svtg2-3vq9b_devops_6d4bc0d2-b34f-46fc-9c9d-65537cb2bff8_0
    84538d2017c2 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 "/pause" About a minute ago Up 58 seconds k8s_POD_kubernetes-jenkins-slave-68-91jbw-svtg2-3vq9b_devops_6d4bc0d2-b34f-46fc-9c9d-65537cb2bff8_0

    Finished: SUCCESS
WeiyiGeek.custom-jenkins-jnlp

WeiyiGeek.custom-jenkins-jnlp