[TOC]
描述: 有了上一章的基础"4.Jenkins进阶之分布式架构环境配置"介绍，本章主要介绍Kubernetes 的插件使用实例, 以及自定义 Slave 的 Jenkins-Jnlp 容器镜像模板(重点)，在进行K8s动态节点生成时拉取使用，极大的节约了系统资源。
描述: 前面已经对Jenkins集成Kubernetes进行部署和配置，本章将进行介绍与实践如何使用该插件以及帮助文档的记录;
说明: Kubernetes-plugin 它是在Kubernetes集群中运行动态代理的Jenkins插件, 该插件为每个启动的代理创建一个Kubernetes Pod，由Docker映像定义运行，并在每次构建后停止。
备注: 代理是使用JNLP启动的，因此预期图像会自动连接到Jenkins主机，以下是重要的环境变量（我们说过不建议使用自己jnlp覆盖默认的jnlp容器）。
[TOC]
描述: 有了上一章的基础"4.Jenkins进阶之分布式架构环境配置"介绍，本章主要介绍Kubernetes 的插件使用实例, 以及自定义 Slave 的 Jenkins-Jnlp 容器镜像模板(重点)，在进行K8s动态节点生成时拉取使用，极大的节约了系统资源。
描述: 前面已经对Jenkins集成Kubernetes进行部署和配置，本章将进行介绍与实践如何使用该插件以及帮助文档的记录;
说明: Kubernetes-plugin 它是在Kubernetes集群中运行动态代理的Jenkins插件, 该插件为每个启动的代理创建一个Kubernetes Pod，由Docker映像定义运行，并在每次构建后停止。
备注: 代理是使用JNLP启动的，因此预期图像会自动连接到Jenkins主机，以下是重要的环境变量（我们说过不建议使用自己jnlp覆盖默认的jnlp容器）。1
2
3
4
5# jenkins/inbound-agent
JENKINS_URL: Jenkins web interface url
JENKINS_SECRET: the secret key for authentication
JENKINS_AGENT_NAME: the name of the Jenkins agent
JENKINS_NAME: the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)
该镜像是Jenkins自定义jnlp容器模板，主要用于Jenkins工作节点容器化使用，主要包含Gitlab_release发布、 docker 容器管理、kubectl 集群管理功能。我们可以直接在docker 的环境中 docker pull weiyigeek/alpine-jenkins-jnlp下来
1.先决条件
2.填写Kubernetes插件配置
3.支持的凭证
描述: podTemplate 是用于创建代理的pod的模板可以通过用户界面或管道进行配置, 无论哪种方式它都可以访问以下字段：
envVar 环境变量其值是内联定义的, secretEnvVar 一个环境变量其值是从Kubernetes机密派生的。)Can be 'never()', 'onFailure()', 'always()', or 'default()'),如果为空则按照下面的active Deadline Seconds参数进行。20) runAsUser : 用户ID，用于将pod中的所有容器运行为。
21) runAsGroup : 组ID，用于将Pod中的所有容器运行为。
描述: containerTemplate 是一个将被添加到pod中的容器模板它属于container中数组对象。同样它可以通过用户界面或管道进行配置，并允许你设置以下字段:
containerTemplate - 容器模板
补充和覆盖在pod级别设置的env var）。Liveness Probe Usage1
2
3
4
5
6
7
8
9
10
11
12
13
14containerTemplate(
name: 'busybox',
image: 'busybox',
ttyEnabled: true,
command: 'cat',
livenessProbe: containerLivenessProbe(
execArgs: 'some --command',
initialDelaySeconds: 30,
timeoutSeconds: 1,
failureThreshold: 3,
periodSeconds: 10,
successThreshold: 1
)
)
Tips : 默认情况下代理连接超时设置为100秒。在某些情况下您想要设置一个不同的值，如果可以可以将system属性设置org.csanchez.jenkins.plugins.kubernetes.PodTemplate.connectionTimeout为一个不同的值
描述： Scripted Pipeline 示例参考:
(1) containerTemplate1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67podTemplate(
cloud: 'kubernetes',
name: 'jenkins-slave',
namespace: 'devops',
label: 'k8s-slave',
// 容器及容器模板定义
containers: [
containerTemplate(
name: 'maven',
image: 'maven:3.6-jdk-8-alpine',
ttyEnabled: true,
command: 'cat',
privileged: false,
alwaysPullImage: false,
workingDir: '/home/jenkins/agent',
resourceRequestCpu: '100m',
resourceLimitCpu: '500m',
resourceRequestMemory: '100Mi',
resourceLimitMemory: '500Mi',
envVars: [
envVar(key: 'MYSQL_ALLOW_EMPTY_PASSWORD', value: 'true')
//, secretEnvVar(key: 'MYSQL_PASSWORD', secretName: 'mysql-secret', secretKey: 'password')
]
//,ports: [portMapping(name: 'mysql', containerPort: 3306, hostPort: 3306)]
)
// ),
// containerTemplate(
// name: 'jnlp',
// image: 'registry.cn-hangzhou.aliyuncs.com/google-containers/jnlp-slave:alpine',
// args: '${computer.jnlpmac} ${computer.name}',
// command: ''
// )
]
,volumes: [
hostPathVolume(hostPath: '/nfsdisk-31/appstorage/mavenRepo', mountPath: '/home/jenkins'),
hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock'),
/*
emptyDirVolume(mountPath: '/etc/mount1', memory: false),
secretVolume(mountPath: '/etc/mount2', secretName: 'my-secret'),
configMapVolume(mountPath: '/etc/mount3', configMapName: 'my-config'),
hostPathVolume(mountPath: '/etc/mount4', hostPath: '/mnt/my-mount'),
nfsVolume(mountPath: '/etc/mount5', serverAddress: '127.0.0.1', serverPath: '/', readOnly: true),
persistentVolumeClaim(mountPath: '/home/jenkins', claimName: 'jenkins', readOnly: false),
*/
],
annotations: [
podAnnotation(key: "maven-pod", value: "Kubernetes-jenkins-Test")
],
showRawYaml: 'flase'
//,defaultContainer: 'maven' # scripted pipeline 不支持该参数
//,imagePullSecrets: [ 'pull-secret' ]
)
{
// 注意此次node中则为Pod模板名称
node ('k8s-slave') {
// container 中指定Pod中容器名称
container('maven') {
stage('maven') {
container('maven') {
sh "hostname && ip addr"
sh ("mvn -version")
sh "env"
}
}
}
}
}
运行结果:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129# Still waiting to schedule task
# Waiting for next available executor on 'jenkins-slave-09vf0-sf3k6'
# 等待下一个可用的执行器启动
Created Pod: kubernetes devops/jenkins-slave-rtjxx-q3nhn
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Scheduled] Successfully assigned devops/jenkins-slave-rtjxx-q3nhn to work-223
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Pulled] Container image "maven:3.6-jdk-8-alpine" already present on machine
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Created] Created container maven | Started container maven
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Pulled] Container image "jenkins/inbound-agent:4.3-4" already present on machine
[Normal][devops/jenkins-slave-rtjxx-q3nhn][Created] Created container jnlp | Started container jnlp
Agent jenkins-slave-rtjxx-q3nhn is provisioned from template jenkins-slave-rtjxx
# ----------------------------------
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
maven-pod: "Kubernetes-jenkins-Test" # 我们添加的注解
buildUrl: "http://jenkins.devops.svc.cluster.local:8080/job/Kubernetes-jenkins-slave/23/"
runUrl: "job/Kubernetes-jenkins-slave/23/"
labels:
jenkins: "slave"
jenkins/label-digest: "823e2bb3db243573786b7a360a483f1acc7a0f96"
jenkins/label: "k8s-slave"
name: "jenkins-slave-rtjxx-q3nhn"
spec:
containers:
- command:
- "cat" # 运行的命令
env:
- name: "MYSQL_ALLOW_EMPTY_PASSWORD" # 运行的环境变量
value: "true"
image: "maven:3.6-jdk-8-alpine"
imagePullPolicy: "IfNotPresent" # 镜像拉取策略
name: "maven" # 容器名称
resources:
limits:
memory: "500Mi"
cpu: "500m"
requests:
memory: "100Mi"
cpu: "100m"
tty: true
volumeMounts: # 卷挂载情况
- mountPath: "/home/jenkins"
name: "volume-0"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-1"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
- env: # jnlp 环境变量
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_AGENT_NAME"
value: "jenkins-slave-rtjxx-q3nhn"
- name: "JENKINS_NAME"
value: "jenkins-slave-rtjxx-q3nhn"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.devops.svc.cluster.local:8080/"
image: "jenkins/inbound-agent:4.3-4"
name: "jnlp" # jenkins-jnlp 容器的名称
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins"
name: "volume-0"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-1"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- hostPath:
path: "/nfsdisk-31/appstorage/mavenRepo"
name: "volume-0"
- hostPath:
path: "/var/run/docker.sock"
name: "volume-1"
- emptyDir:
medium: ""
name: "workspace-volume"
# ------------------------------
+ hostname
# jenkins-slave-rtjxx-q3nhn
+ ip addr
# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
# link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
# inet 127.0.0.1/8 scope host lo
# valid_lft forever preferred_lft forever
# 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
# link/ipip 0.0.0.0 brd 0.0.0.0
# 4: eth0@if162: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue state UP
# link/ether de:c6:df:23:e1:7f brd ff:ff:ff:ff:ff:ff
# inet 172.16.24.220/32 brd 172.16.24.220 scope global eth0
# valid_lft forever preferred_lft forever
+ mvn -version
# Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
# Maven home: /usr/share/maven
# Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
# Default locale: en_US, platform encoding: UTF-8
# OS name: "linux", version: "5.4.0-42-generic", arch: "amd64", family: "unix"
+ env
# JENKINS_HOME=/var/jenkins_home
# POD_CONTAINER=maven
# STAGE_NAME=maven
# KUBERNETES_SERVICE_PORT=443
# JAVA_VERSION=8u212
# MAVEN_HOME=/usr/share/maven
# MAVEN_CONFIG=/root/.m2
Tips : 在PodTemplate模板中设置 showRawYaml: 'flase' 则将不会显示资源清单在日志中;
weiyigeek.top-脚本式流水线k8s插件使用
Tips : 要使用 secretEnvVar 我们必须在 Dashboard -> 凭据 -> 系统 -> 全局凭据 (unrestricted) 进行添加一个Secret-text凭据(未能成功复现);1
Secret text 4fc93636-d8f1-4051-8011-d852873fc740 mysql-secret Secret text mysql-secret
描述: 声明式管道支持需要Jenkins 2.66+, 在Declarative Pipeline使用的 kubernetes 选项说明;1
options are [activeDeadlineSeconds, cloud, containerTemplate, containerTemplates, customWorkspace, defaultContainer, idleMinutes, inheritFrom, instanceCap, label, namespace, nodeSelector, podRetention, serviceAccount, slaveConnectTimeout, supplementalGroups, workingDir, workspaceVolume, yaml, yamlFile, yamlMergeStrategy]
示例1.通过yaml文件资源清单创建1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40pipeline {
agent {
kubernetes {
cloud 'kubernetes'
namespace 'devops'
workingDir '/home/jenkins/agent'
// 继承模板
inheritFrom 'jenkins-slave'
defaultContainer 'maven'
// yamlFile 'KubernetesPod.yaml'
yaml """\
apiVersion:
kind: Pod
metadata:
labels:
jenkins: "slave"
jenkins/label: 'k8s-slave'
spec:
containers:
- name: 'maven'
image: 'maven:3.6-jdk-8-alpine'
imagePullPolicy: 'IfNotPresent' # 镜像拉取策略
command:
- cat
tty: true
""".stripIndent()
}
}
stages {
stage ('declarative Pipeline - kubernetes') {
steps {
echo "declarative Pipeline - kubernetes"
container ('maven') {
sh "echo ${POD_CONTAINER}"
sh "mvn -version"
}
}
}
}
}
执行结果:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
# 'kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8' is offline
# Agent kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8 is provisioned from template Kubernetes-jenkins-slave_49-fcd1s-m3n3q
---
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins.devops.svc.cluster.local:8080/job/Kubernetes-jenkins-slave/49/"
runUrl: "job/Kubernetes-jenkins-slave/49/"
labels:
jenkins: "slave"
jenkins/label: "Kubernetes-jenkins-slave_49-fcd1s"
jenkins/label-digest: "95508e0ca05226ba84989959dc886ad4011cf125"
name: "kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8"
spec:
containers:
- command:
- "cat"
image: "maven:3.6-jdk-8-alpine"
imagePullPolicy: "IfNotPresent"
name: "maven"
tty: true
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_AGENT_NAME"
value: "kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8"
- name: "JENKINS_NAME"
value: "kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.devops.svc.cluster.local:8080/"
image: "jenkins/inbound-agent:4.3-4"
name: "jnlp"
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins/.m2"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
hostNetwork: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- hostPath:
path: "/nfsdisk-31/appstorage/mavenRepo"
name: "volume-0"
- emptyDir:
medium: ""
name: "workspace-volume"
Running on kubernetes-jenkins-slave-49-fcd1s-m3n3q-t33d8 in /home/jenkins/agent/workspace/Kubernetes-jenkins-slave
# [Pipeline] {
# [Pipeline] container
# [Pipeline] {
# [Pipeline] stage
# [Pipeline] { (declarative Pipeline - kubernetes)
[Pipeline] echo
declarative Pipeline - kubernetes
# [Pipeline] container
# [Pipeline] {
# [Pipeline] sh
+ echo maven
maven
# [Pipeline] sh
+ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-42-generic", arch: "amd64", family: "unix"
# [Pipeline] }
# [Pipeline] // container
# [Pipeline] }
# [Pipeline] // stage
# [Pipeline] }
# [Pipeline] // container
# [Pipeline] }
# [Pipeline] // node
# [Pipeline] }
# [Pipeline] // podTemplate
# [Pipeline] End of Pipeline
Finished: SUCCESS
Tips :默认情况下在容器中运行步骤, 步骤将嵌套在隐式container（name）{…}块中，而不是在jnlp容器中执行。1
defaultContainer 'maven'
Tips : 您也可以使用yamlFile将pod模板保存在单独的KubernetesPod.yaml文件中1
2
3
4
5
6
7
8
9
10pipeline {
agent {
kubernetes {
yamlFile 'KubernetesPod.yaml'
}
}
stages {
...
}
}
描述: 下述相关脚本以及文件下载地址请在WeiyiGeek微信公众号回复jenkins-jnlp-dockerfile关键字获取;
自定义的jnlp容器模板主要实现功能:
Tips: 在K8s集群中测试alpine镜像执行相应的安装(需要在Alpine调试新安装软件时使用):1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: alpine-app
spec:
containers:
- name: alpine-app
image: alpine:latest
args:
- sleep
- "100000"
EOF
pod/alpine-app created
$ kubectl get pods -o wide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# alpine-app 1/1 Running 0 116s 10.100.37.194 worker-02 <none> <none>
Dockerfile
自定义jnlp容器模板镜像地址: https://hub.docker.com/r/weiyigeek/alpine-jenkins-jnlp
该镜像是Jenkins自定义jnlp容器模板，主要用于Jenkins工作节点容器化使用，主要包含Gitlab_release发布、 docker 容器管理、kubectl 集群管理功能。我们可以直接在docker 的环境中 docker pull weiyigeek/alpine-jenkins-jnlp下来
说明: 镜像构建文件1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133#----------------------------------------------------------------------#
# Title: Base in Alpine Images Create Custom Jenkins Kubernetes jnlp Images
# Author: WeiyiGeek
# WebSite: https://weiyigeek.top
# MainFunction:
# Install ssh-server docker git openssh tzdata curl tar sudo git ca-certificates wget unzip docker zlib nodejs npm jq
# Install JDK8
# - https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html
# - https://github.com/sgerrand/alpine-pkg-glibc/releases/
# - https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
# Install jnlp
# - https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/
# Install Maven
# - https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
# Install SonarqubeScan
# - https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.5.0.2216-linux.zip
# Install Gitlab Release Version: 0.0.6
# - https://gitlab.com/gitlab-org/release-cli/-/releases
# Install kubernetes cli
# - kubectl Version: 1.19.3
# Version: alpine-3.13.1
# Release: v1.10
# ChangeLog:
# v1.9 - 增加 中文环境
# v1.10 - 增加 node.js 环境支持
#-------------------------------------------------#
FROM alpine:3.13.1
MAINTAINER Jenkins Custom Work Node Jnlp Container - <master@weiyigeek.top> - WeiyiGeek
ARG USERNAME=jenkins \
AGENT_WORKDIR=/home/jenkins \
BASE_DIR=/usr/local \
BASE_BIN=/usr/local/bin \
BASE_URL=http://192.168.12.107:8080 \
LOCALE=locale.md \
JDK_NAME=jdk-8u281-linux-x64 \
JDK_DIR=/usr/local/jdk1.8.0_281 \
GLIBC_NAME=glibc-2.32-r0.apk \
GLIBC_BIN_NAME=glibc-bin-2.32-r0.apk \
GLIBC_I18N_NAME=glibc-i18n-2.32-r0.apk \
MAVEN_NAME=apache-maven-3.6.3-bin \
MAVEN_DIR=/usr/local/apache-maven-3.6.3 \
SONAR_SCANNER_NAME=sonar-scanner-cli-4.5.0.2216-linux \
SONAR_SCANNER_DIR=/usr/local/sonar-scanner-4.5.0.2216-linux \
GITLAB_CLI=release-cli-0.6.0-linux-amd64
ENV LANG=en_US.UTF-8 \
LC_ALL=en_US.UTF-8 \
JAVA_HOME=/usr/local/jdk8 \
JRE_HOME=/usr/local/jdk8/jre \
MAVEN_HOME=/usr/local/maven \
MAVEN_RPEO=/home/jenkins/.m2 \
SONAR_SCANNER_HOME=/usr/local/sonar-scanner \
NODEJS_MODULES=/usr/lib/node_modules
# 用户ROOT切换
USER root
# Shell 命令 - 此种方式极大减少了构建的镜像大小;
# 请根据网络情况选择 Alpine 镜像源 mirrors.aliyun.com 、mirror.tuna.tsinghua.edu.cn
RUN sed -i 's/dl-cdn.alpinelinux.org/mirror.tuna.tsinghua.edu.cn/g' /etc/apk/repositories \
&& apk update \
&& apk add --no-cache openssh tzdata curl tar sudo git ca-certificates wget unzip docker zlib nodejs npm jq \
&& cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& chmod 4755 /bin/busybox \
&& addgroup -g 1000 -S ${USERNAME} \
&& adduser ${USERNAME} -D -g ${USERNAME} -G root -u 1000 -s /bin/sh \
&& echo "jenkins ALL=(root) NOPASSWD:ALL" >> /etc/sudoers \
&& mkdir -p ${AGENT_WORKDIR}/agent ${AGENT_WORKDIR}/.ssh ${AGENT_WORKDIR}/.m2 \
&& wget -q -O /tmp/${GLIBC_NAME} ${BASE_URL}/${GLIBC_NAME} \
&& wget -q -O /tmp/${GLIBC_BIN_NAME} ${BASE_URL}/${GLIBC_BIN_NAME} \
&& wget -q -O /tmp/${GLIBC_I18N_NAME} ${BASE_URL}/${GLIBC_I18N_NAME} \
&& wget -q -O /etc/apk/keys/sgerrand.rsa.pub ${BASE_URL}/sgerrand.rsa.pub \
&& wget -q -O /tmp/${LOCALE} ${BASE_URL}/${LOCALE} \
&& wget -q -O /tmp/${JDK_NAME}.tar.gz ${BASE_URL}/${JDK_NAME}.tar.gz \
&& wget -q -O ${BASE_BIN}/agent.jar ${BASE_URL}/agent.jar \
&& curl -fsSL -o ${BASE_BIN}/jenkins-agent.sh ${BASE_URL}/jenkins-agent.sh \
&& curl -fsSL -o /tmp/${MAVEN_NAME}.tar.gz ${BASE_URL}/${MAVEN_NAME}.tar.gz \
&& curl -fsSL -o /tmp/${SONAR_SCANNER_NAME}.zip ${BASE_URL}/${SONAR_SCANNER_NAME}.zip \
&& curl -fsSL -o /usr/local/bin/release-cli ${BASE_URL}/${GITLAB_CLI} \
&& curl -fsSL -o /usr/local/bin/kubectl ${BASE_URL}/kubectl \
&& sed -i "s/#PermitRootLogin.*/PermitRootLogin yes/g" /etc/ssh/sshd_config \
&& sed -i "s/^#\s*StrictHostKeyChecking ask/StrictHostKeyChecking no/g" /etc/ssh/ssh_config \
&& ssh-keygen -t dsa -P "" -f /etc/ssh/ssh_host_dsa_key \
&& ssh-keygen -t rsa -P "" -f /etc/ssh/ssh_host_rsa_key \
&& ssh-keygen -t ecdsa -P "" -f /etc/ssh/ssh_host_ecdsa_key \
&& ssh-keygen -t ed25519 -P "" -f /etc/ssh/ssh_host_ed25519_key \
&& ssh-keygen -t ed25519 -P "" -C "master@weiyigeek.top" -f /home/jenkins/.ssh/id_ed25519 \
&& apk add /tmp/${GLIBC_NAME} /tmp/${GLIBC_BIN_NAME} /tmp/${GLIBC_I18N_NAME} \
&& tar -zxf /tmp/${JDK_NAME}.tar.gz -C ${BASE_DIR} \
&& mv ${JDK_DIR} ${JAVA_HOME} \
&& tar -zxf /tmp/${MAVEN_NAME}.tar.gz -C ${BASE_DIR} \
&& mv ${MAVEN_DIR} ${MAVEN_HOME} \
&& unzip /tmp/${SONAR_SCANNER_NAME}.zip -d ${BASE_DIR} \
&& mv ${SONAR_SCANNER_DIR} ${SONAR_SCANNER_HOME} \
&& npm config set registry https://registry.npmmirror.com \
&& chmod a+x ${BASE_BIN}/* \
&& chown -R jenkins:jenkins ${BASE_DIR}/ ${AGENT_WORKDIR}/ ${NODEJS_MODULES}/ \
&& echo "root:WeiyiGeek" | chpasswd \
&& echo "jenkins:WeiyiGeek" | chpasswd \
&& cat /tmp/${LOCALE} | xargs -i /usr/glibc-compat/bin/localedef -i {} -f UTF-8 {}.UTF-8 \
&& sed -i "s#use_embedded_jre=true#use_embedded_jre=false#g" ${SONAR_SCANNER_HOME}/bin/sonar-scanner \
&& rm -rf /var/cache/apk/* /tmp/* ${SONAR_SCANNER_HOME}/jre/* \
&& cd ${JAVA_HOME} \
&& rm -rf COPYRIGHT LICENSE README release THIRDPARTYLICENSEREADME-JAVAFX.txt THIRDPARTYLICENSEREADME.txt Welcome.html javafx-src.zip src.zip \
lib/plugin.jar \
lib/ext/jfxrt.jar \
bin/javaws \
lib/javaws.jar \
lib/desktop \
plugin \
lib/deploy* \
lib/*javafx* \
lib/*jfx* \
lib/amd64/libdecora_sse.so \
lib/amd64/libprism_*.so \
lib/amd64/libfxplugins.so \
lib/amd64/libglass.so \
lib/amd64/libgstreamer-lite.so \
lib/amd64/libjavafx*.so \
lib/amd64/libjfx*.so
&& echo "export LANG=zh_CN.UTF-8" > /etc/profile.d/locale.sh
USER jenkins
WORKDIR ${AGENT_WORKDIR}
ENV CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar \
PATH=${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${SONAR_SCANNER_HOME}/bin:$PATH
ENTRYPOINT ["/usr/local/bin/jenkins-agent.sh"]
local.md
描述：让alpine操作系统中文字符支持;1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24tee local.md <<'END'
en_AG
en_AU
en_BW
en_CA
en_DK
en_GB
en_HK
en_IE
en_IN
en_NG
en_NZ
en_PH
en_SG
en_US
en_ZA
en_ZM
en_ZW
zh_CN
zh_HK
zh_SG
zh_TW
zu_ZA
END
jenkins-agent.sh
描述: 容器启动时与jenkins利用Java jnlp 进行连接的脚本;1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91#!/bin/sh
# Usage jenkins-agent.sh [options] -url http://jenkins [SECRET] [AGENT_NAME]
# Optional environment variables :
# * JENKINS_TUNNEL : HOST:PORT for a tunnel to route TCP traffic to jenkins host, when jenkins can't be directly accessed over network
# * JENKINS_URL : alternate jenkins URL
# * JENKINS_SECRET : agent secret, if not set as an argument
# * JENKINS_AGENT_NAME : agent name, if not set as an argument
# * JENKINS_AGENT_WORKDIR : agent work directory, if not set by optional parameter -workDir
# * JENKINS_WEB_SOCKET: true if the connection should be made via WebSocket rather than TCP
# * JENKINS_DIRECT_CONNECTION: Connect directly to this TCP agent port, skipping the HTTP(S) connection parameter download.
# Value: "<HOST>:<PORT>"
# * JENKINS_INSTANCE_IDENTITY: The base64 encoded InstanceIdentity byte array of the Jenkins master. When this is set,
# the agent skips connecting to an HTTP(S) port for connection info.
# * JENKINS_PROTOCOLS: Specify the remoting protocols to attempt when instanceIdentity is provided.
if [ $# -eq 1 ]; then
# if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
exec "$@"
else
# if -tunnel is not provided, try env vars
case "$@" in
*"-tunnel "*) ;;
*)
if [ ! -z "$JENKINS_TUNNEL" ]; then
TUNNEL="-tunnel $JENKINS_TUNNEL"
fi ;;
esac
# if -workDir is not provided, try env vars
if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
case "$@" in
*"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
*)
WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
esac
fi
if [ -n "$JENKINS_URL" ]; then
URL="-url $JENKINS_URL"
fi
if [ -n "$JENKINS_NAME" ]; then
JENKINS_AGENT_NAME="$JENKINS_NAME"
fi
if [ "$JENKINS_WEB_SOCKET" = true ]; then
WEB_SOCKET=-webSocket
fi
if [ -n "$JENKINS_PROTOCOLS" ]; then
PROTOCOLS="-protocols $JENKINS_PROTOCOLS"
fi
if [ -n "$JENKINS_DIRECT_CONNECTION" ]; then
DIRECT="-direct $JENKINS_DIRECT_CONNECTION"
fi
if [ -n "$JENKINS_INSTANCE_IDENTITY" ]; then
INSTANCE_IDENTITY="-instanceIdentity $JENKINS_INSTANCE_IDENTITY"
fi
# if java home is defined, use it
JAVA_BIN="java"
if [ "$JAVA_HOME" ]; then
JAVA_BIN="$JAVA_HOME/bin/java"
fi
# if both required options are defined, do not pass the parameters
OPT_JENKINS_SECRET=""
if [ -n "$JENKINS_SECRET" ]; then
case "$@" in
*"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
*)
OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
esac
fi
OPT_JENKINS_AGENT_NAME=""
if [ -n "$JENKINS_AGENT_NAME" ]; then
case "$@" in
*"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
*)
OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
esac
fi
#TODO: Handle the case when the command-line and Environment variable contain different values.
#It is fine it blows up for now since it should lead to an error anyway.
exec $JAVA_BIN $JAVA_OPTS -cp /usr/local/bin/agent.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $WEB_SOCKET $DIRECT $PROTOCOLS $INSTANCE_IDENTITY $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
fi
镜像构建所需软件
备注: 在Jenkins 2.277版本中添加一个新的节点中获取匹配当前版本的 agent.jar, 或者是在 jenkins 官网 https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting 进行下载;1
2
3~/k8s/jenkins/jnlp-slave$ ls
agent.jar build jdk-8u281-linux-x64.tar.gz Jenkins.zip release-cli-0.6.0-linux-amd64 sonar-scanner-cli-4.5.0.2216-linux.zip
apache-maven-3.6.3-bin.tar.gz glibc-2.32-r0.apk jenkins-agent.sh kubectl sgerrand.rsa.pub
操作流程1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35# (0) 启动一个临时的web服务器（`存放上面的镜像构建所需软件`-非常重要-否则将会导致构建失败）
~/k8s/jenkins/jnlp-slave$ python3 -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /glibc-2.32-r0.apk HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /glibc-bin-2.32-r0.apk HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /glibc-i18n-2.32-r0.apk HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /sgerrand.rsa.pub HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /locale.md HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /jdk-8u281-linux-x64.tar.gz HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /agent.jar HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /jenkins-agent.sh HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /apache-maven-3.6.3-bin.tar.gz HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /sonar-scanner-cli-4.5.0.2216-linux.zip HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /release-cli-0.6.0-linux-amd64 HTTP/1.1" 200 -
# 172.17.0.4 - - [30/Mar/2021 17:32:00] "GET /kubectl HTTP/1.1" 200 -
# (1) 镜像构建 (建议带上构建平台)
docker build -t weiyigeek/alpine-jenkins-jnlp --platform linux/amd64 .
# Successfully built b47b6581b712
# Successfully tagged weiyigeek/alpine-jenkins-jnlp
# (2) 查看构建信息
docker images weiyigeek/alpine-jenkins-jnlp --all
# REPOSITORY TAG IMAGE ID CREATED SIZE
# harbor.weiyigeek.top/devops/jenkins-jnlp 3.13.1-alpine b47b6581b712 About an hour ago 715MB
# (3) 推送镜像
docker push weiyigeek/alpine-jenkins-jnlp
# The push refers to repository [harbor.weiyigeek.top/devops/jenkins-jnlp]
# 059ea3fbd3b3: Pushed
# 1119ff37d4a9: Layer already exists
# 3.13.1-alpine: digest: sha256:1f869c553340c9399c7db9072169600a17ddb0ec41d41d3a4365b8c1571fc201 size: 741
# (4) 采用Ansible各节点拉取镜像
ansible node -m shell -a "docker pull weiyigeek/alpine-jenkins-jnlp"
weiyigeek.top-DockerFile构建自定义Jnlp容器
描述: 此处还是利用Jenkins的kubernetes插件，让jenkins在进job时动态的生成工作节点(Pod)，并在结束后销毁容器。
Step 1.创建流水线 Kubernetes-jenkins-slave Job 在流水线中采用Pipeline Script脚本
1 | pipeline { |
Step 2.配置Pod Templates模板如下因为我们需要继承一哈
1 | 名称：jenkins-slave |
Step 3.在BlueOcen中运行并查看结果
1 | # (1) 可以看见当进行调度时k8s会动态的拉取镜像并运行，当任务结束后会自动销毁Pod |
weiyigeek.top-custom-jenkins-jnlp
1 | #----------------------------------------------------------------------# |
你好看友,欢迎关注博主微信公众号哟! ❤
这将是我持续更新文章的动力源泉，谢谢支持！(๑′ᴗ‵๑)
温馨提示: 未解锁的用户不能粘贴复制文章内容哟!
方式1.请访问本博主的B站【WeiyiGeek】首页关注UP主,
将自动随机获取解锁验证码。
Method 2.Please visit 【My Twitter】. There is an article verification code in the homepage.
方式3.扫一扫下方二维码，关注本站官方公众号
回复：验证码
将获取解锁(有效期7天)本站所有技术文章哟!
@WeiyiGeek - 为了能到远方，脚下的每一步都不能少
欢迎各位志同道合的朋友一起学习交流，如文章有误请在下方留下您宝贵的经验知识，个人邮箱地址【master#weiyigeek.top】或者个人公众号【WeiyiGeek】联系我。
更多文章来源于【WeiyiGeek Blog - 为了能到远方，脚下的每一步都不能少】, 个人首页地址( https://weiyigeek.top )
专栏书写不易，如果您觉得这个专栏还不错的，请给这篇专栏 【点个赞、投个币、收个藏、关个注、转个发、赞个助】，这将对我的肯定，我将持续整理发布更多优质原创文章！。
最后更新时间：
文章原始路径：_posts/系统运维/自动化运维/CI-CD/Jenkins/5.Jenkins进阶之分布式架构扩充知识.md
转载注明出处，原文地址：https://blog.weiyigeek.top/2020/12-31-533.html
本站文章内容遵循 知识共享 署名 - 非商业性 - 相同方式共享 4.0 国际协议