k8s-ocp-yaml
Search…
2019-11-13-openshift jenkins slave pod 自定义模板
比如: 配置maven pod的存储,将maven缓存文件保存在持久化存储上,以加快下次编译速度。 比如: 配置maven nodejs的时区,使输出日志时间准确。 比如: 配置slave pod的资源限额,避免编译过程出现OOM,因为有的应用编译过程是比较吃内存的。 比如: 配置多个slave pod,要支持jdk7,8,10,nodejs 6,8,10,12 等情况。。。

Kubernetes Plug-in

Kubernetes Plug-in 是一个动态生成 slave pod 的插件,用于运行jenkins job,job完成后销毁pod。
此插件已经集成在openshift jenkins中,相关默认配置在jenkins初始化过程会自动处理。
slave pod 默认内置了maven 和 nodejs
如果只是变更maven和nodejs的镜像,我们在openshift project下修改jenkins的template,在env 部分新加两个环境变量 MAVEN_SLAVE_IMAGE NODEJS_SLAVE_IMAGE
TZ 时区变量是为了日志时间准确
1
- name: JENKINS_SERVICE_NAME
2
value: ${JENKINS_SERVICE_NAME}
3
- name: JNLP_SERVICE_NAME
4
value: ${JNLP_SERVICE_NAME}
5
- name: ENABLE_FATAL_ERROR_LOG_FILE
6
value: ${ENABLE_FATAL_ERROR_LOG_FILE}
7
- name: MAVEN_SLAVE_IMAGE
8
value: harbor.test.bocloud.com/openshift/openshift3/jenkins-slave-maven-rhel7:v3.11.141-settings-oraclejdk678-skope
9
- name: NODEJS_SLAVE_IMAGE
10
value: harbor.test.bocloud.com/openshift/openshift3/jenkins-agent-nodejs-6-8-10-12-rhel7:v3.11.141
11
- name: TZ
12
value: Asia/Shanghai
Copied!

OpenShift Sync plug-in

如果要快速便捷自定义 slave pod,openshift 提供了 OpenShift Sync plug-in 插件,此插件默认已安装。
这个插件会在jenkins 启动时候查找project下的具备以下特性的imagestream或者configmap
  • imagestream 具备label role=jenkins-slave
  • imagestream tags 具备 annotation role=jenkins-slave
  • configmap 具备label role=jenkins-slave
具备相关label 的imagestream 或者具备相关annotation 的imagestream tag,会自动更新到kubernetes plugin 配置中,可以和自带的maven与nodejs slave pod一样,直接被jenkins job调用
configmap 方式提供了与kubernetes plugin配置一致的XML格式文件,通过这种方式可以配置kubernetes plugin 支持的所有格式。
kubernetes plugin 支持的参数见插件官方 https://github.com/jenkinsci/kubernetes-plugin
以下为我们在工作中使用到的configmap,配置了资源限额、环境变量、pvc持久化卷 https://raw.githubusercontent.com/cai11745/k8s-ocp-yaml/master/yaml-file/jenkins-agent-pod-template.yaml
1
kind: ConfigMap
2
apiVersion: v1
3
metadata:
4
name: jenkins-agent-pod-template
5
labels:
6
role: jenkins-slave
7
data:
8
template1: |-
9
<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
10
<inheritFrom></inheritFrom>
11
<name>maven</name>
12
<namespace></namespace>
13
<privileged>false</privileged>
14
<capOnlyOnAlivePods>false</capOnlyOnAlivePods>
15
<alwaysPullImage>false</alwaysPullImage>
16
<instanceCap>2147483647</instanceCap>
17
<slaveConnectTimeout>100</slaveConnectTimeout>
18
<idleMinutes>0</idleMinutes>
19
<activeDeadlineSeconds>0</activeDeadlineSeconds>
20
<label>maven</label>
21
<serviceAccount>jenkins</serviceAccount>
22
<nodeSelector></nodeSelector>
23
<nodeUsageMode>NORMAL</nodeUsageMode>
24
<customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
25
<workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
26
<memory>false</memory>
27
</workspaceVolume>
28
<volumes>
29
<org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
30
<mountPath>/home/jenkins/.m2/repository/</mountPath>
31
<claimName>maven-m2</claimName>
32
<readOnly>false</readOnly>
33
</org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
34
</volumes>
35
<containers>
36
<org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
37
<name>jnlp</name>
38
<image>harbor.test.geely.com/openshift/openshift3/jenkins-slave-maven-rhel7:v3.11.141-settings-oraclejdk678-skope</image>
39
<privileged>false</privileged>
40
<alwaysPullImage>true</alwaysPullImage>
41
<workingDir>/tmp</workingDir>
42
<command></command>
43
<args>${computer.jnlpmac} ${computer.name}</args>
44
<ttyEnabled>false</ttyEnabled>
45
<resourceRequestCpu>1</resourceRequestCpu>
46
<resourceRequestMemory>2G</resourceRequestMemory>
47
<resourceLimitCpu>1</resourceLimitCpu>
48
<resourceLimitMemory>2G</resourceLimitMemory>
49
<envVars>
50
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
51
<key>TZ</key>
52
<value>Asia/Shanghai</value>
53
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
54
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
55
<key>LC_ALL</key>
56
<value>en_US.UTF-8</value>
57
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
58
</envVars>
59
<ports/>
60
<livenessProbe>
61
<execArgs></execArgs>
62
<timeoutSeconds>0</timeoutSeconds>
63
<initialDelaySeconds>0</initialDelaySeconds>
64
<failureThreshold>0</failureThreshold>
65
<periodSeconds>0</periodSeconds>
66
<successThreshold>0</successThreshold>
67
</livenessProbe>
68
</org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
69
</containers>
70
<envVars/>
71
<annotations/>
72
<imagePullSecrets/>
73
<nodeProperties/>
74
<yaml></yaml>
75
<podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Default"/>
76
</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
77
template2: |-
78
<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
79
<inheritFrom></inheritFrom>
80
<name>nodejs</name>
81
<namespace></namespace>
82
<privileged>false</privileged>
83
<capOnlyOnAlivePods>false</capOnlyOnAlivePods>
84
<alwaysPullImage>false</alwaysPullImage>
85
<instanceCap>2147483647</instanceCap>
86
<slaveConnectTimeout>100</slaveConnectTimeout>
87
<idleMinutes>0</idleMinutes>
88
<activeDeadlineSeconds>0</activeDeadlineSeconds>
89
<label>nodejs</label>
90
<serviceAccount>jenkins</serviceAccount>
91
<nodeSelector></nodeSelector>
92
<nodeUsageMode>NORMAL</nodeUsageMode>
93
<customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
94
<workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
95
<memory>false</memory>
96
</workspaceVolume>
97
<volumes>
98
<org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
99
<mountPath>/home/jenkins/npm-cache</mountPath>
100
<claimName>npm-cache</claimName>
101
<readOnly>false</readOnly>
102
</org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
103
</volumes>
104
<containers>
105
<org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
106
<name>jnlp</name>
107
<image>harbor.test.geely.com/openshift/openshift3/jenkins-agent-nodejs-6-8-10-12-rhel7:v3.11.141-geely</image>
108
<privileged>false</privileged>
109
<alwaysPullImage>true</alwaysPullImage>
110
<workingDir>/tmp</workingDir>
111
<command></command>
112
<args>${computer.jnlpmac} ${computer.name}</args>
113
<ttyEnabled>false</ttyEnabled>
114
<resourceRequestCpu>1</resourceRequestCpu>
115
<resourceRequestMemory>3G</resourceRequestMemory>
116
<resourceLimitCpu>1</resourceLimitCpu>
117
<resourceLimitMemory>3G</resourceLimitMemory>
118
<envVars>
119
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
120
<key>TZ</key>
121
<value>Asia/Shanghai</value>
122
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
123
</envVars>
124
<ports/>
125
<livenessProbe>
126
<execArgs></execArgs>
127
<timeoutSeconds>0</timeoutSeconds>
128
<initialDelaySeconds>0</initialDelaySeconds>
129
<failureThreshold>0</failureThreshold>
130
<periodSeconds>0</periodSeconds>
131
<successThreshold>0</successThreshold>
132
</livenessProbe>
133
</org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
134
</containers>
135
<envVars/>
136
<annotations/>
137
<imagePullSecrets/>
138
<nodeProperties/>
139
<yaml></yaml>
140
<podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Default"/>
141
</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
Copied!
如果不知道XML参数怎么写,可以找一个jenkins,通过页面配置好slave pod的参数,然后进入jenkins容器,配置的参数保存在这里 /var/lib/jenkins/config.xml
导入configmap,sync 插件会把configmap内容自动同步到jenkins pod template,不需要重启jenkins
需要注意几点: 1. 原来的maven nodejs配置会被覆盖,因为configmap中使用了相同的name 2. 删除configmap,会将jenkins中的两个slave pod 同步删除 3. 不要直接在 jenkins 页面中修改 slave pod的参数,因为一重启就会被configmap重置 4. 如果后面不想被configmap同步,但是想保留配置,在jenkins页面把pod template 名称改掉,就是第一个maven 或者nodejs参数 ,然后把configmap 删了
jenkins-pod-template-changename.png