Jenkins 使用 Kubernetes Plugin 完成持續(xù)構(gòu)建與發(fā)布

icyfire 發(fā)布于2019-06-28 16:54 / 2377人閱讀

摘要：所有需要跑任務(wù)的通過模版動態(tài)創(chuàng)建，當(dāng)任務(wù)執(zhí)行結(jié)束自動刪除。同時也可以用配置完畢，可以點(diǎn)擊按鈕測試是否能夠連接的到，如果顯示則表示連接成功，配置沒有問題。

介紹

基于Kubernetes和Jenkins來實(shí)現(xiàn)CI/CD。所有需要跑任務(wù)的jenkins slave(pod)通過模版動態(tài)創(chuàng)建，當(dāng)任務(wù)執(zhí)行結(jié)束自動刪除。

系統(tǒng)整體架構(gòu)

job流程

環(huán)境

kubernets

jenkins配置

jenkins-deployment.yaml

apiVersion: "apps/v1beta1"
kind: "Deployment"
metadata:
  name: "jenkins"
  labels:
    name: "jenkins"
spec:
  replicas: 1
  template:
    metadata:
      name: "jenkins"
      labels:
        name: "jenkins"
    spec:
      containers:
        - name: jenkins
          image: jenkinsci/jenkins:2.154
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
          env:
            - name: TZ
              value: Asia/Shanghai
          ports:
          - containerPort: 8080 
            name: web
          - containerPort: 50000
            name: agent
      volumes:
        - name: jenkins-home
          nfs:
            path: "/nfs/jenkins/data"
            server: "cpu029.hogpu.cc"
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins

jenkins-account.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["configmap"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

jenkins-service.yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8080
      name: web
      targetPort: 8080
    - port: 50000
      name: agent
      targetPort: 50000
  selector:
    name: jenkins

說明

說明一下：這里 Service 我們暴漏了端口 8080 和 50000，8080 為訪問 Jenkins Server 頁面端口，50000 為創(chuàng)建的 Jenkins Slave 與 Master 建立連接進(jìn)行通信的默認(rèn)端口，如果不暴露的話，Slave 無法跟 Master 建立連接。這里使用 NodePort 方式暴漏端口，并未指定其端口號，由 Kubernetes 系統(tǒng)默認(rèn)分配，當(dāng)然也可以指定不重復(fù)的端口號（范圍在 30000~32767）

創(chuàng)建jenkins

接下來，通過 kubectl 命令行執(zhí)行創(chuàng)建 Jenkins Service。

$ kubectl create namespace kubernetes-plugin
$ kubectl config set-context $(kubectl config current-context) --namespace=kubernetes-plugin
$ kubectl create -f jenkins-deployment.yaml
$ kubectl create -f jenkins-account.yaml
$ kubectl create -f jenkins-service.yaml

ps：

創(chuàng)建一個新的 namespace 為 kubernetes-plugin，并且將當(dāng)前 context 設(shè)置為 kubernetes-plugin namespace 這樣就會自動切換到該空間下。

查看狀態(tài)

jianyu.tian@yz-gpu-k8s004 ~]$ kubectl get deployment,svc,pods
NAME             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/jenkins   1         1         1            1           1h

NAME          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
svc/jenkins   NodePort   10.106.235.91           8080:31051/TCP,50000:30545/TCP   2h

NAME                          READY     STATUS    RESTARTS   AGE
po/jenkins-64564fc5c9-pzlpb   1/1       Running   0          1h

ps：

Jenkins Master 服務(wù)已經(jīng)啟動起來了，并且將端口暴漏到 8080:31051，50000:30545，此時可以通過瀏覽器打開 http://:30645 訪問 Jenkins 頁面了。

jenkins web界面初始化

1.主要對jenkins-plugin插件做說明

安裝完畢后，點(diǎn)擊 “系統(tǒng)管理” —> “系統(tǒng)設(shè)置” —> “新增一個云” —> 選擇 “Kubernetes”，然后填寫 Kubernetes 和 Jenkins 配置信息。

ps：
Name 處默認(rèn)為 kubernetes，也可以修改為其他名稱，如果這里修改了，下邊在執(zhí)行 Job 時指定 podTemplate() 參數(shù) cloud 為其對應(yīng)名稱，否則會找不到，cloud 默認(rèn)值取：kubernetes
Kubernetes URL 處我填寫了 https://kubernetes.default.sv... 這里我填寫了 Kubernetes Service 對應(yīng)的 DNS 記錄，通過該 DNS 記錄可以解析成該 Service 的 Cluster IP，或者直接填寫外部 Kubernetes 的地址 https://:。
Jenkins URL 處我填寫了 http://jenkins.kubernetes-plugin:8080，跟上邊類似，也是使用 Jenkins Service 對應(yīng)的 DNS 記錄，不過要指定為 8080 端口，因?yàn)槲覀冊O(shè)置暴漏 8080 端口。同時也可以用 http://:

配置完畢，可以點(diǎn)擊 “Test Connection” 按鈕測試是否能夠連接的到 Kubernetes，如果顯示 Connection test successful 則表示連接成功，配置沒有問題。

測試

創(chuàng)建一個 Pipeline 類型 Job：

pipeline {
    agent any
  //并行操作
    stages {
        stage("test_all") {
            parallel {
                stage("python3-cuda9.2") {
                    agent {
                        kubernetes {
                            label "mxnet-python3-cuda9"
                            yaml """
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    name: "mxnet-python3-cuda9"
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: hobot.workas
            operator: In
            values:
            - gpu
          - key: kubernetes.io/nvidia-gpu-name
            operator: In
            values:
            - TITAN_V
  containers:
  - name: mxnetone
    image: docker.hobot.cc/dlp/mxnetci:runtime-py3.6-cudnn7.3-cuda9.2-centos7
    imagePullPolicy: Always
    resources:
      limits:
        nvidia.com/gpu: 1
"""
                }
            }
                   stages {
                        stage("拉取代碼") {
                            steps {
                                container("mxnetone") {
                                    checkout(
                                        [
                                            $class: "GitSCM", 
                                            branches: [[name: "nnvm"]], 
                                            browser: [$class: "Phabricator", repo: "rMXNET", repoUrl: ""], 
                                            doGenerateSubmoduleConfigurations: false, extensions: [[$class: "SubmoduleOption", disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: "", trackingSubmodules: false]],
                                            submoduleCfg: [],
                                            userRemoteConfigs: [[credentialsId: "zhaoming_private", url: ""]]
                                        ]
                                    )                   
                                }
                            }
                        }
                        stage("編譯") {
                            steps {
                                container("mxnetone") {
                                    sh """
                                    nvidia-smi
                                    source /root/.bashrc
                                    make deps
                                    echo -e "USE_PROFILER=1
USE_GLOG=0
USE_HDFS=0" >> ./make/config.mk
                                    sed -i "s#USE_CUDA_PATH = /usr/local/cuda-8.0#USE_CUDA_PATH = /usr/local/cuda-9.2#g" ./make/config.mk
                                    make lint
                                    make -j 12
                                    ln -s /home/data ./
                                    make test |  tee unittest.log
                                    """
                                }
                            }
                        }
                        stage("單元測試") {
                            steps {
                              container("mxnetone") {
                                    sh """
                                    cp -rf python/mxnet ./
                                    cp -f lib/libmxnet.so mxnet/
                                    echo "-------Running tests under Python3-------"
                                    python3 -V
                                    python3 `which nosetests` tests/python/train
                                    python3 `which nosetests` -v -d tests/python/unittest
                                       """
                                }
                            }
                        }
                    }
                }

                stage("python2-cuda9.2") {
                    agent {
                        kubernetes {
                            label "mxnet-python2-cuda9"
                            yaml """
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    name: "mxnet-python2-cuda9"
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: hobot.workas
            operator: In
            values:
            - gpu
          - key: kubernetes.io/nvidia-gpu-name
            operator: In
            values:
            - TITAN_V
  containers:
  - name: mxnettwo
    image: docker.hobot.cc/dlp/mxnetci:runtime-cudnn7.3-cuda9.2-centos7
    imagePullPolicy: Always
    resources:
      limits:
        nvidia.com/gpu: 1
"""
            }
        }
                     stages {
                        stage("拉取代碼") {
                            steps {
                                container("mxnettwo") {
                                    checkout(
                                        [
                                            $class: "GitSCM", 
                                            branches: [[name: "nnvm"]], 
                                            browser: [$class: "Phabricator", repo: "rMXNET", repoUrl: ""], 
                                            doGenerateSubmoduleConfigurations: false, extensions: [[$class: "SubmoduleOption", disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: "", trackingSubmodules: false]],
                                            submoduleCfg: [],
                                            userRemoteConfigs: [[credentialsId: "zhaoming_private", url: ""]]
                                        ]
                                    )
                                }
                            }
                        }
                        stage("編譯") {
                            steps {
                                container("mxnettwo") {
                                    sh """
                                    nvidia-smi
                                    pip2 install numpy==1.14.3 -i https://mirrors.aliyun.com/pypi/simple/
                                    source /root/.bashrc
                                    make deps
                                    echo -e "USE_PROFILER=1
USE_GLOG=0
USE_HDFS=0" >> ./make/config.mk
                                    sed -i "s#USE_CUDA_PATH = /usr/local/cuda-8.0#USE_CUDA_PATH = /usr/local/cuda-9.2#g" ./make/config.mk
                                    make lint
                                    make -j 12
                                    ln -s /home/data ./
                                    make test |  tee unittest.log
                                    """
                                }
                            }
                        }
                        stage("單元測試") {
                            steps {
                              container("mxnettwo") {
                                    sh """
                                    cp -rf python/mxnet ./
                                    cp -f lib/libmxnet.so mxnet/
                                    echo "-------Running tests under Python2-------"
                                    python2 -V
                                    python2 `which nosetests` tests/python/train
                                    python2 `which nosetests` -v -d tests/python/unittest
                                       """
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

文檔

https://github.com/jenkinsci/...