Docker Gitlab+Jenkins+Harbor builds a persistent platform operation

CI/CD Overview

CI workflow design

Git code version management system can only be managed by command line

Gitlab has developed a graphical management page based on git. Enterprises use gitlab to manage private code repositories

Github public code management repository

Building gitlab

To build gitlab, create a working directory first, because some data needs to be persisted

[root@www ~]# mkdir -p /gitlab
[root@www ~]# cd /gitlab/
 
docker run -d \
 --name gitlab \
 -p 8443:443 \
 -p 9999:80 \
 -p 9998:22 \
 -v /gitlab/config:/etc/gitlab \
 -v /gitlab/logs:/var/log/gitlab \
 -v /gitlab/data:/var/opt/gitlab \
 -v /etc/localtime:/etc/localtime \
 --restart=always \
 lizhenliang/gitlab-ce-zh:latest

9999 is the web port 8443 web htpps port 9998 ssh port

Persist the containerized data to the local host. This image was built after the early Chinese translation. The default official image is in English. This one is in English. This image is relatively large, about 1G, including databases and message queues, and there are a lot of encapsulated content.

[root@www gitlab]# docker logs 3396d5ccc518 
  - execute /opt/gitlab/bin/gitlab-ctl start postgresql
+psql_host='/var/opt/gitlab/postgresql'

Through the log, you can see which components such as postgresql

The first visit may be a bit slow because there are many components and you may have to wait for 3-5 minutes

The appearance of this page indicates that the gitlab service is started, but other components may not have started successfully. It is best to give 4G physical memory here. If there is only 2G, it cannot start normally. After about eight minutes, it will be as follows

The default username is root. You need to set a new password here. The new password should be at least 8 characters long and can be set to qwerasdf. After updating the password, you can log in with the root username and password qwerasdf.

Create a private project java-demo

Push code to the java-demo repository

Install git on another machine and upload the source code package of the project

[root@localhost ~]# yum install git -y

[root@localhost ~]# unzip tomcat-java-demo-master.zip

Initialize Git Directory

[root@localhost ~]# cd tomcat-java-demo-master
[root@localhost tomcat-java-demo-master]# git init
Initialized empty Git repository in /root/tomcat-java-demo-master/.git/

This git directory will configure some git-related data. After initialization is completed

[root@localhost tomcat-java-demo-master]# cd .git/
[root@localhost .git]# ls
branches config description HEAD hooks info objects refs

Configure git configuration file

This command writes the address of gitlab to the .git/config configuration file

[root@localhost tomcat-java-demo-master]# git remote add origin http://192.168.179.100:9999/root/java-demo.git
 
[root@localhost tomcat-java-demo-master]# cat .git/config You can see that the gitlab address is written in. When you submit the code later, you will read the configuration file [core]
 repositoryformatversion = 0
 filemode = true
 bare = false
 logallrefupdates = true
[remote "origin"]
 url = http://192.168.179.100:9999/root/java-demo.git
 fetch = +refs/heads/*:refs/remotes/origin/*

This . represents all, submitting the current code directory to the temporary storage area

[root@localhost tomcat-java-demo-master]# git add .
[root@localhost tomcat-java-demo-master]# ls
db Dockerfile LICENSE pom.xml README.md src

Submit to the current git repository, because it is a git repository itself (the local one is also a git repository)

[root@localhost tomcat-java-demo-master]# git commit -m 'all'

Push to the remote warehouse, this will read the previous configuration file

[root@localhost tomcat-java-demo-master]# git push origin master
Username for 'http://192.168.179.100:9999': root
Password for 'http://[email protected]:9999': 
Counting objects: 179, done.
Compressing objects: 100% (166/166), done.
Writing objects: 100% (179/179), 1.12 MiB | 0 bytes/s, done.
Total 179 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), done.
To http://192.168.179.100:9999/root/java-demo.git
 * [new branch] master -> master

Deploy Jenkins

[root@localhost jenkins]# ls
apache-maven-3.5.0-bin.tar.gz jdk-8u45-linux-x64.tar.gz
[root@localhost jenkins]# tar xf apache-maven-3.5.0-bin.tar.gz;tar xf jdk-8u45-linux-x64.tar.gz
[root@localhost jenkins]# mv jdk1.8.0_45 /usr/local/jdk
[root@localhost jenkins]# mv apache-maven-3.5.0 /usr/local/maven

The purpose of these two toolkits is to allow Jenkins to use them. Jenkins is deployed using containers, so the two environments on the host machine can be mounted into them. To mount the Maven JDK.

Because we need to build the image in Jenkins, we need to use docker build, so there is docker in docker, that is, build docker in Jenkins. There is no need to study how to install docker in the container. You can directly mount the host's commands and sockets into the container and use the docker commands directly in the container.

-v means persisting Jenkins data to the directory. Jenkins officially stores all data files generated by Jenkins in

/var/jenkins_home

Directory, so you only need to persist this directory. Even if the jenkins container is deleted, the data can still be recovered.

The image used is the latest version, a long-term maintained image

docker run -d --name jenkins -p 80:8080 -p 50000:50000 -u root \
  -v /opt/jenkins_home:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /usr/bin/docker:/usr/bin/docker \
  -v /usr/local/maven:/usr/local/maven \
  -v /usr/local/jdk:/usr/local/jdk \
  -v /etc/localtime:/etc/localtime \
  --restart=always \
  --name jenkins jenkins/jenkins
 
[root@localhost jenkins]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
566d2ec85fe0 jenkins/jenkins "/sbin/tini -- /usr/…" 3 minutes ago Up 3 minutes 0.0.0.0:50000->50000/tcp, 0.0.0.0:80->8080/tcp jenkins

Port 5000 is the master slave port

[root@localhost jenkins]# docker logs 566d2ec85fe0 View logs

Click to select plugin installation. Here, select No plugins are installed. Install Jenkins first and then install whatever plugins you need.

Jenkins pipeline concepts

• Jenkins Pipeline is a set of plugins that supports integration and continuous delivery pipelines in Jenkins;

• Pipeline models simple to complex transport pipelines through specific syntax;

• Declarative: Follows the same syntax as Groovy. pipeline { }

• Scripting: supports most of the features of Groovy and is also a very expressive and flexible tool. node { }

• The definition of a Jenkins Pipeline is written into a text file called a Jenkinsfile

First, you need to install the pipeline plugin and then write the text to model the pipeline. Now we mainly use declarative syntax. What does the text file look like?

This file tells Jenkins what to do. A large stage is divided into many small stages. These stages are divided into pipelines, and these stages form the pipeline. Execute in order from top to bottom until the execution is complete.

The writing of pipeLine is roughly the following processes

After the Pipeline is implemented, there is a case

This template will output each step template according to each step of the pipeline. Such as how long it took and the log. This makes it easy to locate the problem and convenient to view.

Jenkins pipeline plugin installation and pipeline usage

It is very slow when installing plugins. How to speed it up? When Jenkins downloads the plug-in, there are some sources. You can modify the domestic source. It is actually useless to modify it in the advanced plug-in.

[root@localhost jenkins]# cd /opt/jenkins_home/
[root@localhost jenkins_home]# ls
config.xml nodeMonitors.xml
copy_reference_file.log nodes
hudson.model.UpdateCenter.xml plugins
identity.key.enc secret.key
jenkins.install.InstallUtil.lastExecVersion secret.key.not-so-secret
jenkins.install.UpgradeWizard.state secrets
jenkins.model.JenkinsLocationConfiguration.xml updates
jenkins.telemetry.Correlator.xml userContent
jobs users
logs war
[root@localhost jenkins_home]# cd updates/
[root@localhost updates]# pwd
/opt/jenkins_home/updates
 
[root@localhost updates]# ls

default.json actually writes the source of the plugin hudson.tasks.Maven.MavenInstaller and uses sed to modify it

sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && \

sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json

Change to Tsinghua source and restart to take effect

[root@localhost updates]# docker restart jenkins

Jenkins

Install both git and pipeline. It's OK if some of the installations fail, as long as the core key is installed.

After replacing the plugin source, installing the plugin has become significantly faster. You can see that there is an additional pipeline option for the new project. Here, choose pipeline build.

pipeline
  agent any
 
  stages {
    stage('1. Pull the code') {
      steps {
        echo 'Pull code'
      }
    }
        stage('2. Code code code compile') {
      steps {
        echo 'compile'
      }
    }
        stage('3. Code deployment') {
      steps {
        echo 'deploy'
      }
    }
  }
}

You can see the sign

Here is the console output. You can also specify to run the build from a certain stage.

Next, we use pipeline to complete the process design.

Pull the code from git and build the image. Push to harbor and then deploy to docker. This script can be reused

#!/usr/bin/env groovy
 
def registry = "reg.harbor.com"
def project = "welcome"
def app_name = "demo"
def image_name = "${registry}/${project}/${app_name}:${Branch}-${BUILD_NUMBER}"
def git_address = "http://192.168.179.100:9999/root/java-demo.git"
def docker_registry_auth = "c91491d9-91cd-4248-96c8-fc8cc00f7db4"
def git_auth = "86ebc99d-d3fc-4e6e-b938-db1f5a1f3642"
 
pipeline
  agent any
  stages {
    stage('Pull code'){
      steps {
       checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
      }
    }
 
    stage('code compilation'){
      steps {
       sh """
        JAVA_HOME=/usr/local/jdk
        PATH=$JAVA_HOME/bin:/usr/local/maven/bin:$PATH
        mvn clean package -Dmaven.test.skip=true
        """ 
      }
    }
 
    stage('Build image'){
      steps {
        withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
        sh """
         echo '
          FROM ${registry}/library/tomcat:v1
          LABEL maitainer lizhenliang
          RUN rm -rf /usr/local/tomcat/webapps/*
          ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
         ' > Dockerfile
         docker build -t ${image_name} .
         docker login -u ${username} -p '${password}' ${registry}
         docker push ${image_name}
        """
        }
      } 
    }
 
    stage('deploy to Docker'){
      steps {
       sh """
       docker rm -f tomcat-java-demo |true
       docker container run -d --name tomcat-java-demo -p 88:8080 ${image_name}
       """
      }
    }
  }
}

You can see that you are asked to enter the branch here.

Parameterized build, variable names can be referenced in shell scripts or pipelines

      steps {
        echo '${Branch}'
      }

Create harbor credentials, that is, to use jenkin to access the docker harbor warehouse, you need to know the username and password

You can see that there is an additional credential c91491d9-91cd-4248-96c8-fc8cc00f7db4. The credential is defined in the pipeline script def docker_registry_auth = "c91491d9-91cd-4248-96c8-fc8cc00f7db4" This is the credential of Jenkins

You also need the plugin git credentials 86ebc99d-d3fc-4e6e-b938-db1f5a1f3642. Define def git_auth = "86ebc99d-d3fc-4e6e-b938-db1f5a1f3642" in pipeline

Some statements in Pipeline can be automatically generated, such as generating pull code

Just know the general framework of the pipeline and put it in

checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '86ebc99d-d3fc-4e6e-b938-db1f5a1f3642', url: 'http://192.168.179.100:9999/root/java-demo.git']]])

Pull the compiled war package and put it into the environment image

Now configure the Maven source

Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-war-plugin/3.1.0/maven-war-plugin-3.1.0.jar (91 kB at 5.3 kB/s)

Downloading: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-web/2.0.1.RELEASE/spring-boot-starter-web-2.0.1.RELEASE.pom

You can see these things in the background output. When compiling and building, the Maven source is the default network, which is relatively slow. Now we need to modify the source to ensure that the pull speed is faster. Modify it on the host machine, which defines the source from which to pull the corresponding dependency package

[root@localhost ~]# vim /usr/local/maven/conf/settings.xml 
  <mirror>
   <id>central</id>
   <mirrorOf>central</mirrorOf>
   <name>aliyun maven</name>
   <url>https://maven.aliyun.com/repository/public</url>
  </mirror>

After the modification, you do not need to restart the container and rebuild it directly. You can see the built package

[root@localhost ~]# ls /opt/jenkins_home/workspace/test-pipeline/target/ classes generated-sources ly-simple-tomcat-0.0.1-SNAPSHOT ly-simple-tomcat-0.0.1-SNAPSHOT.war maven-archiver maven-status

Job stores data source information

[root@localhost ~]# ls /opt/jenkins_home/jobs/

test test-pipeline

If you need to deploy on other dcker hosts, you can use the ssh command of the ssh plugin or use ansible to start and execute the docker command on another machine to create a container

Off topic

If your harbor provides Https authentication, please copy the certificate to the corresponding directory on your jenkins host. Make sure the jenkins host can log in to the docker harbor warehouse

[root@localhost ~]# mkdir -p /etc/docker/certs.d/reg.harbor.com
[root@localhost ~]# ls
anaconda-ks.cfg reg.harbor.com.pem
 
[root@localhost ~]# cp reg.harbor.com.pem /etc/docker/certs.d/reg.harbor.com/reg.harbor.com.crt
 
[root@localhost ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.179.102 reg.harbor.com
 
[root@localhost ~]# docker login reg.harbor.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 
Login Succeeded

The above article about Docker Gitlab+Jenkins+Harbor to build a persistent platform operation is all the content that the editor shares with you. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM.