Implementation of Jenkins+Docker continuous integration

1. Introduction to Jenkins

Jenkins is an open source software project. It is a continuous integration tool developed based on Java. It is used to monitor continuous repetitive work. It aims to provide an open and easy-to-use software platform to make continuous integration of software possible.

2. Install and deploy Jenkins

Download address: https://jenkins.io/download/

I download the war package here, version: 1.642.3 LTS .war

1. Environmental Information

2. Create a new Jenkins user

[root@osb30 ~]# groupadd jenkins

[root@osb30 ~]# useradd -g jenkins jenkins

[root@osb30 ~]# id jenkins

uid=501(jenkins) gid=501(jenkins) groups=501(jenkins)

[root@osb30 ~]# echo "wisedu" | passwd --stdin jenkins &> /dev/null

3. Jenkins installation method

There are two ways to install Jenkins, Tomcat deployment and Java deployment startup. In this experiment, I take the deployment and startup under tomcat as an example.

(1) Deployment using Tomcat

a. First install tomcat and JAVA, and configure environment variables (this step will not be described here, java configuration is indispensable)

I installed jdk 1.8.0_65 here.

b. Put the jenkins.war file downloaded from the official website into the webapps directory under tomcat, enter the /bin directory of tomcat, and start tomcat to start jenkins.

I am using tomcat8 here.

c. When starting Jenkins, a Jenkins directory will be automatically created under the webapps directory. The access address is: http://localhost:8080/jenkins

[jenkins@osb30 ~]$ tar zxf apache-tomcat-8.0.30.tar.gz

[jenkins@osb30 ~]$ mv jenkins.war apache-tomcat-8.0.30/webapps/

[jenkins@osb30 ~]$ cd apache-tomcat-8.0.30

[jenkins@osb30 apache-tomcat-8.0.30]$ bin/startup.sh

Jenkins home directory: /home/jenkins/.jenkins found at: $user.home/.jenkins

If an error occurs at startup:

Caused by:java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable...

solve:

[jenkins@osb30 ~]$ cd apache-tomcat-8.0.30/bin/

[jenkins@osb30 bin]$ vim catalina.sh

JAVA_OPTS="-Xms1024m -Xmx1024m -Djava.awt.headless=true"

d. Access Jenkins

http://172.16.206.30:8080/jenkins

(2) Java deployment starts Jenkins

Switch to the directory where jenkins.war is stored and enter the following command:

$ java -jar jenkins.war

You can modify the startup port

$ java -jar jenkins.war --httpPort=8000

Then enter http://localhost:8080 in the browser (Firefox or Chrom is recommended). localhost can be the local IP address or the computer name. You can open Jenkins; after changing the port, the port of the access address needs to be changed synchronously.

4. Jenkins authorization and access control

By default, Jenkins does not include any security checks, and anyone can modify Jenkins settings, jobs, and start builds. Obviously, in large-scale companies where multiple departments need to work together in coordination, the absence of any security checks will cause a lot of problems. We can enhance the security of Jenkins in the following ways:

Access Jenkins: http://172.16.206.30:8080/jenkins

Click System Management -> Configure Global Security, click "Enable Security", and you can see that there are many ways to enhance Jenkins authorization and access control:

View the policy:

As shown in the figure above, the default is "Any user can do anything (without any restrictions)".

We select "Jenkins dedicated user database" in "Security domain" and "Allow user registration"; and first click "Any user can do anything (without any restrictions)" in "Authorization policy" to prevent being unable to manage Jenkins after registration. At this time, you can refresh the Jenkins page and see the login and registration buttons in the upper right corner.

(1) Register an administrator account

a. Click Register and register an administrator account first.

b. Click System Management -> Configure Global Security, select "Security Matrix" in "Authorization Policy", add users/groups - add admin account - add all permissions for the admin account, and check the functions you want the anonymous user to know.

[Note]: Anonymous users must enable the read permission here. If not, the subsequent automatic build of the webhook of GitHub or Bitbucket will not have permission.

Check this option and click Save.

After completing this operation, you can log in with the admin account and cancel the logged-in user's permission to do anything.

The above operations can complete the authorization and access control of Jenkins.

5. Jenkins system configuration

Log in to Jenkins - System Management - System Settings, and add the required functional configuration for Jenkins, which includes the following aspects:

(1) JDK configuration

In the jdk option, click "Add JDK", cancel automatic installation, enter the jdk alias (any name), JAVA_HOME should be familiar to everyone, just fill in the HOME location of the java program installed on the server where jenkins is located, fill in different paths according to different operating systems, such as win7 D:\Java\jdk1.8 linux /usr/lib/jvm/jdk1.7.0_51.

Please remember to save after setting.

(2) Add git/svn version control

Select the path of the application that controls the version according to the version used, such as the jdk configuration.

[Note]: If you use Git as the version control repository, Jenkins does not have Git installed by default. We need to select Git in the plug-in management interface and click Install directly.

Click System Management -> Manage Plugins -> Optional Plugins, and enter git in the "Filter" field in the upper right corner to search:

Find Git client plugin and Git plugin, put a √ in front of them, and click to install directly.

Installing plugin. . .

After successful installation, restart Jenkins.

[jenkins@osb30 ~]$ cd apache-tomcat-8.0.30

[jenkins@osb30 apache-tomcat-8.0.30]$ bin/shutdown.sh

[jenkins@osb30 apache-tomcat-8.0.30]$ bin/startup.sh ;tail -f logs/catalina.out

(3) Add Maven configuration to Jenkins

First determine whether Maven is installed on the host where Jenkins is located:

# mvn –version

-bash: mvn: command not found

If not installed, please install Maven first.

a. Install Maven on CentOS

[root@osb30 ~]# cd /usr/local/

[root@osb30 local]# wget http://apache.opencas.org/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

 

[root@osb30 local]# tar zxf apache-maven-3.3.9-bin.tar.gz

[root@osb30 local]# ln -s apache-maven-3.3.9 maven

[root@osb30 local]# vim /etc/profile

# Add the following configuration:

# Maven configuration.

MAVEN_HOME=/usr/local/maven

export PATH=$MAVEN_HOME/bin:$PATH

[root@osb30 local]# source /etc/profile

 

[root@osb30 local]# mvn -version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: /usr/local/maven

Java version: 1.8.0_65, vendor: Oracle Corporation

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

b.Configure Maven in Jenkins

After the installation is complete, log in to Jenkins. Click System Management -> System Settings.

3. Jenkins builds Maven-style jobs

1. Create a new Maven task

Log in to Jenkins and click New. Enter the Item name, select "Build a Maven Project", and click OK.

2. Build task configuration

3. Source code management configuration

Go to the configuration page and find "Source Code Management". I am using svn here, so I enter the address of the repository where the project is located.

4. Build trigger configuration

Under "Source Code Management" is "Build Triggers".

"Build Trigger" is a continuous integration trigger plug-in that can trigger new jobs or pass parameters based on the results of completed builds. The default option is Build whenever a SNAPSHOT dependency is built, which means that the project will be built when the code is updated.

Build periodically and Poll SCM can set up scheduled automatic builds. The difference between the two is as follows:

• Poll SCM: Check source code changes regularly (according to the version number of the SCM software). If there is an update, check out the latest code and then perform a build.
• Build periodically: Build the project regularly (it doesn't care whether the source code has changed).

I have set it to build every 12 hours.

5. Maven build settings

(1) Pre Step

The Pre Steps option is used to configure the work before building, and no changes are made here.

(2) Configure Root POM and Goals and options

Because it is a Maven project, the Build options have settings for Root POM and Goals and options. Root POM: Fill in the location of your project's pom.xml file. Note: it is a relative location. If the file does not exist, there will be a red prompt.

For example, here I am:

(3) Post Steps

After the Maven project is created, we also need to publish the war file to the Alibaba Cloud host every time the build is completed to achieve automatic publishing. We implement automatic publishing by adding a shell.

Find the Execute shell under Post steps:

[Note]: Jenkins executes the shell script as the user jenkins. In some scenarios, please pay attention to the environment variable PATH.

Configure the Alibaba Cloud host to trust the Jenkins host in the intranet:

Because the war package is on the intranet server and the published environment is on the Alibaba Cloud host, you need to configure host mutual trust to prevent the need to enter a password when scp the war package. My intranet server IP is 172.16.206.30, and the external IP is 114.55.29.246.

[jenkins@osb30 ~]$ ssh-keygen -t rsa -f .ssh/id_rsa

[jenkins@osb30 ~]$ ssh-copy-id -i .ssh/id_rsa.pub [email protected]

4. Jenkins email notification settings

1. Configure the built-in email function of Jenkins

(1) Find the system settings

(2) Fill in the system administrator's email address

[Note]: The system administrator's email address must be filled in, otherwise the email cannot be sent, even the test email below cannot be sent.

(3) Find the email notification, enter the SMTP server address, click Advanced, and enter the sender account and password.

(4) Check "Test the configuration by sending a test email" and enter the recipient account

At this point we can already send emails. In the specific job configuration, find "Build Settings" and enter the recipient's email address, but you will find that emails can only be sent when the build fails. You can install the Email Extension Plugin to customize it.

2. Install and use the Email Extension Plugin

(1) Install the Email Extension Plugin

This plugin supports Jenkins version 1.5 and above.

Go to System Management - Plugin Management - Install Email Extension Plugin. It can send build reports based on the results of the build. This plugin supports Jenkins version 1.5 and above.

[Note]: If you use the Email Extension Plugin after installation, you can abandon the built-in email function.

(2) Configure and use the Email Extension Plugin

Click "System Configuration" -> "System Settings".

Find Extended E-mail Notification and enter the following configuration:

【Note】: The User Name and Password above can only be seen after clicking "Advanced Settings". You must enter the username and password, otherwise the email cannot be sent.

Click Save below.

Then go to the job configuration page to activate this plugin. Find the item you want to email and click on it.

Click Configure, click Add Post-Build Action, and select Editable Email Notification.

Attach the build log, click on Advanced Settings.

Configure Triggers:

More detailed introduction: http://www.cnblogs.com/zz0412/p/jenkins_jj_01.html

5. Sonar

Official documentation: http://docs.sonarqube.org/display/SONARQUBE45/Documentation

1. Introduction to sonar

Sonar is an open source platform for code quality management, used to manage the quality of Java source code. Through the plug-in mechanism, Sonar can integrate different testing tools, code analysis tools, and continuous integration tools, such as pmd-cpd, checkstyle, findbugs, and Jenkins. These results are reprocessed through different plug-ins, and changes in code quality are measured in a quantitative way, making it easy to manage code quality for projects of different sizes and types.

Unlike continuous integration tools (such as Hudson/Jenkins), Sonar does not simply display the results of different code checking tools (such as FindBugs, PMD, etc.) directly on the web page, but reprocesses these results through different plug-ins, and measures the changes in code quality in a quantitative way, so that it can easily manage the code quality of projects of different sizes and types.

In terms of support for other tools, Sonar not only provides support for IDE, allowing you to view results online in tools such as Eclipse and IntelliJ IDEA; Sonar also provides interface support for a large number of continuous integration tools, making it very convenient to use Sonar in continuous integration.

In addition, Sonar's plug-ins can also provide support for programming languages ​​​​other than Java, and also have good support for internationalization and report documentation.

2. Environmental requirements

http://docs.sonarqube.org/display/SONAR/Requirements

3. Create a new user

[root@osb30 ~]# groupadd sonar

[root@osb30 ~]# useradd -g sonar sonar

[root@osb30 ~]# id sonar

uid=502(sonar) gid=502(sonar) groups=502(sonar)

[root@osb30 ~]# echo "wisedu" | passwd --stdin sonar &> /dev/null

4. Install JDK

[sonar@osb30 ~]$ java -version

java version "1.8.0_65"

Java(TM) SE Runtime Environment (build 1.8.0_65-b17)

Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

5. Install and configure the database

[root@osb30 ~]# mysql -uroot –p

mysql> CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8_general_ci;

mysql> CREATE USER 'sonar' IDENTIFIED BY 'sonar';

mysql> GRANT ALL ON sonar.* TO 'sonar'@'%' IDENTIFIED BY 'wisedu';

mysql> GRANT ALL ON sonar.* TO 'sonar'@'localhost' IDENTIFIED BY 'wisedu';

mysql> FLUSH PRIVILEGES;

6. Install sonar

The version I use here is SonarQube 4.5.7 (LTS *), upload the software to the home directory of the sonar user.

[sonar@osb30 ~]$ unzip -oq sonarqube-4.5.7.zip

[sonar@osb30 ~]$ vim sonarqube-4.5.7/conf/sonar.properties

Modify the following fields (just configure the database information, nothing else needs to be changed):

sonar.jdbc.username: sonar

sonar.jdbc.password: wisedu

sonar.jdbc.url: jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true

 

# Optional properties

sonar.jdbc.driverClassName: com.mysql.jdbc.Driver

7. Start sonar

Sonar integrates the jetty container by default, which can be started directly to provide services, or it can be built into a war package through a script and deployed in a tomcat container.

The default port of Sonar is "9000", the default context path is "/", the default network interface is "0.0.0.0", and the default administrator account and password are: admin/admin. These parameters can be modified in the configuration file sonar.properties. I changed the port here because port 9000 of this machine is occupied by other programs.

[sonar@osb30 ~]$ vim sonarqube-4.5.7/conf/sonar.properties

sonar.web.port=9003

[sonar@osb30 ~]$ sonarqube-4.5.7/bin/linux-x86-64/sonar.sh start

View the logs:

[sonar@osb30 ~]$ tail -f sonarqube-4.5.7/logs/sonar.log

You can see the initialization statement when starting for the first time:

8. Close Sonar

[sonar@osb30 ~]$ sonarqube-4.5.7/bin/linux-x86-64/sonar.sh stop

9. Access sonar

Enter http://172.16.206.30:9003/ in your browser

The default administrator account and password are: admin/admin.

10. sonar plugin

Sonar supports a variety of plug-ins. The download address of the plug-in is: http://docs.codehaus.org/display/SONAR/Plugin+Library

Upload the downloaded plug-in to the ${SONAR_HOME}extensions\plugins directory and restart sonar.

Sonar integrates the Java Ecosystem plug-in by default, which is a collection of plug-ins:

• Java [sonar-java-plugin]: Java source code analysis, calculation indicators, etc.
• Squid [sonar-squid-java-plugin]: Checks code that violates Sonar defined rules
• Checkstyle [sonar-checkstyle-plugin]: Use CheckStyle to check code that violates the unified coding style
• FindBugs [sonar-findbugs-plugin]: Use FindBugs to check for defective code that violates the rules
• PMD [sonar-pmd-plugin]: Use pmd to check code that violates rules
• Surefire [sonar-surefire-plugin]: Execute unit tests using Surefire
• Cobertura [sonar-cobertura-plugin]: Get code coverage using Cobertura
• JaCoCo [sonar-jacoco-plugin]: Get code coverage using JaCOCO

11. Integration with Jenkins

It can be integrated through Maven or directly with Jenkins. I choose to integrate directly with Jenkins here.

(1) Modify the main Maven configuration file (${MAVEN_HOME}/conf/settings.xml file or ~/.m2/settings.xml file), add access to the Sonar database and Sonar service address, and add the following configuration:

<profile>

<id>sonar</id>

<properties>

    <sonar.jdbc.url>jdbc:mysql://localhost:3306/sonar</sonar.jdbc.url>

    <sonar.jdbc.driver>com.mysql.jdbc.Driver</sonar.jdbc.driver>

    <sonar.jdbc.username>sonar</sonar.jdbc.username>

    <sonar.jdbc.password>sonar</sonar.jdbc.password>

    <sonar.host.url>http://localhost:9003</sonar.host.url> <!-- Sonar server access address-->

</properties>

</profile>

 

<activeProfiles>

    <activeProfile>sonar</activeProfile>

</activeProfiles>
...

(2) In the Jenkins plugin management, choose to install the sonar jenkins plugin, which enables the project to call sonar for code measurement every time it is built.

a. Install the plugin

b. Add sonar configuration to system configuration

Enter the system configuration page to configure the sonar plug-in, as shown below:

Then click Save below.

c. Configure the build project and add Post Build Action

Click the project you want to build and then click Configure on the left.

Find "Post-build actions" at the bottom of the page and select SonarQube.

It is no longer recommended to use SonarQube maven builder. It is preferable to set up SonarQube in the build environment and use a standard Jenkins maven target.

【solve】:
http://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+Jenkins

Modify the Build:

Finally, go to Jenkins to build the project, and check the sonar console after the build:

12. Frequently Asked Questions

After Jenkins builds, sonar scans the code and reports an error:

Solution: Uninstall the sonar JavaScript plugin.

6. Combining Jenkins with Docker

I don’t use Docker Pipeline here. I just execute the shell script after the build is completed, which is more flexible.

1. Deployment process

• R&D push to SVN code base
• Jenkins builds, pulls SVN code and uses Maven to compile and package
• Package the generated code, generate a new version of the image, and push it to the local Docker repository harbor
• Release, the test machine pulls the new version of the image, deletes the original container, and re-runs the new version of the image.

2. Environmental Description

3. Configuration

Since docker is run as the root user on the Jenkins machine, and Jenkins is run as the ordinary user jenkins, you must first configure the jenkins user to be able to use the docker command.

[root@osb30 ~]# visudo

jenkins ALL=(root) NOPASSWD: /usr/bin/docker

Also configure on the Jenkins machine:

# Disable "ssh hostname sudo <cmd>", because it will show the password in clear.

# You have to run "ssh -t hostname sudo <cmd>".

#

#Defaults requiretty

Defaults:jenkins !requiretty

If this is not configured, an error will be reported when executing the following script:

+ cp -f /home/jenkins/.jenkins/workspace/godseyeBranchForNov/godseye-container/target/godseye-container-wisedu.war /home/jenkins/docker-file/godseye_war/godseye.war

+ sudo docker login -u jkzhao -p Wisedu123 -e [email protected] 172.16.206.32

sudo: sorry, you must have a tty to run sudo

Configure on the 172.16.206.32 machine:

#visudo

#

#Defaults requiretty

Defaults:root !requiretty

Otherwise, an error will be reported when executing the script on machine 172.16.206.32:

[SSH] executing...

sudo: sorry, you must have a tty to run sudo

docker: invalid reference format.

4. Install plugins

Log in to Jenkins, click "System Management", click "Manage Plugins", search for the plugin "SSH plugin", and install it.

Log in to Jenkins, click “Credentials”, and click “Add domain”.

Click "System Management", "System Configuration", and find "SSH remote hosts".

5. Configure Post Steps

The other configurations of the project remain unchanged, see the above section.

[Note]: The warehouse and authentication account used in the script need to be created in harbor first.

# Jenkins machine: After the compilation is completed, build generates a new version of the image and pushes it to the remote docker repository # Variables

JENKINS_WAR_HOME='/home/jenkins/.jenkins/workspace/godseyeBranchForNov/godseye-container/target'

DOCKERFILE_HOME='/home/jenkins/docker-file/godseye_war'

HARBOR_IP='172.16.206.32'

REPOSITORIES='godseye_war/godseye'

HARBOR_USER='jkzhao'

HARBOR_USER_PASSWD='Wisedu123'

HARBOR_USER_EMAIL='[email protected]'

 

# Copy the newest war to docker-file directory.

\cp -f ${JENKINS_WAR_HOME}/godseye-container-wisedu.war ${DOCKERFILE_HOME}/godseye.war

 

# Delete image early version.

sudo docker login -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} -e ${HARBOR_USER_EMAIL} ${HARBOR_IP} 

IMAGE_ID=`sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'`

if [ -n "${IMAGE_ID}" ];then

    sudo docker rmi ${IMAGE_ID}

fi

 

# Build image.

cd ${DOCKERFILE_HOME}

TAG=`date +%Y%m%d-%H%M%S`

sudo docker build -t ${HARBOR_IP}/${REPOSITORIES}:${TAG} . &>/dev/null

 

# Push to the harbor registry.

sudo docker push ${HARBOR_IP}/${REPOSITORIES}:${TAG} &>/dev/null 

# Pull the image and publish HARBOR_IP='172.16.206.32'

REPOSITORIES='godseye_war/godseye'

HARBOR_USER='jkzhao'

HARBOR_USER_PASSWD='Wisedu123'

 

# Log in to harbor

#docker login -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} ${HARBOR_IP}

 

# Stop container, and delete the container.

CONTAINER_ID=`docker ps | grep "godseye_web" | awk '{print $1}'`

if [ -n "$CONTAINER_ID" ]; then

    docker stop $CONTAINER_ID

    docker rm $CONTAINER_ID

else #If the container fails to start, you need docker ps -a to find the container CONTAINER_ID=`docker ps -a | grep "godseye_web" | awk '{print $1}'`

    if [ -n "$CONTAINER_ID" ]; then # If this is the first time to pull and run the container on this machine, docker ps -a will not be able to find this container. docker rm $CONTAINER_ID

    fi

fi

 

# Delete godseye_web image early version.

IMAGE_ID=`sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'`

if [ -n "${IMAGE_ID}" ];then

    docker rmi ${IMAGE_ID}

fi

 

# Pull image.

TAG=`curl -s http://${HARBOR_IP}/api/repositories/${REPOSITORIES}/tags | jq '.[-1]' | sed 's/\"//g'` #The last sed is to remove the double quotes before and after the tag docker pull ${HARBOR_IP}/${REPOSITORIES}:${TAG} &>/dev/null

 

# Run.

docker run -d --name godseye_web -p 8080:8080 ${HARBOR_IP}/${REPOSITORIES}:${TAG}

This is the end of this article about the implementation of Jenkins+Docker continuous integration. For more relevant Jenkins Docker continuous integration content, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!