How to use nginx to simulate canary release

This article introduces blue-green deployment and how to use nginx to simulate canary release in the simplest way.

Canary release/grayscale release

The point of canary release is trial and error. The origin of the canary release itself is a tragic story of beautiful creatures of nature being developed in the process of human industry. The canary uses its life to trial and error for the safety of miners. Use very little cost in exchange for overall security. In the practice of continuous deployment, canary is traffic control. It uses very little traffic, such as one percent or one tenth, to verify whether a certain version is normal. If it is not normal, it achieves its function at the lowest cost and reduces the risk. If it is normal, you can gradually increase the weight to 100% and smoothly switch all traffic to the new version. Grayscale release is generally a similar concept. Gray is a transition between black and white. Different from the blue-green deployment, which is neither blue nor green, there is a time period when gray release/canary release coexists, but the corresponding traffic is different. If canary release is different from gray release, the difference should be the purpose. The purpose of canary release is trial and error, while gray release is smooth release. The smooth transition carried out when there is no problem with canary release is gray release.

Simulating a canary release

Next, we use nginx's upstream to simply simulate the canary release scenario. The specific scenario is as follows: the main version is currently active. By adjusting the nginx settings and continuously adjusting the weight of the canary version, a smooth release is finally achieved.

Preparation

Two services are started on ports 7001/7002 respectively to display different information. For the convenience of demonstration, a mirror is made using tornado. Different parameters are passed when the docker container is started to display the differences in services.

docker run -d -p 7001:8080 liumiaocn/tornado:latest python /usr/local/bin/daemon.py "Hello main service： v1 in 7001"
docker run -d -p 7002:8080 liumiaocn/tornado:latest python /usr/local/bin/daemon.py "Hello canary deploy service： v2 in 7002"

Execution log

[root@kong ~]# docker run -d -p 7001:8080 liumiaocn/tornado:latest python /usr/local/bin/daemon.py "Hello main service： v1 in 7001"
28f42bbd21146c520b05ff2226514e62445b4cdd5d82f372b3791fdd47cd602a
[root@kong ~]# docker run -d -p 7002:8080 liumiaocn/tornado:latest python /usr/local/bin/daemon.py "Hello canary deploy service： v2 in 7002"
b86c4b83048d782fadc3edbacc19b73af20dc87f5f4cf37cf348d17c45f0215d
[root@kong ~]# curl http://192.168.163.117:7001
Hello, Service :Hello main service： v1 in 7001
[root@kong ~]# curl http://192.168.163.117:7002
Hello, Service :Hello canary deploy service: v2 in 7002
[root@kong ~]#

Start nginx

[root@kong ~]# docker run -p 9080:80 --name nginx-canary -d nginx
659f15c4d006df6fcd1fab1efe39e25a85c31f3cab1cda67838ddd282669195c
[root@kong ~]# docker ps |grep nginx-canary
659f15c4d006 nginx "nginx -g 'daemon ..." 7 seconds ago Up 7 seconds 0.0.0.0:9080->80/tcp nginx-canary
[root@kong ~]#

nginx code snippet

Prepare the following nginx code snippet and add it to /etc/nginx/conf.d/default.conf of nginx. The simulation method is very simple. Down is used to indicate that the traffic is zero (weight cannot be set to zero in nginx). At the beginning, 100% of the traffic is sent to the main version.

http {
upstream nginx_canary {
  server 192.168.163.117:7001 weight=100;
  server 192.168.163.117:7002 down;
}
server {
  listen 80;
  server_name www.liumiao.cn 192.168.163.117;
  location / {
    proxy_pass http://nginx_canary;
  }
}

How to modify default.conf

This can be achieved by installing vim in the container, modifying it locally and then passing it in via docker cp, or modifying it directly with sed. If you install vim in the container, use the following method

[root@kong ~]# docker exec -it nginx-lb sh
# apt-get update
...omit #apt-get install vim
...omitted

Before modification

# cat default.conf
server {
  listen 80;
  server_name localhost;
  #charset koi8-r;
  #access_log /var/log/nginx/host.access.log main;
  location / {
    root /usr/share/nginx/html;
    index index.html index.htm;
  }
  #error_page 404 /404.html;
  # redirect server error pages to the static page /50x.html
  #
  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
    root /usr/share/nginx/html;
  }
  # proxy the PHP scripts to Apache listening on 127.0.0.1:80
  #
  #location ~ \.php$ {
  # proxy_pass http://127.0.0.1;
  #}
  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  #
  #location ~ \.php$ {
  #root html;
  # fastcgi_pass 127.0.0.1:9000;
  # fastcgi_index index.php;
  # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
  #include fastcgi_params;
  #}
  # deny access to .htaccess files, if Apache's document root
  # concurs with nginx's one
  #
  #location ~ /\.ht {
  # deny all;
  #}
}
#

After

# cat default.conf
upstream nginx_canary {
  server 192.168.163.117:7001 weight=100;
  server 192.168.163.117:7002 down;
}
server {
  listen 80;
  server_name www.liumiao.cn 192.168.163.117;
  #charset koi8-r;
  #access_log /var/log/nginx/host.access.log main;
  location / {
    #root /usr/share/nginx/html;
    #index index.html index.htm;
    proxy_pass http://nginx_canary;
  }
  #error_page 404 /404.html;
  # redirect server error pages to the static page /50x.html
  #
  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
    root /usr/share/nginx/html;
  }
  # proxy the PHP scripts to Apache listening on 127.0.0.1:80
  #
  #location ~ \.php$ {
  # proxy_pass http://127.0.0.1;
  #}
  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  #
  #location ~ \.php$ {
  #root html;
  # fastcgi_pass 127.0.0.1:9000;
  # fastcgi_index index.php;
  # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
  #include fastcgi_params;
  #}
  # deny access to .htaccess files, if Apache's document root
  # concurs with nginx's one
  #
  #location ~ /\.ht {
  # deny all;
  #}
}
#

Reload nginx settings

# nginx -s reload
2018/05/28 05:16:20 [notice] 319#319: signal process started
#

Confirm the result

All 10 calls output v1 in 7001

[root@kong ~]# cnt=0; while [ $cnt -lt 10 ]; do curl http://localhost:9080; let cnt++; done
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello main service： v1 in 7001
[root@kong ~]#

Canary release: Canary version traffic weight is 10%

By adjusting the weight of default.conf and then executing nginx -s reload, the weight of the canary version is adjusted to 10%, and 10% of the traffic will execute the new service.

How to modify default.conf

You only need to adjust the weight of the server in the upstream as follows:

upstream nginx_canary {
  server 192.168.163.117:7001 weight=10;
  server 192.168.163.117:7002 weight=90;
}

Reload nginx settings

# nginx -s reload
2018/05/28 05:20:14 [notice] 330#330: signal process started
#

Confirm the result

[root@kong ~]# cnt=0; while [ $cnt -lt 10 ]; do curl http://localhost:9080; let cnt++; done
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
[root@kong ~]#

Canary release: Canary version traffic weight is 50%

By adjusting the weight of default.conf and then executing nginx -s reload, the weight of the canary version is adjusted to 50%, and 50% of the traffic will execute the new service.

How to modify default.conf

You only need to adjust the weight of the server in the upstream as follows:

upstream nginx_canary {
  server 192.168.163.117:7001 weight=50;
  server 192.168.163.117:7002 weight=50;
}

Reload nginx settings

# nginx -s reload
2018/05/28 05:22:26 [notice] 339#339: signal process started
#

Confirm the result

[root@kong ~]# cnt=0; while [ $cnt -lt 10 ]; do curl http://localhost:9080; let cnt++; done
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
[root@kong ~]#

Canary release: The traffic weight of the canary version is 90%

By adjusting the weight of default.conf and then executing nginx -s reload, the weight of the canary version is adjusted to 90%, and 90% of the traffic will execute the new service.

How to modify default.conf

You only need to adjust the weight of the server in the upstream as follows:

upstream nginx_canary {
  server 192.168.163.117:7001 weight=10;
  server 192.168.163.117:7002 weight=90;
}

Reload nginx settings

# nginx -s reload
2018/05/28 05:24:29 [notice] 346#346: signal process started
#

Confirm the result

[root@kong ~]# cnt=0; while [ $cnt -lt 10 ]; do curl http://localhost:9080; let cnt++; done
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello main service： v1 in 7001
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
[root@kong ~]#

Canary release: Canary version traffic weight is 100%

By adjusting the weight of default.conf and then executing nginx -s reload, the weight of the canary version is adjusted to 100%, and 100% of the traffic will execute the new service.

How to modify default.conf

You only need to adjust the weight of the server in the upstream as follows:

upstream nginx_canary {
  server 192.168.163.117:7001 down;
  server 192.168.163.117:7002 weight=100;
}

Reload nginx settings

# nginx -s reload
2018/05/28 05:26:37 [notice] 353#353: signal process started

Confirm the result

[root@kong ~]# cnt=0; while [ $cnt -lt 10 ]; do curl http://localhost:9080; let cnt++; done
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
Hello, Service :Hello canary deploy service: v2 in 7002
[root@kong ~]#

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM. If you want to learn more about this, please check out the following links