Solution to nginx-ingress-controller log persistence solution

Recently I saw an article on a public account that talked about the application of nginx-ingress-controller. Someone commented below on how to make logs persistent. I just encountered this problem at work, so I put together a solution for your reference.

nginx-ingress-controller log

The log of nginx-ingress-controller includes three parts:

Controller log: Output to stdout. You can configure the output to a file through the --log_dir in the startup parameter. After redirecting to a file, it will automatically rotate, but it will not be automatically cleaned up.
accesslog: Output to stdout. The output file can be configured through the fields in nginx-configuration. Output to file is not automatically rotated or cleaned up
errorlog: Outputs to stderr. The configuration method is similar to accesslog.

Write controller logs to disk

Give nginx-ingress-controller a hostpath: /data/log/nginx_ingress_controller/ maps to /var/log/nginx_ingress_controller/ in the container,
Configure the log-dir and logtostderr parameters for nginx-ingress-controller to redirect logs to /var/log/nginx_ingress_controller/.

The controller logs need to be cleaned up regularly. Since the controller logs are output through klog (k8s.io/klog), the logs will be rolled over, so we can use the script to regularly clean up the log files before a certain time.

Write nginx logs to disk

Modify configmap: nginx-configuration. Configure the output path of accesslog and errorlog to replace the default stdout and stderr. The output path can be consistent with the controller for easy searching.

Both accesslog and errorlog have only one log file. We can use logrotate to rotate the logs and rotate and clean up the logs output to the host machine. Configuration such as:

$ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
  su root list
  rotate 7
  daily
  maxsize 50M
  copytruncate
  missingok
  create 0644 www-data root
}

In the official templates, nginx-ingress-controller starts the container by logging in as user 33 by default, so there is a permission problem when mounting the hostpath path. We need to manually execute chown -R 33:33 /data/log/nginx_ingress_controller on the machine.

Automated ops

When nginx logs are written to disk, points 2 and 3 require manual operation and maintenance. Is there any solution?

The key to the problem is: Is there any way to add a hook before the nginx-ingress-controller container is started to chown the specified directory of the host machine?

You can use initContainer. initcontainer must run to completion and exit successfully before the containers in containers run. Using this k8s feature, we develop a docker image that only executes the following scripts:

#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo "try to set dir: $logdir 's group as $userID"
chown -R $userID:$userID $logdir

The script reads some environment variables to determine which directory needs to be modified and what user group to change to.

Package the script into a docker image and put it in the deploy yaml of nginx-ingress-controller as initcontainers. Note that you need to configure environment variables and volumeMount for the initcontainer.

As for the second point, we noticed that the base image of nginx-ingress-controller comes with logrotate, so the problem is simple. We just need to mount the written logrotate configuration file into the container in the form of configmap.

A deploy yaml is as follows:

---
apiVersion: v1
kind: Service
metadata:
 name: ingress-nginx
 namespace: kube-system
spec:
 type: ClusterIP
 ports:
 - name: http
  port: 80
  targetPort: 80
  protocol: TCP
 - name: https
  port: 443
  targetPort: 443
  protocol: TCP
 selector:
  app: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
 name: default-http-backend
 namespace: kube-system
 labels:
  app: default-http-backend
spec:
 ports:
 - port: 80
  targetPort: 8080
 selector:
  app: default-http-backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: default
 namespace: kube-system
spec:
 backend:
  serviceName: default-http-backend
  servicePort: 80
---
kind: ConfigMap
apiVersion: v1
metadata:
 name: nginx-configuration
 namespace: kube-system
 labels:
  app: ingress-nginx
data:
 use-forwarded-headers: "true"
 # Configure the redirection target of nginx log here access-log-path: /var/log/nginx_ingress_controller/access.log
 error-log-path: /var/log/nginx_ingress_controller/error.log

---

# Create a configmap to configure the rotation strategy of nginx logs, which corresponds to the log file apiVersion: v1 of nginx logs in the container.
data:
 nginx.log: |
  {{ user_nginx_log.host_path }}/access.log {
    rotate {{ user_nginx_log.rotate_count }}
    daily
    maxsize {{ user_nginx_log.rotate_size }}
    minsize 10M
    copytruncate
    missingok
    create 0644 root root
  }
  {{ user_nginx_log.host_path }}/error.log {
    rotate {{ user_nginx_log.rotate_count }}
    daily
    maxsize {{ user_nginx_log.rotate_size }}
    minsize 10M
    copytruncate
    missingok
    create 0644 root root
  }
kind: ConfigMap
metadata:
 name: nginx-ingress-logrotate
 namespace: kube-system
---

kind: ConfigMap
apiVersion: v1
metadata:
 name: tcp-services
 namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
 name: udp-services
 namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: nginx-ingress-serviceaccount
 namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: nginx-ingress-clusterrole
rules:
 -apiGroups:
   - ""
  resources:
   - configmaps
   - endpoints
   - nodes
   - pods
   - secrets
  verbs:
   - list
   - watch
 -apiGroups:
   - ""
  resources:
   - nodes
  verbs:
   - get
 -apiGroups:
   - ""
  resources:
   - services
  verbs:
   - get
   - list
   - watch
 -apiGroups:
   - "extensions"
  resources:
   - ingresses
  verbs:
   - get
   - list
   - watch
 -apiGroups:
   - ""
  resources:
    - events
  verbs:
    - create
    -patch
 -apiGroups:
   - "extensions"
  resources:
   - ingresses/status
  verbs:
   - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
 name: nginx-ingress-role
 namespace: kube-system
rules:
 -apiGroups:
   - ""
  resources:
   - configmaps
   - pods
   - secrets
   - namespaces
  verbs:
   - get
 -apiGroups:
   - ""
  resources:
   - configmaps
  resourceNames:
   # Defaults to "<election-id>-<ingress-class>"
   # Here: "<ingress-controller-leader>-<nginx>"
   # This has to be adapted if you change either parameter
   # when launching the nginx-ingress-controller.
   - "ingress-controller-leader-nginx"
  verbs:
   - get
   - update
 -apiGroups:
   - ""
  resources:
   - configmaps
  verbs:
   - create
 -apiGroups:
   - ""
  resources:
   - endpoints
  verbs:
   - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: nginx-ingress-role-nisa-binding
 namespace: kube-system
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: nginx-ingress-role
subjects:
 - kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: nginx-ingress-clusterrole-nisa-binding
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: nginx-ingress-clusterrole
subjects:
 - kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: ingress-nginx
 namespace: kube-system
spec:
 selector:
  matchLabels:
   app: ingress-nginx
 template:
  metadata:
   labels:
    app: ingress-nginx
   annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
  spec:
   serviceAccountName: nginx-ingress-serviceaccount
   Tolerations:
   - key: dedicated
    value: ingress-nginx
    effect: NoSchedule
   affinity:
    nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
       - key: "system/ingress"
        operator: In
        values:
        - "true"
   dnsPolicy: ClusterFirstWithHostNet
   hostNetwork: true
   # Configure initcontainer to ensure that the permissions of the log directory are configured before the nginx-ingress-controller container is started initContainers:
   - name: adddirperm
    image: "{{ image_registry.addr }}/{{ image.adddirperm }}"
    env:
    - name: LOG_DIR
     value: /var/log/nginx_ingress_controller
    - name: USER_ID
      value: "33"
    volumeMounts:
    - name: logdir
     mountPath: /var/log/nginx_ingress_controller
   containers:
   - name: nginx-ingress-controller
    image: "{{ image_registry.addr }}/{{ image.ingress }}"
    imagePullPolicy: IfNotPresent
    args:
    - /nginx-ingress-controller
    - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
    - --configmap=$(POD_NAMESPACE)/nginx-configuration
    - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
    - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
    - --publish-service=$(POD_NAMESPACE)/ingress-nginx
    - --annotations-prefix=nginx.ingress.kubernetes.io
    
    # Set the output path and method of controller log - --log_dir=/var/log/nginx_ingress_controller
    - --logtostderr=false
    securityContext:
     capabilities:
       drop:
       - ALL
       add:
       -NET_BIND_SERVICE
     # www-data -> 33
     runAsUser: 33
    env:
     - name: POD_NAME
      valueFrom:
       fieldRef:
        fieldPath: metadata.name
     - name: POD_NAMESPACE
      valueFrom:
       fieldRef:
        fieldPath: metadata.namespace
    ports:
    - name: http
     containerPort: 80
    - name: https
     containerPort: 443
    resources:
     requests:
      cpu: 100m
      memory: 256Mi
    livenessProbe:
     failureThreshold: 3
     httpGet:
      path: /healthz
      port: 10254
      scheme: HTTP
     initialDelaySeconds: 10
     periodSeconds: 10
     successThreshold: 1
     timeoutSeconds: 1
    readinessProbe:
     failureThreshold: 3
     httpGet:
      path: /healthz
      port: 10254
      scheme: HTTP
     periodSeconds: 10
     successThreshold: 1
     timeoutSeconds: 1
    volumeMounts:
    #Configure the log output path of the controller component and nginx in the mounted container - name: logdir
     mountPath: /var/log/nginx_ingress_controller
    #Configure the logrotate configuration mount path for nginx logs - name: logrotateconf
     mountPath: /etc/logrotate.d/nginx.log
     subPath: nginx.log
   volumes:
   # The log output path of the controller component and nginx is the hostpath of the host machine
   - name: logdir
    hostPath:
     path: {{ user_nginx_log.host_path }}
     type: ""
   # The rotation configuration file of nginx log comes from configmap
   - name: logrotateconf
    configMap:
     name: nginx-ingress-logrotate
     Items:
     - key: nginx.log
      path: nginx.log
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: default-http-backend
 namespace: kube-system
 labels:
  app: default-http-backend
spec:
 selector:
  matchLabels:
   app: default-http-backend
 template:
  metadata:
   labels:
    app: default-http-backend
  spec:
   terminationGracePeriodSeconds: 60
   Tolerations:
   - key: dedicated
    value: ingress-nginx
    effect: NoSchedule
   affinity:
    nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
       - key: "system/ingress"
        operator: In
        values:
        - "true"
   containers:
   - name: default-http-backend
    # Any image is permissible as long as:
    # 1. It serves a 404 page at /
    # 2. It serves 200 on a /healthz endpoint
    image: "{{ image_registry.addr }}/{{ image.http_backend }}"
    imagePullPolicy: IfNotPresent
    livenessProbe:
     httpGet:
      path: /healthz
      port: 8080
      scheme: HTTP
     initialDelaySeconds: 30
     timeoutSeconds: 5
    ports:
    - containerPort: 8080
    resources:
     limits:
      cpu: 10m
      memory: 20mi
     requests:
      cpu: 10m
      memory: 20mi
---

Finally, some people suggested removing initcontainer and adding a layer based on the original nginx-ingress-controller image, and executing the script for configuring path permissions in this layer. Personally, I think this method is neither beautiful nor convenient. The only benefit is that the deploy yaml is still concise (but configurations such as volumeMount are indispensable). But it depends on personal experience~

This is the end of this article about the solution to the nginx-ingress-controller log persistence solution. For more information about nginx ingress controller log persistence, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!

You may also be interested in: