Detailed explanation of the use of shared memory in nginx

In the nginx process model, tasks such as traffic statistics, traffic control, and data sharing require multiple working processes to work together to complete tasks. Shared memory is an important process communication solution. This article introduces the functions related to shared memory in the nginx code, including the use and precautions of ngx_shmem and ngx_slab, but does not include the memory management algorithm implemented in ngx_slab.

Use of ngx_shmem

The ngx_shmem.c/h file is just a very simple wrapper for the mmap()/munmap() system call or shmget()/shmdt(). Implemented the ngx-style basic library, which can apply for and release a continuous shared memory space. It is generally used for shared data of fixed length. During use, the data length is fixed and will not expand or contract.

typedef struct {
  u_char *addr;
  size_t size;
  ...
} ngx_shm_t;
ngx_int_t ngx_shm_alloc(ngx_shm_t *shm);
void ngx_shm_free(ngx_shm_t *shm);

The usage process of shared memory in ngxin is generally created by the master process, and the worker process obtains the memory pointer by inheritance.

For the use of ngx_shmem, you can refer to some snippets in ngx_event_module_init(). This part of the code creates several variables in shared memory to record the number of requests in each state (accepted/reading/writing...), and performs addition and subtraction statistics on these variables at several key event entries in ngx_event_module. Implement statistics on the current request status of all worker processes.

shm.size = size;
ngx_str_set(&shm.name, "nginx_shared_zone");
shm.log = cycle->log;

if (ngx_shm_alloc(&shm) != NGX_OK) {
  return NGX_ERROR;
}

shared = shm.addr;
...
ngx_stat_accepted = (ngx_atomic_t *) (shared + 3 * cl);
ngx_stat_handled = (ngx_atomic_t *) (shared + 4 * cl);
ngx_stat_requests = (ngx_atomic_t *) (shared + 5 * cl);
ngx_stat_active = (ngx_atomic_t *) (shared + 6 * cl);
ngx_stat_reading = (ngx_atomic_t *) (shared + 7 * cl);
ngx_stat_writing = (ngx_atomic_t *) (shared + 8 * cl);
ngx_stat_waiting = (ngx_atomic_t *) (shared + 9 * cl);

For more details about this feature, see the NGX_STAT_STUB macro definition and the ngx_http_stub_status_module in the code.

Use of ngx_slab

ngx_shmem is a minimalist encapsulation that implements the basic functions of shared memory. However, most of the shared data in our program is not a fixed-size structure, but more like ngx_array, ngx_list, ngx_queue, ngx_rbtree, etc. The size of the data structure can be changed.

We expect to have a memory pool like ngx_pool_t that can dynamically request and release space. ngx_slab is such a structure. In principle, it is similar to the system's malloc() in that it uses a series of algorithms to apply for and release memory segments. It’s just that the object operated by ngx_slab is the shared memory based on ngx_shmem.

Let's take a look at the interface of ngx_slab first

typedef struct {
  ngx_shmtx_t mutex;
  ...
  void *data; /* Generally stores the root data address obtained from the pool (the first data interface requested in the pool) */
  void *addr; /* Use ngx_shmem to obtain the shared memory base address*/
} ngx_slab_pool_t;

void ngx_slab_init(ngx_slab_pool_t *pool);
void *ngx_slab_alloc(ngx_slab_pool_t *pool, size_t size);
void *ngx_slab_alloc_locked(ngx_slab_pool_t *pool, size_t size);
void *ngx_slab_calloc(ngx_slab_pool_t *pool, size_t size);
void *ngx_slab_calloc_locked(ngx_slab_pool_t *pool, size_t size);
void ngx_slab_free(ngx_slab_pool_t *pool, void *p);
void ngx_slab_free_locked(ngx_slab_pool_t *pool, void *p);

It can be seen that the interface is not complicated. The difference between alloc and calloc lies in whether the memory segment obtained by the application is cleared. The interface ending with _locked indicates that the pool being operated has already obtained the lock. In the ngx_slab_pool_t structure, there is a ngx_shmtx_t mutex lock used to synchronize concurrent scenarios where multiple processes access the pool at the same time. Note that ngx_slab_alloc() will first acquire the lock, then apply for space, and finally release the lock. However, ngx_slab_alloc_locked() directly applies for space, assuming that the program has already obtained the lock in other logic.

Using ngx_shmem in nginx development generally requires the following initialization process:

• The module calls the ngx_shared_memory_add() interface during the configuration parsing process to register a shared memory segment. Provides callback functions for shared memory size and memory initialization.
• The framework uses ngx_shmem to apply for memory in ngx_init_cycle(), initializes ngx_slab, and then calls back the initialization function registered by the module.
• Application/interface of the module using ngx_slab

In this process, the ngx_shared_memory_add() interface and the corresponding ngx_shm_zone_t structure are involved.

struct ngx_shm_zone_s {
  void *data;
  ngx_shm_t shm;
  ngx_shm_zone_init_pt init;
  void *tag;
  void *sync;
  ngx_uint_t noreuse; /* unsigned noreuse:1; */
};
ngx_shm_zone_t *ngx_shared_memory_add(ngx_conf_t *cf, ngx_str_t *name,
  size_t size, void *tag);

Among them, it is worth mentioning the noreuse attribute, which controls whether shared memory will be re-applied during the reload process of nginx.

Since the ngx_init_cycle() function is long, you can view the relevant code by looking for the comment /* create shared memory */ or the cycle->shared_memory object.

For more detailed information about the use of ngx_slab, it is recommended to refer to ngx_http_limit_conn_module, which is a module that implements connection limit through shared memory. The module has low complexity and is a good reference example.

References

In-depth understanding of Nginx (2nd Edition) https://book.douban.com/subject/26745255/

ngx_http_limit_conn_module http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.