Linux virtual memory settings tutorial and practice

Linux virtual memory settings tutorial and practice

What is Virtual Memory?

First, I will directly quote an introduction from Wikipedia.

Virtual memory is a technology for computer system memory management. It makes the application think that it has continuous available memory (a continuous and complete address space), but in fact, it is usually divided into multiple physical memory fragments, and some of them are temporarily stored on external disk storage for data exchange when needed.

For variables in C language, we can use the & operator to get its address. Since it is a virtual address, it means that this address is virtual.

The virtual address mechanism is not necessary. In a simple microcontroller, the physical RAM space distribution needs to be specified when the code is compiled. There is no concept of virtual address. The address refers to the physical address in the RAM.

  • Virtual memory (it is called virtual memory because it is relative to the logical memory and physical memory in the system. Logical memory is the memory seen from the process perspective, so it is what programmers care about. Physical memory is the memory seen from the processor perspective and is managed by the operating system. Virtual memory can be said to be a technical means of mapping to these two different perspectives of memory.) technology is a method of on-demand dynamic memory allocation taken over by the operating system. It allows programs to unknowingly use storage space larger than the actual physical space size (in fact, it is to store the storage space required by the program in the form of pages in physical memory and disk). Therefore, virtual memory completely liberates programmers. From then on, programmers do not have to worry too much about the size and loading of the program. They can write programs freely and leave all the tedious things to the operating system.
  • SWAP (meaning "exchange", "physical exchange") partition is the Linux swap partition. Its function is that when there is insufficient memory, the operating system will first store the temporarily unused data in the memory to the swap space of the hard disk, freeing up memory for other programs to run. When the requested data is not in the memory, the system will generate a page interrupt, and the memory manager will then transfer the corresponding memory page from the hard disk back to the physical memory. (It is important to note the difference between it and memory-mapped files. Linux virtual memory or swap partition is a designated area on the disk, namely swap.) The swap partition actually exists on the disk, but the Linux system can use it as memory. When the physical real memory is insufficient, the swap partition can exchange data with the real memory. Simply put, it is to take out a piece of space from the disk as a memory reserve area. Although the disk is used as memory, the speed is still the speed of the disk.

introduce

In our own purchased server environment, we usually buy 1g of memory, but when there are more things installed in the server, the memory will be insufficient. This article will simulate a real situation where the memory is insufficient and how to modify the virtual memory to make the system run normally. Our environment here is to build an ElasticSearch search environment, but our server memory is only 1g. The following will demonstrate how to modify the 1g virtual memory to 4G.

Build ElasticSearch environment

Now our server environment is empty and has nothing. We first upload ElasticSearch to the server and then install jdk and ElasticSearch.

Install JDK

The installation tutorial will be updated later (this article mainly introduces setting up virtual memory. Installing these things is mainly to simulate a state of insufficient memory)

Install ElasticSearch

The installation link will be updated later (this article mainly introduces setting up virtual memory. Installing these things is mainly to simulate a state of insufficient memory)

Start ElasticSearch

When you start ElasticSearch, you will find that an error is reported during startup. The reason is that the current memory of our server cannot meet the memory required by ElasticSearch.

[esyonghu@localhost elasticsearch-6.4.0]$ ./bin/elasticsearch 
[1] 3228
[esyonghu@localhost elasticsearch-6.4.0]$ Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
# An error report file with more information is saved as:
# logs/hs_err_pid3228.log
[esyonghu@localhost elasticsearch-6.4.0]$

Check the memory of our server and use the command free. You can see that the memory of our server is 1g. At this time, we need to modify the virtual memory to solve the problem.

[esyonghu@localhost elasticsearch-6.4.0]$ free -m
  total used free shared buffers cached
Mem: 980 582 397 2 23 245
-/+ buffers/cache: 313 667
Swap: 0 0 0
[esyonghu@localhost elasticsearch-6.4.0]$

Creating a swap file

Enter the /usr directory

[root@localhost usr]$ pwd
/usr
[root@localhost usr]$

Create a swap folder and enter it

[root@localhost usr]# mkdir swap
[root@localhost usr]# cd swap/
[root@localhost swap]# pwd
/usr/swap
[root@localhost swap]#

Create a swapfile file, use the command dd if=/dev/zero of=/usr/swap/swapfile bs=1M count=4096

[root@localhost swap]# dd if=/dev/zero of=/usr/swap/swapfile bs=1M count=4096
Recorded 4096+0 reads Recorded 4096+0 writes 4294967296 bytes (4.3 GB) copied, 15.7479 seconds, 273 MB/sec [root@localhost swap]#

View swap file

Use the command du -sh /usr/swap/swapfile to see that the swap file we created is 4g

[root@localhost swap]# du -sh /usr/swap/swapfile
4.1G /usr/swap/swapfile
[root@localhost swap]#

Set the target to the swap partition file

1. Use the command mkswap /usr/swap/swapfile to set the swapfile file as the swap partition file

[root@localhost swap]# mkswap /usr/swap/swapfile
mkswap: /usr/swap/swapfile: warning: don't erase bootbits sectors
 on whole disk. Use -f to force.
Setting up swapspace version 1, size = 4194300 KiB
no label, UUID=5bd241ff-5375-449d-9975-5fdd429df784
[root@localhost swap]#

Activate the swap area and enable the swap area files immediately

Use the command swapon /usr/swap/swapfile

[root@localhost swap]# swapon /usr/swap/swapfile
[root@localhost swap]#

Use the command free -m to check the current memory. You can see that the Swap partition has become 4095M, which means 4G memory.

[root@localhost swap]# free -m
  total used free shared buffers cached
Mem: 980 910 70 3 8 575
-/+ buffers/cache: 326 654
Swap: 4095 0 4095
[root@localhost swap]#

Set virtual memory to be automatically enabled at startup and add the following command to the etc/fstab file

1. Use the vim editor to open the /etc/fstab file

2. Add the following content to the file

/usr/swap/swapfile2 swap swap defaults 0 0

Restart the server using the reboot command

1. Enter the reboot command to restart

	[root@localhost swap]# reboot

	Broadcast message from [email protected]
		(/dev/pts/1) at 3:56 ...

	The system is going down for reboot NOW!
	[root@localhost swap]# Connection to 192.168.136.142 closed by remote host.
	Connection to 192.168.136.142 closed.
	[Process Completed]

2. After the restart is complete, use the free -m command to check whether the current memory is hung.

[root@localhost swap]# free -m
    total used free shared buffers cached
Mem: 980 910 70 3 8 575
-/+ buffers/cache: 326 654
Swap: 4095 0 4095

Start ElasticSearch again to see if the out of memory error is still reported.

1. Switch to esyonghu to start (I will not introduce why we use es user to start here, this is the knowledge in elasticsearch, here we just use elasticsearch to simulate the situation of insufficient memory), you can see that there is no problem of insufficient memory.

[esyonghu@localhost elasticsearch-6.4.0]$ ./bin/elasticsearch &
[1] 2898
[esyonghu@localhost elasticsearch-6.4.0]$ [2019-03-06T04:00:24,841][INFO ][oenNode ][] initializing ...
[2019-03-06T04:00:24,928][INFO ][oeeNodeEnvironment ] [dMy5nR5] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [7.6gb], net total_space [17.3gb], types [rootfs]
[2019-03-06T04:00:24,928][INFO ][oeeNodeEnvironment ] [dMy5nR5] heap size [1.9gb], compressed ordinary object pointers [true]
[2019-03-06T04:00:25,018][INFO ][oenNode ] [dMy5nR5] node name derived from node ID [dMy5nR5fThaBb-Q2T0txdA]; set [node.name] to override
[2019-03-06T04:00:25,018][INFO ][oenNode ] [dMy5nR5] version[6.4.0], pid[2898], build[default/tar/595516e/2018-08-17T23:18:47.308994Z], OS[Linux/2.6.32-696.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2019-03-06T04:00:25,018][INFO ][oenNode ] [dMy5nR5] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.24Q3S9AE, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/esyonghu/elasticsearch-6.4.0, -Des.path.conf=/home/esyonghu/elasticsearch-6.4.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-03-06T04:00:28,022][INFO ][oepPluginsService ] [dMy5nR5] loaded module [aggs-matrix-stats]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [analysis-common]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [ingest-common]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [lang-expression]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [lang-mustache]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [lang-painless]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [mapper-extras]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [parent-join]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [percolator]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [rank-eval]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [reindex]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [repository-url]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [transport-netty4]
[2019-03-06T04:00:28,023][INFO ][oepPluginsService ] [dMy5nR5] loaded module [tribe]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-core]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-deprecation]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-graph]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-logstash]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-ml]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-monitoring]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-rollup]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-security]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-sql]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-upgrade]
[2019-03-06T04:00:28,024][INFO ][oepPluginsService ] [dMy5nR5] loaded module [x-pack-watcher]
[2019-03-06T04:00:28,025][INFO ][oepPluginsService ] [dMy5nR5] loaded plugin [analysis-ik]
[2019-03-06T04:00:28,025][INFO ][oepPluginsService ] [dMy5nR5] loaded plugin [analysis-pinyin]
[2019-03-06T04:00:31,315][INFO ][oexsasFileRolesStore] [dMy5nR5] parsed [0] roles from file [/home/esyonghu/elasticsearch-6.4.0/config/roles.yml]
[2019-03-06T04:00:32,017][INFO ][oexmjplCppLogMessageHandler] [controller/2947] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2019-03-06T04:00:32,495][DEBUG][oeaActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-03-06T04:00:32,768][INFO ][oedDiscoveryModule ] [dMy5nR5] using discovery type [zen]
[2019-03-06T04:00:33,628][INFO ][oenNode ] [dMy5nR5] initialized
[2019-03-06T04:00:33,628][INFO ][oenNode ] [dMy5nR5] starting ...
[2019-03-06T04:00:33,860][INFO ][oetTransportService ] [dMy5nR5] publish_address {192.168.136.142:9300}, bound_addresses {[::]:9300}
[2019-03-06T04:00:33,884][INFO ][oebBootstrapChecks ] [dMy5nR5] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-03-06T04:00:36,995][INFO ][oecsMasterService ] [dMy5nR5] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {dMy5nR5}{dMy5nR5fThaBb-Q2T0txdA}{ldgTZ1XZSfOpda9uP4treA}{192.168.136.142}{192.168.136.142:9300}{ml.machine_memory=1028210688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-03-06T04:00:37,003][INFO ][oecsClusterApplierService] [dMy5nR5] new_master {dMy5nR5}{dMy5nR5fThaBb-Q2T0txdA}{ldgTZ1XZSfOpda9uP4treA}{192.168.136.142}{192.168.136.142:9300}{ml.machine_memory=1028210688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {dMy5nR5}{dMy5nR5fThaBb-Q2T0txdA}{ldgTZ1XZSfOpda9uP4treA}{192.168.136.142}{192.168.136.142:9300}{ml.machine_memory=1028210688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2019-03-06T04:00:37,058][INFO ][oexstnSecurityNetty4HttpServerTransport] [dMy5nR5] publish_address {192.168.136.142:9200}, bound_addresses {[::]:9200}
[2019-03-06T04:00:37,058][INFO ][oenNode ] [dMy5nR5] started
[2019-03-06T04:00:37,177][INFO ][owadMonitor ] try load config from /home/esyonghu/elasticsearch-6.4.0/config/analysis-ik/IKAnalyzer.cfg.xml
[2019-03-06T04:00:37,179][INFO ][owadMonitor ] try load config from /home/esyonghu/elasticsearch-6.4.0/plugins/ik/config/IKAnalyzer.cfg.xml
[2019-03-06T04:00:37,888][INFO ][oemjJvmGcMonitorService] [dMy5nR5] [gc][4] overhead, spent [486ms] collecting in the last [1.2s]
[2019-03-06T04:00:38,435][WARN ][oexsasmNativeRoleMappingStore] [dMy5nR5] Failed to clear cache for realms [[]]
[2019-03-06T04:00:38,469][INFO ][oelLicenseService ] [dMy5nR5] license [c91cae39-79d7-4a0e-b40b-b1918a45f80c] mode [trial] - valid
[2019-03-06T04:00:38,477][INFO ][oegGatewayService ] [dMy5nR5] recovered [5] indices into cluster_state
[2019-03-06T04:00:38,902][WARN ][oexsasmNativeRoleMappingStore] [dMy5nR5] Failed to clear cache for realms [[]]
[2019-03-06T04:00:39,106][INFO ][oecraAllocationService] [dMy5nR5] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[mynote2][2]] ...]).

2. Now use free -m to check memory usage. You can see that 1.7G of swap has been used.

[esyonghu@localhost elasticsearch-6.4.0]$ free -m
    total used free shared buffers cached
Mem: 980 916 64 0 3 33
-/+ buffers/cache: 880 100
Swap: 4095 1735 2360
[esyonghu@localhost elasticsearch-6.4.0]$

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. If you have any questions, you can leave a message to communicate. Thank you for your support for 123WORDPRESS.COM.

You may also be interested in:
  • Linux system diagnostics: memory basics in depth
  • How to Check Memory Usage in Linux
  • Why does the Linux system eat up my "memory"?
  • Solve the Linux system v shared memory problem
  • Linux system to view CPU, machine model, memory and other information
  • Linux kernel device driver memory management notes
  • Methods for optimizing Oracle database with large memory pages in Linux
  • A brief discussion on Linux virtual memory

<<:  Summary of common sql statements in Mysql

>>:  CocosCreator Skeleton Animation Dragon Bones

Recommend

CSS example code for setting scroll bar style

The CSS implementation code for setting the scrol...

MySQL Daemon failed to start error solution

MySQL Daemon failed to start error solution A few...

Steps to run ASP.NET Core in Docker container

There are too much knowledge to learn recently, a...

Implementation of inserting millions of records into MySQL database within 10 seconds

First, let’s think about a question: To insert su...

Solve the installation problem of mysql8.0.19 winx64 version

MySQL is an open source, small relational databas...

Vue implements the product tab of the product details page function

This article example shares the specific code of ...

How to implement https with nginx and openssl

If the server data is not encrypted and authentic...

How to backup MySQL regularly and upload it to Qiniu

In most application scenarios, we need to back up...

Tutorial on installing MySQL on Alibaba Cloud Centos 7.5

It seems that the mysql-sever file for installing...

Specific usage of textarea's disabled and readonly attributes

disabled definition and usage The disabled attrib...

Docker-compose quickly builds steps for Docker private warehouse

Create docker-compose.yml and fill in the followi...

How to quickly build a static website on Alibaba Cloud

Preface: As a junior programmer, I dream of build...

Complete steps to implement face recognition login in Ubuntu

1. Install Howdy: howdy project address sudo add-...

Detailed explanation of Linux text editor Vim

Vim is a powerful full-screen text editor and the...

Detailed example of MySQL data storage process parameters

There are three types of MySQL stored procedure p...