Explanation of the working mechanism of namenode and secondarynamenode in Hadoop

1) Process

2) FSImage and Edits

Nodenode is the brain of HDFS. It maintains the directory tree of the entire file system and all the files and directories in the directory tree. This information is stored in the file system in two types of files: one is the namespace image (also called file system image, File System Image, FSImage), which is a complete snapshot of HDFS metadata. Each time NameNode starts, the latest namespace image is loaded by default. The other is the edit log of the command space image.

The FSImage file is actually a permanent checkpoint of the file system metadata, but not every write operation will update this file. Because FSImage is a large file, if write operations are performed frequently, the system will run extremely slowly. The solution is that NameNode only writes the changes in advance to the log, that is, writes the edit log of the namespace image. As time goes by, the edit log will become larger and larger. Once a failure occurs, it will take a lot of time to roll back the operation. So just like traditional relational databases, it is necessary to merge FSImage and edit log regularly. If the merge operation is performed by NameNode, then NameNode may not be able to provide sufficient resources when providing services for the cluster. In order to completely solve this problem, SecondaryNameNode is born.

3) Phase 1: namenode startup

• (1) After the namenode is started and formatted for the first time, the fsimage and edits files are created. If it is not the first startup, load the edit log and image file directly into the memory.
• (2) The client requests to add, delete, or modify metadata.
• (3) Namenode records operation logs and updates rolling logs.
• (4) Namenode adds, deletes, modifies and queries data in memory.

4) Phase 2: Secondary NameNode Work

• (1) Secondary NameNode asks NameNode whether a checkpoint is required. Bring it back to namenode directly to check the result.
• (2) Secondary NameNode requests to execute checkpoint.
• (3) The Secondary NameNode instructs the NameNode to perform a rolling update of the edits log being written.
• (4) The Secondary NameNode loads the FSImage file, replays the edit log, merges it into the FSImage, compresses the new FSImage file, and writes it to disk.
• (5) Copy fsimage to namenode.
• (6) Namenode renames fsimage to fsimage.

By default, this process occurs once an hour, or when the NameNode edit log file reaches the default size of 64MB.

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM. If you want to learn more about this, please check out the following links