In-depth understanding of the implementation principle of require loader

Preface

We often say that node is not a new programming language, it is just a runtime for javascript. You can simply understand runtime as the environment for running javascript. In most cases we will run JavaScript in the browser. With the emergence of node, we can run JavaScript in node, which means that wherever node or browser is installed, we can run JavaScript there.

1. Implementation of node modularization

Node has its own modular mechanism. Each file is a separate module, and it follows the CommonJS specification, that is, it imports modules using require and exports modules through module.export.
The operating mechanism of the node module is also very simple. In fact, it is to wrap a layer of function outside each module. With the wrapping of function, the scope isolation between codes can be achieved.

You may say that I didn’t wrap the function when I wrote the code. Yes, that’s true. This layer of function is automatically implemented for us by node. Let’s test it.

We create a new js file and print a non-existent variable in the first line. For example, we print window here, but there is no window in node.

console.log(window);

When you execute the file through node, you will find the error message as follows. (Please use the system default cmd to execute the command).

(function (exports, require, module, __filename, __dirname) { console.log(window);
ReferenceError: window is not defined
    at Object.<anonymous> (/Users/choice/Desktop/node/main.js:1:75)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
    at startup (internal/bootstrap/node.js:279:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:752:3)

You can see that there is a self-executing function at the top level of the error report, which contains the commonly used global variables such as exports, require, module, __filename, and __dirname.

I introduced this in my previous article "The Development History of Front-end Modularity". Self-executing functions are also one of the solutions for implementing front-end modularization. In the early days when there was no modular system in the front-end, self-executing functions can solve the namespace problem very well, and other modules that the module depends on can be passed in through parameters. The cmd and amd specifications also rely on self-executing functions.

In the module system, each file is a module. Each module automatically has a function outside it and defines the export method module.exports or exports, as well as the import method require.

let moduleA = (function() {
    module.exports = Promise;
    return module.exports;
})();

2.require loading module

require relies on the fs module in node to load module files, and fs.readFile reads a string.

In javascript, we can use eval or new Function to convert a string into js code to run.

eval

const name = 'yd';
const str = 'const a = 123; console.log(name)';
eval(str); // yd;

new Function

New Function receives a string to be executed and returns a new function. Calling this new function string will execute it. If this function needs to pass parameters, you can pass the parameters one by one when creating new Function, and finally pass in the string to be executed. For example, here we pass in parameter b, the string str to be executed.

const b = 3;
const str = 'let a = 1; return a + b';
const fun = new Function('b', str);
console.log(fun(b, str)); // 4

You can see that both eval and Function instantiation can be used to execute javascript strings, and it seems that they can both implement require module loading. However, they were not chosen to implement modularity in Node. The reason is very simple because they all have a fatal problem, which is that they are easily affected by variables that do not belong to them.

As shown below, a is not defined in the str string, but the a variable defined above can be used. This is obviously wrong. In the modular mechanism, the str string should have its own independent running space, and variables that do not exist in itself cannot be used directly.

const a = 1;

const str = 'console.log(a)';

eval(str);

const func = new Function(str);
func();

Node has a concept of a VM virtual environment, which is used to run additional JS files. It can ensure the independence of JavaScript execution and will not be affected by external factors.

vm built-in module

Although we defined hello externally, str is an independent module and is not in the village hello variable, so an error will be reported directly.

// Import vm module, no need to install, node self-built module const vm = require('vm');
const hello = 'yd';
const str = 'console.log(hello)';
wm.runInThisContext(str); // Error

Therefore, node can use vm to execute javascript modules. This ensures the independence of the modules.

3.Require code implementation

Before introducing the require code implementation, let's review the usage of two node modules because they will be used below.

The path module

Used to process file paths.

basename: base path, if there is a file path, it is not the base path, the base path is 1.js

extname: Get the extension name

dirname: parent directory

join: concatenate paths

resolve: The absolute path of the current folder. Be careful not to add / at the end when using it.

__dirname: The path of the folder where the current file is located

__filename: The absolute path of the current file

const path = require('path', 's');
console.log(path.basename('1.js'));
console.log(path.extname('2.txt'));
console.log(path.dirname('2.txt'));
console.log(path.join('a/b/c', 'd/e/f')); // a/b/c/d/e/
console.log(path.resolve('2.txt'));

fs module

Used to operate files or folders, such as reading, writing, adding, deleting, etc. Commonly used methods are readFile and readFileSync, which are asynchronous and synchronous file reading respectively.

const fs = require('fs');
const buffer = fs.readFileSync('./name.txt', 'utf8'); // If no encoding is passed, binary will be output console.log(buffer);

fs.access: Determine whether a file exists. The exists method provided by node10 has been deprecated because it does not comply with the node specification, so we use access to determine whether a file exists.

try {
    fs.accessSync('./name.txt');
} catch(e) {
    // File does not exist}

4. Manually implement the require module loader

First, import the dependent module path, fs, vm, and create a Require function that receives a modulePath parameter, which indicates the file path to be imported.

// Import dependency const path = require('path'); // Path operation const fs = require('fs'); // File reading const vm = require('vm'); // File execution // Define import class, parameter is module path function Require(modulePath) {
    ...
}

Get the absolute path of the module in Require, which is convenient for loading the module using fs. Here we use new Module to abstract the reading of the module content, and use tryModuleLoad to load the module content. We will implement Module and tryModuleLoad later. The return value of Require should be the content of the module, that is, module.exports.

// Define the import class, the parameter is the module path function Require(modulePath) {
    // Get the absolute path to be loaded let absPathname = path.resolve(__dirname, modulePath);
    // Create a module and create a new Module instance const module = new Module(absPathname);
    // Load the current module tryModuleLoad(module);
    // Return exports object return module.exports;
}

The implementation of Module is very simple, which is to create an exports object for the module. When tryModuleLoad is executed, the content is added to exports. The id is the absolute path of the module.

// Define module, add file id and exports attribute function Module(id) {
    this.id = id;
    // The read file content will be placed in exports this.exports = {};
}

We have said before that the node module runs in a function. Here we mount the static property wrapper to the Module, which defines the string of this function. The wrapper is an array, and the first element of the array is the parameter part of the function, including exports, module. Require, __dirname, __filename, which are all global variables commonly used in our modules. Note that the Require parameter passed in here is the Require we defined ourselves.

The second parameter is the end of the function. Both parts are strings. When using them, we just wrap them outside the module string.

Module.wrapper = [
    "(function(exports, module, Require, __dirname, __filename) {",
    "})"
]

_extensions is used to use different loading methods for different module extensions. For example, the loading methods of JSON and javascript are definitely different. JSON is parsed using JSON.parse.

JavaScript is run using vm.runInThisContext. We can see that fs.readFileSync passes in module.id, which means that the id stored in our Module definition is the absolute path of the module. The content read is a string. We use Module.wrapper to wrap it, which is equivalent to wrapping a function outside this module, thus realizing private scope.

Use call to execute the fn function. The first parameter changes the running this. We pass in module.exports. The following parameters are the parameters wrapped outside the function, exports, module, Require, __dirname, __filename

Module._extensions = {
    '.js'(module) {
        const content = fs.readFileSync(module.id, 'utf8');
        const fnStr = Module.wrapper[0] + content + Module.wrapper[1];
        const fn = vm.runInThisContext(fnStr);
        fn.call(module.exports, module.exports, module, Require,_filename,_dirname);
    },
    '.json'(module) {
        const json = fs.readFileSync(module.id, 'utf8');
        module.exports = JSON.parse(json); // Put the result of the file in the exports property}
}

The tryModuleLoad function receives the module object, obtains the module suffix through path.extname, and then uses Module._extensions to load the module.

//Define module loading method function tryModuleLoad(module) {
    // Get the extension name const extension = path.extname(module.id);
    // Load the current module by suffix Module._extensions[extension](module);
}

At this point, we have basically finished writing the Require loading mechanism. Let's take a look at it again. When Require loads a module, pass in the module name and use path.resolve(__dirname, modulePath) in the Require method to get the absolute path of the file. Then create a module object through new Module instantiation, store the absolute path of the module in the id attribute of the module, and create the exports attribute in the module as a json object.

Use the tryModuleLoad method to load the module. In tryModuleLoad, use path.extname to get the file extension, and then execute the corresponding module loading mechanism based on the extension.

The loaded module will eventually be mounted in module.exports. After tryModuleLoad is executed, module.exports already exists, so just return directly.

// Import dependency const path = require('path'); // Path operation const fs = require('fs'); // File reading const vm = require('vm'); // File execution // Define import class, parameter is module path function Require(modulePath) {
    // Get the absolute path to be loaded let absPathname = path.resolve(__dirname, modulePath);
    // Create a module and create a new Module instance const module = new Module(absPathname);
    // Load the current module tryModuleLoad(module);
    // Return exports object return module.exports;
}
// Define module, add file id and exports attribute function Module(id) {
    this.id = id;
    // The read file content will be placed in exports this.exports = {};
}
// Define the function that wraps the module content Module.wrapper = [
    "(function(exports, module, Require, __dirname, __filename) {",
    "})"
]
// Define the extension name. Different extension names have different loading methods. Implement js and json
Module._extensions = {
    '.js'(module) {
        const content = fs.readFileSync(module.id, 'utf8');
        const fnStr = Module.wrapper[0] + content + Module.wrapper[1];
        const fn = vm.runInThisContext(fnStr);
        fn.call(module.exports, module.exports, module, Require,_filename,_dirname);
    },
    '.json'(module) {
        const json = fs.readFileSync(module.id, 'utf8');
        module.exports = JSON.parse(json); // Put the result of the file in the exports property}
}
//Define module loading method function tryModuleLoad(module) {
    // Get the extension name const extension = path.extname(module.id);
    // Load the current module by suffix Module._extensions[extension](module);
}

5. Add cache to the module

Adding cache is also relatively simple. When loading a file, put the file into the cache. When loading a module, check whether it exists in the cache. If it exists, use it directly. If it does not exist, re-load it and put it into the cache after loading.

// Define the import class, the parameter is the module path function Require(modulePath) {
    // Get the absolute path to be loaded let absPathname = path.resolve(__dirname, modulePath);
    // Read from the cache, if it exists, return the result directly if (Module._cache[absPathname]) {
        return Module._cache[absPathname].exports;
    }
    // Try to load the current module tryModuleLoad(module);
    // Create a module and create a new Module instance const module = new Module(absPathname);
    // Add cache Module._cache[absPathname] = module;
    // Load the current module tryModuleLoad(module);
    // Return exports object return module.exports;
}

6. Automatically complete the path

Automatically add a suffix to the module to load the module without suffix. In fact, if the file has no suffix, it will traverse all the suffixes to see if the file exists.

// Define the import class, the parameter is the module path function Require(modulePath) {
    // Get the absolute path to be loaded let absPathname = path.resolve(__dirname, modulePath);
    // Get all suffix names const extNames = Object.keys(Module._extensions);
    let index = 0;
    //Store the original file path const oldPath = absPathname;
    function findExt(absPathname) {
        if (index === extNames.length) {
           return throw new Error('file does not exist');
        }
        try {
            fs.accessSync(absPathname);
            return absPathname;
        } catch(e) {
            const ext = extNames[index++];
            findExt(oldPath + ext);
        }
    }
    // Recursively append the suffix name to determine whether the file exists absPathname = findExt(absPathname);
    // Read from the cache, if it exists, return the result directly if (Module._cache[absPathname]) {
        return Module._cache[absPathname].exports;
    }
    // Try to load the current module tryModuleLoad(module);
    // Create a module and create a new Module instance const module = new Module(absPathname);
    // Add cache Module._cache[absPathname] = module;
    // Load the current module tryModuleLoad(module);
    // Return exports object return module.exports;
}

7. Analysis and implementation steps

1. Import related modules and create a Require method.

2. Extract through Module._load method, which is used to load the module.

3.Module.resolveFilename converts the relative path into an absolute path.

4. Cache module Module._cache, do not load the same module repeatedly to improve performance.

5. Create module id: The saved content is exports = {} which is equivalent to this.

6. Use tryModuleLoad(module, filename) to try to load the module.

7.Module._extensions uses read files.

8.Module.wrap: Wrap the read js with a function.

9. Run the obtained string using runInThisContext.

10. Let the string execute and adapt this to exports.

Summarize

This is the end of this article about the implementation principle of require loader. For more information about the principle of require loader, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!