Detailed explanation of NodeJS modularity

1. Introduction

We know that Node.js is modularized based on the CommonJS specification. Modularization is an indispensable tool for complex business scenarios. Perhaps you use it frequently but have never had a systematic understanding of it. So today we will talk about some things you need to know about Node.js modularization and explore the modularization of Node.js.

2. Main text

In Node.js, two modules are built-in for modular management. These two modules are also two keywords we are very familiar with: require and module. Built-in means that we can use these two modules in the global scope without having to reference them like other modules.

No need to require('require') or require('module')

It is not difficult to refer to a module in Node.js. It is very simple:

const config = require('/path/to/file')

But in fact, this simple code performs a total of five steps:

Understanding these five steps will help us understand the basic principles of Node.js modularity and also help us identify some pitfalls. Let’s briefly summarize what these five steps do:

• Resolving: Find the target module to be referenced and generate an absolute path.
• Loading: Determine the type of the module content to be referenced, which may be a .json file, a .js file, or a .node file.
• Wrapping: As the name suggests, wrapping the referenced module. By packaging, modules are given private scope.
• Evaluating: The loaded module is actually parsed and processed.
• Caching: Cache modules, which allows us to import the same module without repeating the above steps.

Some students may have understood these five steps and are familiar with these principles, while some students may have more doubts in their minds. In any case, the following content will analyze the above execution steps in detail, hoping to help everyone answer questions or consolidate knowledge and fill in the gaps.

By the way, if necessary, you can build an experimental directory like me and conduct experiments following the Demo.

2.1 What is a module?

To understand modularity, you need to first take a look at what a module is.

We know that in Node.js, files are modules. We just mentioned that modules can be .js, .json or .node files. By referencing them, we can get tool functions, variables, configurations, etc., but what is its specific structure? Simply execute the following command in the command line to see the module, that is, the structure of the module object:

~/learn-node $ node

> module

Module {

id: '<repl>',

exports: {},

parent: undefined,

filename: null,

loaded: false,

children: [],

paths: [ ... ] }

It can be seen that a module is just an ordinary object, but there are several special attribute values ​​in the structure that we need to understand one by one. Some attributes, such as id, parent, filename, and children, do not even need to be explained and can be understood by their literal meaning.

The following content will help you understand the meaning and function of these fields.

2.2 Resolving

After having a general understanding of what modules are, we start with the first step, Resolving, to understand the principle of modularization, that is, how Node.js finds the target module and generates the absolute path of the target module.

So why did we print the module object just now to let everyone understand the structure of the module? Because there are two field values ​​id, paths and Resolving steps that are closely related. Let’s take a look together.

First, the id attribute:

Each module has an id attribute, which is usually the full path of the module. Node.js can use this value to identify and locate the module. But there is no specific module here, we just output the module structure in the command line, so it is the default <repl> value (repl means interactive interpreter).

Next is the paths attribute:

What is the function of this paths attribute? Node.js allows us to reference modules in a variety of ways, such as relative paths, absolute paths, and preset paths (which will be explained soon). Suppose we need to reference a module called find-me, how does require help us find this module?

require('find-me')

Let's print out what's in paths:

~/learn-node $ node
> module.paths
[ '/Users/samer/learn-node/repl/node_modules',
  '/Users/samer/learn-node/node_modules',
  '/Users/samer/node_modules',
  '/Users/node_modules',
  '/node_modules',
  '/Users/samer/.node_modules',
  '/Users/samer/.node_libraries',
  '/usr/local/Cellar/node/7.7.1/lib/node' ]
  

Ok, it is actually a bunch of system absolute paths. These paths represent the possible locations of all target modules, and they are ordered, which means that Node.js will search all the paths listed in paths in order. If the module is found, it will output the absolute path of the module for subsequent use.

Now we know that Node.js will search for modules in this pile of directories. We try to execute require('find-me') to find the find-me module. Since we have not placed the find-me module in any directory, Node.js cannot find the target module after traversing all directories, so it reports the error Cannot find module 'find-me'. You may often see this error:

~/learn-node $ node

> require('find-me')

Error: Cannot find module 'find-me'

at Function.Module._resolveFilename (module.js:470:15)

at Function.Module._load (module.js:418:25)

at Module.require (module.js:498:17)

at require (internal/module.js:20:19)

at repl:1:1

at ContextifyScript.Script.runInThisContext (vm.js:23:33)

at REPLServer.defaultEval (repl.js:336:29)

at bound (domain.js:280:14)

at REPLServer.runBound [as eval] (domain.js:293:12)

at REPLServer.onLine (repl.js:533:10)

Now, you can try to put the find-me module you need to reference in any of the above directories. Here we create a node_modules directory and create a find-me.js file so that Node.js can find it:

~/learn-node $ mkdir node_modules

~/learn-node $ echo "console.log('I am not lost');" > node_modules/find-me.js

~/learn-node $ node

> require('find-me');

I am not lost

{}

>

After manually creating the find-me.js file, Node.js found the target module. Of course, when the find-me module is found in the local node_modules directory of Node.js, it will not continue to search in subsequent directories.

Students with Node.js development experience will find that when referencing a module, it is not necessary to specify an exact file. You can also reference the target module by referencing the directory, for example:

~/learn-node $ mkdir -p node_modules/find-me

~/learn-node $ echo "console.log('Found again.');" > node_modules/find-me/index.js

~/learn-node $ node

> require('find-me');

Found again.

{}

>

The index.js file in the find-me directory will be automatically imported.

Of course, this is subject to rule restrictions. The reason why Node.js can find the index.js file in the find-me directory is because the default module import rule is to look for the index.js file when the specific file name is missing. We can also change the import rules (by modifying package.json), such as index -> main:

~/learn-node $ echo "console.log('I rule');" > node_modules/find-me/main.js

~/learn-node $ echo '{ "name": "find-me-folder", "main": "main.js" }' > node_modules/find-me/package.json

~/learn-node $ node

> require('find-me');

I rule

{}

>

2.3, require.resolve

If you just want to introduce a module into your project without executing it immediately, you can use the require.resolve method, which is similar to the require method, except that the introduced module method will not be executed:

> require.resolve('find-me');

'/Users/samer/learn-node/node_modules/find-me/start.js'

> require.resolve('not-there');

Error: Cannot find module 'not-there'

at Function.Module._resolveFilename (module.js:470:15)

at Function.resolve (internal/module.js:27:19)

at repl:1:9

at ContextifyScript.Script.runInThisContext (vm.js:23:33)

at REPLServer.defaultEval (repl.js:336:29)

at bound (domain.js:280:14)

at REPLServer.runBound [as eval] (domain.js:293:12)

at REPLServer.onLine (repl.js:533:10)

at emitOne (events.js:101:20)

at REPLServer.emit (events.js:191:7)

>

As you can see, if the module is found, Node.js will print the full path of the module, if not found, it will report an error.

Now that we know how Node.js looks for modules, let’s look at how Node.js loads modules.

2.4. Parent-child dependency between modules

We express the reference relationship between modules as parent-child dependency relationship.

Simply create a lib/util.js file and add a console.log statement to indicate that this is a referenced submodule.

~/learn-node $ mkdir lib

~/learn-node $ echo "console.log('In util');" > lib/util.js

Also enter a line of console.log statements in index.js to identify this as a parent module and reference the lib/util.js just created as a submodule.

~/learn-node $ echo "require('./lib/util'); console.log('In index, parent', module);" > index.js

Execute index.js to see the dependencies between them:

~/learn-node $ node index.js

In util

In index <ref *1> Module {

id: '.',

path: '/Users/samer/',

exports: {},

parent: null,

filename: '/Users/samer/index.js',

loaded: false,

children: [

Module {

id: '/Users/samer/lib/util.js',

path: '/Users/samer/lib',

exports: {},

parent: [Circular *1],

filename: '/Users/samer/lib/util.js',

loaded: true,

children: [],

paths: [Array]

}

],

paths: [...]

}

Here we focus on two properties related to dependencies: children and parent.

In the printed result, the children field contains the imported util.js module, which indicates that util.js is a submodule that index.js depends on.

But if we look closely at the parent property of the util.js module, we find that the value Circular appears here. The reason is that when we print the module information, a circular dependency is generated. The parent module information is printed in the child module information, and the child module information is printed in the parent module information. Therefore, Node.js simply marks it as Circular.

Why do we need to understand parent-child dependencies? Because this is related to how Node.js handles circular dependencies, it will be described in detail later.

Before looking at the problem of dealing with circular dependencies, we need to understand two key concepts: exports and module.exports.

2.5. exports, module.exports

exports:

Exports is a special object that can be used directly as a global variable in Node.js without declaration. It is actually a reference to module.exports. By modifying exports, the purpose of modifying module.exports can be achieved.

Exports is also a property value in the module structure just printed, but the values ​​just printed are all empty objects because we did not operate on it in the file. Now we can try to simply assign a value to it:

// Add a new line exports.id = 'lib/util' at the beginning of lib/util.js;
  
// Add a new line exports.id = 'index' at the beginning of index.js;

Execute index.js:

~/learn-node $ node index.js

In index Module {

id: '.',

exports: { id: 'index' },

loaded: false,

... }

In util Module {

id: '/Users/samer/learn-node/lib/util.js',

exports: { id: 'lib/util' },

parent:

Module {

id: '.',

exports: { id: 'index' },

loaded: false,

... },

loaded: false,

... }

You can see that the two id attributes just added are successfully added to the exports object. We can also add any attributes other than id, just like operating ordinary objects. Of course, we can also turn exports into a function, for example:

exports = function() {}

module.exports:

The module.exports object is actually what we finally get through require. When we write a module, what value we assign to module.exports is what others will get when they reference the module. For example, combined with the previous operation on lib/util:

const util = require('./lib/util');
  
console.log('UTIL:', util);
  
// Output result UTIL: { id: 'lib/util' }

Since we just assigned {id: 'lib/util'} to module.exports through the exports object, the result of require changes accordingly.

Now we have a general understanding of what exports and module.exports are, but there is a small detail to note, that is, module loading in Node.js is a synchronous process.

Let's look back at the loaded attribute in the module structure. This attribute indicates whether the module has been loaded. This attribute can be used to easily verify the synchronization of Node.js module loading.

When the module has been loaded, the loaded value should be true. But so far every time we print module, its status is false. This is actually because in Node.js, module loading is synchronous. When we have not completed the loading action (loading action includes marking the module, including marking the loaded attribute), the printed result is the default loaded: false.

We use setImmediate to help us verify this information:

// In index.js
setImmediate(() => {
  console.log('The index.js module object is now loaded!', module)
});
Click and drag to move The index.js module object is now loaded! Module {
  id: '.',
  exports: [Function],
  parent: null,
  filename: '/Users/samer/learn-node/index.js',
  loaded: true,
  children:
   [Module {
       id: '/Users/samer/learn-node/lib/util.js',
       exports: [Object],
       parent: [Circular],
       filename: '/Users/samer/learn-node/lib/util.js',
       loaded: true,
       children: [],
       paths: [Object] } ],
  paths:
   [ '/Users/samer/learn-node/node_modules',
     '/Users/samer/node_modules',
     '/Users/node_modules',
     '/node_modules' ] }

Ok, since console.log is placed after loading is completed (marked), the loading status now becomes loaded: true. This fully verifies that Node.js module loading is a synchronous process.

Now that we know about exports, module.exports, and synchronization of module loading, let’s take a look at how Node.js handles circular dependencies of modules.

2.6. Module circular dependency

In the above content, we learned that there is a parent-child dependency relationship between modules. What will Node.js do if a circular dependency occurs between modules? Suppose there are two modules, module1.js and module2.js, and they reference each other as follows:

// lib/module1.js
  
exports.a = 1;
  
require('./module2'); // Reference here exports.b = 2;
exports.c = 3;
  
// lib/module2.js
  
const Module1 = require('./module1');
console.log('Module1 is partially loaded here', Module1); // reference module1 and print it

Try running module1.js and you can see the output:

~/learn-node $ node lib/module1.js

Module1 is partially loaded here { a: 1 }

The result only outputs {a: 1}, while {b: 2, c: 3} is missing. Looking closely at module1.js, we find that we added a reference to module2.js in the middle of module1.js, before exports.b = 2 and exports.c = 3 are executed. If we call this location the location where a circular dependency occurs, then the result we get is the property exported before the circular dependency occurs. This is also based on the conclusion that module loading in Node.js is a synchronous process, which we have verified above.

Node.js handles circular dependencies in a simple way. In the process of loading the module, the exports object will be gradually built and the exports will be assigned values. If we import a module before it is fully loaded, we will only get some of the exports object properties.

2.7. .json and .node

In Node.js, we can use require not only to reference JavaScript files, but also to reference JSON or C++ plug-ins (.json and .node files). We don't even need to explicitly declare the corresponding file suffix.

You can also see the file types supported by require in the command line:

~ % node

> require.extensions

[Object: null prototype] {

'.js': [Function (anonymous)],

'.json': [Function (anonymous)],

'.node': [Function (anonymous)]

}

When we use require to reference a module, Node.js will first check whether there is a .js file. If not found, it will then match the .json file. If still not found, it will finally try to match the .node file. But in general, to avoid confusion and unclear reference intent, you can follow the rule of explicitly specifying the suffix when referencing .json or .node files, and omitting the suffix when referencing .js (optional, or both).

.json file:

It is very common to reference .json files. For example, static configuration in some projects is easier to manage by using .json files to store, for example:

{
  "host": "localhost",
  "port": 8080
}

Referencing it or using it is easy:

const { host, port } = require('./config');
console.log(`Server will run at http://${host}:${port}`)

The output is as follows:

Server will run at http://localhost:8080

.node file:

The .node file is converted from a C++ file. The official website provides a simple hello plug-in implemented in C++, which exposes a hello() method and outputs the string world. If necessary, you can jump to the link to learn more and conduct experiments.

We can use node-gyp to compile and build .cc files into .node files. The process is also very simple. We only need to configure a binding.gyp file. I won't go into detail here, you just need to know that after generating the .node file, you can reference the file normally and use the methods in it.

For example, after converting hello() to the addon.node file, reference and use it:

const addon = require('./addon');
console.log(addon.hello());

2.8 Wrapping

In fact, in the above content, we explained the first two steps of referencing a module in Node.js: Resolving and Loading, which respectively solve the problems of module path and loading. Next, let’s take a look at what Wrapping does.

Wrapping means packaging, and the object of packaging is all the code we wrote in the module. That is to say, when we reference a module, we actually go through a layer of "transparent" packaging.

To understand this packaging process, you must first understand the difference between exports and module.exports.

exports is a reference to module.exports. We can use exports in modules to export properties, but we cannot directly replace it. For example:

exports.id = 42; // ok, now exports points to module.exports, which is equivalent to modifying module.exports.
exports = { id: 42 }; // Useless, just points it to the { id: 42 } object, no actual changes to module.exports.
module.exports = { id: 42 }; // OK, operate module.exports directly.

You may wonder why the exports object seems to be a global object for every module, but it can distinguish which module the exported object comes from. How is this done?

Before understanding the wrapping process, let's take a look at a small example:

// In a.js
var value = 'global'
  
// In b.js
console.log(value) // Output: global
  
// In c.js
console.log(value) // Output: global
  
// In index.html
...
<script src="a.js"></script>
<script src="b.js"></script>
<script src="c.js"></script>

When we define a value in the a.js script, this value is globally visible, and the subsequently introduced b.js and c.js can access the value. But this is not the case in Node.js modules. Variables defined in one module have private scope and cannot be directly accessed in other modules. How does this private scope come about?

The answer is simple. Before compiling the module, Node.js wraps the contents of the module in a function and implements private scope through function scope.

By require('module').wrapper you can print out the wrapper properties:

~ $ node

> require('module').wrapper

[ '(function (exports, require, module, __filename, __dirname) { ',

'\n});' ]

>

Node.js will not directly execute any code in the file, but it will execute the code through this wrapped function, which gives each of our modules a private scope and will not affect each other.

This wrapper function has five parameters: exports, require, module, __filename, __dirname. We can access and print these parameters directly through the arguments parameter:

./learn-node $ echo "console.log(arguments)" > index.js

~/learn-node $ node index.js

{ '0': {},

'1':

{ [Function: require]

resolve: [Function: resolve],

main:

Module {

id: '.',

exports: {},

parent: null,

filename: '/Users/samer/index.js',

loaded: false,

children: [],

paths: [Object] },

extensions: { ... },

cache: { '/Users/samer/index.js': [Object] } },

'2':

Module {

id: '.',

exports: {},

parent: null,

filename: '/Users/samer/index.js',

loaded: false,

children: [],

paths: [ ... ] },

'3': '/Users/samer/index.js',

'4': '/Users/samer' }

Let's take a brief look at these parameters. The first parameter exports is initially empty (unassigned). The second and third parameters require and module are instances related to the module we referenced. They are not global. The fourth and fifth parameters __filename and __dirname represent the file path and directory respectively.

The entire wrapped function does approximately the same thing:

unction (require, module, __filename, __dirname) {
  let exports = module.exports;
    
  // Your Code...
    
  return module.exports;
}

In short, wrapping is to privatize our module scope and expose variables or methods with module.exports as the return value for use.

2.9 Cache

Caching is easy to understand. Let's take a look at it through an example:

echo 'console.log(`log something.`)' > index.js

// In node repl

> require('./index.js')

log something.

{}

> require('./index.js')

{}

>

As you can see, the same module is referenced twice, but the information is only printed once. This is because the cache is used for the second reference, and there is no need to reload the module.

Print require.cache to see the current cache information:

> require.cache

[Object: null prototype] {

'/Users/samer/index.js': Module {

id: '/Users/samer/index.js',

path: '/Users/samer/',

exports: {},

parent: Module {

id: '<repl>',

path: '.',

exports: {},

parent: undefined,

filename: null,

loaded: false,

children: [Array],

paths: [Array]

},

filename: '/Users/samer/index.js',

loaded: true,

children: [],

paths: [

'/Users/samer/learn-node/repl/node_modules',

'/Users/samer/learn-node/node_modules',

'/Users/samer/node_modules',

'/Users/node_modules',

'/node_modules',

'/Users/samer/.node_modules',

'/Users/samer/.node_libraries',

'/usr/local/Cellar/node/7.7.1/lib/node'

]

}

}

You can see that the index.js file just referenced is in the cache, so the module will not be reloaded. Of course, we can also clear the cache content by deleting require.cache to achieve the purpose of reloading, but I will not demonstrate it here.

Conclusion

This article outlines some basic principles and common sense that you need to know when using Node.js modularization, hoping to help everyone have a clearer understanding of Node.js modularization. But the deeper details are not explained in this article, such as the processing logic inside the wrapper function, the problem of synchronous loading of CommonJS, the difference with ES modules, etc. You can explore more of these unmentioned contents outside of this article.

The above is a detailed explanation of NodeJS modularization. For more information about NodeJS modularization, please pay attention to other related articles on 123WORDPRESS.COM!