Implementing CommonJS modularity in browsers without compilation/server

introduction
1. What is one-click.js
2. How do packaging tools work?
3. Challenges we face

3.1. Dependency Analysis
3.2. Scope Isolation
3.3. Providing module runtime

4. Conclusion

introduction

I often visit Github. In addition to some large projects with extremely high stars, I also find many interesting small projects on Github. The projects or ideas are very interesting, or have good technical points, and reading them is rewarding. So I plan to compile it into a "Roaming Github" series, occasionally sharing and interpreting interesting projects that I come across on Github. This series focuses on explaining the principles without going into the details of the source code.

Okay, let’s get to the point. The repository to be introduced in this issue is called one-click.js.

1. What is one-click.js

one-click.js is a very interesting library. Github introduces it like this:

We know that if you want Commonjs modular code to run normally in the browser, you usually need a build/packaging tool, such as webpack, rollup, etc. One-click.js allows you to run the CommonJS-based module system normally in the browser without the need for these build tools.

Furthermore, you don't even need to start a server. For example, you can try to clone the one-click.js project and just double-click (open with a browser) the example/index.html file to run it.

There is a sentence in the Repo that summarizes its functions:

Use CommonJS modules directly in the browser with no build step and no web server.

For example,

Assume that in the current directory (demo/) now, we have three "module" files:

demo/plus.js:

// plus.js
module.exports = function plus(a, b) {
    return a + b;
}

demo/divide.js:

//divide.js
module.exports = function divide(a, b) {
    return a / b;
}

And the entry module file demo/main.js:

// main.js
const plus = require('./plus.js');
const divide = require('./divide.js');
console.log(divide(12, add(1, 2)));
// output: 4

The common usage is to specify the entry, compile it into a bundle with webpack, and then reference it in the browser. One-click.js allows you to discard all of this and just use it in HTML:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>one click example</title>
</head>
<body>
    <script type="text/JavaScript" src="./one-click.js" data-main="./main.js"></script>
</body>
</html>

Note the use of the script tag, where data-main specifies the entry file. At this time, directly open this local HTML file with a browser and the result 7 can be output normally.

2. How do packaging tools work?

The previous section introduced the function of one-click.js - the core is to achieve front-end modularization without packaging/building.

Before introducing its internal implementation, let's first understand what the packaging tool does. As the saying goes, if you know yourself and your enemy, you can fight a hundred battles with no danger of defeat.

Again, our three JavaScript files.

plus.js:

// plus.js
module.exports = function plus(a, b) {
    return a + b;
}

divide.js:

//divide.js
module.exports = function divide(a, b) {
    return a / b;
}

And the entry module main.js:

// main.js
const plus = require('./plus.js');
const divide = require('./divide.js');
console.log(divide(12, add(1, 2)));
// output: 4

Recall that when we use webpack, we specify the entry point (main.js). webpack will package a bundle (for example, bundle.js) based on the entry. Finally, we can import the processed bundle.js into the page. At this time, bundle.js has added a lot of webpack's "private goods" in addition to the source code.

Let's briefly sort out the work involved in webpack:

Dependency analysis: First, when packaging, webpack will obtain the module dependencies based on the syntax analysis results. Simply put, in CommonJS, the submodules that the current module depends on are obtained based on the parsed require syntax.
Scope isolation and variable injection: For each module file, webpack will wrap it in a function. This can not only inject variables such as module and require, but also isolate the scope to prevent global pollution of variables.
Provide module runtime: Finally, in order to effectively execute require and exports, a set of runtime code is also needed to implement module loading, execution, export and other functions.

If you don't understand the above 2 and 3 items, you can learn about the module runtime design of webpack from this article.

3. Challenges we face

Without a build tool, running a module using CommonJS directly in the browser requires finding a way to accomplish the three tasks mentioned above:

Dependency Analysis
Scope isolation and variable injection
Providing module runtime

Solving these three problems is the core task of one-click.js. Let’s take a look at how to solve them one by one.

3.1. Dependency Analysis

This is a troublesome problem. If you want to load modules correctly, you must know the dependencies between modules accurately. For example, the three module files mentioned above - main.js depends on plus.js and divide.js, so when running the code in main.js, you need to ensure that plus.js and divide.js have been loaded into the browser environment. However, the problem is that without compilation tools, we naturally cannot automatically know the dependencies between modules.

For a module library like RequireJS, it declares the dependencies of the current module in the code, and then uses asynchronous loading plus callback. Apparently, the CommonJS specification does not have such an asynchronous API.

One-click.js uses a tricky but costly way to analyze dependencies - loading the module file twice. When the module file is loaded for the first time, a mock require method is provided for the module file. Whenever the module calls this method, it can know which submodules the current module depends on in require.

// main.js
const plus = require('./plus.js');
const divide = require('./divide.js');
console.log(minus(12, add(1, 2)));

For example, in main.js above, we can provide a require method like the following:

const recordedFieldAccessesByRequireCall = {};
const require = function collect(modPath) {
    recordedFieldAccessesByRequireCall[modPath] = true;
    var script = document.createElement('script');
    script.src = modPath;
    document.body.appendChild(script);
};

After main.js is loaded, it will do two things:

Record the submodules that the current module depends on;
Load the submodule.

In this way, we can record the dependencies of the current module in recordedFieldAccessesByRequireCall and load the submodules at the same time. Submodules can also be recursively operated until no new dependencies appear. Finally, integrating the recordedFieldAccessesByRequireCall of each module is our dependency relationship.

In addition, if we want to know which methods in the submodules are actually called by main.js, we can use Proxy to return a proxy object and count further dependencies:

const require = function collect(modPath) {
    recordedFieldAccessesByRequireCall[modPath] = [];
    var megaProxy = new Proxy(function(){}, {
        get: function(target, prop, receiver) {
            if(prop == Symbol.toPrimitive) {
                return function() {0;};
            }
            return megaProxy;
        }
    });
    var recordFieldAccess = new Proxy(function(){}, {
        get: function(target, prop, receiver) {
            window.recordedFieldAccessesByRequireCall[modPath].push(prop);
            return megaProxy;
        }
    });
    // ...some other processing return recordFieldAccess;
};

The above code will record the attributes used when you get the attributes of the imported module.

The loading of all the modules above is the first pass of what we call "loading twice", which is used to analyze dependencies. The second time, you just need to "reverse" load the module based on the dependency of the entry module. For example, if main.js depends on plus.js and divide.js, the actual loading order is plus.js->divide.js->main.js.

It is worth mentioning that during the first loading of all modules, these modules will basically report errors when executed (because the loading order of dependencies is wrong). We will ignore the execution errors and only focus on the analysis of dependencies. After getting the dependencies, reload all module files in the correct order. There is a more complete implementation in one-click.js. The method is called scrapeModuleIdempotent. The specific source code can be found here.

At this point you might be thinking: “This is a waste, loading every file twice.”

Indeed, this is also the tradeoff of one-click.js:

In order to make this work offline, One Click needs to initialize your modules twice, once in the background upon page load, in order to map out the dependency graph, and then another time to actually perform the module loading.

3.2. Scope Isolation

We know that modules have a very important feature - the scopes between modules are isolated. For example, for the following ordinary JavaScript script:

// normal script.js
var foo = 123;

When it is loaded into the browser, the foo variable will actually become a global variable and can be accessed through window.foo. This will also cause global pollution, and variables and methods between modules may conflict and overwrite each other.

In the NodeJS environment, due to the use of the CommonJS specification, when a module file like the one above is imported, the scope of the foo variable is only in the source module and will not pollute the global scope. In terms of implementation, NodeJS actually wraps the code in the module with a wrap function. As we all know, a function will form its own scope, thus achieving isolation.

NodeJS will package the source code files when requiring, and packaging tools such as webpack will rewrite the source code files during compilation (also similar packaging). Since one-click.js does not have a compilation tool, rewriting at compile time will definitely not work. So what should we do? Here are two common methods:

3.2.1. Dynamic code execution in JavaScript

One way is to obtain the text content in the script through a fetch request, and then implement dynamic code execution through new Function or eval. Here is an introduction using fetch+new Function:

Still using the above division module divide.js, with a little modification, the source code is as follows:

// When loaded as a script, this variable will become the global variable of window.outerVar, causing pollution var outerVar = 123;

module.exports = function (a, b) {
    return a / b;
}

Now let's implement scope shielding:

const modMap = {};
function require(modPath) {
    if (modMap[modPath]) {
        return modMap[modPath].exports;
    }
}

fetch('./divide.js')
    .then(res => res.text())
    .then(source => {
        const mod = new Function('exports', 'require', 'module', source);
        const modObj = {
            id: 1,
            filename: './divide.js',
            parents: null,
            children: [],
            exports: {}
        };

        mod(modObj.exports, require, modObj);
        modMap['./divide.js'] = modObj;
        return;
    })
    .then(() => {
        const divide = require('./divide.js')
        console.log(divide(10, 2)); // 5
        console.log(window.outerVar); // undefined
    });

The code is very simple. The core is to obtain the source code through fetch, construct it in a function through new Function, and "inject" some module runtime variables into it when calling. In order to ensure the smooth running of the code, a simple require method is also provided to implement module reference.

Of course, the above is a solution, but it does not work under the goal of one-click.js. Because one-click.js also aims to run without a server (offline), fetch requests are invalid.

So how does one-click.js handle this? Let’s take a look at the following:

3.2.2. Another way to isolate scopes

Generally speaking, the need for isolation is very similar to that of a sandbox, and a common way to create a sandbox on the front end is with an iframe. For the sake of convenience, we call the window actually used by the user the "main window" and the iframe embedded in it the "sub-window". Due to the natural characteristics of iframe, each child window has its own window object and is isolated from each other, so it will not pollute the main window or each other.

The following still takes loading the divide.js module as an example. First we construct an iframe to load the script:

var iframe = document.createElement("iframe");
iframe.style = "display:none !important";
document.body.appendChild(iframe);
var doc = iframe.contentWindow.document;
var htmlStr = `
    <html><head><title></title></head></body>
    <script src="./divide.js"></script></body></html>
`;
doc.open();
doc.write(htmlStr);
doc.close();

This way you can load module scripts in an "isolated scope". But it obviously doesn't work properly yet, so the next step is to complete its module import and export functions. The problem that module export needs to solve is to allow the main window to access the module object in the subwindow. So we can mount the script in the child window to the variable of the main window after it is loaded and run.

Modify the above code:

// ...omit repeated code var htmlStr = `
    <html><head><title></title></head></body>
    <scrip>
        window.require = parent.window.require;
        window.exports = window.module.exports = undefined;
    </script>
    <script src="./divide.js"></script>
    <scrip>
        if (window.module.exports !== undefined) {
            parent.window.modObj['./divide.js'] = window.module.exports;
        }
    </script>
    </body></html>
`;
// ...omit repeated code

The core is to achieve "penetration" between the main window and the child window through methods like parent.window:

Mount the subwindow's object onto the main window;
It also supports the function of the sub-window calling the methods in the main window.

The above is just a rough implementation of the principle. If you are interested in more rigorous implementation details, you can see the loadModuleForModuleData method in the source code.

It is worth mentioning that in "3.1. Dependency Analysis" it is mentioned that all modules should be loaded once to obtain dependencies, and this part of the loading is also carried out in the iframe, which also needs to prevent "pollution".

3.3. Providing module runtime

The runtime version of a module includes constructing a module object, storing the module object, and providing a module import method (require). The various implementations of module runtime are generally similar. What needs to be noted here is that if the isolation method uses iframe, some runtime methods and objects need to be passed between the main window and the subwindow.

Of course, the details may also require support for module path resolution, circular dependency processing, error handling, etc. Since the implementation of this part is similar to many libraries, or is not particularly core, it will not be introduced in detail here.

4. Conclusion

Finally, let's summarize the general operation process:

1. First, get the entry module from the page. In one-click.js, it is document.querySelector("script[data-main]").dataset.main;

2. Load the module in the iframe and collect module dependencies in the require method until no new dependencies appear;

3. After the collection is completed, the complete dependency graph is obtained;

4. According to the dependency graph, "reverse" load the corresponding module files, use iframe to isolate the scope, and pay attention to passing the module runtime in the main window to each sub-window;

5. Finally, when the entry script is loaded, all dependencies are ready and can be executed directly.

In general, without the help of build tools and servers, it is difficult to implement dependency analysis and scope isolation. One-click.js solves these problems using the technical means mentioned above.

So, can one-click.js be used in production environments? Obviously not.

Do not use this in production. The only purpose of this utility is to make local development simpler.

So pay attention, the author also said that the purpose of this library is only to facilitate local development. Of course, we can also learn about some of these technical means as learning materials. Interested friends can visit the one-click.js repository to learn more.

The above is the details of implementing CommonJS modularization in browsers without compilation/server. For more information about implementing CommonJS modularization without compilation/server, please pay attention to other related articles on 123WORDPRESS.COM!

You may also be interested in:

Understanding the differences between CommonJS and ES Modules using webpack
Node's module specification for CommonJS
Understanding CommonJS module loading
Differences between module handling in ES6 and CommonJS
Detailed discussion of the differences between commonjs modules and es6 modules
In-depth understanding of Commonjs specifications and Node module implementation
Use Browserify to implement CommonJS browser loading method
Share a super useful JavaScript packaging and compression tool
Detailed explanation of how to use webpack to package JS

<<: A brief discussion on MySQL index optimization analysis

>>: Detailed explanation of identifying files with the same content on Linux