Detailed explanation of Nodejs array queue and forEach application

This article mainly records the problems and solutions caused by array characteristics encountered in the Nodejs development process, as well as the flexible application of arrays.

The test results of this article are based on node v6.9.5

Arrays and Queues

The push/shift method of the array object can be used to implement the first-in-first-out feature of the queue, for example:

>a=[]
[]
>a.push(2.3.4)
3
>a.push(2)
3
>a
[2.3.4.2]
>a.shift()
2
>a
>[3.4.2]

Arrays and forEach

There are two common ways to delete an array: delete and using the splice method. The difference between them needs to be made clear.

If you want to completely remove an element from an array, use splice:

> a=[1,2,3]
[ 1, 2, 3 ]
> a.splice(1,1)
[ 2 ]
> a
[ 1, 3 ]
> a.length
2
> a.forEach(function(item, index){console.info("index[", index,"]:", item)});
index[ 0 ]: 1
index[ 1 ]: 3
undefined
>

So, what is the effect of executing forEach after deleting an element object using delete?

forEach's processing mechanism for arrays containing empty elements

The test results are as follows

> a=[1,2,3]
[ 1, 2, 3 ]
> delete a[1]
true
> a
[ 1, , 3 ]
> a.length
3
> a.forEach(function(item, index){console.info("index[", index,"]:", item)});
index[ 0 ]: 1
index[ 2 ]: 3
undefined

From the test results, forEach does not traverse any item whose value is undefined. In practical applications, it is a big challenge to determine whether forEach has ended.

To solve the asynchronous feature application of forEach, you can add a prototype to the array to manage and set valid data by yourself;

The effect is as follows:

> a=[1,2,3]
[ 1, 2, 3 ]
> a.validnum=3
3
> delete a[2]
true
> a.validnum=2
2
> a
[ 1, 2, , validnum: 2 ]
> a.length
3
> a.validnum
2
> a.forEach(function(item, index){console.info("index[", index,"]:", item)});
index[ 0 ]: 1
index[ 1 ]: 2
undefined
>

Supplement: Node.js array forEach synchronously processes context statements

I am used to the C language way of thinking, and when I first came into contact with Node.js, its asynchronous processing gave me a headache.

When writing code, you may encounter a scenario where you need to process the elements in an array in a loop and then execute a last operation after all the processing is completed. However, the asynchronous nature of JS will cause this last statement to execute first, so take some time to study forEach.

Talk is cheap. Show me the code.

forEach Usage

forEach is used to traverse the array structure. I saw someone say that forEach is implemented using for at the bottom layer. I didn’t study it in depth, but at least the effect seems to be the same. The three parameters of forEach's callback function are: value, sequence number and original array. The sequence number starts from 0.

(() => {
  let arr = [2, 3, 1];
  arr.forEach(function (value, index, array) {
    console.log(value);
    console.log(index);
    console.log(array);
    console.log('-----');
  });
})();

Output

2
0
[ 2, 3, 1 ]
-----
3
1
[ 2, 3, 1 ]
-----
1
2
[ 2, 3, 1 ]
-----

From the results, it can be seen that the multiple loops of forEach are synchronized, that is, they are executed in sequence. But when I think about it being JS, it feels impossible to synchronize. . You can verify it.

forEach asynchronously processes multiple loops

This time, a timer task is added to forEach, and each loop operation is delayed by the time related to value to simulate a more time-consuming operation.

(() => {
  let arr = [2, 3, 1];
  arr.forEach(function (value, index, array) {
    setTimeout(function () {
      console.log(value);
    }, value*100);
  });
})();

Output

1
2
3

From the results, we can see that the task with the shortest time is completed first, and the tasks of each loop are not executed in the order of the loop, that is, multiple loops are processed asynchronously.

forEach context is also executed asynchronously

Back to the problem mentioned at the beginning, regardless of whether the multiple loops are executed in sequence, I need to execute a piece of data after all the tasks in forEach are completed to notify me that all the tasks are completed.

(() => {
  let arr = [2, 3, 1];
  arr.forEach(function (value, index, array) {
    setTimeout(function () {
      console.log(value);
    }, value*100);
  });
  console.log('All the work is done');
})();

Output

All the work is done
1
2
3

From the results, the context statements are not synchronized either. The task in the forEach loop is not completed before notifying that all tasks are completed, which is obviously not in line with expectations.

I read many blogs about this problem, but couldn't find a suitable solution. Finally, I could only use Promise.all to barely implement this function.

Promise.all implements synchronous processing of forEach context statements

Change the above code to the Promise.all structure. At the end of each loop, resolve() is called. We know that the then function of Promise.all will only be triggered when all Promises are executed, which seems to meet our needs.

(() => {
  let arr = [2, 3, 1];
  let proArr = [];
  arr.forEach(function (value, index) {
    proArr[index] = new Promise(function (resolve) {
      setTimeout(function () {
        console.log(value);
        resolve();
      }, value*100);
    });
  });
  Promise.all(proArr).then(()=>{
    console.log('All the work is done');
  })
})();

Output

1
2
3
All the work is done

Judging from the results, our needs were met.

Possible problems

Thinking of the asynchronous characteristics of JS, I suddenly realized that there might be a problem with this method.

Here, each time forEach enters, the Promise array is assigned. This operation time should be very short. The final Promise.all statement is called only after the assignment is completed in the three loops.

However, if the array is very large and the loop assignment operation is very time-consuming, if only half of the assignment operation is completed, then the Promise array passed in when the last Promise.all is executed may not be the array containing all the Promises.

In this case, Promise.all will only wait for half of the operations. When Promise.all is waiting, it is unknown whether the Promise assigned to the back of the array will be waited for.

I have just started using JS and don’t understand the implementation mechanism, so I can only experiment to verify whether this problem exists. Next, let's make this array bigger. Please forgive me for using the most fool-proof way to make it bigger.

(() => {
  let arr = [2, 3, 1, 2, 3, 1, 2, 3, 1, 2]; // 10
  arr = arr.concat(arr); // 2^1 * 10
  arr = arr.concat(arr); // 2^2 * 10
  arr = arr.concat(arr); // 2^3
  arr = arr.concat(arr); // 2^4
  arr = arr.concat(arr); // 2^5
  arr = arr.concat(arr);
  arr = arr.concat(arr);
  arr = arr.concat(arr);
  arr = arr.concat(arr);
  arr = arr.concat(arr); // 2^10
  arr = arr.concat(arr);
  arr = arr.concat(arr);
  arr = arr.concat(arr);
  arr = arr.concat(arr);
  arr = arr.concat(arr); // 2^15
  arr = arr.concat(arr);
  arr = arr.concat(arr); // 2^17 * 10
// arr = arr.concat(arr); // 2^18 * 10
  console.log(arr.length);
  let proArr = [];
  arr.forEach(function (value, index) {
    proArr[index] = new Promise(function (resolve) {
      setTimeout(function () {
        console.log(value);
        resolve();
      }, value*100);
    });
  });
  Promise.all(proArr).then(()=>{
    console.log('All the work is done');
    console.log(arr.length);
  }).catch(function (err) {
    console.log(err);
  })
})();

After testing on my computer, when the array length is 2^18 * 10, Promise reports an error RangeError: Too many elements passed to Promise.all.

When the array length is 2^17 * 10, or 2621440, it will run normally. After testing several times, the last executed command output All the work is done is always output at the end (because the terminal buffer is too small, the output result is redirected to a file for viewing using node xx.js > log.txt redirection).

Of course, there will not be such a large array in the application. Judging from the results, the above problems do not exist in actual applications.

That is to say, Promise.all can be used to implement synchronous processing of forEach context statements.

The above is my personal experience. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM. If there are any mistakes or incomplete considerations, please feel free to correct me.