Detailed explanation of asynchronous generators and asynchronous iterations in Node.js

Preface

Generator functions have been in JavaScript since before async/await was introduced, which means that there are a lot of things to be aware of when creating asynchronous generators (generators that always return a Promise and can be awaited).

Today, we’ll look at asynchronous generators and their close cousin, asynchronous iteration.

Note: Although these concepts should apply to all modern JavaScript implementations, all code in this article was developed and tested against Node.js versions 10, 12, and 14.

Asynchronous generator functions

Take a look at this little program:

// File: main.js
const createGenerator = function*(){
 yield 'a'
 yield 'b'
 yield 'c'
}

const main = () => {
 const generator = createGenerator()
 for (const item of generator) {
 console.log(item)
 }
}
main()

This code defines a generator function, creates a generator object using the function, and then iterates over the generator object using a for ... of loop. Pretty standard stuff - although you'd never use a generator for something this trivial in real life. If you are not familiar with generators and for ... of loops, please see the articles Javascript Generators and ES6 Loops and Iterables. Before using asynchronous generators, you need to have a solid understanding of generators and for...of loops.

Suppose we want to use await in our generator function. Node.js supports this feature as long as we declare the function with the async keyword. If you are not familiar with asynchronous functions, then take a look at the article Writing Asynchronous Tasks in Modern JavaScript.

Let's modify the program and use await in the generator.

// File: main.js
const createGenerator = async function*(){
 yield await new Promise((r) => r('a'))
 yield 'b'
 yield 'c'
}

const main = () => {
 const generator = createGenerator()
 for (const item of generator) {
 console.log(item)
 }
}
main()

Also, in real life, you wouldn't do this - you'd probably await a function from a third-party API or library. In order to make it easier for everyone to grasp, our examples are kept as simple as possible.

If you try to run the above program, you will encounter the problem:

$ node main.js
/Users/alanstorm/Desktop/main.js:9
 for (const item of generator) {
 ^
TypeError: generator is not iterable

JavaScript tells us that this generator is "not iterable". At first glance, it might seem that making a generator function asynchronous also means that the generator it produces is not iterable. This is a bit confusing, since the purpose of a generator is to produce objects that are "programmatically" iterable.

Next, figure out what happened.

Check the generator

If you look at the iterable objects of JavaScript generators [1]. An object implements the iterator protocol when it has a next method, and the next method returns an object with a value property, a done property, or both a value and a done property.

If we compare the generator objects returned by an asynchronous generator function with those returned by a regular generator function using the following code:

// File: test-program.js
const createGenerator = function*(){
 yield 'a'
 yield 'b'
 yield 'c'
}

const createAsyncGenerator = async function*(){
 yield await new Promise((r) => r('a'))
 yield 'b'
 yield 'c'
}

const main = () => {
 const generator = createGenerator()
 const asyncGenerator = createAsyncGenerator()

 console.log('generator:',generator[Symbol.iterator])
 console.log('asyncGenerator',asyncGenerator[Symbol.iterator])
}
main()

You will see that the former does not have a Symbol.iterator method, but the latter does.

$ node test-program.js
generator: [Function: [Symbol.iterator]]
asyncGenerator undefined

Both generator objects have a next method. If you modify the test code to call this next method:

// File: test-program.js

/* ... */

const main = () => {
 const generator = createGenerator()
 const asyncGenerator = createAsyncGenerator()

 console.log('generator:',generator.next())
 console.log('asyncGenerator',asyncGenerator.next())
}
main()

You will see another problem:

$ node test-program.js
generator: { value: 'a', done: false }
asyncGenerator Promise { <pending> }

In order for an object to be iterable, the next method needs to return an object with value and done properties. An async function will always return a Promise object. This feature applies to generators created with asynchronous functions - these asynchronous generators always yield a Promise object.

This behavior makes it impossible for async generators to implement the JavaScript iteration protocol.

Asynchronous iteration

Fortunately, there is a way to resolve this contradiction. If you look at the constructor or class returned by an async generator

// File: test-program.js
/* ... */
const main = () => {
 const generator = createGenerator()
 const asyncGenerator = createAsyncGenerator()

 console.log('asyncGenerator',asyncGenerator)
}

You can see that it is an object whose type or class or constructor is AsyncGenerator instead of Generator:

asyncGenerator Object [AsyncGenerator] {}

Although this object may not be iterable, it is asynchronously iterable.

For an object to be asynchronously iterable, it must implement a Symbol.asyncIterator method. This method must return an object that implements the asynchronous version of the iterator protocol. That is, the object must have a next method that returns a Promise, and that promise must eventually resolve to an object with done and value properties.

An AsyncGenerator object satisfies all of these conditions.

This leaves the question - how can we iterate over an object that is not iterable but can be iterated asynchronously?

for await … of loop

An asynchronous iterable can be manually iterated using only the generator's next method. (Note that the main function here is now async main - this allows us to use await inside the function)

// File: main.js
const createAsyncGenerator = async function*(){
 yield await new Promise((r) => r('a'))
 yield 'b'
 yield 'c'
}

const main = async () => {
 const asyncGenerator = createAsyncGenerator()

 let result = { done:false }
 while(!result.done) {
 result = await asyncGenerator.next()
 if(result.done) { continue; }
 console.log(result.value)
 }
}
main()

However, this is not the most straightforward looping mechanism. I don't like the while loop condition, nor do I want to manually check result.done. Additionally, the result.done variable must exist in the scope of both the inner and outer blocks.

Fortunately most (maybe all?) JavaScript implementations that support asynchronous iterators also support the special for await ... of loop syntax. For example:

const createAsyncGenerator = async function*(){
 yield await new Promise((r) => r('a'))
 yield 'b'
 yield 'c'
}

const main = async () => {
 const asyncGenerator = createAsyncGenerator()
 for await(const item of asyncGenerator) {
 console.log(item)
 }
}
main()

If you run the above code, you will see that the asynchronous generator and iterable object are successfully looped, and the fully resolved value of the Promise is obtained in the loop body.

$ node main.js
a
b
c

The for await ... of loop prefers objects that implement the asynchronous iterator protocol. But you can use it to iterate over any kind of iterable object.

for await(const item of [1,2,3]) {
 console.log(item)
}

When you use for await, Node.js will first look for the Symbol.asyncIterator method on the object. If it can't find one, it falls back to using the Symbol.iterator method.

Non-Linear Code Execution

Like await, the for await loop introduces non-linear code execution into your program. That is, your code will run in a different order than it was written.

When your program first encounters a for await loop, it will call next on your object.

The object will yield a promise, then execution of the code will leave your async function and your program execution will continue outside of that function.

Once your promise is resolved, code execution will return to the loop body with this value.

When the loop ends and it's time to proceed to the next trip, Node.js calls next on the object. This call will produce another promise and code execution will leave your function again. This pattern repeats until the Promise resolves to an object with done being true, and then execution continues with the code after the for await loop.

The following example illustrates this point:

let count = 0
const getCount = () => {
 count++
 return `${count}.`
}

const createAsyncGenerator = async function*() {
 console.log(getCount() + 'entering createAsyncGenerator')

 console.log(getCount() + 'about to yield a')
 yield await new Promise((r)=>r('a'))

 console.log(getCount() + 're-entering createAsyncGenerator')
 console.log(getCount() + 'about to yield b')
 yield 'b'

 console.log(getCount() + 're-entering createAsyncGenerator')
 console.log(getCount() + 'about to yield c')
 yield 'c'

 console.log(getCount() + 're-entering createAsyncGenerator')
 console.log(getCount() + 'exiting createAsyncGenerator')
}

const main = async () => {
 console.log(getCount() + 'entering main')

 const asyncGenerator = createAsyncGenerator()
 console.log(getCount() + 'starting for await loop')
 for await(const item of asyncGenerator) {
 console.log(getCount() + 'entering for await loop')
 console.log(getCount() + item)
 console.log(getCount() + 'exiting for await loop')
 }
 console.log(getCount() + 'done with for await loop')
 console.log(getCount() + 'leaving main')
}

console.log(getCount() + 'before calling main')
main()
console.log(getCount() + 'after calling main')

In this code, you use numbered logging statements, which allow you to track its execution. As an exercise, you should run the program yourself and see what the results are.

Asynchronous iteration is a powerful technique that can cause confusion in your program's execution if you don't know how it works.

Summarize

This is the end of this article about asynchronous generators and asynchronous iterations in Node.js. For more relevant content about asynchronous generators and asynchronous iterations in Node.js, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!