Inside Node.js: Exploring Asynchronous I/O
Tags: node, javascript, programming, linux
Published: 2023-12-11 18:22:39

- [Introduction](#introduction)
- [How Node Handles Asynchronous Code](#how-node-handles-asynchronous-code)
- [Asynchronous Operations: What Are They?](#asynchronous-operations-what-are-they)
   * [Blocking vs Non-Blocking Asynchronous Operation](#blocking-vs-non-blocking-asynchronous-operation)
- [Experiments with Blocking Functions](#experiments-with-blocking-functions)
- [Experiments with Non-Blocking Functions](#experiments-with-non-blocking-functions)
- [Non-Blocking Asynchronous Operations and OS](#non-blocking-asynchronous-operations-and-os)
   * [Understanding File Descriptors](#understanding-file-descriptors)
     + [What is a FD?](#what-is-a-fd)
     + [FD and Non-Blocking I/O](#fd-and-non-blocking-io)
   * [Monitoring FDs with syscalls](#monitoring-fds-with-syscalls)
     + [Understanding select](#understanding-select)
     + [Epoll](#epoll)
     + [io_uring](#io_uring)
- [Conclusion](#conclusion)

## Introduction
Recently, I've been studying asynchronous code execution in Node.js.

I ended up learning (and writing) a lot, from an article about [how the Event Loop works](https://dev.to/ocodista/javascript-event-loop-breaking-down-the-mystery-2c9f) to a Twitter thread explaining [who waits for the http request to finish](https://twitter.com/ocodista/status/1696684507917631841?s=20).

If you want, you can also access the mind map I created before writing this post by clicking [here](https://whimsical.com/node-async-CzpmdNE7HMzsp5uPDeJpve).

Now, let's get to the point!

## How Node Handles Asynchronous Code
In Node:

- All JavaScript code is executed in the main thread.
- The **libuv** library is responsible for handling I/O (In/Out) operations, i.e., **asynchronous** operations.
- By default, libuv provides 4 _worker threads_ for Node.js.
  - These threads will only be used when **blocking** asynchronous operations are performed, in which case they will block one of the libuv threads (which are OS threads) instead of the main Node execution thread.
- There are both blocking and non-blocking operations, and most of the current asynchronous operations are **non-blocking**.

## Asynchronous Operations: What Are They?
Generally, there's confusion when it comes to asynchronous operations.

Many believe it means something happens in the background, in parallel, at the same time, or in another thread.

In reality, an asynchronous operation is an operation that won't return now, but later.

They depend on communication with external agents, and these agents might not have an immediate response to your request.

We're talking about I/O (input/output) operations.

Examples:
- **Reading a file**: data _leaves_ the disk and _enters_ the application.
- **Writing to a file**: data _leaves_ the application and _enters_ the disk.
- **Network Operations**
  - HTTP requests, for example.
  - The application **sends** an _http request_ to some server and **receives** the data.


![Node calls libuv, libuv calls syscalls, event loop runs on the main thread](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v4ve8g8hmkyilvezttqw.png)

### Blocking vs Non-Blocking Asynchronous Operation
In the modern world, ~~people don't talk to each other~~ most asynchronous operations are non-blocking.

But wait, does that mean:
- libuv provides 4 threads (_by default_).
- they "take care of" the **blocking** I/O operations.
- the vast majority of operations are **non-blocking**.

Seems kind of useless, right?


![Libuv worker threads for blocking asynchronous ops](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ch6qfruusrpzgmm8cyls.png)

With this question in mind, I decided to do some experiments.

## Experiments with Blocking Functions
First, I tested an asynchronous CPU-intensive function, one of the **rare** asynchronous **blocking** functions in Node.

The used code was as follows:
```javascript
// index.js
import { pbkdf2 } from "crypto";

const TEN_MILLIONS = 1e7;

// CPU-intensive asynchronous function
// Goal: Block a worker thread
// Original goal: Generate a passphrase
// The third parameter is the number of iterations
// In this example, we are passing 10 million
function runSlowCryptoFunction(callback) {
  pbkdf2("secret", "

salt", TEN_MILLIONS, 64, "sha512", callback);
}

// Here we want to know how many worker threads libuv will use
console.log(`Thread pool size is ${process.env.UV_THREADPOOL_SIZE}`);

const runAsyncBlockingOperations = () => {
  const startDate = new Date();
  const runAsyncBlockingOperation = (runIndex) => {
    runSlowCryptoFunction(() => {
      const ms = new Date() - startDate;
      console.log(`Finished run ${runIndex} in ${ms/1000}s`);
    });
  }
  runAsyncBlockingOperation(1);
  runAsyncBlockingOperation(2);
};

runAsyncBlockingOperations();
```
To validate the operation, I ran the command:
```bash
UV_THREADPOOL_SIZE=1 node index.js
```

**IMPORTANT:**
- UV_THREADPOOL_SIZE: It's an environment variable that determines how many libuv _worker threads_ Node will start.

The result was:
```bash
Thread pool size is 1
Finished run 1 in 3.063s
Finished run 2 in 6.094s
```
That is, with only one thread, each execution took ~3 seconds, and they occurred sequentially. One after the other.

Now, I decided to do the following test:
```bash
UV_THREADPOOL_SIZE=2 node index.js
```

And the result was:
```bash
Thread pool size is 2
Finished run 2 in 3.225s
Finished run 1 in 3.243s
```

With that, it's proven that LIBUV's _Worker Threads_ in Node.js handle blocking asynchronous operations.

But what about the **non-blocking** ones? If no one waits for them, how do they work?

I decided to write another function to test it.

### Experiments with Non-Blocking Functions
The _fetch_ function (native to Node) performs a non-blocking network asynchronous operation.

With the following code, I redid the test of the first experiment:

```javascript
//non-blocking.js
// Here we want to know how many worker threads libuv will use
console.log(`Thread pool size is ${process.env.UV_THREADPOOL_SIZE}`);

const startDate = new Date();
fetch("https://www.google.com").then(() => {
  const ms = new Date() - startDate;
  console.log(`Fetch 1 returned in ${ms / 1000}s`);
});

fetch("https://www.google.com").then(() => {
  const ms = new Date() - startDate;
  console.log(`Fetch 2 returned in ${ms / 1000}s`);
});
```

And I executed the script with the following command:
```bash
UV_THREADPOOL_SIZE=1 node non-blocking.js
```

The result was:
```bash
Thread pool size is 1
Fetch 1 returned in 0.391s
Fetch 2 returned in 0.396s
```

So, I decided to test with two threads, to see if anything changed:

```bash
UV_THREADPOOL_SIZE=2 node non-blocking.js
```

And then:
```
Thread pool size is 2
Fetch 2 returned in 0.402s
Fetch 1 returned in 0.407s
```

With this, I observed that:

> Having more threads running in LIBUV does not help in the execution of non-blocking asynchronous operations.

But then, I questioned again, if no libuv thread is "waiting" for the request to return, how does this work?

My friend, that's when I fell into a gigantic hole of research and knowledge about the operation of:

### Non-Blocking Asynchronous Operations and OS
The Operating System has evolved quite a bit over the years to deal with non-blocking I/O operations, this is done through _syscalls_, they are:

- **select/poll**: These are the traditional ways of dealing with non-blocking I/O and are generally considered less efficient.
- **IOCP**: Used in Windows for asynchronous operations.
- **kqueue**: A method for MacOS and BSD.
- **epoll**: Efficient and used in Linux. Unlike select, it is not limited by the number of FDs.
- **io_uring**: An evolution of epoll, bringing performance improvements and a queue-based approach.

To understand better, we need to dive into the details of non-blocking I/O operations.

### Understanding File Descriptors
To explain non-blocking I/O, I need to quickly explain the concept of File Descriptors (FDs).

#### What is a FD?
It's a numerical index of a table maintained by the _kernel_, where each record has:

- Resource type (such as file, socket, device).
- Current position of the file pointer.
- Permissions and flags, defining modes like read or write.
- Reference to the resource's data structure in the kernel.

They are fundamental for I/O management.

#### FD and Non-Blocking I/O
When initiating a non-blocking I/O operation, Linux associates an FD with it without interrupting (blocking) the process's execution.

For example:

Imagine you want to read the contents of a very large file.

**Blocking approach**:
- The process calls the _read file_ function.
- The process waits while the OS reads the file's content.
  - The process is **blocked** until the OS finishes.

**Non-blocking approach**:
- The process requests _asynchronous_ read.
- The OS starts reading the content and returns an FD to the process.
- The process isn't locked up and can do other things.
- Periodically, the process calls a _syscall_ to check if the reading is finished.

The process decides the mode of reading through the [fcntl](https://man7.org/linux/man-pages/man2/fcntl.2.html) function with the _O_NONBLOCK_ flag, but this is secondary at the moment.

### Monitoring FDs with syscalls
To efficiently observe multiple FDs, OSs rely on some _syscalls_:

#### **Understanding select:**
- Receives a list of FDs.
- Blocks the process until one or more FDs are ready for the specified operation (read, write, exception).
- After the syscall returns, the program can iterate over the FDs to identify those ready for I/O.
- Uses a search algorithm that is O(n).
  - Inefficient, slow, tiresome with many FDs.

#### **Epoll**
An evolution of _select_, it uses a self-balancing tree to store the FDs, making access time almost constant, O(1).

Pretty fancy!

How it works:
- Create an epoll instance with `epoll_create`.
- Associate the FDs with this instance using `epoll_ctl`.
- Use `epoll_wait` to wait for activity on any of the FDs.
- Has a timeout parameter.
  - Extremely important and well utilized by the libuv _Event Loop_!

![Comparison of time between select and epoll](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ysxrok3wm49fk52xlgwv.png)

#### **Io_uring**
This is a game-changer.

While **epoll** significantly improved the performance of searching and handling FDs, io_uring rethinks the entire nature of I/O operations.

And so, after understanding how it works, I wondered how nobody thought of this before!!!

Recapping:
- **select**: Receives a list of FDs, stores them sequentially (like an array), and checks each one for changes or activity, with complexity O(n).
- **epoll**: Receives a list of FDs, stores them using a self-balancing tree, does not check each one individually, is more efficient, and does the same as **select** but with complexity O(1).

Historically, the process was responsible for iterating over the returned FDs to know which have finished or not.

- **io_uring**: What? Return a list? Do polling? Are you kidding? Ever heard of **queues**?

It works using two main queues, in the form of rings (hence the name io-**ring**).

- 1 for submitting tasks.
- 1 for completed tasks.

Simple, right?

The process, when starting an I/O operation, **queues** the operation using the _io_uring_ structure.

Then, instead of calling _select_ or _epoll_ and iterating over the returned FDs, the process can choose to be notified when an I/O operation is completed.

Polling? No. Queues!

### Conclusion
With this knowledge, I now understand precisely the path Node takes to perform an asynchronous operation.

If it's blocking:
- Executes the asynchronous operation using libuv.
- Adds it to a libuv _worker thread_.
- The _worker thread_ is blocked, waiting for the operation to finish.
- Once finished, the thread is responsible for placing the result in the _Event Loop_ in the MacroTasks queue.
- The callback is executed on the main thread.

If it's non-blocking:
- Executes the asynchronous operation using libuv.
- Libuv performs a non-blocking I/O _syscall_.
- Performs _polling_ with the FDs until they resolve (epoll).
- From version 20.3.0, uses io_uring.
  - Queue-based approach for submission/completed operations.
- Upon receiving the event of operation completion:
  - libuv takes care of executing the callback on the main thread.