Smooth Upgrade: Implementation and Usage
Tags: opensource
Published: 2024-05-10 07:08:50

In the past, upgrading JuiceFS versions or modifying mount parameters required users to remount the file system. This causes temporary interruptions in service. Once the underlying file system service is interrupted, it affects the normal operation of other components in the system. To avoid such issues, we’ve introduced the smooth upgrade feature in the [latest release of JuiceFS 1.2](https://github.com/juicedata/juicefs/releases/tag/v1.2.0-beta1). This allows users to restart or upgrade without service downtime.
In this post, we’ll share how we implemented the smooth upgrade feature and how to use it.

## Implementation
### FUSE request handling process
To upgrade without interrupting read and write requests, it's essential to understand the entire process of user read/write requests.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rovd4q6yd5wsn5c4l1xq.png)



As shown above, when a user uses `juicefs mount xxxxx /jfs` and opens `/jfs/a.txt`, the main behaviors of JuiceFS and the operating system are as follows:

1. After the user runs `juicefs mount`, the `go-fuse` module of JuiceFS opens `/dev/fuse` to obtain `mount fd`.
2. The user initiates an `open` system call, which enters the virtual file system (VFS) layer. The VFS layer then forwards the request to the FUSE layer.
3. The FUSE layer writes the `open` request into `mount fd` in the corresponding format according to the protocol.
4. The `go-fuse` module reads `mount fd` and calls the corresponding implementation in JuiceFS after parsing the request.
5. `go-fuse` writes the processing result of this request into `mount fd` according to the format.
6. The FUSE module in kernel space reads `mount fd` to obtain the processing result of this request.

Through the example process above, you can see that the `/dev/fuse` device file serves as the communication bridge between the JuiceFS user-space file system and the kernel. The two communicate via the file handle `mount fd` obtained when the `go-fuse` module opens the `/dev/fuse` device file during executing the `juicefs mount` command. The kernel writes requests and JuiceFS uses `go-fuse` to process and respond to requests. Their relationship can be viewed as a client-server protocol, where the kernel acts as the client requesting service, and the user-space JuiceFS plays the role of the server that handles these requests.

### How to pass file handles (mount fd) between processes

By analyzing the FUSE request handling process, it's evident that if we can ensure kernel requests don't get lost or encounter errors, we can smoothly complete the upgrade of JuiceFS. Even if requests are delayed in processing, it’s acceptable.

The key here lies in maintaining the consistency of the file handle (`mount fd`) obtained during the initial mount. The new process must inherit and use the `mount fd` of the old process, because from the kernel's perspective, nothing has changed. Therefore, the process is transparent to the application.

Note that every time the FUSE device is opened, a new session is allocated. A new session cannot access resources opened in previous sessions. Therefore, to achieve a smooth upgrade, we must maintain the old `mount fd`.

Before delving into the specifics of how to achieve `mount fd` consistency, it's essential to understand that starting from version 1.2, JuiceFS has been split into two processes: a daemon process and a service process. When the service process exits abnormally, the daemon restarts the service process. During this process, read-only requests remain unaffected, but write requests fail. During smooth upgrades, between new and old daemon processes, and between the daemon and service processes, we use [Unix domain sockets](https://en.wikipedia.org/wiki/Unix_domain_socket) to pass file handles.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/axgzn57vzzt83f70ko0o.png)




The figure above shows how `mount fd` and status information are transmitted during a smooth upgrade. The specific steps, labeled from 1 to 7, are marked accordingly in the figure.

1. The old service process obtains `mount fd` by opening the `/dev/fuse` device during the initial mount.
2. The old service process passes `mount fd` to the old daemon process.
3. The new daemon process obtains `mount fd` from the old daemon process.
4. The new daemon process sends a SIGHUP signal to the old service process. At this point, the old service process stops receiving new FUSE requests and flushes the data already written to the object storage. If certain requests are not completed within 3 seconds, they will be interrupted.
5. Upon receiving the SIGHUP signal, the old service process generates an intermediate status file in the `/tmp` directory.
6. The new service process loads the intermediate status file.
7. The new service process obtains `mount fd` from the new daemon process.

During the upgrade process, JuiceFS persists all newly written data in memory to the object storage and saves the open file handles to disk. After the new process starts, it restores these open files, ensuring that subsequent FUSE operations proceed without issues. If any FUSE request fails to complete during the upgrade, it will be forcibly interrupted. This may impact the application. It's recommended to perform upgrade operations during periods of low workload.

Following the steps above, the new JuiceFS service process obtains the `mount fd` previously used by the old JuiceFS. The new service process can then handle requests read from the `mount fd`. The workflow is as follows:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nre31qk6kg3f5io3tuqb.png)



## Usage
To use the smooth upgrade feature, you only need to use the new binary or parameters and remount.

### Client upgrade
For example, if there is a running mount process `juicefs mount redis://127.0.0.1:6379/0 /mnt/jfs -d`, and you want to deploy a new JuiceFS client without unmounting the mount point, you can follow these steps:

 1.Back up the current binary:

```bash
cp juicefs juicefs.bak
```

 2.Download the new binary to replace the current `juicefs` binary.

 3.Use the `juicefs mount` command to complete the smooth upgrade:

```bash
juicefs mount redis://127.0.0.1:6379/0 /mnt/jfs -d
```

### Adjust mount parameters dynamically
In the process above, if you want to adjust the log level to `debug` without unmounting the mount point, you can execute the following command:

```bash
# Adjust the log level.
juicefs mount redis://127.0.0.1:6379/0 /mnt/jfs --debug -d
```

## Note

1. Currently, smooth upgrade in v1.2-beta1 is only supported on Linux platforms. When v1.2 is officially released, macOS platforms will be supported simultaneously.
2. Smooth upgrade requires the JuiceFS client versions of both the new and old processes to be at least v1.2.
4. The FUSE parameters in the new mount parameters should remain consistent with the old mount parameters. Otherwise, the smooth upgrade will continue to overwrite the mount on the current mount point.
4. During the upgrade process, FUSE requests that are not completed within 3 seconds may be interrupted. Some applications may be affected if they cannot automatically retry interrupted file system operations.

Feel free to download and try out [JuiceFS v1.2.0-beta1](https://github.com/juicedata/juicefs/releases/tag/v1.2.0-beta1). If you have any feedback or suggestions, you can join [JuiceFS discussions on GitHub](https://github.com/juicedata/juicefs/discussions) and our [community on Slack](https://juicefs.slack.com/ssb/redirect).