YugabyteDB: fs_data_dirs
Tags: yugabytedb, configuration
Published: 2023-03-20 14:23:31

YugabyteDB, like every database, allows to specify the directory where the database files are stored. This parameter, or gflag, in YugabyteDB is `fs_data_dirs`. 

Of course the directory can also be a mount point for a filesystem, so that the Operating System and the binaries are completely independent from the database files.

## Tablet server and Master
The parameter `fs_data_dirs` is a setting for both the tablet server, as well as the master. Both store rocksdb database files. 

It is not a problem to have both the master and the tablet server running on the same machine/node, and it also is not a problem to have them both be pointing to the same directory with `fs_data_dirs`. They will create their individual, separate, `master` and `tserver` directories, where they will store and manipulate their files.

## `fs_data_dirs` contents
Inside the directory set by the `fs_data_dirs` parameter, the tablet server will create when freshly started:
- pg_data (only if `enable_ysql` is `true`)
- yb-data/tserver

The master server creates:
- yb-data/master

Inside the `yb-data/tserver` and `yb-data/master` directories, a directory structure is created which contains:
- wals: the YugabyteDB write ahead log.
- tablet-meta: tablet metadata.
- data: the rocksdb database directories.
- consensus-meta: RAFT/consensus metadata.
- logs: textual log files of the server and optionally of the PostgreSQL logfile.

The information that is stored for each tablet in the `tablet-metadata` directory links wal and rocksdb together, for which it is using the operating system paths. 

## ADDING a directory to `fs_data_dirs`
If you want to add another directory or path for storing data, you can do that by specifying a second directory separated by a comma without a space, such as: `--fs_data_dirs=/mnt/d0,mnt/d1`.

If you add a directory and restart the tablet server, it will create the directory `yb-data/tserver` in the path specified in `fs_data_dirs`, and create the directories `wals`, `tablet-meta`, `data` and `consensus-meta` in the created directory.

The first thing to notice is that the `logs` directory is not in the list of directories in `yb-data/tserver`. The reason is the logs will be stored in the first directory that the server finds in `fs_data_dirs`.

## How YugabyteDB works with multiple data directories
When multiple data directories are set in `fs_data_dirs`, there is no dynamic or automatic moving or load balancing that will happen.

If a directory is added to `fs_data_dirs` later on, it will be initialised, and the allocations in the first directory will stay where they are.

Also, when an object is created, and therefore its tablet(s) which require wal, tablet-meta, rocksdb and consensus-meta allocations, these will still be created in the first directory for the first tablet; only when a second tablet is specified, it will be allocated in the second directory, a third in the first directory (if two are specified), a fourth in the second directory again, and so on.

This means that if you are running with a filesystem mounted at the path specified in `fs_data_dirs`, and the filesystem is getting filled up, and it's impossible to extend the filesystem (if you can extend the partition that contains the filesystem, XFS allows online resizing), adding a second data directory is quite likely not the solution.

## REMOVING or CHANGING a directory in `fs_data_dirs`
Currently, we do NOT support removing or changing a directory specified in `fs_data_dirs`. The reason is that in the metadata, the full path is specified, and thus moving it will break the specified paths. 

If you must change a path, or do something else that forces you to change the paths, the best way of doing that is adding a tablet server node to the cluster with the correct configuration for `fs_data_dirs`, and blacklisting the problem tablet server. This will make the tablets to move away from the problem node, and the tablet replicas be spread over the cluster again.

## Conclusion
YugabyteDB stores metadata that contains the paths, so you cannot move YugabyteDB datafiles to another directory and add that to `fs_data_dirs` to circumvent a filesystem from filling up. 

Adding a directory to `fs_data_dirs` is possible, but this only will get tablets placed if multiple are specified during creation, and therefore is not a way to balance filesystem space usage.

If you run into problems with the path specified in `fs_data_dirs`, add a node or nodes to the cluster which are sufficiently configured to store the data, and blacklist the problem node or nodes to make them move the tablets. 

The idea of running YugabyteDB is to run simple reasonably small nodes with local (SSD) drives, and scale up the cluster, not the nodes. Also removing or changing a node should be a normal non-intrusive action, which is why dealing with `fs_data_dirs` in this way is sufficient.