Repository: apache/rocketmq

Issue #4803: isSpaceToDelete method is not robust enough when some config change
Created: 2022-08-09T12:45:37Z
isSpaceToDelete method is not robust enough when some config change

Duplicate issue #4824:  Support consumequeue, indexfile multiple directories storage  mode
Created: 2022-08-15T03:55:31Z
In RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support, CommitLog File has supported multi-directories storage mode.
In this case, the commitlog can be stored on multiple disks, but consumequeue/index is still stored on one disk. However, the size of consumequeue/index can reach tens of Gbs, or even 100 Gbs, and the space used can't be easily ignored.
The current commitLog multipath needs to be configured independently, as follows:
storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog
Therefore, I optimized the multi-path mode, and the improvements are as follows:
● Add multi-path mode for consumequeue.
● Add multi-path mode for indexfile.
● Optimize the method to get the commitlog path. It is not necessary to specify storePathCommitLog separately, and generate the storePathCommitLog from storePathRootDir.
● Add a Timed thread to check path capacity. When select store path, it will try to avoid writing to a nearly full disk.
● Loading  metadata in multi-path. By default, the config file is stored in the first path, and the config bak file is stored in the second path. For example: storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config. At startup, scan all paths and load the config file from the latest path of dataversion.（This ensures that when the path is changed, the latest version can be loaded）
● Loading checkpoint file and abort file in multi-path. The loading path of the checkpoint file and the abort file is the same as that of the config file.
Chinese Version / 中文版本
在RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support 当中，CommitLog文件已经支持了多文件路径的存储模式。
在这个情况下，commitLog可以被存储至多块磁盘当中，但是consumequeue和index依然存储于单盘上。然而consumequeue 和 index的文件大小也可能达到数十G或者上百G的大小。这样的存储空间并不能被忽略。
当前的commitLog的多路径需要被额外配置，如下所示：
storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog
因此，我对多路径模式做了一定的优化，改进主要如下：
● consumequeue的存储增加多路径模式。
● indexfile的存储增加多路径模式。
● 优化commitlog获取文件路径的方式。不必要单独设置storePathCommitLog，storePathCommitLog可以从storePathRootDir根目录下直接生成。
● 增加一个定时线程检查路径的容量。当选择写入的路径的时候，能尽量避免写入至快满的磁盘中。
● 多路径下的元数据加载。默认将config文件夹写入至第1个路径中，config文件夹的bak文件写入至第2个路径。例如：storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config。在启动的时候，扫描所有路径，选择dataversion最新的所在路径作为初始加载路径。（这样能确保路径被更改时，能够从最新的路径下加载）
● 加载checkpoint文件和abort文件。checkpoint文件和abort文件的加载路径与config路径保持一致。

Closing comment (2022-08-15T06:37:20Z): @shengminw
In your RIP:

So the deletion should be executed when any of the disks reached the threshold (set by diskMaxUsedSpaceRatio)

Is this a good idea? Attempting to delete expiring data is pretty expensive, which may compete IOPS with normal data persistence, causing unexpected latency spikes.
Another issue: similar to #4803 Watermarks measured in percent are inferior to those measured in kilobytes(megabytes) as Linux kernel does.  The reclamation of RocketMQ should follow the same way.