Compressed bitmaps indexes are used in databases and search engines. A wide range of bitmap compression techniques has been proposed, almost all relying primarily on run-length encoding (RLE), including BBC and WAH. However, on unsorted data, we can get superior performance with a hybrid compression technique that uses both uncompressed bitmaps and packed arrays inside a two-level tree. An instance of this technique, Roaring, has recently been proposed. Due to its good performance, it has been adopted by several production platforms (e.g., Apache Lucene, Apache Kylin and Druid). Yet there are cases where run-length encoded bitmaps are smaller than the original Roaring bitmapstypically when the data is sorted so that the bitmaps contain long compressible runs. To better cope with these cases, we build a new Roaring hybrid that combines uncompressed bitmaps, packed arrays and RLE compressed segments. The result is a new Roaring format that compresses better. Overall, our new implementation of Roaring can be several times faster (up to two orders of magnitude) than the implementations of traditional RLE-based alternatives (WAH, Concise, EWAH) while compressing better. We review the design choices and optimizations that make these good results possible.
More recently, Roaring has been extended to also support chunks of run-length encoded data REF .
Consistently faster and smaller compressed bitmaps with Roaring
Software: Practice and Experience Volume 46, Issue 11, pages 1547-1569, November 2016
Computer Science