Compression @ SwayDB
Tags: java, kotlin, scala, storage
Published: 2020-03-25 12:13:44

---
title: Compression @ SwayDB
published: true
description: Compression configuration in SwayDB
tags: java, kotlin, scala, storage
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/7k6cxlderre2cxwe3xez.png
---

__Enabling full compression can reduce data size by 89.3%!__

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/dqg8jwbzwny2htdph95l.png)

Compression is __important__ for performance because more data can be stored & read into memory within a [block](https://en.wikipedia.org/wiki/Block_(data_storage)), it reduces IOps and the cost of hosting data on servers/cloud. But __over-compression__ can get expensive due to time required for decompression which could effect read performance but it can also be useful for applications that need __cold storage__ or just __high compression__.

A __configurable compression strategy__ is required to support __unique storage requirements__ allowing us to define __how much__ & __how often__ we want to compress data. For example: if your data file (Segment) is of size 100MB with 100K key-values you can express your compression requirements like

-   Compress the full 100MB data file.
-   Compress data every 4MB.
-   Compress data every 1000 key-values.
-   Compress all data but reset compression every 10<sup>th</sup>  key-value
-   Do not compress all data but compress every 10<sup>th</sup>  key-value.
-   Compress  `binary-search-indexes`  &  `hash-indexes`  but not other data-blocks.

__Note:__ __data-block__ is referred to a logical set of bytes (`Array<Byte>`) that are stored within a Segment like indexes, keys, values etc. A Segment itself is a data-block that stores other data-blocks within itself (`Array<Array<Byte>>`).

## Compression strategies

You can combine, enable or disable any or all of the following compression strategies

-   Internal-compression includes __prefix compression__ and __duplicate value elimination__.
-   External-compression uses __LZ4__ and/or __Snappy__ which can applied selectively to parts of a file or to the entire file.

## Prefix compression

Prefix compression stores all keys in a compressed group format in their sorted order. Reading a single key from that group requires decompressing all keys that exist before the searched key within the group, so for __read performance__ it is useful to leave some keys uncompressed. You can also prefix compress all keys if you just want __high compression__.

The following  `Map`  is configured to compress __4__ keys into a group and starts a new group every __5<sup>th</sup>__ key. The input  `boolean`  parameter named  `keysOnly`  is set to  `true`  which applies prefix-compression to keys only, if  `false`, it applies prefix-compression to keys and all __metadata__ that gets written with that key which would result in higher compression.

```java
Map<Integer, String, Void> map =  
  MapConfig  
    .functionsOff(Paths.get("myMap"), intSerializer(), stringSerializer())  
    .setSortedKeyIndex(  
      SortedKeyIndex  
        .builder()  
        .prefixCompression(new PrefixCompression.Enable(true, PrefixCompression.resetCompressionAt(5)))  
        ...
    )  
    .get();  

map.put(1, "one");  
map.get(1); //Optional[one]

```

In the following configuration prefix compression is applied to every __5<sup>th</sup>__ key.  

```java
.prefixCompression(new PrefixCompression.Enable(false, PrefixCompression.compressAt(5)))
```

Prefix compression can also be __disabled__ which optionally allows optimising sorted-index for direct binary-search without creating a dedicated binary-search byte array. You can read more about  `normaliseIndexForBinarySearch`  [here](https://swaydb.simer.au/configurations/sortedKeyIndex/?language=java/).  

```java
.prefixCompression(new PrefixCompression.Disable(false))
```

## Duplicate value elimination

__Time-series__ or __events data__ like weather, electricity, solar etc often contains duplicate values. Such duplicate values can be __detected and eliminated__ with the following configuration.  

```java
ValuesConfig  
  .builder()  
  .compressDuplicateValues(true)  
  .compressDuplicateRangeValues(true)
```

Duplicate value elimination is very __cost effective__ because it does not create or leave decompression markers on compressed data, instead all decompression information for that key is embedded within an already existing 1-2 bytes space.

Range values created by the range APIs like  [remove-range](https://swaydb.simer.au/api/write/remove-range/?language=java/),  [update-range](https://swaydb.simer.au/api/write/update-range/?language=java/)  &  [expire-range](https://swaydb.simer.au/api/write/expire-range/?language=java/)  are most likely to have duplicate values and can be eliminated/compressed with the  `compressDuplicateRangeValues(true)`  config.

## External compression

___Every data-block written into Segment file is compressible! A Segment file is nothing special but just another data-block that stores other data-blocks within itself___.

You will find a  `compression`  property that configures external compression in __all data-blocks__ that form a Segment - [SortedKeyIndex](https://swaydb.simer.au/configurations/sortedKeyIndex/?language=java/),  [RandomKeyIndex](https://swaydb.simer.au/configurations/randomKeyIndex/?language=java/),  [BinarySearchIndex](https://swaydb.simer.au/configurations/binarySearchIndex/?language=java/),  [MightContainIndex](https://swaydb.simer.au/configurations/mightContainKeyIndex/?language=java/),  [ValuesConfig](https://swaydb.simer.au/configurations/valuesConfig/?language=java/)  &  [SegmentConfig](https://swaydb.simer.au/configurations/segmentConfig/?language=java/).

All __LZ4__ instances and __Snappy__ are both supported.

The following snippet demos how to apply compression to a [SortedKeyIndex/Linear-search-index](https://swaydb.simer.au/configurations/sortedKeyIndex/?language=java/). It tries to compress with LZ4 first at minimum 20.0% compression savings, if the compression was lower than 20.0% then Snappy is tried with the same.  

```java
SortedKeyIndex
  .builder()
  .compressions((UncompressedBlockInfo info) ->
    Arrays.asList(
      //try running LZ4 with minimum 20.0% compression  
      Compression.lz4Pair(
        new Pair(LZ4Instance.fastestJavaInstance(), new LZ4Compressor.Fast(20.0)),
        new Pair(LZ4Instance.fastestJavaInstance(), LZ4Decompressor.fastDecompressor())
      ),
      //if not try Snappy  
      new Compression.Snappy(20.0)
    )
    ...
  ) 
```

`UncompressedBlockInfo`  provides the data size (`info.uncompressedSize()`) of the data-block being compressed which can optionally be used to determine if it should be compressed or not. For example: if the data size is already too small then you can disable compression by returning  `Collections.emptyList()`.  

```java
.compression(
  (UncompressedBlockInfo blockInfo) -> {
    if (blockInfo.uncompressedSize() < StorageUnits.mb(1)) {
      return Collections.emptyList();
    } else { //else do compression
      return {your compression};
    }
  }

```

__How to apply compression at a file Level?__ Similar to above you can apply file level compression with  `SegmentConfig`.  

```java
.setSegmentConfig(  
  SegmentConfig  
    .builder()  
    ...  
    .compression((UncompressedBlockInfo info) ->
       {your compression config here}
    )  
)

```

## How to limit compression by size & key-value count?

The property `minSegmentSize` sets the __compressible size__ of Segment and if the above `compression` property is defined for a data-block then that data gets compressed every `minSegmentSize` data block.

The property `maxKeyValuesPerSegment` also controls the compressible limit of a Segment which, along with `minSegmentSize` enables checks to limit the maximum __count/number__ of key-values to store within a compressible Segment.

```java
.setSegmentConfig(  
  SegmentConfig  
    .builder()  
    .minSegmentSize(StorageUnits.mb(4))
    .maxKeyValuesPerSegment(100000)
    ...
)
```

## Summary

SwayDB's compression is highly configurable and can be tuned for unique storage requirements with different tradeoffs.

## GitHub repos

- [SwayDB](https://github.com/simerplaha/SwayDB) on GitHub.
- [Java](https://github.com/simerplaha/SwayDB.java.examples) examples repo.
- [Kotlin](https://github.com/simerplaha/SwayDB.kotlin.examples) examples repo.
- [Scala](https://github.com/simerplaha/SwayDB.scala.examples) examples repo.
- [Documentation](https://swaydb.simer.au/).