A lightweight kv storage engine based on Golang.
Published: 2022-03-19 09:43:12



### What is Bottle🤔?

> 👨🏻‍💻: Bottle is a lightweight kv storage engine based on LSM-TREE.

Project Home pages: [https://bottle.ibyte.me/](https://bottle.ibyte.me/)

### Project Introduction

The first thing to note is that Bottle is a KV embedded storage engine, not a KV database.

### Other

I am a novice, self-taught go language programming, and then realized this project through the bitcask paper, I hope you can give me a star, thank you, my English is not very good.

![hierarchy](https://tva1.sinaimg.cn/large/e6c9d24egy1gzfrmt7qo4j21c20u0tai.jpg)


> The function implementation of this project is completely based on the [`bitcask`](https://blog.ibyte.me/post/bitcask-kvbase/) paper. 

Interested friends can go and see this course. If you think it is good, you can give me a small star ⭐ Thank you.

### Install Module🚀

You just need to install the Bottle module in your project to use:

```shell
go get -u github.com/auula/bottle
```

### Bottle Source code🌲

```shell
.
├── LICENSE
├── README.md
├── bottle-logo.svg
├── bottle.go           # Main File
├── bottle_test.go      
├── encoding.go         # Data encoding module
├── encrypted.go        # Data encrypted module
├── encrypted_test.go
├── example             # Exmaple code
│   ├── config.yaml
│   ├── demo_exchange.go
│   └── demo_put_get.go
├── go.mod
├── go.sum
├── gopher-bottle.svg
├── hashed.go          # Hashed function
├── item.go            # Data log Entry
└── option.go          # Bottle Configure 

1 directory, 17 files

```

### Data Directory 📑

Since the bottle design is based on a single-process program, each storage instance corresponds to a `data` directory, `data` is the log merge structure data directory, and `index` is the `index` data version.

Log consolving structural data The current version is merged every time the data is started. The default is that all data files under the DATA data folder occupies the total and more than `1GB` of more than `1GB` will trigger a merge, and the data that is not used after the merger is discarded.

Of course, if the dirty data merge requirements are not reached, the data file is archived in the size of the configured, each data has a version number, and is set to read-only mount, the process work directory structure is as follows:

```shell
./testdata
├── data
│   └── 1.data
└── index
    ├── 1646378326.index
    └── 1646378328.index

2 directories, 3 files
```
When the storage engine starts working, the folder and files can only be operated by this process to ensure data security.

### Basic API ⚙

Below is a usage example of a basic data manipulation I wrote:

```go
package main

import (
	"fmt"
	"github.com/auula/bottle"
)

func init() {
	// Open a storage instance by default configuration
	err := bottle.Open(bottle.DefaultOption)
	// And handle possible errors
	if err != nil {
		panic(err)
	}
}

// Userinfo test data structure
type Userinfo struct {
	Name  string
	Age   uint8
	Skill []string
}

func main() {

	// PUT Data
	bottle.Put([]byte("foo"), []byte("66.6"))

	// If it is converted to string, it is a string
	fmt.Println(bottle.Get([]byte("foo")).String())

	// If there is no default, it is 0
	fmt.Println(bottle.Get([]byte("foo")).Int())

	// If it is not success, it is false.
	fmt.Println(bottle.Get([]byte("foo")).Bool())

	// If it is not success, it is 0.0
	fmt.Println(bottle.Get([]byte("foo")).Float())

	user := Userinfo{
		Name:  "Leon Ding",
		Age:   22,
		Skill: []string{"Java", "Go", "Rust"},
	}

	var u Userinfo

	// Save the data object by BSON, and set the timeout for 5 seconds
    // TTL timeout can not set demand
	bottle.Put([]byte("user"), bottle.Bson(&user), bottle.TTL(5))

	// Analysis of structures by unwrap
	bottle.Get([]byte("user")).Unwrap(&u)

	// Print value
	fmt.Println(u)

	// Remove key 'foo'
	bottle.Remove([]byte("foo"))

	// Close handleable error that may happen
	if err := bottle.Close(); err != nil {
		fmt.Println(err)
	}
}
```

The following is a separate variable assignment and catch handling errors:

```go
    data := bottle.Get([]byte("user"))

	if data.IsError() {
		fmt.Println(data.Err)
	} else {
		fmt.Println(data.Value)
	}
```

### Encryptor🔐

The data `encryptor` records the value of the data, that is, the block encryption at the field level, instead of encrypting the entire file once, which will cause performance consumption, so the block data segment encryption method is adopted.

The following example uses the [`bottle.SetEncryptor(Encryptor,[]byte)`](https://github.com/auula/bottle/blob/main/option.go#L66) function to set the data encryptor and configure the 16-bit data encryption key.

```go
func init() {
    bottle.SetEncryptor(bottle.AES(), []byte("1234567890123456"))
}
```

You can also customize the interface to implement the data encryptor:

```go
// SourceData for encryption and decryption
type SourceData struct {
    Data   []byte
    Secret []byte
}

// Encryptor used for data encryption and decryption operation
type Encryptor interface {
    Encode(sd *SourceData) error
    Decode(sd *SourceData) error
}
```

The following code is the implementation code of the built-in `AES` encryptor, just implement the [`bottle.Encryptor`](https://github.com/auula/bottle/blob/main/encrypted.go#21) interface, and the data source is the [`bottle.SourceData`](https://github.com/auula/bottle/blob/main/encrypted.go#15) structure field:

```go
// AESEncryptor Implement the Encryptor interface
type AESEncryptor struct{}

// Encode source data encode
func (AESEncryptor) Encode(sd *SourceData) error {
    sd.Data = aesEncrypt(sd.Data, sd.Secret)
    return nil
}

// Decode source data decode
func (AESEncryptor) Decode(sd *SourceData) error {
    sd.Data = aesDecrypt(sd.Data, sd.Secret)
    return nil
}
```
The specific encryptor implementation code can be viewed in [`encrypted.go`](https://github.com/auula/bottle/blob/main/encrypted.go)

### Hashed 🧰

If you need to customize the salad function, you can implement the [bottle.Hashed](https://github.com/auula/bottle/blob/main/hashed.go#10) interface:

```go
type Hashed interface {
    Sum64([]byte) uint64
}
```
Then complete your hash function configuration by built-in [`bottle.SetHashFunc(hash Hashed)`](https://github.com/auula/bottle/blob/main/option.go#85) settings.

### Index Size 🔎

The index pre-set size will greatly affect your program acception and read data. If you can expect the index size required to run during initialization, and in initialization, you can decreaseThe performance issues that run data migration and expansion brought by the program during operation.

```go
func init() {
    // setting indexes size
    bottle.SetIndexSize(1000)
}
```
### Configuration Information

You can also use the default configuration, you can initialize your storage engine with the structure of the built-in [`bottle.Option`](https://github.com/auula/bottle/blob/main/option.go#14), and the configuration instance is as follows:

```go
func init() {
    // Custom configuration information
    option := bottle.Option{
        // Working directory
        Directory:       "./data",
        // Algorithm opens encryption
        Enable:          true,
        // Customize the secret, you can use a built-in secret
        Secret:          bottle.Secret,
        // Custom data size, storage unit is KB
        DataFileMaxSize: 1048576,
    }
    // Custom configuration information
    bottle.Open(option)
}
```

Of course, you can also use the built-in [`bottle.Load(path string)`](https://github.com/auula/bottle/blob/main/bottle.go#157)  function to load the configuration file to start Bottle, the configuration file format is `YAML`, the configurable items are as follows:

```go
# Bottle config options
Enable: TRUE
Secret: "1234567890123456"
Directory: "./testdata"
DataFileMaxSize: 536870912
```

It is important to note that the key implemented by the built-in encryptor must be `16` bits, if you are a custom implementation encrypler to set your custom encryprators through [`bottle.SetEncryptor(Encryptor,[]byte)`](https://github.com/auula/bottle/blob/main/option.go#L66), then this secretThe number of digits will not be restricted.

### Vision 🎉

If you have any questions about this project, you can contact me from the GitHub home page, you can also initiate a `pull request`, after your code is merged, you will appear in the following list~