Discovering and exploring mmap using Go
Tags: go, database, virtualmemory, mmap
Published: 2021-01-14 20:01:07

Recently I've come to know the concept of **memory-mapped files** while watching a lecture of the course [Intro to Database Systems](https://15445.courses.cs.cmu.edu/fall2019/) of [Andy Pavlo](https://twitter.com/andy_pavlo) on database storage. One of the main problems a database storage engine has to solve is **how to deal with data in disk that is bigger than the available memory**. At a higher level, the main purpose of a disk-oriented storage engine is to manipulate data files in a disk. But if we assume that the data in the disk will eventually get bigger than the available memory, we cannot simply load the whole data file into memory, do the change, and write it back to disk.

This is not a new problem in Computer Science. When operational systems were being developed in the early 1960s, a similar problem was faced: **how can we run programs stored in disk that are larger than the available memory?** A solution to this problem was made by a group in Manchester, implemented on the [Atlas Computer](https://en.wikipedia.org/wiki/Atlas_(computer)), in 1961. It was called *virtual memory*. The *virtual memory* gives a running program the illusion that it has big enough memory, despite the fact that the computer does not have enough.

We are not going to go deep on how *virtual memory* works. Just have in mind that when a program is accessing memory it is accessing the *virtual memory*. And maybe the data the program is trying to access is not actually in memory, but it does not matter. The operational system will make pretend that it is by going to disk, and putting it there, and replace an old chunk of memory that is not going to be used.

So, one of the ways a database storage engine can solve the larger than memory problem is to make use of *virtual memory* and the concept of **memory-mapped files**.

In Linux, we can make this use by using the system call [mmap](https://man7.org/linux/man-pages/man2/mmap.2.html) that lets you map a file, no matter how big, directly into memory. If your program needs to manipulate the file, all it needs is to manipulate the memory. The operating system handles the writes to disk for you.

In some occasions, programmers find this method more convenient than the usual system calls: [open](https://man7.org/linux/man-pages/man2/open.2.html), [read](https://man7.org/linux/man-pages/man2/read.2.html), [write](https://man7.org/linux/man-pages/man2/write.2.html), [lseek](https://man7.org/linux/man-pages/man2/lseek.2.html) and [close](https://man7.org/linux/man-pages/man2/close.2.html).

### A simple demonstration

Here is a small example of how you can take advantage of this in Go using the package [mmap-go](https://github.com/edsrzf/mmap-go):

    package main
    
    import (
    	"os"
    	"fmt"
    	"github.com/edsrzf/mmap-go"
    )
    
    func main() {
    	f, _ := os.OpenFile("./file", os.O_RDWR, 0644)
    	defer f.Close()
    	
    	mmap, _ := mmap.Map(f, mmap.RDWR, 0 )
    	defer mmap.Unmap()
    	fmt.Println(string(mmap))
    	
    	mmap[0] = 'X'
    	mmap.Flush()
    }

![asciicast](https://asciinema.org/a/pRS8PvTRHksnCVQgSOWvPBF3a.svg)
The beauty is that we could have a much bigger file, and the solution would still work. We would not have to worry about managing memory in order to avoid it filling up.

### Detailing *mmap* capabilites

We're going to explore more *mmap* functionalities from the point of view of the API provided by [mmap-go](https://github.com/edsrzf/mmap-go). There are probably more features that the [native syscall](https://godoc.org/golang.org/x/sys/unix#Mmap) provides that this library does not implement.

#### The `prot` argument

Here is the `mmap.Map` signature

    func Map(f *os.File, prot, flags int) (MMap, error) 
    

Let's look at `prot` first. The `prot` argument lets you specify the protection levels of your mapping: `RDONLY`, `RDWR`, `EXEC` are the options provided for `mmap-go`. These levels are pretty straightforward, `RDONLY` means you can only read from the mapping, `RDWR` means you can also write, and `EXEC` means you can execute code on that mapping.  Here is the description of `prot` from the Linux `man`:

    The prot argument describes the desired memory protection of the
    mapping (and must not conflict with the open mode of the file).
    It is either PROT_NONE or the bitwise OR of one or more of the
    following flags:
    
    PROT_EXEC
        Pages may be executed.
    
    PROT_READ
        Pages may be read.
    
    PROT_WRITE
        Pages may be written.
    
    PROT_NONE
        Pages may not be accessed.
    

In the [unix package](https://godoc.org/golang.org/x/sys/unix), those flags are: `unix.PROT_EXEC`, `unix.PROT_READ`, `unix.PROT_WRITE` and `unix.PROT_NONE`.

#### Experimenting with `PROT_EXEC` flag

I've become intrigued by the `EXEC` flag and wanted to see an example of how that works. I've Google and could not find any example. So I tried a search in Github by `PROT_EXEC` and found a good example in `C`: [MMapExecDemo](https://github.com/onesmash/MMapExecDemo). I replicated this example in `Go` using `mmap-go`.

The first step was to create a function that I wanted to be put in memory by `mmap` allocation, compile it, and get its assembly opcodes.

I created the `inc` function in `inc.go` file

    package inc
    
    func inc(n int) int {
    	return n + 1
    }
    
    

compiled it with `go tool compile -S -N inc.go`, then got its assembly by calling `go tool objdump -S inc.o`.

    func inc(n int) int {
      0x22b                 48c744241000000000      MOVQ $0x0, 0x10(SP)
            return n + 1
      0x234                 488b442408              MOVQ 0x8(SP), AX
      0x239                 48ffc0                  INCQ AX
      0x23c                 4889442410              MOVQ AX, 0x10(SP)
      0x241                 c3                      RET
    

With this, we can build represent our function in bytes on our code

    code := []byte{
            0x48, 0xc7, 0x44, 0x24, 0x10, 0x00, 0x00, 0x00, 0x00,
    		0x48, 0x8b, 0x44, 0x24, 0x08,
    		0x48, 0xff, 0xc0,
    		0x48, 0x89, 0x44, 0x24, 0x10,
    		0xc3,
    }
    

We allocate our memory with `mmap`.

    memory, err := mmap.MapRegion(nil, len(code), mmap.EXEC|mmap.RDWR, mmap.ANON, 0)
    if err != nil {
        panic(err)
    }
    

In this call, we're using a more complete function called `MapRegion` that lets you specify how much memory you are allocating (`Map` allocates the size of the underlying file) and the offset of the file.

In the beginning, we said that the main purpose of `mmap` was to create a mapping between a file and memory. But in this call we are not indicating any file. `mmap` can be used just a regular memory allocater by setting `nil` to the `*os.File` argument and `mmap.ANON` to the `flags` argument. We will talk about more `mmap.ANON`. Since we are not mapping any file, the offset is `0`.

So we have memory allocated with the same size of our code `len(code)`. Since we set the flag `mmap.RDWR`, we can copy our `code` to `memory`.

    copy(memory, code)
    

We have the code of our `inc` function in memory. In order to execute it, we have to cast that memory address to a function with a signature that matches the signature of our compiled `inc`.

    memory_ptr := &memory
    ptr := unsafe.Pointer(&memory_ptr)
    inc := *(*func(int) int)(ptr)
    

When we call `inc`, we are executing the code we put in memory. That only works because of the flag `mmap.EXEC`. If that flag was not set, a `segmentation violation` would occur.

    fmt.Println(inc(10)) // Prints 11
    

I don't know if this is a real use case. I just wanted to see what it meant to execute code that you put in memory. And there are probably other ways of achieving the same with regular memory allocation and calls to [mprotect](https://man7.org/linux/man-pages/man2/mprotect.2.html).

One question that may come up is: but the code is already in the `code` variable, can't we just execute it? No, because the memory static allocated to `code` is not executable. Can we make it executable? I've tried to use [mprotect](https://man7.org/linux/man-pages/man2/mprotect.2.html) on it but still got `segmentation violation`.

Here is the full working [gist](https://gist.github.com/brunoac/b9ff4ad46c27926e5e4f078133d0de79).

#### The `flags` argument

We can have many processes mapping the same memory region. This argument lets us decide about the visibility of the updates happening in the mapping. There are many flags, and you can check them out at [mmap](https://man7.org/linux/man-pages/man2/mmap.2.html). The important ones are `unix.MAP_SHARED`, `unix.MAP_PRIVATE` and `unix.MAP_ANON`.

`MAP_SHARED` means that changes to the mapping are visible to all processes and will also occur at the underlying mapped file, although we cannot control when.

`MAP_PRIVATE` means the changes are private and other processes will not see them. And also, they are not carried through to the underlying file.

`MAP_ANON` means that there is not going to be a mapped file. It is useful for sub-processes communication with shared memory.

I've got confused about the `mmap-go` library implementation. It only provides the `mmap.ANON` flag, that we used in the above example. If you want your mapping to be private, you can set the `mmap.COPY` flag to the `prot` argument. Anyways, you can always use the flags provided by the `unix` package implementation.

#### Locking and flushing

Two other nice methods, `Lock` and `Flush`, are provided by the API of `mmap-go`. The `Lock` method calls the [mlock](https://man7.org/linux/man-pages/man2/mlock.2.html) system call that prevents the mapping to be paged out to disk. And the `Flush` method calls the [msync](https://man7.org/linux/man-pages/man2/msync.2.html) system call that forces the data in memory to be written to disk. This is a good way to trying to have more control over how and when data is flushed to disk.

### Wrapping up

I felt kind of stupid of knowing about `mmap` after so long. I don't remember it being brought in my college class. For some reason, I felt amazed by it and its capabilities and decided to dig deeper. I like databases and I'm aiming to get a better grasp of them. This means that `mmap` cannot go unnoticed from my learning. For future posts, I'll try to bring about the benefits and drawbacks of using `mmap`, which projects use it, and what kind of problems it is suited for.

Even though the `mmap` can be used to solve that database problem we stated in the beginning, and many modern databases use it, [Andy Pavlo](https://twitter.com/andy_pavlo) advocates against it and have three lecture on how to databases, that don't use `mmap`, manage data.

If you like this kind of content, follow me on [twitter](https://twitter.com/brunocalza). You may find more related stuff there.