Running Linux Programs as Unikernels on macOS
Tags: unikernels, rust, blockchain, linux
Published: 2020-02-10 21:17:34

---
title: Running Linux Programs as Unikernels on macOS
published: true
description: OPS can run any linux program on osx but there are a few caveats to understand how.
tags: unikernels, rust, blockchain, linux
cover_image: https://i.imgur.com/yuEMMoH.jpg
---

Someone emailed me the other day and was having some trouble getting
their program to work under OPS.

He had downloaded a binary release from github and knowing that OPS only runs linux binaries chose the linux version. Further, there are numerous
examples of people running linux programs under ops on osx. Hell, just do a

```bash
ops pkg list
```

for a short list.

However, for some reason it just wasn't working for him.

```bash
user@users-MacBook-Pro Desktop % ops run solana-validator    
*errors.errorString libstdc++.so.6: file does not exist
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:100 (0x4c55779)
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:129
(0x4c5607d)
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:261
(0x4c52d0e)
/Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:24
(0x4c50f5f)
/Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:15 (0x4ca4189)
/Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:118 (0x4ca4f75)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
(0x4c8bf3e)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
(0x4c8cb5b)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
(0x4ca8d88)
/Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
(0x4ca8d83)
/usr/local/go/src/runtime/proc.go:200 (0x402f57c)
  main: return
/usr/local/go/src/runtime/asm_amd64.s:1337 (0x405a671)
  goexit: GLOBL shifts<>(SB),RODATA,$256

panic: libstdc++.so.6: file does not exist
```

After I saw the word mac and then libstdc++ I knew immediately what was wrong but I also realized that it might not be readily apparent to many people and so this is what this blogpost is about.

The binary in question is dynamically linked. What this means is that
there are several libraries that the dynamic loader (ld) will try and
load at runtime versus statically linked where all the libraries are
packaged inside the binary. This leads to a fatter binary but you can be
assured that everything is present when needed. I won't jump into a
static/dynamic flamewar as they each have their own usecases.

I should also point out that normally you can't run linux binaries directly on a mac - this is actually a uniquely interesting unikernel aspect as we don't need to boot up vagrant or a linux vm. Traditional virtualization software virtualizes the operating system. You can see using a tool like [OPS](https://ops.city) as virtualizing the application instead.

One of the reasons I decided to write this post was not just the email but I've noticed in the past few years I've seen quite a lot of people that use containers and go to refer to their programs as statically linked when it is in fact absolutely not.

Let's look at this example go webserver:

```bash
package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Welcome to my website!")
	})

	fs := http.FileServer(http.Dir("static/"))
	http.Handle("/static/", http.StripPrefix("/static/", fs))

	http.ListenAndServe(":8080", nil)
}
```

Fairly simple - all it does is listen on 8080 and serve up requests. If we compile it:

```bash
go build
```

We see that by default it will be dynamically linked against a few libs:

```
eyberg@box:~/z$ ldd z
        linux-vdso.so.1 (0x00007ffe8b1db000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2e091f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2e08e03000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2e09413000)
eyberg@box:~/z$ file z
z: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped
```

VDSO gives us a fast clock. Libpthread gives us threads. Libc contains everything you'd find in section 3 of the man pages and finally LD is our loader.

You __can__ link your go programs statically via something like this:

```bash
eyberg@box:~/z$ go build  -buildmode=pie -ldflags "-linkmode external -extldflags -static"
# _/home/eyberg/z
/tmp/go-link-057347370/000004.o: In function `_cgo_7e1b3c2abc8d_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
eyberg@box:~/z$ ldd z
        not a dynamic executable
eyberg@box:~/z$ file z
z: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=0671d22e52baf812ed9f500a8d9ee7848b8c809e, not stripped
```

Both ldd && file now report that it is statically linked. However, you'll notice (at least in go 1.12) it now outputs a message stating that getaddrinfo is still required! That's cause even though ldd output and file is telling us that it is indeed static we still rely on libnss for dns. You can force go to use its built in resolver instead.

Getaddrinfo is found here:

```bash
eyberg@box:~/s/solana-release/bin$ readelf --dyn-syms /lib/x86_64-linux-gnu/libc-2.27.so  | grep getaddr
   873: 0000000000107bc0  3261 FUNC    GLOBAL DEFAULT   13 getaddrinfo@@GLIBC_2.2.5
  1601: 00000000001413c0    43 FUNC    GLOBAL DEFAULT   13 inet6_rth_getaddr@@GLIBC_2.5
```

You'll note that your libraries are probably stripped which means you can't use something like nm to find the symbols but readelf surfaces them up in the .dynsym section.

Also, even that is not enough - you'll probably make use of various functions found in these files as well:

```
eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_files.so.2
/lib/x86_64-linux-gnu/libnss_files.so.2
eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_dns.so.2
/lib/x86_64-linux-gnu/libnss_dns.so.2
```

The point here is that just sticking a go program in a container does not inherently make your binary 'statically linked' and the problem is that, that is not what a lot of people have been stating and this source of
confusion is probably what leads to misunderstandings such as this.

So let's go back to our example at the beginning of the article.

The program in question is the [solana blockchain](https://github.com/solana-labs/solana).

You can download and unzip like so:

```
wget https://github.com/solana-labs/solana/releases/download/v0.23.2/solana-release-x86_64-unknown-linux-gnu.tar.bz2
bunzip2 solana-release-x86_64-unknown-linux-gnu.tar.bz2
tar xf solana-release-x86_64-unknown-linux-gnu.tar
```

You can see from the ldd output that it is trying to load a library that is dynamically linked (libstdc++) and that's what our error in the trace was.

```Bash
eyberg@box:~/s/solana-release/bin$ ldd solana
        linux-vdso.so.1 (0x00007ffe338df000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06976a8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f06984a2000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06974a4000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f069729c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f069707d000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0696e65000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0696ac7000)
```


As you can see this program works out of the box on linux.

```bash
eyberg@box:~/s/solana-release/bin$ ops run -c config.json solana-validator
[solana-validator --ledger /]
booting /home/eyberg/.ops/images/solana-validator.img ...
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
assigned: 10.0.2.15
solana-validator 0.23.2
log file: solana-validator-7wgZy3ozTRyvLtRN5o3Xh6m5aEvEo2XBxP5yPfaCny1t-20200210-204507.log

eyberg@box:~/s/solana-release/bin$ cat config.json
{
  "Args": ["solana-validator", "--ledger", "/"]
}
```

This is all fine but if you want to run it on osx you'll need to do one
of two things:

1) Either build from source and statically link it (so ldd doesn't show
output).


or

2) Download the libraries it's linked to and manually add them to the
filesystem.

The reason this works out of the box on linux is that [OPS](https://ops.city) looks at and
try to load the libraries as we build the disk image. This isn't
possible on a mac cause mac uses mach-o which is a different format that [Nanos](https://github.com/nanovms/nanos) does not and probably won't ever support as [nanos](https://github.com/nanovms/nanos) is explicitly designed to run linux server-side programs.

For example when we build common packages like nginx or node, you'll see from the [ops](https://github.com/nanovms/ops) output that all required libraries are placed on the filesystem in a known location.

```
➜  ~    ops pkg contents node_v13.6.0
File :/node
File :/package.manifest
Dir :/sysroot
Dir :/sysroot/lib
Dir :/sysroot/lib/x86_64-linux-gnu
File :/sysroot/lib/x86_64-linux-gnu/libc.so.6
File :/sysroot/lib/x86_64-linux-gnu/libdl.so.2
File :/sysroot/lib/x86_64-linux-gnu/libgcc_s.so.1
File :/sysroot/lib/x86_64-linux-gnu/libm.so.6
File :/sysroot/lib/x86_64-linux-gnu/libnss_dns.so.2
File :/sysroot/lib/x86_64-linux-gnu/libnss_files.so.2
File :/sysroot/lib/x86_64-linux-gnu/libpthread.so.0
Dir :/sysroot/lib64
File :/sysroot/lib64/ld-linux-x86-64.so.2
Dir :/sysroot/proc
File :/sysroot/proc/meminfo
Dir :/sysroot/usr
Dir :/sysroot/usr/lib
Dir :/sysroot/usr/lib/x86_64-linux-gnu
File :/sysroot/usr/lib/x86_64-linux-gnu/libstdc++.so.6
```

Without getting into the semantics of building your own package [which you can find here](https://github.com/nanovms/ops/blob/master/PACKAGES.md) the easiest way to do this with your own app is in your project directory you can create a directory:

```
mkdir -p usr/lib
```

then include it in your config.json that you pass to ops when it builds your image.

```json
{
"Dirs":["usr"]
}
```

```
ops run -c config.json myprogram
```

After that you can always grab the image that was created in ~/.ops/images.

Hope this clears up any confusion.