Compiling a Go program into a native binary for Nintendo Switch™
Tags: go, ebiten, gamedev, nintendoswitch
Published: 2022-01-03 08:35:41

---
title: Compiling a Go program into a native binary for Nintendo Switch™
published: true
description: Hajime has succeeded in compiling a Go program into a native binary for Nintendo Switch, and also running a game there.
tags: go, ebiten, gamedev, nintendoswitch
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpz0im8vlauldwbwvjyz.png
---

# This article has been moved to [ebiten.org](https://ebiten.org/blog/native_compiling_for_nintendo_switch.html) due to some complicated business reasons.

*This is an English translation of [my article in Japanese](https://zenn.dev/hajimehoshi/articles/72f027db464280).*

## tl;dr

Previously, we compiled a Go program into a WebAssembly and then converted it into C++ files to make it run on Nintendo Switch. Now, I have succeeded in compiling a Go program into a native binary for Nintendo Switch, and also running a game there. I replaced system calls with C function calls using the `-overlay` option. Also, I have developed a new package [Hitsumabushi](https://github.com/hajimehoshi/hitsumabushi) to generate JSON content for this.

## Caution

This article and the open-source projects in this article are based only on publicly available information. Hajime is responsible for this article's content. Please do not ask Nintendo about this article.

## Background

I have been developing a 2D game engine called [Ebiten](https://ebiten.org) in my spare time. I have succeeded in porting this to Nintendo Switch and [the Nintendo Switch version of "Bear's Restaurant"](https://odencat.com/bearsrestaurant/switch/en.html) was released in 2021.

![Bear's Restaurant](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5iz6fqqusigk9goikx6y.png)
<figcaption>Copyright 2021 Odencat Inc.</figcaption>

The method was to compile a Go program into a WebAssembly (Wasm) binary and then convert it to C++ files. See [the presentation slides from GoConference 2021 Autumn](https://docs.google.com/presentation/d/e/2PACX-1vTMRSmuWjhpOx3DIgetfi72jcOGvlqPU5z0Nps24YN6dxaBbu4dWm0FXS2f--D4G2b1aAvTmfqNA2IG/pub?start=false&loop=false&delayms=3000) for more details. The advantages were low uncertainty, low maintenance cost, and high portability. Once I developed the tool, its maintenance cost was pretty small as Wasm's specification is stable. On the other hand, the disadvantages were bad performance and long compiling time. Not only that performance was worse than native, but GC also suspended the game due to a single thread.

Compiling a Go program into a native binary for Nintendo Switch without using Wasm was quite uncertain and a rocky road. Of course, Go doesn't support Nintendo Switch officially. And naturally, Nintendo Switch's source code and binary formats are not open. Even if I hit an issue, it'd be possible that there would not be any clues to help me solve it. However, if I knew that I were to succeed, performance would be better than ever, and compiling speed would be as fast as Go. So I thought it was worth a shot and have been doing some experiments intermittently for one year.

## Strategy

The strategy is basically to replace system calls with C function calls in the runtime and the standard library. The system calls part is OS-dependent, and if I replace it with something portable, Go should work everywhere in theory. It seems pretty easy, doesn't it? Well, it was a lot more challenging than I expected...

The graphic below describes what I had to do. The left side is a structure of a structural overview of standard Go compiling. System calls work on specific systems and of course, this doesn't work on Nintendo Switch. So I had to replace them with standard C function calls like the right side.

![Replacing system calls with C function calls](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/083w9s2mh16xbmbkq2nl.png)
<figcaption>Replacing system calls with C function calls</figcaption>

And, there is another action item to adjust the binary format that the Go compiler generates to fit with Nintendo Switch. So in summary, the action items were as follows:

1. Replacing system calls with standard C function and/or pthread function calls
2. Adjust the ELF format that the Go compiler generates.

For replacing system calls, of course, system calls do not correspond one-to-one with C functions. And, there are too many system calls to implement. So, I replaced system calls one by one by finding which ones refused to work on an actual Nintendo Switch device.

The Go compiler can generate only formats that the Go compiler officially supports. For example, when a target is Linux, the format is ELF. Can Nintendo Switch support ELF? make a long story short, yes, I managed it. I won't describe the details about 2. here [^elf].

[^elf]: It's due to complicated business reasons.

What I have to do is create a `.a` file via the Go compiler with `GOOS=linux GOARCH=arm64` and `-buildmode=c-archive`, and then link it with other object files and libraries via Nintendo Switch compiler. The reason why I don't use `-buildmode=default` is that there are some items I have to do around an entry point. IMO, in general, it is more portable to depend on the platform for an entry point.

System calls are defined basically in the standard library, especially `runtime` and `syscall` packages. So, how did I rewrite them? In this project, I adopted the `-overlay` option.

## Hitsumabushi - rewriting the runtime with the `-overlay` option

`go build`'s `-overlay` is an option that overwrites Go files to be compiled. I overwrote Go files in the runtime with this option. This is [the official document's explanation](https://pkg.go.dev/cmd/go):

```
-overlay file
	read a JSON config file that provides an overlay for build operations.
	The file is a JSON struct with a single field, named 'Replace', that
	maps each disk file path (a string) to its backing file path, so that
	a build will run as if the disk file path exists with the contents
	given by the backing file paths, or as if the disk file path does not
	exist if its backing file path is empty. Support for the -overlay flag
	has some limitations: importantly, cgo files included from outside the
	include path must be in the same directory as the Go package they are
	included from, and overlays will not appear when binaries and tests are
	run through go run and go test respectively.
```

This is the format to give `-overlay`:

```json
{
  "Replace": {
    "/usr/local/go/src/runtime/os_linux.go": "/home/hajimehoshi/my_os_linux.go"
  }
}
```

If you build a Go program with this, `os_linux.go`'s content in `runtime` is replaced with `my_os_linux.go`'s. Pretty handy, isn’t it?

Managing this JSON file as it is is not portable. A location where Go is installed depends on environments, and then the target files' locations vary. Plus, you very rarely have to replace the full contents of a file, and in most cases, it is enough to replace some functions. As such, it is troublesome to update source files to match each Go version update.

So, I developed a new package to generate a JSON for this project. This is [Hitsumabushi (ひつまぶし)](https://github.com/hajimehoshi/hitsumabushi) [^hitsumabushi]. I adopted this name because I wanted a name ending with 'bushi' as a play on libc (ree-boo-shee (りぶしー) in Japanese pronunciation), because this is one of the primary things that Hitsumabushi deals with. There was another candidate I was considering, Katsuobushi (かつおぶし) [^katsuobushi], but I won’t get into that...

{% github hajimehoshi/hitsumabushi no-readme %}

[^hitsumabushi]: Hitsumabushi is [Japanese food](https://en.wikipedia.org/wiki/Unadon#Variations).
[^katsuobushi]: Katsuobushi is [yet another Japanese food](https://en.wikipedia.org/wiki/Katsuobushi).

Hitsumabushi is a very simple package defining an API like this:

```go
// GenOverlayJSON generates JSON content that can be passed
// to -overlay based on the given options, or returns an error
// when an error occurs.
//
// There are some options like specifying command arguments
// and specifying the number of CPU.
func GenOverlayJSON(options ...Option) ([]byte, error)
```

## Implementation of Hitsumabushi

I have created an original patch format for Hitsumabushi that looks like this:

```go
//--from
func getRandomData(r []byte) {
	if startupRandomData != nil {
		n := copy(r, startupRandomData)
		extendRandom(r, n)
		return
	}
	fd := open(&urandom_dev[0], 0 /* O_RDONLY */, 0)
	n := read(fd, unsafe.Pointer(&r[0]), int32(len(r)))
	closefd(fd)
	extendRandom(r, int(n))
}
//--to
// Use getRandomData in os_plan9.go.

//go:nosplit
func getRandomData(r []byte) {
	// inspired by wyrand see hash32.go for detail
	t := nanotime()
	v := getg().m.procid ^ uint64(t)

	for len(r) > 0 {
		v ^= 0xa0761d6478bd642f
		v *= 0xe7037ed1a0b428db
		size := 8
		if len(r) < 8 {
			size = len(r)
		}
		for i := 0; i < size; i++ {
			r[i] = byte(v >> (8 * i))
		}
		r = r[size:]
		v = v>>32 | v<<32
	}
}
```

The part after `//--from` and the part after `//--to` represent a replacing source and a target respectively. The reason why I invented my simple format is that the existing patch formats don't assume to be modified by a human being. In the above example, Linux's `getRandomData` implementation is replaced with Plan 9's. Linux's `getRandomData` uses `/dev/urandom` and this is not potable[^urandom]. This patch format saves some amount of work to manage the differences I want to replace. Of course, the cost to keep up with the Go version updates doesn't become zero even with this, but it should help a lot.

[^urandom]: There is another solution, making a pseudo `/dev/urandom` file, but I didn't adopt this. There is no other good way than using a platform-specific API.

Hitsumabushi creates modified files with this format and puts them in a temporary directory. It uses the files as the content of JSON (the replacing source file names).

Note that Hitsumabushi rewrites the standard library and the runtime, and the Go compiler is not the target to rewrite. In other words, the regular Go compiler is used as is.

The replacements by Hitsumabushi are only the standard C function calls and pthread function calls. It never deals with platform-specific APIs[^oss]. So, ideally, **Hitsumabushi should enable a Go program to run on any platform, regardless of whether or not the Go compiler originally supports it**.

[^oss]: The main reason is portability, but there is also another compelling reason: I wouldn't be able to make it open-source if it used a platform-specific API.

## Replacements

### Calling C functions from `runtime`

It is not an easy task to call a C function from `runtime`. In a usual Go program, you can call a C function easily with Cgo. However, `runtime` cannot use Cgo. Using Cgo means to depend on `runtime/cgo`, and `runtime/cgo` depends on `runtime`, so this would be a circular dependency.

To get straight to the point, `libcCall` makes it possible to call a C function from `runtime`. Some environments like `GOOS=darwin` already do this.

In addition, various [compiler directives](https://pkg.go.dev/cmd/compile#hdr-Compiler_Directives) are required.

* `//go:nosplit`: Skips an overflow in the stack.
* `//go:cgo_unsafe_args`: Treats Go arguments as C arguments.
* `//go:linkname`: Treats something defined in another package as if it was defined in this package. Or, it treats something defined in this package as if it was defined in another package. It ignores whether the symbol is exported or not. Very useful!
* `//go:cgo_import_static`: Static-links a C function and makes it possible to treat the symbol value in Go.

Let's see an actual example. To call the `write` system call from `runtime`, a function called `write1` is defined on the Go side.

```go
// An excerpt from runtime/stubs2.go in Go 1.17.5

//go:noescape
func write1(fd uintptr, p unsafe.Pointer, n int32) int32
```

```asm
// An excerpt from runtime/sys_linux_arm64.s in Go 1.17.5

TEXT runtime·write1(SB),NOSPLIT|NOFRAME,$0-28
	MOVD	fd+0(FP), R0
	MOVD	p+8(FP), R1
	MOVW	n+16(FP), R2
	MOVD	$SYS_write, R8
	SVC
	MOVW	R0, ret+24(FP)
	RET
```

In the case of 64bit ARM, `SVC` is used to invoke a system call.

Let's replace this with a C function call by `libcCall` and compiler directives.

```go
// An excerpt from runtime/stubs2.go after Hitsumabushi's replacement

//go:nosplit
//go:cgo_unsafe_args
func write1(fd uintptr, p unsafe.Pointer, n int32) int32 {
	return libcCall(unsafe.Pointer(abi.FuncPCABI0(write1_trampoline)), unsafe.Pointer(&fd))
}
func write1_trampoline(fd uintptr, p unsafe.Pointer, n int32) int32
```

```go
// An excerpt from runtime/os_linux.go after Hitsumabushi's replacement

//go:linkname c_write1 c_write1
//go:cgo_import_static c_write1
var c_write1 byte
```

```asm
// An excerpt from runtime/sys_linux_arm64.s after Hitsumabushi's replacement

TEXT runtime·write1_trampoline(SB),NOSPLIT,$0-28
	MOVD	8(R0), R1	// p
	MOVW	16(R0), R2	// n
	MOVD	0(R0), R0	// fd
	BL	c_write1(SB)
	RET
```

```c
// An excerpt from runtime/cgo/gcc_linux_arm64.c after Hitsumabushi's replacement

int32_t c_write1(uintptr_t fd, void *p, int32_t n) {
  static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
  int32_t ret = 0;
  pthread_mutex_lock(&m);
  switch (fd) {
  case 1:
    ret = fwrite(p, 1, n, stdout);
    fflush(stdout);
    break;
  case 2:
    ret = fwrite(p, 1, n, stderr);
    fflush(stderr);
    break;
  default:
    fprintf(stderr, "syscall write(%lu, %p, %d) is not implemented\n", fd, p, n);
    break;
  }
  pthread_mutex_unlock(&m);
  return ret;
}
```

By the way, `libcCall` is not defined on `GOOS=linux`. I had to rewrite `//go:build` in `runtime/sys_libc.go` properly.

If you forcibly call a C function using assembly without `libcCall`, a C stack will be on the current Goroutine's stack. Then, you might find very mysterious errors. I don't recommend invoking a C function without `libcCall`.

### Ignoring signals

Hitsumabushi ignores all signals. For example, [`sigaltstack` and `sigprocmask` in `runtime` are empty](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/1.17/runtime/os_linux.go.patch#L165-L180). There are standard C functions that deal with signals, but they are not implemented in some environments.

As a side effect, accessing a nil pointer caused SEGV, and `recover`-ing it became impossible. A program dies without panic messages, even. This is inconvenient to some extent, but we have to put in the effort to avoid this issue in production environments.

### Implementing a pseudo file system

Even when a Go program does nothing, the runtime might access the file system. On Linux, apparently these files are read from the runtime:

* `/proc/self/auxv` (Information about e.g. a page size)
* `/sys/kernel/mm/transparent_hugepage/hpage_pmd_size` (Huge Page Size)

I hand-crafted some content for both. For example, I used `0` for Huge Page Size since it worked. For the implementation, see [Hitsumabushi's `c_open`](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/1.17/runtime/cgo/gcc_linux_arm64.c.patch#L437-L454).

For writing files, I implemented only a standard output and a standard error. Both just use `fprintf`. Without them, even `println` doesn't work. I decided not to implement reading and writing other files for now. For the implementation, see [Hitsumabushi's `c_write1`](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/1.17/runtime/cgo/gcc_linux_arm64.c.patch#L480-L499).

### Implementing a pseudo memory system

In Go's heap memory management, [`mmap`](https://man7.org/linux/man-pages/man2/mmap.2.html) system call is the bottom layer on Linux. Go manages virtual memory allocated there. `munmap` is called for unused regions.

There are [4 states](https://cs.opensource.google/go/go/+/refs/tags/go1.17.5:src/runtime/malloc.go;l=349-360) of a heap memory region and these states transition as in the diagram below. When the state is 'Ready', the region is available.

![The state transition diagram of Go's memory](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3vxhb0t1twu5qs0p6v0x.png)
<figcaption>The state transition diagram of Go's memory</figcaption>

Go specifies an address in virtual memory and uses an allocated memory region with the address. However, there is no standard C function to allocate memory with a specific address. That's unfortunate.

There are some platforms where it is impossible to allocate memory with a specific address: Plan 9 and Wasm. Hitsumabushi referred to them and implemented a 'corner-cutting' memory system. It referred to [the Wasm version](https://cs.opensource.google/go/go/+/refs/tags/go1.17.5:src/runtime/mem_js.go) in particular, which is the simplest implementation. I won't describe the details here, but basically, the implementation is as shown in the following list. For an actual source, see [Hitsumabushi's `mem_linux.go`](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/1.17/runtime/mem_linux.go).

* `sysAlloc`: Calls `sysReserve` and `sysMap`.
* `sysMap`: Increments the total size record of heap memory.
* `sysFree`: Decrements the total size record of heap memory.
* `sysReserve`: Calls `calloc`.
* The other functions do nothing.

As you can see, there is a call of `calloc` but no call of `free`. It is impossible to `free` a part of a region allocated by `calloc`. This means that memory usage is monotonically increased. Originally, the method to make an Ebiten application work on Nintendo Switch was to convert Go to C++ via Wasm, and memory usage was also monotonically increased there[^wasm]. It didn’t end up making things worse, at the very least, so I’ve compromised with this solution so far, but I would like to fix this in the future…

[^wasm]: To be exact, about 2G of memory was allocated first and was used without additional allocations.

### Implementing pseudo `futex`

[`futex`](https://man7.org/linux/man-pages/man2/futex.2.html) is the bottom layer of the part that handles sleeping and waking up threads. Of course, the standard C functions and pthread functions cannot invoke `futex` directly. So, I had to mimic the behavior of `futex` with pthread. Originally, pthread itself is implemented with `futex`, so I had to do the opposite thing.

There are [two ways](https://cs.opensource.google/go/go/+/refs/tags/go1.17.5:src/runtime/os_linux.go;l=17-24) to use `futex` via Go.

* `futexsleep(uint32 *addr, uint32 val)`: Makes the thread sleep when `addr` is `val`.
* `futexwake(uint32 *addr)`: Wakes up the thread which sleeps with `addr`.

In Hitsumabushi, I added a simple implementation like this. For an actual source, see [Hitsumabushi's `pseudo_futex`](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/1.17/runtime/cgo/gcc_linux_arm64.c.patch#L321-L385).

```c
// A pseudo code
pseudo_futex(void* uaddr, int32_t val) {
  static pthread_cond_t cond; // A condition variable
  
  switch (mode) {
  case sleep:
    if (*uaddr == val) {
      cond_wait(&cond); // Sleep
    }
    break;
  case wake:
    cond_broadcast(&cond); // Wake up all the threads sleeping with cond.
    break;
  }
}
```

When `wake` is called, it will wake up not only the necessary threads, but all the threads. If you want to wake up only necessary threads, you would need to manage multiple condition variables for each `uaddr`, which would be cumbersome. Such unnecessary waking up is called [spurious wakeup](https://en.wikipedia.org/wiki/Spurious_wakeup). [This is explicitly expected in Go source code](https://cs.opensource.google/go/go/+/refs/tags/go1.17.5:src/runtime/os_linux.go;l=41-42), so this is not problematic. However, performance might be degraded.

### Adjusting the number of CPU cores

The number of CPU cores is determined by the result of the  [`sched_getaffinity`](https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html) system call. There is no corresponding standard C function, so I gave Hitsumabushi an option to specify the number of cores to `GenOverlayJSON`. For the actual source, see [Hitsumabushi's `c_sched_getaffinity`](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/overlay.go#L177-L208)

There were some environments where an application froze with 2 or more CPU cores specified. It's because a thread could use only one core by default. Thus, I had to call [`pthread_setaffinity_np`](https://man7.org/linux/man-pages/man3/pthread_setaffinity_np.3.html) explicitly. In Hitsumabushi, I added a hack to call `pthread_setaffinity_np` just after [`pthread_create`](https://man7.org/linux/man-pages/man3/pthread_create.3.html). For the actual source, see [Hitsumabushi's overlay.go](https://github.com/hajimehoshi/hitsumabushi/blob/033f91b0b848e44349a91ccd28d6436bc22d0c44/overlay.go#L217-L247). As an aside, it was quite hard to find this solution. I can’t tell you how happy I was to finally solve this conundrum.

### Entry point

Hitsumabushi is assumed to be used with `-buildmode=c-archive`. The generated file is a C library, and even `main` is not called. If you want to call `main`, you have to define a C function and call `main` explicitly inside. Calling `main` explicitly does not make sense usually, but I think it is practical for `c-archive`.

```go
package main

import "C"

//export GoMain
func GoMain() {
	main()
}
```

```c
// Call the entry point in Go in the entry point in C.
int main() {
  GoMain();
  return 0;
}
```

## Results

* **I managed to get a game called "[Innovation 2007](https://github.com/hajimehoshi/go-inovation)" working on an actual Nintendo Switch device**. Controller support, touch inputting, and audio all work perfectly. Innovation 2007 uses most of Ebiten's features, so I'm sure other games would work as well.
* Compiling speed became much faster. Before this solution, it took 5 to 10 minutes to full-build a C++ project, but now it only takes less than 10 seconds. This is awesome!
* Suspensions by GC seem to have disappeared.
* I now have to update whenever a new version of Go is released. This is an acceptable compromise to me. From my past experiments, I don’t expect any major changes anyway.

## Remarks

This is a side note, but the implementation of Go's runtime has ample accumulation of knowledge about modern OSes and is very insightful. I think it can teach you a significant amount about computer science. That said, it can be quite daunting to read it without a purpose, so I recommend doing so with some sort of modification project in mind.

Thanks to the near-success of this project, the method I presented in the Go Conference is now becoming outdated. This was inevitable, obviously, but it still makes me feel a little sad to see that hard work go obsolete.

## Future works

I'll continue polishing this so that a proper game can be released for Nintendo Switch. As I described first, there is a high level of uncertainty in this project. Until a game is released, I cannot anticipate what kind of issues will occur, and I always have to be on high alert. Even in the worst case scenario, however, I know we can continue to release the game with the help of [go2cpp](https://github.com/hajimehoshi/go2cpp), which is reassuring. Still, with all the hard work I’ve put into this already, I really want to release a game with Hitsumabushi and see it achieve some actual results.

## Acknowledgments

Thanks to the kind folks over in the PySpa community for all their technical advice. I’d also like to express my gratitude to Daigo, [President of Odencat Inc.](https://odencat.com/), who kindly uses Ebiten for Nintendo Switch. Thank you very much.

Happy new year!