#![no_std] with WASI is more complicated than I thought it would be
Tags: rust, webassembly, devjournal
Published: 2021-06-03 05:11:01

This post is meant to document my experience using `#![no_std]` in Rust with [WASI][wasi], which was **way** more complicated than I thought it would. Perhaps this can help to make the documentation better, who knows? :P

So I was wanting to learn more about WASI (WebAssembly System Interface), which is basically like POSIX interfaces for WASI, with capabilities for better security. I decided that perhaps making a "Hello, world!" example in Rust with `#[no_std]` would be a good idea.

It was... difficult. There were a *ton* of roadblocks.

So, let's start this.

I started with a basic shell of a `#![no_std]` program:
```rust
#![no_std]
#![no_main]

use core::panic::PanicInfo;

#[panic_handler]
fn panic_handler(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn _start() -> ! {
    loop {}
}
```

In the `Cargo.toml` I set `panic = "abort"`:
```toml
[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"
```

I use [`cargo-wasi`][cw] to build it:
```shell
$ cargo wasi build
   Compiling wasi-testing v0.1.0 (/home/james/Code/Rust/wasi-testing)
error: linking with `rust-lld` failed: exit status: 1
  |
  = note: ... really long command line invocation ...
  = note: rust-lld: error: duplicate symbol: _start
          >>> defined in /home/james/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o
          >>> defined in /home/james/Code/Rust/wasi-testing/target/wasm32-wasi/debug/deps/wasi_testing.5awycwmt43f1nxf9.rcgu.o
          

error: aborting due to previous error

error: could not compile `wasi-testing`

To learn more, run the command again with --verbose.
```
Uh... what?

So what happened is that `_start` is already defined by this mysterious `crt1-command.o`. I didn't learn until much after that this is from [`wasi-libc`](https://github.com/WebAssembly/wasi-libc/blob/main/libc-bottom-half/crt/crt1-command.c):
```c
#include <wasi/api.h>
#include <stdlib.h>
extern void __wasm_call_ctors(void);
extern int __original_main(void);
extern void __wasm_call_dtors(void);

__attribute__((export_name("_start")))
void _start(void) {
    // Call `__original_main` which will either be the application's zero-argument
    // `__original_main` function or a libc routine which calls `__main_void`.
    // TODO: Call `main` directly once we no longer have to support old compilers.
    int r = __original_main();

    // If main exited successfully, just return, otherwise call `exit`.
    if (r != 0) {
        exit(r);
    }
}
```

```c
// TODO: Call `main` directly once we no longer have to support old compilers.
```
This one comment is what caused 99% of my pain with starting the actual program.

At the time, I didn't know about this, so I just tried changing the name of `_start` to `__original_main`:
```rust
#[no_mangle]
pub extern "C" fn __original_main() -> ! {
    loop {}
}
```

I build it, and I get this error:
```
error: linking with `rust-lld` failed: exit status: 1
  |
  = note: ... really long command line invocation ...
  = note: rust-lld: error: function signature mismatch: __original_main
          >>> defined as () -> i32 in /home/james/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o
          >>> defined as () -> void in /home/james/Code/Rust/wasi-testing/target/wasm32-wasi/debug/deps/wasi_testing.5awycwmt43f1nxf9.rcgu.o
```

Ok, looks like this function must return an `i32`. That's fine, I guess. I assumed that this function's return value is the exit code, which I eventually learn is true.

```rust
#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    0
}
```

I compile it, and it works.

Now I run it:
```
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::exit` has not been defined
```
Uh... ok then.
At this point I decide that I should probably use WASI to define this exit function, instead of just endlessly looping. So I link in `proc_exit`. According to the [WASI docs][wasidoc], `proc_exit` takes in an `exitcode` (a u32) and returns nothing. Perfect!

So I link it in. Notice the deadly mistake? I didn't, yet.
```rust
extern "C" {
    fn proc_exit(code: u32);
}
```

Then I define `fn exit() -> !`:
```rust
#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe { proc_exit(code); }
    loop {} // (hopefully) never reached
}
```

It builds, but it doesn't work:
```
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::proc_exit` has not been defined
```
Hm.

So I try defining `proc_exit` as such:
```rust
pub mod internal {
    #[no_mangle]
    #[export_name = "proc_exit"]
    pub unsafe extern "C" fn proc_exit(code: u32) {
        super::proc_exit(code);
    }
}
```

I run it, and... uh what?
```
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to invoke command default
    1: wasm trap: call stack exhausted
       wasm backtrace:
           0: 0xffffffff - <unknown>!proc_exit
           1:   0xd8 - <unknown>!proc_exit
           2:   0xd8 - <unknown>!proc_exit
           3:   0xd8 - <unknown>!proc_exit
           4:   0xd8 - <unknown>!proc_exit
           5:   0xd8 - <unknown>!proc_exit
           6:   0xd8 - <unknown>!proc_exit
           7:   0xd8 - <unknown>!proc_exit
           ... a TON of lines cut ...
           32682:   0xd8 - <unknown>!proc_exit
           32683:  0x10b - <unknown>!exit
           32684:  0x122 - <unknown>!_start
         note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information
```

Uh ok. At this point I realized I was probably making it call into itself endlessly.

At this point, I look into [Bytecode Alliance's WASI rust bindings](https://github.com/bytecodealliance/wasi), used by Rust's `std`.

I look multiple times, and only about the third time do I notice [this fatal line](https://github.com/bytecodealliance/wasi/blob/main/src/lib_generated.rs#L2142):
```rust
    #[link(wasm_import_module = "wasi_snapshot_preview1")]
    extern "C" {
        // ... //
    }
```

I add that to my `extern "C"` block and remove the weird `internal::proc_exit` function. Here's the full code at this point:
```rust
#![no_std]
#![no_main]

use core::panic::PanicInfo;

#[panic_handler]
fn panic_handler(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    0
}

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe {
        proc_exit(code);
    }
    loop {} // (hopefully) never reached
}

#[link(wasm_import_module = "wasi_snapshot_preview1")]
extern "C" {
    fn proc_exit(code: u32);
}
```

I build and run it, and it works. Since it prints nothing to the screen, I change the `0` in `__original_main` to a different value that I can observe from the terminal. The specific number doesn't matter, just that it's not zero.

```shell
$ wasmtime target/wasm32-wasi/debug/wasi-testing.wasm
$ echo $status # I use fish, with POSIX-compliant shells it's `$!`.
42
```

And it works! Ok, time to do something a bit more complex. I'm going to work on a print function.

I use `fd_write`, and since I don't want to look into how interface types work, I just look at the `wasi` crate again, with some help from the [wasmtime WASI WAT tutorial](https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-tutorial.md#web-assembly-text-example).

I link it in (adding some helper structs, aliases, and enums which I've omitted for brevity, but can be seen in the full code at the end of the post):
```rust
// extern "C" block from before
extern "C" {
    pub fn fd_write(
        fd: Fd,
        ciovecs: *const Ciovec,
        n_ciovecs: Size,
        n_written: *mut Size,
    ) -> Errno;
}
```


I then create a `print` function:
```rust
fn print(s: &str) -> Result<Size, Errno> {
    assert!(
        s.len() <= u32::MAX as usize,
        "please don't store >4GB of text in a string, thanks"
    );
    let ciovec = Ciovec {
        buf: s.as_ptr(),
        buf_len: s.len() as u32,
    };
    let ciovec_ptr = &ciovec as *const Ciovec;
    let mut n_written: Size = 0;
    let errno = unsafe { fd_write(1, ciovec_ptr, 1, &mut n_written as *mut Size) };
    if errno == Errno::Esuccess {
        return Ok(n_written);
    }
    Err(errno)
```

Here's how this works:
- I first assert that the string is not larger than the 32-bit limit, which would be logically incorrect and potentially even unsound to try and print.
- I then construct the ciovec (const IO vector) with the buffer (pointer to the string as a `*const u8`), and the length as a `u32`.
- I then get a pointer to that ciovec on the stack.
- I create a variable on the stack, `n_written`.
- I call `fd_write` with the file descriptor (1 = stdout), the pointer to the ciovecs, the length of the ciovec array (1), as you can print multiple ciovec arrays at once as an optimization for buffered writes, I think, and finally a mutable reference to the `n_written` variable on the stack.
- If the `errno` returned did not indicate an error (`0` = success), then I return `Ok(n_written)`.
- Otherwise I return the `errno`.

I call the function within the `__original_main` function (changing the return value to `0`, again):
```rust
#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    print("Hello, world!\n").unwrap();
    0
}
```

At this point, since I'm calling `.unwrap()`, I'm now realizing I should probably have a better `#[panic_handler]`. So I change it now:
```rust
#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    loop {}
}
```

I then again realize that `format!` can't be used in `#![no_std]` contexts without `alloc`.

I add this to the top of the file to indicate that it should use `alloc`:
```rust
extern crate alloc;
```

Then I import `format!` from `alloc`:
```rust
use alloc::format;
```

I build it:
```
error: no global memory allocator found but one is required; link to std or add `#[global_allocator]` to a static item that implements the GlobalAlloc trait.

error: `#[alloc_error_handler]` function required, but not found.

note: Use `#![feature(default_alloc_error_handler)]` for a default error handler.
```

Ok. Now I need to add in a global allocator. I use [`wee_alloc`](https://github.com/rustwasm/wee_alloc) since I know it works pretty well with WASM:
`Cargo.toml`
```toml
[dependencies]
wee_alloc = "0.4.5"
```

`main.rs`
```rust
use wee_alloc::WeeAlloc;

#[global_allocator]
pub static ALLOC: WeeAlloc = WeeAlloc::INIT;
```

And then I need to create an allocator error handler:
```rust
#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    loop {}
}
```

This means now that I have to add the feature flag for `#[alloc_error_handler]`, meaning that I now have to use the nightly branch:
```rust
#![feature(alloc_error_handler)]
```

I add a `panic!()` in the main function to test it.

I run it:
```
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::memcpy` has not been defined
```

Um. Ok, I remember this kind of thing. I do some searching and then rediscover [`compiler_builtins`](https://github.com/rust-lang/compiler-builtins).

I add it to the program:
`Cargo.toml`
```toml
[dependencies]
# ...
compiler_builtins = "0.1.44"
```

`main.rs`
```rust
extern crate compiler_builtins;
```

It still doesn't work, however. It still complains that `memcpy` is not defined. I then realize that I have to enable the `mem` feature:
```toml
[dependencies]
# ...
compiler_builtins = { version = "0.1.44", features = ["mem"] }
```

And it works! Well, kind of:
```
Hello, world!
panicked at 'explicit panic', src/main.rs:31:5
```

It unfortunately, however, infinitely hangs thanks to the infinite `loop {}`. I decide to use `proc_raise` to send an abort signal:
```rust
extern "C" {
    // ...
    fn proc_raise(sig: Signal) -> Errno;
}
```

You can see the definition of `Signal` and `Errno` in the full code at the end of this post.

I then define a function, `abort()`, because I'll be using this same logic within the `alloc_error_handler` so that when allocation fails, it doesn't silently hang.

```rust
fn abort() -> ! {
    unsafe { proc_raise(Signal::Abrt); }
    loop {} // should not ever be reached, but required for the `!` return type
}
```

I then also change the two error/panic handlers to use this now:
```rust
#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    abort()
}

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    abort()
}
```

And it works! Well, kind of.
```
Hello, world!
panicked at 'explicit panic', src/main.rs:31:5
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to invoke command default
    1: proc_raise unsupported
       wasm backtrace:
           0: 0xa692 - <unknown>!wasi_testing::abort::h7ce8efa81892bb51
           1: 0x4b6a - <unknown>!rust_begin_unwind
           2: 0xacf8 - <unknown>!core::panicking::panic_fmt::h3e7a769e079b4878
           3: 0xa3d6 - <unknown>!core::panicking::panic::h0c30bb50c6755362
           4: 0x8431 - <unknown>!__original_main
           5: 0xb136 - <unknown>!_start
       note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information
```

`proc_raise` isn't supported by wasmtime yet, but it *is* part of the WASI API, so I should probably use it as it's the best method at the moment, and may be supported at some point.

And that is the hellish journey that I took to get WASI working with `#![no_std]` in Rust.

Here is the full code at this point:
```rust
#![feature(alloc_error_handler)]
#![allow(dead_code)]
#![no_std]
#![no_main]

extern crate alloc;
extern crate compiler_builtins;

use alloc::format;
use core::panic::PanicInfo;
use wee_alloc::WeeAlloc;

#[global_allocator]
pub static ALLOC: WeeAlloc = WeeAlloc::INIT;

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    abort()
}

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    abort()
}

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    print("Hello, world!\n").unwrap();
    0
}

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe {
        proc_exit(code);
    }
    loop {} // (hopefully) never reached
}

fn abort() -> ! {
    unsafe {
        proc_raise(Signal::Abrt);
    }
    loop {} // should not ever be reached, but required for the `!` return type
}

#[repr(u16)]
#[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum Errno {
    Esuccess,
    E2big,
    Eacces,
    Eaddrinuse,
    Eaddrnotavail,
    Eafnosupport,
    Eagain,
    Ealready,
    Ebadf,
    Ebadmsg,
    Ebusy,
    Ecanceled,
    Echild,
    Econnaborted,
    Econnrefused,
    Econnreset,
    Edeadlk,
    Edestaddrreq,
    Edom,
    Edquot,
    Eexist,
    Efault,
    Efbig,
    Ehostunreach,
    Eidrm,
    Eilseq,
    Einprogress,
    Eintr,
    Einval,
    Eio,
    Eisconn,
    Eisdir,
    Eloop,
    Emfile,
    Emlink,
    Emsgsize,
    Emultihop,
    Enametoolong,
    Enetdown,
    Enetreset,
    Enetunreach,
    Enfile,
    Enobufs,
    Enodev,
    Enoent,
    Enoexec,
    Enolck,
    Enolink,
    Enomem,
    Enomsg,
    Enoprotoopt,
    Enospc,
    Enosys,
    Enotconn,
    Enotdir,
    Enotempty,
    Enotrecoverable,
    Enotsock,
    Enotsup,
    Enotty,
    Enxio,
    Eoverflow,
    Eownerdead,
    Eperm,
    Epipe,
    Eproto,
    Eprotonosupport,
    Eprototype,
    Erange,
    Erofs,
    Espipe,
    Esrch,
    Estale,
    Etimedout,
    Etxtbsy,
    Exdev,
    Enotcapable,
}

pub type Size = u32;
pub type Fd = u32;
pub type ExitCode = u32;

#[repr(C)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct Ciovec {
    pub buf: *const u8,
    pub buf_len: Size,
}

#[repr(u8)]
pub enum Signal {
    None,
    Hup,
    Int,
    Quit,
    Ill,
    Trap,
    Abrt,
    Bus,
    Fpe,
    Kill,
    Usr1,
    Segv,
    Usr2,
    Pipe,
    Alrm,
    Term,
    Chld,
    Cont,
    Stop,
    Tstp,
    Ttin,
    Ttou,
    Urg,
    Xcpu,
    Xfsz,
    Vtalrm,
    Prof,
    Winch,
    Poll,
    Pwr,
    Sys,
}

#[link(wasm_import_module = "wasi_snapshot_preview1")]
extern "C" {
    pub fn fd_write(fd: Fd, ciovecs: *const Ciovec, n_ciovecs: Size, n_written: *mut Size)
        -> Errno;

    pub fn proc_raise(sig: Signal) -> Errno;

    pub fn proc_exit(rval: ExitCode);
}

fn print(s: &str) -> Result<Size, Errno> {
    assert!(
        s.len() <= u32::MAX as usize,
        "please don't store >4GB of text in a string, thanks"
    );
    let ciovec = Ciovec {
        buf: s.as_ptr(),
        buf_len: s.len() as u32,
    };
    let ciovec_ptr = &ciovec as *const Ciovec;
    let mut n_written: Size = 0;
    let errno = unsafe { fd_write(1, ciovec_ptr, 1, &mut n_written as *mut Size) };
    if errno == Errno::Esuccess {
        return Ok(n_written);
    }
    Err(errno)
}
```

And the `Cargo.toml`:
```toml
[package]
name = "wasi-testing"
version = "0.1.0"
authors = ["ThePuzzlemaker <tpzker@thepuzzlemaker.info>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wee_alloc = "0.4.5"
compiler_builtins = { version = "=0.1.43", features = ["mem"] }

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"
```

[wasi]: https://wasi.dev/
[wasidoc]: https://github.com/WebAssembly/WASI/blob/main/phases/snapshot/docs.md
[cw]: https://github.com/bytecodealliance/cargo-wasi
[wasmtime]: https://github.com/bytecodealliance/wasmtime