How I used wasm-pack to build a WebAssembly module for an AudioWorkletProcessor
Tags: rust, javascript, webassembly
Published: 2023-05-15 15:45:00

If you haven't been following this series, I have been building a synthesizer in [Rust](https://www.rust-lang.org/) and [WebAssembly](https://webassembly.org/). The advantage of using WebAssembly is that I can take advantage of all of Rust's features while simultaneously avoiding the pitfalls of trying to use the operating system's complex libraries for playing audio.

## My Last Solution Was Not Ideal

[In the last post](https://dev.to/speratus/i-built-this-despite-a-flaw-in-rusts-webassembly-toolchain-38p2), I arrived at a working, but less than ideal solution for importing WebAssembly into an `AudioWorkletProcessor`. Even though I couldn't initially figure out how to import a WebAssembly module into a [worklet](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Using_AudioWorklet) using `wasm-pack`, the process of doing so without `wasm-pack` enabled me to identify a potential solution that uses `wasm-pack`.
The next step was trying to implement it.

## A Better Idea

Once I understood the process of importing a WebAssembly module into a worklet, doing so using `wasm-pack` proved to be fairly simple. There are two general steps:

1. Compile the WebAssembly module using the `wasm-pack` `web` target. The `web` target creates JavaScript code that works without a bundler or Node.
2. Copy the JavaScript into the `AudioWorkletProcessor` file and adapt it to the context of `AudioWorkletGlobalScope`.

## Explanation

Since the `web` target is meant to work without the need for a bundler, the resulting JvaScript code can be copied into the `AudioWorkletProcessor` file and will only require minor changes. This approach is currently the best option because the `AudioWorkletGlobalScope` lacks the features required to import WebAssembly modules.

## Implementing the Solution

The first step is to build the Rust code with the `web`
```sh
wasm-pack build --target web
```

Once the code is compiled, I opened the JavaScript file named "<LIBRARY_NAME>_bg.js". This file contains the WebAssembly bindings generated by `wasm-pack`. I copied everything in this file and pasted into my `AudioWorkletProcessor` class. This is the only way to get the JavaScript bindings into the `AudioWorkletGlobalScope`.

Next, I copied the `.wasm` file to a location where the browser could load it using `fetch()`. At the top of my worklet file, I had the following code which I knew would need to be adjusted.
```js
// processor.js

let wasm;

const cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });

cachedTextDecoder.decode();

let cachedUint8Memory0 = null;

function getUint8Memory0() {
    if (cachedUint8Memory0 === null || cachedUint8Memory0.byteLength === 0) {
        cachedUint8Memory0 = new Uint8Array(wasm.memory.buffer);
    }
    return cachedUint8Memory0;
}

// wasm-pack generated code ...

export { initSync }
export default init;

class WebSynthProcessor extends AudioWorkletProcessor {
    // ...
}
```

I replaced this code with the following:
```js
// processor.js

let wasm;

let cachedTextDecoder;

// decode call will be made later

let cachedUint8Memory0 = null;

function getUint8Memory0() {
    if (cachedUint8Memory0 === null || cachedUint8Memory0.byteLength === 0) {
        cachedUint8Memory0 = new Uint8Array(wasm.memory.buffer);
    }
    return cachedUint8Memory0;
}

// wasm-pack generated JavaScript

// Notice, I removed the exports.

class WebSynthProcessor extends AudioWorkletProcessor {
    // ...
}
```

An Instance of `TextDecoder` class needs to be available in the `AudioWorkletGlobalScope`, but the audio worklet scope does not allow the `TextDecoder` class's constructor to be used. I used the `AudioWorkletProcessor`'s `MessagePort` to give the audio worklet a `TextDecoder` instance.

### Setting Up the WebAssembly Module

In order to pass the WebAssembly module to the `AudioWorkletProcessor`, I knew I would need to call `WebAssembly.compile()` in the main thread and pass it to my audio processor. To do this, I needed to call both `fetch()` and `WebAssembly.compileStreaming()` like this:
```js
// index.js

// We will need the worklet node later.
let node = new AudioWorkletNode(context, 'web-synth-proto');

WebAssembly.compileStreaming(fetch('/path/to/library.wasm')).then(module => {
    // Pass the module to the AudioWorkletProcessor
});
```

I originally did this in an `async` function so that I could use `await` to get the return value directly. Here, I'm using a callback with a promise because this pattern is more common.
Once I had a `WebAssembly` module instance, I passed it to the `AudioWorkletProcessor` using the `MessagePort` instance like this:
```js
// index.js

WebAssembly.compileStreaming(fetch('/path/to/library.wasm')).then(module => {
    // Now that the module has been created, it can be passed to the processor.
    node.port.postMessage({type: 'init-wasm', wasmData: module});
});
```

### Using the Module in the AudioWorkletProcessor

To use the WebAssembly module, it needed to be instantiated from inside the audio worklet thread. The first step in this process was to set up a message listener for for the `MessagePort` instance. I did this in the constructor of the `AudioWorkletProcessor`.
```js
// processor.js

class WebSynthProcessor extends AudioWorkletProcessor {
    constructor(options) {
        super(options);
       

        this.port.onmessage = event => this.onmessage(event.data);
        
    }

    onmessage(data) {
        // Check to make sure the message we receive is the correct type.
        if (data.type === 'init-wasm') {                                 
            // Finish instantiating the WebAssembly module
        }
    }
}
```

Now that there is an `onmessage` listener for the `MessagePort`, the worklet can listen for the message sent by the main thread.
```js
// processor.js

class WebSynthProcessor extends AudioWorkletProcessor {
    constructor(options) {
        super(options);

        this.port.onmessage = event => this.onmessage(event.data);
        
    }

    onmessage(data) {
        // Check to make sure the message we receive is the correct type.
        if (data.type === 'init-wasm') {                                 
            cachedTextDecoder = data.decoder;
    
            // Load returns a promise
            load(data.wasmData, getImports()).then(mod => {
                // Once the WebAssembly module has been instantiated, it needs to be finalized
                // so that it can be accessed later.
                finalizeInit(mod.instance, mod.module);
            });
        }
    }
}
```

Once the worklet receives the WebAssembly module from the main thread, it needs to call the `load()` function copied from the wasm-pack output so that the WebAssembly module can be properly instantiated. Finally, the audio processor thread needs to call the `finalizeInit()` function to ensure that it can continue to access the WebAssembly code.

Now that all the setup is complete, the audio thread can use the `SineOsc` struct by calling the JavaScript wrapper functions.
Here is how I setup the oscillator:
```js
// processor.js

class WebSynthProcessor extends AudioWorkletProcessor {
    constructor(options) {
        super(options);

        this.port.onmessage = event => this.onmessage(event.data);
        
    }

    onmessage(data) {
        // Check to make sure the message we receive is the correct type.
        if (data.type === 'init-wasm') {                                 
            cachedTextDecoder = data.decoder;
    
            // Load returns a promise
            load(data.wasmData, getImports()).then(mod => {
                // Once the WebAssembly module has been instantiated, it needs to be finalized
                // so that it can be accessed later.
                finalizeInit(mod.instance, mod.module);

                // Create the oscillator instance.
                this.osc = SineOsc.new(sampleRate);
            });
        }
    }
}
```

Now that the audio thread has created an oscillator, that oscillator can be called to do the actual sound generation.
```js
// processor.js

class WebSynthProcessor extends AudioWorkletProcessor {
    
    // ...

    process(inputs, outputs, parameters) {
        if (typeof this.osc !== 'undefined' && this.osc !== null) {
            // Get the first output channel.
            let output = outputs[0];

            output.forEach(channel => {
                // populate the channel's buffer with samples from the WebAssembly code.
                for (let i = 0; i < channel.length; i++) {
                    // Remember the first argument for the sample method is the pitch we want to 
                    // synthesize and the second argument is the volume.
                    let sample = this.osc.sample(440, 0.5);

                    channel[i] = sample;
                }
            });
        } else {
            console.log('wasm not instantiated yet');
        }
        return true;
    }
}
```

## What Next?

At this point, I have demonstrated how to  adapt `wasm-pack`'s output so that it can be used in an `AudioWorkletProcessor` file. The next step is to start building out the features necessary for a usable synthesizer.

> _If you want to read more about this project, feel free to follow me or read my other posts in this series._

*Introduction to the series:*
{% embed https://dev.to/speratus/building-a-browser-based-synthesizer-using-rust-and-webassembly-3kpl %}

*Source code for this prototype:*
{% embed https://github.com/speratus/web-synth-proto2 %}