Turbocharge your application development using WebAssembly with SingleStoreDB
Tags: singlestoredb, webassembly, sentimentanalysis, rust
Published: 2022-10-04 21:05:57

## Abstract
WebAssembly (Wasm) is a binary instruction format for a stack-based virtual machine. Wasm enables developers to use existing code from programming languages, such as C, C++ and Rust, as part of their application development process. However, Wasm is not just for the web, and today is moving in exciting new directions. For example, one use-case would be to run Wasm code in a database system on the data already stored in the database system - an example of co-locating computation with data. Using Wasm to extend the capabilities of a database system opens up opportunities to develop many new applications. SingleStoreDB supports Wasm through [Code Engine](https://docs.singlestore.com/managed-service/en/reference/code-engine---powered-by-wasm.html) and, in this article, we'll see how to build a Wasm UDF to perform sentiment analysis on data already stored in SingleStoreDB.

## Introduction
We'll need to perform a few steps to prepare our development environment, and the following sections will show how to do this. We'll also use Rust to create our Wasm UDF. Our code example will build upon the example described in the SingleStore blog post [[r]evolution Summer 2022: Bring Application Logic to Your Data With SingleStoreDB Code Engine for Wasm](https://www.singlestore.com/blog/revolution-2022-code-engine-for-wasm/).

## Create a SingleStoreDB Cloud account
To use Wasm on SingleStoreDB Cloud, we'll first create a free account from the [portal](https://portal.singlestore.com/). After we have logged in to the portal, we need to check that we can see the option to **Create a new workspace group** in the left-hand navigation pane, as shown in Figure 1.

![Figure 1. Create a new workspace group.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/exxu3axy5glbb7m0emms.png)
<center>Figure 1. Create a new workspace group.</center>

In the centre of the webpage, we should also see the option to **Create Workspace**, as shown in Figure 2.

![Figure 2. Create Workspace.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wqlgw33jj1rm1kbx2sh5.png)
<center>Figure 2. Create Workspace.</center>

We'll click the **Create Workspace** button. On the next page, we'll be presented with three options:

1. **Workspace Group Name**. We'll call our group **Wasm Demo Group**.
2. **Cloud Provider**. We have the option to use AWS, GCP or Azure. In this article, we'll use **AWS**.
3. **Region**. A drop-down menu provides a list of supported regions. We can choose a region or use the default.

At the bottom of the webpage, we'll click **Next**.

On the next page, we'll be presented with three options:

1. **Workspace Details**. We'll call our workspace **wasm-demo**.
2. **Size**. S-00 will be fine for initial testing.
3. **Confirm Workspace Group**. In the drop-down menu, we should check that the **Wasm Demo Group** is selected.

At the bottom of the page, we'll click **Create Workspace**.

After a short time, the Workspace Group should be successfully created and available.

Under the **Overview** tab (Figure 3),

![Figure 3. Overview.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yuvf75992eh6tahqx6lq.png)

<center>Figure 3. Overview.</center>

if we scroll down, we'll see our **wasm-demo** workspace, as shown in Figure 4.

![Figure 4. wasm-demo workspace.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ykb5pccwp1kd0k13rir2.png)

<center>Figure 4. wasm-demo workspace.</center>

Using the **Connect** pulldown, we'll select **Connect Directly** to obtain the information we need to connect to SingleStoreDB Cloud using various clients, such as a MySQL CLI client. The CLI command should be similar to the following:

```shell
mysql -u admin \
      -h <TO DO> \
      -P 3306 \
      --default-auth=mysql_native_password \
      -p
```

We need a minor variation of this command so that we can also use local files, if necessary, as follows:

```shell
mysql --local-infile \
      -u admin \
      -h <TO DO> \
      -P 3306 \
      --default-auth=mysql_native_password \
      -p
```

The `<TO DO>` for host (`-h`) being replaced with the value obtained from SingleStoreDB Cloud.

In Figure 3, we can also see **Access** and **Firewall** tabs. We'll change these as follows:

- **Access**. We'll create a password using the **Generate Strong Password** button and save the password in a safe place.
- **Firewall**. For initial setup and testing, we'll allow **Access from anywhere**. We can change this later.

## Setup local Wasm development environment
We can quickly create a local Wasm development environment using a few steps. However, SingleStore also provides a Docker Container as an alternative. We'll discuss this alternative later in this article.

### Install the software
First, we'll download the [wasi-sdk](https://github.com/WebAssembly/wasi-sdk/releases). We'll use **wasi-sdk-16.0-linux.tar.gz**, the latest version available when writing this article. We'll move the file to the `/opt` directory and unpack it as follows:

```shell
cd /opt

sudo cp /path/to/wasi-sdk-16.0-linux.tar.gz .

sudo tar xzvf wasi-sdk-16.0-linux.tar.gz
```

We'll replace `/path/to/` with the actual path where we downloaded the file. We'll also need to ensure that we add the `bin` directory to our `PATH` variable, as follows:

```shell
export PATH=/opt/wasi-sdk-16.0/bin:$PATH
```

Second, we'll download and install the Rust toolchain, as follows:

```shell
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

To configure the current shell, we'll need to run:

```shell
source "$HOME/.cargo/env"
```

Finally, we'll install `wit-bindgen`, as follows:

```shell
cargo install --git https://github.com/bytecodealliance/wit-bindgen wit-bindgen-cli
```

To deploy our Wasm module to SingleStoreDB, we'll use the `pushwasm` tool. First, we'll clone the GitHub repo to a convenient location:

```shell
git clone https://github.com/singlestore-labs/pushwasm
```

Next, we'll change to the `pushwasm` directory and build the code, as follows:

```shell
cd pushwasm

cargo build --release
```

A new file should be written to `target/release/pushwasm`, and this should be added to our `PATH` variable.

**NOTE**: We may also need to run the following to ensure a successful `pushwasm` build:

```shell
sudo apt install libssl-dev
```

### Initialise the source tree
Next, let's create a new directory called `workdir` in our home folder:

```shell
cd

mkdir workdir

cd workdir
```

From the `workdir`, we'll now create a skeletal Rust source tree, as follows:

```shell
cargo init --vcs none --lib
```

and we'll need to add `wasm32-wasi` to the Rust toolchain as it is not installed by default:

```shell
rustup target add wasm32-wasi
```

### Create the .wit file
The code we'll use below for our Wasm UDF is also available on [GitHub](https://github.com/singlestore-labs/singlestore-wasm-toolkit/tree/main/examples/rust/sentimentable).

In our `workdir`, we'll now create a file called `sentimentable.wit` that contains the interface definition. In this file, we'll add the following:

```
record polarity-scores {
    compound: float64,
    positive: float64,
    negative: float64,
    neutral: float64,
}

sentimentable: func(input: string) -> list<polarity-scores>
```

We'll define a function `sentimentable` that will take a string, perform sentiment analysis on that string and return a list of polarity scores.

### Implement and compile
In our `workdir` we'll replace the existing contents of `Cargo.toml` with the following code:

```
[package]
name = "sentimentable"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wit-bindgen-rust = { git = "https://github.com/bytecodealliance/wit-bindgen.git", rev = "60e3c5b41e616fee239304d92128e117dd9be0a7" }
vader_sentiment = { git = "https://github.com/ckw017/vader-sentiment-rust" }
lazy_static = "1.4.0"

[lib]
crate-type = ["cdylib"]
```

Now we need to add the code for `sentimentable`, so we'll navigate to the `src` directory in our `workdir` and locate the `lib.rs` file. In the `lib.rs` file, we'll replace the existing contents with the following code:

```rust
wit_bindgen_rust::export!("sentimentable.wit");
use crate::sentimentable::PolarityScores;
struct Sentimentable;
impl sentimentable::Sentimentable for Sentimentable {

    fn sentimentable(input: String) -> Vec<PolarityScores> {
        lazy_static::lazy_static! {
            static ref ANALYZER: vader_sentiment::SentimentIntensityAnalyzer<'static> =
                vader_sentiment::SentimentIntensityAnalyzer::new();
        }

        let scores = ANALYZER.polarity_scores(input.as_str());
        vec![PolarityScores {
            compound: scores["compound"],
            positive: scores["pos"],
            negative: scores["neg"],
            neutral: scores["neu"],
        }]
    }
}
```

Our code uses [VADER](https://github.com/cjhutto/vaderSentiment) (Valence Aware Dictionary and sEntiment Reasoner). VADER is a lexicon and rule-based sentiment analysis tool that can interpret and classify emotions.

Next, we'll go back up one directory level:

```shell
cd ..
```

We'll now build the Wasm module:

```shell
cargo build --target wasm32-wasi --release
```

A new Wasm file should be written to `target/wasm32-wasi/release/sentimentable.wasm`.

### Deploy
From our MySQL CLI client, we'll create a new database and switch to it, as follows:

```sql
CREATE DATABASE demo;

USE demo;
```

Next, from the command line, we'll use the `pushwasm` tool to push our Wasm module into SingleStoreDB, as follows:

```shell
pushwasm --tvf \
         --prompt \
         --wit ./sentimentable.wit \
         mysql://admin@<TO DO>/demo \
         ./target/wasm32-wasi/release/sentimentable.wasm \
         sentimentable
```

The `<TO DO>` for the host being replaced with the value obtained from SingleStoreDB Cloud. We'll be prompted for the password we created earlier when creating the workspace.

After a short time, we should see the following message:

```
Wasm TVF 'sentimentable' was created successfully.
```

From our MySQL CLI client, we can also check that the function was created using:

```sql
SHOW FUNCTIONS;
```

### Run in the database
We can quickly test our function, as follows:

```sql
SELECT * FROM sentimentable('The movie was great');
```

The result should be similar to the following:

```
+------------------+------------------+----------+-----------------+
| compound         | positive         | negative | neutral         |
+------------------+------------------+----------+-----------------+
| 0.62489332693894 | 0.57746478873239 |        0 | 0.4225352112676 |
+------------------+------------------+----------+-----------------+
1 row in set (0.10 sec)
```

VADER can consider capitalisation, so we can try:

```sql
SELECT * FROM sentimentable('The movie was GREAT!');
```

The result should be similar to the following:

```
+------------------+------------------+----------+-----------------+
| compound         | positive         | negative | neutral         |
+------------------+------------------+----------+-----------------+
| 0.72902590497990 | 0.63076923076923 |        0 | 0.3692307692307 |
+------------------+------------------+----------+-----------------+
1 row in set (0.11 sec)
```

We can see that the values have changed, showing a stronger positive sentiment expressed by capitalisation.

## Bonus: Use the Wasm UDF with real sentiment data
We can try a larger-scale test of the Wasm UDF using real sentiment data. One popular dataset is the [Large Movie Review Dataset](https://ai.stanford.edu/~amaas/data/sentiment/):

```
@InProceedings{maas-EtAl:2011:ACL-HLT2011,
  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
  title     = {Learning Word Vectors for Sentiment Analysis},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  month     = {June},
  year      = {2011},
  address   = {Portland, Oregon, USA},
  publisher = {Association for Computational Linguistics},
  pages     = {142--150},
  url       = {http://www.aclweb.org/anthology/P11-1015}
}
```

We'll download a [compressed file](https://github.com/VeryFatBoy/wasm/blob/main/imdb-reviews/imdb-reviews.zip) from GitHub and extract the CSV file. The CSV file contains 25,000 rows consisting of two columns:

1. Text
2. Sentiment (0 = positive, 1 = negative)

In our MySQL CLI client, we'll create a table:

```sql
USE demo;

CREATE TABLE imdb_reviews (
   text TEXT,
   sentiment INT
);
```

We'll now load the data into SingleStoreDB:

```sql
USE demo;

LOAD DATA LOCAL INFILE '/path/to/imdb-reviews.csv'
INTO TABLE imdb_reviews
IGNORE 1 LINES
COLUMNS TERMINATED BY ','
ENCLOSED BY '"';
```

We'll replace `/path/to/` with the actual path to the CSV file.

Once the data are loaded, we can run queries such as the following:

```sql
SELECT SUBSTRING(i.text, 1, 30) AS text,
  FORMAT(s.compound, 3) cpd,
  FORMAT(s.positive, 3) pos,
  FORMAT(s.negative, 3) neg,
  FORMAT(s.neutral, 3) ntrl
FROM imdb_reviews i, sentimentable(i.text) s
LIMIT 10;
```

The result should be similar to the following:

```
+--------------------------------+--------+-------+-------+-------+
| text                           | cpd    | pos   | neg   | ntrl  |
+--------------------------------+--------+-------+-------+-------+
| This is an interesting left tu | 0.996  | 0.126 | 0.060 | 0.813 |
| I don't think any movie of Van | 0.976  | 0.242 | 0.105 | 0.653 |
| This has to be THE WORST film  | -0.572 | 0.103 | 0.129 | 0.768 |
| When I saw this trailer on TV  | 0.968  | 0.155 | 0.014 | 0.831 |
| As an adult I really did enjoy | 0.896  | 0.209 | 0.046 | 0.744 |
| Another example of the unique  | 0.979  | 0.280 | 0.000 | 0.720 |
| This film was a waste of time, | 0.678  | 0.126 | 0.073 | 0.801 |
| I thought this was a splendid  | 0.988  | 0.142 | 0.085 | 0.774 |
| I can only believe that Garson | 0.968  | 0.244 | 0.119 | 0.637 |
| Gods...where to start. I was o | -0.956 | 0.092 | 0.131 | 0.777 |
+--------------------------------+--------+-------+-------+-------+
10 rows in set (0.28 sec)
```

## Next steps
The [Code Engine - Powered by Wasm](https://docs.singlestore.com/managed-service/en/reference/code-engine---powered-by-wasm.html) website is a great place to learn more about Wasm support in SingleStoreDB.

SingleStore offers a free [Wasm training course](https://training.singlestore.com/learn/course/745/code-engine-powered-by-wasm). The course will show the participant how to set up a development environment using a Container. The course will also guide the participant through various examples that are also available through the [SingleStore Wasm Toolkit](https://github.com/singlestore-labs/singlestore-wasm-toolkit).

A [Wasm tutorial](https://singlestore-labs.github.io/singlestore-wasm-toolkit/html/Tutorial-Overview.html) is also available that works through several examples in various programming languages.

The SingleStore YouTube channel also contains a video about [Code Engine Powered by Wasm](https://www.youtube.com/watch?v=1KlS5YK5TA8) from the Summer 2022 launch event.

## Summary
Wasm support in SingleStoreDB Cloud is available today and can be used by anyone. SingleStore is one of the first database vendors to provide Wasm support. Through the worked example in this article, we have seen that Wasm provides great potential to extend the capabilities of the database engine in a safe and controlled way. It also opens up the possibilities to reuse code that developers may have built in other programming languages.

Contact us at SingleStore and let us know how you are using Wasm in SingleStoreDB. We'd love to hear about all the great use cases!

## Acknowledgements
I thank my colleague [Peter Vetere](https://www.linkedin.com/in/pvetere/) at SingleStore for his assistance and patience during the development of the code example in this article.