Running a Javascript VM in Golang: Data Transforms via User Scripts
Tags: javascript, backend, go
Published: 2024-06-28 15:39:57

---
title: Running a Javascript VM in Golang: Data Transforms via User Scripts
published: true
date: 2024-06-28 15:32:13 UTC
tags: javascript,backend,go,golang
canonical_url: https://didil.medium.com/running-a-javascript-vm-in-golang-data-transforms-via-user-scripts-5560b0ced0d5
---

![Javascript VM in Golang](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x6wuv9f9va8skgn2hu22.png)
_Javascript VM in Golang_

### Introduction

Integrating user-defined scripts into backend systems can significantly enhance the flexibility and power of your applications. In this article, we will explore how to run a **JavaScript Virtual Machine** (VM) within a **Go** (Golang) application. Specifically in this example, we’ll focus on performing **HTTP body and headers transformations** via user-provided scripts.

We will be using the [**goja**](https://github.com/dop251/goja) library, a JavaScript VM written in Go, to execute JavaScript code. This setup allows developers to provide custom transformation logic for http payloads and headers.

### The Project

[Inhooks](https://github.com/didil/inhooks) is an open source project aiming to build a lightweight incoming webhooks gateway solution. As some users have requested a way to **transform http messages** before forwarding them to targets, I have been looking for a proper solution.

![Simplified Inhooks Architecture for Transforms](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6va9myxko4a3j3kndemi.png)
_Simplified Inhooks Architecture for Transforms_

### High-Level Overview

Our system listens for incoming HTTP requests, processes the payloads using a JavaScript VM, and applies user-defined transformations to the HTTP body and headers. This allows dynamic and customizable data processing, which can be very useful in various integration scenarios.

#### Features

- **Execute JavaScript in a Go application:** Run Javascript user scripts provided at runtime.
- **Transform HTTP payloads and headers** : Give users a powerful and easy-to-use method to modify incoming requests data.
- **Security and Error handling** : The VM is isolated and does not have access to the network or file system. Robust error handling ensures smooth operation and easy debugging.

#### Code Walkthrough

Let’s dive into the implementation. Below is a snippet of the core transformation function written in Go. This function takes in the original HTTP body and headers, applies the user-defined transformation script, and returns the transformed data.

```
package main

import (
    "fmt"
    "github.com/dop251/goja"
    "net/http"
    "encoding/json"
)

// transformPayload function to apply user-defined JavaScript transformations
func transformPayload(jsScript string, body []byte, headers http.Header) ([]byte, http.Header, error) {
    // Create a new JavaScript VM
    vm := goja.New()

    // Set the HTTP body in the VM
    bodyStr := string(body)
    err := vm.Set("bodyStr", bodyStr)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to set bodyStr: %w", err)
    }

    // Marshal headers to JSON and set in the VM
    headersStr, err := json.Marshal(headers)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to marshal headers to JSON: %w", err)
    }
    err = vm.Set("headersStr", string(headersStr))
    if err != nil {
        return nil, nil, fmt.Errorf("failed to set headersStr: %w", err)
    }

    // Prepare the full script with user function
    fullScript := fmt.Sprintf(`
        /* User Function */
        %s
        /* End User Function */

        const headers = JSON.parse(headersStr);
        var results = transform(bodyStr, headers);
        [results[0], results[1]];
    `, jsScript)

    // Run the script
    val, err := vm.RunString(fullScript)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to execute JavaScript: %w", err)
    }

    // Get the results from the script
    results := val.Export().([]interface{})
    if len(results) != 2 {
        return nil, nil, fmt.Errorf("expected 2 results in js transform, got %d", len(results))
    }

    // Extract and type assert the transformed payload and headers
    transformedPayloadStr, ok := results[0].(string)
    if !ok {
        return nil, nil, fmt.Errorf("expected payload to be of type string, got %T", results[0])
    }
    transformedHeadersTemp, ok := results[1].(map[string]interface{})
    if !ok {
        return nil, nil, fmt.Errorf("expected headers to be of type map[string]interface{}, got %T", results[1])
    }

    // Rebuild the header object
    transformedHeaders := http.Header{}
    for k, values := range transformedHeadersTemp {
        valuesArr, ok := values.([]interface{})
        if !ok {
            return nil, nil, fmt.Errorf("expected header values to be of type []string, got %T", values)
        }

        stringValuesArr := make([]string, len(valuesArr))
        for i, value := range valuesArr {
            stringValue, ok := value.(string)
            if !ok {
                return nil, nil, fmt.Errorf("expected header value to be of type string, got %T", value)
            }
            stringValuesArr[i] = stringValue
        }
        transformedHeaders[k] = stringValuesArr
    }

    // Return the final results
    return []byte(transformedPayloadStr), transformedHeaders, nil
}
```

What the code above does is:

1. **Setting Up the VM:** We initialize a new JavaScript VM using _goja.New()_ and set the HTTP body and headers as variables within the VM.
2. **Running the Script:** The user-defined script is embedded within a template that parses and transforms the body and headers.
3. **Handling Results** : The transformed payload and headers are extracted, type-checked, and returned.

The users then provide a user script such as the one below in the Inhooks configuration file. This example handles a JSON body, adds a http header _X-INHOOKS-TRANSFORMED_, converts the _body.msg_ field to upper case if it is present and deletes the _body.my\_dummy\_key_ field.

```
      function transform(bodyStr, headers) {
        const body = JSON.parse(bodyStr);

        // add a header
        headers["X-INHOOKS-TRANSFORMED"] = ["1"];
        // capitalize the message if present
        if (body.msg) {
          body.msg = body.msg.toUpperCase();
        }
        // delete a key from the body
        delete body.my_dummy_key;

        return [JSON.stringify(body), headers];
      }
```

The following unit tests make sure the code is working properly:

```
package services

import (
 "context"
 "net/http"
 "testing"
 "time"

 "github.com/didil/inhooks/pkg/lib"
 "github.com/didil/inhooks/pkg/models"
 "github.com/stretchr/testify/assert"
)

func TestMessageTransformer_Transform_Javascript(t *testing.T) {
 config := &lib.TransformConfig{
  JavascriptTimeout: 5000 * time.Millisecond,
 }

 mt := NewMessageTransformer(config)

 m := &models.Message{
  Payload: []byte(`{
    "name": "John Doe",
    "age": 30,
    "locations": ["New York", "London", "Tokyo"],
    "scores": [85, 90, 78, 92]
   }`),
  HttpHeaders: http.Header{
   "Content-Type": []string{"application/json"},
   "X-Request-Id": []string{"123"},
   "Authorization": []string{"Bearer token123"},
  },
 }
 transformDefinition := &models.TransformDefinition{
  Type: models.TransformTypeJavascript,
  Script: `
   function transform(bodyStr, headers) {
    const body = JSON.parse(bodyStr);
    body.username = body.name;
    delete body.name;
    delete body.age;
    body.location_count = body.locations.length;
    body.average_score = body.scores.reduce((a, b) => a + b, 0) / body.scores.length;
    headers["X-AUTH-TOKEN"] = [headers.Authorization[0].split(' ')[1]];
    delete headers.Authorization;
    return [JSON.stringify(body), headers];
   }
   `,
 }
 err := mt.Transform(context.Background(), transformDefinition, m)
 assert.NoError(t, err)

 assert.JSONEq(t, `{"username":"John Doe","locations":["New York","London","Tokyo"],"scores":[85,90,78,92],"average_score":86.25,"location_count":3}`, string(m.Payload))
 assert.Equal(t, http.Header{"Content-Type": []string{"application/json"}, "X-AUTH-TOKEN": []string{"token123"}, "X-Request-Id": []string{"123"}}, m.HttpHeaders)
}

func TestMessageTransformer_Transform_Javascript_Error(t *testing.T) {
 config := &lib.TransformConfig{
  JavascriptTimeout: 5000 * time.Millisecond,
 }

 mt := NewMessageTransformer(config)

 m := &models.Message{
  Payload: []byte(`{
    "name": "John Doe",
    "age": 30,
    "locations": ["New York", "London", "Tokyo"],
    "scores": [85, 90, 78, 92]
   }`),
  HttpHeaders: http.Header{
   "Content-Type": []string{"application/json"},
   "X-Request-Id": []string{"123"},
   "Authorization": []string{"Bearer token123"},
  },
 }
 transformDefinition := &models.TransformDefinition{
  Type: models.TransformTypeJavascript,
  Script: `
   function transform(bodyStr, headers) {
    const body = JSON.parse(bodyStr);
    throw new Error("random error while in the transform function");
    return [JSON.stringify(body), headers];
   }
   `,
 }
 err := mt.Transform(context.Background(), transformDefinition, m)
 assert.ErrorContains(t, err, "failed to transform message: failed to execute JavaScript: Error: random error while in the transform function")
}
```

### **Conclusion**

Adding a JavaScript VM to the Go project turned out to be quite easy and allows easily customizable data transformations. The **goja** library provides an efficient and straightforward way to execute JavaScript code in Go without using V8/CGO. Another option would have been to use lua scripts via [**gopher-lua**](https://github.com/yuin/gopher-lua) for example. That could be an idea for a future project.

Feel free to try out the code, adapt it to your own needs and share your feedback. Happy coding!