Write once, run anywhere with Deno and Dnt
Tags: typescript, deno, node, cloudflare
Published: 2022-07-06 12:34:33

Since multiple non-browser JavaScript runtimes has been emerged as well as web browsers, I've tried Deno-based  module development which can support multiple web browsers, Node.js and Cloudflare Workers as well. I will share my efforts as a case study.

## Introduction
I recently implemented a TypeScript module named hpke-js:

{% embed https://github.com/dajiaji/hpke-js %}

[HPKE (Hybrid Public Key Encryption)](https://datatracker.ietf.org/doc/html/rfc9180) is, roughly speaking, a standard for exchanging public keys to create a shared key for secure end-to-end encryption. One of my goals was to implement this on the [Web Cryptography API]() and guarantee that it works with multiple JS runtimes that support this API (major web browsers, Node.js, Deno, Cloudflare Workers), in other words, to properly incorporate testing in all runtimes into CI/CD.

I started out implementing it as an npm package, but in the process of making it Deno-compatible, I made a major rewrite to make it a Deno-based structure. This allowed me to build a much cleaner development and CI/CD flow for TypeScript/JavaScript modules that work on Chrome, Firefox, Safari, Node.js, Cloudflare Workers, and Deno.

Specifically,  make my codebase be for Deno, use Deno built-in formatter, linter and testing, and use [dnt (Deno to Node Transform)](https://github.com/denoland/dnt) to generate an npm package containing ESM code and to test generated code. For testing in the browser, deploy the test content linked to the generated ESM code to Github Pages and use [playwright/test](https://playwright.dev/docs/api/class-test). For Cloudflare Workers, use [wrangler](https://github.com/cloudflare/wrangler) to set up a worker locally for testing. A schematic diagram is shown below.


![Overview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jd74czxk7yd3ihe38k9l.png)

I made the building/testing flow in this diagram be done by using Github Actions at pull request time and merge time to the master branch, and also made the flow including deployment be done at the release time by using Github Actions as well.

In this article, I will introduce the definition and overview of Deno-based "JS runtime-independent module" development, the various tools used and their settings, and CI/CD on Github, using hpke-js as an example to build the above flow.

This article is intended for modules that use APIs provided by JS runtimes, such as the Web Cryptography API, but still want to ensure portability. If it is obvious that your module is runtime-independent, there is no need to build a CI like the one introduced here.

## Table of Contents
- [Definition](#definition)
- [JS Runtime-Independent Module Development](#js-runtime-independent-module-development)
    - [Register your module to major Registries](#register-your-module-to-major-registries)
    - [Directory Structure](#directory-structure)
- [Tools and the Settings](#tools-and-the-settings)
    - [deno](#deno)
    - [dnt](#dnt)
    - [playwright/test](#playwrighttest)
    - [wrangler](#wrangler)
    - [eggs](#eggs)
- [CI/CD on Github](#ci-cd-on-github)
    - [CI for Deno](#ci-for-deno)
    - [CI for Browsers](#ci-for-browsers)
    - [CI for Node.js](#ci-for-nodejs)
    - [CI for Cloudflare Workers](#ci-for-cloudflare-workers)
    - [Delivery](#delivery)
- [Remaining Problems](#remaining-problems)
- [Conclusion](#conclusion)

## Definition
In this article, "JS runtime-independent modules" refers to modules that, after release, will be available in each JS runtime as follows:

**Browsers**: It is available in ESM format in browsers from major CDN services (esm.sh, Skypack, etc.). It has been tested and guaranteed to work in Chrome (Blink), Firefox (Gecko), and Safari (WebKit) before release.

```html
<script type="module">
  import * as hpke from "https://esm.sh/hpke-js@0.13.0";
  // import * as hpke from "https://cdn.skypack.dev/hpke-js@0.13.0";
</script>
```

**Node.js**: It can be installed with `npm` or `yarn` and is available in both ESM and CommonJS formats. It has been tested and guaranteed to work with all Node.js versions that claim to support it.

```js
// CommonJS
const hpke = require("hpke-js");
// or ESM
// import * as hpke from "hpke-js";
```

**Deno**: It can be installed via major registries such as `deno.land` and `nest.land`. It has been tested and guaranteed to work with all Deno major versions (currently only `1.x`) that claim to support it.

```js
import * as hpke from "https://deno.land/x/hpke@0.13.0/mod.ts";
// import * as hpke from "https://x.nest.land/hpke@0.13.0/mod.ts";
```

**Cloudflare Workers**: The single-filed module that is downloaded from various CDNs or emitted by `deno bundle`, can be included in a Cloudflare Worker package and can be used.

```sh
# download from a CDN (esm.sh)
curl -o $YOUR_PATH/hpke.js https://esm.sh/v86/hpke-js@0.13.0/es2022/hpke-js.js
# or downlaod a minified version from a CDN
curl -o $YOUR_PATH/hpke.js https://esm.sh/v86/hpke-js@0.13.0/es2022/hpke.min.js
# or use `deno bundle`
deno bundle https://deno.land/x/hpke@0.13.0/mod.ts > $YOUR_PATH/hpke.js
```

```js
// then import and use it
import * as hpke from "./hpke.js";
```

## JS Runtime-Independent Module Development
As mentioned in [Introduction](#introduction), the point is to develop it as a Deno module and use [dnt (Deno to Node Transform)](https://github.com/denoland/dnt) to convert it into code that works with other JS runtimes.

All you need to do is read the official documentation ([README](https://github.com/denoland/dnt) and [doc.deno](https://doc.deno.land/https://deno.land/x/dnt/mod.ts)) and develop with portability in mind, but here are the main points to keep in mind, in my opinion:

- Basically, do not use Deno-dependent funcitons. However, if you have to use a Deno namespace feature or any other feature that affects portability, check to see if it has a shim that is injected when converting to an npm package with dnt (see [node_deno_shims](https://github.com/denoland/node_deno_shims). For example, the implementation status of the shims is listed up [here](https://github.com/denoland/node_deno_shims/blob/main/packages/shim-deno/PROGRESS.md)). Using shim will ensure that it works on Node.js.
- If your module has dependent packages, use [esm.sh](https://esm.sh/) or [Skypack](https://www.skypack.dev/) as much as possible. If there are corresponding npm packages, dnt will map them to the `dependencies` in the output package.json. In other words, they are treated as external modules.
- The entry point of the module should be `mod.ts` compliant with customary in Deno.
- Since git tags are used for versioning in `deno.land`, make the tag name [SemVer](https://semver.org/) compliant (e.g., `1.2.3`). `v1.2.3` is also fine, but this will cause inconsistencies in the way of specifying versions in various CDNs (sometimes with `v` and sometimes without). I recommend that you use `1.2.3` without `v`.
- If you want to output CommonJS/UMD format modules, do not use Top-level await.

NOTE: It should go without saying, but please keep in mind that even though a shim is provided as a mitigation/workaround, the basic premise is that portability cannot be basically ensured if non-standardized proprietary functions of a runtime are used.

## Register your module to major Registries
To develop a JS runtime-independent module, you should register your module to the following two registries in advance:

- [npmjs.com](https://www.npmjs.com/)
- [deno.land](https://deno.land/x)

Registration with `npmjs` is mandatory, and deploying here will also deploy to various CDNs([esm.sh](https://esm.sh/)、[Skypack](https://www.skypack.dev/)、[unpkg.com](https://unpkg.com/), etc.).

As a Deno module, we would still like to be able to distribute it in `deno.land`. You can register it by clicking `Publish a module` from the link above and following the instructions; note that a Github repository is required. Note that in this article, we will register the Deno module not only in `deno.land` but also in `nest.land`. It seems that `nest.land` is a blockchain-based immutable registry.

- [nest.land](https://nest.land/)

Another point to keep in mind is once you have decided on a module name, you should make sure that it is not registered in any of the above registries, and then pre-register it (I failed to do this...).

## Directory Structure
We will get down to business here. The next section will introduce the various tools and their settings, but before that, let's take a look at the directory structure of `hpke-js` and its important files.

In the past, we have to prepare package.json, package-lock.json, esbuild scripts and configuration files for eslint, jest, typescript, typedoc, etc. It tended to get messy. But after changing to Deno-based developement, it is a little cleaner. There are four configuration files in the top directory, but egg.json is not important, so there are only three files.

- deno.json: settings for deno.
- dnt.ts: configuration and execution script for dnt.
- import-map.json: for aggregating version descriptions of dependent libraries.
- egg.json: for deploying to nest.land, not necessary if only deno.land is needed.

```sh
.
├── deno.json
├── dnt.ts
├── egg.json
├── import-map.json
├── mod.ts
├── README.md
├── src
│   └── *.ts
└── test
    ├── *.test.ts  # Unit tests for Deno, which can be transformed and executed for other runtimes.
    ├── pages      # E2E Test contents for browsers.
    │   ├── index.html
    │   └── src
    ├── playwright # E2E tests for Deno.
    │   ├── hpke.spec.ts
    │   ├── package.json
    │   └── playwright.config.ts
    └── wrangler   # E2E tests for Cloudflare Workers.
        ├── hpke.spec.ts
        ├── package.json
        ├── src
        │   └── index.js
        └── wrangler.toml
```

## Tools and the Settings
I'll introduce the following tools but do not explain how to install or how to use them basically. Please refer to the official documentation for each. Basically, I will only put my setup and introduce some key points.
- deno
- dnt
- playwright/test
- wrangler
- eggs

## deno
I like that deno has a built-in formatter (fmt), linter (lint), test (test), and documentation (doc). It is very `Cargo` like.

The deno configuration file (`deno.json`) is optional and does not need to be present, but for development efficiency, it is better to register a series of commands used in development and CI in tasks and so on. 

First of all, I'll put [hpke-js/deno.json](https://github.com/dajiaji/hpke-js/blob/main/deno.json).

```json
{
  "fmt": {
    "files": {
      "include": [
        "README.md",
        "CHANGES.md",
        "deno.json",
        "dnt.ts",
        "egg.json",
        "import-map.json",
        "samples/",
        "src/",
        "test/"
      ],
      "exclude": [
        "samples/node/node_modules",
        "samples/ts-node/node_modules",
        "src/bundles",
        "test/playwright/node_modules",
        "test/wrangler"
      ]
    }
  },
  "lint": {
    "files": {
      "include": ["samples/", "src/", "test/"],
      "exclude": [
        "samples/node/node_modules",
        "samples/ts-node/node_modules",
        "src/bundles",
        "test/playwright/node_modules",
        "test/wrangler"
      ]
    }
  },
  "importMap": "./import-map.json",
  "tasks": {
    "test": "deno fmt && deno lint && deno test test -A --fail-fast --doc --coverage=coverage --jobs --allow-read",
    "dnt": "deno run -A dnt.ts $(git describe --tags $(git rev-list --tags --max-count=1))",
    "cov": "deno coverage ./coverage --lcov --exclude='test' --exclude='bundles'",
    "minify": "deno bundle ./mod.ts | esbuild --minify"
  }
}
```

The points are as follows:
- `fmt` supports markdown and json, so README.md and so on should be included in the target.
- Since hpke-js uses npm for e2e testing and so on, exclude node_module from `fmt` and `lint`. 
- If you use imprt-map, you should use `"importMap": ". /import-map.json"` is required.
- In `tasks.test`, both `deno fmt` and `deno lint` are executed at once.
- In `tasks.dnt`, specify the version to put in package.json with `$(git describe...) `.

## dnt
dnt (Deno to Node Transform) is a build tool that creates npm packages from code for Deno. It is best to look at the official documentation ([README](https://github.com/denoland/dnt) and [doc.deno](https://doc.deno.land/https://deno.land/x/dnt@0.27.0/mod.ts)). 

But I'll expose [hpke-js/dnt.ts](https://github.com/dajiaji/hpke-js/blob/main/dnt.ts) as an example.

```ts
import { build, emptyDir } from "dnt";

await emptyDir("./npm");

await build({
  entryPoints: ["./mod.ts"],
  outDir: "./npm",
  typeCheck: true,
  test: true,
  declaration: true,
  scriptModule: "umd",
  importMap: "./import-map.json",
  compilerOptions: {
    lib: ["es2021", "dom"],
  },
  shims: {
    deno: "dev",
  },
  package: {
    name: "hpke-js",
    version: Deno.args[0],
    description:
      "A Hybrid Public Key Encryption (HPKE) module for web browsers, Node.js and Deno",
    repository: {
      type: "git",
      url: "git+https://github.com/dajiaji/hpke-js.git",
    },
    homepage: "https://github.com/dajiaji/hpke-js#readme",
    license: "MIT",
    main: "./script/mod.js",
    types: "./types/mod.d.ts",
    exports: {
      ".": {
        "import": "./esm/mod.js",
        "require": "./script/mod.js",
      },
      "./package.json": "./package.json",
    },
    keywords: [
      "hpke",
      // ...省略
    ],
    engines: {
      "node": ">=16.0.0",
    },
    author: "Ajitomi Daisuke",
    bugs: {
      url: "https://github.com/dajiaji/hpke-js/issues",
    },
  },
});

// post build steps
Deno.copyFileSync("LICENSE", "npm/LICENSE");
Deno.copyFileSync("README.md", "npm/README.md");
```

The points are as follows:
- If you want to emit UMD code, you should use `scriptModule: "umd"`.
- If you use imprt-map, you should use `"importMap": ". /import-map.json"` is required.

## playwright/test
This was my first time to use [playwright/test](https://playwright.dev/docs/api/class-test) and found it great. I was surprised at how easy it is to do E2E testing using a browser nowadays.

My [hpke-js/test/playwright/playwright.config.ts](https://github.com/dajiaji/hpke-js/blob/main/test/playwright/playwright.config.ts) is as follows:

```ts
import { devices, PlaywrightTestConfig } from "@playwright/test";

const config: PlaywrightTestConfig = {
  projects: [
    {
      name: "chromium",
      use: { ...devices["Desktop Chrome"] },
    },
    {
      name: "firefox",
      use: { ...devices["Desktop Firefox"] },
    },
    {
      name: "webkit",
      use: { ...devices["Desktop Safari"] },
    },
  ],
};
export default config;
```

For now, I activate `chromium`, `firefox` and `webkit` and I think it pretty much cover various browser environments.

Test code ([hpke-js/test/playwright/hpke.spec.ts](https://github.com/dajiaji/hpke-js/blob/main/test/playwright/hpke.spec.ts)) is as follows. Just 9 lines.

```ts
import { expect, test } from "@playwright/test";

test("basic test", async ({ page }) => {
  await page.goto("https://dajiaji.github.io/hpke-js/");
  await page.click("text=run");
  await page.waitForTimeout(5000);
  await expect(page.locator("id=pass")).toHaveText("45");
  await expect(page.locator("id=fail")).toHaveText("0");
});
```

Basically, since the functionality of the module has been confirmed to some extent exhaustively by unit tests, in E2E using the actual environment, we have prepared test contents that use the Web Cryptography API with all HPKE cipher suite combinations (KEM: 5 types * KDF: 3 types * AEAD: 3 types = 45) and just hit the test button and see the results.

## wrangler
[wrangler](https://github.com/cloudflare/wrangler) is a CLI tool for Cloudflare Workers.

We could have done the same test for browsers, but for Cloudflare Workers, we implemented a test API with the following interface:

```
/test?kem={KEM_ID}&kdf={KDF_ID}&aead={AEAD_ID}
```

I ran this as a local server with `wrangler dev --local=true` and used `deno test` to perform E2E testing against this server. As with `playwright/test` above, I just ran a basic test scenario to check the Web Cryptography API calls with all combinations of the HPKE ciphersuites.


- test API implementation: [hpke-js/test/wrangler/src/index.js](https://github.com/dajiaji/hpke-js/blob/main/test/wrangler/src/index.js)
- E2E test executed via `deno test`: [hpke-js/test/wrangler/hpke.spec.ts](https://github.com/dajiaji/hpke-js/blob/main/test/wrangler/hpke.spec.ts)

## eggs
[eggs](https://github.com/nestdotland/eggs) is a CLI tool to deploy a package to `nest.land`. My setting file is ([hpke-js/egg.json](https://github.com/dajiaji/hpke-js/blob/main/egg.json)) as follows. It's like a package.json.

```json
{
  "$schema": "https://x.nest.land/eggs@0.3.4/src/schema.json",
  "name": "hpke",
  "entry": "./mod.ts",
  "description": "A Hybrid Public Key Encryption (HPKE) module for web browsers, Node.js and Deno.",
  "homepage": "https://github.com/dajiaji/hpke-js",
  "files": [
    "./src/**/*.ts",
    "./src/**/*.js",
    "README.md",
    "LICENSE"
  ],
  "checkFormat": false,
  "checkTests": false,
  "checkInstallation": false,
  "check": true,
  "ignore": [],
  "unlisted": false
}
```

The points are as follows:
- You can define version information in `eggs.json`, but as with `dnt`, pass the latest tag information with the command argument (see `eggs publish` in [Delivery](#derivery)).

## CI/CD on Github
Using the various tools described in the previous section, the flows in the diagram in [Introduction](#introduction) are straightforwardly dropped into Github Actions. In this section, I show each yml file for the following GitHub Actions.
- CI for Deno
- CI for Browsers
- CI for Node.js
- CI for Cloudflare Workers
- Delivery

### CI for Deno
[hpke-js/.github/workflows/ci.yml](https://github.com/dajiaji/hpke-js/blob/main/.github/workflows/ci.yml)

Basically, I just run "deno task test" and "deno task cov" defined in `deno.json` described before.
In addition, I'm using [CodeCov](https://about.codecov.io/) for visualizing the coverage of the test.

```yml
name: Deno CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - name: Run deno test
        run: |
          deno fmt --check
          deno task test
          deno task cov > coverage.lcov
      - uses: codecov/codecov-action@v2
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: ./coverage.lcov
          flags: unittests
```

### CI for Browsers

[hpke-js/.github/workflows/ci_browser.yml](https://github.com/dajiaji/hpke-js/blob/main/.github/workflows/ci_browser.yml)

I deploy test content in the `pages` job and run E2E test in the `playwright-test` job.

```yml
name: Browser CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  pages:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - run: |
          deno task dnt
          cp npm/esm/*.js test/pages/src/
          cp -rf npm/esm/src test/pages/src/
      - uses: peaceiris/actions-gh-pages@v3
        with:
          deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
          publish_dir: ./test/pages

  playwright-test:
    needs: pages
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - uses: microsoft/playwright-github-action@v1
      - working-directory: ./test/playwright
        run: npm install && npx playwright install && npx playwright test
```

### CI for Node.js

[hpke-js/.github/workflows/ci_node.yml](https://github.com/dajiaji/hpke-js/blob/main/.github/workflows/ci_node.yml)

I run `deno task dnt` and `deno task minify` on multiple versions of Node.js (`16.x`, `17.x`, `18.x`).

```yml
name: Node.js CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [16.x, 17.x, 18.x]

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - name: Run dnt & minify
        run: |
          npm install -g esbuild
          deno task dnt
          deno task minify > ./npm/hpke.min.js
```

In addition, considering the size limitation of Cloudflare Workers, we tried to minify the JS file by esbuild to make it as compact as possible, but it did not make much sense as a result, because, for example, esm.sh, one of the deploying destinations, creates a minified JS file. hpke-js example has a normal size of 12KB, a minified version by esbuild of 6KB, and an esm.sh version of 6.5KB.

### CI for Cloudflare Workers

[hpke-js/.github/workflows/ci_cfw.yml](https://github.com/dajiaji/hpke-js/blob/main/.github/workflows/ci_cfw.yml)

I run `wrangler dev --local=true` via `npm start` as a background task and then run `deno test`.

```yml
name: Cloudflare Workers CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - uses: actions/setup-node@v3
        with:
          node-version: v16.x
      - run: deno bundle mod.ts test/wrangler/src/hpke.js
      - name: Run test
        working-directory: ./test/wrangler
        run: |
          npm install
          nohup npm start &
          deno test hpke.spec.ts --allow-net
```

### Delivery

[hpke-js/.github/workflows/publish.yml](https://github.com/dajiaji/hpke-js/blob/main/.github/workflows/publish.yml)

Deployments to `npmjs.com` and `nest.land` are performed with this Github Actions.
Deployment to `deno.land` is done at the time of tag creation via the API of deno.land registered in WebHook (set at the time of module registration).

```yml
name: Publish

on:
  release:
    types: [created]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: v16.x
          registry-url: https://registry.npmjs.org/
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - name: Run eggs
        run: |
          deno install -A --unstable https://x.nest.land/eggs@0.3.4/eggs.ts
          eggs link ${{ secrets.NEST_API_KEY }}
          eggs publish --yes --version $(git describe --tags $(git rev-list --tags --max-count=1))
      - name: Run dnt & minify
        run: |
          npm install -g esbuild
          deno task dnt
          deno task minify > ./npm/hpke.min.js
      - working-directory: ./npm
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{secrets.NPM_TOKEN}}
```

## Remaining Problems
I have set up the CI/CD flows above, but I would like to add what I feel are some of the issues.

- `dependabot` integration is currently not possible.
  - I think this is the biggest disadvantage of using Deno (in my opinion), and I would like to let dependabot update the dependency packages in import-map.json.
- Tests at the time of transformation by `dnt` cannot be executed in parallel.
  - Unit tests in `hpke-js` take a long time to execute because of the huge number of test vectors in the standard, so.
- To begin with, the current situation where there are many major JavaScript runtimes.

## Conclusion
The current situation where there are many JS runtime is still hard. As mentioned in this article, the use of `dnt` and Github Actions can alleviate some of the difficulty, but I would still like to see more portability ensured within the framework of standardization.

We have high expectations for the [W3C Winter CG](https://www.w3.org/community/wintercg/).

Thank you for reading my poor English. See you!