Control Your Custom Cloud Resources with Pulumi
Tags: iac, cloud, typescript, tutorial
Published: 2021-07-31 16:53:52

---
title: Control Your Custom Cloud Resources with Pulumi
published: true
description: A dynamic resource tutorial for Pulumi, we use Upstash as an example.
tags: iac, cloud, typescript, tutorial
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rsah3kvef4qjmiw7dyso.jpg
---

When I began backend development, I mainly worked with AWS, but recently I started looking into other cloud providers. Not the big ones like Azure or Google, but Auth0, Fauna, and Upstash.

While they don't offer as many services like AWS, they often bring attractive alternatives and a better developer experience. The problem is, not all of them are supported by IaC tools.

I'm using Pulumi to deploy and provision my cloud resources because it's provider-independent and allows me to use TypeScript, and in turn, JavaScript to define my infrastructure.

In this article, I will show you how to define your classes, so you can deploy whatever cloud resource you need. I will use Upstash as an example because deploying a serverless Redis database is probably as simple as it gets and still has practical relevance.

> Note: The content here is based on a [video by Cloudspeak](https://www.youtube.com/watch?v=H4nehfvCLm8). The main difference is that I used Upstash, a real cloud database as example. Also, I include the `diff` method and the video talks about the `check` and `read` methods, which I didn't cover here. Check it out!.

## Prerequisites

You need an [Pulumi](https://app.pulumi.com/signin) and [Upstash](https://console.upstash.com/login) account. Also, the [Pulumi CLI](https://www.pulumi.com/docs/get-started/install/) and [Node.js](https://nodejs.org/en/) installed.

## The Goal

We want to write the following code, and the `pulumi up` command should lead to a serverless Redis database deployed with Upstash.

```typescript
import * as upstash from "./upstash"

const redis = new upstash.Database("my-database", {
  region: "eu-west-1",
})

export const dbName = redis.name
export const dbEndpoint = redis.endpoint
export const dbPort = redis.port
export const dbPassword = redis.password
```

## What We Need

Pulumi allows the implementation of custom resources in the Go programming language; these will then be compiled so that they're accessible from other languages, like TypeScript.

But it also has the concept of a dynamic resource, a custom resource implemented in TypeScript and only available in TypeScript. 

A custom resource built with a dynamic Pulumi resource consists of two classes: The dynamic resource and a dynamic resource provider.

In the following code, I defined these two classes for our Redis database.

```typescript
class Database extends pulumi.dynamic.Resource {
  ...
}

class DatabaseProvider implements pulumi.dynamic.ResourceProvider {
  ...
}
```

The `Database` uses the `DatabaseProvider`, and as we saw in the goal code example, the users only have to access the `Database` in their code to create a new instance of that resource.

The `Database` is used to define the inputs and outputs of our resource. So it doesn't know anything about the underlying service. 

The `DatabaseProvider` handles all the interaction with the cloud service API to create, update, and delete the resource.

These two classes also run in different processes, so the only way to communicate between them are the methods the `pulumi.dynamic.ResourceProvider` interface defines.

## Implementing the `Database` Class

Upstash provides a [developer API](https://developer.upstash.com/#databases), that allows managing databases programmatically. This API takes five inputs. 

A `name`, that is a `string`.

A `region`, which currently only supports `"eu-west-1"`, and `"us-east-1"`. 

A `type`, which can be `"free"` or `"standard"`.

And an email and API key that needs to be supplied via HTTP basic authentication.

Our `Database` class will take these as inputs via its resource `name` and `props` in the constructor arguments. Since I'm using TypeScript here, I also tried to type the inputs correctly so we don't use invalid values.

The `DatabaseInputs` interface wraps all types in a `pulumi.Input`. This allows the `Database` class to accept the basic types directly or outputs from other resources. The provider will get the unwrapped outputs later.

The `public` attributes of the class are the outputs we can use later. In this case, we will use the inputs as outputs, but I also added some of the values that the API responds with after creation and update to connect to the database later.

```typescript
type Region = "eu-west-1" | "us-east-1"
type Type = "free" | "standard"
type State = "active" | "deleted"

interface DatabaseInputs {
  name?: pulumi.Input<string>
  region: pulumi.Input<Region>
  type?: pulumi.Input<Type>
}

export class Database extends pulumi.dynamic.Resource {

  // Outputs from our inputs
  public readonly name!: pulumi.Output<string>
  public readonly region!: pulumi.Output<Region>
  public readonly type!: pulumi.Output<Type>

  // Outputs from API response
  public readonly port!: pulumi.Output<number>
  public readonly state!: pulumi.Output<State>
  public readonly password!: pulumi.Output<string>
  public readonly endpoint!: pulumi.Output<string>

  constructor(name: string, props: DatabaseInputs, opts?: any) {
    const config = new pulumi.Config("upstash")

    if (props.name === undefined) props.name = name
    if (props.type === undefined) props.type = "free"

    super(
      new DatabaseProvider(),
      name,
      {
        ...props,

        port: null,
        state: null,
        password: null,
        endpoint: null,

        upstashEmail: config.require("email"),
        upstashApiKey: config.require("apiKey"),
      },
      opts
    )
  }
}
```

The constructor takes three arguments.

A `name`, which is the resource name Pulumi will use. In this case, I re-used it for the database name in the `props`, because the Upstash API doesn't allow to change it right now. If we rename the resource, Pulumi will attempt to delete the database with the old name and create a  database with the new name. So, that would be following the Upstash API anyway.

A `props` argument, which defines the configuration of our database. Currently, only three are available: `name`, `region`, `type`.

> Note: You can only have one free database.

A `opts` argument, which defines Pulumi specific behavior for the resource. Like, protect it that it can't be accidentally deleted or with some parent resource. In this tutorial, that's not important, so I typed it `any`.

In the constructor, we gather the config values from the `upstash` namespace, so we don't have to add the email and API key as `props`, when creating the resource.

The calls to `config.require` expect a `upstash:email` and `upstash:apiKey` to be present in the `Pulumi.dev.yaml` file.

The email you used to create your Upstash account and the API key can be created in [the Upstash console](https://console.upstash.com/pages/account/apikeys).

The values can be set with these CLI commands:

    $ pulumi config set upstash:email <YOUR_EMAIL>
    $ pulumi config set upstash:apiKey <YOUR_KEY> --secret

The constructor has to call `super` with an instance of the `DatabaseProvider`, the resource `name`, the `props`, and the `opts`.

The `props` also have to include null-ish values for all the outputs we will later get from the server and the values we got from the config. They will be available as arguments for the methods of the `DatabaseProvider` later. 

It's crucial to remember the order of fields in the third object passed to `super`. These will be serialized and given to the provider class in another process. If the fields have a different order later, this will lead to an error where the resource is deployed, but Pulumi thinks the deployment failed. 

> Note: The two classes run in different processes, so the `props` are the only way to give the provider data. Calls to `pulumi.Config` aren't possible in the provider either.

The `Database` class knows nothing about Upstash, it's only responsible for telling Pulumi there is a new resource, and it has specific inputs and outputs. This way, we can interact with our resources as we would with all the other resources provided by Pulumi and seen in the goal section initially.

## Implementing the `DatabaseProvider` Class

The next step is to implement the `DatabaseProvider`, which will do the actual work.

The `DatabaseProvider` has to implement the `pulumi.dynamic.ResourceProvider` interface, which only requires a `create` method to be implemented.

For our database, we will need four methods: `create`, `update`, `delete`, and `diff`.

The first time we run `pulumi up` with our `Database` resource, the `create` method will be called.

After that, the `diff` method will be called to check if anything of importance changed and if it can be applied to our resource via an `update` call or if it requires us to replace the resource, which essentially means a call to `create` and `delete` or `delete` and `create`.

Let's go over every method in detail, starting with `create`. I also provided the types again, so we know what is what.

> Note: The types aren't wrapped in `pulumi.Input` anymore.

```typescript
type Region = "eu-west-1" | "us-east-1"
type Type = "free" | "standard"
type State = "active" | "deleted"

interface DatabaseProviderInputs {
  name: string
  region: Region
  type: Type

  port: number
  state: State
  password: string
  endpoint: string

  upstashEmail: string
  upstashApiKey: string
}

...

async create(inputs: DatabaseProviderInputs) {
    const { data } = await axios.post(
      "https://api.upstash.com/v1/database",
      { database_name: inputs.name, region: inputs.region, type: inputs.type },
      {
        auth: {
          username: inputs.upstashEmail,
          password: inputs.upstashApiKey,
        },
      }
    )

    return {
      id: data.database_id,
      outs: {
        ...inputs,

        port: data.port,
        state: data.state,
        password: data.password,
        endpoint: data.endpoint,

        upstashEmail: inputs.upstashEmail,
        upstashApiKey: inputs.upstashApiKey,
      },
    }
  }

  ...
```

The method is asynchronous, which means it returns a promise. This allows us to access the network or the disk to deploy our database resource.

In this case, we need to call the Upstash developer API via HTTP. I'm using the [axios](https://www.npmjs.com/package/axios) package to make things a bit more convenient.

The `inputs` argument holds all data we need to call the API. It's a POST request with the name, region, and type in the body and the API credentials supplied via HTTP basic authentication.

If everything went well, we could use the `data` attribute of our result to return the new state of our database resource to Pulumi.

The `database_id` becomes our Pulumi ID, and the rest will be put in the `outs` object. This object needs the same field order as the object in the `super` call before. Otherwise, we get an error even if the deployment worked.

The axios client will throw an error when it didn't receive a 200 HTTP status. This will automatically be picked up by Pulumi and displayed to the user.

Next in line is the `delete` method. Which will be called when our database has to go.

```typescript
  async delete(id: string, inputs: DatabaseProviderInputs) {
    await axios.delete(`https://api.upstash.com/v1/database/${id}`, {
      auth: {
        username: inputs.upstashEmail,
        password: inputs.upstashApiKey,
      },
    })
  }
```

We use the first argument with a DELETE request, but depending on the API you use, it could be that you need other values. That's why the method also has access to the `inputs` again.

The `diff` method is used for updates.

There are two types of updates—regular updates and replacements.

A regular update will use the existing database and change its configuration.

A replacement will delete the database and redeploy it with a new configuration. We can decide if we delete the old database first and then create a new one or if we want to make the new one first and then delete the old one. In our Upstash use-case is essential to delete first because we can only have one free database.

If we didn't implement the `update` method, a replacement is the only available option. But let's not get ahead of us and look at the `diff` method first.

```typescript
  async diff(
    id: string,
    olds: DatabaseProviderInputs,
    news: DatabaseProviderInputs
  ) {
    const diffResult: any = { changes: false }

    const replaces = []
    if (olds.name !== news.name) replaces.push("name")
    if (olds.region !== news.region) replaces.push("region")
    if (replaces.length > 0) diffResult.replaces = replaces

    if (
      replaces.length > 0 ||
      olds.upstashEmail !== news.upstashEmail ||
      olds.upstashApiKey !== news.upstashApiKey
    ) {
      diffResult.changes = true
      diffResult.deleteBeforeReplace = true
    }

    return diffResult
  }
```

The method has to return a `diffResult` which contains a `changes` boolean field that tells Pulumi something has changed. 

The `diffResult` can also contain a `replaces` string array field. If this field is present, Pulumi will go for a replacement. If `replaces` is missing, Pulumi will try a regular update.

The `replaces` field is a list of all inputs that were responsible for the replacement. In the Upstash case, only the type can be upgraded from `"free"` to `"standard"`. A `name` and `region` change requires a replacement.

Pulumi will inform the user about the inputs that caused the replacement before the change will be executed, so they don't delete their database by accident.

The `deleteBeforeReplace` field will change the order of creating and delete, so the database is gone before creating a new one.

The last method is `update`, which will try to change the database configuration in place. 

```typescript
  async update(
    id: string,
    olds: DatabaseProviderInputs,
    news: DatabaseProviderInputs
  ) {
    if (news.type !== olds.type) {
      await axios.post(
        `https://api.upstash.com/v1/database/${id}/upgrade`,
        { type: news.type },
        {
          auth: {
            username: news.upstashEmail,
            password: news.upstashApiKey,
          },
        }
      )
    }
    return {
      outs: {
        ...news,

        port: olds.port,
        state: olds.state,
        password: olds.password,
        endpoint: olds.endpoint,

        upstashEmail: news.upstashEmail,
        upstashApiKey: news.upstashApiKey,
      },
    }
  }
```

The method gets the old inputs and the new inputs, and the Pulumi ID of the resource. There is nothing special happening here, just an API call to the `upgrade` endpoint if the `type` changed.

Again, the `outs` need to be in the correct order; otherwise: fails that lead to inconsistencies.

## Conclusion

Dynamic resources are tricky in terms of input field ordering and wrapping types correctly, but overall they are pretty flexible.

If you can define a resource or a task that needs to be done at deployment in terms of `create`, `update`, `delete`, and `diff` it can be implemented as a dynamic resource.

I think TypeScript makes things incredibly convenient. We can define many of the inputs as static types that will increase developer experience when using the resource and notify us that things are wrong before we even start deploying.