How to Wrap Your Terraform Provider for Pulumi
Tags: terraform, api, openapi, go
Published: 2023-10-13 16:10:24

In this post, we'll show you how to make a Terraform provider available on Pulumi. The article assumes some prior knowledge of Terraform and Pulumi, as well as fluency in Go. If you are new to Terraform providers, please check out [our Terraform documentation](/docs/create-terraform/).

While we provide instructions for building a Pulumi provider here, the resultant provider is not intended for production use. If you want to maintain a Pulumi provider as part of your product offering, we recommend you get in touch with us. Without a partner, providers can become a significant ongoing cost.

## Why Users Are Switching From Terraform to Pulumi

Following the recent [HashiCorp license change](https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license), many users are exploring Pulumi as an alternative to Terraform.

The license change came as a surprise. In response, many companies are considering alternatives to manage their infrastructure-as-code setup. Given the scale of the Terraform ecosystem, it's unlikely that the license change will lead to Terraform disappearing. However, we can expect to see some fragmentation in the market.

If you're used to Terraform, you might notice that Pulumi has fewer providers available. Currently, Pulumi offers 125 providers in their registry, while Terraform boasts an incredible 3,511.

We know the comparison isn't completely fair, though – some users take the "everything-as-code" approach to the extreme, with providers for [ordering pizza](https://registry.terraform.io/providers/MNThomson/dominos/latest/docs), [building Factorio factories](https://github.com/efokschaner/terraform-provider-factorio), and [placing blocks in Minecraft](https://registry.terraform.io/providers/HashiCraft/minecraft/latest). But even accounting for hobbyist providers, it's clear that Terraform has significantly more third-party support.

So, let's look at how we can shrink that provider gap...

## How Pulumi Differs From Terraform

Pulumi and Terraform are both infrastructure-as-code tools, but they [differ in many ways](https://www.pulumi.com/docs/concepts/vs/terraform/), most importantly in the languages they support.

Terraform programs are defined in the declarative HashiCorp Configuration Language (HCL), while Pulumi allows users to create imperative programs using familiar programming languages like Python, Go, JavaScript, TypeScript, C#, and Java.

This difference has some benefits and drawbacks. With Pulumi's imperative approach, users have more control over how their infrastructure is defined and can write complex logic that isn't easily expressed in Terraform's declarative language. However, this also means that Pulumi code can be less readable than Terraform code.

## Bridging Terraform and Pulumi

Pulumi provides two tools to help maintainers build bridge providers. The first is [Pulumi Terraform Bridge](https://github.com/pulumi/pulumi-terraform-bridge), which creates Pulumi SDKs based on a Terraform provider schema. The second repository, [Terraform Bridge Provider Boilerplate](https://github.com/pulumi/pulumi-tf-provider-boilerplate), is a template for building a new Pulumi provider based on a Terraform provider.

While creating a new provider, we'll use the Terraform Bridge Provider Boilerplate, but we'll often call functions from the Pulumi Terraform Bridge.

Pulumi Terraform Bridge is actively maintained, so bear in mind that the requirements and steps below may change with time.

## How the Pulumi Terraform Bridge Works

Pulumi Terraform Bridge plays an important role during two distinct phases: design time and runtime.

During design time, Pulumi Terraform Bridge inspects a Terraform provider schema, then generates Pulumi SDKs in multiple languages.

At runtime, the bridge connects Pulumi to the underlying resource via the Terraform provider schema. This way, the Terraform provider schema continues to perform validation and calculates differences between the state in Pulumi and the resource state.

Pulumi Terraform Bridge does not use the Terraform provider binaries. Instead, it creates a Pulumi provider based only on a Terraform provider's Go modules and provider schema.

## Step by Step: Creating a Terraform Bridge Provider in Pulumi

The process of creating a Pulumi provider differs slightly depending on how the Terraform provider was created. In the past, most Terraform providers were based on [Terraform Plugin SDK](https://github.com/hashicorp/terraform-plugin-sdk). More recent providers are usually based on [Terraform Plugin Framework](https://github.com/hashicorp/terraform-plugin-framework).

The steps you'll follow depend on your Terraform provider. Inspect the `go.mod` file in your provider to see whether it depends on `github.com/hashicorp/terraform-plugin-sdk` or `github.com/hashicorp/terraform-plugin-framework`.

### Prerequisites

We manually installed the required tools before noticing that the Terraform Bridge Provider Boilerplate contains a Dockerfile to set up a development image. The manual process wasn't too painful, and either method should work.

The following must be available in your `$PATH`:

- [`pulumictl`](https://github.com/pulumi/pulumictl#installation)
- [Pulumi](https://www.pulumi.com/docs/install/)
- [Go 1.17](https://golang.org/dl/) or 1.latest
- [NodeJS](https://nodejs.org/en/) 14.x – we recommend using [nvm](https://github.com/nvm-sh/nvm) to manage Node.js installations
- [Yarn](https://yarnpkg.com/)
- [TypeScript](https://www.typescriptlang.org/)
- [Python](https://www.python.org/downloads/) (called as `python3`) – for recent versions of macOS, the system-installed version is fine
- [.NET](https://dotnet.microsoft.com/download)


### Terraform Plugin SDK to Pulumi

As an example of a Terraform provider based on Terraform Plugin SDK, we'll use this [Spotify Terraform provider](https://github.com/conradludgate/terraform-provider-spotify), a small provider for managing Spotify playlists with Terraform.

To create a new Pulumi provider, we'll start with the [Terraform Bridge Provider Boilerplate](https://github.com/pulumi/pulumi-tf-provider-boilerplate).

#### Clone the Terraform Bridge Provider Boilerplate

Go to the [Terraform Bridge Provider Boilerplate](https://github.com/pulumi/pulumi-tf-provider-boilerplate) repository in GitHub and click on the green **Use this template** button. Select **Create a new repository** from the dropdown.

Select your organization as the owner of the new repository (we'll use `speakeasy-api`) and create a name for your Pulumi provider. In Pulumi, it is conventional to use `pulumi-` followed by the resource name in lowercase as the provider name. We'll use `pulumi-spotify`.

Click **Create repository**.

Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.

In the terminal, replace `speakeasy-api` with your GitHub organization name in the code below and run it:

```bash
git clone git@github.com:speakeasy-api/pulumi-spotify.git
cd pulumi-spotify
```

#### Rename Pulumi-Specific Strings in the Boilerplate

The Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi. We need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.

The `Makefile` has a `prepare` command that handles some of the string replacement. In the terminal, replace `speakeasy-api` with your GitHub organization name in the command below and run it:

```bash
make prepare NAME=spotify REPOSITORY=github.com/speakeasy-api/pulumi-spotify
```

The `make` command will print the `sed` commands it ran to replace the boilerplate strings in two files in the repo.

Next, we'll use `sed` to replace strings in the rest of the repo:

In the terminal, replace `speakeasy-api` with your GitHub organization name, then run:

```bash
find . -not \( -name '.git' -prune \) -not -name 'Makefile' -type f -exec sed -i '' 's|github.com/pulumi/pulumi-spotify|github.com/speakeasy-api/pulumi-spotify|g' {} \;
```

In the `Makefile`, replace the `ORG` variable with the name of your GitHub organization.

Finally, in the `provider/resources.go` file, manually replace the values in the `tfbridge.ProviderInfo` struct. Many of these values define names and other fields in the resulting Pulumi SDK packages. Set the `GitHubOrg` to `conradludgate`.

#### Import Your Terraform Provider

Back in the `provider/resources.go` file, replace the `github.com/terraform-providers/terraform-provider-spotify/spotify` import with `github.com/conradludgate/terraform-provider-spotify/spotify`.

In a terminal run the following to change into the provider directory and install requirements.

```bash
cd provider
go mod tidy
```

#### Fix a Dependency Version

This temporary step is a workaround related to the version of `terraform-plugin-sdk` imported in the boilerplate.

During our testing, we encountered a bug in `terraform-plugin-sdk` that was fixed in `v2.0.0-20230710100801-03a71d0fca3d`.

In `provider/go.mod`, replace `replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20220824175045-450992f2f5b9` with `replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20230710100801-03a71d0fca3d`. Note the difference in the version strings.

#### Remove Outdated `make` Step

This temporary step removes a single line from the `Makefile` that copies a nonexistent `scripts` directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a `scripts` folder containing `install-pulumi-plugin.js`, but [Pulumi no longer generates these files](https://github.com/pulumi/pulumi/issues/13195#issuecomment-1703318022).

In the `Makefile`, remove the `cp -R scripts/ bin && \` line from `build_nodejs`:

```
makefile
build_nodejs:: VERSION := $(shell pulumictl get version --language javascript)
build_nodejs:: install_plugins tfgen # build the node sdk
	$(WORKING_DIR)/bin/$(TFGEN) nodejs --overlays provider/overlays/nodejs --out sdk/nodejs/
	cd sdk/nodejs/ && \
        yarn install && \
        yarn run tsc && \
		cp -R scripts/ bin && \ # remove this line
        cp ../../README.md ../../LICENSE package.json yarn.lock ./bin/ && \
		sed -i.bak -e "s/\$${VERSION}/$(VERSION)/g" ./bin/package.json
```

#### Remove the Nonexistent `prov.MustApplyAutoAliasing()` Function

In `provider/resources.go`, remove the line `prov.MustApplyAutoAliasing()` from the `Provider()` function.

#### Build the Generator

In the terminal, run:

```bash
make tfgen
```

Go will build the `pulumi-tfgen-spotify` binary. You can safely ignore any warnings about missing documentation. This can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide because the Pulumi boilerplate code does not include this step yet.

#### Build the Provider

In the terminal, run:

```bash
make provider
```

Go now builds the `pulumi-resource-spotify` binary and outputs the same warnings as before.

#### Build the SDKs

In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.

In the terminal, run:

```bash
make build_sdks
```

You can find the generated SDKs in the new `sdk` directory in your repository.

### Terraform Plugin Framework to Pulumi

As we mentioned earlier, more recent Terraform plugins are based on Terraform Plugin Framework instead of Terraform Plugin SDK. The way Terraform Plugin Framework structures Go code adds a few extra steps when bridging a plugin to Pulumi.

The difference is significant enough that Pulumi Terraform Bridge includes a dedicated tool called [Pulumi Bridge for Terraform Plugin Framework](https://github.com/pulumi/pulumi-terraform-bridge/blob/master/pf/README.md) in its main repository.

As with providers created using Terraform Plugin SDK, Pulumi Bridge for Terraform Plugin Framework needs to create a new Go binary that calls the Terraform plugin's new provider function. Plugins created with Terraform Plugin Framework define their new provider functions in an _internal_ package, which means we can't import the package directly.

To work around this, we'll create a shim that imports the internal package and exposes a function to our bridge.

But we're getting ahead of ourselves. Let's look at a step-by-step example.

#### Airbyte Terraform Provider

For this example, we'll bridge the [Airbyte Terraform provider](https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs) to Pulumi. While not the focus of this guide, it is worth mentioning that this provider was [entirely generated by Speakeasy](https://www.speakeasyapi.dev/post/case-study-airbyte).

#### Clone the Terraform Bridge Provider Boilerplate

Go to the [Terraform Bridge Provider Boilerplate](https://github.com/pulumi/pulumi-tf-provider-boilerplate) repository in GitHub and click on the green **Use this template** button. Select **Create a new repository** from the dropdown.

Select your organization as the owner of the new repository (we'll use `speakeasy-api` again) and create a name for your Pulumi provider. Let's use `pulumi-airbyte`.

Click **Create repository**.

Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.

In the terminal, replace `speakeasy-api` with your GitHub organization name in the code below and run it:

```bash
git clone git@github.com:speakeasy-api/pulumi-airbyte.git
cd pulumi-airbyte
```

#### Rename Pulumi-Specific Strings in the Boilerplate

You'll remember that the Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi, so we need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.

The `Makefile` has a `prepare` command that handles some of the string replacement. In the terminal, replace `speakeasy-api` with your GitHub organization name in the command below and run it:

```bash
make prepare NAME=airbyte REPOSITORY=github.com/speakeasy-api/pulumi-airbyte
```

The `make` command will print the `sed` commands it ran to replace the boilerplate strings in two files in the repo.

Next, we'll use `sed` to replace strings in the rest of the repo.

In the terminal, replace `speakeasy-api` with your GitHub organization name, then run:

```bash
find . -not \( -name '.git' -prune \) -not -name 'Makefile' -type f -exec sed -i '' 's|github.com/pulumi/pulumi-airbyte|github.com/speakeasy-api/pulumi-airbyte|g' {} \;
```

In the `Makefile`, replace the `ORG` variable with the name of your GitHub organization.

In the `provider/resources.go` file, manually replace the values in the `tfbridge.ProviderInfo` struct. Many of these values define names and other fields in the resulting Pulumi SDK packages.

Most importantly, in `provider/resources.go`, replace `fmt.Sprintf("github.com/pulumi/pulumi-%[1]s/sdk/", mainPkg),` with `fmt.Sprintf("github.com/speakeasy-api/pulumi-%[1]s/sdk/", mainPkg),`:

```go-diff
  ImportBasePath: filepath.Join(
-   fmt.Sprintf("github.com/pulumi/pulumi-%[1]s/sdk/", mainPkg),
+   fmt.Sprintf("github.com/speakeasy-api/pulumi-%[1]s/sdk/", mainPkg),
    tfbridge.GetModuleMajorVersion(version.Version),
```

Remember to replace `speakeasy-api` with your organization name.

#### Remove Outdated `make` Step

This temporary step removes a single line from the `Makefile` that copies a nonexistent `scripts` directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a `scripts` folder containing `install-pulumi-plugin.js`, but [Pulumi no longer generates these files](https://github.com/pulumi/pulumi/issues/13195#issuecomment-1703318022).

In the `Makefile`, remove the `cp -R scripts/ bin && \` line from `build_nodejs`:

```makefile
build_nodejs:: VERSION := $(shell pulumictl get version --language javascript)
build_nodejs:: install_plugins tfgen # build the node sdk
	$(WORKING_DIR)/bin/$(TFGEN) nodejs --overlays provider/overlays/nodejs --out sdk/nodejs/
	cd sdk/nodejs/ && \
        yarn install && \
        yarn run tsc && \
		cp -R scripts/ bin && \ # remove this line
        cp ../../README.md ../../LICENSE package.json yarn.lock ./bin/ && \
		sed -i.bak -e "s/\$${VERSION}/$(VERSION)/g" ./bin/package.json
```

#### Remove the Nonexistent ‘prov.MustApplyAutoAliasing()’ Function

In `provider/resources.go`, remove the line `prov.MustApplyAutoAliasing()` from the `Provider()` function.

Clone the `terraform-provider-airbyte` repository (replace the destination with your destination):

```bash
git clone git@github.com:airbytehq/terraform-provider-airbyte.git /Users/speakeasy/terraform-provider-airbyte
cd /Users/speakeasy/terraform-provider-airbyte
```

Replace the module name:

```bash
sed -i '' '1s|.*|module github.com/airbytehq/terraform-provider-airbyte|' go.mod
```

Replace the module name in imports:

```bash
find . -name '*.go' -print0 | xargs -0 sed -i '' 's|airbyte/internal|github.com/airbytehq/terraform-provider-airbyte/internal|g'
```

In the steps below, replace `/Users/speakeasy/terraform-provider-airbyte` with your local `terraform-provider-airbyte` repository.

#### Create a Shim To Import the Internal New Provider Function

Start with a new directory called `provider/shim` in your `pulumi-airbyte` project:

```bash
mkdir provider/shim
```

Add a `go.mod` file to this directory with the following contents:

```go
module github.com/airbytehq/terraform-provider-airbyte/shim

go 1.18

// TODO: This replacement is necessary only because airbytehq/terraform-provider-airbyte is not in https://pkg.go.dev/ and the module name is not an FQDN
// So to work around this we clone the terraform-provider-airbyte code and rewrite references to it to the local version.
replace github.com/airbytehq/terraform-provider-airbyte => /Users/speakeasy/terraform-provider-airbyte

require (
  github.com/airbytehq/terraform-provider-airbyte v0.0.0
  github.com/hashicorp/terraform-plugin-framework v1.3.5
)
```

Now we'll add `shim.go` to this directory:

```go
package shim

import (
	tfpf "github.com/hashicorp/terraform-plugin-framework/provider"
	"github.com/airbytehq/terraform-provider-airbyte/internal/provider"
)

func NewProvider() tfpf.Provider {
	return provider.New("dev")()
}
```

#### Add Shim Requirements

To have Go gather the requirements for our shim module, run the following from the root of the project:

```bash
cd provider/shim
go mod tidy
cd ../..
```

#### Import the New Shim Provider and the Terraform Package Framework Bridge

In `provider/resources.go`, edit your imports to look like this (replace `speakeasy-api` with your organization name):

```go
import (
	"fmt"
	"path/filepath"

	"github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge"
	"github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge/tokens"
	shim "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim"
	// Import the Pulumi Terraform Framework Bridge:
	pf "github.com/pulumi/pulumi-terraform-bridge/pf/tfbridge"
	"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
	"github.com/speakeasy-api/pulumi-airbyte/provider/pkg/version"
	// Import our shim:
	airbyteshim "github.com/airbytehq/terraform-provider-airbyte/shim"
)
```

#### Instantiate the Shimmed Provider

In `provider/resources.go`, replace `shimv2.NewProvider(airbyte.Provider())` with `pf.ShimProvider(airbyteshim.NewProvider())`:

```go-diff
func Provider() tfbridge.ProviderInfo {
    // Instantiate the Terraform provider
-   p := shimv2.NewProvider(airbyte.Provider())
+   p := pf.ShimProvider(airbyteshim.NewProvider())
```

#### Add the Shim Module as a Requirement

Edit `provider/go.mod`, and change the requirements to the following:

```go
require (
	github.com/airbytehq/terraform-provider-airbyte/shim v0.0.0
	github.com/pulumi/pulumi-terraform-bridge/pf v0.17.0
	github.com/pulumi/pulumi-terraform-bridge/v3 v3.61.0
)
```

Also in `provider/go.mod`, replace `github.com/airbytehq/terraform-provider-airbyte/shim` with `./shim` as shown below, to let the Go compiler look for the shim in our local repository:

```go
replace (
	github.com/airbytehq/terraform-provider-airbyte/shim => ./shim
  // TODO: The following replacement is only necessary because the terraform-provider-airbyte Go module is not named github.com/airbytehq/terraform-provider-airbyte and it does not appear in the Go package registry
  // This replaces references to github.com/airbytehq/terraform-provider-airbyte with a reference to our local terraform-provider-airbyte repository
  github.com/airbytehq/terraform-provider-airbyte => /Users/speakeasy/terraform-provider-airbyte
)
```

#### Install Go Requirements

From the root of the project, run:

```bash
cd provider
go mod tidy
cd ..
```

#### Build the Generator

In the terminal, run:

```bash
make tfgen
```

Go will build the `pulumi-tfgen-airbyte` binary. You can safely ignore any warnings about missing documentation. The missing documentation warnings can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide as the Pulumi boilerplate code does not include this step yet.

#### Build the Provider

In the terminal, run:

```bash
make provider
```

Go now builds the `pulumi-resource-airbyte` binary and outputs the same warnings as before.

#### Build the SDKs

In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.

In the terminal, run:

```bash
make build_sdks
```

You can find the generated SDKs in the new `sdk` directory in your repository.

## Summary

We hope this comparison of Terraform and Pulumi and our step-by-step guide to bridging a Terraform provider into Pulumi has been useful to you. You should now be able to create a Pulumi provider based on your Terraform provider.

Speakeasy can help you generate a Terraform provider based on your OpenAPI specifications. Follow [our documentation](https://speakeasyapi.dev/docs/create-terraform/) to enter this exciting ecosystem.

Speakeasy also has a Pulumi target in beta. [Join our Slack community](https://join.slack.com/t/speakeasy-dev/shared_invite/zt-1cwb3flxz-lS5SyZxAsF_3NOq5xc8Cjw) to discuss our upcoming Pulumi release or for expert advice on Terraform providers.