Extending Terraform functionality with a custom data source - part 1
Published: 2022-10-02 18:24:29

I want to talk about writing a Terraform provider, but instead of focusing on creating a _resource_, as in many articles on the Internet, I want to cover a _data source_.

There usually are two reasons to write a terraform provider:

- To wrap a remote API to manage remote infrastructure as code
- To create additional utility functions in Terraform

I will show you how to extend Terraform functionality and add an additional utility function using a custom provider.

This article will also give your a bare-bone setup to start with plugin development that you can use for your project. Sounds good? Let's dive into it.

## Configure your dev environment

To start, you need to set up the Go environment. Depending on the OS you are using, instructions could be different. Take a look [here](https://go.dev/doc/install) to find instructions for your system. Examples in this article are for OSX because that's what I use.

```
brew install go
```
You, of course, will need to install Terraform. If you haven't tried [tfenv](https://github.com/tfutils/tfenv), it is a straightforward way to manage different versions of Terraform on your local machine.

```
brew install tfenv
tfenv install 1.1.9
```
There are currently two standards for building Terraform plugins: the old one [SDKv2](https://www.terraform.io/plugin/sdkv2) and the new one [Plugin Framework](https://www.terraform.io/plugin/framework). Though Plugin Framework is the future, it was still under technical review when I wrote this, so we will look into SDKv2 as it is a more stable way of creating a plugin currently.

Hashicorp has a [scaffolding](https://github.com/hashicorp/terraform-provider-scaffolding) repo to help you start developing a plugin, and it makes sense to review it. But I would like us to start with an empty repo and add files as we go along. I think it will be easier to learn how to write a plugin this way.

First, we should choose the name for our provider and follow the pattern when the plugin name is always `terraform-provider-<name>`. Even though we don't connect to any external provider's API, we still call it a Terraform provider.

It's essential to pick a name in the beginning because later on, this name will be used all over the code, and it will be a hassle to change it. The name will be required as a prefix to any resource or data source you will define, so it's better if the name is concise and has no special symbols. I will call our provider `test`, and the project name is `terramform-provider-test`.

Let's create a new folder and fill it out with the primary files we will need for this project.

```
mkdir terraform-provider-test
cd terraform-provider-test
touch main.go provider.go main.tf
```

That's the list of files we will start with. Typically you would follow folder structure as in the [scaffolding repo,](https://github.com/hashicorp/terraform-provider-scaffolding), placing Go files inside the `internal/provider` folder and Terraform files inside the `examples` folder. But let us first understand how it all works together before imposing this structure.

We will look into the content of the files in the next section. But before we do that, there is one last thing to set up. Terraform has a particular configuration in your user's Home folder called `.terraformrc`. You probably don't have it if you never developed for Terraform. This file will help us create so-called developer overrides, which are needed to specify the path to the plugin object file which Terraform can load in.

```
touch ~/.terraformrc
```

Fill it out with the path to the project folder you have created.
```
provider_installation {
  dev_overrides {
    "test" = "/Users/~your~path~/terraform-provider-test/"
  }

  direct {}
}
```

## Plugin entry point

First thing, let's initialise Go dependency tracking.
```
go mod init terraform-provider-test
```
And now, we are ready to create an entry point for our plugin, starting with the `main.go` file.
```
package main

import (
	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
	"github.com/hashicorp/terraform-plugin-sdk/v2/plugin"
)

func main() {
	plugin.Serve(&plugin.ServeOpts{
		ProviderFunc: func() *schema.Provider {
			return Provider()
		},
	})
}
```
In this file, we start the plugin and initialise a Provider object. Let's create that object in `provider.go`.
```
package main

import (
	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func Provider() *schema.Provider {
	return &schema.Provider{
		Schema: nil,
		DataSourcesMap: nil,
		ProviderMetaSchema: map[string]*schema.Schema{},
		TerraformVersion: "",
	}
}
```
An empty Provider that doesn't do anything just to make the project build successfully. After that, install SDKv2 dependencies.
```
go get github.com/hashicorp/terraform-plugin-sdk/v2/plugin
go get github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema
```
And build the plugin.
```
go build -o terraform-provider-test
```
If you see a `terraform-provider-test` executable in your folder, congratulations, you built your first Terraform plugin. Now we can try using it with some Terraform code.

## Our Terraform code

We will now look at creating a simple data source that takes some information in and allows printing something out. It will be easier to write code when you know what kind of result you expect, so let's map out our idea in Terraform code before implementing it in Go. The following goes into `main.tf`.

```
terraform {
  required_providers {
    test = {
      version = "0"
      source  = "test"
    }
  }
}

data "test_msg" "my_msg" {
  message = "My output message"
}

```

I thought of doing something very simple for starters. As you can see from the code, the data source takes in some text information in the `message` field. It doesn't have to do anything with this message yet. We will look at how to load that plugin in and make `terraform plan` pass on that example.

When you develop a plugin, you don't have to initialise your repository code with `terraform init`, just use `plan`.

```
tfenv use 1.1.9
terraform plan
```
It should fail with something like this.
```
╷
│ Error: Invalid data source
│
│   on main.tf line 10, in data "test_msg" "my_msg":
│   10: data "test_msg" "my_msg" {
│
│ The provider hashicorp/test does not support the data source "test_msg".
```
This error indicates that Terraform could load our provider in memory and communicate with it, meaning all previous steps were correct. And we are finally ready to add the data source code.

## Define the input values schema

Terraform plugin SDK helps you define and validate schema for the input values for resources you create. When you make a resource, you start by defining its schema. Let's open a new file called `data_source_msg.go` and create our schema.

```
package main

import (
	"context"

	"github.com/hashicorp/terraform-plugin-log/tflog"
	"github.com/hashicorp/terraform-plugin-sdk/v2/diag"
	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func dataSourceMsg() *schema.Resource {
	return &schema.Resource{
		Description: "Output a custom message",

		ReadContext: dataSourceMsgRead,

		Schema: map[string]*schema.Schema{
			"message": {
				Type:     schema.TypeString,
				Required: true,
			},
		},
	}
}

func dataSourceMsgRead(ctx context.Context, d *schema.ResourceData, meta any) diag.Diagnostics {
	tflog.Info(ctx, "test-msg data source has been loaded")
	return nil
}
```
In this file, we define the data source schema, and the read function, which will be called by Terraform when it finds your data source in the infra configuration.

Now return to the `provider.go` file and add `DataSourcesMap` to point to the resource we have just defined.
```
DataSourcesMap: map[string]*schema.Resource{
  "test_msg":  dataSourceMsg(),
}
```
So our plugin exposes a new data source at this point, and we can test that everything works by building it and running `terraform plan`.

```
go build -o terraform-provider-test
TF_LOG=INFO terraform plan
```
[tflog](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-log/tflog) is the standard way to debug your provider code, and it comes in handy here to print a little message once the provider is loaded. We should be able to see the message in the output log.
```
2022-10-02T20:06:55.212+0200 [INFO]  provider.terraform-provider-test: configuring server automatic mTLS: timestamp=2022-10-02T20:06:55.211+0200
2022-10-02T20:06:55.250+0200 [INFO]  provider.terraform-provider-test: test-msg data source has been loaded: tf_provider_addr=provider tf_req_id=dd639fe4-d927-04fc-0d17-cf1c89fe8f9e @caller=/terraform-provider-test/data_source_msg.go:27 @module=provider tf_rpc=ReadDataSource tf_data_source_type=test_msg timestamp=2022-10-02T20:06:55.250+0200
2022-10-02T20:06:55.252+0200 [INFO]  backend/local: plan operation completed

No changes. Your infrastructure matches the configuration.
```
## Conclusion

If everything worked out well for you in this article, you should have set up the basis for Terraform plugin development. In the next part, I will show you how to add more functionality to your plugin.

## Reference

- https://www.terraform.io/plugin/sdkv2
- https://github.com/hashicorp/terraform-provider-scaffolding