Learning some terraform for Proxmox
Published: 2022-12-06 09:12:56

## Introduction


In an effort to capture some of the effort that goes into setting up tests infrastructures on proxmox, i decided to learn some terraform to formalize infrastructure deployment on proxmox as terraform configs.



## Personal objectives

- Build a terraform provider from source and use it from locally (this will be useful in case the need to further extend the provider with additional features that the proxmox API provides)
- Deploy n-lxc containers using terraform on proxmox

---


## The Provider

#### 1. Building the Provider

I choose [Telmate/Proxmox](https://github.com/Telmate/terraform-provider-proxmox) provider as it one of the most popular on hashicorp.

This provider has a quiet comprehensive friendly structure, with an entrypoint `main.go` and `package proxmox` dedicated to proxmox's specific logic.


![peek at the code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pzmqhbh52g454w9b9wxt.png)

And surprisingly easy to install. After installing the `go compiler` and `build-essentials` package on ubuntu, `make` does the trick.


![building terraform from source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ug63etoiyts9idzb43hn.png)


The resulting provider binary `terraform-provider-proxmox` is stored in `./bin` directory.


![terraform-provider-proxmox binary built in bin directory](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/05fspgtfhp3y8tatgfn3.png)




#### 2. Where does terraform search/find provider binaries

By default terraform will directly fetch providers from Hashicorp and cache them in `~/.terraform/plugins`.

What we can do if we want to run a custom built provider is to put it in a directory structure similar to the one below.


![directory strucuture for terraform plugins on system](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sokh9p7ntkwu6wllcm9j.png)


We can now invoke this provider in our terraform files as follows.


```tf
# lxc-example.tf
terraform {
  required_providers {
    proxmox = {
      source  = "terraform.local/telmate/proxmox"
    }
  }
}

provider "proxmox" {
    pm_tls_insecure = true
    pm_api_url = "https://localhost:8006/api2/json"
    pm_password = "donthackmeplease"
    pm_user = "root@hostname"
}
```

---


##  The containers

#### 1. A minimal lxc container

At this point, to get a little confidence in my understanding of this provider i wanted to create a simple terraform file to spin for me an lxc container in a designated pool on proxmox.

So  this exercise consisted of creating an lxc container:

- within a choosen Pool (pools are kind of resource groupings in proxmox with common administrative policies, this pool was created because proxmox allows for visually grouping resources into pools, hence primarily for aesthetic purposes)
- with 1024MB of RAM
- 2vCPU
- is connected to the internet over virtual switch `vmbr0`
- starts immediately after creation.
- is based on lxc ubuntu 22.04 template


To achieve this we can read  the example terraform file `terraform-provider-proxmox/examples/lxc_example.tf` to spin an lxc container provided by the code base. However, we could get more insight about the terraform directives which will help us achieve our requirements by reading directly the file `terraform-provider-proxmox/proxmox/resource_lxc.go` as it is an entrypoint into the Schema definition of an lxc resource from the point of view of terraform (it basically maps the way terraform understands what an lxc container is to what proxmox understands it to be).

Similarly  the file `terraform-provider-proxmox/proxmox/resource_pool.go`  provides the Schema definition of  a `Pool`.

- Create Pool `TestPool1`

```tf
# lxc-example.tf
resource "proxmox_pool" "pool-test" {
  poolid = "TestPool1"
  comment = "Just a test pool"
}
```

- The lxc container

```tf
resource "proxmox_lxc" "lxc-test" {
  pool = proxmox_pool.pool-test.poolid # depends on pool-test
  target_node = "pve" # node of your cluster on which deployment should be done
  cores = 2
  memory = 1024
  hostname = "bot"
  password = "rootroot"
  network  {
    name = "eth0"
    bridge = "vmbr0"
    ip = "dhcp"
  }
  start = true # start after creation

  # using ubuntu container template
  ostemplate = "local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst"
  rootfs {
    storage = "local-lvm"
    size = "8G"
  }
}
```

- `ostemplate` points to the volume on which is found the lxc template of your container, and follows some kind of promox's address notation.


![ostemplate is special location to lxc template](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d6lcovexdf8qvq82byv3.png)

This special notation ultimately resolves to local filesystem paths, as shown below for the address storage notation for our `ubuntu-22.04-standard_22.04-1_amd64.tar.zst` template.


![ostemplate resolves to location in /var/lib/vz/template/cache of system](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gzih4b8tk0adqeyyhvkj.png)

- `rootfs` specifies the `storage` (where the vm disk would be stored) and its `size`


![rootfs is local-lvm in my case](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ydpmz8uhctrez6uxgw8l.png)


**Testing:**


```sh
$ terraform init
$ terraform plan  # just to check
$ terraform apply
```


![Testing the Deployment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/32zehcwnea9d4tjd2kty.png)



![Results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g7inb7tcfmd8jmrledfu.png)

I developed an overly simple mental model through studying this code base, which might be useful in case you find yourself in need to extend/add support for some proxmox feature to this code base


![Mental Model for Telmate/proxmox](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpojn0blc1zrj5lke1mz.png)

- `terraform cli` depends on `Telmate/terraform-provider-proxmox` for Schema definitions.
- `Telmate/terraform-provider-proxmox` binds the Schema definitions to promox JSON API using `Telmate/proxmox-api-go`
- `Telmate/proxmox-api-go` ultimately interacts with the proxmox server via its API.

 The uniform naming conventions used throughout the Telmate codebase makes so that a Schema directive's name is exactly the same name as in the Proxmox Core API.
 So we an directly refer ourselves to the proxmox API docs to get description / information about the role/purpose of a Schema directive.


#### 2. Spinning 20 lxc containers

We spin `n=20` containers, we can use terraform's `count` meta-argument.

```tf
resource "proxmox_pool" "pool-test" {
  poolid = "TestPool1"
  comment = "Just a test pool"
}

resource "proxmox_lxc" "lxc-test" {
  count = 20 # create 20 lxc containers
  pool = proxmox_pool.pool-test.poolid # depends on pool-test
  target_node = "pve" # node of your cluster on which deployment should be done
  cores = 1
  memory = 512
  hostname = "bot-${count.index}"
  password = "rootroot"

  network  {
    name = "eth0"
    bridge = "vmbr0"
    ip = "dhcp"
  }
  start = true # start after creation

  # using ubuntu container template
  ostemplate = "local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst"
  rootfs {
    storage = "local-lvm"
    size = "8G"
  }
}
```

After running `terraform apply` we get


![Terraform apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4w5cndgj124t1k3qn6nm.png)

We have them, 20 containers created in pool `TestPool1`

![20 lxc containers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vthovh5q76z2zlagko9z.png)

Just as easily as we created them, we can destroy them back using `terraform destroy`


![Terraform destroy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/urjlcyvzumt1jk74ugno.png)


## References

- https://pve.proxmox.com/pve-docs/api-viewer/#/nodes/{node}/lxc
- https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_volumes
- https://developer.hashicorp.com/terraform/language/meta-arguments/depends_on (resource dependency)
- https://developer.hashicorp.com/terraform/language/meta-arguments/count