Creating virtual machines with LXD
Tags: tutorial, virtualmachine, lxd, docker
Published: 2023-07-29 22:58:14

## Intro

My original goal was to show you [how you can create your home lab](https://dev.to/rimelek/the-first-ansible-playbook-579h), but before we can start to automate everything using Ansible, we also have to learn how we can do it manually. In this post I will show you how you can create virtual machines with LXD in a way that really helps you to do it quickly even without Ansible.

## Table of contents

- [Why LXD?](#why-lxd)
- [Can you also say anything bad about LXD?](#can-you-also-say-anything-bad-about-lxd)
- [Install LXD](#install-lxd)
- [Preparation before initialization](#preparation-before-initialization)
- [Initializing / Configuring LXD](#initializing-configuring-lxd)
- [Non-interactive initialization](#noninteractive-initialization)
  - [Remove LXD](#remove-lxd)
  - [Use the previously saved config to initialize LXD](#use-the-previously-saved-config-to-initialize-lxd)
- [Let's see what we have now](#lets-see-what-we-have-now)
- [First virtual machine](#first-virtual-machine)
- [Overcommitting and how to avoid that](#overcommitting-and-how-to-avoid-that)
- [Use profiles to install similar virtual machines easily](#use-profiles-to-install-similar-virtual-machines-easily)
- [Checking the installed virtual machines](#checking-the-installed-virtual-machines)
- [Conclusion](#conclusion)

## Why LXD?

[» Back to table of contents «](#table-of-contents)

Even though I had some issues with LXD recently, it is still a great tool with a similar interface to Docker. When I say "similar", don't expect "the same"! LXD is not Docker but some concepts are the same. Docker is for creating light-weight containers with the possibility to add a simple virtual machine layer for better isolation (See [Kata container runtime](https://katacontainers.io/)) and LXD is for running full-fledged Linux distributions inside containers, and it can also run KVM-based virtual machines using qemu.

You can also use cloud-init to automatically add users and SSH keys to all of your virtual machines, and of course cloud-init can also be used to install some packages when starting the virtual machine the first time. Many modern solutions either support LXD or are even based on it.

There are long-term supported versions (LTS) and their end of life date is basically the same as the end of life of the LTS Ubuntu versions.

## Can you also say anything bad about LXD?

[» Back to table of contents «](#table-of-contents)

Well, some of you might not like [snap packages](https://snapcraft.io/) and indeed I never recommend using snap to install Docker, but Docker is not developed by Canonical (the developers of Snap) and LXD is. Unless you want to build LXD from source, you need to use the snap package on Ubuntu, but some distributions provide other ways as well.

I was still not able to move virtual machines between LXD servers while the virtual machines were running and I can't say that I was completely satisfied by the error messages while I was working on a solution. If you want to know more about my issue, you can read about that on the Ubuntu forum:

{% embed https://discourse.ubuntu.com/t/moving-running-lxd-virtual-machines-from-one-server-to-another/37009 %}

If that didn't scare you away you can still use it to easily create virtual machines on one server or even move them to other servers after you stopped them, and you can use the same interface to create containers.

## Install LXD

[» Back to table of contents «](#table-of-contents)

First you will need a Linux operating system. I will use an Ubuntu 22.04 server that supports Snap. You can check if your operating system supports it: [Install the daemon](https://documentation.ubuntu.com/lxd/en/latest/installing/#other-installation-options)

You can also check the other installation methods provided by some distributions: [Other installation options](https://documentation.ubuntu.com/lxd/en/latest/installing/#other-installation-options)

Let's see the Snap way:

```bash
sudo snap install lxd --channel 5.15/stable
```

The above command will install the currently latest stable version, however that is not the LTS version with Long-term support, so in a production system you would need to upgrade or reinstall it every month. That is not recommended.

To find out which is the LTS version before Canonical decided to move LXD out from the LinuxContainers project, you could check the [releases category on linuxcontainers.org](https://discuss.linuxcontainers.org/tag/release). However, since the last LTS versions were 3.0, 4.0 and 5.0, it is safe to say that the next LTS release will be 6.0.

To check the list of available channels and versions, you can use the following command:

```bash
snap info lxd
```

## Preparation before initialization

[» Back to table of contents «](#table-of-contents)

When you install LXD, it is just the Snap package without any configuration. Before you can start to run containers and virtual machines you need to configure some LXD networks, storages and profiles default parameters, so you won't have to configure every instance from scratch. Instead, you can just assign profiles to instances.

Even before that, we need to decide how we are going to do that. One of the first things you need to decide is what [storage driver](https://documentation.ubuntu.com/lxd/en/latest/reference/storage_drivers/#storage-drivers) you want to use. ZFS is one of the recommended drivers and that is also the default. If you skip preparation entirely and just press enter at every step of the initialization you will have a ZFS storage pool, except that would use a virtual disk image file which is good for testing, but to have the best performance, you should use a dedicated physical disk or partition. I have two physical disks. An HDD at `/dev/sda` and an SSD at `/dev/nvme0n1`. I used the SSD for Windows and the HDD for Linux. Well, that's not a recommendation, just a fact. If you plan to use your Linux more frequently and for tasks that require faster disks, choose SSD. In the following output you will see how I partitioned my HDD which also shows how badly I designed it as I have only 15GiB for my root partition which is not enough when I install many snap packages which I need to have on my host and not in virtual machines. I can handle it, but it requires extra work.

The following output is a part of the output of `lsblk`

```text
sda           8:0    0 465.8G  0 disk
├─sda1        8:1    0     1M  0 part
├─sda2        8:2    0    15G  0 part /
├─sda3        8:3    0     2G  0 part /boot
├─sda4        8:4    0    30G  0 part /home
├─sda5        8:5    0     3G  0 part /var/log
├─sda6        8:6    0   300G  0 part
└─sda7        8:7    0 115.8G  0 part /mnt/data
```

`sda6` is not mounted anywhere because that is what I use for the ZFS storage pool. If you have multiple disks or partitions you can also add more than one to the same ZFS storage pool, but as far as I know, LXD can initialize it with only one. Although I have only one, I will use a method that can be used with multiple disks as well. We need to create a ZFS storage pool even before initializing LXD. Because paths like `/dev/sda6` can change at every boot, it is better to use a persistent name when we add the disk to the storage pool. If the name of the partition is `sda6` as mine, you can use the following command to find persistent aliases for it:

```bash
disk=/dev/sda6
find -L /dev/disk/ -samefile "$disk"
```

In my case the output is this:

```text
/dev/disk/by-label/local
/dev/disk/by-uuid/2675692044005731665
/dev/disk/by-partuuid/af18d4b7-4bf0-44d9-92df-7321b1565ce5
/dev/disk/by-path/pci-0000:00:17.0-ata-5.0-part6
/dev/disk/by-path/pci-0000:00:17.0-ata-5-part6
/dev/disk/by-id/scsi-SATA_Samsung_SSD_850_S2RBNX0J103301N-part6
/dev/disk/by-id/scsi-0ATA_Samsung_SSD_850_S2RBNX0J103301N-part6
/dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
/dev/disk/by-id/wwn-0x5002538d41a17ca1-part6
/dev/disk/by-id/scsi-35002538d41a17ca1-part6
```

Labeling disks could be very convenient, but it can also be dangerous if you choose a bad label. Like the one you can see in the first line which is so general you won't know what that means. I assume it was automatically created by LXD because the LXD storage pool (not ZFS pool) is called "local" when we install an LXD cluster not an individual LXD server. If I created it, I should be ashamed...

I will use `/dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6`

Before we can start working with ZFS without LXD, we need to install `zfsutils-linux` (at least this is how it is called on Ubuntu 22.04)

```bash
sudo apt install zfsutils-linux
```

It will install Systemd services and some management commands like `zfs` and `zpool`. Let's define the name of the ZFS pool and the list of the disks that you want to add:

```bash
name="lxd-default"
disks=(
  /dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
)
```

If you want to add more disks, you can just add more lines in the bash array. You can also choose a different name. LXD would create it as "default", but I like to see what kind of default that pool is, so I use the `lxd-` prefix. The next step is creating the pool:

```bash
sudo zpool create "$name" "${disks[@]}"
```

Check if you have done it correctly:

```bash
zpool list
```

You should see something like this:

```text
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
lxd-default   298G   106K   298G        -         -     0%     0%  1.00x    ONLINE  -
```

ZFS works with datasets. Before installing LXD, there will be only one:

```bash
zfs list
```

Output:

```text
NAME          USED  AVAIL     REFER  MOUNTPOINT
lxd-default   106K   289G       24K  /lxd-default
```

That's it. ZFS can do a lot of things and work as a software RAID, which would require a lot more configuration. For now that simple pool will be enough for us.

## Initializing / configuring LXD

[» Back to table of contents «](#table-of-contents)

You can initialize LXD interactively or non-interactively. In this post we will discuss mainly the interactive way. At the moment you can also find a short description about this in my tutorial called Learn Docker. I wrote that part to be able to compare LXD and Docker: [Learn Docker: LXD](https://learn-docker.it-sziget.hu/en/latest/pages/intro/lxd.html). There is a pretty good description about the initialization steps [in the documentation as well.](https://documentation.ubuntu.com/lxd/en/latest/howto/initialize/) which covers the latest version that can change in the future. To start the initialization for version 5.15, run the following command:

```bash
sudo lxd init
```

First question:

> Would you like to use LXD clustering? (yes/no) [default=no]:

In this case we don't want an LXD cluster, so just press enter to get the next question:

> Do you want to configure a new storage pool? (yes/no) [default=yes]:

Well, in this case yes, we want. Note that this is not a ZFS pool, but a storage pool which will use a ZFS pool. The default answer is "yes" so just press enter again.

> Name of the new storage pool [default=default]:

The default name will be fine, but you can also change it to any name you like. I will leave it as it is and press enter.

> Name of the storage backend to use (zfs, btrfs, ceph, dir, lvm) [default=zfs]:

Again, the default storage driver is what I want so let's press enter again.

> Create a new ZFS pool? (yes/no) [default=yes]:

Since we have already created a ZFS pool, it's time to type "no" (without the quotation marks) and press enter.

> Name of the existing ZFS pool or dataset:

There is no default value. Since we named our pool as "lxd-default" let's type that and press enter.

> Would you like to connect to a MAAS server? (yes/no) [default=no]:

MAAS is something I want to write about in the future, but now we don't have any, so press enter once again.

> Would you like to create a new local network bridge? (yes/no) [default=yes]:

A local network bridge will be available only inside your local machine similar to how Docker networks work. You could also use LAN network or even VLANs, but that is something you should try after you have learnt to work with LXD locally.

> What should the new bridge be called? [default=lxdbr0]:

Again, this is similar to the default Docker bridge which is "docker0". The default value, "lxdbr0" will be perfect so press enter.

> What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:

LXD will try to find unused subnets in your network to configure for the local bridge. Although using antivirus is less frequent on Linux workstations in case you are installing LXD on your Laptop, I found out that ESET antivirus blocks this process and the initialization fails. I can't give you a solution here, so if you know one, please share in the comment section. You could answer "no" and continue with the initialization before configuring the network manually, but there is no guarantee that the `lxc` commands (yes, `lxc`) will work properly either.

Let's assume you don't have antivirus, or it does not interfere with LXD and press enter without changing the default choice.

> What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:

IPv6 is optional and I usually disable it since I rarely need it locally and disabling it also helps me not to worry about the difficulties of IPv6 especially when I am learning about something else. If you want to disable it as I do, type "none" and press enter.

> Would you like the LXD server to be available over the network? (yes/no) [default=no]:

The default value is "no" which is fine, since you usually want to log in to the server, or you are already logged in to your laptop, and you can use the local socket. It also seems more stable so if you don't want to manage your LXD server remotely, just press enter and [enable it anytime you need it later](https://documentation.ubuntu.com/lxd/en/latest/howto/server_expose/).

> Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:

LXD can automatically update already downloaded images. This is something I always change to "no", but you are free to choose "yes". I disable it because I had bad experience with automatically updated images before, although it was not with LXD but LibVirt. Nevertheless, I don't like to be surprised by new bugs when I repeat the installation process frequently in my home lab, so I choose "no" even though I know if I don't update my images, it could be a security risk, but I can update the images manually before creating a new virtual machine and I can also upgrade the packages after starting the VM. If you want to follow my way, type "no" and press enter.

> Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

This is the last step. We already configured everything we could this way, and now we can type "yes" and press enter to see the generated configuration which we can save and reload later when we want to reinstall LXD non-interactively.

My output was this:

```yaml
config:
  images.auto_update_interval: "0"
networks:
- config:
    ipv4.address: auto
    ipv6.address: none
  description: ""
  name: lxdbr0
  type: ""
  project: default
storage_pools:
- config:
    source: lxd-default
  description: ""
  name: default
  driver: zfs
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      network: lxdbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null
```

Save this file anywhere you like. I will just leave it in my home in this case at `$HOME/lxd-init.yml`

Now if you run `zfs list` again, you will see more datasets under "lxd-default".

```text
NAME                                   USED  AVAIL     REFER  MOUNTPOINT
lxd-default                            603K   289G       24K  legacy
lxd-default/buckets                     24K   289G       24K  legacy
lxd-default/containers                  24K   289G       24K  legacy
lxd-default/custom                      24K   289G       24K  legacy
lxd-default/deleted                    144K   289G       24K  legacy
lxd-default/deleted/buckets             24K   289G       24K  legacy
lxd-default/deleted/containers          24K   289G       24K  legacy
lxd-default/deleted/custom              24K   289G       24K  legacy
lxd-default/deleted/images              24K   289G       24K  legacy
lxd-default/deleted/virtual-machines    24K   289G       24K  legacy
lxd-default/images                      24K   289G       24K  legacy
lxd-default/virtual-machines            24K   289G       24K  legacy
```

## Non-interactive initialization

You can skip this step, but I want to leave this here for you in case you want to reinstall LXD.

### Remove LXD

[» Back to table of contents «](#table-of-contents)

Before you can try to reinitialize LXD, let's remove it first. You just run the following command to remove it but also save it in the Snap cache:

```bash
sudo snap remove lxd
```

Since leftover snap packages can take a lot of space, I usually like to "purge" the package without caching:

```bash
sudo snap remove lxd --purge
```

If you forget to add `--purge`, you can list the packages in the cache:

```bash
snap saved
```

Output

```bash
Set  Snap  Age    Version       Rev    Size    Notes
8    lxd   12.4s  5.15-3fe7435  25086  56.3kB  auto
```

And use the "Set" number to "forget" it. In my case the number is "8".

```bash
sudo snap forget 8
```

### Use the previously saved config to initialize LXD

[» Back to table of contents «](#table-of-contents)

Now we can reinstall LXD:

```bash
sudo snap install lxd --channel 5.15/stable
```

and reinitialize it non-interactively:

```bash
lxd init --preseed <$HOME/lxd-init.yml
```

This command can fail if the disk still have the ZFS filesystem:

```text
Error: Failed to create storage pool "default": Provided ZFS pool (or dataset) isn't empty, run "sudo zfs list -r lxd-default" to see existing entries
```

In order to fix this, you need to remove the subsets in the zfs pool using `zfs destroy lxd-default/<dataset name>` or you can remove the pool and recreate it:

```bash
sudo zpool destroy lxd-default
sudo zpool create "$name" "${disks[@]}"
```

See previous sections for the value of the variables.
Now you can run the init command again:

```bash
lxd init --preseed <$HOME/lxd-init.yml
```

## Let's see what we have now

[» Back to table of contents «](#table-of-contents)

If you try to list instances (containers and virtual machines)

```bash
lxc list
```

you will get the following output:

```text
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm

+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
```

As you can see LXD helps you to start your first VM, although it will be shown only once. There are just a few parameters because profiles contain some defaults. If you checked the generated init config carefully, you have already seen the default profile in it, but you can also get it later.

```bash
lxc profile list
```

Output:

```text
+---------+---------------------+---------+
|  NAME   |     DESCRIPTION     | USED BY |
+---------+---------------------+---------+
| default | Default LXD profile | 0       |
+---------+---------------------+---------+
```

Or get the content of the default profile:

```bash
lxc profile show default
```

Output:

```yaml
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by: []
```

You have a default network and a root disk for the operating system from the default LXD storage pool. The "show" and "list" subcommands work for these objects as well:

```bash
lxc network list
lxc network show lxdbr0
```

or

```bash
lxc storage list
lxc storage show default
```

But you could also edit them with the "edit" subcommand or create a new object with the "create" subcommand. For more details, please check the help from command line like this:

```bash
lxc storage create --help
```

or visit the [official documentation](https://documentation.ubuntu.com/lxd).

And finally I want to show you the command to list remote repositories which are similar to Docker registries if you know that concept, however, a remote can also be another LXD server which you want to control from a management server.

```bash
lxc remote list
```

Output

```text
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
```

## First virtual machine

[» Back to table of contents «](#table-of-contents)

The easiest way of creating virtual machines was shown by the first `lxc list` command, but that would create the VM with a random name as `docker run` creates containers without using the `--name` option, so I want to set the name as well passing it as the second argument after the image reference:

```bash
lxc launch --vm ubuntu:22.04 ubuntu-jammy-server
```

That would work, but I also want to override the CPU and memory limits and also the size of the root disk. If you started it already, you can remove it with the following command:

```bash
lxc delete ubuntu-jammy-server --force
```

Now the limits:

```bash
lxc launch --vm ubuntu:22.04 ubuntu-jammy-server \
  --config limits.memory=2GiB \
  --config limits.cpu=2 \
  --device root,size=20GiB
```

I believe the memory and cpu limits are obvious. The last line sets the size of the root disk which is 10GiB by default, but we changed it to 20GiB. If you start to run the following command quickly:

```bash
lxc exec ubuntu-jammy-server lsblk
```

it will first fail because of the LXD agent is still not running:

```text
Error: LXD VM agent isn't currently running
```

or show you a smaller root partition:

```text
sda       8:0    0    20G  0 disk
├─sda1    8:1    0   2.1G  0 part /
├─sda14   8:14   0     4M  0 part
└─sda15   8:15   0   106M  0 part /boot/efi
```

because it takes time to resize the partition after the initialization automatically. Eventually you should see something like this:

```text
sda       8:0    0    20G  0 disk
├─sda1    8:1    0  19.9G  0 part /
├─sda14   8:14   0     4M  0 part
└─sda15   8:15   0   106M  0 part /boot/efi
```

Note: If you want the root partition to be 20GiB and not the disk, you need to add the sizes of the other partitions to the size of the root disk in the `lxc launch` command.

If you run `lxc list` quickly instead of the above-mentioned lsblk, it can produce different outputs until the VM fully starts.

```text
+---------------------+---------+------+------+-----------------+-----------+
|        NAME         |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+---------------------+---------+------+------+-----------------+-----------+
| ubuntu-jammy-server | RUNNING |      |      | VIRTUAL-MACHINE | 0         |
+---------------------+---------+------+------+-----------------+-----------+
```

or

```text
+---------------------+---------+------------------------+------+-----------------+-----------+
|        NAME         |  STATE  |          IPV4          | IPV6 |      TYPE       | SNAPSHOTS |
+---------------------+---------+------------------------+------+-----------------+-----------+
| ubuntu-jammy-server | RUNNING | 10.177.180.83 (enp5s0) |      | VIRTUAL-MACHINE | 0         |
+---------------------+---------+------------------------+------+-----------------+-----------+
```

The IP address can appear as "IP (eth0)", disappear and reappear as "IP (enp5s0)" or similar. This is because during boot Ubuntu changes the name of the network interface. After that, the LXD agent process starts in the virtual machine, and you can run a bash inside similarly to `docker exec`, but this is not a surprise now as we already used `lxc exec` above. The difference is that we have an interactive terminal by default.

```bash
lxc exec ubuntu-jammy-server bash
```

Although the above command would work, sometimes it is better to define the commands after a double dash so the flags of the executable would not be passed to `lxc`

```bash
lxc exec ubuntu-jammy-server -- bash --help
```

## Overcommitting and how to avoid that

[» Back to table of contents «](#table-of-contents)

Even though this post is not for production systems, there is one thing that I feel I have to mention. There is a concept called "overcommitting". It means if you add up the amount of memory, CPUs or the size of root disks you assigned to each virtual machine, you get more than you physically have. Sometimes it could be useful for example when you know that each virtual machine will need  10 gigabytes of memory occasionally, but never at the same time. As you could see we actually used `limits.memory` and `limits.cpu` meaning that these are just limits, but don't guarantee that the required CPU resources are available at the moment. When you check the amount of resources in the virtual machine you will still see that as if you had what you requested. To make sure overcommitting doesn't happen you need to do the calculation to find out the amount of safely assignable resources before you create the new virtual machine or before change the configuration later. To add up all the memory you assigned to LXD virtual machines you can run the following command and get a number in gibibytes:

```bash
lxc list --format json \
  | jq -r '.[].config."limits.memory"' \
  | numfmt --suffix B --from iec-i --to-unit=1073741824 \
  | cut -dB -f1 \
  | paste -s -d+ - \
  | bc
```

or

```bash
lxc list --format json \
  | jq -r '.[].config."limits.memory"' \
  | numfmt --suffix B --from iec-i --to-unit=1073741824 \
  | cut -dB -f1 \
  | jq -s add
```

If the above commands didn't work for you, it could be because they require the [jq command](https://jqlang.github.io/jq/) and [numfmt](https://manpages.ubuntu.com/manpages/focal/man1/numfmt.1.html) as part of [coreutils](https://howtoinstall.co/en/coreutils).

Now that you know that, doing the same with the CPU resources will be much easier due to the lack of units as it is just a number:

```bash
lxc list --format json \
  | jq -r '.[].config."limits.cpu"' \
  | paste -s -d+ - \
  | bc
```

or

```bash
lxc list --format json \
  | jq -r '.[].config."limits.cpu"' \
  | jq -s add
```

Let's do it with the root disks:

```bash
lxc list --format json \
  | jq -r '.[].devices.root | select(.type == "disk") | .size ' \
  | numfmt --suffix B --from iec-i --to-unit=1073741824 \
  | cut -dB -f1 \
  | paste -s -d+ - \
  | bc
```

or

```bash
lxc list --format json \
  | jq -r '.[].devices.root | select(.type == "disk") | .size ' \
  | numfmt --suffix B --from iec-i --to-unit=1073741824 \
  | cut -dB -f1 \
  | jq -s add
```

Unfortunately this works only if you defined the resource limits in the instance config and not in a profile. Not to mention when you didn't define it anywhere or when you have multiple projects (like namespaces in Kubernetes) or multiple storage pools. So to really avoid overcommitting you need to do more than I could show you in this post.

## Use profiles to install similar virtual machines easily

[» Back to table of contents «](#table-of-contents)

Up until now everything we did was based on the default profile. When we wanted more memory, cpus or a bigger root disk, we used the `--config` or `--device` options to override the default values. Let's use profiles to make it easier. I like Docker, so I will create a `docker-host` profile, but first I need volumes (new virtual disks) for `/var/lib/docker` in the virtual machine.

```bash
pool_name="default"
volume_name="docker-01-docker-data-root"
size="50GiB"

lxc storage volume create --type "block" "$pool_name" "$volume_name" "size=$size"
```

The volume name could be actually anything, but I chose the format of `<vm_name>-<device_name>`, so when I list the volumes, I will know which volume is used by which docker host.

Now repeat the creation with `docker-02`:

```bash
pool_name="default"
volume_name="docker-02-docker-data-root"
size="50GiB"

lxc storage volume create --type "block" "$pool_name" "$volume_name" "size=$size"
```

To list the volumes, run the following command:

```bash
pool_name="default"
lxc storage volume list "$pool_name" type=custom
```

Output:

```text
+--------+----------------------------+-------------+--------------+---------+
|  TYPE  |            NAME            | DESCRIPTION | CONTENT-TYPE | USED BY |
+--------+----------------------------+-------------+--------------+---------+
| custom | docker-01-docker-data-root |             | block        | 0       |
+--------+----------------------------+-------------+--------------+---------+
| custom | docker-02-docker-data-root |             | block        | 0       |
+--------+----------------------------+-------------+--------------+---------+
```

We also need a new profile named as "docker-host"

```bash
lxc profile create docker-host
```

This will be an empty profile, but we can change it soon. Save the following content in a file named as `lxd-profile-docker-host.yml`:

```yaml
config:
  boot.autostart: "false"
  limits.cpu: "4"
  limits.memory: 4GiB
  cloud-init.user-data: |
    #cloud-config
    users:
      - name: manager
        lock_passwd: false
        groups: sudo
        shell: /bin/bash
        passwd: "$6$5PNw7C5RTRxcv96e$CfGSFJZ/y4vMtO.wAs.fE59fXGJT.65.rEnjNnYgNa5axrvHS8B0X53sMaoCeoCOb9PuZMnzbBaZkvodzwH/s0"

    runcmd:
      - apt-get update
      - apt-get install -y jq
      - |
        /bin/bash -c '
          disk_path=$(lsblk --json --output-all | jq --raw-output ".blockdevices[] | select(.serial == \"lxd_docker--data--root\") | .path");
          e2label "$disk_path" || mkfs.ext4 "$disk_path";
          e2label "$disk_path" docker-data-root;
          echo "LABEL=docker-data-root  /var/lib/docker  ext4  defaults  0  0" >> /etc/fstab;
          mkdir /var/lib/docker
          mount "LABEL=docker-data-root";
        '
      # see: https://docs.docker.com/engine/install/ubuntu/
      - sudo apt-get update
      - sudo apt-get install -y ca-certificates curl gnupg
      - sudo install -m 0755 -d /etc/apt/keyrings
      - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
      - sudo chmod a+r /etc/apt/keyrings/docker.gpg
      - |
        echo \
          "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
          "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" \
          | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
      - sudo apt-get update
      - |
        /bin/bash -c '
          VERSION_STRING="5:24.0.4-1~ubuntu.22.04~jammy" \
          && sudo apt-get install -y \
              docker-ce=$VERSION_STRING \
              docker-ce-cli=$VERSION_STRING \
              containerd.io \
              docker-buildx-plugin \
              docker-compose-plugin \
          && sudo apt-mark hold docker-ce docker-ce-cli
        '
description: Docker Host
devices:
  root:
    path: /
    pool: default
    size: 20GiB
    type: disk
  docker-data-root:
    pool: default
    type: disk
    source: this_is_a_placeholder__keep_it_here
```

This profile will work only with Ubuntu 22.04 instances, but you can modify it to remove all the references to the version number like the held package versions. Let's update the `docker-host` profile:

```bash
lxc profile edit docker-host <lxd-profile-docker-host.yml
```

At the beginning, we have memory and cpu resources and also an autostart setting which is similar to Docker's restart policy. I set it to "false", so when I start my host, the virtual machine will not start automatically:

```yaml
config:
  boot.autostart: "false"
  limits.cpu: "4"
  limits.memory: 4GiB
```

Then we use cloud-init to create a user and run commands at first boot. The value of `cloud-init.user-data` is a string. That's why we use a pipe character after the parameter name. This is a yaml featue, not LXD and not Cloud init.

```yaml
cloud-init.user-data: |
  #cloud-config
```

Now let's see the user creation:

```yaml
users:
  - name: manager
    lock_passwd: false
    groups: sudo
    shell: /bin/bash
    passwd: "$6$5PNw7C5RTRxcv96e$CfGSFJZ/y4vMtO.wAs.fE59fXGJT.65.rEnjNnYgNa5axrvHS8B0X53sMaoCeoCOb9PuZMnzbBaZkvodzwH/s0"
```

The value of `passwd` is a password hash which I generated with the following command:

```bash
openssl passwd -6
```

I got a prompt and set the password to be "password". The output was the hash which I added to the user definition quoted

Then we use cloud-init's `runcmd` which will run at first boot. The value of `runcmd` will be a list. Sometimes we have to use special syntax, because not everything is valid in this yaml list that would be valid in a shell script. For example for some reason I don't know yet, environment variables can't be defined unless you run everything directly as an argument of a shell. This could have a similar reason to what I mentioned in another post [about Docker's CMD, ENTRYPOINT and SHELL instructions](https://dev.to/rimelek/constructing-commands-to-run-in-docker-containers-2g2i).

That's why you can see this syntax in the list:

```yaml
- |
  /bin/bash -c '
    disk_path=$(lsblk --json --output-all | jq --raw-output ".blockdevices[] | select(.serial == \"lxd_docker--data--root\") | .path");
    e2label "$disk_path" || mkfs.ext4 "$disk_path";
    e2label "$disk_path" docker-data-root;
    echo "LABEL=docker-data-root  /var/lib/docker  ext4  defaults  0  0" >> /etc/fstab;
    mkdir /var/lib/docker
    mount "LABEL=docker-data-root";
  '
```

I use a similar method where I install Docker:

```yaml
- |
  /bin/bash -c '
    VERSION_STRING="5:24.0.4-1~ubuntu.22.04~jammy" \
    && sudo apt-get install -y \
         docker-ce=$VERSION_STRING \
         docker-ce-cli=$VERSION_STRING \
         containerd.io \
         docker-buildx-plugin \
         docker-compose-plugin \
    && sudo apt-mark hold docker-ce docker-ce-cli
  '
```

This is where you could change the version Docker, or you can just remove the version completely if you always want the latest:

```yaml
- |
  sudo apt-get install -y \
    docker-ce \
    docker-ce-cli \
    containerd.io \
    docker-buildx-plugin \
    docker-compose-plugin \
  && sudo apt-mark hold docker-ce docker-ce-cli
```

Both solutions will make sure you don't upgrade Docker accidentally which is very important in production.

In the devices section, we also set the size of the root disk and define a non-existent volume just as a placeholder to have a valid profile, but we will need to override the volume name when we create the virtual machine. It helps us to have fewer arguments in the command line and also make sure we don't forget to add a disk for the docker data since it will throw an error if we leave the non-existent volume in the definition without overriding it from command line.

Note: This trick can also be problematic if you want to add the profile to an existing virtual machine, because then you can't override the value before adding the profile, but you can't add the profile because the volume does not exist. If this is a problem for you, you can just remove that `docker-data-root` section and add the device manually later. 

Now create the virtual machine:

```bash
lxc launch ubuntu:22.04 docker-01 --vm \
  --profile default \
  --profile docker-host \
  --device docker-data-root,pool=default \
  --device docker-data-root,type=disk \
  --device docker-data-root,source=docker-01-docker-data-root 
```

Optionally you can see the boot process:

```bash
lxc console docker-01
```

You may need to press enter to get the login prompt after the installation, but the virtual machine doesn't have a user with a password, so it will just show you if the installation process was finished.

In order to exit the console, press `CTRL+a` and after that press 'q'.

Now you can repeat the same commands with `docker-02` for which you need to mount `docker-02-docker-data-root` as docker data root.

## Checking the installed virtual machines

[» Back to table of contents «](#table-of-contents)

It's time to confirm whether the installation was successful. I will only do it with `docker-01` the process is the same with any VM.

We know we wanted to mount a new disk to `/var/lib/docker`. The `df` command can show us what disk was mounted to a specific folder, so let's run the following commands:

```bash
lxc exec docker-01 -- df -h /var/lib/docker
```

The output should be only one line (and the header):

```text
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         49G  280K   47G   1% /var/lib/docker
```

Check if the size is what you expected. To be honest, I did not expect the size to be just 49G since I specifically asked for 50GiB. You can also see that the available size is 47G, so the difference is much more than the size of the used disk space which is 280K. The fact is that zfs needs some space for itself. Assuming you named everything I did in this tutorial, run the following command to get some information of the dataset of the volume:

```bash
zfs list lxd-default/custom/default_docker-01-docker-data-root
```

Output

```text
NAME                                                    USED  AVAIL     REFER  MOUNTPOINT
lxd-default/custom/default_docker-01-docker-data-root  1.04G   260G     1.04G  -
```

Here you can see 1.04G is used on the volume, but it is not on the filesystem, so it could explain why we got only 49G and not 50G. Although I'm still not sure why is the difference between the disk size and the available size. Again, this is something that you need to consider when you define the size of the volume.

You also want to know if Docker was installed successfully.

```bash
lxc exec docker-01 docker version
```

If it works, you finally have a virtual machine with Docker inside on a dedicated volume installed automatically. You can use the same profile to add more virtual machines like this

## Conclusion

[» Back to table of contents «](#table-of-contents)

So even though I explained some LXD-related issues in this post, profiles can make virtual machine creation easier and the interface is user-friendly. As of LXD 5.14, Canonical also provide a web-based GUI, so you can install desktop operating systems and get a web console with graphical interface.

{% embed https://github.com/canonical/lxd-ui %}

At the time of writing this post Canonical [released a preview version of LXD 5.16](https://discourse.ubuntu.com/t/lxd-5-16-has-been-released/37150), so you can also install that and fall back to the version mentioned in this post only if the instructions don't work with the new version.

Have you been using LXD for a while, and you have a solution for some of the issues I mention? I would love to hear from you in the comment section! :) Are you a beginner, and you still don't understand something in the post? The comment section is yours too so sharing your opinions and questions you can help me create better tutorials in the future.