Running Vagrant on an M1 Apple Silicon using Docker
Tags: docker, vagrant, linux, arm64
Published: 2021-04-22 01:42:02

I've recently started at a new job, and I'm fortunate enough to have a new M1 Macbook Air. I'd heard from the wider community that everything "just works" on the M1, even though it's not an Intel chip. "Even Docker" they said. "Wow" I thought, this will be seamless, incredible.

So I sit there on my first day ready to get started. NPM installs everything fine, my python packages all install and both the backend and frontend are running, too easy! Then I hit an issue. This application requires a service which is run using Vagrant on Virtualbox.

Virtualbox very much doesn't support Apple Silicon. It's unclear whether Virtualbox will ever support Apple Silicon. The preferred alternative is a product by VMWare which also doesn't (yet) support Apple Silicon. I'm in a bit of a pickle.

But! Never fear! I can see that Vagrant also supports Docker. So over the next three days I dive into documentation, I frantically search the web for "vagrant docker M1" and "vagrant docker config" and "vagrant docker network issue" and "vagrant docker systemctl". After shaving at least three seperate yaks, and configuring a monstrosity I got it all running! Here's some lessons I learned.

_Disclaimer: I am very much **not** a linux / ops / devops person. My experience with linux is limited but there's not much help out there for this issue so I'm doing my best._


# Docker and Vagrant are frenemies

Docker and Vagrant have very different philosophies about how you should run your development setup. Docker's ideal is a minimalist setup, with just enough to run the single process you need. If you want more processes, Docker wants you to create more containers. Vagrant prefers a maximalist style, install everything on the one virtual machine and get it all going together. 

To get Docker going you'll need a `Dockerfile` this describes the setup needed to create the Docker container. For our Docker container to be friends with Vagrant we're going to configure it as if its a traditional linux machine. By default you'll need at least `sshd` (which lets you SSH into the machine). I also needed `systemd` (which runs services). This is very much not how you should do Docker normally.

Here's the `Dockerfile` I used:

```sh
# Docker image to use with Vagrant
# Aims to be as similar to normal Vagrant usage as possible
# Adds Puppet, SSH daemon, Systemd
# Adapted from https://github.com/BashtonLtd/docker-vagrant-images/blob/master/ubuntu1404/Dockerfile

FROM ubuntu:18.04
ENV container docker
RUN apt-get update -y && apt-get dist-upgrade -y

# Install system dependencies, you may not need all of these
RUN apt-get install -y --no-install-recommends ssh sudo libffi-dev systemd openssh-client

# Needed to run systemd
# VOLUME [ "/sys/fs/cgroup" ]
# Doesn't appear to be necessary? See comments

RUN apt-get -y install puppet

# Add vagrant user and key for SSH
RUN useradd --create-home -s /bin/bash vagrant
RUN echo -n 'vagrant:vagrant' | chpasswd
RUN echo 'vagrant ALL = NOPASSWD: ALL' > /etc/sudoers.d/vagrant
RUN chmod 440 /etc/sudoers.d/vagrant
RUN mkdir -p /home/vagrant/.ssh
RUN chmod 700 /home/vagrant/.ssh
RUN echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ==" > /home/vagrant/.ssh/authorized_keys
RUN chmod 600 /home/vagrant/.ssh/authorized_keys
RUN chown -R vagrant:vagrant /home/vagrant/.ssh
RUN sed -i -e 's/Defaults.*requiretty/#&/' /etc/sudoers
RUN sed -i -e 's/\(UsePAM \)yes/\1 no/' /etc/ssh/sshd_config

# Start SSH
RUN mkdir /var/run/sshd
EXPOSE 22
RUN /usr/sbin/sshd

# Start Systemd (systemctl)
CMD ["/lib/systemd/systemd"]
```

If you don't need `systemd` in your setup then you can remove that and change the `RUN` of `sshd` to:

```
CMD ["/usr/sbin/sshd", "-D"]
```

All of this sets up a Docker container which doesn't work like a regular Docker container. It runs more like a virtual machine. This means it will be difficult to manage using the normal Docker commands. Once you take it down it will also be difficult to get up again. Best to control it with Vagrant. I've had to delete the container and re-create it with Vagrant many times.


# Vagrant is pretty easy to configure

To get Vagrant running using your Dockerfile you just need to add a little bit of configuration:

```ruby
Vagrant.configure(2) do |config|

# ... your existing config

  # Custom configuration for docker
  config.vm.provider "docker" do |docker, override|
    # docker doesnt use boxes
    override.vm.box = nil
    
    # this is where your Dockerfile lives
    docker.build_dir = "."

    # Make sure it sets up ssh with the Dockerfile
    # Vagrant is pretty dependent on ssh
    override.ssh.insert_key = true
    docker.has_ssh = true

    # Configure Docker to allow access to more resources
    docker.privileged = true
  end

# ...

end
```

You may need to override some other settings here. The `override` variable allows you to change top-level settings, while the `docker` variable allows you to setup docker-specific stuff. Have a read of the [vagrant docker provider](https://www.vagrantup.com/docs/providers/docker) and [vagrant docker provisioning](https://www.vagrantup.com/docs/provisioning/docker) documentation.

Once you've set this up, you can run your Vagrant machine with:

```
$ vagrant up --provider=docker
```

And if you need to do extra things inside the machine, you can ssh in with:

```
$ vagrant ssh
```

You should have `sudo` access as well if you need it.


# Vagrant doesn't like to run multiple boxes

When I first ran Vagrant, before I realised Virtualbox wouldn't work it started and then failed. Then when I tried to run vagrant with the Docker provider it gave me errors about not having multiple boxes. I also couldn't delete my box because virtualbox wasn't running.

You can just delete the files in `.vagrant` to resolve that. It seems like Vagrant doesn't have a good way of resolving this itself.


# You can re-provision Vagrant

Sometimes I found my setup would break, in my case it wasn't too hard to just run the Vagrant provisioning script again.

```
$ vagrant up --provision
```

Once that was complete some of my issues got resolved. It's not a good idea to blindly do this, but in my case I'm just trying to get it running.


# Docker's public network bridge feature is broken

It doesn't seem to work on Mac! So if your Vagrant config has something like:

```
config.vm.network "public_network"
```

Try to see if you can replace it with:

```
config.vm.network "private_network", type: "dhcp"
```

In my case this substitution was sufficient, but you may have other requirements.


# Your linux binaries need to be arm64

I ran into some issues because our provisioning scripts assumed that the architecture of the system would be `amd64` (Intel and AMD 64-bit processors). It was a pretty safe bet that the CPU was `amd64` up until Apple introduced their M1 chip! 

Running linux on the M1 chip uses the `arm64` architecture instead. As a handy shortcut you can get the architecture of the current machine in linux with:

```
$ dpkg --print-architecture
```

I used this in shell scripts by setting a variable:

```sh
readonly ARCH=`dpkg --print-architecture`
```

And then substituting that in when binaries were downloaded and installed, usually that was relatively simple, e.g.:

```
wget http://somedomain.com/some_binary_linux_amd64.zip
```

Became

```
wget "http://somedomain.com/some_binary_linux_${ARCH}.zip"
```

# I learned a lot - but I still don't know much

I learned a lot about Docker (and how not to do things) Vagrant (and how to hack it) and some various linux tricks trying to get this all working. However I'm very much not a linux, Docker or Vagrant person, I'm just trying to make things work. So if I've written anything here horribly wrong, or extremely misguided, please throw down a comment.

If this helped you I'd love to hear it! I got stuck googling around for how to do this, and I imagine there are other people out there in the same boat.