Earthly: Beyond Docker
Tags: earthly, cicd, docker, tutorial
Published: 2021-12-07 17:30:50

We've been using containers for software packaging for a long time. But let’s face it, Docker's user experience isn't really the greatest. Building and testing usually entails having a bunch of Makefiles, fiddly Dockerfiles, a bit of black magic, and a bundle of Bash scripts. It’s a path that can lead into insanity.

And it’s a shame because Docker could be much friendlier. [Earthly](https://earthly.dev/) developers certainly believe in this: they have created a tool that fills the space between Docker and Make.

At the end of this tutorial, you’ll know how Earthly works. As a bonus, you'll have a pipeline that builds, tests, and pushes a Docker image into a remote repository.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gk54dxip1cv0n5ehpekn.png)

## How Earthly works

Earthly is a Docker-based build tool. It does not, however, replace language-specific tools like Maven, Gradle, or Webpack. Instead, it leverages and integrates with them — acting as a glue.

Using Docker containers as the core mechanic to achieve repeatability, every Earthly command runs in an isolated environment and is effortlessly parallelized whenever possible.

Earthly consists of the CLI binary and a Docker image called [earthly/buildkit](https://hub.docker.com/r/earthly/buildkitd), based on [Docker’s BuildKitd](https://docs.docker.com/develop/develop-images/build_enhancements/).


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sbsn08t94nqt1rh4uem1.png)

Earthly wants you to get the same experience everywhere. BuildKit’s role is to yield portable Docker-compatible layers. Thus, we get the same result whether on our local machine and in our CI systems.

## Does Earthly deliver on its promises?

Earthly promises better, repeatable builds. But, does it deliver? Here are some of the best features and some of the problems found while doing a test drive.

Pros:

- Earthly, like Docker, is language-agnostic. Its syntax feels familiar since it mixes Make and Dockerfiles.
- Their site has good documentation.
- Configuration is low maintenance. There are only a few settings to tweak.
- Useful even in the cases you don’t need Docker. Container images are only one of the possible outputs. For instance, you can use Earthly to test and compile binaries in a clean environment.
- The import system works well in multirepo and [monorepos](https://semaphoreci.com/blog/what-is-monorepo).

Cons:

- It doesn’t actually replace Makefile or Docker Compose. And you will still need some shell scripts.
- It’s slower than native Docker because it relies on copying files instead of mounts. Of course, this is by design to ensure repeatability.
- Multiplatform builds sometimes use [emulation](https://docs.earthly.dev/docs/guides/multi-platform), so it may be too slow for some use cases.
- Error messages can get really confusing.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u7hzyehcwkzjw9tan231.png)

## Setting up Earthly

Earthly configuration files are called, as typical in software engineering, `Earthfiles`. Opening one reminds us of Dockerfiles with a twist of Makefiles.

Earthly takes the syntax from Docker and expands it for a general use case. While Dockerfiles are only meant to produce container images, Earthly can generate artifacts.

The following Earthfile was taken from Earthly’s official guide. It describes two targets: `build` and `docker`.

``` dockerfile
FROM golang:1.15-alpine3.13
WORKDIR /go-example

build:
    COPY main.go .
    RUN go build -o build/go-example main.go
    SAVE ARTIFACT build/go-example /go-example AS LOCAL build/go-example

docker:
    COPY +build/go-example .
    ENTRYPOINT ["/go-example/go-example"]
    SAVE IMAGE go-example:latest
```

The `build` target compiles a Go program inside a container, while `docker` copies it into a production-ready image that can be pushed into any container registry and run anywhere.

The `FROM` keyword works as you would expect if familiar with Dockerfiles. Likewise, `COPY`, `ENTRYPOINT`, `WORKDIR`, and `RUN`, behave similarly. Two concepts deserve a pause to explain further:

- **targets**: the thing we want to build. Earthly uses a target-based system inspired by Make.
- **references**: targets can reference other targets. As in Make, Earthly will sort dependencies and do the right thing.
- **extended commands**: Earthly features specialized commands such as SAVE IMAGE and SAVE ARTIFACT.

To run this Earthfile we need to run: `earthly +TARGET`. The first time, Earthly will pull the image and create a cache volume. The cache will persist the build files between runs.

``` bash
$ earthly +build

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
golang:1.15-alpine3.13 | --> Load metadata linux/arm64
             context | --> local context .
               +base | --> FROM golang:1.15-alpine3.13
             context | transferred 1 file(s) for context . (2.1 MB, 4 file/dir stats)
               +base | *cached* --> WORKDIR /go-example
              +build | *cached* --> COPY main.go .
              +build | *cached* --> RUN go build -o build/go-example main.go
              output | --> exporting outputs
================================ SUCCESS [main] ================================
              +build | Artifact +build/go-example as local build/go-example

```

Or we can directly go to the Docker generation target with:

``` bash
$ earthly +docker

               +base | --> FROM golang:1.15-alpine3.13
               +base | *cached* --> WORKDIR /go-example
             context | transferred 3 file(s) for context . (2.1 MB, 4 file/dir stats)
              +build | *cached* --> COPY main.go .
              +build | *cached* --> RUN go build -o build/go-example main.go
              +build | *cached* --> SAVE ARTIFACT build/go-example +build/go-example AS LOCAL build/go-example
             +docker | --> COPY +build/go-example ./
              output | [██████████] exporting layers ... 100%
              output | [          ] exporting manifest
================================ SUCCESS [main] ================================
             +docker | Image +docker as go-example:latest
              +build | Artifact +build/go-example as local build/go-example

```

Since the `docker` target depends on `build`, Earthly first runs the build step and then proceeds to build and export the image into the local Docker directory.

## Extending Earthfiles

Let’s try Earthly in a more involved example. Please go ahead and fork this [demo repository](https://github.com/TomFern/semaphore-demo-earthly).

The provided Dockerfile builds a Node Alpine container image.

``` dockerfile
FROM node:14.17-alpine3.12
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY src src
EXPOSE 3000
ENTRYPOINT ["npm", "start"]
```

You can build it with: `docker build . -t mydemo` and then run it as: `docker run -it -p 3000:3000 mydemo`. Browsing the localhost on port 3000 should print a "hello, world" message.

How would we go about turning this into a working Earthfile? Let’s start one from scratch.

The first two lines remain the same and will be used for every target.

``` dockerfile
FROM node:14.17-alpine3.12
WORKDIR /app
```

Now we add the build target, which downloads dependencies and copies the source.

``` makefile
build:
    COPY package.json package-lock.json ./
    COPY --if-exists node_modules node_modules
    RUN npm install
    COPY src src
```

We have added a `COPY` statement to copy the node_modules folder into the build environment.

Since npm runs in the container, we need to copy its output files back outside. We do this with Earthly’s special command `SAVE ARTIFACT ... AS LOCAL`.

``` makefile
    SAVE ARTIFACT node_modules AS LOCAL ./node_modules
    SAVE ARTIFACT package.json AS LOCAL ./package.json
    SAVE ARTIFACT package-lock.json AS LOCAL ./package-lock.json
```

Running `earthly +build` should install the Node dependencies. The end result of this process is similar to what we would get if we run `npm install` directly, with the added benefit that we get the same outcome everywhere.

Next, we should create the Docker image. We can achieve this with:

``` makefile
docker:
    FROM +build
    EXPOSE 3000
    ENTRYPOINT ["npm", "start"]
    SAVE IMAGE semaphore-demo-earthly:latest
```

The new target picks up from where the build stopped. It adds the start command and exports the image into the host’s local Docker registry.

## Testing with Earthly

One of the most powerful features Earthly brings is the ability to extend Dockerfiles with tests. Let’s add unit tests by extending `build` with a target that runs: `npm test`.

``` makefile
tests:
    FROM +build
    COPY spec spec
    RUN npm test
```

Now we can run `earthly +tests` to get the results.

Adding a linting target is also trivial:

``` makefile
lint:
    FROM +build
    COPY .jshintrc ./
    RUN npm run lint
```

To run more involved tests, such as integration tests or end-to-end tests, we can reuse existing Docker Compose manifests in order to bring up other containers during the integration stage.

``` makefile
integration-tests:
    FROM +build
    COPY docker-compose.yml ./
    COPY integration-tests integration-tests
    WITH DOCKER --compose docker-compose.yml --service db
        RUN sleep 10 && npm run integration-tests
    END
```

Earthly uses a Docker-in-Docker approach to start helper containers:

1. Copies `docker-compose.yml` and the integration tests in the main container (the application container).
2. Starts a PostgreSQL container inside the main container. The special `WITH DOCKER` command accepts *precisely one* `RUN` statement.
3. Runs the integration tests.

Our Earthfile is good enough for now. Let’s upload it into the repository so we can configure a CI/CD pipeline in the next section.

``` bash
$ git add Earthfile
$ git commit -m "push Earthfile"
$ git push origin master
```

## Earthly Continuous Integration Pipeline

We are now all set to test the application with Earthly and [continuous integration](https://semaphoreci.com/continuous-integration). Our pipeline will pull to and push from Docker Hub, something that will require authentication with the service.

Authentication in Semaphore happens via [secrets](https://docs.semaphoreci.com/guided-tour/environment-variables-and-secrets/). To create an encrypted secret, go to your organization menu and click on Settings > Secrets. Then, press **Create secret** and type the variables `DOCKER_USERNAME` and `DOCKER_PASSWORD` as shown below:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6px8ln8gxbxatr18gaky.png)

Next, initialize the forked demo repository. The procedure for adding a project is detailed in the [getting started guide](https://docs.semaphoreci.com/guided-tour/getting-started/). Go check it out if this is the first time using Semaphore.

Since Earthly doesn’t come preinstalled in the [Ubuntu CI image](https://docs.semaphoreci.com/ci-cd-environment/ubuntu-20.04-image/), we need to install it for every job. The simplest way of achieving this is with a [pipeline-level prologue](https://docs.semaphoreci.com/reference/pipeline-yaml-reference/#an-example-of-using-global_job_config-property). The commands in the global prologue are executed before every job in the pipeline.

The Earthly installation command for Linux is:

``` bash
sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'
```

Click on the [pipeline](https://semaphoreci.com/blog/cicd-pipeline) and type the line in the prologue section.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jlblk97tf9xy7pxjdo13.png)

### Build block

The first block will start a [build stage](https://semaphoreci.com/blog/build-stage). In it, we will download and cache Node dependencies.

Type the following commands in the job:

``` bash
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly --ci +build
cache store
```

We’ll run these commands or a variation of them in every successive job in the pipeline:

- **docker login**: logs into Docker Hub with the credentials defined in the secret.
- **checkout**: clones the repository into the [CI machine](https://docs.semaphoreci.com/ci-cd-environment/machine-types/).
- **cache**: With the [cache](https://docs.semaphoreci.com/reference/toolbox-reference/#cache) command, we can store `node_modules` in the project-level storage provided by Semaphore.
- **earthly** **ci**: runs Earthly with some CI-optimized settings.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dw8wafetc53jg38k5jf3.png)

To finish the block, enable the “dockerhub” secret, so the variable is decrypted and exported in the job.

### Tests block

The next block will run fast tests such as unit tests and linting. Add the following commands in the [block’s prologue](https://docs.semaphoreci.com/reference/pipeline-yaml-reference/#prologue):

``` bash
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/andh5vcrei046d327yrk.png)

Next, create two jobs:

```
earthly --ci +tests

```

And:

```
earthly --ci +lint
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xilc70iwujl1nmqlnawf.png)

### Integration tests block

The last job will run the integration test. Type the following commands. The `-P` switch is needed to enable [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) inside Docker (needed for Docker-in-Docker).

```
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly --ci -P +integration-tests
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e9xucd7wmv2l74x5dldj.png)

Don’t forget to check that the secret is enabled in all blocks. When ready, run the workflow to try the CI pipeline.



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/modj6t6tfp5mluibquxv.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0wrbl9l52qjrisjfmgyi.png)

## Continuous Delivery with Earthly

The final result of a CI/CD pipeline is to deliver something that can be deployed. Here’s where the [continuous delivery](https://semaphoreci.com/cicd) part comes into play. We’ll push the image to a remote repository so it can be later deployed in production.

A Semaphore workflow can span multiple pipelines, as long as they’re connected by [promotions](https://docs.semaphoreci.com/essentials/deploying-with-promotions/).  Go back to the workflow editor and click on **Add Promotion**. Click the new pipeline and copy the global Earthly install commands, again:

```
sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'
```

The block in the pipeline will contain a single job to generate and push the Docker image into Docker Hub. The commands are as follows:

```
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly +docker
docker tag semaphore-demo-earthly $DOCKER_USERNAME/semaphore-demo-earthly
docker push $DOCKER_USERNAME/semaphore-demo-earthly
```

We use `earthly +docker` to export the image into the CI machine, then a combination of `docker tag` and `docker push` to send it to the remote registry.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ic0m73cw50d3gw3ldvst.png)

The only thing left is to enable the Docker Hub secret … and we’re done!

Rerun the workflow and click on promote to try pushing the image.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z4srdw75s6vnn3rp5fbm.png)

## What do you think?

The Earthly project is young and under heavy development, with many features still in the experimental stage (we stuck to stable features during the course of the post). It’s not perfect, but it feels like a massive step in the right direction.

Skill up your Docker knowledge with these tutorials:

- [Kubernetes vs. Docker: Understanding Containers in 2021](https://semaphoreci.com/blog/kubernetes-vs-docker)
- [How To Deploy a Go Web Application with Docker](https://semaphoreci.com/community/tutorials/how-to-deploy-a-go-web-application-with-docker)
- [Dockerizing a Node.js Web Application](https://semaphoreci.com/community/tutorials/dockerizing-a-node-js-web-application)
- [Dockerizing a Ruby on Rails Application](https://semaphoreci.com/community/tutorials/dockerizing-a-ruby-on-rails-application)

Have you tried Earthly? Tell us about your experience.