Deploying a Hugo Site with Git Submodules on AWS Amplify
Tags: hugo, git, submodules, amplify
Published: 2020-03-26 14:34:51

---
title: Deploying a Hugo Site with Git Submodules on AWS Amplify
published: true
date: 2020-03-25 12:34:55 UTC
tags: [ "hugo", "git", "submodules", "amplify" ]
canonical_url: https://rewritable.me/o81kt9y8zb/
---
In the course of deploying my site to AWS Amplify, I ran into a few hiccups, attributable to both my project configuration and to the behavior of Hugo itself.

After completing the simple setup process, Amplify ran the initial deployment and showed green check marks across the board, but the site was "unreachable" when I opened the URL.

The default build configuration contains a single command: "hugo".

```txt
# amplify.yml

version: 0.1
frontend:
  phases:
    build:
      commands:
        - hugo
  artifacts:
    baseDirectory: public
    files:
      - '**/*'
  cache:
    paths: []
```

I tried reproducing the problem locally, by following the same steps:

1. Clone the repo.
1. Change to the project directory.
1. Run `hugo`.

This revealed the problem:

```
WARN 2020/03/24 01:56:23 found no layout file for "HTML" for "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2020/03/24 01:56:23 found no layout file for "HTML" for "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2020/03/24 01:56:23 found no layout file for "HTML" for "taxonomyTerm": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2020/03/24 01:56:23 found no layout file for "HTML" for "taxonomyTerm": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2020/03/24 01:56:23 found no layout file for "HTML" for "taxonomyTerm": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2020/03/24 01:56:23 found no layout file for "HTML" for "taxonomyTerm": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.

| EN
+------------------+----+
Pages            |  6
Paginator pages  |  0
Non-page files   |  0
Static files     |  1
Processed images |  0
Aliases          |  1
Sitemaps         |  1
Cleaned          |  0
Total in 11 ms
```

In other words, even with template files missing (in this case, all of them), Hugo still "successfully" builds the site. (While it wasn't the root of the problem, this is a little terrifying; I looked for an option for `hugo` to fail on warnings, but had no such luck.)

> In UNIX convention, success or failure of a command is signaled by the "exit code" (0 for success, non-zero for failure). A build will fail if any one of its commands fail. Despite that the output above looks very much like a failure to a human, the 0 exit code means "success" to the builder.

## Cloning Submodules

In any case, the reason the template files were missing is that my theme is separate repo, included as a Git submodule.  This enables me to develop both as separate public projects without one polluting the other. As I learned, though, running `git clone` will not include a repo's submodules by default (notice the empty `rewritable.hugo` directory below):

```txt 
$ git clone git@gitlab.com:rewritable/rewritable.me.git

...

$ tree rewritable.me/themes/ -L 2
rewritable.me/themes/
└── rewritable.hugo
```

It's possible to clone a repo along with its submodules (and their submodules), all in one command, by passing the `--recurse-submodules` option:

```
$ git clone --recurse-submodules git@gitlab.com:rewritable/rewritable.me.git

...

$ tree rewritable.me/themes/ -L 2
rewritable.me/themes/
└── rewritable.hugo
    ├── archetypes
    ├── assets
    ├── layouts
    ├── UNLICENSE.md
    ├── config.toml
    └── theme.toml
```

In my case, I needed a command to pull them down after having cloned and changed to the directory of the main repo (the context in which the `commands` in `amplify.yml` are executed).

That command is:

```
git submodule update --init
```

However, even my theme submodule contains a submodule (the Feather icons repo), which would not be pulled by the above command. For that, there's the `--recursive` flag:

```
git submodule update --init --recursive
```

One last detail: pulling down my small theme repo along with its entire Git history is no big deal, but the history of popular open source projects can be expansive. When I began developing my theme, I had included the Bulma CSS framework; now there's only Feather, but I could have included both or more.

Downloading the full history of even one such project can take seconds, and those add up to minutes of CPU time, which cost money (not to mention my patience).

In Git parlance, cloning only the latest state of a repo, with no history, is called a "shallow clone". This is achieved by passing the `--depth` flag, allowing to specify the desired depth of Git history in terms of commits (starting from and including the latest commit):

```
git clone --depth 1 <repository>
```

The `--depth` flag can also be passed to `git submodule update`:

```
git submodule update --init --recursive --depth 1
```

> `git clone` accepts a `--shallow-submodules` flag to achieve the same:

I updated the build commands like so:

```txt 
# amplify.yml

version: 0.1
frontend:
  phases:
    build:
      commands:
        - git submodule update --init --recursive --depth 1
        - hugo
  artifacts:
    baseDirectory: public
    files:
      - '**/*'
  cache:
    paths: []
```

## Building With an Arbitrary Version of Hugo

The next build failed with a new error: "this feature is not available in your current Hugo version".

```
# Executing command: hugo
Building sites …
ERROR 2020/03/24 08:55:10 error: failed to transform resource: TOCSS: failed to transform "css/main.scss" (text/x-scss): this feature is not available in your current Hugo version, see https://goo.gl/YMrWcn for more information
```

The [linked documentation](https://gohugo.io/troubleshooting/faq/#i-get-tocss--this-feature-is-not-available-in-your-current-hugo-version) explains that SCSS processing is available only in the extended version, which is not installed by default on AWS.

From their Dockerfile:

```txt 
## Install Hugo
RUN wget https://github.com/gohugoio/hugo/releases/download/v${VERSION_HUGO}/hugo_${VERSION_HUGO}_Linux-64bit.tar.gz && \
    tar -xf hugo_${VERSION_HUGO}_Linux-64bit.tar.gz hugo -C / && \
    mv /hugo /usr/bin/hugo && \
    rm -rf hugo_${VERSION_HUGO}_Linux-64bit.tar.gz
```

Amplify does [provide an option](https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html#setup-live-updates) to configure the value of the `VERSION_HUGO` environment variable, but it wasn't a viable option here.

The latest extended release archive looks like this:

```
hugo_extended_0.68.3_Linux-64bit.tar.gz
```

So setting `VERSION_HUGO` to `extended_0.68.3` would result in the correct file name. However, `VERSION_HUGO` is interpolated in two places:

```
.../download/v${VERSION_HUGO}/hugo_${VERSION_HUGO}_Linux-64bit.tar.gz
```

Setting it to `extended_0.68.3` would look for the archive under the nonexistent subdirectory, `vextended_0.68.3`:

```
.../download/vextended_0.68.3/hugo_extended_0.68.3_Linux-64bit.tar.gz
```

The real URL resembles the following:

```
.../download/v0.68.3/hugo_extended_0.68.3_Linux-64bit.tar.gz
```

I opted for the solution [described in the Hugo docs](https://gohugo.io/hosting-and-deployment/hosting-on-aws-amplify/#using-a-newer-version-of-hugo), which amounts to breaking the bash one-liner from the `RUN` step in the Dockerfile...

```
RUN wget https://github.com/gohugoio/hugo/releases/download/v${VERSION_HUGO}/hugo_${VERSION_HUGO}_Linux-64bit.tar.gz && \
    tar -xf hugo_${VERSION_HUGO}_Linux-64bit.tar.gz hugo -C / && \
    mv /hugo /usr/bin/hugo && \
    rm -rf hugo_${VERSION_HUGO}_Linux-64bit.tar.gz
```

...into a series of steps within the build configuration (build commands follow the same logic specified by `&&`, where the next command in sequence is only executed if the previous was successful).

```txt 
# amplify.yml

version: 0.1
frontend:
  phases:
    build:
      commands:
        - git submodule update --init --recursive --depth 1
        - wget https://github.com/gohugoio/hugo/releases/download/v0.68.3/hugo_extended_0.68.3_Linux-64bit.tar.gz
        - tar -xf hugo_extended_0.68.3_Linux-64bit.tar.gz hugo
        - mv hugo /usr/bin/hugo
        - rm -rf hugo_extended_0.68.3_Linux-64bit.tar.gz
        - hugo
  artifacts:
    baseDirectory: public
    files:
      - '**/*'
  cache:
    paths: []
```

> The one change was to extract `hugo` to the local directory, not the root directory (so, removing the `- C /` option from the `tar` command , and the leading slash in `mv /hugo`).
> 
> I'm not sure why (probably something to do with sandboxing or permissions), but extracting to the root directory did not work. The `mv` command failed because `/hugo` did not exist.

And that was it! The next build was a success. I'm now able to publish articles or deploy any other updates with a simple `git push` to the master branch.