How to use CI to update Git submodules 🧩
Tags: devops, github, git, programming
Published: 2021-05-07 11:25:54

---
title: How to use CI to update Git submodules 🧩
published: true
date: 2021-04-22 23:15:00 UTC
tags: devops, github, git, programming
canonical_url: https://www.benjaminrancourt.ca/how-to-use-ci-to-update-git-submodules
cover_image: https://www.benjaminrancourt.ca/images/how-to-use-ci-to-update-git-submodules.jpg
---

If you are using **[Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules)** in your project, you might want to keep them up to date so that their latest version is available. Unfortunately, by design, Git keeps track of the submodules using their **commit SHA** (ex. `eb41d76`) and it must be committed to your repository.

And it seems their is not a way yet to always use the latest submodule commit without updating their reference with a command like `git submodule update --remote`... 😐

It is not really convenient when you want to automate your workflow and have to manually checkout the code to update the submodule SHA, right? 🤔

Imagine if you could only checkout your project containing submodules and you have the guarantee that they are at their latest version...😌

I am aware that sometimes, you do not want that a submodule to be able to introduce a breaking change, but if you are 99.99% sure that it cannot break your project, would not it be fun to forget that you have to update their references to use the latest changes? 🤭

It turned out to be not as simple as I would have like, but I managed to make it work. 😇

## Solutions

To simplify my explanation, let's says we have two projects:  `projectA` and `projectB`, and `projectB` has `projectA` as one of its submodules. 😬

### #1 - Update the submodules from within

Do you remember the `.git:push` job I wrote in my post [How to push to a Git repository from a GitLab CI pipeline](https://www.benjaminrancourt.ca/how-to-push-to-a-git-repository-from-a-gitlab-ci-pipeline)? Well, I have found another case to use it! 😅

```yaml
update:submodule:
  extends: .git:rsync
  rules:
    - if: $UPDATE_SUBMODULE == "true"
  script:
    - cd "${CI_COMMIT_SHA}"
    - git submodule update --init --recursive
    - git submodule update --remote
    - git status
  stage: install

```
<figcaption>Job used in <code>projectB</code> to update all its submodules</figcaption>

Since I did not want all other jobs to run when I was running this job, I add the following code to other jobs:

```yaml
  rules:
    - if: $UPDATE_SUBMODULE == "true"
      when: never

```
<figcaption>Do not run these jobs if the <code>$UPDATE_SUBMODULE</code> is true</figcaption>

By using a [**pipeline schedule**](https://docs.gitlab.com/ee/ci/pipelines/schedules.html) with the `$UPDATE_SUBMODULE` set, this solution was working. My submodules have been updated according to the interval I wanted.

![](https://www.benjaminrancourt.ca/images/image.jpg)
<figcaption>he first pipeline was executed to update a submodule, while the second was skipped because it had the <code>-o ci.skip</code> Git flag</figcaption>

![](https://www.benjaminrancourt.ca/images/image-1.jpg)
<figcaption>A commit that updates a subdirectory to a new SHA</figcaption>

Unfortunately, this solution requires that the `projectB` schedule run some times after an update to the `projectA`, otherwise the latest commit of the `projectA` will have to wait for the next schedule to be taken into account...

Since I did not wanted to introduce any delay and I did not want to depend on the condition that my `projectA` pipeline be completed in less than X minutes, I keep looking for a better solution. 🔎

### #2 - Trigger the pipeline of `projectB` from `projectA`

If we want the `projectB` to know when the `projectA` has been updated, one solution is for `projectA` to notify the `projectB`.

Fortunately, GitLab offers a specific keyword to launch a downstream pipeline: [**trigger**](https://docs.gitlab.com/ee/ci/yaml/#trigger). It is pretty straightforward, because you just need to configure two more keywords:

- **project** : path of the second project
- **branch** : branch to update of the second project

like in the example below.

```yaml
update:references:
  trigger:
    branch: '${CI_DEFAULT_BRANCH}'
    project: ranb2002/projectB
    strategy: depend
  stage: trigger
  variables:
    UPDATE_SUBMODULE: 'true'
```
<figcaption>Use of the trigger keyword in <code>projectA</code> GitLab CI file</figcaption>

Since I do not want to trigger the entire pipeline, I am also able to pass variables from one pipeline to another, like the `UPDATE_SUBMODULE` variable used in the previous solution.

There is also another keyword, **strategy** , that when set to `depends`, that [links the repositories](https://docs.gitlab.com/ee/ci/yaml/#linking-pipelines-with-triggerstrategy) togethers. This can be useful if you want to know whether or not the second pipeline was successful from the upstream repository.

![](https://www.benjaminrancourt.ca/images/image-2.jpg)<figcaption>Pipeline of <code>projectA</code> (upstream), with the pipeline of <code>projectB</code> (downstream)</figcaption>

### #3 - Use the GitLab API to trigger the downstream pipeline

I can not remember what I did wrong, but the previous solution did not work for me as I thought it would. I finally managed to fix it, but that was after finding another solution by using a [specific GitLab API endpoint](https://docs.gitlab.com/ee/ci/triggers/#triggering-a-pipeline):

```yaml
trigger:downstream:
  script:
    - curl
      --request POST
      --form "ref=${CI_DEFAULT_BRANCH}"
      --form "token=${CI_JOB_TOKEN}"
      --form "variables[UPDATE_SUBMODULE]=true"
      "${CI_SERVER_URL}/api/v4/projects/${DOWNSTREAM_PROJECT_ID}/trigger/pipeline"
  stage: update

```
<figcaption>Use of the GitLab API to trigger the downstream pipeline</figcaption>

It is a little more verbose than the previous example, but you just need to provide the ID of the project to update (`DOWNSTREAM_PROJECT_ID`) as the [other variables are **automatically predefined**](https://www.benjaminrancourt.ca/gitlab-ci-predefined-variables) by GitLab CI.

### #4 - Use the GitLab API to update directly a submodule

While searching on the Internet, I came across another GitLab API endpoint that did exactly what I wanted, without the need to adapt two pipelines: the **[Repository submodules API](https://docs.gitlab.com/ee/api/repository_submodules.html)**!

> As I finally choose this way to update my submodules, I improved the example from the documentation to meet my criteria, so be careful, as the example below may look complicated.

For a much simpler example, I suggest you read the [official documentation on the subject](https://docs.gitlab.com/ee/api/repository_submodules.html). 🛡️

```yaml
trigger:
  image: curlimages/curl:7.76.1
  script:
    # Put the private-token with a colon into a variable to escape the "Linefeed-Limbo" (https://stackoverflow.com/a/51187502)
    - PRIVATE_TOKEN="PRIVATE-TOKEN:"

    # Get the current datetime (in local time with the TZ variable)
    - CURRENT_DATE="$(date +'%F %T')"

    # Add it to each commit message of the repository (and make sure there is no carriage returns)
    - MESSAGE="${COMMIT_MESSAGE:-$CI_COMMIT_MESSAGE} (${CURRENT_DATE})"
    - MESSAGE=$(echo "${MESSAGE}"|tr -d '\n')

    # Use the Repository submodules API to update the submodule reference in the repository
    - curl
      --data "branch=${BRANCH:-$CI_DEFAULT_BRANCH}&commit_sha=${COMMIT_SHA:-$CI_COMMIT_SHA}&commit_message=${MESSAGE}"
      --header "${PRIVATE_TOKEN} ${GITLAB_TOKEN}"
      --request PUT
      "${CI_SERVER_URL}/api/v4/projects/${PROJECT_ID}/repository/submodules/${SUBMODULE_PATH}"
  stage: trigger
  rules:
    - changes:
        - ghost/
        - images/
  variables:
    PROJECT_ID: '20554554'
    SUBMODULE_PATH: 'ghost'
    TZ: ':America/Toronto'

```

Now, I am sure sure you want an explication of the above code, right? 😅

First, the heart of the solution is the `curl` command, where I call the GitLab endpoint. Let's break down all of the variables used here:

- `SUBMODULE_PATH`: which represents the name of the submodule exactly as it was commited into the `projectB` repository
- `PROJECT_ID`: the `projectB` ID
- `CI_SERVER_URL`: the URL of GitLab
- `GITLAB_TOKEN`: a personal GitLab token, with write permissions on the API
- `MESSAGE`: the commit message I want (more explanations below)

There is also a lot of line just to set the `MESSAGE` variable, because I want to add the current date to it (in my local time zone). If the `COMMIT_MESSAGE` is not provided, it defaults to `CI_COMMIT_MESSAGE` which is the latest commit message of the `projectA` repository.

### Conclusion

You have now four ways to update a Git submodule in a GitLab CI pipeline! Which one will you choose? 😉