Golang, GitLab & Private Repos
Published: 2023-10-24 01:54:05

Private Git repositories can pose challenges when integrating them into your Continuous Integration/Continuous Deployment (CI/CD) process, primarily because their restricted access complicates the incorporation of necessary credentials. This article details the difficulties we encountered in this regard and outlines the strategies we employed to overcome them.

In each of the recommended scenarios that follow, the principles are:
* export `GOPRIVATE=gitlab.com` so Go tools know where your private repositories are located
* set `git config` rewrite to use SSH instead of HTTPS for GOPRIVATE repos - the assumption here is that you access GitLab via SSH when building locally
* populate a `$HOME/.netrc` file for when Go tools use the curl library to access GOPRIVATE git repos

## GitLab CI Access Tokens & Variables

The techniques detailed below offer a means to facilitate the utilization of one project's resources within another during the build process. For more comprehensive information, please consult GitLab's official documentation.

If you encounter the following error while attempting to access a private repository, carefully review and implement the techniques described below:

```shell
git ls-remote -q origin in /go/pkg/mod/cache/vcs/ce35c7dd2f2e6b60868b3692964f5657730fb86df44930b66cbef124db489041: exit status 128:
	remote: The project you were looking for could not be found or you don't have permission to view it.
	fatal: repository 'https://gitlab.com/your-repo.git/' not found
```

### Opt#1: CI_JOB_TOKEN
This is the recommended solution as it leverages built-in credentials. The CI_JOB_TOKEN environment variable is configured to serve as an authentication token, enabling CI jobs to access Git repositories and artifacts. To activate this variable within your job stages, ensure that, for each project to be accessed via `go get`, the "**Token Access/Limit access to this project**" toggle is disabled in each project's CI/CD settings.

To make use of the token in your job stages, include the following global section in your `.gitlab-ci.yml` file:

```yaml
before_script:
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/".insteadOf https://gitlab.com/
  - go env -w GOPRIVATE=gitlab.com/${CI_PROJECT_NAMESPACE}
  - echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > $HOME/.netrc
```

### Opt#2: Personal access token
In this approach, the GitLab parent project should define the CI/CD variables CI_ACCESS_USER and CI_ACCESS_TOKEN. These variables should be populated by the user who generates a personal access token, subsequently storing both the token and the username in these variables, which are inherited by sub-projects.

It's worth noting that GitLab access tokens have a maximum lifespan of one year.

```yaml
before_script:
  - go env -w GOPRIVATE=gitlab.com/${CI_PROJECT_NAMESPACE}
  - echo -e "machine gitlab.com\nlogin ${CI_ACCESS_USER}\npassword ${CI_ACCESS_TOKEN}" > $HOME/.netrc
```

### Local settings
These settings are essential when working on projects locally, as they involve SSH-based access to GitLab.

```yaml
git config --global url."ssh://git@gitlab.com/".insteadOf "https://git@gitlab.com/"
export GOPRIVATE="gitlab.com"
```

To ensure `go get` functions correctly, even if you can use GitLab with SSH locally, you must obtain a GitLab personal access token and store it in the file `$HOME/.netrc` with the following format:

```text
machine gitlab.com
login <username>
password <your access token>
```

Make sure to export `GOPRIVATE=gitlab.com` before executing 'go get.'

## Docker usage
Transferring the $CI_JOB_TOKEN into a multi-stage Dockerfile for building in the first stage and copying the binary in the second stage has proven to be a challenging endeavor. Therefore, the following compromise is proposed which uses a single-stage Docker build:

* Get the CI compile stage to work with CI_JOB_TOKEN and generate the compiled binary as a stage artifact.
* Separate the Docker-related tasks into a subsequent stage that depends on the artifact produced in the compile stage. The Dockerfile should assume the binary's location in the same path where it was created during the compile stage.
* Ensure that the same Docker base image is used for both stages to prevent potential issues with architectures that may not support the built binary.

The provided `.gitlab-ci.yml` configuration exercises these principles:

```yaml
stages:
  - build
  - release

.before_script: &beforeScript
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/".insteadOf https://gitlab.com/
  - go env -w GOPRIVATE=gitlab.com/${CI_PROJECT_NAMESPACE}
  - echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > $HOME/.netrc

before_script:
  - *beforeScript

compile:
  image: amd64/golang:alpine
  stage: build
  before_script:
    - apk add --no-cache git build-base openssh upx
    - *beforeScript
  except:
    - main
  script:
    - make build
  artifacts:
    paths:
      - myapp

# automatically build the docker image after the new tag is created
build-image:
  image: amd64/docker:latest
  stage: release
  only:
    - tags
  dependencies:
    - # copy in artifacts from compile job in build stage
    - compile
  services:
    - docker:dind
  before_script:
    - apk add --no-cache git build-base openssh upx
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_AZURE_REGISTRY
  script:
    # get the version tag; CI_COMMIT_REF is branch or tag of commit
    - tag=$CI_COMMIT_REF_NAME
    - CI_REGISTRY_IMAGE="$CI_AZURE_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME"
    - docker build --pull -t "$CI_REGISTRY_IMAGE:${tag}" .
    - docker push "$CI_REGISTRY_IMAGE:${tag}"
```

## Troubleshooting
Below are further hints to figure out why you cannot pull your private repos during build. They are presented as a series of commands without any explanation. We hope they provide you with some level of comfort.

```shell
* ssh -T git@gitlab # responds with your username if validated
* go get -v -x gitlab.com/your-private-repo
* GIT_CURL_VERBOSE=2 GIT_TRACE=2  git ls-remote -q origin
* GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL git@gitlab.com 'git-upload-pack '\''/your-private-repo.git'\'''
```