A gopher’s journey to the center of container images
Tags: containers, go, docker, tutorial
Published: 2023-11-23 15:17:30

## Blissful past...

A couple of years ago, I would never have thought that I would get that interested in the underlying structure of containers, not to mention going into the journey of building one in Golang.

I was living the blissful life of an engineer who simply uses `podman pull` or `docker push`, creates `ContainerFile` or `Dockerfile`, then runs command lines to build images from such files... sitting back and watching the standard output list the layers being built, then pushed one by one with nice digests to the registry of my choosing, under the tag of my choosing.

## All changed when...

All this changed when I started contributing to [oc-mirror](https://github.com/openshift/oc-mirror), around a year ago. oc-mirror is a plugin of OpenShift's CLI, and targets disconnected clusters. It mirrors all images needed by such clusters in order to install and upgrade OpenShift as well as all its Day-2 operators from operator catalogs. 
Suddenly, the underground world of containers unraveled. 

Most of the logic of oc-mirror is about extracting metadata from images such as release images and operator catalog images, interpreting the contents of these images in order to determine the list of images that constitute a release or an operator, and later copy those images to an archive or to a partially disconnected registry.

Nevertheless, some of the activities also include building multi-arch images. This is the case for the graph image. Without going into the details of [what this image is useful for](https://docs.openshift.com/container-platform/4.14/updating/updating-restricted-network-cluster/restricted-network-update-osus.html), let's just say that the graph image is simply a UBI9 image, to which we copy some metadata in `/var/lib`, and whose `CMD` we modify, so that this image can become an init container for the disconnected cluster to use.

## Let's start building

In this article, we are roughly trying to build in Golang the equivalent of this very simple Containerfile:
```
FROM registry.access.redhat.com/ubi9/ubi:latest

RUN curl -L -o cincinnati-graph-data.tar.gz https://api.openshift.com/api/upgrades_info/graph-data

RUN mkdir -p /var/lib/cincinnati-graph-data && tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati-graph-data/ --no-overwrite-dir --no-same-owner

CMD ["/bin/bash", "-c" ,"exec cp -rp /var/lib/cincinnati-graph-data/* /var/lib/cincinnati/graph-data"]
```

I'll try to describe the three paths I explored to achieve this task. I'm aware these are probably not the only possibilities, and probably not always adapted to what your context is: 
* [Context A - having root capabilities: using `containers/buildah`](#contexta)
* Context B - no root capabilities: 
  * [Using `containers/buildah` in a secure namespace](#contextb1)
  * [Using `google/go-container-registry`](#contextb2)

### <a id="contexta"></a> Context A - having root capabilities: Using `containers/buildah`

For the task of building the graph image, my first idea was to rely on [buildah](https://github.com/containers/buildah).
In fact, our design was already heavily relying on [containers/image](https://github.com/containers/image) for all things regarding copying images from one registry to the other, or from one registry to an archive. The obvious choice was to use the same suite of modules in order to keep dependencies to a minimum. 

My implementation effort was greatly guided by Buildah's tutorial [4-Include in your build tool](https://github.com/containers/buildah/blob/main/docs/tutorials/04-include-in-your-build-tool.md).

I'm assuming here that the golang binary that I'm building can have root privileges. If this is not your context, and you'd like to run this binary as non-root, you will need a special setup of the `builder` (which you can find in the [next section](#contextb1)).

With the assumption that root privileges are available, the implementation is fairly simple. As you'll see below, each instruction of the Containerfile has an equivalent method in the `builder` interface.

I encountered one small gotcha: Any files or folders that you want to copy/add to the image need to be in the current working directory. 

For our development, this was a little incovenience: why would someone using the tool in his home directory suddenly end up with Openshift's upgrade graph metadata poluting his home?! But this could easily be worked around by cleaning up in a `defer` statement when the `builder` was done (regardless of the build outcome: success or failure).

All the code is available [here](https://github.com/sherine-k/graph-data-image-builder/tree/main). 

Now let's break down what needs to be done:

#### Initializing the builder - FROM instruction

I want to initialize the `builder` on ubi9 image. This is passed in the `BuilderOptions` like this:
```
const(
	graphBaseImage              string = "registry.access.redhat.com/ubi9/ubi:latest"
)
// ... truncated code
builderOpts := buildah.BuilderOptions{
  FromImage:    graphBaseImage,
  Capabilities: capabilitiesForRoot,
  Logger:       logger,
}
builder, err := buildah.NewBuilder(context.TODO(), buildStore, builderOpts)
```
#### Adding a layer - ADD instruction

Given that I have prepared the files that need to be copied to the image in `graphDataUntarFolder`, I can add the content of the whole folder using `builder.Add`. The `AddAndCopyOptions` can help set the userID and groupID owning these files and folders inside the container. 
```
	addOptions := buildah.AddAndCopyOptions{Chown: "0:0", PreserveOwnership: false}
	addErr := builder.Add(graphDataDir, false, addOptions, graphDataUntarFolder)
```
#### Updating the command - CMD instruction

Next, we want to setup the command of the container image. This is rather straightforward:
```
	builder.SetCmd([]string{"/bin/bash", "-c", fmt.Sprintf("exec cp -rp %s/* %s", graphDataDir, graphDataMountPath)})
```

#### Building and pushing

It's now time to build the image and push it. By default, you can push to the store by first preparing the image reference like so:
```
imageRef, err := is.Transport.ParseStoreReference(buildStore, "docker.io/myusername/my-image")
```
But in my case, I opted for pushing it directly to the destination registry, like so:
```
	imageRef, err := alltransports.ParseImageName("docker://localhost:7000/" + graphImageName)
  // ... truncated code
	imageId, _, _, err := builder.Commit(context.TODO(), imageRef, buildah.CommitOptions{})
```

### <a id="contextb1"></a>Context B - using `buildah` as non root

oc-mirror being a CLI plugin, it should not require any extra root permissions in order to build images. 

Buildah provides a way to run as non-root. But before we delve into that, a small parenthesis on the configuration of the store that Buildah uses:

#### Store defaults
Buildah relies on a build store for keeping track of layers, images pulled, built, etc. For setting up the build store, I simply used all the default setups available in the `buildah` module, like so:
```
	logger := logrus.New()
	logger.Level = logrus.DebugLevel
	buildStoreOptions, err := storage.DefaultStoreOptionsAutoDetectUID()
  // ... truncated code
	conf, err := config.Default()
  // ... truncated code
	capabilitiesForRoot, err := conf.Capabilities("root", nil, nil)
  // ... truncated code
	buildStore, err := storage.GetStore(buildStoreOptions)
	// ... truncated code
	defer buildStore.Shutdown(false)
	builderOpts := buildah.BuilderOptions{
		FromImage:    graphBaseImage,
		Capabilities: capabilitiesForRoot,
		Logger:       logger,
	}
	builder, err := buildah.NewBuilder(context.TODO(), buildStore, builderOpts)
  // ... truncated code
```

#### Setup for non-root execution

In order to integrate the `buildah` module to your golang product without root privileges, buildah's recommendation is to pause the execution of the go binary, create a user namespace where it could be root, and re-execute the binary in that user namespace.

This is achieved by adding the following lines in `main.go`, as early as you can in the `main` function:
```
if buildah.InitReexec() {
  return
}
unshare.MaybeReexecUsingUserNamespace(false)
```

This has to be added in the `main` function: you have to keep in mind that the execution will restart from the beginning, so any initializations will be done a second time. 

##### Impacts on debugging

Re-executing has a few impacts on the way we debug our code: 
This modifies the debugging process: In order to debug, I had to launch dlv debugger in a user namespace: 
```
podman unshare dlv debug --headless --listen=:43987 main.go 
```
PS: if you need to pass arguments to `main`, you can add ` -- ` to the command above, then append any arguments you have.

Once the command above is triggered, it is possible to use delve to debug (either using dlv directly or attaching to it with a client).

If you use VSCode, it is possible to attach it to the dlv process running in the background. This is achieved by adding the following code to the configurations[] inside of the launch.json:
```
{
    "name": "Attach Package",
    "type": "go",
    "debugAdapter": "dlv-dap",
    "request": "attach",
    "mode": "remote",
    "host": "localhost",
    "port": 43987,
},
{
    "name": "Attach Tests",
    "type": "go",
    "debugAdapter": "dlv-dap",
    "request": "attach",
    "mode": "remote",
    "host": "localhost",
    "port": 43987,
}
```

##### Impacts on users

Finally, for the use cases where our binary must run in a container, or in a pod on a Kubernetes cluster, it is important to setup securityContext and to list all the capabilities necessary to be able to run the binary inside the container. Among these capabilities, you need to include `CAP_SETGID` and `CAP_SETUID`. Other capabilities might as well be needed. 

#### Full code

[graph-data-image-builder](https://github.com/sherine-k/graph-data-image-builder/tree/main)

### <a id="contextb2"></a>Context B - Using `go-containerregistry` as non-root

I also explored another module, [go-containerregistry](https://github.com/google/go-containerregistry), in order to build images without root privileges. The approach is completely different, and we can manipulate each component of the container image separately. This can present an advantage, if you're looking for a way to fine tune things.

#### Preparing for use of `go-container-registry`

In order to start using the `remote` package of `go-container-registry` to pull/push images, you need to set :

* `nameOptions`: `StrictValidation` vs `WeakValidation`, and the possibility for default registries to be used while referring to container images
* `remoteOptions`: which group all configurations related to pulling and pushing images, such as:
  * connection proxies, timeouts, keepAlives, use of http2 or http1.1
  * configuration files containing credentials for registries 
  * TLS verification explicit disabling (if needed)

```
	nameOptions := []name.Option{
		name.StrictValidation,
	}
	remoteOptions := []remote.Option{
		remote.WithAuthFromKeychain(authn.DefaultKeychain), // this will try to find .docker/config first, $XDG_RUNTIME_DIR/containers/auth.json second
		remote.WithContext(context.TODO()),
		// doesn't seem possible to use registries.conf here.
	}
```

#### Pulling the origin image

Each image we want to build needs to be copied to a folder of your choosing on local disk. That folder (`layoutDir`) will contain the image layout, with any manifest-list, oci index, manifest, config, and layers...

This is achieved by using [`remote`](https://github.com/google/go-containerregistry/pkg/v1/remote) and [`layout`](https://github.com/google/go-containerregistry/pkg/v1/layout) like so:

```
  imgRef := "registry.access.redhat.com/ubi9/ubi:latest"
	ref, err := name.ParseReference(imgRef, b.NameOpts...)
	if err != nil {
		return "nil", err
	}
	idx, err := remote.Index(ref, b.RemoteOpts...)
	if err != nil {
		return "", err
	}
	layoutPath:= layout.Write(layoutDir, idx)
	return layoutPath
```

#### Creating a layer 

Adding a layer from a tar can be achieved very easily using [`tarball`](https://github.com/google/go-containerregistry/pkg/v1/tarball).

Given that outputFile is a string containing the path to a tar file, `LayerFromFile` automatically untars the tar file contents and constructs a layer from that.

`outputFile` could be anywhere on the filesystem. There are no restrictions to it being saved to the working directory like in `buildah`.

```
  layerOptions := []tarball.LayerOption{}
  layer, err := tarball.LayerFromFile(outputFile, layerOptions...)
  if err != nil {
    return nil, err
  }
```

#### Updating the command

For changing anything inside an image, [`mutate`](https://github.com/google/go-containerregistry/pkg/v1/mutate) is needed.

This is slightly more complicated than what this snippet shows, due to the fact that an image might be a dockerv2-2 manifest list or oci index, itself containing several manifests (image for a specific architecture and OS). 

In order to modify the command for the multi-arch image, we'd need to update the config of each of the underlying manifests.

But let's keep that out for now, and focus on how to modify the command for a single manifest. The full code is [here](https://github.com/openshift/oc-mirror/blob/1c8f538897c88011c51ab53ea5073547521f0676/v2/pkg/imagebuilder/builder.go#L114).

```
// layoutPath is the result of layout.Write from the previous snippet
idx, err := layoutPath.ImageIndex()
if err != nil {
	return err
}
idxManifest, err := idx.IndexManifest()
if err != nil {
	return err
}
manifest := idxManifest.Manifests[0]
currentHash := *manifest.Digest.DeepCopy()
img, err := idx.Image(currentHash)
cfg, err := img.ConfigFile()
if err != nil {
  return nil, err
}
cfg.Config.Cmd = cmd
img, err = mutate.Config(img, cfg.Config)
if err != nil {
  return nil, err
}
```

#### Building and pushing
##### Adding the layer
Same as for the modification of the command, adding a layer is achieved with [`mutate`](https://github.com/google/go-containerregistry/pkg/v1/mutate).

```
// `img` is the single arch image from the index. We get it by calling `idx.Image(currentHash)` like in the previous snippet
// `layer` is the 
additions := make([]mutate.Addendum, 0, len(layers))
for _, layer := range layers {
  additions = append(additions, mutate.Addendum{Layer: layer, MediaType: mt})
}
img, err = mutate.Append(img, additions...)
if err != nil {
  return nil, err
  }
```

##### Building new manifests and index
Once a layer is added, or a Config modified, the manifest of the image should be updated. To be more exact, we need to remove the old manifest from the index, and add a new one.

This is done by creating a new descriptor for the `img` that was updated in previous snippets

```
desc, err := partial.Descriptor(img)
if err != nil {
	return nil, err
}
```

Next, we need to update the image index, by replacing the descriptor:

```
add := mutate.IndexAddendum{
	Add:        img,
	Descriptor: *desc,
}
modifiedIndex := mutate.AppendManifests(mutate.RemoveManifests(idx, match.Digests(currentHash)), add)
resultIdx = modifiedIndex
```

#### Full code

* https://github.com/openshift/oc-mirror/blob/1c8f538897c88011c51ab53ea5073547521f0676/v2/pkg/release/graph.go#L17
* https://github.com/openshift/oc-mirror/blob/1c8f538897c88011c51ab53ea5073547521f0676/v2/pkg/imagebuilder/builder.go#L114

## Conclusion

Using `buildah` is much more simple: out of the box, it has support for multi-arch image building, as well as support for [registries.conf](https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md), which was a requirement for our product.

Furthermore, and like shown in this blog entry, each Containerfile instruction maps to a builder method. This makes the builder very easy to use.

`go-containerregistry` has all the necessary interfaces and methods to manipulate all the building blocks of container images, regardless of their format (dockerv2-1, dockerv2-2 or oci). It is probably worth investigating whether another golang module builds on top of `go-containerregistry` and provides an experience closer to that of a `builder`, abstracting away all the lower level changes, and allowing for building multi-arch images easily. But that's a subject for a next blog...