Using multiple repositories in your CI builds
Tags: programming
Published: 2022-08-16 21:38:40

[Djinn CI](https://about.djinn-ci.com) makes working with multiple repositoriesin a build simple via the [sources](https://docs.djinn-ci.com/user/manifest/#sources)parameter in the build manifest. This allows you to specify multiple Git respositories to clone into your build environment. Each source would be a URL that could be cloned via `git clone`. With most CI platforms, a build's manifest is typically tied to the source code repository itself. With Djinn CI, whilst you can have a build manifest in a source code repository, the CI server itself doesn't really have an understanding of that repository. Instead, it simply looks at the sources in the manifest that is specified, and clones each of them into the build environment.

## Defining multiple sources in a manifest

Like with the [previous][0] post, we're going to use [djinn-ci/imgsrv][1] as an example of using multiple sources in a build manifest. If we look at the top of the manifest file, we will see that it requires three repositories to build. These are, the source code for djinn-ci/imgsrv itself, [golang/tools][2], and [valyala/quicktemplate][3], defined like so,

[0]: https://blog.djinn-ci.com/showcase/2022/08/06/running-your-ci-builds-without-the-server
[1]: https://github.com/djinn-ci/imgsrv
[2]: https://github.com/golang/tools
[3]: https://github.com/valyala/quicktemplate

    sources:
    - https://github.com/djinn-ci/imgsrv.git
    - https://github.com/golang/tools
    - https://github.com/valyala/quicktemplate

during execution of the manifest, all of the above source repositories will be cloned. If submitting this through the user interface for the build server to run, then each of these will have its own `clone.n` job associated with it, where `n` is the number counting up from `1`.

## SSH keys and private repositories

As previously mentioned, the URLs specified in the `sources` parameter will be given to `git clone` for cloning in the build environment. This means you can use the SSH format when specifying a source URL,

    git@github.com:org/private-repo

this would be preferable in instances where you would want to clone from a private repository. This would however require creating an [SSH key][4] to use as a sort of deployment key when cloning into the environment.

[4]: https://docs.djinn-ci.com/user/keys/

Let's consider the above example, when cloning from `github.com`. We would create a [deploy key][5] for that repository, and upload it to Djinn CI. When doing so, we will be able to specify some custom SSH configuration to associate with that key, this is what will allow us to make use of it when it is cloned in the build,

[5]: https://docs.github.com/en/developers/overview/managing-deploy-keys

    Host github.com
        User git
        IdentityFile /root/.ssh/id_deploy_private_repo

with the key in place, along with the config, we would then be able to use the repository `org/private-repo` in our builds. However, an issue can arise here, assume we want to clone from another private repository, we could add the same deployment key there, but this might not be preferable if you're concerned about the security implications. So instead, you would create another key for that repository. With two deploy keys in place, one for each repository, this raises the question, how would you associate each key with each repository?

This could be done by making use of the `.ssh_config` file format and `Host` matching. Assume the two private repositories are `private-repo` and `acme-repo`, both exist under `org` on `github.com`. And they each have a deployment key named `id_deploy_private_repo` and `id_deploy_acme_repo`. Then we could give each key the following SSH configuration,

    Host private-repo.github.com
        User git
        IdentityFile /root/.ssh/id_deploy_private_repo

    Host acme-repo.github.com
        User git
        IdentityFile /root/.ssh/id_deploy_acme_repo

then in the build manifest we could give the following sources,

    sources:
    - git@private-repo.github.com:org/private-repo
    - git@acme-repo.github.com:org/acme-repo

it's a bit of a hacky work around, but it works if you would prefer to use different deploy keys for different private repositories.

### More information

To learn more about Djinn CI, visit the [about page][6] which gives an overview of the features of the platform, and be sure to checkout the [user documentation][7] too.

[6]: https://about.djinn-ci.com
[7]: https://docs.djinn-ci.com/user