How we do automated dependency updates in 2020
Published: 2020-07-30 13:27:17

As the number of projects grew in our company, it became difficult to update manually our dependencies. Early 2019, we looked for a solution to automate this process. We tried a few but ended up being overwhelmed by a flood of PRs and too many branches clogging up our repositories, despite the fact that, in practice, we would always accept pull requests (PRs), but it quickly became a hassle more than a help.

So, we decided to write our own dependency update tool with these requirements in mind:
1. dependencies should be updated on the same branch, one after the other, and in the end, only commits that passed CI should be merged in
1. there's no need for PRs: if the bot can watch update commits run through our CI, it can tell whether it's safe to merge the update branch to the main branch
1. still, we should be able to configure what to do after an update via simple plugins. Send a Slack message, an email, open an issue... you name it. 
1. package managers support should be made with plugins so we can add as many as we want
1. we should be able to connect multiple Git servers: we host some projects on our internal Gitlab and others on Github, and we want to view the updates in a single place
1. all our secrets and private registries are already configured in our CI, we should not have to reconfigure that anywhere else

So we wrote it, we've used it, we've named it [Pmbot](https://pmbot.io), and we'd like to share it with you ! It's free, and self hosted.

Now, get yourself a cup of coffee, and let's see how to get started !

## Installing Pmbot locally with Docker compose

We'll use the following `docker-compose.yml`:

{% gist https://gist.github.com/gempain/94c091a7781222913627ee8db8eebcd8 %}

Note that `APP_UI_URL` and `PMBOT_API_URL` are not strictly mandatory, but it'll make your life easier when copy/pasting code snippets from the Pmbot UI. Use the IP or host name of your machine

Now, run `docker-compose up -d`, and browse `http://192.168.1.54:9118`. You should get to the setup screen:

![Pmbot setup screen](https://dev-to-uploads.s3.amazonaws.com/i/npejway2z48hf6sq06bh.png)

Just fill in the registration form and click **Create account**.

## Connect Pmbot to your Git server

We'll start off by selecting the type of Git server we're adding. In this case, we're using a self hosted Gitlab instance, but you can also connect to Github, and we'll be adding support for more Git servers in the coming weeks.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/dx1gc7je8jd8kirjgih4.png)

We'll now add Pmbot as [a new OAuth2 application](https://docs.gitlab.com/ee/integration/oauth_provider.html) in Gitlab so we can get our **client ID** and **client secret**, which we can use to complete the Git provider **OAuth configuration**:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/m4og3ilafvt5nsx1q1t6.png)

Now, just click **Run test**. Pmbot will check the connection and synchronize your projects.

Finally, click **Add Git provider**.

## Connect Pmbot to your CI server

We're using Gitlab CI in this case, but you can use Drone (we wrote [a blog post](https://blog.pmbot.io/how-to-use-pmbot-with-drone-ci/) on this topic), and we're planning to add compatibility with more CI providers in the coming weeks.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/oqquv2330hg8th6dxtt6.png)

Now, just click **Add CI provider**, and you'll now be in your dashboard.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jbzmn9qcccqerchtero0.png)

## Configure updates for a Git repository

We'll add our first project. Click **Add first project**, and then select the Git repository to update.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9kd77w7i93ofwb96rcn9.png)

Then, we'll tell Pmbot to update this project on our CI server, and we'll configure the settings so Pmbot can trigger jobs on our CI.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/uxylhsug2ruezitmbmjh.png)

We'll tell Pmbot to update the **master** branch **every day** and then click **Save**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/en3otq96gm6tg1128ice.png)

Finally, we'll click **Enable project**. The UI now shows us what the next steps are.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o3u8yaiqzze91r4otxbf.png)

## Add `.pmbot.yml`

Pmbot uses a `.pmbot.yml` to configure package managers and actions. We'll add the following one to our Git repository:

{% gist https://gist.github.com/gempain/cee8d297301e404102dbcedfbc8aebfc %}

Here, we'll be updating our Npm dependencies with the official [**npm** plugin](https://docs.pmbot.io/package-managers/npm#npm), but you could use any of the existing [package manager plugins](https://docs.pmbot.io/plugins/plugin-list#package-managers) or even [write your own](https://docs.pmbot.io/plugins/custom-plugin#custom-plugins). At the moment, we've written plugins for [Npm](https://docs.pmbot.io/package-managers/npm), [Go](https://docs.pmbot.io/package-managers/go) and a very basic one for [Maven](https://docs.pmbot.io/package-managers/maven), but we'll add more in the coming weeks.

We're also using an **action plugin** named [auto-merge](https://docs.pmbot.io/actions/auto-merge) and we're telling Pmbot to execute it when a project update is **successful** or **partially** successful. This plugin is pure greatness: it automatically merges the update branch to the branch from which the update started, and it's fully configurable, so checkout [our docs](https://docs.pmbot.io/actions/auto-merge) for more info. Checkout the [complete list](https://docs.pmbot.io/plugins/plugin-list#actions) of action plugins for more goodness, we've already written cool things like sending a Slack message or calling a webhook. Best of all, you can [write your own action plugins](https://docs.pmbot.io/plugins/custom-plugin#creating-a-custom-plugin) with just a few lines of code !

## Update your CI config file

We're using Gitlab CI, but you can find instructions for using [Drone CI](https://drone.io/) either in [our docs](https://docs.pmbot.io/ci-providers/drone-ci#ci-config-file) or on [a blog post we wrote about it](https://blog.pmbot.io/how-to-use-pmbot-with-drone-ci/).

We'll update our `.gitlab-ci.yml`:
- add an `update` job which runs the `pmbot` CLI
- add jobs for notifying Pmbot of CI output
- conditionally enable/disable jobs so Pmbot can decide which ones to run

{% gist https://gist.github.com/gempain/5c5741d3682d4f1b0edc442b05677000 %}

Values for `PMBOT_URL`, `PMBOT_PROJECT_ID` and `PROJECT_TOKEN` are displayed in your Pmbot UI, so this is specific to your Pmbot project.

Also, notice how we control which jobs to run by using an environment variable. That's how Pmbot uses your CI to run updates inside single jobs.

## Allow Pmbot to commit and push

Finally, we'll have to allow the `pmbot` CLI to commit and push to our repository. Generate an SSH keypair with

```bash
ssh-keygen -b 2048 -t rsa -f /tmp/sshkey -q -N ""
```

or click **Generate for me** in the UI:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/vevjkeh01ymlg1dcyj2b.png)

You'll need to add the **public key** as a **deploy key** in your Gitlab project, and provide the `pmbot` CLI with the **private key**. We suggest setting a CI secret environment variable of type **file** named `PMBOT_SSH_PRIVATE_KEY` which our CLI will pick up automatically, or you can use the [`--ssh-private-key`](https://docs.pmbot.io/core/pmbot-cli#ssh-private-key) option.

*Tip: any option of the `pmbot` CLI can be provided using environment variables (uppercase, underscored, starting with `PMBOT_`). More info in [our docs](https://docs.pmbot.io/core/pmbot-cli#pmbot-cli-reference).*

Done !

Now, just click **Run first update**:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ku937ok029dotngkx92o.png)

The update will start. It may take a few minutes, as every time Pmbot updates a package, it waits for the commit to run through your CI.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hcdh2xyw2tfn0926gsyy.png)

When the update is done, you'll see all the actions executed.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/1pokfmz7sdua77kovwyg.png)

Check your Git history, you'll see a new commit from your beloved Pmbot.

*Tip: clicking any commit id will open the corresponding page in your Git server, and clicking any pipeline status bubble will open it in your CI platform.*

## Conclusion

Pmbot takes a relatively new approach to automated dependency updates. We've tried to make it follow the flow that we used to manually update the dependencies of our projects, and we hope you'll like the way we've made it work.

We're aware that the project setup process is a bit tedious at first, and we'll work on making it smoother and reducing the number of manual steps, but know that once you've added your first project, adding the next ones takes a lot less time as you can copy/paste most of the code. You can also leverage global CI variables to reduce configuration of the `pmbot` CLI.

We're looking forward to having feedback on this tool, and we hope that it'll change your coding life as much as it changed ours.

Thanks a lot, and happy updates !

Read further:
- [official website](https://pmbot.io)
- [documentation](https://docs.pmbot.io)
- [blog](https://blog.pmbot.io)
- [bugs and feature requests](https://github.com/pmbot-io/issues)
- [follow us on twitter](https://twitter.com/pmbotio)