Hey! You broke our pipeline!
Tags: azuredevopspipelines, azuredevops
Published: 2023-03-02 10:50:42

---
title: Hey! You broke our pipeline!
published: true
date: 2023-01-12 17:21:53 UTC
tags: AzureDevOpsPipelines,AzureDevOps
canonical_url: https://jessehouwing.net/azure-pipelines-you-broke-our-pipeline/
---

 ![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/212787247_d716ebad02_k.jpg)

One of the advantages of Azure Pipelines is that it automatically updates your tasks to the latest minor version. That way you don't have anything to stay up-to-date. But this also has disadvantages. Both Microsoft and Extension Authors can accidentally break your pipelines.

I've personally shipped a version of my [Variable Toolbox](https://marketplace.visualstudio.com/items?itemName=jessehouwing.jessehouwing-vsts-variable-tasks) extension that broke 10.000 developers at a single corporation that relied on my pipeline tasks.

And yesterday [Microsoft had a world-wide outage due to the npmAuthenticate@0 task suffering from an unexpected breaking change](https://github.com/microsoft/azure-pipelines-tasks/issues/17584).

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-9.png)

Most people don't know what to do when something like this happens. But luckily there are a few things you can do to fix thing yourself when you find yourself in this situation.

## Pin to an older version

If you're using YAML pipelines, then the simplest way to get unblocked, is to [**pin your workflow to a specific version of the task**](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/tasks?view=azure-devops&tabs=yaml#task-versions). This is one of the big advantages of YAML based pipelines. By default Azure Pipelines will reference a task by its major version `@0` in the case of the `npmAuthenticate` task, but you can specify `@0.208.1` or any other previous version instead:

```
# Broken
- step: npmAuthenticate@0
  inputs:
    workingDirectory: $(Build.SourcesDirectory)
    
# Fixed
- step: npmAuthenticate@0.208.1
  inputs:
    workingDirectory: $(Build.SourcesDirectory)
```

The easiest way to find the version to enter here is to look at the last pipeline that succeeded, the **Initialize Job** step will list the exact versions used:

```
2022-12-23T09:05:03.7682146Z ##[section]Starting: Initialize job
2022-12-23T09:05:03.7684106Z Current agent version: '2.213.2'
2022-12-23T09:05:03.7719717Z Current image version: '20221215.2'
2022-12-23T09:05:03.8121128Z Download all required tasks.
...
2022-12-23T09:05:03.8228407Z Downloading task: npmAuthenticate (0.208.1)
2022-12-23T09:05:06.6642168Z ##[section]Finishing: Initialize job
```

## Overwrite the broken task

But if you're using classic build and release pipelines (UI based), then you can't pin to a minor version. In that case you'll need to upload a version of the task that works. You can use `tfx-cli` to do that:

```
TFS Cross Platform Command Line Interface v0.12.0
Copyright Microsoft Corporation

Syntax:
tfx build tasks upload --arg1 arg1val1 arg1val2[...] --arg2 arg2val1 arg2val2[...]

Command: upload
Upload a Build Task.

Arguments:
  --task-path Local path to a Build Task.
  --task-zip-path Local path to an already zipped task
  --overwrite Overwrite existing Build Task.
```

But first you'll need to get a copy of the task that works. Your Azure DevOps instance already has a copy of that task, you just need to know where to find it:

```
$pat = "{{AZURE DEVOPS PAT TOKEN}}"
$org = "{{AZURE DEVOPS ORG NAME}}"
$url = "https://dev.azure.com/$org"
$header = @{authorization = "Basic $([Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(".:$pat")))"}

$taskName = "npmAuthenticate"
$taskid = "ad884ca2-732e-4b85-b2d3-ed71bcbd2788"
$taskversion = "0.208.1"
$taskZip = "$taskName.$taskid-$taskversion.zip"

Invoke-WebRequest -Uri "$url/_apis/distributedtask/tasks/$taskid/$taskversion" -OutFile "$taskZip" -Headers $header
```

Or you can [grab a copy of an older version from my azure-pipelines-tasks-zips repository](https://github.com/jessehouwing/azure-pipelines-tasks-zips/releases):

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-10.png)

Now in order to "fix" the broken version we'll have to overwrite it. To do so we're going to patch the older working version with the broken version's version number.

First extract the zip file, then update the `task.json` file and set the version number to the one you want to overwrite (in our case `0.214.1`):

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-11.png)

Put the `task.json` back into the zip file.

We're now ready to upload the task. We can use `tfx-cli` to do that.

We first need to delete the existing task using `tfx build task delete --task-id` (**warning this will delete all older versions)**:

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-13.png)

And then we can upload the patched task `tfx build tasks upload --overwrite --task-zips-path file-we-just-patched.zip`:

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-12.png)

Instead of deleting the existing task and re-uploading it, we can also patch the taskzip with a version number that's newer than the broken task, in our case `0.214.2` and then upload that.

As you can see here, that's what the Azure Pipelines team does too, but they roll out the task through their own provisioning process, they either hotfix the task, or simply roll back their changes:

[Comparing master...releases/m215 · microsoft/azure-pipelines-tasks
![Hey! You broke our pipeline!](https://opengraph.githubassets.com/00a07616ebd7e6d6eb85d5d5c8ce185b9922b66b9ff2762560d2124a0a4b6a1d/microsoft/azure-pipelines-tasks)
](https://github.com/microsoft/azure-pipelines-tasks/compare/master...releases/m215)

## Temporarily replace the broken task

Instead of overwriting, we can also install a temporary version of the task with a different `id` and `name`, reconfigure our pipelines to use the temporary replacement and switch back when the Azure Pipelines team has fixed theirs.

The process is very similar to replacing the existing task, but in this case we need to make a few more changes, we need to replace the `id` (choose any guid), `name` and update the `friendlyName` properties in **both** `task.json` **and** `task.loc.json`:

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-14.png)

Or you can [download a pre-patched (`-sxs`) version from the `azure-pipelines-tasks-zips` repository](https://github.com/jessehouwing/azure-pipelines-tasks-zips/releases).

Then we can upload the patched file:

![Hey! You broke our pipeline!](https://jessehouwing.net/content/images/2023/01/image-15.png)

But now you'll need to go through all of your pipelines and replace the old task with the temporary replacement. A bit of scripting against the REST API would make that a quick change.