How to read Azure Dev Ops logs from Node.js using REST API
Tags: node, azdo, zip, rest
Published: 2020-04-22 14:17:52

---
title: How to read Azure Dev Ops logs from Node.js using REST API
published: true
description: 
tags: nodejs, azdo, zip, rest
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/n9b407v7gig6cqffc4u9.jpg
---

## Step by step guide

> Getting some logs can't be that hard, can it? I can surely do that in a few lines of code!

Well, that's what I though... initially. It turns out it's more than just calling a GET endpoint.

Some of it is that the response from the logs endpoint is actually a zip file. And inside that there are multiple file entries - one for each pipeline task. Plus there's the authorization part. So...

This article will take you step by step from a blank file to having the logs from your Azure Dev Ops Release pipeline. As you may know they are available online, but to get to them there are a couple of steps/clicks one must go through. You may want to get the logs and process them programmatically. For example I had to check if a particular string is part of the Release pipeline logs.

Prerequisites:

- node js - install [Windows/MacOS](https://nodejs.org/en/download/) or [Linux](https://www.devroom.io/2011/10/24/installing-node-js-and-npm-on-ubuntu-debian/)
- Azure Dev Ops account - sign up [here](https://azure.microsoft.com/en-us/services/devops/) (don't worry it's _free_)

## What we'll do

- Start with a blank node.js project and include dependencies - [axios](//npmjs.com/packages/axios) and [yauzl](//npmjs.com/packages/yauzl).
- Get a personal access token(PAT) from Azure Dev Ops and store it in an environment variable. Use that for authorization.
- Get the zipped logs via the [Azure Dev Ops REST API](https://docs.microsoft.com/en-us/rest/api/azure/devops/).
- Unzip in memory and read the text contents.
- We'll read logs out of a Release Pipeline run, but at the end there is a section on how to convert the script and read a Build Pipeline script.

> If you only want the finished script here's the gist for reading logs from a [Release Pipeline](https://gist.github.com/gparlakov/426820697dc2574da6e6f6f6b31d5498) and [Build Pipeline](https://gist.github.com/gparlakov/95e41ebee68c99baf36f36c12278deaf). I've left reminders `// TODO Replace with your own` for the variables.

## My setup
I'll be using [ts-node](//npmjs.com/packages/ts-node) because I prefer typescript safety and don't want to have to deal with the transpilation step. So instead of `node index.js` I'll do `ts-node index.ts`. The script should work as plain js, after the types are removed if you so prefer.

My shell is `bash` running inside Windows Subsystem for Linux ([WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10)).

## 1. Start

In a folder `azdo-logs` initialize a node package:

```bash
mkdir azdo-logs
cd azdo-logs
npm init -y
```

Expect to see an output similar to:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/l0fbpdomngwable86bpc.png)

Create `index.ts` file and include these lines:

```ts
/// <reference types="node" />

const accessToken = process.env.AZURE_ACCESS_TOKEN;

if (accessToken == null || accessToken === '') {
  throw new Error('Please provide an access token');
} else {
  console.log('token is present!');
}
```

We'd like to be sure the token is there, safely hidden in your **private environment variable** and NOT checked in with the code!

The `reference` on top gives us access to the nodejs types. You might need to install them as a dev dependency:
```bash
npm i @types/node -D
```

## 2. Add dependencies

Globally install `ts-node` and `typescript` to execute our script.

```bash
npm i -g ts-node typescript
```

Install `axios` and `yauzl` in our package. The flag `-s` will save  them to our `package.json`. And `@types/yauzl` will give us typing, adding it to `devDependencies` with the `-D` flag

```bash
npm i axios yauzl -s
npm i @types/yauzl -D
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8yjlvxcznqdfzs48f7wc.png)

This is how `package.json` looks like now:

```json
{
  "name": "azdo-logs",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "axios": "^0.19.2",
    "yauzl": "^2.10.0"
  },
  "devDependencies": {
    "@types/yauzl": "^2.9.1"
  }
}
```

## 3. Get Azure Dev Ops token

It can be acquired via the profile menu

- open personal access tokens page 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/91zh2vgwpyhb6ldbt6nr.png)
- create a new access token with Release `Read` permission 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5m15mxd5m398dq8klass.png)
- store it because you will not be able to see it anymore(_you'll be able to recreate it if you lose it_) 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3ej02n36wt4pxsbeb59l.png)

Finally place that in an environment variable on you local machine or in safe storage (e.g. [secret environment variable](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=classic%2Cbatch#secret-variables))

```bash
export AZURE_ACCESS_TOKEN = "token-placeholder-not-actual-thing"; # replace token-placeholder-not-actual-thing with your token
```

or Windows command line

```bat
set AZURE_ACCESS_TOKEN="token-placeholder-not-actual-thing";
```

I've added this line to my `.bashrc` file so the PAT is available on bash start and I don't have to remember to `export` it every time I start a terminal. 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/dehrtnmb6l0rhnj4d0ih.png)
On Windows you can add it to your Environment Variables. _Keep in mind you'll need to restart your session (logout/login) for the env variables to take effect._

Now run `ts-node index.ts` and you should see 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5qlp0il9ux9bdg4jgwiv.png)
For more details on how to get personal access token see [this link](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page).

Ok - we now have the token and dependencies!

### 4. Get the Azure Dev Ops Organization and Project

To get the logs, we'll need the organization and project names as well as the release id we'd like to read the logs of. The latter would increment for each subsequent run - release 1, 2, 3 so we'd need to provide it per call. For this example, I'll target the [release pipeline](https://dev.azure.com/gparlakov/Scuri/_releaseProgress?_a=release-environment-logs&releaseId=58&environmentId=87) for a package I maintain.

The project name and organization we can get from the Azure Dev Ops UI: 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/tl6ph1g593765ev5tu3p.png)

In my case, it's organization 'gparlakov' and project 'Scuri'. Add those lines in `index.ts` and replace with your org and project names:

```ts
const project = 'Scuri';
const organization = 'gparlakov';
```

### 5. Authenticate

To get authorization for the API endpoint using a personal access token (PAT) we need to send a header with the token encoded in base64 format adhering to a [specific contract](https://docs.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-5.1#assemble-the-request). Add the following at the and of `index.ts`:

```ts
const headers = {
  Authorization: `Basic ${Buffer.from(`PAT:${this.token}`).toString('base64')}`,
  'X-TFS-FedAuthRedirect': 'Suppress', // we can't handle auth redirect so - suppress
};

export const axiosInstance = axios.create({
  baseURL: `https://vsrm.dev.azure.com/${organization}/${project}/_apis/`,
  headers: headers,
});
```

We need to import the axios module, at the top of `index.ts`

```ts
import axios from 'axios';
```

### 6. Get the logs for your release

For this example, I'll use an actual release with the `id` of `58` (replace with your own). Appending to `index.ts`:

```ts
const releaseId = 58;
axiosInstance
  .get(`release/releases/${releaseId}/logs`, {
    responseType: 'stream',
  })
  .then((logs) => {
    if (logs.status != 200) {
      throw new Error('logs missing');
    }
    console.log('Received bytes:', logs.data.read().length);
  });
```

Running `ts-node index.ts` should yield something similar to:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/vqg1gwl37gpnqbzrh46b.png)

That proves we are authorized to use this REST API endpoint!

### 7. Unzip the logs

Delete or comment out the `console.log` line - we'll not need it for now and change the `axiosInstance` call so it looks like this:

```ts
axiosInstance
  .get(`release/releases/${releaseId}/logs`, {
    responseType: 'stream',
  })
  .then((logs) => {
    if (logs.status != 200) {
      throw new Error('logs missing');
    }
    return readLogs(logs.data);
  })
  .then(({ logs }) => {
    console.log(logs);
  });
```

and finally the `readLogs` function:

```ts
function readLogs
  zipBuffer: NodeJS.ReadableStream
): Promise<{ logs: string; skippedFor: Error[] }> {
  // we'll reject the promise when we can't read anything from the zip
  // and resolve it when we could read (some) plus add the errors for the skipped parts
  // in the end we'd like to say - yes the logs contain the Proof OR no the logs do not contain the proof but there were skipped parts
  return new Promise((res, rej) => {
    const es: Error[] = [];
    const zipChunks: any[] = [];

    zipBuffer.on('data', (d) => zipChunks.push(d));
    zipBuffer.on('end', () => {
      yauzl.fromBuffer(Buffer.concat(zipChunks), { lazyEntries: true }, function (err, zipfile) {
        // can not even open the archive just reject the promise
        if (err) {
          rej(err);
        }
        if (zipfile != null) {
          const chunks: any[] = [];

          zipfile.on('entry', function (entry) {
            if (/\/$/.test(entry.fileName)) {
              // Directory file names end with '/'.
              // Note that entries for directories themselves are optional.
              // An entry's fileName implicitly requires its parent directories to exist.
              zipfile.readEntry();
            } else {
              // file entry
              zipfile.openReadStream(entry, function (err, readStream) {
                if (err) {
                  es.push(err);
                  // skip this one - could not read it from zip
                  zipfile.readEntry();
                }
                if (readStream == null) {
                  // just skip - could not get a read stream from it
                  es.push(
                    new Error(
                      'Could not create a readable stream for the log ' + (entry || {}).fileName ||
                        '<missing file name>'
                    )
                  );
                  zipfile.readEntry();
                } else {
                  readStream.on('data', (c) => chunks.push(c));
                  readStream.on('error', (e) => {
                    es.push(e);
                    // skip this one - could not read it from zip
                    zipfile.readEntry();
                  });
                  readStream.on('end', function () {
                    zipfile.readEntry();
                  });
                }
              });
            }
          });

          zipfile.once('end', function () {
            zipfile.close();
            res({ logs: Buffer.concat(chunks).toString('utf8'), skippedFor: es });
          });

          zipfile.readEntry();
        } else {
          // can't read the archive - reject the promise
          rej(new Error('Could not read the zipfile contents'));
        }
      });
    });
  });
}
```

There seems to be a lot going on here. It boils down to working with 3 streams.

- First, we read the `zipFile` push into the `zipChunks` and concat those into a `Buffer`.
- Then, use that `Buffer` in the `yauzl.fromBuffer()` call which returns an object that has a `readEntry()` method. I think of it as a `next`, because it reads the next entry in the archive.
- We get a `readStream` for each zip file entry. That is a `ReadableStream` that we push into the `chunks`.
- Finally, we concat all files' chunks into a buffer and read a string out of it:
  ```ts
  Buffer.concat(chunks).toString('utf8');
  ```

### Done!

We now have a `string` variable containing all our logs!

Here's a gist of the final [index.ts](https://gist.github.com/gparlakov/426820697dc2574da6e6f6f6b31d5498). I've left reminders `// TODO Replace with your own` for the variables.

### Reading a build pipeline logs

To read the logs from a build pipeline, we would need to

1. Add the "Build: Read" permission to our token or issue a new one with that permission:
   ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/csbuv3rlsad1pyi6wwjq.png)
2. Change a bit (just remove one piece) the auth logic:
   ```ts
   const headers = {
     Authorization: `Basic ${Buffer.from(`:${this.token}`).toString('base64')}`,
     'X-TFS-FedAuthRedirect': 'Suppress', // we can't handle auth redirect so - suppress
   };
   ```
3. Change the base URL:
   ```ts
   export const axiosInstance = axios.create({
     baseURL: `https://dev.azure.com/${organization}/${project}/_apis/`,
     headers: headers,
   });
   ```
4. Change the endpoint address and provide a build number (in my case I'll use [this build](https://dev.azure.com/gparlakov/Scuri/_build/results?buildId=208&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9))
   ```ts
   const buildId = 200;
   axiosInstance.get(`build/builds/${buildId}/logs`, {
     responseType: 'stream',
     headers: {
       accept: 'application/zip',
     },
   });
   ```
Here's the final [script](https://gist.github.com/gparlakov/95e41ebee68c99baf36f36c12278deaf).

### Memory consumption note

This whole approach keeps a few buffers in memory, basically copying the zip file a few times* in memory. Considering that we are reading pipeline logs, this should not be a problem. I expect they won't be too large. If that's a problem for you, store the archive locally (though that may be a [security consern](https://en.wikipedia.org/wiki/Zip_bomb) as Samuel Attard @marshallofsound pointed out) and then use the other method of `yauzl`

```ts
logs.data.pipe(fs.createWriteStream('my-temp-zip-file.zip'))

yauzl.open('my-temp-zip-file.zip', { lazyEntries: true }, function(err, zipfile) {
    //... same code from here on down
```

 *the response stream, the chunks, the buffer, the zip content chunks, their buffer and finally the string - for a 50k log we end up using 250k RAM.
## Resources

- Restful API [docs](https://docs.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-5.1) - really helpful
- nodejs [client](https://github.com/microsoft/azure-devops-node-api) for the API (but its around 116k minified+GZipped! according to [bundlephobia](https://bundlephobia.com/result?p=azure-devops-node-api@10.1.1) ~830k worth of script for your runtime to parse - **for each request**)
- Docs for [axios](https://github.com/axios/axios#request-config)
- Docs for [yauzl](https://github.com/thejoshwolfe/yauzl)