axios/got faceoff: uploading a file to Amazon WorkDocs using Amazon S3 pre-signed url
Tags: aws, javascript, productivity
Published: 2020-05-18 22:11:56

## Overview
If you are using [Amazon WorkDocs](https://aws.amazon.com/workdocs/?amazon-workdocs-whats-new.sort-by=item.additionalFields.postDateTime&amazon-workdocs-whats-new.sort-order=desc) as your managed cloud based content management and/or storage system, and if you are planning to automate tasks such as integrate it with other document/content storage systems, then you must have come across the use case of uploading a file. After some automation, you should be able to do this at scale. In a future post, I will share a detailed reference architecture on how to build such an integrated system. 

The following sections demonstrate the various aspects of the app, starting with setting up a simple Node.js app. However, there are some prerequisites 
- AWS IAM user with sufficient privileges e.g. I am using my development account and I have created a user with admin privileges without AWS Management console access, and I rotate its access keys regularly. For more, read [AWS IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html)
- an existing [Amazon WorkDocs site](https://www.youtube.com/watch?v=IMiA2euydn0)
- [Install](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) and [configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) AWS CLI with [named profiles](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html)


## Initialize npm project
I have used the following commands to initialize a new npm project

```
➜ mkdir workdocs-sample && cd workdocs-sample
➜ npm init
➜ npm install aws-sdk axios form-data got
➜ touch index.js
```

After initialization, my folder structure looks like this:
```
➜  workdocs-sample ls
da-quiz-storage-result.pdf 
index.js                   
node_modules               
package-lock.json          
package.json               
yarn.lock
```

## Initialize the WorkDocs client
Setup AWS credentials in `index.js`. For more information, read [best practices to use AWS credentials in your development environment](https://dev.to/sahays/best-practices-to-use-aws-access-key-and-secret-in-your-development-environment-23ki)

```js
const AWS = require("aws-sdk");
const credentials = new AWS.SharedIniFileCredentials({ profile: "default" });
AWS.config.credentials = credentials;
```

In addition to that, you'll need the following declarations
```js
const got = require("got");
const fs = require("fs");
const FormData = require("form-data");
const workdocs = new AWS.WorkDocs();
```

Finally, initialize the WorkDocs client

```js
const workdocs = new AWS.WorkDocs();
```

## Steps to upload a file
To upload a file to a WorkDocs folder you need the following:
- a folder ID to upload
  - to get the root folder ID, you need to make a call to `describeUsers` API
  - if you have created new folders at the root, then you need to call `describeFolderContents` with the root folder ID
- call `initiateDocumentVersionUpload` with the folder ID, name of the file, and optionally, a content type. It returns an Amazon S3 pre-signed upload url, document ID, and a version ID among other things
- use `got` to upload the file to the returned `uploadUrl`
- call `updateDocumentVersion` with document ID, version ID, and set `VersionStatus` to `ACTIVE` 

### Get the root folder ID
Every user has a root folder which can contain one or more children - nothing fancy, just the usual nested folder structure. The root folder has an ID that can used to create folders inside it. Using the `describeUsers` API call, we'll get the root folder ID for the user defined by the `query` parameter. You can look up `OrganizationId` from your Amazon WorkDocs AWS console.

```js
const describeUsers = async () => {
  const user = await workdocs
    .describeUsers({
      OrganizationId: "d-92672xxxxx", // your WorkDocs organization Id
      Query: "sahays", // name of an existing WorkDocs user
    })
    .promise();
  return user;
};
```

### Initialize upload
The following code uses `initiateDocumentVersionUpload` to initiate the process of uploading a file. The api requires `ParentFolderId` to upload the file to, and a `Name`. It returns a `documentId` for the document, `versionId` for the first version of the document, `uploadUrl` containing the Amazon S3 pre-signed url, and `signedHeaders` containing the `content-type` and `x-amz-server-side-encryption` encryption type.

```js
const initUpload = async ({ folderId, filename }) => {
  try {
    console.log("initUpload");
    const contentType = "application/octet-stream";
    const initResult = await workdocs
      .initiateDocumentVersionUpload({
        ParentFolderId: folderId,
        Name: filename,
        ContentType: contentType,
        ContentCreatedTimestamp: new Date(),
        ContentModifiedTimestamp: new Date(),
      })
      .promise();
    const documentId = initResult.Metadata.Id;
    const versionId = initResult.Metadata.LatestVersionMetadata.Id;
    const { UploadUrl, SignedHeaders } = initResult.UploadMetadata;
    console.log("initUpload complete");
    return {
      documentId,
      versionId,
      uploadUrl: UploadUrl,
      signedHeaders: SignedHeaders,
    };
  } catch (e) {
    console.log("failed initUpload", e);
    throw e;
  }
};
```

The header looks like the following:
```js
headers: {
    'Content-Type': 'application/octet-stream',
    'x-amz-server-side-encryption': 'AES256'
  }
```

### Upload a file using `got`
The following code uses `got` [npm library](https://www.npmjs.com/package/got) to upload a local file. Please note, we are using a `PUT` request. The file is appended to `FormData` using a file stream object. The headers retrieved from the previous call `initiateDocumentVersionUpload` is used to set a `PUT` request header.

```js
const uploadFile = async ({ filename, signedHeaders, uploadUrl }) => {
  try {
    if (fs.existsSync(filename)) {
      console.log("reading file stream");
      const fileStream = fs.createReadStream(filename);
      console.log("preparing form data");
      const formData = new FormData();
      formData.append(filename, fileStream);
      console.log("uploading to ", uploadUrl);
      const extendParams = {
        headers: signedHeaders,
      };
      console.log("got extendParams", extendParams);
      const client = got.extend(extendParams);
      await client.put(uploadUrl, {
        body: formData,
      });
      console.log("upload complete");
    } else {
      console.log("file doesn't exist");
      throw "file doesn't exist";
    }
  } catch (e) {
    console.error("failed uploadFile", e);
    throw e;
  }
};
```

### Update document version
This important step completes the file upload transaction by setting the `VersionStatus` to `ACTIVE` which tells Amazon WorkDocs to mark the just uploaded file as the most recent/active version.

```js
const updateVersion = async ({ documentId, versionId }) => {
  try {
    await workdocs
      .updateDocumentVersion({
        DocumentId: documentId,
        VersionId: versionId,
        VersionStatus: "ACTIVE",
      })
      .promise();
    console.log("document version updated");
  } catch (e) {
    console.log("failed updateVersion", e);
    throw e;
  }
};
```

## Time for that faceoff: `got` vs `axios`
![face/off](https://cdn.pixabay.com/photo/2019/01/25/13/22/cars-3954498_1280.jpg)

Let's take a look at `axios` invocation first.

```js
await axios.put(uploadUrl, formData, {
        headers: signedHeaders
      });
```

This results in Amazon S3 rejecting the request with the following error:

```xml
<Error>
<Code>NotImplemented</Code>
<Message>A header you provided implies functionality that is not implemented</Message>
<Header>Transfer-Encoding</Header>
<RequestId>016D6B18F95E6923</RequestId><HostId>QgYnoYEQTZR4jG7wvdLfAe6lcd2Tg+/eAOeHLvtM+CamqyDxZX8p7CV4ZL+Hph7+IOUiFJkayT8=</HostId>
</Error>
```

The server returns a `501: not implemented` response
```js
response: {
    status: 501,
    statusText: 'Not Implemented',
    headers: {
      'x-amz-request-id': '016D6B18F95E6923',
      'x-amz-id-2': 'QgYnoYEQTZR4jG7wvdLfAe6lcd2Tg+/eAOeHLvtM+CamqyDxZX8p7CV4ZL+Hph7+IOUiFJkayT8=',
      'content-type': 'application/xml',
      'transfer-encoding': 'chunked', // extra header
      date: 'Mon, 18 May 2020 22:00:24 GMT',
      connection: 'close',
      server: 'AmazonS3'
    },...
}
```

Now, let's take a look at the `got` invocation:

```js
const extendParams = {
  headers: signedHeaders,
};
console.log("got extendParams", extendParams);
const client = got.extend(extendParams);
await client.put(uploadUrl, {
  body: formData,
});
```
This results in a successful `200: OK` response with the same inputs

## Bring it all together

The following is the entry point function that runs as a result of running index.js using `node index.js`

```js
const start = async () => {
  try {
    const user = await describeUsers();
    const rootFolderId = user.Users[0].RootFolderId;
    const filename = "da-quiz-storage-result.pdf";
    const {
      documentId,
      versionId,
      uploadUrl,
      signedHeaders,
    } = await initUpload({ folderId: rootFolderId, filename });
    await uploadFile({ filename, signedHeaders, uploadUrl });
    await updateVersion({ documentId, versionId });
  } catch (e) {
    console.error(e);
  }
};

start();
```

## Finally
After running `node index.js` in your terminal, you'll see an output similar to the following:
```
initUpload
initUpload complete
reading file stream
preparing form data
uploading to  https://gb-us-west-2-prod-doc-source.s3.us-west-2.amazonaws.com/1b45f47aa1c4d1d1c1f0978587e10f1e56ce801824ca5d5fce0565dea6f76baf/1589767973739-0d3c7a46986cfe7d0fd8beec8258628a8b6ca0e9b0f412afafcdaf9c6aa7a00e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200518T021253Z&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-server-side-encryption&X-Amz-Expires=900&X-Amz-Credential=AKIAIM5HWZT6CVS2WHIA%2F20200518%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=025e9ed29fe7f8ab85593c51a4a09b396909de47ea1e893148df14e3435ea080
got extendParams {
  headers: {
    'Content-Type': 'application/octet-stream',
    'x-amz-server-side-encryption': 'AES256'
  }
}
upload complete
document version updated
```

The file `da-quiz-storage-result.pdf` is now uploaded as shown in this screenshot:
![uploaded-file](https://dev-to-uploads.s3.amazonaws.com/i/o7yyi960mz0217ibf13e.png)