Step-by-Step Guide: Uploading and Downloading Binary Files to AWS S3 v3 using serverless JavaScript and Codehooks.io
Tags: node, aws, serverless, javascript
Published: 2023-05-24 17:01:33

Are you looking for an API to upload and download binary files to AWS S3 v3 using Node.js? Look no further! In this step-by-step guide, we will walk you through the process of achieving this seamlessly with a Codehooks.io API and easy deployment to the cloud.

The complete source code is available at [GitHub](https://github.com/RestDB/codehooks-io-examples/tree/main/aws-s3)

## Set Up AWS S3

To begin, make sure you have an AWS account and an S3 bucket created. Note down the bucket name and region for future reference. 

Then get your secure access credential to use with the AWS S3 API, read more about how to in [this excellent Medium guide](https://medium.com/@shamnad.p.s/how-to-create-an-s3-bucket-and-aws-access-key-id-and-secret-access-key-for-accessing-it-5653b6e54337) from [@shamnad.p.s](https://medium.com/@shamnad.p.s).

## Create a codehooks.io project and install required Packages

If you're new to Codehooks.io you can read the [getting started guide here] (https://codehooks.io/docs/).

Install the official Codehooks command line interface (CLI) first.

```bash
npm install codehooks -g
```

Then, after logging in, create a new project, in this example we'll name our project `s3project`. Choose any project title you prefer.

```bash
coho create s3project
cd s3project
```

In your project directory, initialise and install the necessary packages by running the following commands:

```bash
npm init es6 -y
npm install codehooks-js
npm install @aws-sdk/client-s3
```

## Writing the Code

Now, let's dive into the code to handle the upload and download files to AWS S3.

### Import the required packages:

```js
import { app } from 'codehooks-js'
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3'
import { PassThrough } from 'stream'
```

### Configure the AWS SDK with your AWS credentials and region:

Use the CLI or the Admin UI to add your secret environment variables. Note the `--encrypted` flag to prevent reading the actual data content of the variables.

```bash
coho set-env AWS_ACCESS_KEY_ID 'YOUR_KEY' --encrypted
coho set-env AWS_SECRET_ACCESS_KEY 'YOUR_SECRET' --encrypted
coho set-env AWS_BUCKET 'YOUR_BUCKET' --encrypted
coho set-env AWS_REGION 'YOUR_REGION' --encrypted
```

Use the CLI command `coho info` to inspect and verify that the variables are set correctly.

With the necessary secrets in place, we can create the code to access these in the application as regular `process.env` values.

```js
const { AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_BUCKET } = process.env;

const s3config = {
    region: AWS_REGION,
    AWS_ACCESS_KEY_ID,
    AWS_SECRET_ACCESS_KEY
}

const s3client = new S3Client(s3config);
```

The next step is to create an API that process file uploads from any client and store files in an AWS S3 bucket.

### API to Upload a binary file to AWS S3:
The API to upload a data stream to AWS S3 uses the `PutObjectCommand` with a stream that we can pipe data from the client into.

```js
// API to POST a binary data stream
app.post('/upload/single', async (req, res) => {
    try {
        // get size, type and filename from destructured header values
        const { 'content-length': ContentLength, 'content-type': ContentType, filename } = req.headers;
        
        const input = {
            "Bucket": AWS_BUCKET,
            "Key": `tmp/${filename}`,  // emulate file system /bucketname/tmp/filename
            "Body": new PassThrough(), // stream to pipe data through
            "ContentLength": ContentLength,
            "ContentType": ContentType
        };
        // pipe binary request data to S3 stream
        req.pipe(input.Body); 
        // create put command
        const command = new PutObjectCommand(input); 
        // execute put object command
        const response = await s3client.send(command); 
        // return data to client
        res.json(response);
    } catch (error) {
        // some error occured, return 400 status to client
        res.status(400).end(error.message)
    }
})
```

Let's test the upload API with curl by uploading a test file from a local filesystem. Use `coho info --examples` to inspect your project API endpoint. In this example it's: `https://s3project-kvqx.api.codehooks.io/dev/upload/single`

```bash
curl --location 'https://s3project-kvqx.api.codehooks.io/dev/upload/single' \
--header 'x-apikey: XXXXX' \
--header 'filename: einstein.jpg' \
--header 'Content-Type: image/jpeg' \
--data '@./Einstein_tongue.jpg'
```

### API to Download a binary file from AWS S3:

The API to download a data stream from AWS S3 uses the `GetObjectCommand` with a stream that can pipe data back to the client.

```js
// API to GET a binary data stream from AWS S3
app.get('/download/:file', async (req, res) => {
    try {
        // filename from route
        const { file } = req.params;
        const input = {
            "Bucket": AWS_BUCKET,
            "Key": `tmp/${decodeURI(file)}` // decode filename and store in /bucket/tmp/file
        };
        // Create get command
        const command = new GetObjectCommand(input);
        // Send get command
        const response = await s3client.send(command);
        // set content-type
        res.set('content-type', response.ContentType)
        // stream data back to client
        response.Body.pipe(res.writable)
    } catch (error) {
        // Woops
        res.status(400).end(error.message)
    }
})
```

### Easy deployment to the cloud:

After finishing the code for the AWS S3 API, we simply deploy the application to the cloud with this command:

```bash
coho deploy
```
The command should output the following on success.

```bash
Project: s3project-kvqx  Space: dev
Deployed Codehook successfully! 🙌 
```

Now that we understand the basic operations required to upload and download binary files to AWS S3, let's go ahead and create a simple web client that uses the API.

Happy days 🙌

## Client web app to test the API

Let's create a simple (and ugly) web page to upload and download a binary file to AWS S3 through the Codehooks API.

```html
<html>
<body>
    <h2>AWS S3 upload and download API demo</h2>
    <label> upload binary file to read here </label>
    <input type="file" id="inputFile">
    <div id="mydiv"></div>
    <script src="test.js"></script>
</body>
</html>
```
Screen shot of the ugly web page:

![web page example to upload binary file to AWS S3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/407s7r48nz7d1rlcskxa.png)

Note that the web page refers to a JavaScript `test.js` which contains the following code.

```js
/*
* Web client to upload a binary file to a Codehooks.io AWS S3 API
*/

var URL = 'https://<YOUR_PROJECT_ID_HERE>.api.codehooks.io';
var APIKEY = 'YOUR_API_TOKEN_HERE';

async function uploadFile(file, cb) {
    var myHeaders = new Headers();
    myHeaders.append("x-apikey", APIKEY);
    myHeaders.append("filename", file.name);
    myHeaders.append("content", file.type);
    myHeaders.append("content-type", file.type);
    myHeaders.append("content-length", file.size);

    var requestOptions = {
        method: 'POST',
        headers: myHeaders,
        body: file, // file stream
        redirect: 'follow'
    };

    try {
        var response = await fetch(`${URL}/dev/upload/single`, requestOptions)
        console.log(response.status, response.statusText);
        var result = await response.text();
        console.log(result);
        cb(result);
    } catch (error) {
        console.error(error)
    }
}

// HTML5 file event
function fileChange(theEvent) {
    var theFile = theEvent.target.files[0]; // target the loaded file
    uploadFile(theFile, (result) => {
        var mydiv = document.getElementById("myDiv");
        var aTag = document.createElement('a');
        aTag.setAttribute('href',`${URL}/dev/download/${theFile.name}`);
        aTag.setAttribute('target','_blank');
        aTag.innerText = theFile.name;
        mydiv.appendChild(aTag);
        mydiv.appendChild(document.createElement("br"));
        if (theFile.type.startsWith('image')) {
            var imgTag = document.createElement('img');
            imgTag.setAttribute('src',`${URL}/dev/download/${theFile.name}`);
            imgTag.setAttribute('width','200px');
            mydiv.appendChild(imgTag);
            mydiv.appendChild(document.createElement("br"));
        }
    });    
}
// get html element for file: <input type="file" id="inputFile">
document.getElementById("inputFile").addEventListener("change", fileChange, false); // listener for file upload button
```

You'll find the complete example source code for AWS S3 upload and download of binary files at our [Github account here](https://github.com/RestDB/codehooks-io-examples/tree/main/aws-s3)


Remember to replace `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` and `AWS_BUCKET` with your actual AWS credentials, bucket and region.

### Links
* [AWS S3 v3 SDK](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/)
* [Codehooks.io](https://codehooks.io/)
* [Codehooks docs](https://codehooks.io/docs)
* [Codehooks blog](https://codehooks.io/blog)

If you have read this far and even tried out the example code in your own project, you're all set to conquer the world with apps and API's that uses the power of AWS S3 for binary file persistence and much more.

Happy coding!