🤯 Private S3 HTTP Server
Tags: node, javascript, tutorial, aws
Published: 2021-11-09 13:12:04

## What are we solving here?

In my AWS adventures, I've come across use cases where it'd be awesome to have easy access internally to some files in AWS without making something public....or creating an S3 bucket behind a Cloudfront behind a WAF with a whitelist rule.....
<img width="100%" style="width:100%"src="https://media4.giphy.com/media/fRhSHzQ4NXOdrHIZJd/giphy.gif?cid=5a38a5a26jegtp8ckhec88yx10kjr2a714zeeykn1smhy4mk&rid=giphy.gif&ct=g"/>


Anyway, `http-server` is an awesome package that does this exact thing for file shares. So I figured I'd fork the code base and replace all the file server bits with S3 calls.
<img width="100%" style="width:100%"src="https://media2.giphy.com/media/3o6Zt4HU9uwXmXSAuI/giphy.gif?cid=5a38a5a2i6b3n37860wozrt2tf2epxoq271f6f563rguvxjz&rid=giphy.gif&ct=g"/>

Opening the hood of http-server, I realized for my specific use case, most of the code has features I didn't need. So, I opted to create `s3-http-server` from scratch with http-server as inspiration. 

What follows is an explanation of the interesting bits.

## A look at the 🥩🥔 code

Firstly, the stack used for this solution:

- nodejs - javascript runtime
- express - http server
- pug - template engine

The most important features are listing, downloading, and uploading objects.

### Listing Objects

The code snippet for this is straight forward, but ensuring you are only returning objects AND prefixes at a given level is a little obscure. Below is the excerpt of code from the express route's async function handler - 

```javascript
const AWS = require("aws-sdk");
const s3 = new AWS.S3();
const Bucket = 'MyBucket'
const Prefix = ''

...
    const s3Res = await s3
      .listObjectsV2({
        Bucket,
        Prefix,
        Delimiter: "/",
        MaxKeys: 1000,
      })
      .promise();
    const data = [
      ...s3Res.CommonPrefixes.map(({ Prefix }) => ({
        Key: Prefix,
        isDir: true,
      })),
      ...s3Res.Contents.filter((c) => c.Key !== Prefix).map(({ Key }) => ({
        Key,
        isDir: false,
      })),
    ];
...

```

The first part returns a list of s3 objects at a given prefix. Note that in a large bucket, you would want to handle pagination of the objects as well.

The shenanigans creating the data variable is the good part. If, for example, you call `listObjectsV2` with a Prefix of "" (the root of the bucket), you only get objects in the return object's `Content` property. To get the prefixes at the root (or anywhere else) you'll need to look at the `CommonPrefixes` property. 

### Downloading Objects

The downloading of an object in S3 is a fun implementation as well. Here is an abstracted excerpt of that code.

```javascript
...
      const readStream = new stream.PassThrough();
      const fileName = ...
      res.set("Content-disposition", "attachment; filename=" + fileName);
      s3.getObject({
        Bucket: bucket,
        Key: decodeURI(req.originalUrl.substring(1)),
      })
        .on("error", (err) => {
          console.log(err);
        })
        .on("httpData", (chunk) => {
          readStream.push(chunk);
        })
        .on("httpDone", () => {
          readStream.end();
        })
        .send();
      readStream.pipe(res);
...
```
This works by creating a pass through stream. We then call `getObject` and configure a listener for `httpData`. Each time the listener function fires the stream gets the current chunk pushed into it. Finally we pipe the stream to the express response stream.

### Uploading Objects

The client website also allows uploading objects into the current S3 prefix.

```javascript
...
    const form = new formidable.IncomingForm();
    form.parse(req, async function (err, fields, files) {
      const { originalFilename, filepath } = files.filetoupload;
      const rawData = fs.readFileSync(filepath);
      await s3
        .upload({
          Bucket: bucket,
          Key: req.originalUrl.substring(1) + originalFilename,
          Body: rawData,
        })
        .promise();
...
```
We leverage the `formidable` package to simplify the file upload. Simply call the `IncomingForm` function and follow that up with the `parse` function on the return object. The callback passed to the `parse` function is where the magic happens. We get a local file path of the file that has been uploaded and the file name. We then read the uploaded file  into memory and create a key using the current prefix and supplied file name which is all passed to the `s3.upload` function.


## Using the npm package

Here's the [s3-http-server](https://github.com/rowlinsonmike/s3-http-server) repo if you'd like to look at the code base in full.

Install the package with npm
```bash
npm install s3-http-server --global
```

*Make sure you have AWS keys available in the environment*

Run the following to fire up the server

```bash
s3-http-server my-bucket-name
```

Navigate to `http://localhost:8080` and start browsing your bucket.


## Wrap up

Thanks for reading. There are a few libraries in the wild that serve a similar purpose. This was my take on it. I'll add features like deletion of objects and website serving in the future. Let me know if you have any suggestions. 

Peace ✌️