Upload Files to S3 in Node.js
Tags: node, javascript, aws
Published: 2020-01-15 11:49:02

<p>My excitement to implement file upload didn't take too long to turn into a struggle, then dread but finally, a victorious push to the finish. This is my humble attempt to help you skip the line and jump directly to the third phase.</p>

If you are someone who is here for a specific piece of information, you may skip to any of the below:

[1. Upload file as a whole using Multer](#multer)

[2. Upload in chunks with Busboy](#busboy)

---

### Let's begin by making an uploader for Vue.js


<p>First off, let's allow our user to upload a file using Vue.js so that it may reach our API.</p>

For that, we starts with the tag:

```javascript
<input type="file" :accept="allowedMimes" ref="inputFile" @change="onChange"/>
```

The above `input` tag allows a user to upload a single file. Once a file is selected, the `onChange` method is called with the file data.

The `onChange` method looks like below:

```javascript
function onChange() {
  const data = new FormData();
  for (const [key, value] of Object.entries(this.options)) {
    data.append(key, value);
  }

  const file = this.$refs.inputFile.files[0];
  data.append('file', fileToUpload, file.name);
  const {data: res} = await axios.post(API`/files`, data);
}
```

With this, our front-end is good to go and now, we are ready to send our file off to S3.
</p>

---

### Multer-S3 saves the day<a name="multer" />

<p>This approach will let you upload a file directly to AWS S3, without having to do anything in between.</p>

##### __When to use this approach__:

* You want to pipe your data to a location in your S3 bucket without modifying or accessing the file bytes. In short, this method will pipe your whole file without you having to do anything.


<p>Here's how the basic skeleton looks like. It contains your multer declaration and the API endpoint.</p>

```javascript
const upload = multer({});

router.post('/file', upload.single('file'), async (req, res) => {

});

```

We start by specifying the `upload` method:

```javascript
const multer = require('multer');
const multerS3 = require('multer-s3');

const upload = multer({
  storage: multerS3({
    s3, // instance of your S3 bucket
    contentDisposition: 'attachment',
    contentType: multerS3.AUTO_CONTENT_TYPE,
    bucket(req, file, callb) {
      // logic to dynamically select bucket
      // or a simple `bucket: __bucket-name__,`
      callb(null, '_my_bucket_');
    },
    metadata(req, file, cb) {
      cb(null, {
        'X-Content-Type-Options': 'nosniff',
        'Content-Security-Policy': 'default-src none; sandbox',
        'X-Content-Security-Policy': 'default-src none; sandbox',
      });
    },
    async key(req, file, abCallback) {
      try {
        // logic to dynamically select key or destination
        abCallback(null, ' _dest/key_');
      } catch (err) {
        abCallback(err);
      }
    },
  }),
  limits: {}, // object with custom limits like file size,
  fileFilter: filterFiles, // method returns true or false after filtering the file
});
```

We then pass it as a middleware to our API end-point.
```javascript
router.post('/file', upload.single('file'), async (req, res) => {
    // you can access all the FormData variables here using req.file._var_name
});
```
This is it! All the data pertaining to your S3 upload will be available under the `req.file` variable.

With that, we have successfully uploaded your file to s3, the easy way.

---

### When save the day with Busboy<a name="busboy" />

<p>Then comes a situation where you want to have access of the bytes you are piping to your S3 bucket, before the actual upload happens. You might want to compress them, uncompress them, check for virus, or fulfil any other endless requirements. I decided to use `Busboy` here, it's a tried, tested and an easy to use library. Other options you may go for are libraries like `Formidable` or `Multiparty`.


##### __When to use this approach__:

* You want to access the file chunks, modify them or use them before you pipe them to your S3 bucket.

Here's how the basic structure looks like. It again, contains the basic definition along with our usual API endpoint.

```javascript
const busboyUpload = (req) => {};

router.post('/file', async (req, res) => {
});
```

So, let's dive right in. The Busboy is called as a method from our API with the `request` as its parameter as defined below.

```javascript
router.post('/file', async (req, res) => {
  try {
    const uploadedFileData = await busboyUpload(req);
    req.file = uploadedFileData;
    res.sendStatus(200);
  } catch (err) {
    res.sendStatus(500);
  }
}

```

Our Busboy uploader will be set up in a simple and straight forward manner.

* We start by returning a Promise and initiate our Busboy instance along with the basic structure.
```javascript
const busboyUpload = (req) => new Promise((resolve, reject) => {
  const busboy = new Busboy({});
});
```
---

* We then define an array that will help us check whether the upload has finished or not. This will allow us to return a suitable response.

```javascript
const fileUploadPromise = [];
```

---

* In this next step, we will work on the actual file. We define the listener that executes when a file is encountered.

```javascript
busboy.on('file', async (fieldname, file, filename, encoding, mimetype) => {
  // check for conditions and set your logic here
  // s3Bucket = '_Bucket_';
  // s3Key = '_Key_';
  // check file size and file type here
});
```

---

* Inside the `onFile` listener above, we will upload to S3 using `Read` and `PassThrough` stream. The way our streams and S3 upload will be defined is:

```javascript
const { Readable, PassThrough } = require('stream');
const s3 = require('@/utils/awsConnect').getS3();

const passToS3 = new PassThrough();
const fileReadStream = new Readable({
  read(size) {
    if (!size) this.push(null);
    else this.push();
  },
});
fileUploadPromise.push(new Promise((res, rej) => {
  s3.upload({
    Bucket: bucket,
    Key: key,
    Body: passToS3,
    contentDisposition: 'attachment',
  }, (err, data) => {
    if (err) {
      rej();
    } else {
      res({ ...data, originalname: filename, mimetype });
    }
  });
}));
fileReadStream.pipe(passToS3);
```

__Whats happening here:__ We create the `Read` stream, pass it to `PassThrough` and after creating `PassThrough` we pipe it to the S3 upload function. Before beginning the upload, we push it as a Promise to the `fileUploadPromise` array we created earlier.

---

* To begin the file upload, we define the following listeners inside our `onFile` listener. On a chunk/data event, we push the same to the `Read` stream that will in turn push it to our S3.

```javascript
file.on('data', async (data) => {
  fileReadStream.push(Buffer.from(nextChunk));
});
file.on('end', () => {
  fileReadStream.push(null);
});
```
---

* Lastly, we define our `onFinish` event, pipe the request to BusBoy, sit back and relax. You will notice, we wait for the `fileUploadPromise` to complete here before we send a response back.

```javascript
busboy.on('finish', () => {
  Promise.all(fileUploadPromise).then((data) => {
    resolve(data[0]);
  })
    .catch((err) => {
      reject(err);
    });
});
req.pipe(busboy);
```
---

In the end this is how your `BusBoyUpload` structure should look like.

```javascript
const busboyUpload = (req) => new Promise((resolve, reject) => {
  const busboy = new Busboy({ });
  busboy.on('file', async (fieldname, file, filename, encoding, mimetype) => {
    fileReadStream.pipe(passToS3);
    file.on('data', async (data) => {
    });
    file.on('end', () => {
    });
  });
  busboy.on('finish', () => {
  });
  req.pipe(busboy);
});
```
---

With this, you are well set to upload files to S3 the right way.
---

Or, you could even use the npm package I created: https://www.npmjs.com/package/@losttracker/s3-uploader

Thanks for reading! :)