Upload file to Express server
Tags: javascript, node, webdev
Published: 2021-07-31 02:41:48

I was working on creating a proof of concept RabbitMQ data pipeline in Node where a web app would upload a large csv file to an Express server and the server would stream its content into the pipeline in JSON.

There are two possibilities of uploading a file
1) Send entire file
2) Stream file


#### Send Entire File
Send entire csv file from browser
```javascript
fetch('http://localhost:3000/upload', { // Your POST endpoint
  method: 'POST',
  headers: {
    'Content-Type': 'text/csv' // Not necessary if extension is csv
  },
  body: file // This is your file object
})
.then(success => console.log(success)) // Handle the success response object
.catch(error => console.log(error)) // Handle the error response object
``` 

The two important points int the server are
1. How to handle request
2. How to stream csv file content as json into pipeline

To get a stream of JSON objects from the csv file, create a stream and pipe that stream into `fast-csv`.

The resulting code
```javascript
const app = require('express')()
const textBodyParser = require('body-parser').text
const csv = require('fast-csv')
const { Readable } = require('stream')

// Handle very large file
app.use(text({ type: 'text/csv', limit: '500mb' }))

app.post('/upload', (req, res) => {
  const content = Readable.from(req.body)
  content
    .pipe(csv.parse({ headers: true }))
    .on('data', (data) => {
      console.log(data) // Handle JSON object
    })
  res.sendStatus(200)
})
``` 
  
*A lot of the tutorials suggest that one use `express-fileupload`.It doesn't work if the csv file is not streamed*.


#### Stream File
Stream csv file from browser
```javascript
// Important that file is sent as FormData
const data = new FormData()
data.append('file', file)
fetch('http://localhost:3000/upload', {
  method: 'POST',
  body: data,
})
.then((success) => console.log(success)) // Handle the success response object
.catch((error) => console.log(error)) // Handle the error response object => console.log(error)) // Handle the error response object
```

For the server to handle the stream, the HTTP request must have the header `Content-Type: multipart/form-data; boundary=aBoundaryString`, more info found [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types#multipartform-data).
By sending the file as form data we can avoid having to specify this header.The browser will take care of it.

Use `busboy` to get the file stream and pipe that to `fast-csv` to get a stream of JSON objects.

The resulting code

```javascript
app.post('/upload', (req, res) => {
  const busboy = new Busboy({ headers: req.headers })
  // Busboy gives us a lot information regarding the file
  busboy.on('file', (__, file) => {
    file.pipe(csv.parse({ headers: true })).on('data', (row) => {
      // Handle data here. Row is a csv row in JSON
      console.log('Row in JSON', row) 
    })
    file.on('end', function () {
      // Handle end case here
      console.log('file ended')
    })
  })
  busboy.on('finish', function () {
    res.writeHead(303, { Connection: 'close', Location: '/' })
    res.end()
  })
  req.pipe(busboy)
})
```