File storage microservice for your Express.js Application
Tags: microservices, node, programming, express
Published: 2023-04-04 07:01:41



In today's world of technology, microservices have become an essential part of web development. Microservices are small, independent applications that can communicate with each other to create larger and more complex but efficient systems. One common use case for microservices is file storage. In this blog post, we will discuss how to set up [Total.js OpenFiles](https://docs.totaljs.com/openfiles/) as a file storage microservice for an Express application in 3 steps:

* first, we will learn about Total.js OpenFiles, why we need it and how to install it.
    
* Secondly, we will see the first approach (the bad one) of how to connect Total.js OpenFiles as a microservice to an express application.
    
* And finally, we will see the second approach (the good and recommended one) of how to connect it.
    

This blog post includes both code implementations and system design illustrations for a better understanding of some simple concepts of system microservices architectures. Make sure you read till the end to not miss one experience ahead in microservices development.

### [**Total.js OpenFiles**](https://docs.totaljs.com/openfiles/)

**OpenFiles** is a lightweight app for storing files from 3rd party applications. The application is built entirely with the Total.js Framework and its functionality is simple: the Files are stored on HDD in the Total.js FileStorage that we talked about in my previous blog post (Read [here](https://docs.totaljs.com/openfiles/)). In addition, you can easily browse and filter stored files in the OpenFiles setup interface.

![Total.js OpenFiles Interface for browsing files](https://cdn.hashnode.com/res/hashnode/image/upload/v1680191865837/4e0d3939-b207-4a5e-b226-3264d9d8f918.png)

The most interesting thing about that application is its API. Yes, OpenFiles exposes an API that allows third-party applications to store files in it.

[![Total.js OpenFiles API endpoints](https://cdn.hashnode.com/res/hashnode/image/upload/v1680213294434/8a54c649-b794-4b94-adfe-b08557e5960e.png)](https://docs.totaljs.com/openfiles/#suti001cs41d)

**Installation and setup**

Installing Total.js OpenFiles is easy. I have prepared a tutorial video about how to install it. Installing it on your machine is necessary for the rest of this tutorial. Don't forget to SUBSCRIBE to my channel.

{% embed https://youtu.be/9lLJ6qUxSdI %}

## **Connect OpenFiles to Express.js (Method 1)**

Here is the most interesting part of the tutorial. In the previous section, we followed the youtube video tutorial and we now have an awesome file storage microservice running on our machine or wherever. Now we can try the first approach (the bad one) to make an express.js application store files in that file storage. To have a clear idea of what we do, let us first illustrate the design.

### System design

![System design of file storage microservice with Total.js OpenFiles](https://cdn.hashnode.com/res/hashnode/image/upload/v1680255132791/8d6eea27-ae1c-47df-b884-a2c1ee723df5.png)

### The express.js app implementation

First, we need to initiate a simple express.js application and install dependencies via `npm`.

```bash
# Create folder
$ mkdir expressapp && cd expressapp
# Init application
$ npm init -y
# Install dependencies
$ npm install express total4 multer ejs
# Create endpoint file
$ touch index.js
#create folder for views
$ mkdir views
```

Then, we create an express.js server app by putting the following in `index.js`:

```javascript
require('total4');
const Express = require('express');
const multer = require('multer');
// URL of OpenFile instance + auth token (see video)
const Url = 'http://localhost:8000/upload/photos?token=yourtoken&hostname=1';

const app = Express();
const PORT = 3000;
const storage = multer.diskStorage({ 
    destination: function(req, file, cb) {
        return cb(null, 'temp')
    },
    filename: function(req, file, cb) {
        return cb(null, NOW.format('yyyyMMdd_HHmmss') + '_' + file.originalname)
    }
});

const upload = multer({ storage: storage });

app.set('view engine', 'ejs');
app.set('views', PATH.views());

app.use(Express.json());
app.get('/', function(req, res) {
    res.render('index', { info: '', url: '' });
});

app.post('/upload', upload.single('photo'), async function(req, res) {
   var data =  req.body;
   var file = req.file;
   var response;

   // Send file to The Total.js OpenFiles
   if (file) 
        response  = await RESTBuilder.POST(Url, { }).file('file', PATH.root(file.path)).promise();


    // save data
    data.id = UID();
    data.search = (data.name + ' ' + data.phone).toSearch();
    data.photo = response ? response.url : '';
    data.dtcreated = NOW;
    NOSQL('users').insert(data, true).callback(function(err, resp) {
        res.render('index', { info:response.name, url: response.url});
    });
});
app.listen(PORT);
```

Now we need to add an `index.ejs` view a file in the `views` folder:

```xml
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
	<meta http-equiv="X-UA-Compatible" content="IE=10" />
	<meta name="format-detection" content="telephone=no"/>
	<meta name="viewport" content="width=1024, user-scalable=yes" />
	<meta name="robots" content="all,follow" />
	<style type="text/css">
		body { padding: 50px; margin: 0; font:normal 12px Arial; color: gray }
		.paging { font-size: 11px; height: 20px }
		.paging a { float: left; margin-right: 5px; }
		.paging div { float: right; }
		.mb5 { margin-bottom: 5px; }
	</style>
</head>
<body>
<div style="background-color:#F0F0F0;padding:3px 10px">
	Uploaded: <b style="color:black"><%= info %></b>
	Image: <br>
	<% if (url) { %>
		<img src="<%= url %>" width="150" alt="Image preview">
	<% } %>
</div>
<br />
<form method="POST" enctype="multipart/form-data" action="/upload">
	<input type="text" name="name" placeholder="Name" /> <br><br>
	<input type="email" name="email" placeholder="Email" /><br><br>
	<input type="phone" name="phone" placeholder="Phone" /><br><br>
	<input type="file" name="photo" /><br><br><br>
	<button>SUBMIT</button>
</form>
</body>
</html>
```

Finally, we can start our Express server and make some tests:

```bash
$ node index.js
```

After some testing, you should see a simple UI like this:

![Simple UI output after testing](https://cdn.hashnode.com/res/hashnode/image/upload/v1680251696779/5f29a221-0681-4d73-b5a1-5f52763cd684.png)

The file is successfully uploaded and the user information is stored in the database too.  
However, this approach for connecting a file storage microservice has problems:

* Using an express server to proxy the file makes it slow because the file is processed twice, and DNS is resolved twice too.
    
* Bringing `multer` in this project could be avoided to make the express application very lightweight, and also avoid storing files in a temporary folder.
    
* Bringing `total4` is a very bad idea, [Total.js](https://totaljs.com) is an independent framework for web development. Bringing the entire Total.js framework into another framework (Express) is working fine but this is not a good practice.
    

This first approach is working fine, but it is not very good practice, which brings us to the second approach which is far better.

## **Connect OpenFiles to Express.js (Method 2)**

The first method has some shortcomings, so we will reshape our system design to make better microservice architecture.

### System design

This time we are sending the file from the web browser to our Total.js OpenFiles microservice. It will return the URL and we will be able to push the URL and the form data to our express application. Just like the following illustration:

![The best system design of file storage microservice with Total.js OpenFiles](https://cdn.hashnode.com/res/hashnode/image/upload/v1680255250360/06b763e4-898d-4876-8f3e-3163050d8dd5.png)

### The express.js app implementation

Now that we do not send files to OpenFiles via express application, the express app is simplified and better. The `index.js` file is simplified as follows:

```javascript
require('total4');
const Express = require('express');

const app = Express();
const PORT = 3000;

app.set('view engine', 'ejs');
app.set('views', PATH.views());

app.use(Express.json());
app.get('/', function(req, res) {
    res.render('index', { info: 'info', url: '' });
});

app.post('/upload', async function(req, res) {
   var data =  req.body;
   console.log(data);
    // save data
    data.id = UID();
    data.search = (data.name + ' ' + data.phone).toSearch();
    data.dtcreated = NOW;
    NOSQL('users').insert(data, true).callback(function(err, resp) {
        res.json({ success: true, value: data.id });
    });
});


app.listen(PORT);
```

PS: we still have `require('total4')` just because of the NOSQL database that we are using, but if you would like to use the Mysql database as described in the system design, then `total4` is completely useless. Then you have the following in the `views/index.ejs` file:

```xml
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
	<meta http-equiv="X-UA-Compatible" content="IE=10" />
	<meta name="format-detection" content="telephone=no"/>
	<meta name="viewport" content="width=1024, user-scalable=yes" />
	<meta name="robots" content="all,follow" />
	<script src="//cdn.componentator.com/spa.min@19.js"></script>
	<link rel="stylesheet" href="//cdn.componentator.com/spa.min@19.css" />
	<style type="text/css">
		body { padding: 50px; margin: 0; font:normal 12px Arial; color: gray }
		.paging { font-size: 11px; height: 20px }
		.paging a { float: left; margin-right: 5px; }
		.paging div { float: right; }
		.mb5 { margin-bottom: 5px; }
		.button { border: 0; margin: 15px 0 0; background-color: #F0F0F0; height: 34px; padding: 0 15px; color: #000; cursor: pointer; font-family: Arial; line-height: 34px; vertical-align: middle; outline: 0; font-size: 14px; text-decoration: none; transition: all 0.3s; float: left; width: 100%; }
		.button:disabled { background-color: #7c98d5; }
		.button i { width: 15px; text-align: center; margin-right: 5px; }
	</style>
</head>
<body>
<ui-component name="upload"></ui-component>
<br />


<div class="container">
	<ui-plugin name="Form" path="form" >
		<div class="grid-2">
			<ui-component name="input" path="?.name" config="required:1;">Name</ui-component>
			<ui-component name="input" path="?.email" config="required:1;type:email">Email</ui-component>
			<ui-component name="input" path="?.phone" config="required:1;type:phone">Phone</ui-component>
			<ui-component name="input" path="?.photo" config="required:1;type:url;disabled:1">Photo url</ui-component>
			<span></span>
			<ui-component name="preview" path="preview" config="width:300;height:200;url:[openfiles];icon:folder;preview:value => value.url;">Preview</ui-component>
		</div>
		<ui-component name="submit" path="?" config="url:POST /upload/">
			<button class="button" name="submit" disabled="disabled">SUBMIT FORM</button>
		</ui-component>
	</ui-plugin>
</div>
<script>
	var preview = {};
	var form = {};
	ENV('openfiles', 'http://localhost:8000/base64/photos?token=yourtoken&hostname=1');
	console.log(ENV());
	WATCH('preview', function(path, value) {
		SET('form.photo', value.url);
	});
</script>

</body>
</html>
```

This time we are importing a `CDN` at the head of the HTML document: it's the client-side library of Total.js. We try to leverage its powerful [data-binding](https://docs.totaljs.com/components/40d04001gw51c/) features and [free UI components](https://componentator.com/) for uploading the image directly from the browser to the OpenFiles microservice. You should see the following result:

![Output of connecting Total.js OpenFiles as microservice](https://cdn.hashnode.com/res/hashnode/image/upload/v1680290368712/b123ed97-c74e-4bf7-a962-a119e1476477.jpeg)

In summary, we have installed and learned about Total.js OpenFiles before connecting it to an Express.js application as a microservice, in two different approaches. The first approach is good but may become challenging with time, especially as the application is scaling. The second approach is not perfect but at least solves many problems that the first approach is more likely to accounter. I hope it has been helpful. Feel free to play with the code, connect it to any other backend language you want and if you find out some ideas of how to improve any of those system designs, then feel free to leave some comments. Like, share and subscribe.

* [**Make the Total.js Platform better, support it!**](https://www.totaljs.com/support/)
    
* [**Have any questions? Contact us**](https://www.totaljs.com/contact/)
    
* [**Join Total.js Telegram**](https://t.me/totaljs)