Let's create microservices system from scratch - part 2
Tags: architecture, microservices, beginners
Published: 2021-01-18 10:02:01

We ended my [previous article](https://dev.to/digitalcrafting/let-s-create-microservices-system-from-scratch-series-intro-24g4) with a basic architecture diagram as well as a tech stack which we want to use and a rough idea for the first milestone we want to achieve.

Having that, I happily started coding. Created basic REST Apis for few services (just "I'm alive"), dockerized them and even connected *Gateway* to *Service discovery* using docker-compose which was pretty easy. But that's not what we are going to talk about in this article. We are going to talk about something that I feel is not stressed enough when talking about microservices - *how do you manage them*? 

Granted, for some of us developing services, there will be whole another team of DevOps Engineers managing and running everything BUT there is still an issue of testing this thing locally. 

For some reason I thought that simply using Docker will solve all my problems. Boy, was I wrong.

# The problem
The problem lies in the number of things you need to do to start even the simplest test case. Suppose we need to start just 2 services connected to each other using Docker, say `api-gateway` and `user-service`, for each of the services we need to:
1. Remove old docker image
2. Rebuild the service
3. Rebuild the docker image
4. Start the docker image

Which means **8** steps for **2** services. It gets worse if you forget to run docker image with `--rm` flag set - you will have to clean up the docker containers (unless you force clear everything later). Also, step 1 is required because docker image is rebuilt when it detects the changes to `Dockerfile` which will not happen if we only rebuild the app. Even with Docker-compose, the *remove old image* still applies since, at least for me, Docker-compose will sometimes not pick up changes. 

With 3 services it becomes unbearable and we start to create some scripts to ease the task. And so, we create the script, and then another, and another... After a while, there comes a problem of running these scripts one by one which basically means we are back to square one, only this time, we run scripts and not individual commands. Then we write scripts to run other scripts... and suddenly! A lightbulb moment! We should create a  [CLI Tool](https://en.wikipedia.org/wiki/Command-line_interface)! And that's what I did.

# The solution

The [CLI Tool](https://en.wikipedia.org/wiki/Command-line_interface) will allow us to run everything from terminal, but we need to take it into account when designing the architecture of each service. In order to do that, we need to back up a little (not too much fortunately, we didn't write that much code yet) and design a proper structure for our services so that we can easily add new ones in the future. 

As for how do we design it, there are few questions we need to answer regarding the CLI:

### What do we need the CLI to do ?
1. Rebuild the projects and docker images
2. Run the projects and docker images
3. Run Docker-compose configurations

Simple enough. 

### What do we need in order to be able to do that ? 
Well, it might depend on your needs but since services can be build and deployed individually by different teams (that's the whole point, really) I came up with following list:

1. `build` script for each service
2. `start` script for each service
3. `Dockerfile` for each service
4. `docker-compose.yml` files to run different configurations

### What environments are we going for ?
Now, we actually have **2** environments we want to build for: **local** and **docker**, for my project both will be build with **dev** configuration. To achieve that, I'll use **profiles** (like `maven` profiles):
1. `dev-local`
2. `dev-docker`

Which will determine what properties are going to be used in our services.

### Project directory structure
Having decided on that, it's time for the services folder structure. Ultimately, the structure I went for looks like this:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/65oprtimhwpm9z4qc8ug.png)

The first level is *language directory*. It serves as the **root** for the projects in that language, thanks to which all python services will use one *virtual environment*, java services will have common `pom.xml` and so on.

The second level is the service itself, containing `Dockerfile` for each service, since that is the standard.

The third level are the directories:
* *src* which contains the source files of the service
* *scripts* which contains `build.sh` and `start.sh` scripts for each service

Each script will handle the profiles on its own, the profile will be passed as a parameter, ex:

`build.sh dev-local`

Having that out of the way...

### In what language do we write the CLI ?

My first thought was to simply do it in *bash* scripting language since it comes by default in linux but, ultimately, I went with mix of *bash* and *Python* with `click` library. *Python* also comes with linux nowadays (at least with most popular choices) and it only lacks the `click` library, which is easily installed using `pip`.

Most of the CLI will be written in Python for ease of writing and the simple `start` and `build` scripts will be written in bash.

These are the sources I used to write a simple CLI:
* [pymbook](https://pymbook.readthedocs.io/en/latest/click.html)
* [click website](https://click.palletsprojects.com/en/7.x/)
* plus occasional google search which usually led to concrete subpage of [click website](https://click.palletsprojects.com/en/7.x/)
* and of course *bash* tutorials

### What the interface will look like ?

It can be done in a various different way, what I settled for, for now, is something similar to *docker* commands. I divided the CLI into 3 major commands, each with subcommand:
* local (build|start) (--services|-s) SERVICES
* docker (build|start) (--services|-s) SERVICES
* docker-compose
 * start - for now it only runs `docker-compose up` 
 * stop - for now it only run `docker-compose down`

For the version 0.0.1 of the CLI, I left the services config inside the script but I intend to move it to some `json` or `yml` file in the future. And, of course, the division I did might change in the future.

The source code for the CLI can be seen [here](https://github.com/DigitalCrafting/microservices-the-one-project/blob/master/cli/theonecli)

# Summary

If you decide to move from monolith to microservices, before you start coding, think how are you going to build and run them, both locally and to production. Thinks of it as the foundation of the whole architecture. Without it, the number of microservices would soon make the development unbearable and the project would crumble.

Hope you enjoyed this one and that it will help you :wink:
The source code for the whole project is on [my github](https://github.com/DigitalCrafting/microservices-the-one-project). See you in the [next part](https://dev.to/digitalcrafting/let-s-create-microservices-system-from-scratch-part-3-42hi).