Ghost in A shell - part III : to Production, with performance
Tags: tutorial, ghost, productivity, docker
Published: 2018-06-04 20:44:59

---
title: Ghost in A shell - part III : to Production, with performance
published: true
tags: tutorial,ghost,productivity,docker
canonical_url: https://manu.breton.ch/blog/deploying-ghost-part-3-production-performance/
---

 ![Ghost in A shell - part III : to Production, with performance](https://manu.breton.ch/blog/content/images/2018/06/Ghost-in-A-shell.png)


I am actually very happy with the outcome of the [previous blog](https://dev.to/ebreton/ghost-in-a-shell---part-ii---https-is-the-norm-1jj4) for _my_ production: it has a proxy, it serves on HTTPs, sqlite3 is not so bad after all... and for my 7.5 users, it is just _great_.

Your definition of "great" might be slightly different...

> What about defining _production greatness_ as a setup that will give satisfaction based on:
> 
> 1. performance (covered in this blog)
> 2. ease of maintenance (covered in next blog)
> 3. reliability (not sure that this will fit into the scope of the Tutorial)

Soooo... let's start with performance.

# Philosophy

I am **not** going to dive into the depth of the performance theme, too many things to cover. Let me just suggest this path: to focus on setting up a toolbox that will allow you

1. to run performance/load tests easily (one command sounds good, as usual; it will be `make gatling`)
2. to view the output metrics easily (like in a web browser, thank you gatling!)

The idea behind is to base our decisions upon relative metrics, which means we are going to compare the results of the tests before and after making some changes (idealy: any change). Changes will be validated according to the variation of the results

But enough talking, let's make sure we are ready to go...

# Pre-requisites

## a) from the previous episodes...

I will make the assumption that you have been through my two first blogs:

1. [Part I : localhost](https://dev.to/ebreton/ghost-in-a-shell---part-i--localhost-5he9), that gets you familiar with the main commands of the _Makefile_, and focus on localhost
2. [Part II : HTTPs is the norm](https://dev.to/ebreton/ghost-in-a-shell---part-ii---https-is-the-norm-1jj4), that shows you how to spawn ghost instances fearlessly on one server

There are no new dependencies, which means you are good to go with `docker`, `make` and the two repos [prod-stack](https://github.com/ebreton/prod-stack) and [ghost-in-a-shell](https://github.com/ebreton/ghost-in-a-shell).

## b) if you wish to send emails (probability is high)

The Ghost Team makes some recommendations to send email, one solution being to [leverage MailGun](https://docs.ghost.org/docs/mail-config#section-mailgun). Follow the link, and the steps behind this link.

That will give you credentials to send emails through their platform. Those credentials will go into _./etc/prod.env_:

```
$ cp etc/prod.env.sample etc/prod.env
$ vi etc/prod.env
  ...
  # see https://docs.ghost.org/docs/mail-config#section-mailgun
  # you will find an updated screencast to understand exactly where to find these details
  MAILGUN_LOGIN=(Sandbox) default SMTP Login
  MAILGUN_PASSWORD=(Sandbox) default Password

```

Those variables will be used when spawning an instance in "production mode", as we can see within the Makefile:

```
prod:
    ...
    -e mail__transport=SMTP \	
    -e mail __options__ service=Mailgun \
    -e mail __options__ auth__user=${MAILGUN_LOGIN} \
    -e mail __options__ auth__pass=${MAILGUN_PASSWORD} \
    ...

```

Because we are now dealing with a 'production' environment, the setup is not as easy as it was when using your SMTP server... hence a few "surprises" on the way:

- MailGun will ask you for a credit card, even if you keep using their free plan
- With the default login and password, you will have to declare all the recipients you wish to send emails to (and they will have to approve receiving those emails)
- To go around this limitation, you will have to set up your domain within MailGun. (and you will need to modify your DNS zone for this)

Therefore, if you prefer not using MailGun, you can remove the 4 lines above from the `make prod` command, or adapt it to another service provider (let me know how it went if you do, please!)

Next step is setting up the DB...

# Switching the DB

## a) objective

We wish to make use of MySQL instead of sqlite3, because it is [recommended](https://docs.ghost.org/docs/install#section-install-mysql) by the Ghost team. Fair enough.

## b) setup

The prod-stack already runs a MariaDB in a container, as defined in the [docker-compose.yml](https://github.com/ebreton/prod-stack/blob/master/docker-compose.yml#L29) file:

```
  db-shared:
    image: mariadb:latest
    container_name: db-shared
    restart: always
    env_file:
      - $PWD/etc/db.env
    volumes:
      - db_data:/var/lib/mysql

```

There is only one piece of customization, the _db.env_ file which holds the MYSQL\_ROOT\_PASSWORD. This file has been generated for you when you installed the prod-stack by running `make`. Here is the reminder:

```
# from prod-stack root directory
$ make
cp etc/traefik.toml.sample etc/traefik.toml
cp etc/db.sample.env etc/db.env
sed -i s/password/auMr9jsMrPnX0Oatb9j4yX2AYBN2jTQV/g etc/db.env
Generated etc/db.env
...

```

Well, I guess you know what is coming: copy the value from _prod-stack/etc/db.env_ and paste it in your _ghost-in-a-shell/etc/prod.env_ file. It now should look like:

```
# password set for db-shared, as defined in prod-stack/etc/db.env
MYSQL_ROOT_PASSWORD=auMr9jsMrPnX0Oatb9j4yX2AYBN2jTQV

# see https://docs.ghost.org/docs/mail-config#section-mailgun
# you will find an updated screencast to understand exactly where to find these details
MAILGUN_LOGIN=postmaster@your.domain.com
MAILGUN_PASSWORD=your-key

```

Done? You are ready to make use of a MariaDB instead of sqlite3 thanks to the following line in the Makefile.

```
prod:
    ...
	-e database__client=mysql \
	-e database __connection__ host=db-shared \
	-e database __connection__ user=root \
	-e database __connection__ password=${MYSQL_ROOT_PASSWORD} \
	-e database __connection__ database=${NAME} \
    ...

```

As a note, you can see that the connection to the DB server is hard-coded: it is made through the container name `db-shared` which is defined in the _prod-stack/[docker-compose.yml](https://github.com/ebreton/prod-stack/blob/master/docker-compose.yml#L29)_ as seen previously.

# `make prod`

In the two sections above, we have created a _etc/prod.env_ file with three environment variables. This allow us to call `make prod`, which creates a ghost instance:

- behind the Nginx+Traefik proxy (same as with `make qa`)
- accessible through HTTPs (same as with `make qa`)
- using MailGun to send emails ( **new** )
- using a MariaDB instead of sqlite3 ( **new** )

Because we are in a relative mindset (remember the philosophy?), we are going to compare our new set up, performance-wise, with the old one...

1. define a new blog NAME for the exercise: `export NAME=fire-at-blog`
2. spawn the instance with sqlite3: `make qa` (we will use the default content to keep it simple)
3. measure performance (and produce our first report): `make gatling`

![Ghost in A shell - part III : to Production, with performance](https://manu.breton.ch/blog/content/images/2018/06/with.sqlite3.png)

1. start again same blog using MariaDB: `make stop prod` (you could also go for `make stop prod logs` target to follow the logs and see when the instance is ready)
2. measure performance again: `make gatling`

![Ghost in A shell - part III : to Production, with performance](https://manu.breton.ch/blog/content/images/2018/06/with-mariadb.png)

1. compare the two reports written in the folder _./gatling-results_

We do not really care about the absolute values of any of the two reports. We are interested by the differences. Whatever the expectations were.. we now have the detailed figures with one command. And we can access directly in our browser the reports.

For illustation purpose, I have only shown the high level figures for requests and active... but you have much more (thank you [gatling](https://gatling.io)!)

Of course, that will not alone make a decision. Let's not forget that we also want ease of maintenance (sqlite3 might be better here), and reliability (MariaDB gets several good points there)

> Most important: we have a clear view on the performance of one solution vs the other

# `make gatling`

The magic in the section above come from the command `make gatling`, which is actually quite simple:

```
gatling:
	docker run -it --rm \
		-v $(shell pwd)/etc/gatling-conf.scala:/opt/gatling/user-files/simulations/ghost/GhostFrontend.scala \
		-v $(shell pwd)/gatling-results:/opt/gatling/results \
		-e JAVA_OPTS="-Dusers=${GATLING_USERS} -Dramp=${GATLING_RAMP} -DbaseUrl=${GATLING_BASE_URL}" \
		--network=proxy \
		denvazh/gatling -m -s ghost.GhostFrontend

```

We are using a docker container which encapsulate all the (Java) layers needed by gatling.

- `-v $(shell pwd)/etc/gatling-conf.scala ...` and mounts a few volumes for input (scenario file)
- `-v $(shell pwd)/gatling-results ...` mounts the volume for the output.
- `-e JAVA_OPTS="-Dusers=${GATLING_USERS} ...` adapts the script according to the environment variables (more details below)
- `--network=proxy` allows the container to directly load-test its beloved siblings (e.g. the blog instances)

The scenario file, _./etc/gatling-conf.scala_, has 4 sequences (self explicit):

```
atOnceUsers(nbUsers),
constantUsersPerSec(5) during (myRamp seconds),
rampUsers(nbUsers) over (myRamp seconds),
heavisideUsers(10) over (myRamp seconds)

```

You can see that two variables (`nbUsers` & `myRamp`) are available to give you a bit of flexibility in the loads you are expecting. Default values are provided in the Makefile and can ve overridden as usual.

```
GATLING_BASE_URL?=${PROTOCOL}://${NAME}:2368/${URI}
GATLING_USERS?=3
GATLING_RAMP?=5

```

By default, the galing container which targets the container that runs your blog instance. (hence the `--network=proxy` parameter in the gatling command). Here is a more realistic example:

`GATLING_BASE_URL=https://your.qa.com/blog GATLING_USERS=20 make gatling`

You might wonder what will be the "full URL(s)" since we make use of a "...\_BASE\_URL". You will have to change the configuration file here (or use another one). They are so far coded as the following array, which is use as many time as necessary (thanks to the last bit `.circular`).

```
  val uriFeeder = Array(
    Map("URIKey" -> s"welcome"),
    Map("URIKey" -> s"the-editor"),
    Map("URIKey" -> s"using-tags"),
    Map("URIKey" -> s"managing-users"),
    Map("URIKey" -> s"private-sites"),
    Map("URIKey" -> s"advanced-markdown"),
    Map("URIKey" -> s"themes")
  ).circular

```

By the way, we could have chosen some other strategy than `.circular` (e.g. `random`), but I have chosen a repeatable + predictable strategy here, for the purpose of being able to compare the results with a controlled scenario.

Let's apply this knowledge to load-test our proxy

# Playing a bit with Nginx

Keeping in line with this methodology, we could have a closer look at the proxy (Nginx + Traefik) and compare our configuration with another one with NGinx cache... We will use the configuration _prod-stack/etc/000-proxy-with-cache.conf.sample_ for this purpose:

```
proxy_cache_path /var/cache levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=1g;
    
server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;

  location / {
    proxy_pass http://traefik/;
    proxy_set_header Host $host;
    proxy_buffering on;
    proxy_cache STATIC;
    proxy_cache_valid 200 1d;
    proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
  }
}

```

The tricky part is

- to continue playing around locally, while
- making gatling access the blog through NGinx, and not directly from his container to the other container...

For this purpose, we are going to use the `GATLING_BASE_URL` variable along with the internal IP of our computer. This will make the gatling container query the blog through the host (which means through Nginx and Traefik) instead of heading straight at the ghost container.

> Watch out: 127.0.0.1 will not work (neither localhost) since the container will query itself in that case, not the ghost instances.

Let's say our internal IP is 192.168.178.93... here is the process:

1. run a first container without cache: `NAME=no-cache-blog make qa logs`
2. once launched, run a gatling container: `NAME=no-cache-blog GATLING_BASE_URL=http://192.168.178.93/no-cache-blog make gatling`
3. activate Nginx cache in the prod stack
```
$ cd /your/path/to/prod-stak/etc/conf.d
$ rm 000-default.conf
$ cp ../000-proxy-with-cache.conf.sample 000-proxy-with-cache.conf

# at this point you can stop and start the stack again, 
# or simply go inside the nginx container and reload the conf:
$ docker exec -it nginx-entrypoint bash
root@bef39f4ad198:/# nginx -s reload
... signal process started

```
4. run a second container with cache: `NAME=with-cache-blog make qa logs`
5. once launched, run gatling again: `NAME=with-cache-blog GATLING_BASE_URL=http://192.168.178.93/with-cache make-blog gatling`
6. look at the results (no cache on left hand side, with cache on right hand side)

![Ghost in A shell - part III : to Production, with performance](https://manu.breton.ch/blog/content/images/2018/06/with-or-without.png)

Performances are better with cache (as expected), but content could now "lag" and that could impact negatively the user experience... On the other end it could also impact positively the ease of maintenance, allowing us minor downtime of the blog itself (if well cached).

Again, the final choice will depend on you. At least, you will have metrics to compare the performances of the various solutions.

# Keep in mind, and avoid...

Avoid changing multiple criteria when comparing performances.

For instance, doing the following comparison is a bad idea :

| `make dev` | `make qa` | `make prod` |
| --- | --- | --- |
| some performance | other perf. | yet-another perf. |

The results are actually useless because you have multiple changes between the different sets of configurations: you are changing at the same time the DB, the proxy, and the email settings (potentially some cache layer). Therefore you will be blind of the individual effects of each of them.

# Conclusion

You now can run three modes of ghost instances, for development, testing or production:

| Command | Description | Remarks |
| --- | --- | --- |
| `make dev` | straightforward (local) installation if you simply wish to bridge a container port on your host (3001 by default) | also known as `make` |
| `make qa` | The former being not really convenient if you wish to serve on standard ports (80 or 443), and if you want anyone to access your blog easily, this command sets up a traefik router | recommended usage of [prod-stack](https://github.com/ebreton/prod-stack) |
| `make prod` | Same as above, using a MariaDB instead of SQLite and an email provider (Mailgun) | requires a (free) account at Mailgun |

The (more) important get-away of this blog though, is the simple command `make gatling`, and using it to validate (small and well identified) changes along the way.

A good practice would actually be to integrate the command in the continuous integration workflow, and to raise alerts if the performance goes down

In the next blog, I will introduce a few helpers for the maintenance... in order to keep a bit of order in all the instances we can now spawn here and there