RetroFeed Work Log #2 - Basic Package Setup
Tags: raynor, rpc, retrofeed, worklog
Published: 2019-01-20 15:26:11

---
title: RetroFeed Work Log #2 - Basic Package Setup
published: true
tags: raynor,rpc,retrofeed,work-log
canonical_url: https://horia141.com/retrofeed-2.html
series: "RetroFeed Dev Log"
---

We left off in [Idea & Basic Setup](https://horia141.com/retrofeed.html) with a project idea and a general tech plan. The current and next two articles will cover a bunch of foundation-level issues for any Internet application project. More precisely we’re going to:

- Setup a repository with a basic structure, establish various processes and link the some developer-oriented services. This will be covered in this article.
- Setup the application part of the project. We’ll deal with the web framework, setting up sessions, migrations, authentication, Docker, server-side rendering etc. There’s a lot of meaty stuff here, and even if the application itself won’t _do_ that much at the end, a lot of the bricks will have been laid. This will be covered in the next article.
- Setup the infrastructure part of the project. We’ll deal with setting up GCP, constructing a “runtime environment” for deployment and building actual build pipeline to make it happen. This will be covered in the third and last article of this section.

At the end of the process we’ll basically have a very simple web application, with probably two screens - a “hello world” main page, and a “logged in” page, showing the name of the user. However I’m expecting all of the processes & foundational issues to be decided and solved. From how to run tests locally, to monitoring or rolling back a bad release. As mentioned in the previous article, expect _modern tools, but conservative choices_.

Since that’s out of the way, let’s start with the proper “part one”.

You can check out the code [here](https://github.com/horia141/retrofeed/tree/v0.0.2). It is hosted on GitHub and it’s a single repository. It’s tagged as `v0.0.2`. I’ll keep referencing exact versions in the future as well, rather than just any commit.

Recapping the [last post](https://horia141.com/retrofeed.html), the main architectural choices are:

> The software stack is going to be Node with TypeScript for the backend, NestJs with Express as a web framework, Raynor with Raynor RPC for NestJs for RPC-ish type stuff. I’ll use Postgres as a datastore with Redis for caching. I’ll use Knex.js as a data access library. I’ll use React & it’s ecosystem for the frontend bits. The local environment will be run off Docker, with docker-compose. Deployment will be done on Google’s cloud. I’ll use the managed SQL and Redis offerings, and run Docker in production. But I won’t go the Kubernetes route, and simply run the services on a compute node with docker run. Infrastructure as code via terraform, of course. Other services I’ll be using are GitHub for code hosting, Travis.Ci for builds, NPM for package hosting, Codecov.io for coverage reports, Cypress.io for end to end testing, Auth0 for easy authentication, and GitHub’s API for repository stuff. The code’s going to be released under the MIT license and I’ll try to be as open as possible about the things I’m doing.

A bunch of these can already be seen in the repository at this stage. The file structure looks something like this:

```
.
├── .gitignore
├── .nvmrc
├── .travis.yml
├── LICENSE.txt
├── README.md
├── docker-compose.yml
├── package.json
├── scripts
│ ├── lint.sh
│ ├── send-coverage.sh
│ ├── serve-watch.sh
│ ├── serve.sh
│ ├── test-watch.sh
│ └── test.sh
├── src
│ └── index.ts
├── test
│ └── index-test.ts
├── tsconfig.json
├── tslint.json
├── tsnyc.json
└── yarn.lock
```

There’s not much to it in fact LOC wise. `.gitignore`, `README.md` and `LICENSE.txt `are pretty boilerplate. We use `yarn` as a package manager, `tslint` with it’s default and quite strict ruleset, `istanbul`/`nyc` for coverage reports etc.

I’ll highlight some of the more interesting files and choices, but feel free to poke around to see how things tick.

The `package.json` looks something like this:

```
{
    "name": "retrofeed",
    "version": "0.0.2",
    "description": "Manage updates for your project's dependencies",
    "author": "Horia Coman horia141@gmail.com",
    "scripts": {
        "serve": "./scripts/serve.sh",
        "serve:watch": "./scripts/serve-watch.sh",
        "test": "./scripts/test.sh",
        "test:watch": "./scripts/test-watch.sh",
        "lint": "./scripts/lint.sh"
    },
    "devDependencies": // Not shown
    "dependencies": // Not shown
}
```

The `scripts` section is interesting. It only references the `./scripts` directory. This is where I hold all of the _actions_ one can do as a developer. And ideally, everything you’d want to do - from running a test, to doing a deploy, to regenerating [LetsEncrypt](https://letsencrypt.org/) certificates will have a script here. The current structure is:

```
scripts/
├── lint.sh
├── send-coverage.sh
├── serve-watch.sh
├── serve.sh
├── test-watch.sh
└── test.sh
```

Notice that not all of them are mapped to the `scripts` key. So you can say `yarn lint`, but not `yarn send-coverage`. This is intentional. It highlights the ones you should be using locally. And while you could run the others, those are meant for the CI server or the production environment. Conversely however, there shouldn’t be anything you’d want to run which wouldn’t have a script.

For example, `serve-watch.sh` looks like (a bit after `v0.0.2`):

```
#!/bin/bash

set -e # Fail on first command

source scripts/setup-env.sh # Sets up environment variables

npx ts-node-dev --inspect=${INSPECT_PORT} -- src/index.ts # Does the action
```

Notice that `npx` is needed, cause the script is meant to also be run outside of `yarn serve:watch`.

The code itself is minimal. There’s just one `src/index.ts` file with a single function, and a corresponding `test/index-test.ts` file with tests for it. We use `mocha`, `chai`, `sinon` etc for testing, since these are quite standard in the JavaScript world. More code is going to come later.

A thing to notice is that there’s no `compile.sh` or `build.sh` script. TypeScript tooling is good enough that there’s a equivalents of `node` (`ts-node`) or `node-dev` (`ts-node-dev`) which allow skipping the classical compilation step entirely. So this gets rid of a lot of complexity and can’t really see a downside to it – perhaps performance in production. But I doubt it.

One interesting thing is the set of integrations we have. They’re in turn:

- [Travis.CI](https://travis-ci.org/horia141/retrofeed) as a build tool. Every commit in every branch results in a build. But only tagged commits with a version number get pushed to npm. Checkout the [`.travis.yml`](https://github.com/horia141/retrofeed/blob/v0.0.2/.travis.yml) for the build configuration. It’s pretty standard - some setup, a run of `yarn lint` and `yarn test`, a test success action of sending coverage results to coveralls, and a tagged deploy action of pushing to NPM.
- [NPM](https://www.npmjs.com/package/retrofeed) as a package repository. Packages are pushed automatically from Travis, on a green and tagged build.
- [Coveralls.io](https://coveralls.io/github/horia141/retrofeed) for managing coverage reports. There’s a bit of magic involved in getting the _right_ data with the setup. Every successful build send coverage data, and at some point tests are going to fail if coverage isn’t good enough.
- [Cypress.io](https://dashboard.cypress.io/#/projects/a4ib2i/runs) for end to end testing. This integration isn’t yet visible here, since there’s nothing to end to end test.
- [Ngrok](https://ngrok.com/) for making the current server instance available publicly. Again, this isn’t setup here yet, but it’s just a neat tool.

Anyway, that’s it for now. As I’ve said, not much happening, but a lot of preparation for it to happen. In the next parts we’re going to actually have something serving as a web application, and a full cloud deployment for it. See you soon!

Checkout all the articles in the series [here](https//horia141.com/retrofeed.html).