A thousand deployments later...
Tags: fullstack, python, vue, astro
Published: 2019-07-29 06:08:12

---
title: A thousand deployments later...
published: true
description: The story behind my solopreneur project "arcsecond.io"
tags: full-stack, python, vuejs, astro
---

[Originally published [in my blog](https://onekiloparsec.dev/2019/07/28/a-thousand-deployments-later/)].


<p class="has-drop-cap">A bit more than four years after its inception, two years after its real start, a thousand  deployments, about 10 000 commits, more than 1200 unit and integration tests, more than 100 REST API endpoints, <a href="https://www.arcsecond.io">Arcsecond.io</a> is approaching v1.0. What a journey, <a href="https://onekiloparsec.dev/2015/12/29/the-story-of-6-years-of-development-of-iobserve/">again</a>!</p>



<p>The more I code arcsecond.io, the more I love it, and feel confidence for the years to come. Since the beginning, I knew this endeavour would last for multiple years, and not just be a hobby project. After all, I've been developing a <a href="https://apps.apple.com/us/app/iobserve/id424693907">macOS app called iObserve</a> for more than 8 years. Thus, I built Arcsecond.io with these long-term objectives: maximum reliability, permanent deployment capability, permanent clean and tested code. And of course: utmost care for the feedback of users and astronomers worldwide. For this, I knew I will have to learn and master whatever it needs, whatever its complexity or technological layer, and develop and master the permanent operations of a pretty nice full-stack software system. This article explains the important choices in the technical stack used in arcsecond.io.</p>



<p class="has-drop-cap">Let me start by showcasing a bit one thing I am pretty proud of today. Although I am very happy to have ported the core of my macOS app iObserve on the web (<a href="https://www.arcsecond.io/iobserve">check it out</a>), one of the coolest feature of arcsecond.io today is <strong>Worldwide Observing Activities</strong>, that is, the possibility to <strong>know in real-time</strong> what is being observed in the (largest but not only) observatories in the world, and some scientific satellites! </p>



<figure class="wp-block-image"><img src="https://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-27-at-14.47.31-1024x685.png" alt="" class="wp-image-455"/><figcaption>Live observing activities occuring worldwide and in space!</figcaption></figure>



<p>Even better: you can subscribe to it, and be notified right away if something matches your subscription. Say, you want to be notified when the HST starts an observation, or when an Programme is being performed for a given observer, or when a telescope instrument is being used, or when a region of the sky is being observed, etc. Boum. You've got mail! How cool is that? </p>



<ul class="wp-block-gallery columns-2 is-cropped"><li class="blocks-gallery-item"><figure><img src="http://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-27-at-20.59.21-1024x677.png" alt="" data-id="467" data-link="http://onekiloparsec.dev/?attachment_id=467" class="wp-image-467"/><figcaption>Example of a mail for a SWIFT observation</figcaption></figure></li><li class="blocks-gallery-item"><figure><img src="http://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-27-at-21.00.48-1024x733.png" alt="" data-id="468" data-link="http://onekiloparsec.dev/?attachment_id=468" class="wp-image-468"/><figcaption>Example of a mail for a La Silla Observatory observation</figcaption></figure></li></ul>



<p>To achieve this result, it is a long effort, and a rather lengthy list of decisions, optimizations, trials and errors. But the result is what I wanted: even if the user experience is not perfect (some links aren't always here, aren't always pointing to complete info etc), its potential is clearly enormous. Now, I have to "fill" it with more connectors.</p>



<p>Ok, let's dive in. <strong>We'll start with the backend, then move on with the infrastructure, the frontend and finally the code organisation.</strong></p>



<h2>The backend</h2>



<figure class="wp-block-image"><img src="https://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-27-at-17.31.59-1024x685.png" alt="" class="wp-image-462"/><figcaption>Ok, not the most beautiful webpage of the Internet. But its beauty is what's inside.</figcaption></figure>



<p>The backend server is written with <a href="https://www.djangoproject.com">Django</a> 2.2 and Python 3.7 Originally, arcsecond.io started with Django 1.5 or something, and Python 2. The migration to Python 3 was made more than a year ago, and it was clearly a good choice. Almost no third-party libraries broke during the migration. At the beginning, I evaluated the possibility to use <a href="https://palletsprojects.com/p/flask/">Flask</a> instead of Django. But Django comes with a lot of things built-in, in particular its automatic admin portal, which proved to be invaluable over time.</p>



<p>The design choice of the backend was to make a RESTful purely data-centric server. Hence, the rather obvious choice was to use the fantastic <a href="https://www.django-rest-framework.org">Django REST Framework</a>. It is an essential part of arcsecond.io. It is not the purpose of this article to explain REST principles, but simply let me say <em>it is a mindset</em>, and the DRF allows you to deploy this way of doing with lots of ease. </p>



<p>One of the key reason of using Python in the backend was obviously its seamless integration with scientific and astronomical libraries. In particular, it wouldn't be possible without <a href="http://astropy.org/">AstroPy</a> (and its affiliated packages) and <a href="https://numpy.org">NumPy</a>. Once arcsecond.io is making some benefits (hopefully some day!), I will redirect part of it to help fund these projects.</p>



<p class="has-drop-cap">The projects declares 52 direct dependencies (among which <code>django-cors-headers</code> for managing HTTP headers easily, <code>django-cryptography</code> to encrypt some model fields, <code>django-otp</code> to protect the production admin route with 2FA, <code>django-allauth</code> and <code>django-rest-auth</code> to handle the complex process of auth, <code>django-storages</code> to handle multiple storages remotely, <code>psycopg2</code> to talk to the PostgresQL DB, <code><a href="https://2.python-requests.org/en/master/">requests</a></code> and <code><a href="https://pyvo.readthedocs.io">pyvo</a></code> to ease the writing of the multiple connectors...). </p>



<p>More importantly, the backend relies on<code><a href="http://www.celeryproject.org">celery</a></code> and <code><a href="https://github.com/django/channels">channels</a></code> (with the ASGI <code><a href="https://github.com/django/daphne">daphne</a></code> server), two libraries of crucial importance.</p>



<p>Celery is a distributed task queue. When I was a newbie at the very beginning of this full-stack journey, I asked myself: Why would I need such a thing? Until I realised that a lot, if not most, of the activity of a backend server could be different from serving HTML pages or serialized data. In arcsecond.io the activity is not stable enough to say if the background Celery worker is more important than the Dahne server. However, with permanent parsing of Data Archives, such as ESO's,  Gemini's and HST's, as well as Satellite schedules such as Swift's, and the creation of a permanent flux of observing activities, a great part of the load of the "arcsecond.io" backend is occuring in the background tasks handled by Celery.</p>



<p>The second library is the official Django-supported project to handle more than just HTTP requests, but also WebSockets. And even if we have only one read-only WebSocket route opened in arcsecond.io so far (the live activities here: <a href="wss://api.arcsecond.io/activities/live">api.arcsecond.io/activities/live</a>), the migration from a simple WSGI pure-HTTP server (it was gunicorn for a longtime) to <code>channels</code> and <code>daphne</code>, was not a simple one. We had 2 main difficulties. One was that the <a href="https://github.com/django/channels_redis"><code>channels_redis</code></a> library (providing the backing store for the <code>channels</code> layer of the server) <a href="https://github.com/django/channels_redis/issues/125">had a bug</a> that was causing the creation of numerous connections in our infrastructure, way beyond what is expected. For a long time we had various errors pointing in various directions, without knowing the true cause of the problem. But once fixed (recently), the result was quite impressive!</p>



<figure class="wp-block-image"><img src="https://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-07-at-22.12.07-1024x766.png" alt="" class="wp-image-457"/><figcaption>Evolution over time of the number of Redis connections used by our channels / daphne server. See how dramatic the effect of a bugfix had.</figcaption></figure>



<p>The second difficulty was to understand the underlying mechanism of WebSockets, and how one could provide automatically and similarly serialized JSON data. In other words, how to use our carefully crafted REST serializers to format the data in the WebSocket tube the same way it was for HTTP requests. We use the open-source (very early-stage but clean and well written) <a href="https://github.com/hishnash/djangochannelsrestframework">djangochannelsrestframework</a> library for that.</p>



<p>As for the tests, we use the <a href="https://pytest.org/en/latest/">pytest</a> library, with the standard python mocks when necessary, and these additional and <em>very</em> useful libraries:</p>



<ul><li><a href="http://stevepulec.com/freezegun/">freezegun</a> : to easily write tests anywhere you want in time,</li><li><a href="https://factoryboy.readthedocs.io/en/latest/">factory_boy</a> : to easily create dozens of instances during your tests setup,</li><li><a href="https://github.com/edwinlunando/django-naomi">django-naomi</a> : to easily see in a browser during development what the emails your backend sent look like,</li><li><a href="https://github.com/kevin1024/vcrpy">vcrpy</a> : to very easily replay the "cassettes", that is, replay real network requests.</li></ul>



<h2>The infrastructure</h2>


<!-- wp:image {"id":464} -->
<figure class="wp-block-image"><img src="https://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-27-at-17.56.29-1024x225.png" alt="" class="wp-image-464"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Arcsecond.io has two servers: one for the backend, and one for the frontend. The two are deployed on the well-known <a href="https://en.wikipedia.org/wiki/Platform_as_a_service">PaaS</a> <a href="http://heroku.com/">Heroku</a>. Why Heroku? Well, because a colleague of mine, when I was working back in Switzerland, just mentionned it's the easiest way of deploying a server. And it is still mostly true today, I am very happy of this decision (even if today I would consider using a serverless stack, amazed by the products of <a href="https://zeit.co">zeit.co</a> for instance).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Some of the many advantages of using Heroku:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Large marketplace of one-click-installation add-ons for bazillions of auxiliary services (storage, logging, monitoring...)</li><li>Deploy with a simple <code>git push</code>.</li><li>A very easy-to-use CLI allowing you to interact and connect to your servers.</li><li>A very easy way to have your hostnames using certificates (using <a href="https://letsencrypt.org">letsencrypt</a>)</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>So here is the infrastructure of arcsecond.io:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>A Daphne server (one Heroku dyno - 7$/month),</li><li>A Celery instance for a background worker (one Heroku dyno - 7$/month),</li><li>A celery-beat scheduler (one Heroku dyno - 7$/month),</li><li>A production <a href="https://www.postgresql.org">PostgresQL</a> DB instance, automatically backed up every hour  - 9$/month,</li><li>A basic <a href="http://papertrailapp.com/">Papertrail</a> instance for logging,</li><li>A basic <a href="https://sendgrid.com">Sendgrid</a> instance for sending emails,</li><li>A basic <a href="https://redislabs.com">RedisCloud</a> instance for the backing store of the Daphne server,</li><li>A basic Heroku Redis instance for the HTTP cache,</li><li>An <a href="https://expressjs.com">Express.js</a> server for the frontend (one Heroku dyno - 7$/month),</li><li>Multiple Amazon S3 buckets for the storage of the data (price follows data volume),</li><li>The domains are bought on the french cloud provider <a href="http://ovh.com">ovh.com</a> (~40$ / year for a <code>.io</code> domain),</li><li>I used to pay the Pro level of <a href="https://github.com">GitHub</a> to store private repositories, but then it became free,</li><li>There are also 2 other free Heroku dynos used for the staging environnements (one for the back, one for the front).</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Total price: about 40$ + S3 storage (+ a bit of googlemaps...) every month! One could say it's not much and it's a lot at the same time. 40 bucks for all these services is not much indeed. But these are mostly first or second-level entries of the services, and as soon as some money flows in, I will move up the level of some of them. <strong>Thanks to Heroku it will be as simple as moving a cursor!</strong> It's "a lot" in the sense that for now, arcsecond.io is totally free for everyone. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There is one key part of the infrastructure that is not mentionned yet: <a href="http://circleci.com/">CircleCI</a>. The code of the backend is never pushed in production directly. It always goes through the automatic testing and deployment by CircleCI on a staging environment, then on production.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>As for the frontend, the code is not going through an external CI service, for one  reason. The Javascript code is split in various private <a href="https://nodejs.org/en/">NodeJS</a> modules (see below), and I couldn't wrap my head around the idea to pay an additional Amazon EC2 instance to hold a <a href="http://verdaccio.org">Verdaccio</a> server, or use the premium mode of <a href="http://npmjs.com/">npmjs.com</a> "just for 5 small JS libs". So I use a Verdaccio server that runs on my iMac at home.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So far, I count about 750 deploys for the backend, and 260 for the frontend.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One couldn't finish this part without mentionning the use of the fantastic crash-reporter <a href="http://sentry.io/">Sentry</a>. The richness and the ease of use of this tool is really great, and its help to find and correct bugs is invaluable. Even if Django comes with an automatic emailing when an error occurs (all 500 status code ends up in my Inbox), Sentry helps you read the code stack and identify more clearly what happened. And it also provides very easy management of problems, grouping identical ones, alerting when closed ones reappear etc.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph {"dropCap":true} -->
<p class="has-drop-cap">This "software system" is in production since more than a year now, and has handled already quite a few visit spikes (hello <a href="https://www.producthunt.com/posts/arcsecond">ProductHunt.</a>..). It is ready to support a much larger load without having to change its underlying organisation.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>The frontend</h2>
<!-- /wp:heading -->

<!-- wp:image {"id":465} -->
<figure class="wp-block-image"><img src="https://onekiloparsec.dev/wp-content/uploads/2019/07/Screenshot-2019-07-27-at-17.59.13-1024x685.png" alt="" class="wp-image-465"/><figcaption>I never imagined I could build up such a complex view in the browser.</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The frontend is a <a href="https://vuejs.org">Vuejs</a> <a href="https://en.wikipedia.org/wiki/Single-page_application">Single-Page Application</a> (SPA). The discovery of Vue.js was simply a life-saver. Originally, the frontend of arcsecond.io was written with AngularJS, the first version of the now-called <a href="http://angular.io">Angular</a> frontend framework (wich is one of the  Trilogy, with Vuejs and <a href="https://reactjs.org">Reactjs</a>). But the codebase was enormous for the result, and hard to maintain. I tried at least 5 times to migrate to Angular 2, but always failed. And the JSX language of Reactjs appeared to me like a frightning thing and a prohibitive cost to pay (I still find it is a terrible idea for readibility). </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Then came Vuejs. Simple. Easy to learn. Natural. No cruft, no crazy syntax (simple <code>v-</code> prefixes like <code>v-if</code>, compared to <code>*ngIf</code> for Angular – really, guys?), a textbook example of how a documentation must be written and displayed (yeah, Python my friend, your documentation is damn difficult to <em>work with</em>). Then came the v3 of the Vuejs <a href="https://cli.vuejs.org">Command-Line Interface</a>. Oh man. Even the migration of the project from its original shape to the Vuejs-CLI v3 organisation was a breeze. And thanks God, it abstracts away all <a href="https://webpack.js.org">Webpack</a> insane complexity (for non-frontend developers, webpack is crazy difficult to learn). To me, Vuejs is what the web development should have always been.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The tests are written with <a href="https://jestjs.io">Jest</a>, and <a href="https://vue-test-utils.vuejs.org">vue-test-utils</a>. I have made several attemps with <a href="https://www.cypress.io">Cypress</a> to make end-to-end tests, but I had trouble installing a stable version on my iMac. Moreover, one could go a long way with Jest, so I am concentrating on unit tests for now. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One last piece that could be categorized as "frontend" in the sense of being a consumer of the APIs and their data is the Arcsecond Command-Line-Interface (CLI), open-source and <a href="https://github.com/arcsecond-io/cli">freely available</a> on GitHub. For a long time I thought it was enough to provide REST APIs of interesting data, I would not commit myself to another Python script / module like every scientist on this planet... But reality is stronger, and I am very happy to have build such tool. It helps integrate api.arcsecond.io more easily into custom workflows.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>The organisation of the code </h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To that day, the backend repository contains 4200 commits, and the frontend repository 4900. Yes, I tend to make a lot of microcommits. The project started for real only two years ago, when I was able to really craft a serious beyond-hello-world frontend app for the long-term. The first commit of the Vuejs frontend was made on Tuesday February 28, 2017. But the first commit of the Python backend was on Tuesday Apr 15, 2014! (Always good to start a project on Tuesdays.)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In the code, I have 295 Vuejs components (<code>.vue</code> files) that allow to build a webapp, which weights 3.1 MB in the end. It is not using Server-Side-Rendering (<a href="https://ssr.vuejs.org">SSR</a>) so far, and it's pretty difficult to transform it right now (I have special handlers for dynamic routes for Organisation Portals...). But I intend to increase at least the number of pages using prerendering (see SSR link above). On the backend, even if there is only one "arcsecond" Django monolyth app, the code is split into 24 different Django subapps (which all have their own models, serializers, views, urls, sometimes tasks and/or connectors...). </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Yes, the backend app is a monolyth, even if the industry is breezing with microservices. Actually, given the amount of relationships between all the models, the real work of development and its operations appeared to me <em>a lot easier</em> within a monolyth, rater than distinct microservices. I am alone (Eric my friend and partner mostly helps with advices, tests and comments), and arcsecond.io is a kind of product that requires an large amount of development first before starting to be useful. I just can't spend more time to something else than development.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For the IDEs, I use the pro version of <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> since the beginning for the backend (~80$/year). For the frontend I tried <a href="https://www.sublimetext.com">Sublime</a>, <a href="https://atom.io">Atom</a>, <a href="http://vscode">VSCode</a>, <a href="https://www.jetbrains.com/webstorm/">WebStorm</a> over the years, and finally use PyCharm too, since it has an excellent Vuejs plugin. Finally, I couldn't commit so fast and so easily, and find the history of the changes so quickly without the excellent Git client <a href="https://www.git-tower.com/mac">Tower</a> (~65$/year –&nbsp;that's quite a lot just to make commits... but I see Tower more like both a debugging and a management tool at the same time).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The GitHub repositories (repos, in the developers jargon..) are organised as follow:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>A single arcsecond-back repo for the backend, sometimes with the help of forks of open-source libraries,</li><li>An arcsecond-front repo for the frontend, but using 6 private repos for custom Vuejs libaries,</li><li>The <a href="https://github.com/onekiloparsec/AA.js">AA.js open-source repo</a> for the Astronomical Algorithms,</li><li>The <a href="https://github.com/arcsecond-io/cli">Arcsecond CLI</a> open-source repo,</li><li>The <a href="http://github.com/arcsecond-io/public-issues/issues">public-issues</a> repo to collect and work on public issues, found by my or users.</li><li>The <a href="https://github.com/arcsecond-io/standard-stars-catalogues">standard-stars-catalogues</a> repo for holding an open-source and reliable source of standard stars catalogues.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>That's it. Thanks for reading down to these lines! I hope you had learn a bit what it takes to start a "small but growing" project that grabs quite a large fraction of your daily awake time.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>–– Clear skies to all!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>P.S. The beautiful image of the starry night is from G. Hüdepohl (<a rel="noreferrer noopener" href="http://atacamaphoto.com/" target="_blank">atacamaphoto.com</a>)/ESO. </p>
<!-- /wp:paragraph -->