Polysemy, one year later
Tags: haskell, development, architecture, programming
Published: 2021-09-10 15:29:17

# Polysemy, one year later
One year ago, we began the development of Hetchr. We made a number of technical and architectural decisions, and this blog post intends to discuss one of them: the adoption of [polysemy](https://hackage.haskell.org/package/polysemy).

This article does not aim to explain how _polysemy_ works, but in short, it's a library that allows algebraic effects composition. While using [`Monad transformers`](https://www.fpcomplete.com/haskell/tutorial/monad-transformers/) forces each layer to be composed one with another, Polysemy provides a Monad that is responsible for the composition, letting effects stay independent from one to another.

## Context
Hetchr is a SaaS product that centralizes the key features of many collaborative development tools (Github, Jira, Gitlab, and so on).

From a technical point of view: we talk to external APIs and we store metadata over AWS.

## Motivations
From the beginning, we planned to support a large number of different APIs, leading to various choices:

* Having a big monad (or stick with `IO`) for everything: this is an issue in terms of expressivity, maintainability, and would make programmatic tests difficult and slow.
* Having a [`mtl`](https://hackage.haskell.org/package/mtl)-style Monad transformer: which might lead to composition difficulties.
* Picking an effect system library: it was new to me at the time but the concept looked promising, such as an orthogonal way of dealing with effects.

## Retrospective
Today, we have worked roughly one year on Hetchr and 7 developers have contributed to it (2-3 at the same time, and months with freelancers), leading to a product with 60k lines of code.

We have 19 effects and 35 interpreters.

### What worked
_Fined grained interface implementation segregation is perceptible_.
Where previously we used to encapsulate technical code implementation in functions (such as an HTTP call), we can create an effect that does not expose the implementation. This allows us to not be coupled too much to our providers (or their SDK version).

_Effects exposition makes mistakes more obvious_.
With dedicated effects, signatures reflect what functions are doing. So if a function has effects related to Github in code dealing with users, it is a clear hint the code does more than it aims to do. Also when a piece of code relies on only a small subset of an effect, it can be considered too large.

_Interpreters are easy to write, easy to (re)use_.
One of the development strategies is to have a really short feedback loop. Keeping the full product building and testing as fast as possible (around one minute on a developer's machine, 10 minutes to build/test/deploy on the continuous integration). So we have interpreters that target the real infrastructure (AWS DynamoDB, AWS Kinesis, ElasticSearch), and interpreters in-memory for tests. At some point we wanted a local version of the backend to facilitate front-end development, it takes us little effort to pull the code and pick the right interpreters. For example, we picked the in-memory interpreters for stores but we took the logging used in production. Extensibility is great, adding effects and interpreters is linear.

_Interpreters can be composed_.
Not only we can have interpreters depending on each other (we have an interpreter of an _event store_ effect, which yields a _registry_ effect, a _time_ effect, and a _log_ effect), but we are also able to transform effects on the fly (for buffering, checking the number of calls), without changing the underlying code.

_Most important effects exist already_.
Most of the heavy lifting is done and we can easily reuse it in our specific interpreters.

### What did not work
_Initial effects are costly to build_.
While the principle is simple, setting up the infrastructure is a bit tedious.

_Errors handling is difficult_.
While Haskell is not known for the simplicity of its compilation errors, _polysemy_ makes them worse. Even with the [polysemy-plugin](https://hackage.haskell.org/package/polysemy-plugin), if we forgot to apply one value to an effect, we might end up with an error suggesting a missing effect. Sometimes we even get an incomprehensive error when it happens around effect declaration.

_Onboarding is painful_. 
I find effects systems quite simple to grasp (compared to monad transformers), but I think there is a lack of culture, feedback, and examples around it. Consequently, whenever we onboarded people, it took more time than we expected, while our codebase is full of examples.

### What can be improved
_The documentation is good enough_.
It contains all we need to work with the wide range of effects, but we need more articles (and even books) on it. This will improve the culture around it and will reduce the _onboarding effort_.

_Having more guidance on the design process_.
While we aim for a very iterative _Type Driven Design_, it is especially true in effects systems. For example, I usually iterate three to four times over a piece of code before it is stabilized, and when it comes to effects, it usually takes seven iterations.

_Better warnings would be great_.
Especially for redundant constraints, which is helpful for not effects-related code, but is necessary for effects-related code. It is quite easy to start our code, add an effect, change the code, drop this effect, and keep the constraint. In the end, we keep carrying this effect, which pollutes the type signature and forces us to interpret it.

## Conclusion
Despite the initial investment and learning the basics, I think that choosing _polysemy_ was one of our best moves. It helps us to have better structure in our code, giving it good stability.