Brazilian "Rinha de Backend" challenge #1 - How To Fail
Published: 2023-08-28 13:41:04

Father's Day at Brazil I received a message from a friend of mine talking about a tech challenge. [A quite simple one: Write a API, that has at least 2 replicas, a load balancer and a database, so someone can try to tear it down with a Stress Test.](https://github.com/zanfranceschi/rinha-de-backend-2023-q3)

I though to myself. _That_ is the next level of **CRUD developer**, so let's try it out.

[The initial idea](https://github.com/lukas8219/rinha-be-2023-q3/commit/b374c30cc4d0f0c302378a21ead5debf1ec6fcbd), by which the [commit history](https://github.com/lukas8219/rinha-be-2023-q3/commits/main) made it clear I mistakenly called it "finished", was:

- NodeJS Express
- Postgres
 - [Indexes for FTS (`gin(to_tsvector(lan, field));`)](https://github.com/lukas8219/rinha-be-2023-q3/commit/b374c30cc4d0f0c302378a21ead5debf1ec6fcbd#diff-c79e7bf3fe4949f3acfa89d636f82f6e794a2348c0c25763fd5fcb05a3380bb9R24)
 - [Stack being a Array field](https://github.com/lukas8219/rinha-be-2023-q3/commit/b374c30cc4d0f0c302378a21ead5debf1ec6fcbd#diff-c79e7bf3fe4949f3acfa89d636f82f6e794a2348c0c25763fd5fcb05a3380bb9R20)
- Batching
 - [All read requests to database](https://github.com/lukas8219/rinha-be-2023-q3/commit/b374c30cc4d0f0c302378a21ead5debf1ec6fcbd#diff-c79e7bf3fe4949f3acfa89d636f82f6e794a2348c0c25763fd5fcb05a3380bb9R113)

I knew this configuration was far away from aiming for any TOP 20 but I had _confidence_ it wouldn't fail that much.

After my first NGINX configuration and Docker-Compose YAML created I boldly applied to the challenge.

Sitting at the top of my confidence and not-so worried about it I noticed the competitors publishing screenshots of their own benchmarks.I had realized how much I had underestimated it.

Gatling installed locally, lets give it a shot. To be fair...How bad can my application perform, right?


![Fantastical failure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/unkbwsyag86k3kakwzbe.png)

Lets not talk about how disappointed I got but how worthy it was to try the tool locally.

### So, what was happening?


Besides a bunch of [premature closed connections](https://stackoverflow.com/questions/36488688/nginx-upstream-prematurely-closed-connection-while-reading-response-header-from), no healthy upstream, 60000ms timeouts, there was no way I my application was bottlenecking that much...

I tried a lot of crazy stuff:

- Clustering - via [PM2](https://github.com/lukas8219/rinha-be-2023-q3/blob/7e0d606bcc5550a07976d15dae8e71d85b011a1e/Dockerfile#L8) or [native cluster module](https://github.com/lukas8219/rinha-be-2023-q3/blob/7e0d606bcc5550a07976d15dae8e71d85b011a1e/index.js#L63).
- Closing [slow requests](https://github.com/lukas8219/rinha-be-2023-q3/blob/7e0d606bcc5550a07976d15dae8e71d85b011a1e/index.js#L80) early
- Adjusted some [business rules](https://github.com/lukas8219/rinha-be-2023-q3/blob/7e0d606bcc5550a07976d15dae8e71d85b011a1e/database.js#L97) which were off
- [Reconfigured NGINX](https://github.com/lukas8219/rinha-be-2023-q3/commit/7e0d606bcc5550a07976d15dae8e71d85b011a1e#diff-1f91f64bfa3fb0a1ece881887af0a2356b8c529a96fb91c1894745b2a1a009aaR4)
- Changing LIBUV `process.env.UV_THREADPOOL_SIZE = os.cpus().length`

I was very confident I could get at least into 20% of success...Come on...

![api errors after running gatling](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/scb52d356ymhvofjrdbn.jpeg)

I had to do _something_. I was totally blind and didn't knew where to start.

After running a `watch "docker stats"` I was able to finally get a grasp on it. My database was being **hammered**.

![database bottlenecking locally](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xp2jlryoabectuqvhpsz.png)

Time to take it seriously. More than competing with my fellow colleagues _I wanted to compete with myself_. _I knew I could do **better**_. It was a long time I got so excited about something I knew nothing about.

In that late Sunday I already _knew_ Monday would be _one of those hyperfocus ADHD days_.

[Next chapter](https://dev.to/lukas8219/brazilian-rinha-de-backend-challenge-2-the-improvement-1a9h)