How did we tackle Airtable's API limits at FINN?
Tags: airtable, nocode, productivity, database
Published: 2021-12-17 14:16:03

![Airtable logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/058c8qca6gbn9b0z5kgp.png)

We started to use [Airtable](https://airtable.com/) at the early stages as I described in my previous article. Today, you will be reading how we tackled [Airtable](https://airtable.com/)'s API limits while the company grows to 10.000 car subscribers since 2019. 

![Andreas Wixler (CTO) initiated the Slack channel for Airtable Action Squad](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kactokl0fuqk4irk9pc6.png)
(Andreas Wixler (CTO) initiated the Slack channel for [Airtable](https://airtable.com/) Action Squad)

After we hit the limit, we were no longer able to provide a reliable service to our customers. Therefore, we initiated an immediate action squad functioning cross-departments with the lead of [Christian](https://www.linkedin.com/in/cpoetter/) (VP Engineering - B2B), [Tome](https://www.linkedin.com/in/tomepejoski/) (Tech Lead - Customer Product), and [Alexander](https://www.linkedin.com/in/alexanderyasnogor/) (VP Engineering - Data & Finance). I was representing the user acquisition team. Initially, we planned a sprint for two weeks, created our Kanban board in Jira, identified issues, and scheduled regular meetings for status updates on the tasks. 

Before tackling the API limits, we created an [Airtable](https://airtable.com/) migration plan and focused on certain topics for investigation earlier before we hit the issues. So, it's quite important to be ready when you hit the limit. But, we were not 100% ready when the issue occurred, it took a few weeks to solve the problems. Still, the performance we gained by the speed of moving fast while we grow was pushed by [Airtable](https://airtable.com/).

These were the core points we focused on earlier when we already decided on [NocoDB](https://www.nocodb.com/) as an alternative for [Airtable](https://airtable.com/):

- One database-instance per department (PostgreSQL) - AWS Aurora PostgreSQL
- Hourly backups
- Unlimited databases per department
- Connected with Google Workspace SSO
- Role-based control is department responsibility (VP Engineering of the Department, Department Head)
- [Integromat](https://integromat.com) - [NocoDB](https://www.nocodb.com/) Module - Customer Product write [Integromat](https://integromat.com) Module
Retool - Connecting via API (recommend)

So we were getting ready but not as fast as we were growing. Customers just loved us. :)

## Analysis of failure points

First, we sat down and looked for every scenario in [Integromat](https://integromat.com) and sometimes analyzed the API calls from the logs to have some metrics in front of us. We have a Slack bot that alerts us whenever there is an issue in [Airtable](https://airtable.com/) modules (API limits). Every person checked their scenarios which they own. Some people came up with alternative ideas for better performance. We took the notes down. 

These were the main issues we realized in our flows within all departments:

- [Airtable](https://airtable.com/)'s every base has a request limit of 5 per second. 
- [Airtable](https://airtable.com/) gives a warning at the top of a base because we reached and crossed total row limits. Sad. :/
- We almost collected everything in one base. This is because the company started with a simple base and then grew fast while the number of employees and orders increased quickly. We realized this issue earlier and created the new projects on different bases already. 
- We were using tons of modules in [Integromat](https://integromat.com) running in parallel (we have more than 15 workers in our [Integromat](https://integromat.com)). Also, due to the pagination requests in some modules because of data being too big, we were increasing the risk of hitting timeout errors. 
- Our checkout process was using services that were talking to [Airtable](https://airtable.com/) directly and synchronously. If there was a limit/timeout issue, the customer couldn't proceed further. 
- [Airtable](https://airtable.com/) is calculating every formula, within a base, when there is an edit, creation, update, or deletion of data. This causes a delay and timeout issue from time to time. Sometimes the response is timed out, but the action is successful. This can also deceive our [Integromat](https://integromat.com) modules. 

## Work we have done to fix the issue(s)


![Timeline of the total number of records in Vehicle Subscription Management (VSM) main base - decreasing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oj5dge55tbutk3de9hjs.png)


![Timeline of the total number of tables in Vehicle Subscription Management (VSM) main base - decreasing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jvm8erff5jj7x079f43m.png)

Most of our [Integromat](https://integromat.com) scenarios are using [Airtable](https://airtable.com/) for read-only operations. When [Airtable](https://airtable.com/) introduced their [Airtable Sync](https://support.airtable.com/hc/en-us/articles/360052521254-Airtable-Sync-Basic-setup), we immediately increased the number of bases by taking away some responsibilities (tables, records) from the main base into different bases.  From these bases, we created synced tables from the main base and used these bases as the reference point so we could gain their API request limit for read-only operations. Each department created its own small bases for read-only operations and updated its [Integromat](https://integromat.com) scenarios. 

While doing this, we also started to use Sync Inc to have a copy of our [Airtable](https://airtable.com/) base synced in PostgreSQL. Initially, we set the limit of writing at Sync to 1, and gradually we increased this limit (max 5) while monitoring our [Airtable](https://airtable.com/) performance and data quality. This way we could connect this solution to our own [NocoDB](https://www.nocodb.com/) instance. After our beloved engineer [Ishtiaque](https://www.linkedin.com/in/ishtiaquezafar/) (Senior Engineer - Operations) created an [Integromat](https://integromat.com) module for [NocoDB](https://www.nocodb.com/), we started reading from our own PostgreSQL instance UNLIMITED and we initiated the migration to the [NocoDB](https://www.nocodb.com/) this way. We first started with Sync solution, then switched to [NocoDB](https://www.nocodb.com/). 

We had to update our [NocoDB](https://www.nocodb.com/) modules. We used [Integromat](https://integromat.com) API to search for certain [Airtable](https://airtable.com/) modules and then replaced them with [NocoDB](https://www.nocodb.com/) modules. [Dina](https://www.linkedin.com/in/dinalivia/) (Software Engineer - Operations) and Vijay (our [NocoDB](https://www.nocodb.com/) guru) contributed heavily to this. <3

I moved our calls to the [Airtable](https://airtable.com/) from our checkout process. We created an [Integromat](https://integromat.com) module that would book the car for the customer asynchronously while the customer could proceed with their process - no waiting. 

We analyzed company-wide each column in each table to detect if they are being used or not. It was a huge company effort because dependencies were too many, all spread over different tools. We removed all unneeded columns successfully. After that, we also optimized the formulas that we use and tried to decrease dependencies in formulas and between columns. 

![Airtable timeouts per day](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zfv0ce12bmkkx2gg2qfi.png)

Thanks, everyone involved!

## Highlights and achievements

- More than halved the numbers of records in VSM
Halved the numbers of tables in VSM
- Reduced massively the # numbers of [Airtable](https://airtable.com/) timeouts per day
- Switch to read from [NocoDB](https://www.nocodb.com/) directly where ever possible and write to [Airtable](https://airtable.com/)
- Proof of concept migration of car sales to [NocoDB](https://www.nocodb.com/)
- Last but not least, worked for the first time all together cross-department, with a great time spirit and achieving the goals set at the beginning

Thanks, [Emanuele](https://www.linkedin.com/in/emanuele-gennusa/) (Senior Product Manager - Operations) for managing our sprints!

While we move away from [Airtable](https://airtable.com/) with our core data, we will be still using it for some time. [NocoDB](https://www.nocodb.com/) is still in the early stages. We built a dedicated team to work on this open-source project to fit our needs by the time. At [FINN](https://www.finn.auto), we will be always using No Code - whether [Airtable](https://airtable.com/) or not. 

------------

I will write about how we are using [NocoDB](https://www.nocodb.com/) at [FINN](https://wwww.finn.auto) in one of my next articles. 

Stay tuned and don't forget to like/follow! :)