How did we build a usage billing tool?
Building billing software might seem straightforward, but the real challenge. Mainly with a usage-based billing platform where massive data processing is crucial. Here’s how we crafted Togai's architecture, a usage billing platform, and the tech decisions we made along the way.
Togai’s Backend Architecture From day one, Togai was modeled as a microservices-based architecture. While monoliths could have sped up development, our commitment to delivering high throughput (nearly unlimited) necessitated a microservices approach. This gave us the flexibility to scale critical components in our event processing pipeline as needed.

Tech Choices: Language & Framework We chose Kotlin for most of Togai’s backend due to its strong typing, expressive syntax, and JVM compatibility, allowing us to leverage powerful third-party libraries. Key components include:

Ktor: A Kotlin HTTP web framework, excellent for building high-throughput systems. 2. JooQ: Enables type-safe SQL queries, keeping us in the strongly-typed realm. 3. Flyway: Manages DB migrations seamlessly. 4. HikariCP: Optimizes DB connections with a lightweight pooling library. 5. Logback, Koin, JsonLogic, Cohort: Handle logging, dependency injection, JSON logic, and health checks respectively, each contributing to our robust backend infrastructure.

Kotlin's coroutines proved invaluable, enabling us to achieve 60 TPS with minimal resources—just a t4g.small instance with 20% CPU usage. This performance, with Kotlin’s efficiency, solidified our choice.

However, Togai’s backend also includes a NodeJS-based service for event ingestion. NodeJS was chosen for its asynchronous I/O capabilities, ideal for validating and queuing usage events. We addressed interoperability challenges using OpenAPI, allowing seamless integration across different languages and simplifying tasks like input validation and API generation.

Frontend Our frontend is built with Vue3, but we’ll dive into its architecture in a separate post.

Database We heavily rely on PostgreSQL, appreciating its rich features and community support. For time-series data, we opted for TimescaleDB, which outperformed other solutions in terms of ingestion rates and query performance. Being an extension of Postgres, TimescaleDB gave us confidence in its scalability.

Messaging For event processing, we needed a high-performance message queue. After evaluating Kafka and NATS, we chose NATS for its smaller memory footprint and superior scalability. NATS supports over 5000 messages per second with minimal resource consumption, making it a cost-effective solution for Togai. We also implemented a backup system using AWS SQS for added reliability.

Deployment Togai is deployed on AWS, leveraging Terraform and Chef for infrastructure management, ensuring our components like NATS, Nginx, Redis, and PostgreSQL are consistently configured.

Local Development Setting up Togai’s local environment is complex. We use Docker Compose, though performance issues are slowing down development. We’re exploring alternatives and will share our findings in a future post.

Testing We prioritize local integration tests over pure unit tests, allowing us to deliver high-quality features quickly. This approach has proven effective, as seen in our open-source identity and access management service.

Operations

Vector: We chose Vector for log collection due to its low memory footprint. NewRelic: Used for monitoring app servers and infrastructure. Emails: Alerts are currently managed via email, though we’re considering moving to a more robust notification system like PagerDuty. Other Tools We use Trello, GitHub, and Slack for team logistics.

Our tech choices reflect our design principles, familiarity, and the need for development speed. While these decisions are opinionated, they’ve proven effective for Togai’s unique requirements. What'd you think of our choices? Let me know.