Building for Scale? Make a R.A.D.I.C.A.L. System!
Tags: techonology, webdev, devops, machinelearning
Published: 2018-12-26 23:56:44

---
title: Building for Scale? Make a R.A.D.I.C.A.L. System!
published: true
description: Creating a R.A.D.I.C.A.L. (Reactive, Automated, Distributed, Intelligent, Complex, Agile, and Lean) System
tags: techonology, webdev, devops, machinelearning
cover_image: https://thepracticaldev.s3.amazonaws.com/i/gxb2j12sxd1d3dv3igz2.jpg
---

The year is ending, 2019 is almost here and soon the 2010s will come to an end. This decade was marked by services from companies like Netflix, Google and Facebook thriving and becoming part of our daily lives.

For us in the tech field, these companies are examples. They build and operate systems that handle **billions** of users, 24/7, all around the world. Just think about it. That's a scale hard to grasp and many would doubt we could have so many people connected just a decade ago. 

In this post I want to introduce a new concept came up with a couple days ago: **R.A.D.I.C.A.L. systems**, that is, systems that are **R**eactive, **A**utomated, **D**istributed, **I**ntelligent, **C**omplex, **A**gile, and **L**ean.

I invented this acronym as a way to assemble these techniques that big companies are using to create successful systems that scale. I've been reading [lots of books](https://renatocf.xyz/goodreads) and watching [lots of talks](https://renatocf.xyz/youtube) about these subjects and I realized there was no term to describe systems that mix all of them. 

Most of the concepts here are not new, but they're becoming more and more popular thanks to the challenge of making systems -- R.A.D.I.C.A.L. systems -- capable of serving a huge number of people.

I'll start backwards by introducing each letter of the acronym in the **reverse order**. I believe this is the most **constructive way** to introduce the concepts. Whenever possible, I'll put links for those interested in reading more about the subjects I'm talking about.

This is my first public post about software engineering, so I hope you enjoy!

---

# 1. `L` stands for _Lean Inception_

Projecting systems from the scratch is hard, mainly when we don't know the business domain that we want to tackle. [Lean Startup](https://en.wikipedia.org/wiki/Lean_startup) by Eric Ries taught us how to **build, measure and learn** to create a **Minimum Viable Product (MVP)** in an environment of uncertainty. On the other hand, [Design Thinking](https://en.wikipedia.org/wiki/Design_thinking) brought to our attention the importance of **research** to have **insights** about problems in order to **ideate** and generate **prototypes** to get a specific solution. How can we mix these mindsets to create a new project?

[Lean Inception](https://martinfowler.com/articles/lean-inception/) by Paulo Caroli is a methodology that joins some techniques from Lean Startup and Design Thinking to start a new product. It does that by aligning the ideas of all stakeholders, product owners and developers involved in the project. The result (by the end of the one week session) is a set of **MVP Canvas** that describe the main goals, advantages and drawbacks of each iteration of development. They are accompanied by **user stories** that can be used by developers during the construction of the system.

A **R.A.D.I.C.A.L system** should use the **L**ean Inception to choose which and when features should be built.

# 2. `A` stands for _Agile Development_

The [Agile Manifesto](https://agilemanifesto.org/) is 17 years old and being agile is no longer hipster but rather mainstream. It's hard to find a developer who does not agree with the core values of agility, where **individuals and interactions**, **working software**, **customer collaboration** and **responding to change** are valued respectively more than processes and tools, comprehensive documentation, contract negotiation and following a plan.

The flexibility of Agile made it the default mindset when companies start developing -- whichever techniques they choose to use, prepackaged in methodologies such as [Extreme Programming](https://en.wikipedia.org/wiki/Extreme_programming) (more focused on the developer-side), [Scrum](https://en.wikipedia.org/wiki/Scrum_%28software_development%29) (more focused on the management side), [Kanban](https://en.wikipedia.org/wiki/Kanban) (focused on the organization of tasks), and many others.

A **R.A.D.I.C.A.L system** should use the **A**gile Development to optimize how and where features should be built.

# 3. `C` stands for _Complex Event Processing_

Millions to billions of users produce many, many (did I say many?) data, all the time, and we need a way to handle it. In an era of [Internet of Things](https://en.wikipedia.org/wiki/Internet_of_Things) where more and more devices are connected together, we need a way to handle all the **signals** they produce when interacting with humans and other machines.

[Complex Event Processing](https://en.wikipedia.org/wiki/Complex_event_processing) is a theory that advances many (big) data processing techniques to establish a well-defined method to deal with **infinite data sets**, generating ways to query, transform and mix different streams. It gives a framework to **abstract** dataflows, processes and exceptions that are happening inside a system.

A **R.A.D.I.C.A.L system** should use **C**omplex Event Processing techniques to track and analyze the data inside it.

# 4. `I` stands for _Intelligent Data Transformation_

[Artificial Intelligence](https://en.wikipedia.org/wiki/Artificial_intelligence) and, in particular, [Machine Learning](https://en.wikipedia.org/wiki/Machine_learning) are trending in the last couple years since humans can no longer grasp the amount of data we are producing. **Pattern Recognition**, **Computer Vision**, **Natural Language Processing** are some of the areas that AI/ML are making the most impact.

The result is a new generation of systems that make things that we could only dream of a decade ago, such as [voice assistants](https://en.wikipedia.org/wiki/Virtual_assistant), [autonomous cars](https://en.wikipedia.org/wiki/Autonomous_car) and [reusable rockets](https://en.wikipedia.org/wiki/Reusable_launch_system). As the ecosystem matures, using algorithms and models from these areas to find out valuable information to your product is no longer a task only for PhDs.

A **R.A.D.I.C.A.L system** should use **I**ntelligent Data Transformation techniques to learn and generate insights from the data inside it.

# 5. `D` stands for _Distributed Environments_

Some say that [Cloud Computing](https://en.wikipedia.org/wiki/Cloud_computing) is making to computation what the first power plants made to electricity. We no longer need to lose time maintaining our own servers to run and scale a system for new users. The cloud, however, brought many new challenges for developers. We **no longer have the control** of when a machine will be available, neither where exactly our systems are running.

Distributed systems are as old as the first networks. It is a cheap solution to increase processing power and the cloud made it much cheaper. However, the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem) made the words **availability** and **consistency** known by every developer that deals with these systems. This mix of theoretical and practical challenges pushed the creation of a new generation of tools that encapsulate the best patterns to manage these environments.

A **R.A.D.I.C.A.L system** should use **D**istributed Environments to be scalable in a simple and unrestrained manner.

# 6. `A` stands for _Automated Infrastructure_

If servers are out of our hands (thanks to the cloud), **automation** is very important to guarantee that we can run our services anywhere (particularly with the least vendor lock-in as possible). [Infrastructure as Code](https://en.wikipedia.org/wiki/Infrastructure_as_code) was the first step to enable developers to take part of operators responsibility, leading to the much broader [Cloud-Native](https://en.wikipedia.org/wiki/Native_cloud_application) movement (which includes technologies for **virtualization**, **orchestration**, **observability** and many other buzzwords) and the [Serverless](https://en.wikipedia.org/wiki/Serverless_computing) movement (which tries to free developers to think in nothing but their business logic).

At the same time, [DevOps](https://en.wikipedia.org/wiki/DevOps) is shining as a mindset to close the gap between development (creating new things) and operations (maintaining existing things), allowing the **full ownership** of products by those who created them. This heavily relies on and allows [Continuous Delivery](https://en.wikipedia.org/wiki/Continuous_delivery) of new features, where they can be **packaged**, **tested** and **deployed** with shorter cycles.

A **R.A.D.I.C.A.L system** should use **A**utomated Infrastructure to be deployable in a safe and reproducible manner.

# 7. `R` stands for _Reactive Microservices Architecture_

Complex business cases may demand large systems that span many business subdomains. [Domain-Driven Design](https://en.wikipedia.org/wiki/Domain-driven_design) by Eric Evans brought the concept of a **ubiquitous language** that should be used to describe these systems and **bounded contexts** that divide them. These are the building blocks for a [Microservices architecture](https://en.wikipedia.org/wiki/Microservices), that explores their semi-independence to create a system composed by many independent services which can be developed by standalone teams.

But building microservices is not a silver bullet for modern systems since there is a set of [prerequisites](https://www.martinfowler.com/bliki/MicroservicePrerequisites.html) in order to adopt them instead of a monolith. Anyway, the architecture still can become a [big ball of mud](https://en.wikipedia.org/wiki/Big_ball_of_mud). The [Reactive Manifesto](https://www.reactivemanifesto.org/) indicates how creating **responsive**, **resilient**, **elastic** and **message-driven** services may help to strength systems for scale.

A **R.A.D.I.C.A.L system** should use **R**eactive Microservices Architecture to encode complex business models and domain-specific knowledge.

---

That's it! What do you think about **R.A.D.I.C.A.L systems**? I'm sure they are not easy to make, and that grasping every concept requires lots of learning -- but it's been a very nice journey.

I want to thank my friends Fernando, Rafael and Victor for the valuable feedback. This post is just scratching the surface, but I want to highlight that there are systems -- R.A.D.I.C.A.L. systems -- that mix all these concepts.

If you want to discuss more about these ideas, please reach me at [Twitter](https://renatocf.xyz/twitter) or [LinkedIn](https://renatocf.xyz/linkedin).