Server Environments
Tags: systems, programming, testing
Published: 2023-02-02 00:46:44

When maintaining large software systems, you will likely have multiple environments with names like Prod, Staging, Dev, Eval, UAT, Daily, Nightly, or some remix of these names. To distinguish _this_ type of environment from the dozen other things in software development that we give that same name, these are often formally referred to as **Deployment Environments**.

One question I've _never_ been asked directly is: _"What is an Environment?"_ This is surprising; because not understanding what Deployment Environments _actually are_ is one of the most common pitfalls I see in my day-to-day work.

## What is a Deployment Environment

A **Deployment Environment** is a _consistently connected_ set of

1. processes
2. datastores, and
3. any ecosystem around them (e.g., cron jobs, analytics, etc.)[^1]

making up a fully functioning software system.

[^1]: I include the rest of the ecosystem around this for completeness, but often it's sufficient to think of a Deployment Environment simply as a consistently connected set of processes & datastores.

This definition is quite intuitive, but the devil is in the details. In this case, that detail is the phrase _"consistently connected"_.

### What _consistently connected_ entails

Ideally, environments should be _perfectly isolated_; no data or RPC should leak between processes and data stores across environments during normal operation[^2].

[^2]: Explicit processes that exist _beyond_ the bounds of any environment may purposely interact with multiple environments. For example, it might be desirable to sync or seed some test data between environments, etc.

#### A Case Study

_This concrete example convinces the uninitiated that you should **never** mix dependencies across environments (e.g., Dev instances should only call other Dev instances; never prod instances). If this is already intuitive for you, you can
skip this section and [go straight to the principles of consistent connectedness](https://blog.eyas.sh/2023/02/server-environments/#principles)._

![An Architecture Diagram representing a simple To-do system. An HTTP endpoint calls TodoApi and PeopleApi. Todo API is connected to a database called TodoStore, while PeopleApi is connected to some People graph service called PeopleStore. There's also a ReminderService that reads TodoStore and sends push notifications to the HTTP endpoint. The user communicates only with the HTTP endpoint via a browser.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tgldskz52ah1hms605av.png)

Imagine the architecture above represents a system you maintain. Each box here represents either a _service_ or a _datastore_. Right now, you have one instance of each of these endpoints, and they're connected as you see above. Let's say these systems are connected by directly addressing each other (e.g., by calling specific URIs for each service). Real people are about to use this service, but you want to continue deploying more recent versions.

You might decide to create a set of _Dev_ instances to help you out.

You might wonder: _How many instances do I need to set up and configure to have a viable Dev environment?_

What is _viable_ will certainly depend on which endpoints you care about testing.

Let's assume you want to test `TodoApi`. This service:

-   Calls `PeopleApi`
-   Reads and writes to `TodoStore`
-   Is _called by_ `PeopleApi`
-   Is _called by_ `HttpBackend`

Generally, if you can exercise the service you're interested in directly, you might not care about its callers.

Next, you have to decide:

-   which `PeopleApi` should this instance call, and
-   which `TodoStore` should this instance read and write to?

Remember that `PeopleApi` will call `TodoApi` back. It's very tough (and almost _always_ wrong) to try to get away with calling the _Production_ instance of `PeopleApi` from your _Dev_ instance of `TodoApi`; the production instance you call might mutate the production state, or it might call back the production instance of `TodoApi` instead of you. You might convince yourself it's harmless, but more often than not, you'll be met with subtle glitchy behavior at best and serious bugs or user data leaks at worst.

Instead, you'll want an entirely separate _Dev_ instance of `PeopleApi` in its own right. As you configure **this dev instance of `PeopleApi`**, you will have **only one correct choice** for which `TodoApi` to call: the Dev instance we just created.

We will also likely want a separate `TodoStore` database to be available to the Dev instance of `TodoApi`, with totally separate tasks, etc. This allows us to make sure none of our _read/write_ testing has the potential to affect production users.

This line of reasoning applies recursively and can help you arrive at some general principles.

### Principles of consistent connectedness

**Read more about the principles of consistent connectedness in [the full post](https://blog.eyas.sh/2023/02/server-environments/)**.