How to never forget you're in production
Tags: devops, sre, tutorial, linux
Published: 2020-08-11 02:57:14


## The Scenario
You now have production access!
You shell into the server and .... huh

```
Welcome to Ubuntu 50.01.1

  System information as of Sun Aug  9 20:29:10 UTC 2050

  System load:  0.2                Processes:           149
  Usage of /:   42.4% of 16 bits   Users logged in:     0
  Memory usage: 67%                IP address for ens5: 8.8.8.8
  Swap usage:   0%

 * Are you ready for Kubernetes 5.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.


Last login: Sun Aug  9 20:22:21 2050 from 192.16.184.0
bob@dns0:~$
```

It looks exactly the same as any server†. Pretty boring, really. Months go by, and shelling into production becomes old hat to you. One day you're tasked with deleting everything in a test server. A quick `rm -rf *`, and bam, it's done.

Oh wait.

You were on the production server.

![](https://dev-to-uploads.s3.amazonaws.com/i/x52q68xv4mozpotnuobh.png)
<center>"Maybe if I control-z I can undo it"</center>

This was just a hypothetical example, but it could happen! It's important to visually make your production servers distinct as to reduce the chance of error. Doing so is very easy, and can potentially stop a total catastrophe.

## The Tutorial
Put the below line in your `~/.profile` file:
`export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "`

Replace `{{ domain }}` with the name of your environment.

Now you can see the environment in your prompt:

`caleb@web0.cloud20:~$ `

For production environments at [15Five](https://15five.com) we use a distinct color and put the environment name in uppercase:

`export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.\[\033[01;31m\]{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/fkl7s8el5yxbedzujfzx.png)

Feel free to customize this to your liking - you can even put emojis in your prompt!

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/54ozh3yim73qtbadig6a.png)

You can automate this for all your servers by using [Ansible](https://github.com/ansible/ansible):

```yaml

# key_names is a list of users

# https://serverfault.com/questions/322997/how-do-you-make-it-obvious-you-are-on-a-production-system
# http://abload.de/img/bash-color-chartmxjbp.png

- name: Set terminal prompt to have VPC name
  lineinfile:
    dest: /home/{{ item }}/.profile
    regexp: '^export PS1='
    line: 'export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "'
  with_items: "{{ key_names }}"
  when: env_name != 'production'

- name: Set terminal prompt to have red uppercase VPC name in production
  lineinfile:
    dest: /home/{{ item }}/.profile
    regexp: '^export PS1='
    line: 'export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.\[\033[01;31m\]{{ domain|upper }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "'  # noqa: 204
  with_items: "{{ key_names }}"
  when: env_name == 'production'
```

## But what if I accidentally run Ansible against the wrong environment?

You can catch that too by using a Ansible [callback plugin](https://docs.ansible.com/ansible/latest/plugins/callback.html)! Assuming you have a environment variable in your terminal for the environment you are running against, you can verify that the Ansible inventory matches the environment variable.

{% gist  https://gist.github.com/caleb15/152b1624106e697076e9c9c0538bb847 %}

You probably have to adapt the file for how you handle secrets and deployments via Ansible.

---

† Assuming your other servers are google DNS servers from the future that work with just 16 bits of storage. You know, the usual.

Fun fact: You can actually [recover data](https://unix.stackexchange.com/a/101247/420478) from a `rm -rf`!


<!-- *oh no.jpg* -->
<!-- nuclear bomb picture? -->

<!-- So you've been granted production access. The keys to the kingdom, absolute power in the tip of your hands. As Julius Caeser said, *quod non est, qualitercumque media dicentem iustus aliquid Im 'ut glaciem canetis* -->