Update thousands of linux servers in no time with Ctfreak
Tags: ctfreak, scheduling, sysadmin
Published: 2023-03-24 09:16:40

Are you looking to automate the update of your linux servers?

By launching the updates concurrently?

Without having to use a sledgehammer to swat a fly?

Don’t look any further this tuto is made for you 😉.

## Requirements

- 1000 servers to update (for example) :
  - Having for hostname `serverXXXXX.local` with `XXXXX` ranging from 00001 to 01000
  - Using a linux distribution based on Debian (Debian, Ubuntu, Linux mint, …)
  - Accessible via SSH on port 22 with the same private key and an `adminuser` user account that can run `sudo` commands without password prompt
- An instance of [ctfreak](https://ctfreak.com) (very easy to [setup](https://ctfreak.com/docs/install): no dependencies, no db to configure) with an administrator account, the _Free Edition_ will be enough.

## The script

On each server, this shell script will perform the update :

```bash
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive
sudo apt update 
sudo apt upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"
```

We find the classic `apt update/upgrade` enriched with some elements allowing to answer automatically any question likely to interrupt the script.

_NB: If your servers are under Redhat (or any other distribution based on RPM), I invite you to adapt the script accordingly_.

## Ctfreak

The next step will be to use [ctfreak](https://ctfreak.com) to deploy and run via SSH the script on our 1000 servers.

Login to ctfreak with your administrator account.

![Connection](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/connect.webp)

### Adding the SSH Credential

Go to _SSH Credential_ -> _New SSH Credential_ button

![SSH credential](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/sshkey.webp)

Add the SSH private key **MySSHKey** which allows to connect to the servers and validate.

_NB : You can also connect with a password instead of a private key_.

### Adding nodes

A _node_ defines the parameters for connecting to a server via SSH, the nodes are grouped in _source nodes_.

As a reminder, there are 1000 servers, so 1000 nodes to add.

It is actually possible to add the nodes one by one via the web interface, but in this case, you will rather create a dedicated _source node_ which will be fed by a yaml file describing our 1000 nodes.

Let the file `/home/adminuser/ctfreak-nodes.yaml` (readable for the user running the ctfreak instance) :

```yaml
- name: server0001
  tagNames:
    - debian_server
    - linux_server
  username: adminuser
  hostname: server0001.local
  osFamily: UNIX
  sshPort: 22
  
- name: server0002
  tagNames:
    - debian_server
    - linux_server
  username: adminuser
  hostname: server0002.local
  osFamily: UNIX
  sshPort: 22

## ...  
## Complete with nodes 3 to 1000
## ...

```

Go to _Nodes_ -> _New External Node Source_ button

![Connection](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/node-source.webp)

Check that the SSH key **MySSHKey** is selected as _Credential_ (ctfreak will use it to connect to nodes via SSH), then validate to create the _source node_ **Linux Servers** (this source node will resync its nodes from the yaml file every 2 hours).

If the yaml file has been successfully processed, you should see the following list :

![Nodes](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/nodes.webp)

### Adding the task

Go to _Projects_ -> _New Project_ button

![Project](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/project.webp)

Validate to create the **Sysadmin** project.

Click on the _New Task_ button.

![Task](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/task.webp)

Validate to create the **Upgrade debian servers** task.

This task will deploy and execute the shell script :

- on all _nodes_ matching the **#debian\_server** tag
- every 1st day of the month at 4am  

### Execution of the task

To avoid waiting for the 1st of the month to see what happens 😉 , execute the task by clicking the _Execute_ button.

![Execution](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/execution.webp)

The execution is launched, then click on the execution identifier **#T0ZJX** to see its progress for each node.

Once the execution is finished, some nodes may fail (timeout on SSH connection opening, package repository unavailable at the launch of apt-update, DNS issues, …) :

![Execution nodes](https://pelle.link/en/update-thousands-of-servers-with-ctfreak/execution-nodes.webp)

After checking the _Failed only_ filter to only display failed nodes and check the logs to fix the issue (here a DNS problem), you can launch a new execution to **process only failed nodes** by clicking on _Re-execute failed nodes_.

### Conclusion

Here you have a reliable process for updating your servers.

For those who would like to go further, here are some suggestions for improvement:

- Add an email notification in case of failed update.
- Add Redhat servers:
  - Add 1000 new nodes with tags **redhat\_server** , **linux\_server**.
  - Duplicate the **Upgrade debian servers** task to **Upgrade redhat servers** and adapt it accordingly.
  - Create a **Upgrade all servers** task of type **workflow** to launch concurrently (PRO Edition) or sequentially (FREE Edition) the **Upgrade debian servers** and **Upgrade redhat servers** tasks.