Saving Your Linux Machine from Certain Death
Tags: devops, sysadmin, linux, bash
Published: 2020-05-01 07:01:31

_Note: This was originally posted at [martinheinz.dev](https://martinheinz.dev/blog/22)_

Troubleshooting damaged systems is essential skill of every SysAdmin, SRE or DevOps engineer. Everyone of us runs into OS related issues from time to time and it's better to be prepared when things go terribly wrong. It's especially beneficial to be able to identify and act on the issue quickly to prevent any significant damage. To help with that in this article - we will go over a few common problems that you might encounter as well as ways to gather information, troubleshoot and solve these issues.

_Note: This article uses RHEL 8/CentOS. But, examples/concepts below can be applied to any Linux distribution._

## Recovering `root` Password

What if you lose `root` password and you don't have access to any privileged user? If you still have access to the machine, then there is way to solve this _inconvenient_ situation:

First, start by rebooting the machine. When machine starts, hit any key to access boot menu:

![Boot Menu Screenshot](https://dev-to-uploads.s3.amazonaws.com/i/cxby8kid1klbozyixbjh.png)

In boot menu hit `e` to edit boot options. Using arrows move to the line starting with `linux` and append `rd.break` - this _breaks_ boot process early on. Optionally you can also append `enforcing=0`, to pause _SELinux_ enforcing. Next, hit `CTRL+X` to let machine boot.

![Edited Boot Menu Screenshot](https://dev-to-uploads.s3.amazonaws.com/i/hvo2bqgys0ld0my1pos0.png)

After few seconds of booting, you should get shell. At this point you have access to system in _read-only_ mode. So, in order to change anything in the system - like `root` password - we need to make filesystem _read-write_. We can do that by running `mount -o remount,rw /sysroot`. Next thing we need to do is enter _root jail_ using `chroot /sysroot` - this changes _root_ of filesystem to be `/sysroot` instead of `/`.  This is required, so that any further commands we run will be in regards to `/sysroot` directory. Now we can change `root` password using `passwd`.

If you added `enforcing=0` to boot options, you can now hit `CTRL+D` (or type `exit`) and let system fully boot. If not, run `touch /.autorelabel` to trigger _SELinux_ system relabel. This is needed because changing password results in `/etc/password` having incorrect _SELinux_ security context. Therefore we need to relabel whole filesystem during next boot (this can take some time, depending on size of filesystem).

![Password Change Commands](https://dev-to-uploads.s3.amazonaws.com/i/7659gdcr6zdhwp8d2hzc.png)

As an alternative solution, you could also access Linux `debug-shell`. This can be done - again - by accessing _GRUB_ during boot and appending `systemd.debug-shell` instead of `rd.break`. When you let system boot with this option, you will end up in normal shell session, which isn't really helpful. If you however try to access terminal 9 using `CTRL+ALT+F9`, you will open `debug-shell` with full `root` permissions. Here, you can change password normally. At this point you can switch back to normal shell (`CTRL+ALT+F1`) and login. You shouldn't forget to stop the `debug-shell` though, as it is huge vulnerability to the system. You can do that by running `systemctl stop debug-shell.service` (you can still switch back to `debug-shell` but it will be unresponsive - killed-off).

![debug-shell](https://dev-to-uploads.s3.amazonaws.com/i/vwzkejp47zexn9ds5qzz.png)

## Fixing Unmountable Filesystems

Creating new partitions, creating filesystems, mounting filesystem, etc. are common tasks for most SysAdmins. But, even though these are basic tasks it's easy to make mistake which may render your system _unbootable_. Let's see how you can solve problems related _unmountable_ filesystems:

As with previous solutions, we start by rebooting the machine, accessing the boot menu and editing it, this time appending `systemd.unit=emergency.target`. This tells your system to boot into emergency target instead of default one (multi-user or graphical). When system boots and we get shell, we login as root and we again remount the filesystem using `mount -o remount,rw /`. Now we can try mounting all filesystems by running `mount -a`. If there is a problem with mounting some specific filesystem, you might see error message like `mount: /wrong-mount: mount point does not exist.` or `mount /wrong-mount: special device /dev/sdb1 does not exist.`. These kind of an issues need to be fixed inside `/etc/fstab`:

```
/dev/sda1   /mnt           xfs   defaults   0 0
/dev/sdb1   /wrong-mount   xfs   defaults   0 0  <- fix, comment out or remove this line
```

After fixing the issue in `/etc/fstab` run `systemctl daemon-reload`, so that `systemd` picks up the changes. Now, run `mount -a` again. If the issue was indeed fixed, you should see no error (_no news, is good news_). You can now exit using `CTRL+D` and let system boot normally.

Aside from mistyped device or mount point name, you might also encounter issues with _VDO (Virtual Data Optimizer)_ or _Stratis_, which require extra mount arguments e.g. `x-systemd.requires=vdo.service` or `x-systemd.requires=stratisd.service`, without which system won't boot properly. Another common and easily fixable mistake might be missing quote when using `UUID="...` to specify device (use `/etc/fstab` syntax highlighting, it can save you lot problems).

## Troubleshooting SELinux Problems

This one is not a life and death kind of a situation, but it can cause a lot problems, so it's beneficial to be able to identify it quickly when it happens.

It's important to realise that most of the time, _SELinux_ is doing it's job correctly. But it might just happen that you are trying to achieve something _SELinux_ doesn't expect. Some of the problem you might encounter may include issues with incorrect file context, for example after moving file from one place to another. Sometimes the issue might be with overly restrictive policies (_SELinux_ booleans) or blocked service ports.

One can troubleshoot all of these problems by first temporarily changing _SELinux_ to non-enforcing mode using `setenforce 0` and retrying the action that wasn't working previously. If the problem got fixed by switching _SELinux_ to non-enforcing mode, then we know that the problem was caused by some _SELinux_ violation.

Now, if we turn _SELinux_ back on using `setenforce 1`, we can try to analyze and fix the violation:

First install `setroubleshoot-server` using `yum -y install setroubleshoot-server`. This troubleshooting server will listen to `/var/log/audit/audit.log` and send summary messages to `/var/log/messages`. Next, to analyze these messages run `grep sealert /var/log/messages` which should give you messages like this:

```shell
...
Apr  6 08:03:46 server1 setroubleshoot[24548]: SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket port 8012. For complete SELinux messages run: sealert -l d2c359a9-e9c5-48cb-af8b-ba73d982f21b
```

As an example here, I configured `httpd` to run on port `8012` which is blocked because of _SELinux_ service's allowed ports. If we were not aware of this, then it would be quite hard to find root cause of this issue. The output above can help with that. We can see description of the _SELinux_ violation as well as command that can help us troubleshoot further, so let's try it out:

```shell
[root@server1 ~]# sealert -l d2c359a9-e9c5-48cb-af8b-ba73d982f21b
SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket port 8012.

*****  Plugin bind_ports (92.2 confidence) suggests   ************************

If you want to allow /usr/sbin/httpd to bind to network port 8012
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 8012
    where PORT_TYPE is one of the following: http_cache_port_t, http_port_t, jboss_management_port_t, jboss_messaging_port_t, ntop_port_t, puppet_port_t.

... omitted

Additional Information:
Source Context                system_u:system_r:httpd_t:s0
Target Context                system_u:object_r:unreserved_port_t:s0
Target Objects                port 8012 [ tcp_socket ]
Source                        httpd
Source Path                   /usr/sbin/httpd
Port                          8012
Host                          ...
Source RPM Packages           httpd-2.4.37-10.module+el8+2764+7127e69e.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.14.1-61.el8.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing

... omitted
```

This produces full report of what caused the violation. Including suggested (not necessarily the most appropriate) fix. If you have some experience with _SELinux_ you might realise that most appropriate way to fix this issue is to add the relevant port to _SELinux_ service (`http_port_t`). This can be done by running `semanage port -a -t http_port_t -p tcp 8012`.

This pattern of replicating the violation, looking for `sealert` messages in `var/log/messages` and viewing the report and analyzing the report can be applied to any _SELinux_ violation/problem, not just the one example above. 

Alternatively you can also search directly in `/var/log/audit/audit.log` using `ausearch`. Specific command you would want to run: `ausearch -m AVC -ts recent`. This shows all recent denials. Output should look something like this (same information, little less user friendly):

```shell
time->Mon Apr  6 08:10:56 2020
type=PROCTITLE msg=audit(1586160656.860:235): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=SYSCALL msg=audit(1586160656.860:235): arch=c000003e syscall=49 success=no exit=-13 a0=3 a1=562ebb162280 a2=10 a3=7ffcd89eb21c items=0 ppid=1 pid=24612 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1586160656.860:235): avc:  denied  { name_bind } for  pid=24612 comm="httpd" src=8012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0
```

## Getting Logs From Crashing System

By default, logs stored in `/run/log/journal` are not persisted across system reboots. That might become a problem if you need to debug logs on crashing system. To preserve journal logs we need to modify `/etc/systemd/journald.conf`. More specifically the `Storage` parameter:

```ini
# /etc/systemd/journald.conf

[Journal]
#Storage=auto  # <- This one
#Compress=yes
#Seal=yes
#SplitMode=uid
...
```

By uncommenting and changing `Storage` to `persistent`, we tell `systemd` to store all logs in `var/log/journal`. Aside from this change we also need to run `systemctl reload systemd-journald` to make sure that the change takes effect.

Even though this change will persist logs on your system, it won't keep all of them forever. By default, `journald` is configured to not exceed 10% of filesystem or leave system with less then 15% of free space.

Now, to actually inspect the previously stored logs. First switch to `root` user. Run `journalctl --list-boots`. This will give you list like this:

```shell
-3 620d709db36440dc9a0b165f85f9f346 Sun 2020-03-08 18:58:41 CET—Fri 2020-04-03 08:29:25 CEST
-2 89a7ba7725ab4ac5bc91517d29fbc31e Fri 2020-04-03 08:33:19 CEST—Fri 2020-04-03 08:35:02 CEST
-1 e93ec544729f4008ac126b80b4b0c207 Fri 2020-04-03 08:35:53 CEST—Sun 2020-04-05 15:21:16 CEST
 0 58a205834d2d4d7b9b228d8a126b9e81 Sun 2020-04-05 15:21:40 CEST—Fri 2020-04-10 14:57:49 CEST
```

Based on the dates and times, choose from which boot you want to see logs. For example, to view logs from boot with id `-2` with log level `err` or higher:

```shell
~ $ journalctl -b -2 -p err
-- Logs begin at Sun 2020-03-08 18:58:41 CET, end at Fri 2020-04-10 14:59:59 CEST. --
apr 03 08:33:21 my-pc bluetoothd[1187]: ...
apr 03 08:33:25 my-pc bluetoothd[1187]: ...
apr 03 08:33:25 my-pc kernel: ...
apr 03 08:33:36 my-pc wpa_supplicant[1198]: ...
apr 03 08:33:36 my-pc wpa_supplicant[1198]: ...
...
```

If the logs above are not enough to troubleshoot your issues, then there are other log files to check:

- `/var/log/messages` - Most of the _syslog_ messages should be here.
- `/var/log/boot.log` - System startup messages (Not related to _syslog_)

Alternatively, if you can't boot your machine normally, then you can access `emergency.target` as shown above a view logs there the same way.

## Conclusion

There is a lot more that can go wrong with Linux machine than what I shown in sections above. These examples/approaches though, can be applied to variety of other problems that you might encounter. Also, not all of them are _life-and-death_ kind of a situation, but it's always preferable to be able solve them rather quickly, especially if this problematic machine is a production system. Solving most of the issues depends of getting the right information and being able to restore previous configurations, therefore it's crucial to always store logs and to backup critical files on your system before modifying them.