Citrix XenServer 7.1 CentOS VM’s not starting after Kernel update
Tags: devops, webdev, security, tutorial
Published: 2020-09-01 18:44:18


After I update my CentOS 7 VM kernel, the new kernel stopped booting on my XenServer hypervisor. I’ve been using XenServer version 7.1


Working Kernel version, before `yum update`, was `3.10.0-693` and, after `yum update` Kernel version is `3.10.0-1127.19.1`


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/zm9g0ga7bw6vr03cbu27.jpg)


This issue is due to the kernel which I updated has the `spectre/meltdown` patches in. But our XenServer 7 doesn’t. This difference created the kernel booting problem. Interestingly, this issue only affects CentOS VMs. My other Ubuntu servers can just run updates without issue but CentOS VM’s require the host to be updated or they stop working.   

To ***FIX*** this issue you should update your XenServer to the latest version or else if there are hotfixes available then you should install it.  

Link for XenServer 7.1 hotfixes. [https://support.citrix.com/article/CTX230788](https://support.citrix.com/article/CTX230788)   


To upgrade XenServer 7.1 you should have Citrix License first. But a "free" patch was released for 7.2, due to the massive security issue later.

But if you have been running older XenServer's version, older than 7.1 then you don't have any choice besides updating XenServer to the latest.

Also, CentOS fixed their booting problem on their new Kernel versions which I will discuss later below.

## What is Meltdown and Spectre vulnerabilities?
[https://spectreattack.com](https://spectreattack.com)

Someone having access to the VM will be able to read the memory outside the VM itself, and therefore read data from other VMs on this same physical host. An attacker could enter via other application exploits (webserver etc.) on your VMs. 
#### Am I affected?
Meltdown is using a design flaw into **Intel** CPUs only. This is called by Xen sec team "SP3" (aka rogue data cache load).  
You are impacted only if you are using:
* 64-bits PV type VM (***HVM/PVHVM aren't affected!***)
* Intel CPUs (AMD chip design is a bit different and not affected)
* untrusted VMs, ie untrusted users having VM access (even non-root!)
* All XenServer versions are affected

> 64-bits PV guests are vulnerable because guest and hypervisor share the same address space, but with different privileges. HVM aren't.

#### Mitigate Risk:
Migrate all VM’s from `PV` to `HVM` or upgrade XenServer to latest or install hotfixes if available.   
But if you have been running older version XenServer, older than `7.1` then you have to update your XenServer to resolve this issue.


***Back to CentOS 7 booting issue.***
It seems the booting issue was fixed in the latest CentOS kernel versions. If the XenServer was up-to-date with the hotfixes then the VM’s should not have any problem. But in my case I didn't update my XenServer, so now I’ve to either install the latest CentOS 7 kernel or downgrade to older previous running one.


### Upgrading Kernel Version
Add `ELRepo Repository` first.

Add ELRepo gpg key to the system.
```yaml
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
```
>This is important – CentOS will not allow the installation of an unsigned software package. The GPG key provides a digital signature to verify that the software is authentic.


Now add new ELRepo repository.
```yaml
rpm -Uvh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
```

When it's complete, check all repository enabled on the system, and make sure ELRepo is on the list.
```yaml
yum repolist
```

```yaml
yum --enablerepo=elrepo-kernel install kernel-ml
```
***kernel-ml*** – which indicates a mainline release, with a shorter support term but with more frequent updates.

***OR INSTALL (Install either ml or lt)***

```yaml
yum --enablerepo=elrepo-kernel install kernel-lt
```
***kernel-lt*** – signifying a stable long-term support release.

listout all your existing kernel version on your system.
```yaml
awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg

CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)
```
We have sucessfully installed CentOS latest kernel version `4.4.234-1`. Now set this as a default kernel.

### Set Default Kernel Version
First listout all existing kernel versions.
```yaml
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)
```

>NOTE: The first entry is denoted as `Zero`. Let us modify the Kernel Version to `4.4.234-1` which is at line number `1` but denoted as entry `0`.


`Set default kernel.`
```yaml
grub2-set-default 0
```

Next, `Rebuild grub.cfg` file with `'gurb2-mkconfig'` command.
```yaml
grub2-mkconfig -o /boot/grub2/grub.cfg
```

##### To check default Kernel Version
```yaml
cat /boot/grub2/grubenv |grep saved

saved_entry=CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
```

Reboot Server
```yaml
reboot
```


**Above solution will resolves the booting issue.**

If you don't want to update CentOS kernel to latest one and want to boot from the existing previous working kernel then reboot your system and choose kernel version which was working fine before from `grub menu` and select `enter` and follow the below steps. 


Set Default Kernel Version but first list-out all existing kernel versions.
```yaml
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)
```

>NOTE: The first entry is denoted as `Zero`. So currently the Server is booted to `0th` entry as per the above command output. Let us modify the Kernel Version to `3.10.0-693`  which is at line number `4` but denoted as entry `3`.


`Set default kernel.`
```yaml
grub2-set-default 3
```

Next, `Rebuild grub.cfg` file with `'gurb2-mkconfig'` command.
```yaml
grub2-mkconfig -o /boot/grub2/grub.cfg
```

##### Check default Kernel Version
```yaml
cat /boot/grub2/grubenv | grep saved

saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
```

Reboot Server
```yaml
reboot
```