Recovering after a root-partition resize gone wrong
Tags: linux, lvm, recovery
Published: 2020-12-03 19:26:32

---
title: Recovering after a root-partition resize gone wrong
published: true 
date: 2020-09-29 20:00:00 UTC
tags: linux,lvm,linux,recovery
canonical_url: https://www.zufallsheld.de/2020/09/29/recover-root-partition-resize-gone-wrong/
---

Recently I tried to resize a LVM logical volume with its volume group and the physical disk belonging to the vg.Because the logical volume contained all partitions except for `/boot` I knew it would be hard.

I also knew that I had no backup and no snapshot of the virtual machine. :)

Here’s the initial situation:

```
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 499711 497664 243M 83 Linux
/dev/sda2 501758 204797951 204296194 97,4G 5 Extended
/dev/sda5 501760 204797951 204296192 97,4G 8e Linux LVM

```

Note that `sda5` is a logical partition on top of the extended partition `sda2`.

Here’s the full volume group:

```
root@master:/home/sebastian# vgdisplay
  --- Volume group ---
  VG Name master-vg
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 4
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 2
  Open LV 2
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 97,41 GiB
  PE Size 4,00 MiB
  Total PE 24938
  Alloc PE / Size 24938 / 97,41 GiB
  Free PE / Size 0 / 0
  VG UUID yka0ZI-k7Q3-aeYD-vxU7-xajX-OVOy-qyogee

```

And here is the underlying physical volume:

```
root@master:/home/sebastian# pvdisplay
  --- Physical volume ---
  PV Name /dev/sda5
  VG Name master-vg
  PV Size 97,42 GiB / not usable 0
  Allocatable yes (but full)
  PE Size 4,00 MiB
  Total PE 24938
  Free PE 0
  Allocated PE 24938
  PV UUID TDHGY0-t1IO-qKuf-BQNf-fsWo-abut-PHZ8KJ

```

To resize the disk I used `sfdisk` because I read that `fdisk` alone couldn’t do the job.

Actually I followed this tutorial on Stack Exchange: https://unix.stackexchange.com/questions/379549/resize-extended-partition-lvm/379681#379681

I’ll paste the answer below for completeness:

> To extend your sda5, you need to extend its container too, sda2. Using command-line tools, the simplest way to do this is to use sfdisk:
> 
> `sfdisk /dev/sda`
> 
> This will print the current partition table, which should match what you saw in fdisk:
> 
> ```
> Disk image: 100 GiB, 107374182400 bytes, 209715200 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0xe59ec859
> 
> Old situation:
> 
> Device Boot Start End Sectors Size Id Type
> image1 * 2048 499711 497664 243M 83 Linux
> image2 501758 104855551 104353794 49.8G 5 Extended
> image5 501760 104855551 104353792 49.8G 83 Linux
> 
> Type 'help' to get more information.
> 
> ```
> 
> At the >>> prompt, start re-defining all your partitions:
> 
> ```
> 2048,497664,83,*
> 
> ```
> 
> (this is the first partition: start sector, length in sectors, type, and a \* to make it bootable). sfdisk will respond with
> 
> ```
> Created a new DOS disklabel with disk identifier 0x03408377.
> Created a new partition 1 of type 'Linux' and of size 243 MiB.
> image1 : 2048 499711 (243M) Linux
> 
> ```
> 
> and prompts for image2 (sda2). Enter
> 
> ```
> 501758,,5
> 
> ```
> 
> which tells sfdisk to create an extended partition starting at sector 501758 and occupying all the available space after that; sfdisk will output
> 
> ```
> Created a new partition 2 of type 'Extended' and of size 99.8 GiB.
> image2 : 501758 209715199 (99.8G) Extended
> 
> ```
> 
> For image3, enter simply
> 
> ```
> 501760
> 
> ```
> 
> sfdisk will then output
> 
> ```
> Created a new partition 5 of type 'Linux' and of size 99.8 GiB.
> image5 : 501760 209715199 (99.8G) Linux
> 
> ```
> 
> and prompt for image6, which we don’t need, so enter
> 
> ```
> quit
> 
> ```
> 
> which will cause sfdisk to print the new partition table and ask if you want to write it to disk:
> 
> ```
> New situation:
> 
> Device Boot Start End Sectors Size Id Type
> image1 * 2048 499711 497664 243M 83 Linux
> image2 501758 209715199 209213442 99.8G 5 Extended
> image5 501760 209715199 209213440 99.8G 83 Linux
> 
> Do you want to write this to disk? [Y]es/[N]o:
> 
> ```
> 
> If you’re convinced the starting sectors all match, and sda1 is still OK, press Y to write the partition table and return to your shell.
> 
> Once that’s done, run
> 
> ```
> pvresize /dev/sda5
> 
> ```
> 
> to resize your LVM PV; you should then be able to use the newly-allocated disk space (in new LVs, or to extend existing LVs).

After doing the above steps I rebooted and of course the server did not come back up.I connected with the vmware kvm tools to the server and saw an error message (among others) that told me `volume group not found`. It seemed I destroyed the partition layour or disk.

To fix this issue I mounted a live-CD into the server and booted into it. There I had access to the disk and could try repairs.

The LVM tools (pvscan, pvck) did not find any information regarding lvms on the disks:

```
> pvck /dev/sda5
Could not find LVM label on /dev/sda5

```

Also there was no backup in `/etc/lvm/backup` that I could use. But the internet told me that the configuration of the lvm is stored in the first bytes/sectors of the partition and that it is possible to restore this configuration with `dd`:

```
dd if=/dev/sdc3 bs=512 count=255 skip=1

```

And this did work! I spat out the configuration that contained the volume group, physical disk and logical volumes on this disk.

I then used `pvcreate` to recreate the physical disk and make it known to the lvm:

```
> pvcreate --uuid "U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy" --restore /etc/lvm/backup/testvg /dev/sda5

```

I don’t really know if this command was necessary as I did use `vgcfgbackup` to restore the whole configuration.

```
vgcfgrestore master-vg

```

> vgcfgrestore restores the metadata of a VG from a text back up file produced by vgcfgbackup. This writes VG metadata onto the devices specifed in back up file.

After that I could see the logical volume again and everything looked just like before I tried to resize the partition. I did one final reboot and the system came back online!My disaster recovery worked! Now all I need to do is resize the disk..

Some useful links I found during my repair attempts:

- https://unix.stackexchange.com/questions/420577/how-do-i-recover-a-lost-lvm-volume-group-not-found-by-vgscan-and-lvmdiskscan
- https://unix.stackexchange.com/questions/489182/lvm-pv-missing-after-reboot
- https://unix.stackexchange.com/questions/520688/broken-lvm-physical-volume-after-resizing-partition
- https://unix.stackexchange.com/questions/329582/drives-not-showing-when-sudo-pvdisplay
- https://serverfault.com/questions/275679/rescue-disk-is-unable-to-see-the-lvm-physical-volumes
- https://www.howtoforge.com/recover\_data\_from\_raid\_lvm\_partitions