pypartclone 0.1.5
Python package to read partclone and ntfsclone backup images & utility to mount images as virtual partitions.
Requires Python: >=3.8
Keywords: partclone,ntfsclone,image,backup

# vntfsclone & vpartclone - Mount Image Backups as Virtual Partitions

At a low level, this package provides Python code to read the building blocks -
headers, bitmaps, and blocks - of [partclone](https://partclone.org/) and
[ntfsclone](https://linux.die.net/man/8/ntfsclone) backup
images. These components may be used in other Python projects. Refer to the
[API documentation](https://pypartclone.readthedocs.io/en/latest/api.html)
for a comprehensive description of these components.

## Virtual Partitions

At a higher level, two command-line utilities based on this low-level code are
also included. Its features are as follows:
* They read partclone and ntfsclone images, verify checksums and dump data
structures.
* They mount partclone and ntfsclone images - the backup of a partition - as
virtual partitions.

These virtual partitions have the contents of the partition. They are created
without allocating additional disk space. Just like a restored partition, for
example, a virtual partition can be subjected to a file system consistency
check (`fsck`).

A virtual partition can be mounted as a file system. This is done with the
help of a loop device and allows you to inspect the contents. Individual
files and directories can be copied from image backups.

A virtual partition is read-only and cannot be written to.

## Usage

`vpartclone` and `vntfsclone` are command-line scripts with several options.
They are typically  called with a single-file, uncompressed and unencrypted
partclone or ntfsclone image and the `-m` or `--mountpoint` option:

```
$ mkdir -p ~/mnt; vpartclone nvme0n1p3.img -m ~/mnt

Virtual partition provided as '/home/user/mnt/nvme0n1p3'.

The file system of this virtual partition can be checked with this command:
   ntfsfix --no-action /home/user/mnt/nvme0n1p3

This virtual partition can be mounted as a read-only filesystem at '/home/user/mnt' with this command:
   sudo mount -t ntfs /home/user/mnt/nvme0n1p3 /home/user/mnt -o loop,ro

Forking subprocess to enter event-loop. When done unmount '/home/user/mnt' to quit this event-loop and its subprocess:
   sudo umount /home/user/mnt; umount /home/user/mnt
```

An empty directory `mnt` is created in the home directory and `mnt` is passed to
`vpartclone` with the `-m` or `--mountpoint` option. `vpartclone` will mount the
virtual partition to that mount point. We can check it with the usual
commands:

```
$ ls -lh ~/mnt
total 0
-r--r----- 1 user user 476G Aug 13 13:19 nvme0n1p3
```

This virtual partition looks like a big file. It does not actually allocate
any disk space, though. Note that the virtual partition is write-protected.
It cannot be modified in any way.

We can try to dump its contents:

```
$ xxd -g1 ~/mnt/nvme0n1p3 | head
00000000: eb 52 90 4e 54 46 53 20 20 20 20 00 02 08 00 00  .R.NTFS    .....
00000010: 00 00 00 00 00 f8 00 00 3f 00 ff 00 00 a8 08 00  ........?.......
00000020: 00 00 00 00 80 00 80 00 8e b2 72 3b 00 00 00 00  ..........r;....
00000030: 00 00 0c 00 00 00 00 00 02 00 00 00 00 00 00 00  ................
00000040: f6 00 00 00 01 00 00 00 96 7d 93 64 be 93 64 78  .........}.d..dx
00000050: 00 00 00 00 fa 33 c0 8e d0 bc 00 7c fb 68 c0 07  .....3.....|.h..
00000060: 1f 1e 68 66 00 cb 88 16 0e 00 66 81 3e 03 00 4e  ..hf......f.>..N
00000070: 54 46 53 75 15 b4 41 bb aa 55 cd 13 72 0c 81 fb  TFSu..A..U..r...
00000080: 55 aa 75 06 f7 c1 01 00 75 03 e9 dd 00 1e 83 ec  U.u.....u.......
00000090: 18 68 1a 00 b4 48 8a 16 0e 00 8b f4 16 1f cd 13  .h...H..........
```

This dump absolutely looks like an NTFS partition.

`vpartclone` suggested two commands when it mounted the virtual partition, a
`fsck` command and a mount command for that virtual partition. We will run the
`fsck` command first:

```
$ ntfsfix --no-action /home/user/mnt/nvme0n1p3
Mounting volume... OK
Processing of $MFT and $MFTMirr completed successfully.
Checking the alternate boot sector... BAD
Error: Failed to fix the alternate boot sector
```

Even the `ntfsfix` command accepts this virtual partition as a real partition.

Finally, we mount the virtual partition. Note that we mount it over `~/mnt`.
When we are done, we have to unmount `~/mnt` twice, once with `sudo` for the 
NTFS partition and then a second time as regular user to unmount the virtual
partition.

```
$ sudo mount -t ntfs /home/user/mnt/nvme0n1p3 /home/user/mnt -o loop,ro
[sudo] password for user:
```

There is no message and the NTFS file system of the partition is mounted:

```
$ mount | tail -2
vpartclone on /home/user/mnt type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,default_permissions,allow_other)
/home/user/mnt/nvme0n1p3 on /home/user/mnt type fuseblk (ro,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
```

Finally, we can access the NTFS file system:
```
$ ls ~/mnt/Windows/
 appcompat         csup.txt                    GameBarPresenceWriter   lsasetup.log         Provisioning       SoftwareDistribution   UUS
 apppatch          Cursors                     Globalization           Media                regedit.exe        Speech                 Vss
 AppReadiness      debug                       Help                    mib.bin              Registration       Speech_OneCore         WaaS
 AsPEToolVer.txt   diagerr.xml                 HelpPane.exe            Microsoft.NET        rescache           splwow64.exe           Web
 assembly          diagnostics                 hh.exe                  Migration            Resources          System                 WindowsShell.Manifest
 ASUS              DiagTrack                   IdentityCRL             ModemLogs            SchCache           System32               winhlp32.exe
 ASUS_IMAGE.Ver    diagwrn.xml                 IME                     notepad.exe          schemas            SystemApps             win.ini
 bcastdvr          DigitalLocker               ImmersiveControlPanel   OCR                  security           system.ini             WinSxS
 bfsvc.exe        'Downloaded Program Files'   INF                     OEM                  ServiceProfiles    SystemResources        WMSysPr9.prx
 Boot              DtcInstall.log              InputMethod            'Offline Web Pages'   ServiceState       SystemTemp             write.exe
 bootstat.dat      ELAMBKUP                    Installer               Panther              servicing          SysWOW64               WUModels
 Branding          en-US                       Inst_AsModelCopy.log    Performance          Setup              TAPI
 BrowserCore       es-ES                       L2Schemas               PFRO.log             setupact.log       Tasks
 CbsTemp           explorer.exe                LanguageOverlayCache    PLA                  setuperr.log       Temp
 comsetup.log      Firmware                    LiveKernelReports       PolicyDefinitions    ShellComponents    tracing
 Containers        Fonts                       Log                     Prefetch             ShellExperiences   twain_32
 Core.xml          fr-FR                       Logs                    PrintDialog          SKB                twain_32.dll
```

At this point we can copy files and directories from the virtual partition.

When we are done, we unmount the NTFS partition with sudo:

```
sudo umount ~/mnt
```

and unmount the virtual partition as a regular user:

```
umount ~/mnt
```


## Command-line arguments

Besides the `-m/--mountpoint` options, there are several other options. This
section introduces them all.

```
usage: vpartclone [-h] [-m MOUNTPOINT] [-v] [-d] [-c] [-i INDEX_SIZE] image

Mount partclone image backup as virtual partition.

positional arguments:
  image                 partition image to read

options:
  -h, --help            show this help message and exit
  -m MOUNTPOINT, --mountpoint MOUNTPOINT
                        mount point for virtual partition; an empty directory
  -v, --verbose         dump header and bitmap info
  -d, --debug_fuse      enable FUSE filesystem debug messages
  -c, --crc_check       verify all checksums in image (slow!)
  -i INDEX_SIZE, --index_size INDEX_SIZE
                        Size parameter for building bitmap index; leave
                        unchanged unless memory usage too high.
                        Increase size to reduce memory usage by doubling or
                        quadrupling the number repeatedly (default 1024).
  -q, --quiet           suppress progress bar in crc check
```

### image file

An image written by `partclone` is the only argument needed. For virtual
partitions, this image file must be a regular file. Split files must be
contatenated into a single file and compressed files must be uncompressed.

### verbose

The `-v/--verbose` options cause the header and bitmap information to be dumped.

### mountpoint

The argument of the `-m/--mountpoint` option is an empty directory where the
virtual partition will be created.

### debug_fuse

The `-d/--debug_fuse` option enables debug messages of the filesystem in
userspace (FUSE) code that is invoked for the virtual partition. This option
will cause fuse to run in the foreground. Use another window to unmount the
virtual partition.

### crc_check

The `-c/--crc_check` option requests that all checksums for data blocks be
checked. Enabling this adds a lengthy pass through an entire image file before
creating the virtual partition.

### index_size

The `-i/--index_size` option is available to reduce the memory consumption of
`vpartclone` at the expense of runtime if necessary.

When the virtual partition is active, `vpartclone` must read blocks
from the image file in an any order. Image files are not organized to alow to
quickly look up the location of a given data block in the image file. A bitmap
allows to determine in constant time whether a block is in the image file. If a
block is in the image file, the total number of bits set from the
beginning of that bitmap needs to be counted to determine the location of the
block's data in the image.

The bitmap can be millions, even tens or hundreds of millions of bytes in size.
To avoid counting the bits set in the bitmap from the beginning for each 
block, an index has been implemented. The bitmap is indexed so that for
each block access, only bits in a small range need to be counted. The
`index_size` option specifies the size of this range. It defaults to 1024
bits, which is 128 bytes of the bitmap.

If `vpartclone` ever runs out of memory, this default value can be doubled or
quadrupled. This may double or quadruple the time for each block access but
will reduce the memory usage by the factor of two or four.

Only `vpartclone` has this option. ntfsclone images do not contain bitmaps
and `vntfsclone` does not need this option.

### quiet

The -q/--quiet` option suppressed a progress bar that is shown whenever the
entire image file is read. The entire file is read when `vntfsclone` builds
an index for a virtual partition. The entire file is also read when
`vpartclone` verifies checksums.


## Dependencies

This code requires Python 3.8 or later and the additional packages `tqdm`,
`pyfuse3`, and `pyzstd`.

On Ubuntu Linux - and perhaps other Debian-based distributions - these
dependencies can be installed with:

```
sudo apt install -y python3-pip git pkg-config libfuse3-dev python3-tqdm python3-pyfuse3 python3-lz4
pip install git+https://github.com/joergmlpts/pypartclone

```

The installation of `libfuse3-dev` is not absolutely necessary but it
lets us install `pyfuse3` later in virtual environments.

On other platforms `pip` will install the dependencies:

```
pip install git+https://github.com/joergmlpts/pypartclone

```

where `tqdm` should install without issues but `pyfuse3`, when installed with
`pip`, needs the development package for `fuse3`. This package is called
`libfuse3-dev` or `libfuse3-devel` and it must be installed before `pip` is
invoked as seen above for Ubuntu. The chapter
[Pyfuse3 Installation](http://www.rath.org/pyfuse3-docs/install.html)
has more information about the installation of `pyfuse3`.

Please note that this utility relies on the filesystem in userspace (FUSE)
functionality. It will not run on Windows and all other platforms that do not
have FUSE.

## Python API

Partclone images consist of three components, a header, a bitmap and blocks of
data. A description of the [image file format](https://github.com/Thomas-Tsai/partclone/blob/master/IMAGE_FORMATS.md) is available.

This package has classes to read and process these three components.

### Classes PartClone and NtfsClone

Classes `PartClone` and `NtfsClone` are instantiated with an open file and its
filename. They read the header and bitmap of a partclone or ntfsclone image. If
the file does not contain a supported image, they raise exception
`ImageBackupException`.

```
from partclone.imagebackup import ImageBackupException
from partclone.ntfsclone import NtfsClone
from partclone.partclone import PartClone

with open('sda1.img', 'rb') as file:

    try:

        image = PartClone(file, 'sda1.img')

        print(image)

    except ImageBackupException as e:
        print('Failed to read image:', e)
```

If the image file can be opened, the header is printed like this:

```
Partclone Header
================
partclone version 0.3.24
fs type           BTRFS
fs total size     274,994,298,880 (256.1 GB)
fs total blocks   16,784,320
fs used blocks    968,580 (14.8 GB)     used block count based on super-block
fs_used_bitmap    1,168,685 (17.8 GB)   used block count based on bitmap
fs block size     16384
image version     2
cpu bits          64
checksum mode     CRC32
checksum size     4
checksum blocks   64
checksum reseed   True
bitmap mode       BIT
header_crc32      0xae9d1efd
bitmap            2,098,040 bytes (2.0 MB)
bitmap_crc32      0x6dbca530
blocks_section    at 2,098,154 in img file
block_offset_size 1024
block_offsets     0 instances
```

The image can also be read from a pipe, a regular file is not necessary.

`partclone` images contain a bitmap, `ntfsclone` images do not. The bitmap
represents each block with one bit and indicates whether the block
is in use or not. Only if a block is in use its data is saved to the image file.
There is not much besides the actual bitmap, just a checksum. The members
`block_offset_size` and `block_offsets` have not been read from the image file.
They implement indexing which allows us to read data blocks from the image
quickly and in an arbitray order.

### Reading Blocks

Once the header and bitmap have been read, we can read all used blocks from the
partition. Method `blockReader` reads all used blocks in sequence:

```
image.blockReader(progress_bar: bool = True, verify_crc: bool = False,
                  fn: Optional[Callable[[int,bytes],None]] = None) -> None:
```

By default, `blockReader` shows a progress bar as it reads all the data from
the image. Parameter `progress_bar` can be set to `False` to suppress the
progress bar.

By default, `blockReader` does not verify checksums for the blocks. Parameter
`verity_crc` can be set to `True` to cause `blockReader` to verify checksums.

Parameter `fn` is `None` by default. A function can be passed which will be
called with the offset into the partition and the data for each block. This
function can be used to restore a partition:

```
with open('/dev/sda1', 'rb+') as f_out:

    def write_block(offset: int, block: bytes) -> None:
        f_out.seek(offset)
        f_out.write(block)

    image.blockReader(fn=write_block)
```

The function is only called for used blocks. Unused blocks are not even stored
in the image file. However, since `blockReader` calls the function strictly in
ascending order of the offset, unused blocks can be written as well. The
following code fills them with `0xdeadbeef`, a pattern that is easily
recognized in hex dumps:

```
with open('sda1.vol', 'wb') as f_out:
    block_size  = image.blockSize()
    last_offset = 0
    empty_block = bytes([0xde, 0xad, 0xbe, 0xef] * (block_size // 4))

    def write_unused(offset: int) -> None:
        global last_offset
        while last_offset < offset:
            f_out.write(empty_block)
            last_offset += block_size

    def write_block(offset: int, block: bytes) -> None:
        global last_offset
        write_unused(offset)
        f_out.write(block)
        last_offset = offset + len(block)

    image.blockReader(fn=write_block)
    write_unused(image.totalBlocks() * block_size)
```

Note that `write_block` does not call `f_out.seek` anymore. In this scenario
the output file is written sequentially.

### Class BlockIO

There are situations where the blocks in an image file need to be read in random
order. Class `BlockIO` allows random access to arbitrary ranges of bytes.

The image file cannot be read sequentially in this scenario. It has to be a
regular file; a pipe will not work.

```
from partclone.blockio import BlockIO

blockio = BlockIO(image)

# read 42 bytes at offset 100000 and dump them in hex
print(' '.join(f'{b:02x}' for b in blockio.read_data(offset=100000, size=42)))
```