Raspberry Pi 3B+ ISCSI boot with RaspiOS ARM64
Tags: raspberrypi
Published: 2022-02-14 15:49:37

---
title: Raspberry Pi 3B+ ISCSI boot with RaspiOS ARM64
published: true
date: 2022-02-11 20:32:36 UTC
tags: raspberrypi
canonical_url: https://dashaun.com/posts/raspberry-pi-3b-iscsi-boot-arm64/
---

![Diskless Raspberry Pi 3B+](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x7tjxg73pi2ea0wc6f6r.png)

Raspios 64-bit recently went [GA](https://www.raspberrypi.com/news/raspberry-pi-os-64-bit/) and I recently found [new joy](https://dev.to/dashaun/spring-boot-263-arm64-image-afo) in my Raspberry Pi 3B+ and Raspberry Pi Zero 2 W devices.

Now feels like a great time to revisit diskless booting.

This is what I'm using:

- UniFi for DHCP
- QNAP NAS for TFTP and ISCSI
- Raspberry Pi 3B+


### Install RaspiOS 64-bit

Use `rpi-imager` and install the GA version of 64-bit RaspiOS!

### Enable USB Boot Mode

The devices that I have are 4-6 years old.  I'm not sure if new ones will have this set already or not.

Run this command, the "OTP dump", to see if usb_boot_mode is enabled:

```bash=
vcgencmd otp_dump | grep 17:
```

This is what my OTP dump looked like before usb_boot_mode enabled.

```
17:1020000a
```

If the output looks like this, get the latest firmware:

```bash=
sudo rpi-update
```

I was a little nervous about the `rpi-update`, but I'm also following my own notes.

```
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
 *** Backing up modules 5.10.63-v8+
#############################################################
WARNING: This update bumps to rpi-5.15.y linux tree
See: https://forums.raspberrypi.com/viewtopic.php?t=322879

'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.
##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   173  100   173    0     0    678      0 --:--:-- --:--:-- --:--:--   678
100  123M    0  123M    0     0  4052k      0 --:--:--  0:00:31 --:--:-- 3014k
 *** Updating firmware
 *** Updating kernel modules
 *** depmod 5.15.21-v8+
 *** depmod 5.15.21-v7+
 *** depmod 5.15.21-v7l+
 *** depmod 5.15.21+
 *** Updating VideoCore libraries
 *** Using SoftFP libraries
 *** Updating SDK
 *** Running ldconfig
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to fbcab73d5a20f592aa9f6b0e364757ef131dae27
 *** A reboot is needed to activate the new firmware
```

After rebooting from the `rpi-update` we still need to enable usb_boot_mode, just needed the latest firmware to do so.

Add this line to `/boot/config.txt` it doesn't matter where, but needs its own line.  I put it at the top.

```
program_usb_boot_mode=1
```

Then reboot the device again.  Now the OTP dump should look like this:

```
17:3020000a
```

Now that the OTP dump is set, it is safe to remove the `program_usb_boot_mode=1` line from `/boot/config.txt` file.  There is no need to reboot, but if you remove the line, and reboot, you will see that the OTP dump still shows usb_boot_mode enabled.


### Enable ISCSI module with initramfs tool

These are the steps I took:

```bash= 
# Update repositories
sudo apt update
# Install available upgrades
sudo apt upgrade -y
# Install open-iscsi
sudo apt install open-iscsi
# Start open-iscsi to create the configs that we will use later
sudo systemctl start open-iscsi
# Create initramfs flag file for the iscsi module
sudo touch /etc/iscsi/iscsi.initramfs
# Create a new initramfs image that includes the iscsi module from the current kernel
sudo update-initramfs -v -k `uname -r` -c
```

You should now have an `initrd.img` in your `/boot` directory.

```bash=
ls -latr /boot/initrd.img*
```
The output should look like this:
```
-rwxr-xr-x 1 root root 10487750 Feb 11 10:59 /boot/initrd.img-5.15.21-v8+
```

We will refer to this new `initrd.img` file later!

### Add TFTP options to DHCP server

At this point, the bootloader on our device will accept a network boot, similar to PXE, but not exactly the same.

Setup your DHCP server to point to your TFTP server.  The DHCP protocol has `Option 66` for TFTP Server.  Basically, you want to tell your DHCP client (the Raspberry Pi 3B+) which server to connect to next, to get further instructions.

### Setup TFTP Server

The network boot process for the Raspberry Pi 3B+ is PXE-like.  So it requires a slightly custom setup.  It requires the serial number of the device, to be known in advance.

Grab the serial number of the device:

```bash=
grep Serial /proc/cpuinfo
```
In my case it looks like this:
```
Serial		: 000000002edd5668
```
On the TFTP server, create a directory for the device.  In the case of my QNAP server, the TFTP root is at `/share/tftp` but I'll use the variable `TFTP_ROOT`.

```bash=
# Set the implementation specific TFTP_ROOT
export TFTP_ROOT=/share/tftp
# Create directory using the device serial number, without leading 0s
sudo mkdir $TFTP_ROOT/2edd5668
```

Now, from the TFTP server, copy over the `/boot` partition, of the Raspberry Pi, that includes our new kernel.

```bash=
sudo scp -r pi@2edd5668:/boot/* $TFTP_ROOT/2edd5668/
```
If you want to check your work, the directory should look something like this:
```
COPYING.linux*            bcm2708-rpi-cm.dtb*       bcm2710-rpi-3-b-plus.dtb* bcm2711-rpi-4-b.dtb*      cmdline.txt*              fixup4db.dat*             initrd.img-5.15.21-v8+*   kernel8.img*              start4db.elf*
LICENCE.broadcom*         bcm2708-rpi-zero-w.dtb*   bcm2710-rpi-3-b.dtb*      bcm2711-rpi-400.dtb*      config.txt*               fixup4x.dat*              issue.txt*                overlays/                 start4x.elf*
bcm2708-rpi-b-plus.dtb*   bcm2708-rpi-zero.dtb*     bcm2710-rpi-cm3.dtb*      bcm2711-rpi-cm4.dtb*      fixup.dat*                fixup_cd.dat*             kernel.img*               start.elf*                start_cd.elf*
bcm2708-rpi-b-rev1.dtb*   bcm2709-rpi-2-b.dtb*      bcm2710-rpi-zero-2-w.dtb* bcm2711-rpi-cm4s.dtb*     fixup4.dat*               fixup_db.dat*             kernel7.img*              start4.elf*               start_db.elf*
bcm2708-rpi-b.dtb*        bcm2710-rpi-2-b.dtb*      bcm2710-rpi-zero-2.dtb*   bootcode.bin*             fixup4cd.dat*             fixup_x.dat*              kernel7l.img*             start4cd.elf*             start_x.elf*
```

Now, because this is PXE-like, we need to copy the `bootcode.bin` to the `TFTP_ROOT`

```bash=
sudo cp $TFTP_ROOT/2edd5668/bootcode.bin $TFTP_ROOT/
```

At this point, without an SD Card installed, the Raspberry Pi 3B+ will:

- Ask the DHCP server for an IP address
- Our DHCP server will provide an IP address AND tell the device to visit the TFTP server for more info
- Our device will ask the TFTP server for $TFTP_ROOT/bootcode.bin
- Our device will try to boot from $TFTP_ROOT/2edd5668/

At this point though, our `$TFTP_ROOT/2edd5668/` exactly matches what is on the SD Card in our device. We need to tell the device to use our new ISCSI enabled kernel.

I added the following line to the bottom of `$TFTP_ROOT/2edd5668/config.txt`, but you should use your specific `initrd.img` version.
```
initramfs initrd.img-5.15.21-v8+ followkernel
```


### Create ISCSI LUN for the device

For simplicity, I'm creating a 32GB LUN, with "thick" provisioning on my NAS, and mapping that to an ISCSI Target.  I chose 32GB simply because that is the size of SDcard that I have been using.  I also used the serial number, of the Raspberry Pi, in the naming of the LUN and ISCI target.

To verify that your new ISCSI LUN is accessible we will connect to it from the Raspberry Pi 3B+.


```bash=
# set the IP address of the ISCSI server
export ISCSI_IP=192.168.1.248
# 
sudo iscsiadm -m discovery -t sendtargets -p $ISCSI_IP | grep 2edd5668
```
The output should contain the new ISCSI target.
```
192.168.1.248:3260,1 iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
```
From above the TARGET_IQN is `iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c` and I create an enviroment varialbe to store it.

```bash=
export TARGET_IQN=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
```

### Mount the ISCSI LUN to the Raspberry Pi

Login to the ISCSI LUN, this example is not using CHAP authentication.

```bash=
sudo iscsiadm -m node -l -T $TARGET_IQN -p $ISCSI_IP
```
Success should look similar to this:
```
Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c, portal: 192.168.1.248,3260]
Login to [iface: default, target: iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c, portal: 192.168.1.248,3260] successful.
```
I validate the new ISCSI connection using `sudo fdisk -l` from the Raspberry Pi 3B+ and look for something like this, showing the 32GB LUN attached via ISCSI
```
Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: iSCSI Storage   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 1048576 bytes / 1048576 bytes
```
The LUN is showing up as `/dev/sda` in this case.
```bash=
# Format the LUN attached as /dev/sda to ext4
sudo mkfs.ext4 /dev/sda
```
```
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: f581fa52-f7ea-4c87-91cc-e236d64f63a6
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done
```
```bash=
# Get the UUID of the LUN
sudo blkid /dev/sda
```
```
/dev/sda: UUID="f581fa52-f7ea-4c87-91cc-e236d64f63a6" BLOCK_SIZE="4096" TYPE="ext4"
```
```bash=
# Mount the new disk
sudo mount /dev/sda /mnt
```

### Copy the existing installation to the mounted LUN

This step takes some time.  We are copying all of the files from the `/` partition of the SD card on the Raspberry Pi 3B+ to the network attached LUN.  We are excluding the directories that are mounted during boot.

```bash=
# Copy / partition to /mnt
sudo rsync -avhP --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
# Create directories that are mounted during boot
sudo mkdir /mnt/{dev,proc,sys,boot,mnt}
```

### Remove SD Card details from /etc/fstab

```bash=
cat /mnt/etc/fstab
```
Before:
```
proc            /proc           proc    defaults          0       0
PARTUUID=9633a3a2-01  /boot           vfat    defaults          0       2
PARTUUID=9633a3a2-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
```
Remove the lines containing `PARTUUID` which are referring to the SDCard.

After:
```
proc            /proc           proc    defaults          0       0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
```

### Update the kernel configs on TFTP server

Now that we have updated our filesystem.  We need to revisit kernel that is getting loaded over the network.  We need to tell it how to mount the `/` filesystem via ISCSI.

The ISCSI Initiator, which identifies the Raspberry Pi 3B+ device, can be found at `/etc/iscsi/initiatorname.iscsi`

```bash=
sudo cat /etc/iscsi/initiatorname.iscsi
```
```
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:2ccb20a316
```

On the TFTP server, I'm modifying the `$TFTP_ROOT/2edd5668/cmdline.txt` file.

I'm adding/updating these details for mounting the ISCSI drive:
- ip=::::2edd5668:eth0:dhcp
    - hostname of the device
- root=UUID=f581fa52-f7ea-4c87-91cc-e236d64f63a6
    - From the blkid command against the LUN
- ISCSI_INITIATOR=iqn.1993-08.org.debian:01:2ccb20a316
- ISCSI_TARGET_NAME=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
- ISCSI_TARGET_IP=192.168.1.248
- ISCSI_TARGET_PORT=3260
- rw


Before:
```
console=serial0,115200 console=tty1 root=PARTUUID=9633a3a2-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_enable=pids cgroup_memory=1 systemd.unified_cgroup_hierarchy=0
```
After:
```
console=serial0,115200 console=tty1 root=PARTUUID=9633a3a2-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_enable=pids cgroup_memory=1 systemd.unified_cgroup_hierarchy=0 ISCSI_INITIATOR=iqn.1993-08.org.debian:01:2ccb20a316 ISCSI_TARGET_NAME=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c ISCSI_TARGET_IP=192.168.1.248 ISCSI_TARGET_PORT=3260 rw
```
>This all needs to be on a single line.

### Ready to reboot!

From the Raspberry Pi 3B+ device:

```bash=
# Unmount the drive
sudo umount /mnt
# Disconnect from ISCSI
sudo iscsiadm -m node --logout all
# Shutdown
sudo shutdown now
```
Remove the SD card and restart the device!

### Summary

If you were paying close attention, you may have notice that the `cmdline.txt` also had some non-default settings in it.  Those are there to allow `k3s` to run as well.

This article is very specific to the Raspberry Pi 3B+ and using ISCSI.  Look for a follow-up example using the Raspberry Pi 4.

Soon, I'll document a diskless-pipeline for setting up Raspberry Pi 3B+ and Raspberry Pi 4, that includes using `arkade` and `k3sup` to configure the kubernetes abstraction layer across all of these homelab devices.

### Acknowledgements and References

- https://www.raspberrypi.com/documentation/computers/config_txt.html#boot-options
- https://stuff.drkn.ninja/post/2016/11/08/Net-boot-%28PXE-iSCSI%29-with-a-RaspberryPi-3