Day 4: (Re)Installing NixOS on my Macbook Pro
Tags: nixos, dualbooting, linux
Published: 2023-05-14 00:37:37

If you're following from previous notes, you'll know that I'm starting fresh. I've got a totally fresh install of macOS Ventura and I'm ready to rock and roll. I'm combining a couple different guides to get my setup working and I'm noting everything I do here. Spoiler: I got it working this time! Let's get started!

### Basic Setup

To clarify, my Macbook is an APFS T2 Intel Macbook Pro 2019 15 inch. If you follow this with another device, there may be some differences in steps required. But at least this section is easy:

1. Plug in power
2. Plug in Ethernet via USB-C adapter (optional)
3. Plug in Get 4GB* USB for install media (via USB-C adapter)

*I actually used a 16gb drive this time, but I did use a 4gb in the past and I think it was good enough.

### Create installation media

I downloaded the minimal [nixos-t2-release](https://github.com/t2linux/nixos-t2-iso/releases) and used [USBImager](https://gitlab.com/bztsrc/usbimager) to write the resulting ISO to my usb drive. [Balena Etcher](https://www.balena.io/etcher) is a good choice as well.

### Create new partitions on Macbook Pro SSD

This can be done many ways, but I decided to use the macOS Disk Utility because it seems like the safest way to avoid messing up my APFS partition. Here is the desired endstate:
```
Total Physical SSD Storage: 500GB
0. EFI Partition (<1 GB; Not shown in Disk Utility)
1. APFS Partition (250 GB; macOS has volumes within this)
2. ExFAT Partition (30 GB; CrossData - sharing between OSes)
3. Ext4 Partition (120 GB; NixOS Root)
4. Ext4 Partition (80 GB; NixOS Home)
5. Swap Partition (20 GB; Swap - Memory Overflow for NixOS)
```

I am giving a lot of space to NixOS root because this is where I understand packages and previous generations will be stored. I also witnessed online that people who complain about running out of space in their root partition are mocked by people who claim to be allocating 100+ GB to it. It seems to me that this is a bit of a mental shift from past operating systems. Since all my programs will be nix packages, they can be nicely separated from my personal files and data. Apple has done a pretty neat thing with APFS. macOS's portion of the drive is so tidy because all the operational divisions are handled with dynamically sized `volumes` within the APFS partition. Meanwhile, Microsoft will screw up your EFI partition, so you need a second EFI to mitigate that if you want to triple boot. I don't really understand all that, but I'm just going to use one for now. 

Here's how to create the partitions with macOS GUI Disk Utility. The trick is to click the right items in the right order.

1. View > Show all devices
2. Click the top level APPLE SSD AP0512M Media
3. Click Partition
4. Hit the +; choose `Add Partition` not volume
5. Set the size for the partition
6. Name the partition
7. Select the `ExFat` format
8. Hit Apply & Confirm

_Plus button first, then size, then name, then format._

Do this for each partition. I did it 4 times to create my CrossData partition and 3 partitions for NixOS (NixOS Root, NixOS Home, and Swap). Sometimes there may be an error. I don't know the consequences of that, but I just respond by deleting the new partition and creating it again. One time it took 3 tries, but usually it works the first time.

Looks like this for me now:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1redz3vx0hzi4a5vs4q9.png)

### Extract the Macbook Devices Firmware

We need to get the firmware from macOS somewhere linux can access it. The [t2linux site](https://wiki.t2linux.org/guides/wifi-bluetooth/#on-macos) provides a handy script called `firmware.sh` which has a macOS portion and a Linux portion. The macOS portion does 2 things:

1. Zips the firmware files up into `firmware.tar.gz` on the EFI partition
2. Copies the firmware.sh script into the EFI partition as well

Download the script, `cd` into its directory, and run it. 

Grant execution permission with chmod first.
```
chmod +x firmware.sh
```
Execute it
```
./firmware.sh
```

If you want to do it manually, you can follow the steps on [this guide](https://github.com/Redecorating/archiso-mbp/blob/2898e6f957c6afb239ce466ab3909719098977aa/airootfs/root/wifi-firmware/wifi.md?plain=1#L27). Those steps include the easy part (copying the files over) as well as the hard part (packaging them up appropriately). Looked a little intimidating for me, so I stuck to running the `firmware.sh` script and crossing my fingers.

### Prepare for dual booting

I'm not sure why, but I still couldn't boot into another device without rEFInd. Here's what I had to do to get my boot-up process working.

Boot into recovery mode by holding `CMD+R`
- Use Startup Security Utility to set secure boot to `No Security` and allowed boot media to `Allow booting`. Apple docs on this [here](https://support.apple.com/en-us/HT208198).
- Use Terminal to disable system integrity protection (SIP) by running `csrutil disable`. Apple docs [here](https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection).

Reboot to normal macOS and download rEFInd from the `A binary zip file` sourceforge link from [rodsbooks website](https://www.rodsbooks.com/refind/getting.html).

From its directory, run the install script
```
./refind-install
```

To make life easier, I also set an nvram parameter so that I wouldn't have to hold option `⌥` to boot to startup manager.
```
sudo nvram manufacturing-enter-picker=true
```


### Boot from USB

Time to get down to business. Restart the mac and we should see two boot options: Macintosh HD and EFI Boot. Since I have rEFInd installed, it doesn't matter what I choose at this point, but if you don't, you need to pick EFI Boot. 

When rEFInd loads, what I saw was a list of 4 boot options.

1. macOS from Preboot (ssd)
2. Boot EFI\boot\refind_x64.efi from EFIBOOT (usb)
3. Boot Fallback boot loader from EFIBOOT (usb)
4. Boot Legacy OS from whole disk volume (usb)

Now I don't fully understand these options, but I know to boot from my install media, I need to choose #3. Some text comes up, I get brief GUI boot option and let the timer countdown to 0 to automatically choose the first option. Some green and white matrix code flies by and next thing I know I'm at the NixOS Live CD command line.

### Finish preparing the drive

Welcome to the command line. We should see a blinking underscore after `[nixos@nixos:~]$`. We have a few steps to take before actually installing NixOS. First things first, let's make the font size bigger. 
```
setfont ter-v32n
```
Wow!!! The keyboard works!!! We're on the right track! Next let's take a look at the partitions we prepared earlier

```
lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
```

This gives me a bunch of stuff, take a look:
```
NAME        FSTYPE    SIZE     MOUNTPOINT
loop0       squashfs  800M     /nix/.ro-store
sda         iso9660   14.7G
├─sda1      iso9660   839M     /iso
└─sda2      vfat      3M
nvme0n1               465.9G
├─nvme0n1p1 vfat      300M
├─nvme0n1p2 apfs      232.8G
├─nvme0n1p3 exfat     18.6G
├─nvme0n1p4 exfat     74.5G
├─nvme0n1p5 exfat     111.8G
└─nvme0n1p6 exfat     27.9G
```

Looks familiar, right? Now we need to format the two NixOS partitions as Ext4. We'll keep the CrossData partition as exfat and we'll format the swap partition differently.

First, let's format the swap partition and label it `swap`
```
sudo mkswap -L swap /dev/nvme0n1p3
```
Next, let's format the nixos partition. Graciously, it gave me a confirmation dialog to confirm. That's good because this command wipes the contents of the partition. I don't know what will happen if you wipe the APFS partition. Can't be very good.
```
sudo mkfs.ext4 -L nixos /dev/nvme0n1p5
```
Finally, the partition for the home directory
```
sudo mkfs.ext4 -L home /dev/nvme0n1p4
```
Let's check our work
```
lsblk -o NAME,FSTYPE,SIZE,LABEL,MOUNTPOINT
```
Same command as before, but now with the LABEL column.
```
NAME        FSTYPE    LABEL                         SIZE     MOUNTPOINT
loop0       squashfs  802.6M                                 /nix/.ro-store
sda         iso9660   nixos-minimal-23.05-x86_64    14.7G
├─sda1      iso9660   nixos-minimal-23.05-x86_64     839M    /iso
└─sda2      vfat      EFIBOOT                          3M
nvme0n1                                            465.9G
├─nvme0n1p1 vfat      EFI                            300M
├─nvme0n1p2 apfs                                   232.8G
├─nvme0n1p3 swap      swap                          18.6G
├─nvme0n1p4 ext4      home                          74.5G
├─nvme0n1p5 ext4      nixos                        111.8G
└─nvme0n1p6 exfat     CrossData                     27.9G
```
Looking good! Before we continue mounting, let's follow this complex process for connecting to the internet.

### Loading Wi-Fi firmware to installer

With an ethernet-USBC adapter, you'll automatically be able to connect online, but these steps will still help prepare for regular usage later on, so I suggest following them.

First, let's run that `firmware.sh` script from before. Remember the first time we ran it, it just copied the script file into the EFI partition along with a the zipped firmware files. Thanks to its power of conditional statements, it will do something else when run on Linux. Run the script with this command:
```
./mnt/boot/firmware.sh
```
Hopefully that works for you first try. When I did it, it failed and I ended up going in and reading the script so that I could pick out parts to execute manually. I think my mistake was mounting my EFI partition before running this script, which also mounts it to another location. Since we saved that step for AFTER this one, you should be fine.

Once you've got your firmware installed, you can connect online! Use `wpa_supplicant` and add your network. The same instructions as the [NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-installation-manual-networking).

```
systemctl start wpa_supplicant
```
Enter the wpa_cli with
```
wpa_cli
```
A few more commands...
```
add_network
```
If this returns something other than 0, use that number as the network id for the following commands.
```
set_network 0 ssid "myhomenetwork"
```
```
set_network 0 psk "mypassword"
```
```
set_network 0 key_mgmt WPA-PSK
```
```
enable_network 0
```
If that worked, then you'll see some messages including the word `CONNECTED`. Wonderful. On a Macbook!

### Mounting Partitions

Now we mount the partitions. Consider that right now we are running on the USB. In order to access the Macbook SSD partitions, we need to make them available at some location. Mounting is the process that takes a partition or device and makes it available at a specified location. So we'll be creating `/mnt` as a working directory from where we can access these formatted partitions.

Furthermore, the drive mount statuses are what the NixOS installer will look at when generating the important `hardware-configuration.nix` file.

The order of operations here doesn't matter AFAIK. But we need to mount the root partition (nixos), the home partition (home), and the boot partition (EFI). We'll activate the swap partition because the nixos manual says to do so, even though we have 16GB of RAM in this MBP which really should be enough.

First let's run that lsblk command again
```
lsblk -o NAME,FSTYPE,SIZE,LABEL,MOUNTPOINT
```
None of your new partitions should be mounted at this point. If they are, unmount them. I needed to unmount my boot partition at this point so I ran `sudo umount /dev/nvme0n1p1` then did my lsblk command again to confirm.

If you don't want to keep typing sudo, you can run `sudo -i` to identify yourself as the super user. **WARNING**: Make sure you are using _your_ partition names. They might not be the same as my examples below. Ex: My `nvme0n1p5` might be your `nvme0n1p4`.

Mount the root partition
```
mount /dev/nvme0n1p5 /mnt
```
Create the home directory
```
mkdir /mnt/home
```
Mount the home partition
```
mount /dev/nvme0n1p4 /mnt/home
```
Create a directory to mount the boot/EFI partition
```
mkdir -p /mnt/boot/efi
```
Mount the EFI partition
```
mount /dev/nvme0n1p1 /mnt/boot/efi
```
Activate the swap
```
swapon /dev/nvme0n1p3
```

### Preparing Configuration.nix

Remember how NixOS is a declaratively defined operating system? Everything we're about to install will be declared in configuration.nix. And since we have an internet connection, you could probably just download a valid config file right now and not do anything manually. In fact, here's my [configuration.nix](https://github.com/raymondgh/nix-config/blob/v0.1.1/configuration.nix) that I ended up with after finishing this guide. But if you've come so far with me doing things the hard way, why not continue? Here's the minimal config I want to implement right now:

1. Add general t2 mac hardware support
2. Add wifi drivers
3. Store the home directory contents on my home partition
4. Anything else to get the machine to work
5. Install a desktop environment

First things first, let's generate our starter configuration.nix file. There's a command for that:
```
nixos-generate-config --root /mnt
```
This actually generates two files. We'll only focus on `configuration.nix` for now, as the other one, `hardware-configuration.nix` shouldn't need to be modified. You'll notice in a moment that the main config file imports the hardware config file. Pretty neat. Apparently there's a lot of room to get creative with organizing config. For now we'll try to keep it simple.

Now let's edit it with Nano, the teeny tiny command line text editor. If you haven't already, cd back to root. To do that, I ran `cd ../..` and then `pwd` to confirm that I was in `/`. Really, just try running the next command and navigate with `ls`, `cd`, and `pwd` as needed. It's good to be comfortable with that kind of navigation!
```
nano /mnt/etc/nixos/configuration.nix
```
Up pops a bunch of cyan text. Mostly everything is commented out. It's nice that the installer gives us some guidance with this starter config rather than just an empty file.

The bottom of the screen should show the available controls. Arrow keys to navigate, CTRL+O to save, CTRL+X to exit. 

Let's work top to bottom.

First up is an import that we need to add. The top of your file probably looks like this:
```
imports =
  [ # Include the results of the hardware scan.
    ./hardware-configuration.nix
  ];
```
We need to add a line following the t2linux instructions so that it looks like this:
```
imports =
  [ # Include the results of the hardware scan.
    ./hardware-configuration.nix
    "${builtins.fetchGit { url = "https://github.com/kekrby/nixos-hardware.git"; }}/apple/t2"
  ];
```
I really don't like that the line turns red. But this is what the instructions say and if it doesn't work I'll come back and edit this part of my post. I don't know if the indentation matters, but I'm keeping things aligned and using spaces not tabs.

To the boot.loader section, add one line:
```
boot.loader.efi.efiSysMountPoint = "/boot/efi";
```
Next, networking.hostname. It's commented out right now, which is fine, but how do you want your device to be identified on the network? I'm going to have two nixos devices (laptop and desktop) so I'm going to uncomment this and change the name. So my line now looks like this:
```
networking.hostname = "nixos-mbp"
```
Continuing down the list, we need to choose a network solution: networking.wireless or networking.networkmanager. Uncomment `networking.networkmanager.enable = true;` and move on! Next up, timezone. I think the spelling counts here, so maybe skip it if you don't know yours. I'm setting mine to `time.timeZone = "America/Los_Angeles";`

I'm skipping the network proxy. In the internationalization section, I'm uncommenting only the first line:
```
i18n.defaultLocale = "en_US.UTF-8";
```
Next there's something about X11 windowing system. This is required if you want to use a desktop environment. I certainly do. If you go online you can find arguments about which is best. If you look long enough, you might find two people that agree. What's great about NixOS is that if we want to change what we have, we just come in here to the configuration and change it. 

So I am uncommenting the line
```
services.xserver.enable = true
```
and adding a few more to enable a Plasma desktop environment 
```
# Enable the Plasma Desktop Environment.
services.xserver.displayManager.sddm.enable = true;
services.xserver.desktopManager.plasma5.enable = true;
```
If you want GNOME instead of Plasma, change `sddm` to `gdm` and `plasma5` to `gnome` within those 2 lines.

Next there is some config about how the keyboard should work. I am using a US keyboard and don't need nothin' fancy. Uncommenting the first line does it for me:
```
services.xserver.layout = "us";
```
Skipping the part about CUPS and sound. Yes, sound 🥲. I think I read somewhere that sound is borked on Macbooks, so I will leave that for another day. There's a line to enable touchpad support. I'm not sure if this is the right way to do it, but I'm uncommenting that line as well
```
services.xserver.libinput.enable = true;
```
Then we have a user account. Apparently accounts are managed in config! Awesome! I wonder how passwords work though -- don't want those in config!! Here's what mine looks like
```
users.users.ray = {
    isNormalUser = true;
    description = "Ray";
    extraGroups = [ "networkmanager" "wheel" ];
    packages = with pkgs; [
      firefox
    ];
  };
```
I deleted thunderbird, but why not keep firefox. Below this, I added another important line:
```
# Allow unfree packages
nixpkgs.config.allowUnfree = true;
```
Remember, everything is a package. So if you want to use something [unfree](https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree) like VSCode, you need to allow it.

The [T2Linux guide](https://wiki.t2linux.org/distributions/nixos/installation/) instructs to add this, so let's add it:

```
# Firmware for Broadcom Wi-Fi and Bluetooth
hardware.firmware = [
  (pkgs.stdenvNoCC.mkDerivation {
    name = "brcm-firmware";

    buildCommand = ''
      dir="$out/lib/firmware"
      mkdir -p "$dir"
      cp -r ${./firmware}/* "$dir"
    '';
  })
];
```
*_Note: I have modified one line in the t2linux guide by removing `/files` so that our firmware will be stored at /etc/nixos/firmware/bcrm._

This is another spot where the syntax highlighting is no good. Nano thinks that `/*` is the beginning of a multi-line comment, so the following lines are all blue. Nix doesn't support multi-line comments as far as I can tell, and the expression is still valid. Maybe Nix does support it and it doesn't take effect because of the double apostrophe? Dunno! But this works, let's move on.

Last config change is to add the git package. Since we're relying on the `fetchGit` expression to get the hardware firmware, we need to make sure git is available. You can simply add `git` as a new line under your created user, but I'm going to add it as a system package instead of a user package. AFAIK this just makes it available to everyone instead of just one user. So lower down you'll see

```
# List packages installed in system profile. To search, run:
# $ nix search wget
#   environment.systemPackages = with pkgs; [
#     vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
#     wget
# ];
```
We will add `git` very simply so that it looks like this;
```
# List packages installed in system profile. To search, run:
# $ nix search wget
  environment.systemPackages = with pkgs; [
#   vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
    # wget
    git
  ];
```
_When I first installed, I didn't realize I needed to add git. When I booted into my machine for the first time, I had to use an external keyboard to comment out the fetchGit hardware line near the top, add git, rebuild, then go an uncomment the fetchGit hardware line and rebuild & reboot again._

[This](https://github.com/raymondgh/nix-config/tree/v0.1.1) is what the final configuration.nix and hardware-configuration.nix should look like.

Lastly before installing, we also need to copy the Broadcom wireless firmware into the nixos partition. First create the appropriate directory:
```
mkdir -p /mnt/etc/nixos/firmware/brcm
```
Then issue the copy pasta command:
```
cp -r /lib/firmware/brcm/* /mnt/etc/nixos/firmware/brcm/
```

### Install NixOS

Run
```
nixos-install
```

Another awesome thing about NixOS: if something is wrong, it's gonna halt the installation and tell you right away. You get to fix it _before_ having the chance to screw up your system. I had a couple typos in my configuration.nix when I got to this point. Once it gets going, you'll hear the airplanes take off (the laptop fan). Lots of white text scrolling across the screen. I don't really read it. Looks like a lot of `building '/nix/store/...`. 

Eventually it will prompt you for a new root password and report `Installation finished!`! Woot!


### Boot NixOS on Macbook Pro

We're still running the installer, so let's reboot with this command:
```
reboot
```
If everything went right, Macbook's startup manager will run and show 3 options: EFI Boot (Internal), Macintosh HD (Internal), and EFI BOOT (external).

NixOS should be running now, with access to internet and ability to log in to a user named `root` with the password you set during installation. Once logged in, you can set a password for your user with a command like this
```
passwd ray
```
It'll ask you to type and retype the desired password. Now I don't know how passwords work yet in NixOS, but from what I can tell it's not a fully solved problem. Perhaps we'll explore that together another day.

### Booting macOS and NixOS selectively

I don't know if you'll encounter these, but I had to do a few extra things before I could get into NixOS on the Mac SSD. 

From my installation media, I had to run an additional command to install the bootloader because my original configuration.nix had the boot partition mounted at `/boot` instead of `/boot/efi`. To do that, I followed some steps [here](https://nixos.wiki/wiki/Bootloader#FAQ). Specifically I ran `nixos-enter` from my installation media and then once that gave me the `>` text entry prompt, I ran
```
NIXOS_INSTALL_BOOTLOADER=1 /nix/var/nix/profiles/system/bin/switch-to-configuration boot
```
After that. I could only boot NixOS, not macOS. So I reset the nvram by booting with `CMD+OPTION+P+R`, disabled SIP with `csrutil disable` after booting in recovery mode `CMD+R`, and finally reinstalled rEFInd via macOS.

So now I choose my boot partition successfully with rEFInd. I'd like to use the Mac's native startup manager instead but that'll take some figuring out.

### Congrats!

We got our Macbook Pro dual booting macOS and NixOS. We got our keyboard, touchbar, trackpad, and wireless networks working. Woo!! This post is labeled "day 4" but this took more than 1 day to figure out, I promise. I'm just glad to have this stable starting point from which I can continue to improve my OS. Maybe we'll reach the limits of NixOS eventually, but for now, I'm optimistic.

### To Do

The more I learn, the less I know. Here are some things I want to get back to:

1. Getting a stable & desirable boot management configuration set up
2. NixOS keeps trying to have 2 wired connections but there's only one... Would like to fix that.
3. Getting audio to work
4. If I leave my laptop open and running for a long time and come back, I can't get the screen to turn back on. Want to fix that
5. Setting up a smooth integration between my configuration.nix and a github repository

And of course, installing on my desktop, potentially getting my two environments in sync, and setting up a development environment for some software project or another.

### Thank you!

If you went through all this, I would appreciate your feedback. Anything on what you liked or would have liked to have seen would be great! I can update this post with improvements.