Installing a Custom NixOS Image on a Raspberry Pi

January 23, 2024 | 15 min. read

In my homelab, I have four Raspberry Pis running on NixOS. For the first Pi I installed NixOS on, I went through the traditional boot process:

  1. Find the official NixOS .iso image, and copy it onto a USB drive
  2. Find a spare monitor and keyboard, and plug the Pi into them
  3. Plug in the USB and boot the RasPi off of the NixOS install .iso
  4. Install the OS manually by clicking around a GUI, then edit the configuration.nix file

This definitely works, but it's a huge pain. I have to dig around through my messy cable drawer and assemble an ad-hoc Pi Installation Station, editing the configuration.nix on a fresh install without my usual text editor and tools is annoying, etc. I also use Flakes for all of my desktop/laptop NixOS configs, so unifying all my Pi hosts with the flake outputs would be ideal.

Luckily, it's pretty easy to create custom NixOS images to boot a RasPi from. It doesn't even require a USB drive, or any manual intervention other than inserting an SD card into the Pi. Here's how it's done:

Write the Configuration

The first step, as you might guess, is to write the desired NixOS configuration for the Pi that you want to install NixOS on. Throughout this guide, I'll be using Nix Flakes. The general approach will be the same if you're not using Flakes, but the arguments for a few shell commands will be a little different (and left as an exercise for the reader). Our example flake.nix will look like this:

# flake.nix

{
  description = "my flake setup";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs, ... }@inputs: rec {
  
    # here goes the other flake outputs, if you have any

    nixosConfigurations."pi" = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        ./configuration.nix
      ];
    };
  };
}

Then, you can write your configuration.nix:

# configuration.nix

{ pkgs, ... }:

{
  imports = [
    ./hardware-configuration.nix
  ];
  
  # Use the extlinux boot loader. (NixOS wants to enable GRUB by default)
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;
  
  # networking config. important for ssh!
  networking = {
    hostName = "pi";
    interfaces.end0 = {
      ipv4.addresses = [{
        address = "192.168.1.42";
        prefixLength = 24;
      }];
    };
    defaultGateway = {
      address = "192.168.1.1"; # or whichever IP your router is
      interface = "end0";
    };
    nameservers = [
      "192.168.1.1" # or whichever DNS server you want to use
    ];
  };
  
  # the user account on the machine
  users.users.admin = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
    hashedPassword = "blablabla" # generate with `mkpasswd`
  };

  # Enable the OpenSSH daemon.
  services.openssh.enable = true;

  # I use neovim as my text editor, replace with whatever you like
  environment.systemPackages = with pkgs; [
    neovim
    wget
  ];

  # allows the use of flakes
  nix.package = pkgs.nixFlakes;
  nix.extraOptions = ''
    keep-outputs = true
    keep-derivations = true
    experimental-features = nix-command flakes
  '';

  # this allows you to run `nixos-rebuild --target-host admin@this-machine` from
  # a different host. not used in this tutorial, but handy later.
  nix.settings.trusted-users = [ "admin" ];

  # ergonomics, just in case I need to ssh into
  programs.zsh.enable = true;
  environment.variables = {
    SHELL = "zsh";
    EDITOR = "neovim";
  };
}

This is a big example, but a few key points should be made. SSH is enabled by default to allow us to get a remote shell post-installation. Double check that the IP address for the end0 interface is outside of your network's DHCP range (if applicable), and that the gateway and nameserver IPs are valid for your router and DNS setup.

Also, we're creating a user named admin on the machine, and giving them access to sudo by adding them to the wheel group. Change that if you don't want sudo access or I don't recommend that, and doing so will prevent you from doing remote deployments (which is covered in the last section of this tutorial). . Be sure, though, to set the hashedPassword attribute to the output of mkpasswd -m <hashing algorithm of choice> <your password here>. This circumvents having to run passwd and manually set the password on the Raspberry Pi, so we'll be able to just ssh into the machine as admin from the get-go.

Here's the hardware-configuration.nix that I used. Unfortunately, from what I can tell, there's no way to run nixos-generate-config based on the hardware of a different device, so I had to mostly copy+paste this from the Raspberry Pi 4 that already had NixOS installed on it. I'm also pretty sure this will work on other models of Pi, but not 100% sure. Caveat emptor.

# hardware-configuration.nix

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" "usbhid" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-label/NIXOS_SD"; # this is important!
      fsType = "ext4";
      options = [ "noatime" ];
    };

  swapDevices = [ ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;
  # networking.interfaces.end0.useDHCP = lib.mkDefault true;
  # networking.interfaces.wlan0.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux";
}

Building the SD Image

For the custom image, we'll be building an SD image rather than an .iso. This allows us to just use the SD card that the Raspberry Pi uses as storage to install the operating system. With an .iso, we'd also need a USB drive to burn the .iso onto and then use as the boot media.

To build the SD image, we'll use the nixos-generators tool. Also be sure to stage all of your new files in Git if you haven't already, so that the Flake will be able to find them. Now, we can run:

$ nix run nixpkgs#nixos-generators -- -f sd-aarch64 --flake .#pi --system aarch64-linux -o ./pi.sd

That'll chug for a while, but it should eventually spit out a symlink named pi.sd that links to a directory in /nix/store containing the SD image. We'll have to copy it out of the Nix store and decompress it before we can use it.

$ cd pi.sd/sd-image
$ ls 
nixos-sd-image-24.05.20231124.5a09cb4-aarch64-linux.img.zst
$ cp nixos-sd-image-24.05.20231124.5a09cb4-aarch64-linux.img.zst ~/ && cd ~
$ unzstd -d nixos-sd-image-24.05.20231124.5a09cb4-aarch64-linux.img.zst -o nixos-sd-image.img

You should now see a file named nixos-sd-image.img that's at least a few gigabytes in size. That'll be our custom image that we'll install onto the SD card.

Installing the SD Image

Grab your SD card, and insert it into whatever machine you've build the SD image on. We'll need to find the name of the block device for the SD card, so we know which device to copy the image data onto. You can do this with sudo fdisk -l, and the outputs should look like the following:

$ sudo fdisk -l
Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       2048  2099199  2097152    1G  c W95 FAT32 (LBA)
/dev/mmcblk0p2      2099200 31116287 29017088 13.9G 83 Linux

In that example, the name of the SD card block device is /dev/mmcblk0. The p1 and p2 are just partitions on the device, which we don't need to care about. With that info, we can then run:

$ sudo dd if=~/nixos-sd-image.img of=/dev/mmcblk0 bs=1M status=progress

That's also going to chug along for a while, but you should see the data slowly copy onto the device. After it's done, pop the SD card into the Raspberry Pi and plug in the Ethernet and power cable. Don't worry about grabbing a keyboard and monitor - this should work automagically! The red LED should be solid, and the green LED will blink for a while. To double check that it boots and connects to the network, run ping on the IP that we put in the configuration.nix:

$ ping 192.168.1.42
PING 192.168.1.42 (192.168.1.42) 56(84) bytes of data.
64 bytes from 192.168.1.42: icmp_seq=1 ttl=64 time=0.257 ms
64 bytes from 192.168.1.42: icmp_seq=2 ttl=64 time=0.386 ms
64 bytes from 192.168.1.42: icmp_seq=3 ttl=64 time=0.222 ms
^C
--- 192.168.1.42 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2087ms
rtt min/avg/max/mdev = 0.222/0.288/0.386/0.070 ms

We can also try to SSH into the machine with the password we created for the admin user:

$ ssh admin@192.168.1.42
The authenticity of host '192.168.1.42 (192.168.1.42)' can't be established.
ED25519 key fingerprint is SHA256:ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.42' (ED25519) to the list of known hosts.
(admin@192.168.1.42) Password:
Last login: Tue Jan 23 19:54:47 2024 from 192.168.1.120

[admin@pi:~] $

And we're good to go! The SD image isn't like an ISO install image; the operating system has already been fully installed as the SD card's filesystem and can be treated like any other server now.

Bonus: Installing New Configs Remotely

Another problem that arises when you have more than a few NixOS hosts is configuration drift. Flakes do help with this a little, since you can have every NixOS hosts' configuration as a different output of the same flake and use the flake as a kind of configuration monorepo. Even so, this usually means a workflow something like this:

  1. You want to make a change to one of your servers from the coziness of your desktop/laptop workstation.
  2. Update the Nix file associated with the server, and push the changes to your Git remote.
  3. Time to SSH into the server, pull the new changes, and run nixos-rebuild.
  4. If there's any issues, you change the configuration from the server and rebuild until it's working.

This definitely works, but it can really bite you if you forget to push any changes to your Git remote after making any changes on the server. That's when configuration drift (and a lot of nasty merge conflicts) occur.

Thankfully, there's a few solutions that allow us to never leave the comfort of our workstation when it's time to deploy a NixOS configuration. Unfortunately, most of them are either unofficial, unmaintained, or both. NixOps is a NixOS project, but it's de facto abandoned. Morph is another popular tool for deployments, but it's not affiliated with Nix/NixOS officially and it doesn't look like it supports Flakes. Thankfully, nixos-rebuild has a pretty simple flag, --target-host, that allows us to build a NixOS configuration on one machine and deploy it on another machine over SSH.

To use the --target-host flag, you'll need to double check that you have the nix.settings.trusted-users field in your configuration.nix set to your user (see the above example). You'll also have to use the --use-remote-sudo flag. This is needed because everything will be built on the local machine, and then copied over into the /nix/store of the target host. Of course, you can avoid this whole song and dance by just using the root user on the target host and enabling root access over SSH, but that's not recommended for security reasons.

The magic command is then as follows:

$ nixos-rebuild switch --flake .#pi --target-host admin@192.168.1.42 --use-remote-sudo

In my experience, this works... but not flawlessy. If, like on most systems, you require users to type in a password to use sudo, you'll get hit with the following error:

$ nixos-rebuild switch --flake .#pi --target-host admin@192.168.1.42 --use-remote-sudo
building the system configuration...
(admin@192.168.1.42) Password: 
copying 0 paths...
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

Essentially, the sudo running on 192.168.1.42 can't ask us for our password, since it's running remotely rather than in the same TTY instance. To appease the computer gods, we must invoke the magic env var NIX_SSHOPTS:

$ NIX_SSHOPTS="-o RequestTTY=force" nixos-rebuild switch \
  --flake .#pi \
  --target-host admin@192.168.1.42 \
  --use-remote-sudo

Here's where things get very odd, and I'm not sure if I'm running into a bug or I've just misconfigured something and tooling is tripping over it. Here's a walkthrough of what happens when I run that command:

$ NIX_SSHOPTS="-o RequestTTY=force" nixos-rebuild switch \
  --flake .#pi \
  --target-host admin@192.168.1.42 \
  --use-remote-sudo
building the system configuration...
(admin@192.168.1.42) Password:

We build the NixOS configuration for pi (I've already done so, hence the lack of usual build logs). We then get prompted for the admin password, so we can SSH in and copy the build output to the target host. I type in my password as usual.

...
copying 0 paths...
[sudo] password for admin:

I've already installed this NixOS configuration, so we don't need to copy any new /nix/store paths over. Regardless, I'm prompted for the admin password by sudo as if I did (not the surprising part).

...
Shared connection to heracles.lab.janissary.xyz closed.

The program then hangs, until I press Ctrl-C.

...
[sudo] password for admin:

The program comes back to life, and I get prompted by sudo for my password, again???

...
activating the configuration...
setting up /etc...
reloading user units for admin...
setting up tmpfiles
reloading the following units: -.mount
Shared connection to 192.168.1.42 closed.

...and the target host is able to install the new config, and restart just fine???

This is super duper weird, but it definitely works. I made small but detectable changes to the NixOS config and went through the flow again several times to be sure, and the end result was This felt very much like falling through that one illusory spike pit in Super Metroid . You think you've done something horribly wrong, but it's actually totally fine. .

Conclusion

In a previous blog post, I mentioned trying to install NixOS on a Raspberry Pi via the official ISO and running into weird issues. Thankfully, I was able to dig through some GitHub issues and documentation to find all the info I've laid out above. It worked like a charm for me when Nixifying all of my machines, and I hope it does for you too.

If you happen to have any thoughts/feedback/trouble with this tutorial, feel free to send me an email at the address in my About page.