Building a Homelab, Part 4 - Nixification, Kubernetes

This is the fifth post in a series on the homelab that I've been building. If you haven't read the previous posts, here they are:

• Part 0
• Part 1 - Rackmounting and DNS
• Part 2 - LDAP and Single Sign-On
• Part 3 - Patch Panel, Email, Minecraft Server

Alright, this is the big project that I've been builing up to for a while now. To reiterate previous posts, I have four Raspberry Pis in my homelab that run everything - Jellyfin, qBittorrent, Calibre, PiHole, BIND, etc. I've acquired each of the Pis over the span of a few years and I haven't put in the effort to set up any infrastructure as code, so they've all experienced quite a bit of configuration drift. They each have slightly different versions of my dotfiles, they differ in which utilities and bash scripts are installed, the versions of Raspbian they run are quite varied, etc. It's not too bad to manage four fairly unique hosts and remember which is which, but each new host or VM I add is going to increase complexity exponentially. Essentially, I need to start treating the machines as cattle, not pets.

Additionally, I want to be able to treat all of the machines as identical nodes of a single cluster of compute, rather than configure each machine and allocate services individually. This is primarily for convenience, since I would like to be able to just throw YAML at a container orchestrator until it spits out the services I want. Secondarily, it would be nice to have some resiliency if one of the Pis' SD cards kicks the bucket or something - instead of an outage, whatever services running on the Pi would just be scheduled to run on the other nodes of the cluster (ideally!).

Game Plan

As a result, there are two big projects that I've been working on for a while now. The first is installing NixOS on every machine that hosts a service in the lab (so not the NAS or my router, but all the Pis). I've been using NixOS on my desktop and non-MacOS laptops for about two years now, and it's been awesome. If you haven't heard of NixOS, it's a really cool project Disclaimer: I maintain a few packages in nixpkgs and I've contributed some documentation to the project, so I do have some skin in the game other than being a user. . It's a Linux distro built on top of Nix, a purely functional package manager that aims to make software 100% reproducible (package derivations are also built in a domain-specific functional programming language called nix, which can get a little confusing). Because it's built on top of a really solid package manager that views every build artifact as a pure function of "source code and build tools in, software out", NixOS allows for total declarative management of your operating system. No more fiddling with config files in /etc and losing them, or misremembering if it's foo or foo-dev or foo-git from the AUR that has that one weird thing you need. Every single package and service you want (and every option you want to specify for those packages/services) on your machine is declared in a single configuration.nix file. Obviously, this is very handy for wrangling a fleet of servers like the homelab.

Managing configs is just half the battle, though - managing compute is a whole other beast. For that task, I've been eyeing self-hosting Kubernetes for a while now. I've worked with managed Kubernetes instances quite a bit in the past for work (mainly EKS), but self-hosting is a whole other beast. With EKS, you can basically click a few buttons in the AWS console and all of the hard stuff is done; you can start copy/pasting writing Deployments in a few minutes. When self-hosting, it's so much more work - you need to set up static addresses for each node, set up a Certificate Authority, generate TLS certs, install a CNI plugin, etc. Just looking at Kubernetes the Hard Way is enough to make your head spin.

Nihil Sub Nix Novum

As you might have seen in a previous blog post, I've been gradually installing Nix on all of my machines for a while now, and I have the system down pretty well.

My NixOS configs live in my dotfiles, where I have a single Nix Flake that contains all of my Nixified machines' configs. I've tried pretty hard to template things such that all common functionality (e.g. every machine should use the same fonts, every machine should have Flakes enabled, all the RasPis should be more or less identical, etc.) lives in a common folder (src) that can be used with the imports attribute in each machine's respective NixOS module (example). This isn't totally flawless, though, as some things are pretty identical across machines but just different enough to make a mess of things. For example, NixOS uses the fonts.packages attribute to determine which fonts are installed, but nix-darwin uses fonts.fonts. For the most part I've just let things stay duplicated, but here and there I do explicit pkgs.stdenv.isLinux checks to help determine which attributes need to be present in a NixOS module.

As with any declarative configuration tool, the one really tricky part is secrets management. Part of the fun of having configuration as code is checking your configs into git and linking the repo on /r/unixporn, but this gets a lot trickier when you want your declarative configuration to contain stuff like user passwords or private SSH keys. There's a few different tools in NixOS for thisThis blog post gives a great overview of what's available. , but unfortunately none of them are first-party. For a while I went with just .gitinore'ing a common/secrets.nix file that contains all my sensitive data and doing an import ../secrets.nix to fetch that data in any of the hosts' configs that needed it. This is pretty suboptimal for two reasons:

• When the derivation is built, the secret data is copied into /nix/store and available in plaintext to anyone with permission to view the derivation.
• I'm far too lazy to manually copy around that secrets.nix file across all my machines, so if I want to do rebuild any machine that needs secret data remotely then I need to do it from the machine with the secrets.nix file.

I ended up settling on agenix to properly manage my secrets, and it's been a pretty good experience so far. It's built on top of a general purpose encryption tool called age, and uses SSH keys to asymetrically encrypt/decrypt any secrets you want to add to your NixOS configs. Since secrets are encrypted at rest, you can check them into git repos with a clear conscience. The six-step tutorial in the GitHub repo is literally all you need to get up and running.

That being said (and I'm not 100% on this, please correct me if I'm wrong), I think there's a chicken-and-egg problem that would prevent the kind of zero-click installs that I've been doing so far. With the older secrets.nix method of managing secrets, I could build an image on any machine with the secrets.nix file, burn it onto the installation media, and boot from it on the machine I want to install NixOS on. After that, no config is needed; everything that I need on the machine was included in the install image. With agenix, if you want to share a secret with a machine or user, you have to explicitly state in the config "This SSH public key has access to this secret". That's great and all for granular access controls, but it makes me wonder - what about the scenario I've described above, where I'm writing the configs for a machine that doesn't exist yet (and therefore doesn't have any SSH public keys)?

I guess the simplest solution here is abandoning the "zero-click" part of things, and justing add a few manual steps to the install process. The new flow would basically be something like:

1. Write the new machine's configs, create the install image as usual, boot the new machine from it
2. SSH into the machine
3. Get the newly generated host keys
4. Add the public key to the permitted public keys of the agenix secrets
5. Encrypt and add the secrets to new machine's configs
6. Rebuild the new machine with the updated config and encrypted secrets

That makes me a little sad, though - I really enjoyed being able to stick an SD card into a machine, power it on, and immediately SSH into a fully-configured server. Regardless, it's still a huge step up from doing all of my configuration by hand on multiple machines and having zero record of what's present on any given machine.

K3s, Please

On the declarative compute side of things, things are also going pretty swimmingly. While researching other homelabbers' self-hosted Kubernetes clusters, the suggestion I got again and again was to check out a project called K3s. Turns out, it fits my use case pretty perfectly.

It's a Kubernetes "distribution" optimized for edge and IoT devices (like Raspberry Pis!) that does all of the heavy lifting in terms of installation and setup. In fact, the installation process (for both control plane and worker nodes) is a simple curl | bash commandI know a lot of people absolutely hate that pattern for installing software, but I've gotta be honest: I don't find the arguments super compelling. I've seen a lot of people say that it's a security nightmare, but I fail to see how it differs from running any other arbitrary binary on your system. I mean, downloading a package from the AUR (for example) is basically doing the same thing in that you're sourcing a random binary/bash script that you're really trusting to not act maliciously. If we've learned anything from the xz debacle, it should be that even "properly" packaged software can have pretty gnarly side effects.. k3s also includes some pretty handy stuff out-of-the-box, like Traefik for proxying services and HTTPS.

As part of this project, I also decided to actually learn Kubernetes. Like I mentioned earlier, I have a good amount of experience wrangling managed Kubernetes instances at work, but most of that was following playbooks that others had written or updating pre-existing manifests. I decided to read through Kubernetes Up and Running, which was highly recommended and definitely more approachable than reading through the official docs top-to-bottom. It was a pretty quick read, and (not to brag) didn't actually have too much stuff I didn't already know about. What I did find surprising, however, is how little a vanilla, self-hosted Kubernetes cluster actually implements. If you want to use Ingresses, you have to install a separate Ingress Controller. LoadBalancer type Services don't actually do anything, unless you're running a managed K8s cluster from a cloud provider (or, like k3s, you have something like ServiceLB). Ditto with networking - you're practically required to install a Container Networking Interface plugin like Calico or Flannel to be able to network between pods. I'm sure there's good reasons to keep this functionality out of the K8s project itself (I don't even pretend to understand the inner workings of Kubernetes), but it sure makes learning it harder.

Regardless, having the servers run NixOS made installation even easier. It's literally just the following in each machines' configuration.nix:

  services.k3s = {
    enable = true;
    role = "agent"; # on the server, this is "server" instead
    serverAddr = "https://192.168.1.42:6443";
    tokenFile = config.age.secrets.k3s-token.path;
  };

I didn't want to go with a super-fancy High Availability setup, so I just made heracles (a Raspberry Pi model 4B) the server node and ixion (another 4B) and athena (a 3B) the agent nodes:If you've noticed, the last Pi gorgon isn't part of the cluster. It's only a Model 2B, so it's way too wimpy to run most of the stuff in the lab.

❯ k get nodes
NAME       STATUS   ROLES                       AGE    VERSION
athena     Ready    <none>                      43d    v1.28.6+k3s2
heracles   Ready    control-plane,etcd,master   138d   v1.29.3+k3s1
ixion      Ready    <none>                      116d   v1.27.6+k3s1

Next, I just had to migrate all of my disparate docker-compose.yml files that used to run my services into Kubernetes manifests. This wasn't too hard - most Deployments kind of read like a Docker Compose file but with a little more info - but it did take a lot of copying and pasting. Part of me wanted to utilize Helm to template things and make them more DRY, but I figured I was already reaching the limit of justifiable complexity by running K8s in the first place.

The only tricky part of migrating to k3s was importing my funky Traefik config. You can read about it more in this post, but I have a local BIND instance that is the authoritative DNS server for all of my services' and machines' hostnames (anything in *.lab.janissary.xyz). I could never quite get BIND to cooperate with DNS challenges for ACME, so I had to point Traefik to 1.1.1.1 specifically for ACME challenges. Since CloudFlare lives outside the homelab, it only sees the records delegated to DigitalOcean (which is *.janissary.xyz). I then had to create a wildcard record in DigitalOcean for *.lab.janissary.xyz, so that Lego (the ACME client Traefik uses) can properly create the TXT records that ACME uses to verify ownership of domains. This is kind of a huge hack and misuse of DNS, but it manages to work smoothly enough anytime I need a new TLS cert and I haven't experienced any issues with it yet. Porting this setup to k3s took some digging through the documentation, but I managed to get it working by writing this small HelmChartConfig:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - --api.insecure
      - --accesslog
      - --providers.kubernetescrd
      - --certificatesresolvers.myresolver.acme.dnschallenge.provider=digitalocean
      - --certificatesresolvers.myresolver.acme.dnschallenge.resolvers=1.1.1.1:53
      - --certificatesresolvers.myresolver.acme.email=<my email>
      - --certificatesresolvers.myresolver.acme.storage=/data/acme.json
    env:
      - name: DO_AUTH_TOKEN
        valueFrom:
          secretKeyRef:
            name: digitalocean-auth-token
            key: token

That manifest essentially edits the Traefik deployment that k3s sets up by default. After symlinking that into the k3s server's /var/lib/rancher/k3s/server/manifests/traefik-config.yaml and restarting k3s.service, it was all good to go.

After I migrated the old services into k8s and verified that they were working as expected, I slowly started deleting the old docker-compose.yml files. I also went ahead and installed some new services to test how easy it was compared to the old Docker Compose method, and thankfully it was as easy as I was hoping. A little copy-pasting is involved since each service requires a Deployment, a Service, an IngressRoute, and usually a PersistentVolume and accompanying PersistentVolumeClaim, but it's a breeze to stand up something new and have it automatically schedule to a node in the cluster. Not to mention, I can now check all of this stuff into git.

In fact, it's probably a bit too much of a breeze to stand up new services - I went a little hog wild and installed a ton of new stuff. But I won't go into too much detail - that's for next time ;)