apollo: Main microVM host

{
  <<imports>>

  networking.hostName = "apollo";
  networking.hostId = "fb53aded";
  nixpkgs.hostPlatform = "x86_64-linux";
  system.stateVersion = "25.05";

  <<config>>

  _class = "nixos";
}

apollo used to be to be a pretty pitiful OptiPlex with a 6th-gen i5, but when that started failing, I rebuilt it with some more modern parts, making it sort of a "server of Theseus". With a beefy Ryzen 9 3900X, it became capable of running almost every service in the lab without really breaking a sweat.

boot.initrd.availableKernelModules = [
  "xhci_pci"
  "ahci"
  "nvme"
  "usbhid"
  "sd_mod"
];
boot.kernelModules = [ "kvm-amd" ];
hardware.enableRedistributableFirmware = true;

This comes from nixos-generate-config based on detected hardware in the machine.

fileSystems."/nix" = {
  device = "rpool/crypt/nix";
  fsType = "zfs";
};

mjm.zfs.enable = true;
mjm.zfs.encryption.rootPool = "rpool";

apollo uses ZFS. For important filesystems that need to be mounted in stage 1, I set mountpoint=legacy and mount them with fstab. apollo is using ZFS native encryption with support from my ZFS module.

fileSystems."/persist" = {
  device = "rpool/crypt/persist";
  fsType = "zfs";
  neededForBoot = true;
};

mjm.state.persistDir = "/persist";
system.etc.overlay.mutable = false;

apollo is using an ephemeral root filesystem, so / is a tmpfs and things that must be persisted are bind-mounted from /persist. The use of an ephemeral root also makes it easier to adopt the immutable etc overlay.

boot.zfs.extraPools = [ "slow" ];
mjm.nas.enable = true;

apollo also has two large spinning disks in it that are configured as a mirror in a pool called "slow". Like in other VM hosts, this pool is used for local backups of VM data, but apollo's is also used as a NAS. Files on the NAS are accessible via SMB/CIFS which is supported on basically every platform. Services that need to use files stored on the NAS use microVM shares to mount the necessary directories via virtiofsd.

fileSystems."/boot" = {
  device = "/dev/disk/by-partlabel/boot";
  fsType = "vfat";
  options = [
    "fmask=0077"
    "dmask=0077"
  ];
};

mjm.secureboot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;

apollo uses my secureboot module to set up Lanzaboote for SecureBoot. Unlike the other microVM hosts, since it has more modern hardware, it is able to use measured boot like my workstations do.

Lanzaboote is built on systemd-boot, so it needs the EFI system partition mounted at /boot.

mjm.server.enable = true;

This machine is running as a server, so it needs the common server infrastructure for my lab.

mjm.spire.agent.tpm.publicKeyHash =
  "3d356749d9ddb9de76464708a109c4c71676ea15f10aec23db5c624f7fb878e0";

Any of my machines that has a TPM uses it to attest to the SPIRE server and get its node identity. The public key hash here is specified only to create a registration entry that aliases this public key identity to the hostname of the machine.

mjm.microvm-host.enable = true;
mjm.microvm-host.zfsPrefix = "rpool/crypt";
mjm.networkd.macvlan.enable = true;
mjm.networkd.secondaryLinkName = "lan1";

apollo's primary purpose is to host microVMs (a lot of them), with their data living in the rpool/crypt/microvms dataset. All microVM hosts use MACVLAN to provide network devices to the VMs, and they all do so with a dedicated secondary NIC named lan1 (as opposed to lan0, which is reserved for the host).

nixpkgs.config.permittedInsecurePackages = [
  "olm-3.2.16"
];

Unfortunately, I still use some Matrix bridges that use olm, so I need to allow the insecure package. I do this here instead of in the relevant microVM config because this "nixpkgs.config" option doesn't merge well: if there are multiple entries in different modules, one will clobber the others. That's not generally how things work in NixOS modules, but it is in this case. That can be very surprising, so it makes more sense to declare this once in the host's config rather than risk it if another insecure package needs to be added later.

mjm.remote-builder.enable = true;

My microVM hosts also do double duty as remote builders for CI, specifically for running NixOS VM tests.

mjm.nut = {
  enable = true;
  connectedUPSName = "smart500";
};

All physical hosts in my homelab run NUT to be able to be shutdown properly in the case of power loss. Since I have two UPSes in the rack, each machine is configured with the one it is connected to, so that will be the one that it monitors.

microVMs

=> abra
=> arcanine
=> bellsprout
=> bulbasaur
=> diglett
=> eevee
=> electabuzz
=> espeon
=> flareon
=> gengar
=> goldeen
=> hypno
=> ivysaur
=> jigglypuff
=> jolteon
=> jynx
=> lapras
=> magmar
=> nidoqueen
=> primeape
=> rhyhorn
=> scyther
=> seadra
=> slowbro
=> snorlax
=> spinarak
=> staryu
=> tangela
=> umbreon
=> vaporeon
=> venusaur
=> voltorb

These VMs are configured in their own files which are imported into the host's module:

imports = [
  ./abra.nix
  ./arcanine.nix
  ./bellsprout.nix
  ./bulbasaur.nix
  ./diglett.nix
  ./eevee.nix
  ./electabuzz.nix
  ./espeon.nix
  ./flareon.nix
  ./gengar.nix
  ./goldeen.nix
  ./hypno.nix
  ./ivysaur.nix
  ./jigglypuff.nix
  ./jolteon.nix
  ./jynx.nix
  ./lapras.nix
  ./magmar.nix
  ./nidoqueen.nix
  ./primeape.nix
  ./rhyhorn.nix
  ./scyther.nix
  ./seadra.nix
  ./slowbro.nix
  ./snorlax.nix
  ./spinarak.nix
  ./staryu.nix
  ./tangela.nix
  ./umbreon.nix
  ./vaporeon.nix
  ./venusaur.nix
  ./voltorb.nix
];