Hosts for microVMs

{
  lib,
  config,
  pkgs,
  inputs,
  localModulesPath,
  nodes,
  ...
}:
let
  cfg = config.mjm.microvm-host;

  <<grouped-units>>
in
{
  imports = [
    "${inputs.microvm}/nixos-modules/host"
  ];

  options.mjm.microvm-host = {
    <<options>>
  };

  <<vm-extension>>

  config = lib.mkMerge [
    {
      microvm.host.enable = cfg.enable;
    }
    (lib.mkIf cfg.enable {
      <<config>>
    })
  ];
}

I have several servers whose whole purpose is to run services inside microVMs. This module sets up the infrastructure for that on the host side.

Options

enable = lib.mkEnableOption "MicroVM host";

Not all machines are meant to host microVMs, so this behaviour must be enabled.

zfsPrefix = lib.mkOption {
  type = lib.types.str;
};

The prefix for where microVMs will be stored on disk. This prefix should contain a "microvms" dataset that is mounted at /var/lib/microvms.

VM config extensions

options.microvm.vms = lib.mkOption {
  type = lib.types.attrsOf (
    lib.types.submodule (
      { name, ... }: {
        config = {
          autostart = false;
          specialArgs = {
            inherit
              inputs
              localModulesPath
              name
              nodes
              ;
            hostConfig = config;
          };
          config.imports = [ "${localModulesPath}/nixos" ];
        };
      }
    )
  );
};

There a few options I want to set on every microVM. Because the VMs are defined as an attribute set of submodules, the way to do this is to add another definition of the option, which will be merged with the one from microvm.nix.

My microVMs disable the autostart option, because although I do want them to start automatically, I want more control than the autostart option offers. More on that later.

I pass along several specialArgs received from the host, along with an additional hostConfig argument with the config from the host, so the VM config can make choices based on the host it's running on.

Finally, the VMs automatically import my various NixOS modules, just like the hosts do.

Config

systemd.services =
  let
    vmServices = lib.concatMapAttrs (name: vm: {
      <<install-microvm>>
      <<spiffe-proxy>>
    }) config.microvm.vms;
  in
  {
    <<spiffe-proxy-template>>
  }
  // vmServices;

I need to add/update a few systemd services to set up my microVMs the way I like.

"install-microvm-${name}" = {
  path = [ config.boot.zfs.package ];
  script = lib.mkBefore ''
    zfs create -p ${cfg.zfsPrefix}/microvms/${name}/var
  '';
};

The "install-microvm-$name" service is set up by microvm.nix, and it starts by creating the /var/lib/microvms/$name directory. I want my VMs to each be in their own ZFS dataset, and for the "var" share they all have to be its own as well. Adding this ZFS command to the start of the script for the service does the job without me needing to do it manually, and it ensures that the directory isn't created before I have a chance to make it a dataset.

"microvm-spiffe-proxy@" = {
  description = "SPIFFE Workload API proxy for MicroVM '%i'";
  before = [ "microvm@%i.service" ];
  after = [
    "local-fs.target"
    "microvm-set-booted%i.service"
  ];
  partOf = [ "microvm@%i.service" ];
  restartIfChanged = false;
  serviceConfig = {
    ExecStart = "${pkgs.vm-proxy}/bin/vm-proxy vsock:notify.vsock:9999 unix:${config.mjm.spire.agent.socketPath}";
    WorkingDirectory = "/var/lib/microvms/%i";
    SyslogIdentifier = "microvm-spiffe-proxy@%i";
    Restart = "always";
    RestartSec = "5s";
    User = "microvm";
    Group = "kvm";
    Type = "notify";
  };
};

The microvm-spiffe-proxy@ services exist to support letting microVMs attest themselves as SPIRE nodes. Each microVM will run an instance of this templated service, which just exposes the host's SPIRE agent socket to the VM via VSOCK on port 9999. The VM can then use this connection to request a workload SVID that it can then exchange with the SPIRE server for a node SVID.

"microvm-spiffe-proxy@${name}" = {
  requiredBy = [ "microvm@${name}.service" ];
  serviceConfig.X-RestartIfChanged = [
    ""
    vm.restartIfChanged
  ];
  path = lib.mkForce [ ];
  overrideStrategy = "asDropin";
};

This override for each individual instance of the SPIFFE proxy service mimics others in microvm.nix. It propagates the "restartIfChanged" property from the microVM's configuration, and also introduces a requirement so that the main microvm service pulls in this unit. Setting requiredBy in the template doesn't work for this purpose.

mjm.spire.entries = lib.flip lib.mapAttrs' config.microvm.vms (
  name: vm: {
    name = "spire-exchange-${name}";
    value = {
      spiffe_id = "spire-exchange/${name}";
      parent_id = config.networking.hostName;
      selectors = [
        {
          type = "systemd";
          value = "id:microvm-spiffe-proxy@${name}.service";
        }
      ];
    };
  }
);

For the SPIFFE proxy for each VM to work as expected, each instance needs a SPIRE registration entry. Since the vm-proxy running inside the service is going to be the one connecting to the host's SPIRE agent socket, assigning an entry to the systemd service for the proxy will ensure that's the only SPIFFE ID given for certificates requested by the VM.

These certificates are scoped under a spire-exchange/ prefix, as the server is configured so other IDs that don't match that can't be exchanged for node identities.

microvm.host.startupTimeout = 360;

The default timeout for starting VMs is a little too short in my experience, so I increase it to six minutes.

systemd.targets.microvms-early-pre = {
  description = "Ready to start early MicroVMs";
  before = earlyVmUnits;
  partOf = earlyVmUnits;
};
systemd.targets.microvms-early = {
  description = "MicroVMs that should start first";
  wants = earlyVmUnits ++ [ "microvms-early-pre.target" ];
  wantedBy = earlyVmUnits;
  partOf = earlyVmUnits;
};
systemd.targets.microvms-late-pre = {
  description = "Ready to start late MicroVMs";
  before = lateVmUnits;
  partOf = lateVmUnits;
  wants = [ "microvms-early.target" ];
};
systemd.targets.microvms-late = {
  description = "MicroVMs that should start last";
  wants = lateVmUnits ++ [ "microvms-late-pre.target" ];
  partOf = lateVmUnits;
  wantedBy = lateVmUnits ++ [ "multi-user.target" ];
};

This mess of targets is my attempt to be able to group VMs into two stages. One of my machines runs over 30 microVMs, and trying to start them all at the same time usually causes a bunch of unnecessary contention.

The intent of these targets is that once microvms-early-pre is started, the microVMs in the "early" group can begin to start. Once those VMs have all started, the microvms-early target starts. This then allows the microvms-late-pre target to start, which then allows the microVMs in the "late" group to begin to start. And finally, when all of those are done starting, the microvms-late target starts, signalling the end of starting all microVMs.

The PartOf dependencies exist to ensure the targets get stopped when the VMs get stopped, which combined with the targets being WantedBy the VMs, should continue to enforce the grouping of VMs even when restarting them while activating a new NixOS generation.

earlyVmUnits = unitsForGroup "early";
lateVmUnits = unitsForGroup "late";
unitsForGroup =
  group:
  lib.pipe config.microvm.vms [
    lib.attrNames
    (lib.filter (vm: config.microvm.vms.${vm}.config.config.mjm.profiles.microvm.group == group))
    (lib.map (vm: "microvm@${vm}.service"))
  ];

The target definitions rely on lists of the relevant microvm@ service units, which are defined here based on the mjm.profiles.microvm.group option.

mjm.zfs.backups.roots = [ "${cfg.zfsPrefix}/microvms" ];

microVM data is backed up via the backups support in my own ZFS module.

mjm.deploy.tests = lib.concatMapAttrs (_: vm: vm.config.config.mjm.deploy.tests) config.microvm.vms;

Any NixOS tests that a microVM declares should run as part of a deploy get lifted to the VM host, as that's the machine that the deploy machinery actually sees.