SPIRE server

{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.mjm.spire;
  <<nodeAttestor>>
in
{
  options.mjm.spire.agent = {
    <<options>>
  };

  config = lib.mkIf cfg.agent.enable {
    <<config>>
  };

  _class = "nixos";
}

The SPIRE agent runs on each machine that wants to run workloads that will need SPIFFE identities. It connects to the SPIRE server, attests its own identity, and then supplies identities to workloads running on the machine upon request.

Options

enable = lib.mkEnableOption "SPIRE agent";

The SPIRE agent needs to be enabled on each machine. Usually this is done automatically by the server module.

trustDomain = lib.mkOption {
  type = lib.types.str;
  default = cfg.trustDomain;
};

The agent's trust domain is set automatically from the higher level mjm.spire.trustDomain option.

serverAddress = lib.mkOption {
  type = lib.types.str;
  default = if cfg.server.enable then "127.0.0.1" else "arges.home.mattmoriarity.com";
};

The agent needs to know what SPIRE server to connect to. This defaults to the hostname of the Raspberry Pi that runs my SPIRE server, unless the machine is also running the SPIRE server, in which case it targets itself.

socketPath = lib.mkOption {
  type = lib.types.path;
  default = "/run/spire-agent/api.sock";
  readOnly = true;
};

This read-only option exposes the path of the Unix socket that workloads will use to communicate with the agent. Other modules can read this value instead of hard-coding the path.

joinToken = lib.mkOption {
  type = lib.types.str;
  default = "";
};

I'm using two different node attestors in my lab: join tokens and TPM. Machines that are using join tokens can specify the token using this option. This only needs to be done once per token, since join tokens are invalid after they are used once. From that point on, the agent will keep renewing its identity regularly, and this option can be unset.

tpm.publicKeyHash = lib.mkOption {
  type = lib.types.str;
  default = "";
};

This specifies the hash of the public key of the endorsement key (EK) of the machine's TPM. This hash can be obtained by running get_tpm_pubhash on the machine. If this option is set, the agent will be configured to use the TPM node attestor instead of join tokens, and a SPIFFE ID based on this machine's hostname will be aliased to the SPIFFE ID based on this hash that the attestation will produce.

Config

users.users.spire-agent = {
  isSystemUser = true;
  group = "spire-agent";
};
users.groups.spire-agent = { };

Due to an issue in systemd, the SPIRE agent needs a real non-dynamic user to be able to query systemd about what unit a PID is running under.

=> systemd: DynamicUser is denied access to dbus-daemon after a recent lockup bugfix

nodeAttestor =
  if config.microvm.guest.enable then
    "x509pop"
  else if cfg.agent.tpm.publicKeyHash != "" then
    "tpm"
  else
    "join_token";

Various pieces of the config that follows need to conditionalize on which node attestor is being used for this machine, so a binding for that is defined at the top of the module. MicroVMs use the x509pop (X.509 Proof-of-Possession) attestor, which will be explained a bit more later when it is configured. Other hosts use the TPM attestor if a public key hash is configured, or join tokens otherwise.

systemd.services.spire-agent =
  let
    hcl1 = pkgs.formats.hcl1 { };
    configFile = hcl1.generate "spire-agent.hcl" {
      <<agent-settings>>
    };
  in
  {
    description = "SPIRE Agent";
    wantedBy = [ "multi-user.target" ];
    after = [ "network.target" ] ++ lib.optional cfg.server.enable "spire-server.service";
    startLimitIntervalSec = 0;

    serviceConfig = {
      Type = "exec";
      ExecStart = lib.concatStringsSep " " (
        [
          "${pkgs.spire-agent}/bin/spire-agent"
          "run"
          "-config"
          configFile
        ]
        ++ lib.optionals (nodeAttestor == "join_token") [
          "-joinTokenFile"
          "\${CREDENTIALS_DIRECTORY}/spire_agent_join_token"
        ]
      );
      StateDirectory = "spire-agent";
      RuntimeDirectory = "spire-agent";
      User = "spire-agent";
      Group = "spire-agent";
      SupplementaryGroups =
        lib.optional config.virtualisation.podman.enable "podman"
        ++ lib.optional (nodeAttestor == "tpm") config.security.tpm2.tssGroup;
      Restart = "always";
      RestartSec = "5s";
      LoadCredential = "spire_agent_join_token";

      <<agent-hardening>>
    };
  };

The SPIRE agent needs a systemd service to run. NixOS has a module for this now, but it didn't when I set up SPIRE initially, so I'm doing this the old-fashioned way. That module made some different choices about where to put things and doesn't have the hardening I've added, so there's not a lot of motivation to migrate.

There's a few special things here. If the agent is running on the same machine as the SPIRE server, then it's ordered to start after the server. If using join tokens for attestation, the command-line is adjusted to read it from a systemd credential. And the groups for the process are set according to some specific needs for accessing Podman or the TPM.

Now let's break down the config file for the SPIRE agent.

agent.socket_path = cfg.agent.socketPath;
agent.data_dir = "/var/lib/spire-agent";

First we can set up the paths it will use to store various things. The socket path is under the RuntimeDirectory declared for the service above. Persistent data is under the StateDirectory.

agent.trust_domain = cfg.agent.trustDomain;

The agent needs to know the trust domain it is supposed to operate under.

agent.server_address = cfg.agent.serverAddress;
agent.server_port = 8081;
agent.trust_bundle_path = "${./trust.pem}";

This configures the agent to know how to communicate with the SPIRE server. The trust bundle is important, as when setting up a new agent, it is used to be able to securely communicate with the server. This bundle includes the root CA certificate that is used for all certificates issued by the server, including its own.

plugins.NodeAttestor =
  if nodeAttestor == "x509pop" then
    {
      x509pop.plugin_data = {
        spiffe_endpoint_socket = "unix:/run/spire-exchange-proxy/api.sock";
      };
    }
  else if nodeAttestor == "tpm" then
    {
      tpm = {
        plugin_cmd = "${pkgs.spire-tpm-plugin}/bin/tpm_attestor_agent";
        plugin_data = { };
      };
    }
  else
    {
      join_token = {
        plugin_data = { };
      };
    };

The SPIRE agent must have exactly one node attestor plugin configured. The TPM and join token attestors are pretty self-explanatory, but the x509pop one is worth a little more attention. The way it generally works is that the node must provide a valid X.509 certificate from a particular CA, and then the server will issue an identity to the node based on that certificate.

Conveniently, this plugin has a mode where it uses the SPIRE server's own CA for this. In short, this allows a workload SVID under a particular prefix (/spire-exchange by default) to be exchanged for a node SVID. To provide the certificate to the microVM, the host runs a small proxy for each microVM that exposes the SPIFFE workload API from the host's SPIRE agent to the VM over VSOCK. The service that proxy runs in has an entry so that it provides a /spire-exchange/$name SVID to the guest. Then the guest runs a similar proxy to expose that VSOCK listener over a Unix socket instead, and the SPIRE agent inside the guest is configured to get the X.509 certificate for attestation from that Unix socket.

plugins.KeyManager =
  if nodeAttestor == "join_token" then
    { disk.plugin_data.directory = "/var/lib/spire-agent"; }
  else
    { memory.plugin_data = { }; };

The key manager for the agent is needed to remember the current node SVID. When using join tokens, it's important that this information is persisted to disk, because it's not possible for the agent to reattest without a new join token. The other two attestors do support reattestation though: they both rely on information that the agent should be able to provide at any time. For that reason, it's better for them only keep their keys in memory, so there is not some sensitive material on disk that could be exfiltrated.

plugins.WorkloadAttestor = {
  systemd.plugin_data = { };
}
// lib.optionalAttrs config.virtualisation.podman.enable {
  docker.plugin_data = { };
};

The agent needs workload attestor plugins to determine which identity will be issued to any given process. The most common one used in my lab is the systemd plugin. Individual systemd services can be assigned specific identities by service name.

On my CI runner, I also access the agent from within Podman containers, so on that machine, the docker plugin is enabled as well.

health_checks = {
  listener_enabled = true;
  bind_port = "18080";
};
telemetry.Prometheus = {
  host = "[::]";
  port = 9988;
};

networking.firewall.allowedTCPPorts = [ 9988 ];

A separate listener needs to be enabled for health checks, which will be set up with Consul later. Similar for metrics for Prometheus. Note that Prometheus metrics are scraped from outside the host, so that listener needs to bind externally and have its port allowed through the firewall.

That's all for the SPIRE agent's config file. I've also gone to the trouble of hardening the systemd service.

CapabilityBoundingSet = "";
DeviceAllow = lib.mkIf (nodeAttestor == "tpm") "char-tpm";
DevicePolicy = "closed";
LockPersonality = true;
MemoryDenyWriteExecute = true;
NoNewPrivileges = true;
PrivateDevices = nodeAttestor != "tpm";
PrivateIPC = true;
PrivateMounts = true;
PrivateTmp = true;
PrivateUsers = "identity";
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectSystem = "strict";
RemoveIPC = true;
RestrictAddressFamilies = [
  "AF_INET"
  "AF_INET6"
  "AF_UNIX"
];
RestrictNamespaces = true;
RestrictRealtime = true;
RestrictSUIDSGID = true;
SystemCallArchitectures = "native";
SystemCallErrorNumber = "EPERM";
SystemCallFilter = [
  "@system-service"
  "~@resources @privileged"
];
UMask = "0027";

This is mostly normal systemd hardening boilerplate, but there are a few exceptions due to the use of the TPM. Specifically, when the TPM is in use, private devices cannot be enabled. And while we can still use a closed device policy, that requires explicitly allowing access to TPM character devices.

environment.etc."credstore/spire_agent_join_token".text =
  if (config.microvm.guest.enable || cfg.agent.joinToken == "") then
    "no-token-provided"
  else
    cfg.agent.joinToken;

This is gross and I should really find a better way to handle it. Maybe it will just go away once enough machines are using TPM attestation.

The idea here is to read the join token via a credential provided to the system manager. For microVMs, that will be done via SMBIOS, as doing so does not require altering the NixOS config for the VM. This means that once a VM is attested, the join token can be removed from the config without forcing a reboot of the VM. Honestly, kind of a marginal benefit for the awkwardness imposed here. On normal machines, the token is just placed in a file in /etc/credstore, which systemd will read.

The hackiness largely comes from the fact that the SPIRE agent will error if you provide a join token file that doesn't exist. So to avoid needing the config to change based on whether one is present or not, the credential always needs to exist with something in it. Hence the weird conditional seen above.

I wonder if a better approach would be to start the agent using a script that checks for the file's presence, and decides whether to pass the -joinTokenFile parameter based on that. But it would honestly be better to be rid of join tokens entirely.

mjm.spire.entries."${config.networking.hostName}-tpm" = lib.mkIf (nodeAttestor == "tpm") {
  spiffe_id = config.networking.hostName;
  parent_id = "spire/agent/tpm/${cfg.agent.tpm.publicKeyHash}";
};
security.tpm2.enable = lib.mkIf (nodeAttestor == "tpm") true;

If TPM attestation is being used, then a registration entry is created on the SPIRE server that aliases the host name to the SPIFFE ID that will be assigned based on the hash. This entry doesn't match any workloads, but instead means that entries for workloads can use the hostname as the parent ID. Join tokens get an alias in the same way, but generated imperatively when the token is created. This way, other entries can be ignorant of the specific method used for attestation.

The other TPM-specific consideration here is to enable TPM2 support in NixOS, as this creates the relevant group and udev rules needed for the agent to access the TPM device.

mjm.spire.entries."${config.networking.hostName}-x509pop" = lib.mkIf (nodeAttestor == "x509pop") {
  spiffe_id = config.networking.hostName;
  parent_id = "spire/agent/x509pop/${config.networking.hostName}";
};

Similar to the TPM case above, the ID issued to a microVM using X.509 attestation will have a path scoped to that plugin. This alias makes it match the existing convention of just using the hostname.

mjm.consul.services.spire-agent = {
  metrics.enable = true;
  metrics.port = 9988;

  checks.up = {
    http.path = "/ready";
    http.port = 18080;
  };
};

A Consul service is setup to monitor the health of the agent and configure Prometheus to scrape it for metrics.

environment.systemPackages = [
  (pkgs.execline.writeScriptBin ",spire" "-S0" ''
    ${pkgs.spire-agent}/bin/spire-agent $@
      -socketPath ${cfg.agent.socketPath}
  '')
];

For convenience, I expose a ,spire command on the machine that can be used to communicate with the agent. It prefills the command with the socket path of the agent, so that doesn't need to be specified with every single command invocation.