Vault secrets with SPIFFE and systemd-creds

#| file: services/spire/creds.nix
{
  lib,
  config,
  pkgs,
  ...
}:
let
  cfg = config.mjm.spire;
in
{
  <<options>>

  config = lib.mkIf cfg.agent.enable {
    <<config>>
  };

  _class = "nixos";
}

Any non-trivial deployment of software services needs to manage secrets: bits of configuration that should not be accessible to outside actors. NixOS doesn't have a built-in way to handle secrets, as there isn't really consensus on a best way to do it. There are community options like sops-nix and agenix, which both generally involve keeping encrypted secrets alongside your Nix code and having those secrets decrypted only at runtime. There is the manual approach, where you manually write secrets into files on the machine that the service is configured to read. If you don't want to do either of these, you pretty much need to roll your own.

I've been using Vault for secrets since before I used Nix at all. Around it, I have built a bespoke setup for getting secrets to services running on NixOS. It relies on systemd's credentials mechanism, and particularly the ability to load secrets from a Unix domain socket. Using this, secrets are retrieved dynamically from Vault when a service is started, without the service needing to know anything about Vault. All it needs is the ability to read the secret from a given file path. Authentication to Vault for the service that actually retrieves the secrets is done using SPIFFE JWT tokens.

There are more details about what I've done here in a few other places:

=> nixos/services/secrets module
=> spiffe-creds service code

Options

options.mjm.spire.creds = lib.mkOption {
  default = { };
  type =
    lib.types.attrsOf
    <| lib.types.submodule {
      options = {
        aliases = lib.mkOption {
          type = with lib.types; attrsOf str;
          default = { };
        };
      };
    };
};

mjm.spire.creds is an attribute set where the keys are service names. Under each service, the aliases option can be used to define any aliases need to map a systemd credential name to a particular secret in Vault. This should only be needed in situations where the upstream NixOS module is naming the credentials, so I don't have control over the names to make them match spiffe-creds expectations.

options.systemd.services =
  let
    <<credServiceType>>
    <<credKeyType>>
  in
  lib.mkOption {
    type =
      lib.types.attrsOf
      <| lib.types.submodule (
        { name, config, ... }:
        {
          options.credentials = lib.mkOption {
            default = { };
            type = lib.types.attrsOf (credServiceType name);
          };

          config = {
            <<systemd-service-config>>
          };
        }
      );
  };

This module also extends the normal systemd.services options with an additional option: credentials. This gives a more succinct way to add Vault credentials to a systemd service. The credentials option on a systemd service should be an attribute set where the keys are the names of services that credentials will be pulled from.

credServiceType =
  serviceName:
  with lib.types;
  coercedTo (listOf str) (lib.flip lib.genAttrs (_: { }))
  <| attrsOf (credKeyType "${serviceName}.service");

The value for each service is canonically another attribute set, where this time, the key is the name of a particular secret in Vault for that service. As you'll see below, there aren't any options that need to be manually set under that key, so it's also possible to set the value for the service to a list of names of secrets, and it will be coerced into an attribute set.

credKeyType =
  unit:
  lib.types.submodule (
    {
      name,
      config,
      options,
      ...
    }:
    let
      svcName = options.name.loc |> lib.dropEnd 2 |> lib.last;
    in
    {
      options = {
        name = lib.mkOption {
          type = lib.types.str;
          default = "${svcName}.${lib.replaceStrings [ "/" ] [ "." ] name}";
          readOnly = true;
        };
        path = lib.mkOption {
          type = lib.types.path;
          default = "/run/credentials/${unit}/${config.name}";
          readOnly = true;
        };
        loadCredential = lib.mkOption {
          type = lib.types.str;
          default = "${config.name}:/run/${svcName}-creds.sock";
          readOnly = true;
        };
      };
    }
  );

The value for each secret is a submodule with a few different options on it. All of them are read-only, so they are not supposed to be set. The options here are useful for referencing via config, so a module can easily find the path to a secret on disk for instance.

#| id: systemd-service-config
serviceConfig.LoadCredential =
  config.credentials
  |> lib.attrValues
  |> lib.concatMap lib.attrValues
  |> map (x: x.loadCredential);

Each secret entry for each service becomes an entry in the LoadCredential setting on the service, telling systemd to load the credential of the given name from a socket belonging to the service that owns the secret.

Config

systemd.sockets."spiffe-creds@" = {
  description = "SPIFFE Credentials Helper Socket for '%i'";
  partOf = [ "spiffe-creds@%i.service" ];
  socketConfig = {
    ListenStream = "/run/%i-creds.sock";
    SocketMode = "0600";
  };
};

A service that is using secrets gets a socket at /run/$service-creds.sock that it can use to request secrets in systemd services using LoadCredential. This is defined as a single template unit. The nixos/services module will create drop-ins to add a WantedBy=sockets.target dependency for the individual instances where secrets are enabled.

systemd.services =
  let
    <<dropIns>>
  in
  {
    <<service-template>>
  }
  // dropIns;

This module will define a single template service to match the template socket unit above. It will also create drop-ins as needed to specify aliases for individual services.

#| id: service-template
"spiffe-creds@" = {
  description = "SPIFFE Credentials Helper for '%i'";
  requires = [ "spiffe-creds@%i.socket" ];
  after = [
    "network.target"
    "spiffe-creds@%i.socket"
  ];

  environment = {
    OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318";
    OTEL_RESOURCE_ATTRIBUTES = "deployment.environment.name=prod";
    SPIFFE_ENDPOINT_SOCKET = "unix:${cfg.agent.socketPath}";
    VAULT_ADDR = "https://vault.service.consul:8200";
  };

  serviceConfig = {
    Type = "notify";
    ExecStart = lib.concatStringsSep " " [
      "${pkgs.spiffe-tool}/bin/spiffe-creds"
      "serve"
      "--path"
      "prod/services/%i"
    ];
    DynamicUser = true;

    <<hardening>>
  };
};

The service to handle credential requests is socket-activated: it only runs when a service actually requests credentials from it. It's designed to stay running once activated until a few minutes pass without any requests for credentials, at which point it terminates until a new request starts it again.

The service supports tracing, so there is some OpenTelemetry environment variables to configure that. And it needs to communicate with the SPIFFE workload API and with Vault, so there are environment variables for that as well.

The code for the service itself is described in much more detail on its own page.

=> spiffe-creds

#| id: hardening
CapabilityBoundingSet = "";
DevicePolicy = "closed";
LockPersonality = true;
MemoryDenyWriteExecute = true;
PrivateDevices = true;
PrivateIPC = true;
PrivateUsers = "identity";
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectProc = "invisible";
ProtectSystem = "strict";
RestrictAddressFamilies = [
  "AF_INET"
  "AF_INET6"
  "AF_UNIX"
];
RestrictNamespaces = true;
RestrictRealtime = true;
SystemCallArchitectures = "native";
SystemCallErrorNumber = "EPERM";
SystemCallFilter = [
  "@system-service"
  "~@resources @privileged"
];
UMask = "0077";

This is pretty standard systemd service hardening. The service is locked down pretty tight, but it does need to make network requests to access Vault.

dropIns =
  cfg.creds
  |> lib.filterAttrs (_: { aliases, ... }: aliases != { })
  |> lib.mapAttrs' (
    name:
    { aliases, ... }:
    lib.nameValuePair "spiffe-creds@${name}" {
      overrideStrategy = "asDropin";
      environment.SECRET_ALIASES =
        aliases |> lib.mapAttrsToList (k: v: "${k}=${v}") |> lib.concatStringsSep " ";
    }
  );

Any services that define aliases to map a credential name to a particular secret in Vault will need a drop-in file to set the SECRET_ALIASES environment variable to that particular instance of spiffe-creds, which will then read those and use them when resolving credential names to secrets.