SPIRE server

{
  config,
  lib,
  pkgs,
  nodes,
  ...
}:
let
  cfg = config.mjm.spire;
in
{
  options.mjm.spire.server = {
    enable = lib.mkEnableOption "SPIRE server";
  };

  config = lib.mkIf cfg.server.enable {
    <<config>>
  };
}

The SPIRE server runs on a single machine and provides certificates and keys to workloads running on hosts that run the SPIRE agent.

mjm.services.spire = {
  postgresql = {
    enable = true;
    databases = [ "spire-server" ];
  };
};

The SPIRE server will store its data in a PostgreSQL database, so I enable my common PosgreSQL support.

systemd.services.spire-server =
  let
    hcl1 = pkgs.formats.hcl1 { };
    configFile = hcl1.generate "spire-server.hcl" {
      <<server-settings>>
    };
  in
  {
    description = "SPIRE Server";
    wantedBy = [ "multi-user.target" ];
    after = [
      "network.target"
      "postgresql.service"
    ];
    preStart = ''
      <<server-gen-root-cert>>
    '';
    serviceConfig = {
      Type = "exec";
      ExecStart = "${pkgs.spire-server}/bin/spire-server run -config ${configFile}";
      StateDirectory = "spire-server";
      RuntimeDirectory = "spire-server";
      DynamicUser = true;

      <<server-hardening>>
    };
  };

The SPIRE server needs a systemd service to run. NixOS has a module for this now, but it didn't when I set up SPIRE initially, so I'm doing this the old-fashioned way. That module made some different choices about where to put things and doesn't have the hardening I've added, so there's not a lot of motivation to migrate.

Now let's break down the config file for the SPIRE server.

server.socket_path = "/run/spire-server/api.sock";
server.data_dir = "/var/lib/spire-server";
plugins.KeyManager.disk.plugin_data.keys_path = "/var/lib/spire-server/keys.json";
plugins.UpstreamAuthority.disk.plugin_data = {
  cert_file_path = "/var/lib/spire-server/root.crt";
  key_file_path = "/var/lib/spire-server/root.key";
};

First we can set up all of the paths it will use to store various things. The socket path is under the RuntimeDirectory declared for the service above. The rest are under the StateDirectory.

The keys path for the KeyManager plugin is something that the SPIRE server will generate itself. The UpstreamAuthority plugin on the other hand is meant to read a root CA that comes from outside. To make sure it exists, we use a preStart script on the systemd service to create it.

if [ ! -e /var/lib/spire-server/root.crt ]; then
  ${pkgs.openssl}/bin/openssl req \
    -subj "/C=/ST=/L=/O=/CN=home.mattmoriarity.com" \
    -newkey rsa:2048 -nodes -keyout /var/lib/spire-server/root.key \
    -x509 -days 3650 -out /var/lib/spire-server/root.crt
fi

SPIRE needs a root CA for signing all of its certificates, so if there isn't one present, generate a 10 year certificate for that. Obviously this shouldn't need to happen often, so the only reason it's really in the config like this is it makes it impossible to forget how it should be generated.

Back to the server's config file.

server.trust_domain = "home.mattmoriarity.com";
server.jwt_issuer = "https://spire.midna.dev";

Each SPIRE server is responsible for managing a single trust domain (the authority part of the URIs used for SPIFFE IDs), so that's configured here. Similarly, we also set the issuer set on JWT tokens it produces, which some consumers of those tokens might want to verify.

plugins.NodeAttestor.join_token.plugin_data = { };
server.agent_ttl = "48h";
server.ca_ttl = "288h";

At least one NodeAttestor plugin needs to be set up so that SPIRE agents can attest their identities to the server. I'm using the simplest version of this, which is the join token plugin. In this setup, a token must be generated on the server, which maps a UUID (the token) to a SPIFFE ID for the agent. The agent is configured to use that token to join if it needs to attest. The tokens have a limited TTL and are one-time-use.

When a SPIRE agent attests its identity to the server, it is issued a certificate which it then has to continue to renew regularly. If it fails to renew the certificate in time, it will attempt to reattest. This doesn't work for join tokens, since they are one-time-use, so it's important that the certificates live long enough to tolerate any downtime an agent might experience, or you need to manually generate a new one and reconfigure the agent to use it to get it going again.

Obviously there's a trade-off here between security and convenience, but at this point I've chosen to allow the agent certificates to be valid for 2 days, which allows for decent amounts of unexpected downtime before manual intervention is required. This change requires increasing the TTL for CA certs, so that those can be properly rotated without invalidating the agent certificates prematurely.

plugins.DataStore.sql.plugin_data = {
  database_type = "postgres";
  connection_string = "dbname=spire-server host=/run/postgresql";
};

The SPIRE server will store its data on a PostgreSQL instance running on the same machine. There's no need to manage auth credentials, as it will just connect to a local UNIX socket and use peer authentication.

health_checks = {
  listener_enabled = true;
  bind_port = "8080";
};
telemetry.Prometheus = {
  host = "[::]";
  port = 8082;
};

A separate listener needs to be enabled for health checks, which will be set up with Consul later. Similar for metrics for Prometheus. Note that Prometheus metrics are scraped from outside the host, so that listener needs to bind externally.

That's all for SPIRE's config file. I've also gone to the trouble of hardening the SPIRE server's systemd service.

CapabilityBoundingSet = "";
DevicePolicy = "closed";
LockPersonality = true;
MemoryDenyWriteExecute = true;
PrivateDevices = true;
PrivateIPC = true;
PrivateUsers = "identity";
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectProc = "invisible";
ProtectSystem = "strict";
RestrictAddressFamilies = [
  "AF_INET"
  "AF_INET6"
  "AF_UNIX"
];
RestrictNamespaces = true;
RestrictRealtime = true;
SystemCallArchitectures = "native";
SystemCallErrorNumber = "EPERM";
SystemCallFilter = [
  "@system-service"
  "~@resources @privileged"
];
UMask = "0027";

These Service settings for the systemd service lock it down to only the capabilities it actually needs to do its job.

networking.firewall.allowedTCPPorts = [
  8081
  8082
];

The SPIRE server will listen for HTTPS requests from agents on port 8081. The metrics listener was configured for port 8082. So those ports need to be open in the firewall.

mjm.consul.services.spire-server = {
  port = 8081;

  metrics.enable = true;
  metrics.port = 8082;

  blockDeploy = true;

  checks.up = {
    http.path = "/ready";
    http.port = 8080;
  };
};

While the agents will contact the server machine directly by hostname, it still gets a Consul service registered to enable health checking and metrics scraping. The blockDeploy setting instructs my deployment tooling to wait for this Consul service to be healthy before advancing to deploying the next host. That's useful for the SPIRE server because if it's not healthy, deploying to other machines is likely to cause more problems due to restarted VMs needing to get new certs. Better to stop and figure out what's wrong before continuing.

systemd.services.spire-provision =
  let
    <<provision-entries>>
  in
  {
    description = "Provision SPIRE Registration Entries";
    wantedBy = [ "multi-user.target" ];
    after = [ "spire-server.service" ];
    requires = [ "spire-server.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = "${pkgs.spiffe-tool}/bin/spire-provision ${entriesJson}";
      DynamicUser = true;
      User = "spire-server";
      Restart = "on-failure";
      RestartSec = 5;

      <<provision-hardening>>
    };
  };

In addition to the server itself, I also run a oneshot service to ensure the SPIRE server has all of the registration entries it should. These entries determine what SPIFFE ID (if any) to use for a given workload. When I first set up SPIRE, I would create these manually from the CLI as needed, but it quickly became tedious and hard to keep track of.

While I could have figured out how to incorporate this into my Pulumi setup, that would have required having credentials that could administer the SPIRE server externally. Instead, I opted for this service that runs on the machine itself, so it's using the same SSH auth as machine deploys do.

I wrote a Go tool that reads the desired entries as JSON and compares them to the existing entries on the server, and then makes the necessary changes to achieve the desired state. The entries JSON is generated in Nix and is part of the systemd service config, so this service will re-run whenever the entries change.

Here's how those entries are generated:

entriesJson = lib.pipe nodes [
  lib.attrValues
  (map (node: (lib.filterAttrs (_: v: v.enable) node.config.mjm.spire.entries)))
  lib.mergeAttrsList
  lib.attrValues
  (map (v: lib.removeAttrs v [ "enable" ]))
  (jsonFormat.generate "spire-entries.json")
];
jsonFormat = pkgs.formats.json { };

Put simply, this looks at each host in my config, including microVMs, and gathers all the enabled entries under the mjm.spire.entries option. The entries are merged into a single list, the Nix-only enable option is removed, and the rest is dumped into a JSON file.

CapabilityBoundingSet = "";
DevicePolicy = "closed";
LockPersonality = true;
MemoryDenyWriteExecute = true;
PrivateDevices = true;
PrivateIPC = true;
PrivateUsers = "identity";
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectProc = "invisible";
ProtectSystem = "strict";
RestrictAddressFamilies = [
  "AF_INET"
  "AF_INET6"
  "AF_UNIX"
];
RestrictNamespaces = true;
RestrictRealtime = true;
SystemCallArchitectures = "native";
SystemCallErrorNumber = "EPERM";
SystemCallFilter = [
  "@system-service"
  "~@resources @privileged"
];
UMask = "0027";

Just like the SPIRE server itself, this provisioning service is using hardening options from systemd to lock it down.

environment.systemPackages = [
  (pkgs.execline.writeScriptBin ",spire" "-S0" ''
    ${pkgs.spire-server}/bin/spire-server $@
      -socketPath /run/spire-server/api.sock
  '')
];

For the situations that are not covered by the provisioning service, like generating join tokens, I have a simple ,spire command that prefills the correct socket path. This makes it easier to use the CLI to make API calls to the server.