S3-compatible storage with Garage

{
  pkgs,
  lib,
  config,
  ...
}:
let
  cfg = config.mjm.garage;
in
{
  imports = [
    ./spiffe.nix
  ];

  options.mjm.garage = {
    enable = lib.mkEnableOption "garage";
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };
}

Garage provides S3-compatible object storage to various other services in the lab. It runs in a cluster of three nodes, each running on a different host machine.

Service settings

mjm.services.garage.secrets.enable = true;

Garage needs to store a shared RPC secret key in Vault to be able to securely communicate between nodes.

mjm.services.garage.http = {
  socket = "/run/garage/s3.sock";

  health.path = "/health";
  health.port = 3903;
  health.blockDeploy = true;

  metrics.enable = true;
  metrics.port = 3903;

  ingress = {
    subdomain = "garage";
    authMode = "none";
  };
};

Garage will be configured to listen on a Unix socket for its S3 API, and that's what will be exposed to others via Consul.

A health check endpoint is available on the admin API port. Deploys are blocked until Garage is healthy, as it is likely easier to fix a bad deploy if only one Garage node goes down rather than all three. Prometheus metrics are also exposed on the admin API port.

Garage is exposed externally at https://garage.midna.dev. Since it's implementing the S3 API, it already has an authentication mechanism, so it doesn't need additional protection.

Config

services.garage.enable = true;
services.garage.package = pkgs.garage_2;

Nixpkgs makes both Garage 1.x and 2.x available. I've upgrade to 2.x so I choose that package. My version of 2.x is overridden though to add a patch:

{
  oldpkgs,
  rustPlatform,
}:

oldpkgs.garage_2.overrideAttrs (oldAttrs: rec {
  patches = (oldAttrs.patches or [ ]) ++ [ ./fix-rustls.diff ];
  cargoDeps = rustPlatform.fetchCargoVendor {
    inherit (oldAttrs) src;
    inherit patches;
    hash = "sha256-CfMkvV28XqpD2PUE8VxSMr3ZOiLPE3bTpx7MI5Wpz/I=";
  };
})

The patch fixes a regression in the most recent version that broke Consul discovery.

services.garage.settings = {
  db_engine = "lmdb";
  replication_factor = 3;
  rpc_bind_addr = "[::]:3901";
};
networking.firewall.allowedTCPPorts = [ 3901 ];

These are basic recommended settings for Garage. LMDB is the currently best supported DB engine, though Fjall might replace it in the future. A replication factor of 3 is the minimum to get good clustering properties: all data is stored on three nodes, so two can be lost without losing data, and one can be lost without interrupting reads or writes.

services.garage.settings.s3_api = {
  s3_region = "home";
  api_bind_addr = "/run/garage/s3.sock";
};
systemd.services.garage.serviceConfig.RuntimeDirectory = "garage";

The S3 API listens on a Unix socket, since it will be exposed externally via a tunnel. A Garage node expects to serve a single region, so for my purposes, that region is always "home".

The /run/garage directory isn't normally used, so I need to add it to the systemd service config.

services.garage.settings.consul_discovery = {
  consul_http_addr = "http://127.0.0.1:8500";
  api = "agent";
  service_name = "garage";
  tags = [ "rpc" ];
};

My Garage nodes discover each other via Consul. They connect to the local agent on the machine.

mjm.consul.services.garage.serviceConfig.tags = [ "s3" ];
mjm.ingress.vhosts.garage.upstream.service.tag = "s3";
mjm.spire.entries.tunnel-garage-garage.dns_names = [ "s3.garage.service.consul" ];

The Consul service garage creates for itself advertises the RPC port, not the S3 port. This is why I also have instances registered for the S3 port in the service settings above. Both of these will use the service name "garage", but they will be tagged differently: s3 or rpc. The ingress needs to be configured to use the tag, and the DNS name for the SPIFFE certificate should include the tag as well.

services.garage.settings.admin.api_bind_addr = "[::1]:3903";

The admin API is used for a handful of tasks: health checks, metrics, and managing keys. It's only exposed locally to the node, but for actual administration operations, it still requires a token.

services.garage.extraEnvironment.GARAGE_RPC_SECRET_FILE = "%d/garage.rpc_secret";
systemd.services.garage.credentials.garage.rpc_secret = { };
services.garage.settings.allow_world_readable_secrets = true;

Each Garage node needs to share the same RPC secret to be able to communicate with each other securely. The secret data is pulled from Vault via a systemd credential, and the file with the secret is passed as an environment variable.

The permissions of the secret files from systemd credentials appear too permissive to Garage, because it doesn't realize they are stored within a directory that is inaccessible to other users. So I have to configure Garage to allow those permissions, knowing that they are actually safe.

systemd.services.garage.serviceConfig = {
  Restart = "on-failure";
  RestartSec = 5;
};

I don't want Garage to end up stuck in a failed state: it should attempt to restart if it exits unsuccessfully.

microvm.volumes = [
  {
    image = "data/garage-data.img";
    label = "garage";
    mountPoint = "/var/lib/private/garage/data";
    size = 300 * 1024;
    fsType = "xfs";
  }
];

systemd.services.garage.serviceConfig.StateDirectory = lib.mkForce "garage/meta garage/data";

Garage recommends keeping metadata on an SSD and data on an HDD. To do this, I have a "slow/microvms/$name/data" ZFS dataset on the host that is mounted at "/var/lib/microvms/$name/data". Within it, I create a disk image that is 300G and formatted as XFS, and mount that under the garage state directory. This lets me use the default data and metadata directories for Garage while keeping things split between SSD and HDD.

The one complication is that systemd doesn't like this with just a plain "garage" StateDirectory setting. I believe this has something to do with mount propagation, because that garage directory will now contain another mount within it. The solution is pretty simple though: declare garage/meta and garage/data as separate state directories.

mjm.spire.tunnels.garage.allowedServices = lib.mkForce [ ];

Because it's using my service infrastructure modules, the server tunnel for Garage won't accept traffic from services that don't declare it as an upstream. That's not really necessary, since even after going through the tunnel, the service still needs to provide credentials. So I'm forcing the allowed services list to be empty, which grants access to anyone. A valid SPIFFE certificate is still needed, but any will do.

environment.systemPackages = [
  (pkgs.execline.writeScriptBin "g" "-S0" ''
    systemd-run
      --service-type=exec
      --wait --quiet --pty
      --collect
      -p LoadCredential=garage.rpc_secret:/run/garage-creds.sock
      env GARAGE_RPC_SECRET_FILE=''${CREDENTIALS_DIRECTORY}/garage.rpc_secret
      ${config.services.garage.package}/bin/garage
      $@
  '')
];

The NixOS module for Garage creates a "garage" script that wraps the normal "garage" command to also source an environment file if one is set. For some reason, it doesn't include the extraEnvironment settings, but even if it did, those wouldn't work here because it wouldn't handle the systemd credentials correctly.

So I have my own "g" script that uses systemd-run to load the credential. Unfortunately, even systemd kind of drops the ball here: the options for setting environment variables in the systemd-run command don't have a way to access the credentials directory, so I'm resorting to using the env command to do that properly.

With this script, I can use "g" to administer the Garage cluster. Any "garage" commands the documentation suggests to run can be used by replacing "garage" with "g".

mjm.deploy.tests = {
  inherit (pkgs.nixosTests.garage_2)
    basic
    with-3node-replication
    ;
};

The NixOS tests that Nixpkgs has for Garage will run as part of my CI to hopefully avoid regressions.

Automatic access key provisioning

{
  pkgs,
  lib,
  config,
  nodes,
  ...
}:
let
  cfg = config.mjm.garage;
  trustDomain = config.mjm.spire.trustDomain;
in
{
  config = lib.mkIf cfg.enable {
    <<spiffe-config>>
  };

  _class = "nixos";
}

One of the nice things about using SPIFFE for identity is that it avoids the need to manage storage and delivery of credentials. I prefer to use this everywhere I can, but there are places where that isn't possible. SPIFFE supports X.509 certs and JWT tokens, neither of which can be used with Garage because of its need to be compatible with the S3 API. Garage instead has to use a pair of non-secret key ID and secret key.

So the next best thing is to be able to use an intermediary service to exchange a SPIFFE certificate for Garage credentials. The most transparent way would be as a proxy: a client makes a request to this service, which then makes the request to Garage on the client's behalf after checking permissions. But this would likely confuse clients, who are expecting to access an S3-like API that needs some kind of credentials. Perhaps static ones could be used and ignored, but eh.

Instead, I'm taking advantage of the fact that AWS also has this problem of needing to provide credentials to container workloads. Because of this, its SDKs, as well as third-party ones that want to support this too, support a variety of ways to transparently get credentials. I can expose the same API in my VMs, and those SDKs will fetch credentials from my service and use them automatically.

Each Garage node also runs a sidecar service, uncreatively named spiffe-garage, that checks the client certificate of the incoming request, and returns the credentials for the Garage key whose name matches the SPIFFE ID, if such a key exists. The infrastructure to allow clients to actually connect is setup in the services module.

=> client-side setup for services accessing garage

mjm.services.spiffe-garage = {
  secrets.enable = true;
};

The spiffe-garage service needs its own SPIFFE identity for clients to be able to trust it, and it uses a Vault secret to store its scoped admin token.

systemd.sockets.spiffe-garage = {
  description = "SPIFFE Garage Credential Socket";
  wantedBy = [ "sockets.target" ];
  partOf = [ "spiffe-garage.service" ];
  socketConfig.ListenStream = "[::]:3899";
};

networking.firewall.allowedTCPPorts = [ 3899 ];

spiffe-garage uses a systemd socket to listen on port 3899. This is the actual port exposed externally for clients to connect to. Since spiffe-garage cares about the specific identity of its clients, it can't be exposed with a tunnel, as that would swallow that information. So instead it listens directly and terminates its own TLS.

systemd.services.spiffe-garage = {
  description = "SPIFFE Garage Credential Service";
  wantedBy = [ "multi-user.target" ];
  after = [
    "network.target"
    "spiffe-garage.socket"
    "spire-agent.service"
  ];
  requires = [ "spiffe-garage.socket" ];

  environment = {
    SPIFFE_ENDPOINT_SOCKET = "unix:${config.mjm.spire.agent.socketPath}";
    OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318";
    OTEL_RESOURCE_ATTRIBUTES = "deployment.environment.name=prod";
  };

  credentials.spiffe-garage.admin_token = { };

  serviceConfig = {
    Type = "notify-reload";
    ExecStart =
      let
        <<configFile>>
      in
      "${pkgs.spiffe-tool}/bin/spiffe-garage --config ${configFile}";
    DynamicUser = true;
    Restart = "on-failure";

    <<spiffe-garage-hardening>>
  };
};

The actual systemd service for spiffe-garage is nothing too crazy. Despite being able to be activated by a socket, it's pulled in at boot, as a health check will keep it running constantly anyway and I think it makes for more obvious restart behavior from NixOS activation when it is changed.

The environment is configured for the SPIFFE Go APIs to automatically connect to the right socket for the SPIRE agent. This will be needed to get certificates for the service itself as well as the trust bundle for validating incoming client certificates. The other environment variables support tracing, but the collector they point to is currently not enabled because it introduces a circular dependency: spiffe-garage would depend on Tempo, but Tempo itself stores traces in Garage and therefore depends on spiffe-garage for credentials. Perhaps if Tempo was able to ingest traces without always having a connection to Garage, this cycle could be broken.

To be able to get credential keys from Garage, spiffe-garage needs to use the Garage Admin API with an admin token. So that credential is fetched from Vault. spiffe-garage already knows how to read it from the credentials directory.

jsonFormat = pkgs.formats.json { };
serviceBuckets =
  lib.pipe nodes [
    lib.attrValues
    (map (node: node.config.mjm.services))
    lib.mergeAttrsList
    (lib.filterAttrs (_: s: s.s3.enable))
    (lib.mapAttrs (_: s: s.s3.buckets))
  ]
  // {
    backups = [ "restic-backups" ];
  };
configFile = jsonFormat.generate "spiffe-garage.json" {
  identities = lib.mapAttrs' (
    name: buckets:
    lib.nameValuePair "spiffe://${trustDomain}/svc/${name}" {
      buckets = lib.genAttrs buckets (_: {
        read = true;
        write = true;
      });
    }
  ) serviceBuckets;
};

spiffe-garage creates access keys on-demand as needed. It uses a basic JSON configuration to decide which buckets a key for a given SPIFFE ID should have access to. That configuration is generated here based on the buckets that services declare with mjm.services. The backups service identity doesn't use mjm.services, so it is hardcoded here.

CapabilityBoundingSet = "";
DevicePolicy = "closed";
LockPersonality = true;
MemoryDenyWriteExecute = true;
PrivateDevices = true;
PrivateIPC = true;
PrivateUsers = "identity";
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectProc = "invisible";
RestrictAddressFamilies = [
  "AF_INET"
  "AF_INET6"
  "AF_UNIX"
];
RestrictNamespaces = true;
RestrictRealtime = true;
SystemCallArchitectures = "native";
SystemCallErrorNumber = "EPERM";
SystemCallFilter = [
  "@system-service"
  "~@resources @privileged"
];
UMask = "0077";

This is normal systemd hardening.

mjm.spire.entries.spiffe-garage = {
  spiffe_id = "svc/spiffe-garage";
  selectors = [
    {
      type = "systemd";
      value = "id:spiffe-garage.service";
    }
  ];
  dns_names = [ "spiffe-garage.service.consul" ];
};

A SPIRE registration entry is needed for the spiffe-garage systemd service for it to be able to use the SPIFFE workload API. Since it's serving HTTPS content, it should have a DNS name that matches the address clients would use to connect to it.

mjm.consul.services.spiffe-garage = {
  port = 3899;

  checks.up = {
    script.args = [
      (lib.getExe pkgs.curl)
      "--no-progress-meter"
      "--fail-with-body"
      "--cacert"
      "/run/certs/consul/bundle.pem"
      "--cert"
      "/run/certs/consul/cert.pem"
      "--key"
      "/run/certs/consul/key.pem"
      "--resolve"
      "spiffe-garage.service.consul:3899:127.0.0.1"
      "https://spiffe-garage.service.consul:3899/healthz"
    ];
    checkConfig = {
      failures_before_warning = 2;
      failures_before_critical = 6;
    };
  };
};

To expose spiffe-garage to clients, a Consul service is needed. The health check is unusual because spiffe-garage requires a client certificate in order to connect, and Consul doesn't natively support that for HTTP checks. So instead, a script to call curl is used. An alternative approach could be to expose a plain HTTP endpoint on another socket.