S3-compatible storage for services

{
  lib,
  config,
  ...
}:
let
  cfg = config.mjm.services;

  <<serviceName>>
in
{
  options.mjm.services =
    let
      serviceType = {
        options = {
          <<options>>
        };
      };
    in
    lib.mkOption {
      type = with lib.types; attrsOf (submodule serviceType);
    };

  config = lib.mkIf (serviceName != null) {
    <<config>>
  };

  _class = "nixos";
}

Services that need to use S3-compatible storage should enable this module. By doing so, they will get an HTTP tunnel to Garage running on localhost:3902, and the environment will be automatically set up to provide them with credentials.

Service options

These options are available within the scope of a particular entry of mjm.services.

s3.enable = lib.mkEnableOption "S3-compatible storage";

S3 support must be enabled explicitly.

s3.buckets = lib.mkOption {
  type = with lib.types; listOf str;
};

A service that uses S3-compatbile storage must list the buckets that it will have access to. Bucket access is always read-write to keep things simple. I haven't yet had a use-case for read-only access to buckets.

Determining the service identity

serviceName =
  let
    serviceNames = lib.pipe cfg [
      lib.attrValues
      (lib.filter (s: s.s3.enable))
      (map (s: s.name))
    ];
    count = lib.length serviceNames;
  in
  if count > 1 then
    throw "cannot have more than one garage client in a single VM"
  else if count == 1 then
    lib.head serviceNames
  else if config.mjm.backups != { } then
    "backups"
  else
    null;

Due to the way S3 credentials are automatically provided, only one service identity can be used for S3 access on a given machine. This is fine, because services are split up into their own microVMs, but there is some validation needed for it. As long as a single service enables S3 on a machine, its identity will be used. If none are enabled, but there are backup jobs set up on the machine, then the "backups" service identity, which has access to the backups bucket, will be used. Otherwise, S3 support will not be set up at all on the machine.

It's possible for a service that uses S3 directly to also have backup jobs that need to use it as well. In this situation, the service's Garage key needs to be given read-write access to the backups bucket as well, since that's the key that will be used by those jobs. It's unfortunately not feasible to have the backup jobs use a different service identity.

Config

mjm.spire.tunnels.s3 = {
  id = lib.mkDefault serviceName;
  mode = "client";
  listen.port = 3902;
  target.service = "garage";
  target.tag = "s3";
};

Services access S3 through a local client tunnel on port 3902.

systemd.globalEnvironment.AWS_CONTAINER_CREDENTIALS_RELATIVE_URI = "/creds";

All services running on the machine get this AWS environment variable set which tells them where to get their S3 credentials from. This particular variable is normally set for workloads running in containers on Amazon ECS. When it is set, the value will be appended to "http://169.254.170.2" to form the full HTTP URI to use to fetch credentials. This method of providing credentials is chosen for being pretty easy to implement while having broad compatibility with various S3 SDKs.

systemd.network.networks."10-loopback" = {
  matchConfig.Type = "loopback";
  networkConfig.Address = "169.254.170.2/16";
  linkConfig.RequiredForOnline = false;
};

The 169.254.170.2 IP that AWS containers use needs to be added as a local loopback address, so this adds a networkd config file to do that.

mjm.spire.tunnels.s3-creds = {
  id = lib.mkDefault serviceName;
  mode = "client";
  listen.address = "169.254.170.2:80";
  target.service = "spiffe-garage";
};
systemd.sockets.s3-creds-tunnel.socketConfig.FreeBind = true;

With the IP address configured, this sets up a tunnel to listen on it and send the requests to the spiffe-garage service, which runs on every Garage node. This service will look at the SPIFFE identity used to connect to it and send back the credentials for the Garage key whose name matches the SPIFFE ID. Using a tunnel for this means that clients can use plain HTTP requests and not having to worry about passing the right TLS certificates, which is exactly what AWS clients will do automatically with the environment variable set above.

Because the tunnel listens on a specific IP, it's recommended to use the FreeBind option so that the socket can bind even before the IP address is set on any interface.

mjm.spire.entries =
  let
    hostname = config.networking.hostName;
  in
  lib.mkIf (serviceName == "backups") {
    tunnel-backups-s3-creds.enable = false;
    tunnel-backups-s3.enable = false;

    "tunnel-${hostname}-s3-creds" = {
      spiffe_id = "svc/backups";
      parent_id = hostname;
      selectors = [
        {
          type = "systemd";
          value = "id:s3-creds-tunnel.service";
        }
      ];
    };
    "tunnel-${hostname}-s3" = {
      spiffe_id = "svc/backups";
      parent_id = hostname;
      selectors = [
        {
          type = "systemd";
          value = "id:s3-tunnel.service";
        }
      ];
    };
  };

This is kind of a nasty hack, and there's probably a better way to deal with it.

The issue here is that the tunnels created above will have create SPIRE registration entries for their systemd services, and those entries will use the service identity as the parent: the entries will be valid on any machine that has that service enabled. If the entries for the backups service are created in this way, then they will create a conflict, since machines that use a more specific service that also have backups will get both entries.

The current solution is to disable the entries created for backups, and create new ones instead that are bound to the host identity instead of the service identity. This way, the entries that use the backups identity will only be valid on those specific hosts, and the other S3 hosts will only have an entry for their own specific service identity.