Nix binary cache with attic

{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.mjm.atticd;
in
{
  options.mjm.atticd = {
    enable = lib.mkEnableOption "atticd";
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };

  _class = "nixos";
}

atticd provides a binary cache for Nix packages that I end up needing to build myself.

Service settings

mjm.services.atticd = {
  <<service-settings>>
};

atticd needs more service infrastructure than most, so these settings are broken down into smaller pieces.

postgresql.enable = true;

atticd stores metadata about what it has cached in a PostgreSQL database.

s3.enable = true;
s3.buckets = [ "attic-caches" ];

atticd uses S3-compatible storage to store the actual chunks of cached artifacts.

http = {
  port = 18100;
  health.path = "/";
  health.blockDeploy = true;

  ingress = {
    subdomain = "attic";
    authMode = "none";
  };
};

atticd will listen locally for HTTP connections on port 18100. It doesn't have a dedicated health check endpoint, so the root page is used for that purpose. The attic cache is important enough to using the lab that the deploy blocks until it is healthy. This is maybe unnecessary, as attic going down doesn't affect the ability to deploy other machines and deploying other machines is unlikely to make the problem worse.

atticd is generally accessed through the ingress via https://attic.midna.dev. It handles its own authentication via JWT tokens.

secrets = {
  enable = true;
  templates.env = {
    text = ''
      ATTIC_SERVER_TOKEN_RS256_SECRET_BASE64=''${secret_atticd_token_rs256_secret}
    '';
    secrets = [ "token_rs256_secret" ];
  };
};

atticd only really needs to store one secret: the private key used to sign its JWT tokens. Unfortunately, it only supports providing that secret via an environment variable, so I need to resort to templating to create an environment file for the systemd service to use.

Config

services.atticd.enable = true;
services.atticd.settings.listen = "[::1]:18100";
services.atticd.settings.database.url = "postgresql:atticd?host=/run/postgresql&user=atticd";

atticd is enabled and configured for HTTP and database access as described above.

services.atticd.settings.storage = {
  type = "s3";
  region = "home";
  bucket = "attic-caches";
  # when there's only a single chunk, then attic will redirect to a pre-signed URL for
  # that chunk, so the endpoint needs to be publicly reachable
  endpoint = "https://garage.midna.dev";
};

atticd uses S3-compatible storage from my Garage cluster to store the actual chunks of artifacts. My only region is "home". The attic-caches bucket is used, which is the only bucket that the atticd service should be able to read and write to.

Generally I would prefer to not go through the ingress when accessing Garage: most services that use it configure it as an upstream and use a local tunnel. However, when a cached artifact is made up of only a single chunk, instead of streaming it, atticd will redirect to a pre-signed URL for the artifact in Garage. For this to work, the endpoint configured must be a URL that atticd's clients can use to access Garage.

services.atticd.settings.compression.type = "zstd";
services.atticd.settings.garbage-collection.default-retention-period = "3 months";

atticd's chunks are zstd compressed and retained for three months.

services.atticd.settings.chunking = {
  nar-size-threshold = 524288;
  min-size = 65536;
  avg-size = 524288;
  max-size = 2097152;
};

I've increased the default sizes for chunking the artifacts atticd stores. I didn't write down why I did it almost three years ago.

services.atticd.environmentFile = "/run/atticd-secrets/env";
systemd.services.atticd.wants = [ "atticd-secrets.service" ];
systemd.services.atticd.after = [ "atticd-secrets.service" ];

The private key for the JWT tokens will be passed as an environment variable via the /run/atticd-secrets/env file. This file is generated by atticd-secrets service based on the template that was configured above. That service must run before atticd itself so that the file exists before systemd tries to read it.

systemd.services.atticd.environment.RUST_LOG = "info";

I want a little more logging out of atticd.

mjm.deploy.tests = {
  inherit (pkgs.nixosTests) atticd;
};

My CI will run the NixOS test for atticd from Nixpkgs to ensure its basic functionality is working.