Forgejo Actions runner

{
  config,
  lib,
  pkgs,
  nodes,
  ...
}:
let
  cfg = config.mjm.forgejo.runner;
in
{
  options.mjm.forgejo.runner = {
    enable = lib.mkEnableOption "Forgejo Actions runner";
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };

  _class = "nixos";
}

I rely on a CI pipeline to deploy changes to my homelab. That pipeline is currently powered by Forgejo Actions. I have a machine dedicated to running actions for my own Forgejo instance, and this module sets up that runner.

mjm.services.forgejo-runner = {
  secrets.enable = true;
  secrets.templates = {
    <<secret-templates>>
  };
  secrets.user = "gitea-runner";
};

The runner will need a few secrets from Vault to set itself up correctly. Because of the way those secrets need to be used, they'll be set up via my secret templates mechanism. The specific templates will be covered later.

virtualisation.podman.enable = true;

I run my CI jobs in containers using Podman, so that needs to be enabled on the machine.

virtualisation.containers.containersConf.settings = {
  containers.pids_limit = -1;
};

At some point, I was having an issue with my CI jobs hitting this limit on PIDs, so I've disabled the limit.

networking.firewall.trustedInterfaces = [ "podman*" ];

Trusting the podman networking interfaces in the firewall configuration is needed for two reasons. First, I use the cache action, which relies on a caching service running on the host. Without trusting the interface, the container cannot communicate with the caching service. Second, I've found that DNS resolution does not work in the container without this.

services.gitea-actions-runner = {
  package = pkgs.forgejo-runner;
  instances.main = {
    enable = true;
    name = config.networking.hostName;
    url = "https://git.midna.dev";
    tokenFile = "/run/forgejo-runner-secrets/token";
    labels = [ "nix:docker://node:22-trixie" ];
    settings = {
      log.level = "debug";
      <<runner-settings>>
    };
  };
};

There's no specific NixOS module for the Forgejo runner. Since it's a fork of the Gitea Actions runner, we just use the module for that but replace the package.

I only need one runner instance, since all of my CI jobs run the same way. The "labels" option is what determines both how a job requests running here (by using "runs-on: nix" in the job spec) and how the job is run (in a container using the node:22-trixie image). Note that even though the label says "docker", the NixOS module will see that Podman is enabled on the host instead and will point the runner at Podman's socket instead of Docker's.

The settings for the runner are worth exploring in a little more depth.

runner.capacity = 5;

This runner can do five jobs at a time. The limit is pretty arbitrary, as I either have jobs that either need to be the only one like them running or generate basically no meaningful load. The limit mostly just keeps things from blowing up unexpectedly.

runner.timeout = "4h";
runner.shutdown_timeout = "5m";

Adjust some timeouts for jobs. The default 3 hours timeout can be a bit too short for some of my builds if large compilations are involved.

container.enable_ipv6 = true;

By default, the runner with create a new Podman network for every job it runs, and that network will only be set up for IPv4. This setting has it also set up IPv6, which I use throughout my network and would like to rely on even more.

container.options = lib.concatStringsSep " " [
  "--volume /nix/store:/nix/store:ro"
  "--volume /nix/var/nix/db:/nix/var/nix/db:ro"
  "--volume /nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
  "--volume /etc/ssl/certs:/etc/ssl/certs:ro"
  "--volume /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro"
  "--volume /run/spire-agent:/run/spire-agent:ro"
  "--volume /run/consul:/run/consul:ro"
];
container.valid_volumes = [
  "/nix/store"
  "/nix/var/nix/db"
  "/nix/var/nix/daemon-socket"
  "/etc/ssl/certs"
  "/etc/ssh/ssh_known_hosts"
  "/run/spire-agent"
  "/run/consul"
  "forks"
];

This configuration sets up volumes and mounts needed to make my jobs work. To make Nix work inside the container, a few things are mounted from the host to allow the container to communicate with the host's Nix daemon. Unix sockets for the SPIRE agent and Consul agent running on the host are also mounted into the container.

All of the volumes requested in the options must be included in valid_volumes as well. There's one additional entry in the valid volumes that isn't mounted automatically for all jobs: "forks". Since it's not an absolute paths, it's the name of a Podman volume instead. This volume stores checkouts of Git repos of dependencies that I maintain forks of. The actions I have for keeping those forks up-to-date rely on this volume to avoid recloning large repos with every run.

runner.envs = {
  LANG = "en_US.UTF-8";
  LC_ALL = "en_US.UTF-8";
  LOCALE_ARCHIVE = "/nix/var/nix/profiles/default/lib/locale/locale-archive";

  CONSUL_HTTP_ADDR = "unix:///run/consul/agent.sock";
  VAULT_ADDR = "https://vault.service.consul:8200";
  SPIFFE_ENDPOINT_SOCKET = "unix:///run/spire-agent/api.sock";

  NIX_REMOTE = "daemon";
  NIX = "${config.nix.package}";
  RUNNER_PROFILE = "${pkgs.buildEnv {
    name = "forgejo-runner-path";
    paths = [
      config.nix.package
      pkgs.cacert
      pkgs.git
      pkgs.glibcLocalesUtf8
      pkgs.openssh
    ];
  }}";
};

Finally, a handful of environment variables are set for the runner. First, locale-related variables are set to use UTF-8. I'm not sure these actually are working correctly, since bash still complains about it, but it hasn't caused issues yet so I haven't dug into it.

Next, variables for communicating with Consul, Vault, and SPIRE are set so that jobs can use these services easily. Consul and SPIRE are accessed through the volumes mounted above.

Last, some variables are set related to using Nix from inside the container. The NIX_REMOTE variable is read by Nix and forces communication with the Nix daemon even when it would be unnecessary, so this ensures the container is always doing Nix builds through the host. The other two variables are used by a custom action that my workflows use. In particular, the RUNNER_PROFILE variable will get linked to /nix/var/nix/profiles/default and will be part of the PATH in the jobs, so this is how I can include tools needed by most jobs.

=> setup-nix action

token = {
  text = ''
    TOKEN=''${secret_forgejo_runner_token}
  '';
  secrets = [ "token" ];
};

systemd.services.gitea-runner-main.requires = [ "forgejo-runner-secrets.service" ];
systemd.services.gitea-runner-main.after = [ "forgejo-runner-secrets.service" ];

This secret template will generate the /run/forgejo-runner-secrets/token file that was referenced above when setting up the runner instance. The tokenFile option is supposed to be an environment file that defines the TOKEN environment variable, so this template creates that structure. Of course, the runner needs to run after the token file has been created, so a dependency is added for that.

systemd.services.podman.restartIfChanged = false;
systemd.services.gitea-runner-main.restartIfChanged = false;
mjm.deploy.rebootAutomatically = false;

Normally, updates to systemd services cause a restart of those services, but that will cause problems for Podman and the Forgejo Runner. If they are restarted during a job, it will interrupt the job. It's better if they just keep running with an out-of-date version.

Similarly, my deploy tooling will automatically reboot machines if certain components are updated, but doing that on the CI runner would interrupt the job, so in that case, the new generation is set up to be applied on reboot, but the reboot itself must be done manually.

nix.distributedBuilds = true;
nix.buildMachines =
  let
    mkVmTestBuilder = name: {
      hostName = "${name}.home.mattmoriarity.com";
      system = "x86_64-linux";
      protocol = "ssh-ng";
      maxJobs = 2;
      speedFactor = 1;
      supportedFeatures = [
        "kvm"
        "nixos-test"
      ];
      mandatoryFeatures = [ "nixos-test" ];
    };
  in
  [
    {
      hostName = nodes.niobe.config.mjm.deploy.targetHost;
      system = nodes.niobe.config.nixpkgs.hostPlatform.system;
      protocol = "ssh-ng";
      maxJobs = 6;
      speedFactor = 2;
      supportedFeatures = [
        "nixos-test"
        "benchmark"
        "big-parallel"
      ];
      mandatoryFeatures = [ ];
      publicHostKey = "c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUxEbjh6cDFkL3M1T29HZ0FxdEgxNEtFVHZCRU1IOXBERWY2YzJ5amNkWXMgcm9vdEBuaW9iZQo=";
    }
  ]
  ++ (map mkVmTestBuilder [
    "apollo"
    "artemis"
    "demeter"
  ]);

My Nix builds in CI rely on remote builders for some things. My VPS niobe is an aarch64-linux machine which is used to run Nix builds for both itself and athena, my Apple Silicon machine.

Otherwise, the runner machine runs x86_64 Nix builds itself for the most part. The exception is VM tests, which also get farmed out to the other VM hosts. Honestly, the benefit of this right now is kind of dubious: I don't run that many VM tests, as I'm quick to disable them if they're fussy, and I haven't put the effort into figuring out how to write useful ones for my own modules.

Even though builds run in Podman containers, because they use Nix through the daemon on the host, this configuration must be done on the host rather than within the containers.

programs.ssh.extraConfig = lib.mkAfter ''
  Host ${lib.concatMapStringsSep " " (m: m.hostName) config.nix.buildMachines}
    User mjm
    IdentitiesOnly yes
    IdentityFile /run/forgejo-runner-secrets/key
'';

Remote builds operate over SSH, so a small amount of SSH client configuration is needed to make sure the Nix builds use the correct credentials.

key = {
  text = "\${secret_forgejo_runner_remote_builder_private_key}\n";
  secrets = [ "remote_builder_private_key" ];
};

This very simple secret template writes the file for the SSH key used for the remote builds. A template is used rather than a systemd credential so that the file is there regardless of the lifecycle of a particular service, which makes it easier to test if the connection for remote builds is working or not.

mjm.spire.entries."forgejo-build-${config.networking.hostName}" = {
  spiffe_id = "ci/repo/nix-config";
  parent_id = config.networking.hostName;
  selectors = [
    {
      type = "docker";
      value = "image_id:docker.io/library/node:22-trixie";
    }
  ];
};

This entry creates a SPIFFE identity for the CI jobs. When I used GitLab CI, these were better scoped: the GitLab runner would put useful labels on the containers it created so that you could grant an identity only to jobs for a specific repo. The Forgejo Runner doesn't do this (maybe I should make a PR), so I'm just matching on the image that I've configured the runner to use. That means any job that "runs-on: nix" will be able to use this identity.

The /ci/repo/nix-config identity has pretty broad access in Vault, since it needs to be able to deploy to machines as well as update policies and other infrastructure. I would like to be able to lock this down more.

nix.gc.dates = "Mon *-*-* 10:00:00";

Normally, the Nix garbage collection job on my servers runs daily. That's a bit aggressive though for a machine whose job is to build Nix things, as it causes unnecessary rebuilds or downloads from the cache, particularly because the things being built don't introduce GC roots.

Garbage collection is still important, as the machine doesn't have unlimited space, but it's better off running weekly instead.