ZFS configuration

{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.mjm.zfs;
in
{
  imports = [
    ./encryption.nix
    ./zrepl.nix
  ];

  options.mjm.zfs = {
    <<options>>
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };
}

Several of my servers use ZFS, as it's just unmatched in terms of managing complex storage configuration. These services also used to run Proxmox and TrueNAS, which are both built on ZFS, so when I migrated them to NixOS, it was easier to keep them on ZFS. So, win-win.

Options

enable = lib.mkEnableOption "ZFS";

These ZFS customizations need to be enabled.

arcMax = lib.mkOption {
  type = lib.types.int;
  default = 8 * 1024;
};

By default, ZFS is allowed to use up to 50% of your total RAM for it's Adaptive Replacement Cache. I find this to be a little too much, as it creates memory pressure that can cause VMs to be killed. By default, I give my ZFS machines 8GiB for ARC, but I expose this option so machines with less RAM can tweak it down.

encryption.rootPool = lib.mkOption {
  type = with lib.types; nullOr str;
  default = null;
};

Encryption support is enabled by setting the name of the root pool. This pool should have a zvol at "$pool/keys/default" that is a LUKS volume that will be used to unlock datasets.

backups.roots = lib.mkOption {
  type = with lib.types; listOf str;
  default = [ ];
};

Prefixes for datasets that should be automatically backed up to a local hard drive.

Config

boot.zfs.forceImportRoot = false;

By default, on older state versions, NixOS will import the pool used for the root filesystem with force. This is potentially unsafe, and in general is not really needed, so I'm disabling that to both silence a warning and match the new default for NixOS.

boot.kernelParams = [ "zfs.zfs_arc_max=${toString (cfg.arcMax * 1024 * 1024)}" ];

This sets the max memory used for ARC to the configured number of megabytes.

services.zfs.autoScrub.enable = true;

Scrubbing automatically on a monthly basis is a good idea, particularly when you have mirrors so that any invalid data can hopefully be repaired automatically from the other mirror. While ZFS will also do this kind of repair when data is accessed, scrubbing proactively can discover and fix errors in infrequently accessed data, possibly preventing loss in case the device with valid data later fails.

systemd.services.disable-mglru = {
  description = "Disable Multi-Gen LRU";
  wantedBy = [ "basic.target" ];
  script = ''
    ${pkgs.coreutils-full}/bin/echo n > /sys/kernel/mm/lru_gen/enabled
  '';
  serviceConfig.Type = "oneshot";
  unitConfig.ConditionPathExists = "/sys/kernel/mm/lru_gen/enabled";
};

Modern Linux's multi-gen LRU cache can interact poorly with ZFS's ARC, so I disable it on boot on my ZFS machines with this service.

Encryption

{
  lib,
  config,
  utils,
  ...
}:
let
  cfg = config.mjm.zfs;
  <<zfs-functions>>
in
{
  options.mjm.zfs = {
    <<enc-options>>
  };

  config = lib.mkIf (cfg.enable && cfg.encryption.rootPool != null) {
    <<enc-config>>
  };
}

I have a special setup to use ZFS native encryption where the keyfile is protected by LUKS on zvols in the same pool. This lets me rely on some nice features in LUKS and systemd around unlocking with the TPM while otherwise getting to use some of the nice things about ZFS encryption. My thanks to @ElvishJerrico for sharing his own config for doing similar things, which helped influence what I ended up with here.

boot.initrd.luks.devices.cryptkey = {
  device = "/dev/zvol/${cfg.encryption.rootPool}/keys/default";
};

First, a LUKS device is setup for the volume that contains the keyfile for the ZFS datasets. The unlocked device will be present at /dev/mapper/cryptkey.

boot.initrd.supportedFilesystems.erofs = true;
boot.initrd.systemd.contents."/etc/fstab".text = ''
  /dev/mapper/cryptkey /cryptkey erofs defaults,x-systemd.after=systemd-cryptsetup@cryptkey.service 0 2
'';

An entry in the stage 1 fstab will allow the cryptkey volume to be mounted in stage 1 before the root filesystem is mounted. A mount unit can also accomplish the same task. The cryptkey volume will be use erofs as the filesystem, so it's compact and immutable.

boot.zfs.requestEncryptionCredentials = false;

By default, NixOS will try to unlock encrypted ZFS datasets when importing the pool they belong to. I don't want to do that, because the keyfile for the encrypted datasets actually lives on a zvol on the same pool. So I need to import the pool first, then unlock the LUKS volume, and then come back and unlock the datasets.

boot.initrd.systemd.services."zfs-import-${cfg.encryption.rootPool}" = {
  requiredBy = lib.mkForce [ "zfs-unlock.service" ];
  before = lib.mkForce [
    "cryptsetup-pre.target"
    "shutdown.target"
    "zfs-import.target"
  ];
  wants = [ "cryptsetup-pre.target" ];
};

The NixOS import service for the root pool needs some tweaks to its dependencies. Normally it will have requiredBy and before entries for any manually mounted filesystems in stage 1 that are part of the pool. Those need to go, because the datasets for those mounts will not be unlocked yet once this service is started. Instead, the import needs to happen before cryptsetup, and it needs to be pulled in by the zfs-unlock service: a new service which will be defined next.

boot.initrd.systemd.services.zfs-unlock =
  let
    mounts = getPoolMounts "/sysroot" cfg.encryption.rootPool;
  in
  {
    description = "Load Keys for ZFS";
    requiredBy = [ "zfs-import.target" ] ++ mounts;
    before = [ "zfs-import.target" ] ++ mounts;
    unitConfig = {
      DefaultDependencies = false;
      WantsMountsFor = "/cryptkey";
    };
    script = ''
      ${config.boot.zfs.package}/bin/zfs load-key -a
    '';
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
    };
  };

This new service loads the keys for any encrypted datasets from imported pools. This is the new service that needs to happen before any ZFS filesystems get mounted. It's set up to happen after /cryptkey so the keyfile will be available. And it will happen before the mounts that would normally happen after the import service. The logic for that uses functions that are copied from the NixOS ZFS module:

datasetToPool = x: lib.elemAt (lib.splitString "/" x) 0;

fsToPool = fs: datasetToPool fs.device;

getPoolFilesystems =
  pool: lib.filter (x: x.fsType == "zfs" && (fsToPool x) == pool) config.system.build.fileSystems;

getPoolMounts =
  prefix: pool:
  let
    poolFSes = getPoolFilesystems pool;

    # Remove the "/" suffix because even though most mountpoints
    # won't have it, the "/" mountpoint will, and we can't have the
    # trailing slash in "/sysroot/" in stage 1.
    mountPoint = fs: utils.escapeSystemdPath (prefix + (lib.removeSuffix "/" fs.mountPoint));

    hasUsr = lib.any (fs: fs.mountPoint == "/usr") poolFSes;
  in
  map (x: "${mountPoint x}.mount") poolFSes ++ lib.optional hasUsr "sysusr-usr.mount";

These functions are used to turn the filesystem entries for the system into a list of mount unit names.

boot.initrd.systemd.targets.initrd-switch-root = {
  conflicts = [
    "cryptkey.mount"
    "systemd-cryptsetup@cryptkey.service"
  ];
  after = [
    "cryptkey.mount"
    "systemd-cryptsetup@cryptkey.service"
  ];
};

Finally, these conflicts will cause the cryptkey volume to get locked again before leaving stage 1.

Local backups

{
  config,
  lib,
  ...
}:
let
  cfg = config.mjm.zfs;
in
{
  options.mjm.zfs = {
    <<backup-options>>
  };

  config = lib.mkIf (cfg.enable && cfg.backups.roots != [ ]) {
    <<backup-config>>
  };
}

All of my ZFS systems include a "slow" pool made up of one or more HDDs. One of the purposes of this is to back up the important data on these machines from the SSDs to the HDDs. I use zrepl for this, as it seems like the best of the ZFS backup tools to me.

services.zrepl.enable = true;
services.zrepl.settings.global.logging = [
  {
    type = "stdout";
    level = "info";
    format = "json";
  }
];

Of course, the zrepl service needs to be enabled to do backups.

services.zrepl.settings.jobs =
  let
    encrypted = cfg.encryption.rootPool != null;
  in
  [
    <<sink-job>>
    <<push-job>>
  ];

The backups require two jobs: a push job and a sink job. The push job is the more complex of the two, as it includes the logic for when to take snapshots and how many to keep on both sides. The sink job just receives the snapshots from the push job.

{
  name = "local_sink";
  type = "sink";
  serve = {
    type = "local";
    listener_name = "local_sink";
  };
  root_fs = "slow/backups";
  recv.placeholder.encryption = "off";
}

The sink job puts all of the backups that it receives under "slow/backups". The "serve" section configures it to listen locally under the name "local_sink", which the push job will use.

The encryption setting for placeholder datasets needs to be configured so zrepl knows how to configure any intermediate datasets it needs to create. Setting it to "off" should work for unencrypted datasets as well as raw encrypted sends, which is what I'm using.

{
  name = "backup_to_local";
  type = "push";
  connect = {
    type = "local";
    listener_name = "local_sink";
    client_identity = "local";
  };
  <<push-job-filesystems>>
  <<push-job-snapshotting>>
  <<push-job-pruning>>
}

The push job connects to the listener from the sink job defined above. The client identity "local" becomes a namespace for the backups. So all of my backups end up under "slow/backups/local". The rest of the job configuration is broken down in more detail below.

filesystems = lib.pipe cfg.backups.roots [
  (map (r: {
    "${r}<" = true;
    "${r}" = false;
  }))
  lib.mergeAttrsList
];
send = lib.mkIf encrypted { encrypted = true; };

The job will backup any datasets under the roots that have been configured for the machine. The particular configuration syntax I'm using has the effect that if "foo/bar" is in the roots, then "foo/bar" itself will not be backed up, but every dataset underneath "foo/bar/" will be. send.encrypted is set to true if ZFS encryption was enabled on this machine. This will cause zrepl to use raw sends, so the HDD will receive the same encrypted bytes that are on the SSD, and those datasets will be recoverable with the same keyfile as the originals.

snapshotting = {
  type = "periodic";
  prefix = "zrepl_";
  interval = "1h";
};

zrepl will take a snapshot for each dataset every hour, prefixing the name with "zrepl_" to distinguish them from any manually taken snapshots.

pruning.keep_sender = [
  { type = "not_replicated"; }
  {
    type = "last_n";
    count = 3;
  }
];
pruning.keep_receiver = [
  {
    type = "last_n";
    count = 1;
  }
  {
    type = "grid";
    regex = "^zrepl_.*";
    grid = "4x1h | 7x1d | 4x7d | 2x30d";
  }
];

The trickiest part of the config to get right is the pruning, which is configured separately for the sending side and the receiving side.

For the sender, in general I want to keep the last three snapshots. Even three is probably not necessary, since those same snapshots should end up on the HDD, but for recovering something from a recent snapshot quickly, it's nice to have and isn't very expensive. I also keep anything that isn't replicated to the receiver. This should cover both manual snapshots and any number of snapshots that might build up if there is an issue with replication that goes on for more than a few hours.

For the receiver, I always want to keep the most recent snapshot, whatever it is. Otherwise, I use a grid configuration to easily define a handful of buckets to keep fewer snapshots as they get older.

services.zrepl.settings.global.monitoring = [
  {
    type = "prometheus";
    listen = "[::1]:9811";
  }
];

environment.etc."alloy/zrepl.alloy".source = ./zrepl.alloy;

prometheus.scrape "zrepl" {
  targets = [{"__address__" = "[::1]:9811", instance = constants.hostname}]
  forward_to = [prometheus.remote_write.prod.receiver]
}

zrepl can expose Prometheus metrics, which I use to monitor whether there are any errors in replication. Otherwise, it's easy for those to go unnoticed. Instead of exposing the metrics endpoint externally, I'm using Alloy to scrape it on the local machine and then forward the metrics to the Prometheus instance.