Service discovery with Consul

#| file: services/consul/default.nix
{ config, lib, ... }:
let
  cfg = config.mjm.consul;
in
{
  imports = [
    ./server.nix
    ./services.nix
  ];

  options.mjm.consul = {
    <<options>>
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };

  _class = "nixos";
}

My lab relies on Consul to monitor the health of services and let them discover each other automatically. I have yet to find a good alternative to it that doesn't rely on Kubernetes.

Options

enable = lib.mkEnableOption "consul agent";

Running a Consul agent on a machine needs to be enabled. In most cases, this is done automatically by my server module, so that an agent is running on every server on the LAN.

=> nixos/server module

server.enable = lib.mkEnableOption "consul server";

Most nodes run as simple agents, but three are designated as servers. The servers are the authority on the state of the datacenter and form a Raft cluster together. A meaningful chunk of configuration shown later will be specific to the server nodes.

ipv4Address = lib.mkOption {
  type = lib.types.str;
  default = ''{{ . | include "name" "${config.mjm.networkd.primaryIface}" | include "type" "ipv4" | attr "address" }}'';
};
ipv6Address = lib.mkOption {
  type = lib.types.str;
  default = ''{{ . | include "name" "${config.mjm.networkd.primaryIface}" | include "type" "ipv6" | include "flags" "global unicast" | sort "size" | attr "address" }}'';
};

Each agent in the cluster advertises both an IPv4 and IPv6 address. The address is determined automatically based on the addresses of the primary network interface for the machine. The template for the IPv6 address is a little more complicated, as there are some extra addresses on those that I don't want to use. This template should find the SLAAC address that is chosen based on the machine's MAC address, but only because I've disabled the privacy extensions in my networkd configuration.

certsPath = lib.mkOption {
  type = lib.types.path;
  default = "/run/certs/consul";
};

Consul agents will communicate with each other over mTLS using SPIFFE certificates. A helper will generate these certificates in /run/certs/consul. This option is mainly a convenience for accessing that path where needed.

socketPath = lib.mkOption {
  type = lib.types.path;
  default = "/run/consul/agent.sock";
};

Consul agents will listen over HTTP on a Unix socket whose path can be configured.

Config

This section of configuration applies to both Consul agents and servers, except where explicitly specified otherwise.

services.consul.enable = true;

Of course, the Consul service needs to be enabled.

services.consul.extraConfig.retry_join = lib.mkDefault [
  "bulbasaur.home.mattmoriarity.com"
  "charmander.home.mattmoriarity.com"
  "squirtle.home.mattmoriarity.com"
];

When joining a cluster for the first time, an agent needs to know how to find at least one of the other agents already in the cluster. I configure my agents to attempt to join through each server node. One should be running on each VM host in the lab.

services.consul.extraConfig = {
  advertise_addr_ipv6 = cfg.ipv6Address;
  advertise_addr_ipv4 = cfg.ipv4Address;
  advertise_addr = cfg.ipv4Address;
};

Consul agents listen on several different ports and addresses for various reasons, so it can be hard to keep them all straight. By default, Consul will listen for internal cluster traffic on all IPv4 and IPv6 addresses, but it will only advertise the IPv4 address for the node. I want my nodes to advertise both IPv4 and IPv6 addresses, to later make it easier to transition to IPv6-only. To do that, I configure the advertise addresses explicitly. For now, the primary one use is IPv4 because those addresses are stable while my IPv6 addresses can change if my ISP changes my prefix.

services.consul.extraConfig = {
  addresses = {
    dns = lib.mkDefault "127.0.0.1 ::1";
    http = lib.concatStringsSep " " [
      "127.0.0.1"
      "::1"
      "unix://${cfg.socketPath}"
    ];
  };
  unix_sockets.mode = "666";
  ports.https = lib.mkDefault (-1);
};
systemd.services.consul.serviceConfig.RuntimeDirectory = "consul";

For non-internal traffic, Consul will instead default to listening on 127.0.0.1 for everything, but this can be overridden by setting addresses. I add ::1 for both DNS and HTTP, so that Consul is also listening on IPv6 for those. And for HTTP, there's an additional Unix socket listener, which I prefer to use if possible. Because that socket goes in /run/consul, I configure the systemd service to create that runtime directory.

For non-server agents, there's not really a need to listen for HTTPS traffic at all, so I disable that port entirely.

networking.firewall.allowedTCPPorts = [ 8301 ];
networking.firewall.allowedUDPPorts = [ 8301 ];

Port 8301 is used for gossip communication between Consul nodes. It should be open to external traffic for both TCP and UDP.

mjm.services.consul-client = lib.mkIf (!cfg.server.enable) { };
mjm.spire.certs.consul = {
  id = "consul-client";
  user = "consul";
  systemd.unit = "consul.service";
};

services.consul.extraConfig.auto_reload_config = true;
services.consul.extraConfig.tls.defaults = {
  ca_file = "${cfg.certsPath}/bundle.pem";
  cert_file = "${cfg.certsPath}/cert.pem";
  key_file = "${cfg.certsPath}/key.pem";
  tls_min_version = "TLSv1_3";
  verify_server_hostname = true;
  verify_outgoing = true;
  verify_incoming = true;
};
services.consul.extraConfig.telemetry.certificate.enabled = false;

This configures the certificates to use for mTLS communication between Consul servers and agents. Agents will use a SPIFFE ID for the "consul-client" service, but Consul really only cares if the certificate is valid for the configured CA, which will be the one for my SPIFFE trust domain. Servers will get a more specific identity that is configured later.

The helper service to grab the certificates and write them to disk does not need to do any explicit signalling to Consul to reload, because Consul can be configured to reload those files automatically when they change.

I configure TLS in Consul to use these certs and verify everything, and also use TLS 1.3 since I have enough control over the stack to be sure it is available. Consul also includes telemetry that will emit warnings and errors when certificates are close to expiring. I disable this because it operates on the order of days, and that's much longer than my SPIFFE certs are valid for, so it's basically always firing. I can trust that the certificates will keep renewing: a lot of things will be noticeably wrong in the lab if that is broken.

services.consul.extraConfig.enable_local_script_checks = true;

I rely on some script checks for services, either for bespoke needs or for using curl to check services that only listen on a Unix socket.

services.consul.extraConfig.limits.http_max_conns_per_client = 500;

I don't remember why I increased this limit. The default is 200, but I guess I needed more at some point.

systemd.services.consul.startLimitIntervalSec = 0;
systemd.services.consul.serviceConfig.RestartSec = "5s";

It's very important that Consul starts on any machine where it's configured, so it shouldn't give up retrying.

services.resolved.dnsDelegates.consul.Delegate = {
  DNS = "[::1]:8600";
  Domains = "~consul";
};

Machines running a Consul agent will configure their local resolved to send requests to the .consul domain to their local agent. While my router is also configured to send DNS requests for this domain to the three Consul servers, handling this with the local agent where possible reduces the load on those servers, helping Consul scale better. The router configuration is still useful for workstations that don't have a local agent in the first place.

Server config

#| file: services/consul/server.nix
{ config, lib, ... }:
let
  cfg = config.mjm.consul;
in
{
  config = lib.mkIf (cfg.enable && cfg.server.enable) {
    <<server-config>>
  };

  _class = "nixos";
}

This configuration section applies only to Consul servers, not to most other nodes running as agents.

mjm.services.consul = {
  http.ingress = {
    subdomain = "consul";
    authMode = "proxy";
    proxy.rules = [
      {
        resources = [ "^/v1/health/node/.+$" ];
        methods = [ "GET" ];
        policy = "bypass";
      }
    ];
  };
};
mjm.ingress.vhosts.consul.upstream.service.port = 8501;
services.consul.webUi = true;

The Consul servers will register themselves under the "consul" service name, so I don't need to register them manually. I do want to expose the Consul web UI though, which is available on https://consul.midna.dev. The port needs to be adjusted though, since Consul will advertise its server RPC port, not its HTTPS port. Since I'm not using ACLs with Consul, there's no protection in the UI itself, so I put it behind the authentication proxy. I have a rule to let through API requests to check the health of nodes, because my deploy tooling uses that to check if a deploy can advance to the next machine. In CI, it can do that through the local Consul agent, but when running on a workstation, that's not available, so it goes through the ingress for that.

services.consul.extraConfig = {
  server = true;
  bootstrap_expect = 3;
};

Server nodes must be explicitly configured in the node's config file. When a cluster is first being set up, some coordination is expected to ensure there is only one Raft leader. The bootstrap_expect config sets how many server nodes there are supposed to be in the cluster. With this value set, if there is no existing cluster, the server nodes will wait until all three are up, and then they will elect a leader and it will bootstrap the cluster. This isn't particularly relevant now, as my cluster has been running for a long time now, but it's also fine to have it in there.

services.consul.extraConfig.telemetry = {
  prometheus_retention_time = "1h";
  disable_hostname = true;
};
environment.etc."alloy/consul-server.alloy".source = ./consul.alloy;

//| file: services/consul/consul.alloy
prometheus.scrape "consul_server" {
  targets = [{"__address__" = "[::1]:8500", instance = constants.hostname}]
  metrics_path = "/v1/agent/metrics"
  params = {format = ["prometheus"]}
  forward_to = [prometheus.remote_write.prod.receiver]
}

The Consul servers use Alloy to scrape their metrics locally and send them to Prometheus.

services.consul.extraConfig.addresses.dns = "::";
services.consul.extraConfig.addresses.https = "::";
services.consul.extraConfig.ports.https = 8501;

Server nodes are expected to be accessed externally in ways that normal agents are not. The DNS server is public, though it is mostly accessed via the normal port 53 (see how that works later). The servers also listen for HTTPS requests on port 8501.

networking.firewall.allowedTCPPorts = [
  8300
  8302
  8501
  8600
];
networking.firewall.allowedUDPPorts = [
  8302
  8600
];

Some additional openings in the firewall are also required. Port 8300 is used for server RPC traffic. 8302 is for WAN gossip, which I don't think I use but is still something the servers are listening for. 8501 is HTTPS traffic, and 8600 is for DNS.

networking.nftables.tables.consul-dns = {
  family = "inet";
  content = ''
    chain prerouting {
      type nat hook prerouting priority dstnat;
      udp dport 53 redirect to 8600
      tcp dport 53 redirect to 8600
    }
  '';
};

This firewall config takes incoming traffic to port 53 and redirects it to the Consul DNS server on port 8600. My UniFi router serves as my DNS server on the LAN, and it can be configured to forward DNS requests for a particular domain to specific servers, but it doesn't offer a way to configure the port, so it will send those to port 53. So this firewall rule lets the Consul servers still handle that.

This is only needed for machines that aren't running their own Consul agent. The ones that are can (and do) configure resolved to send requests for the "consul" domain to the local agent.

mjm.spire.entries."consul-server-${config.networking.hostName}" = {
  spiffe_id = "svc/consul";
  parent_id = config.networking.hostName;
  selectors = [
    {
      type = "systemd";
      value = "id:spiffe-certs@consul.service";
    }
  ];
  dns_names = [
    "server.dc1.consul"
    "${config.networking.hostName}.server.dc1.consul"
    "consul.service.consul"
  ];
};

This entry sets up the certificate that the server nodes will use. These use a "consul" service identity rather than "consul-client". Each machine gets its own entry, and the DNS names for the certificate include both "server.dc1.consul" and "$hostname.server.dc1.consul", which are what Consul expects server certificates to have. The "consul.service.consul" name is used for ingress.

Registering services with the agent

Service modules can declare Consul services that will be automatically registered with the local agent.

=> consul/services module