Monitoring metrics with Prometheus

{
  pkgs,
  config,
  lib,
  ...
}:
let
  cfg = config.mjm.prometheus;
in
{
  options.mjm.prometheus = {
    <<options>>
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };
}

To keep an eye on metrics for my lab and to power dashboards in Grafana, I run Prometheus.

Options

enable = lib.mkEnableOption "prometheus";

Prometheus needs to be enabled in order to run.

dnsServers = lib.mkOption {
  type = with lib.types; listOf str;
  default = [ "10.0.0.5" ];
};

This option lists the local DNS servers within the lab so they can be referenced easily later.

Service settings

mjm.services.prometheus = {
  http = {
    inherit (config.services.prometheus) port;
    health.path = "/-/ready";

    metrics.enable = true;

    ingress = {
      subdomain = "metrics";
      authMode = "proxy";
    };
  };
};

Prometheus is exposed externally at https://metrics.midna.dev. It has a health check endpoint that Consul monitors. It has its own metrics endpoint that it will scrape. Prometheus doesn't have a useful authentication mechanism of its own, so it's proxied behind Authelia to protect it.

Config

services.prometheus.enable = true;
services.prometheus.listenAddress = "[::1]";
services.prometheus.webExternalUrl = "https://metrics.midna.dev";

Prometheus listens locally, since it exposed with mutual TLs using a tunnel.

services.prometheus.extraFlags = [ "--web.enable-remote-write-receiver" ];
mjm.spire.tunnels.alloy-prometheus.enable = false;

Many of my metrics are now pushed to Prometheus via Alloy rather than being scraped by Prometheus itself. For this to be possible, the remote write receiver needs to be enabled.

At the same time, the tunnel Alloy uses to push these metrics on most machines should be disabled on this one, since Prometheus is already listening locally and can receive the metrics directly. The tunnel uses the same port, so the configuration in Alloy doesn't need to change for this.

mjm.spire.certs.prometheus = {
  systemd.unit = "prometheus.service";
  systemd.action = "reload-or-restart";
  user = "prometheus";
};

services.prometheus.checkConfig = "syntax-only";

The majority of my Consul services expose their metrics via mutual TLS. Prometheus needs to perform its side of the TLS, so it needs certificates. Usually this is done with tunnels, but since the targets that it needs to connect to are determined dynamically, that's not an option. Instead, certificates will be written to disk and Prometheus will be configured later to read from those files. When the certificates rotate, Prometheus will be reloaded.

Because the configuration will include files that do not exist except at runtime, a full check of the configuration at build-time is not possible, so a syntax-only check is done instead.

mjm.deploy.tests = {
  inherit (pkgs.nixosTests.prometheus) config-reload;
};

My deploy tooling will run a basic Prometheus VM test from Nixpkgs in CI.

Scraping metrics from Consul services

services.prometheus.scrapeConfigs = [
  {
    job_name = "consul-services";
    consul_sd_configs = [ { server = "localhost:8500"; } ];
    tls_config = {
      ca_file = "/run/certs/prometheus/bundle.pem";
      cert_file = "/run/certs/prometheus/cert.pem";
      key_file = "/run/certs/prometheus/key.pem";
      insecure_skip_verify = true;
    };
    relabel_configs = [
      <<relabel-configs>>
    ];
  }
];

The only scraping job in my Prometheus config is for services registered dynamically in Consul. Anything else will be scraped locally by Alloy and pushed to Prometheus.

The job is configured to use the certificates issued for Prometheus above. That's necessary for the majority of services which are exposing their metrics via mTLS. Unfortunately, I do need to disable TLS verification on the client side, because Prometheus doesn't have a way to use relabeling to configure the expected server name.

The relabeling for this is a bit involved, so I've broken it down into the individual steps.

{
  source_labels = [ "__meta_consul_service_metadata_metrics_path" ];
  action = "keep";
  regex = "(.+)";
}
{
  source_labels = [ "__meta_consul_service_metadata_metrics_path" ];
  target_label = "__metrics_path__";
  regex = "(.+)";
}

First, only services with a "metrics_path" metadata key are kept. This is the signal that the service has metrics to scrape. That value is then relabeled to "metrics_path" so that Prometheus will use that path when scraping.

{
  source_labels = [ "__meta_consul_service" ];
  target_label = "service_name";
  regex = "(.+)";
}
{
  source_labels = [ "__meta_consul_node" ];
  target_label = "node_name";
  regex = "(.+)";
}
{
  source_labels = [ "__meta_consul_service_id" ];
  target_label = "instance";
  regex = "(.+)";
}

A few pieces of information from Consul are copied into permanent labels.

{
  source_labels = [
    "__address__"
    "__meta_consul_service_metadata_metrics_port"
  ];
  regex = "([^:]+)(?::\\d+)?;(\\d+)";
  replacement = "$1:$2";
  target_label = "__address__";
}
{
  source_labels = [
    "__address__"
    "__meta_consul_service_metadata_metrics_port"
  ];
  regex = "(\\[[^\\]]+\\])(?::\\d+)?;(\\d+)";
  replacement = "$1:$2";
  target_label = "__address__";
}

Some services expose their metrics over a different port than the main service traffic. For those, I support a "metrics_port" metadata key. If it's present, I need to rewrite the "address" label to use that. This is somewhat complicated by the need to support IPv6 addresses, so there's two rules to tackle that correctly.

{
  source_labels = [ "__meta_consul_service_metadata_metrics_scheme" ];
  target_label = "__scheme__";
  regex = "(.+)";
}

Metrics are assumed to be accessible over plain HTTP by default, but a service can override the scheme using the "metrics_scheme" metadata key.

Monitoring web sites and DNS

I'm using the blackbox exporter for Prometheus to monitor my DNS in the lab as well as the end-to-end health of several of the sites I host.

environment.etc."alloy/blackbox.alloy".source = ./blackbox.alloy;
environment.etc."alloy/blackbox-config.alloy".text =
  let
    jsonFormat = pkgs.formats.json { };
    yamlFormat = pkgs.formats.yaml { };
  in
  ''
    <<blackbox-config>>
  '';

The blackbox exporter is bundled as part of Alloy, so I configure it through that. The configuration is split into two parts: the part that is static and the part that needs substitutions from Nix.

discovery.file "local_dns" {
  files = ["${
    jsonFormat.generate "dns-servers.json" [
      { targets = cfg.dnsServers; }
    ]
  }"]
}

discovery.file "public_dns" {
  files = ["${
    jsonFormat.generate "public-dns-servers.json" [
      {
        targets = [
          "8.8.4.4"
          "8.8.8.8"
          "1.0.0.1"
          "1.1.1.1"
        ];
      }
    ]
  }"]
}

For checking DNS, I have two sets of targets: one for my local DNS server and one for publicly-hosted DNS servers. Checking both means that I can tell whether a problem is specific to my local DNS or if it's more universal.

These discovery.file components produce lists of targets that can then be relabeled and passed to the exporter.

dns_public = {
  prober = "dns";
  timeout = "5s";
  dns = {
    query_name = "www.google.com";
    query_type = "A";
    valid_rcodes = [ "NOERROR" ];
    validate_answer_rrs.fail_if_not_matches_regexp = [ "www\\.google\\.com\\.\t.*\tIN\tA\t.*" ];
  };
};

The blackbox exporter needs to be configured with modules that determine the checks that can be done against a target address. The first module I define is "dns_public", which simply checks if www.google.com resolves through DNS.

discovery.relabel "blackbox_dns_public" {
  targets = array.concat(
    discovery.file.local_dns.targets,
    discovery.file.public_dns.targets,
  )

  rule {
    target_label = "probe_type"
    replacement = "dns"
  }
  rule {
    target_label = "probe_scope"
    replacement = "public"
  }
  rule {
    target_label = "name"
    replacement = "blackbox-dns-public"
  }
  rule {
    target_label = "module"
    replacement = "dns_public"
  }
}

The dns_public module is used against both local and public DNS servers, and ideally it should succeed for both. If it's failing against local but not public, that's a signal that something is wrong with my local DNS. If it's failing against both, that might just mean my internet is down.

The "probe_type" and "probe_scope" labels are my own: my alert rules use them.

dns_private = {
  prober = "dns";
  timeout = "5s";
  dns = {
    query_name = "demeter.home.mattmoriarity.com";
    query_type = "A";
    valid_rcodes = [ "NOERROR" ];
    validate_answer_rrs.fail_if_not_matches_regexp = [
      "demeter\\.home\\.mattmoriarity\\.com\\.\t.*\tIN\tA\t10\\.0\\.2\\.13"
    ];
  };
};

The next module is "dns_private" which checks DNS resolution for one of my local VM servers.

discovery.relabel "blackbox_dns_private" {
  targets = discovery.file.local_dns.targets

  rule {
    target_label = "probe_type"
    replacement = "dns"
  }
  rule {
    target_label = "probe_scope"
    replacement = "private"
  }
  rule {
    target_label = "name"
    replacement = "blackbox-dns-private"
  }
  rule {
    target_label = "module"
    replacement = "dns_private"
  }
}

This module of course is only used against local DNS, since the record it's checking for isn't expected to exist publicly.

dns_ad_blocking = {
  prober = "dns";
  timeout = "5s";
  dns = {
    query_name = "googleadservices.com";
    query_type = "A";
    valid_rcodes = [ "NOERROR" ];
    validate_answer_rrs.fail_if_not_matches_regexp = [
      "googleadservices\\.com\\.\t.*\tIN\tA\t0\\.0\\.0\\.0"
    ];
  };
};

I also have a module to check if my local DNS-based ad-blocking is working. It checks googleadservices.com which should resolve to 0.0.0.0 if ad-blocking is working.

discovery.relabel "blackbox_dns_ad_blocking" {
  targets = discovery.file.local_dns.targets

  rule {
    target_label = "probe_type"
    replacement = "dns"
  }
  rule {
    target_label = "probe_scope"
    replacement = "ad-blocking"
  }
  rule {
    target_label = "name"
    replacement = "blackbox-dns-ad-blocking"
  }
  rule {
    target_label = "module"
    replacement = "dns_ad_blocking"
  }
}

This is another module that only makes sense against local DNS.

https_homelab = {
  prober = "http";
  http = {
    method = "GET";
    headers.Accept = "text/html";
    preferred_ip_protocol = "ip6";
  };
};

The last module is not for DNS. Instead, it expects a successful HTTP response from accessing the target.

discovery.relabel "homelab_https" {
  targets = [
    { "__address__" = "attic.midna.dev" },
    { "__address__" = "atuin.midna.dev" },
    { "__address__" = "auth.midna.dev" },
    { "__address__" = "feeds.midna.dev" },
    { "__address__" = "graphs.midna.dev" },
    { "__address__" = "home.midna.dev" },
    { "__address__" = "pass.midna.dev" },
    { "__address__" = "vault.midna.dev" },
  ]

  rule {
    target_label = "name"
    replacement = "homelab-https"
  }
  rule {
    target_label = "module"
    replacement = "https_homelab"
  }
}

The list of targets here is hard-coded. It's specifically sites that aren't protected by Authelia as a proxy, since those will redirect to a login page when accessed. In the future, I might make this list dynamic based on the vhost configurations, so it includes all subdomains of midna.dev that make sense.

prometheus.exporter.blackbox "default" {
  config_file = "${
    yamlFormat.generate "blackbox-config.yml" {
      modules = {
        <<blackbox-modules>>
      };
    }
  }"
  targets = array.concat(
    discovery.relabel.blackbox_dns_public.output,
    discovery.relabel.blackbox_dns_private.output,
    discovery.relabel.blackbox_dns_ad_blocking.output,
    discovery.relabel.homelab_https.output,
  )
}

With all of the targets relabeled, it's possible to configure the exporter itself. The YAML config file is generated from Nix with all of the modules described above. The targets are just the combination of all of the relabeled ones from above.

discovery.relabel "blackbox_instance" {
  targets = prometheus.exporter.blackbox.default.targets

  rule {
    source_labels = ["__param_target"]
    target_label = "instance"
  }
}

The blackbox exporter takes the targets given to it and reshapes them into valid Prometheus scrape targets. That means turning the address into a target parameter so that the address can point at the exporter itself. I still want to keep track of the target address afterwards though, so I copy that into the instance label.

prometheus.scrape "blackbox" {
  targets = discovery.relabel.blackbox_instance.output
  forward_to = [prometheus.remote_write.prod.receiver]
}

Finally, this final set of targets can be scraped and forwarded to Prometheus.

Monitoring Consul service health

environment.etc."alloy/consul.alloy".source = ./consul.alloy;

prometheus.exporter.consul "default" {
}

prometheus.scrape "consul" {
  targets = prometheus.exporter.consul.default.targets
  forward_to = [prometheus.remote_write.prod.receiver]
}

The Consul exporter for Prometheus exports metrics reflecting the health of services registered in Consul. I use this to alert when services are unhealthy. The exporter is bundled with Alloy and doesn't need any meaningful configuration: it can simply be created and scraped.