Grafana visualization and alerting

#| file: services/grafana/default.nix
{
  config,
  lib,
  ...
}:
let
  cfg = config.mjm.grafana;
  secrets = config.systemd.services.grafana.credentials.grafana;
in
{
  options.mjm.grafana = {
    enable = lib.mkEnableOption "grafana";
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };

  _class = "nixos";
}

Grafana is a common choice for a self-hosted tool for creating dashboards and generally exploring observability data. I use it for monitoring metrics, viewing logs and traces, and alerting.

Service settings

mjm.services.grafana = {
  <<service-settings>>
};

#| id: service-settings
http = {
  socket = "/run/grafana/server.sock";
  health.path = "/api/health";

  metrics.enable = true;

  ingress = {
    subdomain = "graphs";
    authMode = "oidc";
    oidc = {
      name = "Grafana";
      clientId = "7BReUARtsRcF6ypjiA4DcJ3E6fJNjzwheH5Tj1HCLoqfXCQSLHxZJHQ7bAV9U0aU";
      clientSecret = "$argon2id$v=19$m=65536,t=3,p=4$LExwz3BrD2Cu5o1ur61RIw$W4kCJsEG+VuCxeEOI689IEMoiE2r5G2Nwrc+q4fHU0c";
      redirectUris = [ "https://graphs.midna.dev/login/generic_oauth" ];
    };
  };
};

Grafana listens on a Unix domain socket and exposed externally as https://graphs.midna.dev. It has a health check endpoint that Consul uses, and it advertises metrics scraping for Prometheus. Grafana is protected with OpenID Connect for authentication.

#| id: service-settings
postgresql.enable = true;

Grafana stores its data in a PostgreSQL database.

#| id: service-settings
secrets.enable = true;

Grafana stores secrets in Vault for two purposes: its OIDC client secret and its secret key for encrypting saved data source credentials.

#| id: service-settings
upstreams = {
  loki.port = 13100;
  tempo.port = 3200;
  prometheus.port = 19090;
};

Grafana does not store observability data itself: it queries other datasources for that data, so it needs to be able to connect to those datasources. It uses tunnels to communicate with those upstream services via mutual TLS.

The Loki and Prometheus ports are 10000 + the usual port for those services, because Alloy on the same machine will use the usual ports for its own communication with these services. Arguably they could share tunnels, but they are using different service identities, so it feels off to have one using the other's identity.

Config

services.grafana.enable = true;

The Grafana service needs to be enabled on the machine.

services.grafana.settings.server = {
  protocol = "socket";
  socket = "/run/grafana/server.sock";
  socket_mode = "0666";
  domain = "graphs.midna.dev";
  root_url = "https://graphs.midna.dev";
};

As mentioned above, Grafana is configured to listen on a Unix domain socket. I'm not sure if both domain and root_url are needed. I know that domain alone is not enough: without the root_url setting, the OpenID Connect logic won't realize it should use URLs with HTTPS, because Grafana itself is only listening on HTTP.

services.grafana.settings.database = {
  type = "postgres";
  host = "/run/postgresql";
  user = "grafana";
};

Grafana needs to be configured to use the PostgreSQL database.

systemd.services.grafana.credentials.grafana.secret_key = { };
services.grafana.settings.security.secret_key = "$__file{${secrets.secret_key.path}}";

Grafana needs a secret key to use for encrypting sensitive datasource settings like passwords. I'm not sure I have any of these that are actually sensitive, but it needs it anyway.

services.grafana.settings."unified_alerting.state_history.annotations".max_age = "2w";

I'm using the built-in alerting that Grafana has, rather than running a separate Alertmanager instance (which I used to do). I don't remember the specifics, but Grafana generates some annotations related to alerting that it saves in its DB. These really add up over time, and by default never get cleaned up. I set a max age of two weeks so that the database size doesn't grow unbounded over time.

systemd.services.grafana.credentials.grafana."managed/oidc_client_secret" = { };
services.grafana.settings = {
  auth.disable_login_form = true;
  "auth.generic_oauth" = {
    enabled = true;
    name = "Authelia";
    icon = "signin";
    client_id = config.mjm.services.grafana.http.ingress.oidc.clientId;
    client_secret = "$__file{${secrets."managed/oidc_client_secret".path}}";
    scopes = "openid profile email groups";
    auth_url = "https://auth.midna.dev/api/oidc/authorization";
    token_url = "https://auth.midna.dev/api/oidc/token";
    api_url = "https://auth.midna.dev/api/oidc/userinfo";
    jwk_set_url = "https://auth.midna.dev/jwks.json";
    login_attribute_path = "preferred_username";
    groups_attribute_path = "groups";
    name_attribute_path = "name";
    use_pkce = true;
    validate_id_token = true;
    role_attribute_path = "contains(groups[*], 'admins') && 'GrafanaAdmin'";
    allow_assign_grafana_admin = true;
  };
};

This is the Grafana side of the OpenID Connect configuration mentioned above. Most of the configuration is straightforward I think. At the bottom, I've configured Grafana to make anyone in the "admins" group be a server administrator. Otherwise, users will default to being a viewer.

services.grafana.settings."tracing.opentelemetry.otlp".address = "localhost:4317";
mjm.server.alloy.tracing.enable = true;

Grafana supports OpenTelemetry tracing, so I enable that in Alloy and point Grafana to it.