Distributed tracing with Tempo

{
  config,
  lib,
  ...
}:
let
  cfg = config.mjm.tempo;
in
{
  options.mjm.tempo = {
    enable = lib.mkEnableOption "Grafana Tempo";
  };

  config = lib.mkIf cfg.enable {
    <<config>>
  };

  _class = "nixos";
}

Tempo is Grafana's backend for storing and querying trace data. When done well, tracing can offer unmatched ability to debug problems in production. I'm using it wherever it is feasible to do so.

Service settings

mjm.services.tempo = {
  http = {
    port = 3200;
    health.path = "/ready";

    metrics.enable = true;
  };

  s3.enable = true;
  s3.buckets = [ "tempo-traces" ];
};

Tempo listens locally on port 3200, and that traffic is protected by a tunnel and advertised in Consul.

Tempo stores historical traces in a Garage bucket.

Config

services.tempo.enable = true;

The Tempo service of course needs to be enabled.

services.tempo.settings.server = {
  http_listen_address = "::1";
  grpc_listen_address = "127.0.0.1";
};

By default, Tempo will listen on all addresses, so I override this so it only listens locally, forcing external traffic to come from tunnels. Normally, I prefer to do this with IPv6 addresses, but when running in monolithic mode, Tempo seems to try to talk to itself over gRPC assuming an address of 127.0.0.1, which doesn't work if you listen on ::1. If I ever move things to an IPv6 only setup, that'll be something I have to deal with, likely by explicitly configuring all the addresses.

services.tempo.settings.distributor.receivers.otlp.protocols = {
  grpc.endpoint = "[::1]:16317";
  http.endpoint = ":14318";
};
mjm.spire.tunnels.tempo-grpc = {
  id = "tempo";
  mode = "server";
  listen.port = 14317;
  target.port = 16317;
  allowedServices = [ "alloy" ];
};
networking.firewall.allowedTCPPorts = [ 14318 ];

In addition to its own API, Tempo is also configured to expose listeners for the OTLP APIs, both gRPC and HTTP. Most things that send traces don't send them to Tempo directly: they send them to a local OpenTelemetry collector running via Alloy, and Alloy is the one that batches them and sends them to Tempo. Alloy will always use the gRPC endpoint for this. It does so through an mTLS tunnel, so the gRPC endpoint only needs to listen locally.

The HTTP endpoint on the other hand is exposed on the LAN without TLS. This supports being able to send traces from a development machine, which I sometimes find useful. I wish I had a good way to securely send things to Tempo from a workstation, but I haven't figured out the right way to do that yet.

mjm.spire.tunnels.alloy-tempo.enable = false;

Tempo doesn't itself generate traces, so it shouldn't enable tracing on the machine in the first place, meaning that the Tempo tunnel for Alloy should already be disabled. Disabling it here as well ensures that if something else on the machine enables it by mistake, that won't go unnoticed.

services.tempo.settings.live_store = {
  shutdown_marker_dir = "/var/lib/tempo/live-store/shutdown-marker";
  wal.path = "/var/lib/tempo/live-store/traces";
};

The live-store in Tempo handles recent traces: by default, those in the last twenty minutes. These are stored in memory and in a WAL on disk. The default directories it uses for storage need to be adjusted to be under /var/lib/tempo, as do other directories later.

services.tempo.settings.backend_scheduler = {
  provider.compaction.compaction.block_retention = "48h";
  local_work_path = "/var/lib/tempo/scheduler";
};

The backend scheduler has another directory that needs to be adjusted. I'm also adjusting the parameter that controls how long trace data gets kept for. The default is two weeks, but I'm usually interested in things that are more recent, so I reduce it to two days.

services.tempo.settings.storage.trace = {
  backend = "s3";
  s3 = {
    endpoint = "localhost:3902";
    bucket = "tempo-traces";
    region = "home";
    insecure = true;
    forcepathstyle = true;
  };
  wal.path = "/var/lib/tempo/wal";
  local.path = "/var/lib/tempo/blocks";
};

More directories to fix. Otherwise, this is just configuring Tempo to store older traces (ones that aren't currently in the live-store) in Garage in the tempo-traces bucket.