Connecting to upstream services

{
  config,
  lib,
  ...
}:
let
  cfg = config.mjm.services;
  upstreamServices = lib.attrValues (lib.filterAttrs (_: s: s.upstreams != { }) cfg);
in
{
  options.mjm.services =
    let
      serviceType = {
        options = {
          <<options>>
        };
      };
    in
    lib.mkOption {
      type = with lib.types; attrsOf (submodule serviceType);
    };

  config = lib.mkIf (upstreamServices != [ ]) {
    <<config>>
  };

  _class = "nixos";
}

A service that needs to communicate with another service running in the lab can declare that other service as an upstream, providing it with a tunnel to automatically handle that communication over mutual TLS.

In addition to the tunnel created in this module, declaring an upstream also has the effect of configuring the upstream service to allow incoming traffic from this one. That logic is handled by the http module for services.

Service options

These options are available within the scope of a particular entry of mjm.services.

upstreams =
  let
    upstreamType = { name, ... }: {
      options = {
        <<upstream-options>>
      };
    };
  in
  lib.mkOption {
    type = with lib.types; attrsOf (submodule upstreamType);
    default = { };
  };

upstreams is an attribute set where the key is the name of the upstream service. A service can have any number of upstreams. Each upstream has a few options to configure it.

name = lib.mkOption {
  type = lib.types.str;
  default = name;
  readOnly = true;
};

port = lib.mkOption { type = lib.types.port; };

The local TCP port that this service should use to connect to the upstream service.

ipv4Only = lib.mkOption {
  type = lib.types.bool;
  default = false;
};

Flag to make the tunnel for the upstream service listen on 127.0.0.1 instead of [::1]. While I usually prefer to default to using IPv6 for things, some services I use have a bug where if an IPv6 connection fails, they permanently try to use IPv4 for connections from then on, as long as there is a routable IPv4 address available. I can't yet disable IPv4 completely, so until then I need this hack to listen on IPv4 instead for those services.

Example

mjm.services.alice = {
  upstreams.bob.port = 3000;
};

Now the machine running the alice service can make requests to localhost:3000 to reach bob. The connections will use the svc/alice SPIFFE ID, which bob will be automatically configured to allow. Similarly, when alice connects to bob this way, the tunnel will verify that the server is actually using the svc/bob SPIFFE ID, so another service can't register in Consul and then impersonate bob. And the actual alice service doesn't need to manage any certificates to get this secure communication: it can communicate with localhost:3000 in plaintext.

Config

mjm.spire.tunnels = lib.pipe upstreamServices [
  (map (
    { name, upstreams, ... }:
    lib.mapAttrs' (
      upstreamName:
      {
        port,
        ipv4Only,
        ...
      }:
      lib.nameValuePair "${name}-${upstreamName}" {
        id = name;
        mode = "client";
        listen.port = lib.mkIf (!ipv4Only) port;
        listen.address = lib.mkIf ipv4Only "127.0.0.1:${toString port}";
        target.service = upstreamName;
      }
    ) upstreams
  ))
  lib.mergeAttrsList
];

The config here is simple: it creates a client tunnel for each upstream declared for each service. The upstreams option for services is really a small DSL for defining these tunnels in a convenient way.

Note that there is no setting for what port the upstream service is listening on. I use a patch that I created for Ghostunnel to let it use SRV DNS records to resolve the target. Those records include the port number, not just the IP address. This is great because it means that clients don't need to know the port the service is listening on, and services with multiple instances can have each instance listening on a different port without issue.