spiffe-garage

spiffe-garage is a service that provides Garage key credentials to other services based on their SPIFFE ID.

func main() {
	ctx := context.Background()

	<<setup-tracing>>

	var cli CLI
	c := kong.Parse(&cli, kong.BindTo(ctx, (*context.Context)(nil)), kong.Vars{
		"creds_dir": os.Getenv("CREDENTIALS_DIRECTORY"),
	})
	if err := c.Run(); err != nil {
		slog.Error("spiffe-garage failed", "error", err)
		os.Exit(1)
	}
}

This is pretty common setup across my Go programs. I use Kong to make it easy to define CLI subcommands and arguments. Every subcommand has access to the parent context. If running the subcommand produces an error, then the error is logged and the command exits with a failing code.

I pass the CREDENTIALS_DIRECTORY environment variable to Kong as "creds_dir". Flags can then reference creds_dir in their default values.

type CLI struct {
	Serve ServeCmd `cmd:"" default:"withargs"`
}

The CLI type for spiffe-garage has a single subcommand "serve". For convenience, it's marked as the default command. It isn't really necessary to use a subcommand here, but I prefer the extra bit of structure.

exp, err := otlptracehttp.New(ctx)
if err != nil {
	panic(err)
}

res, err := resource.New(ctx, resource.WithFromEnv(),
	resource.WithTelemetrySDK(),
	resource.WithOS(),
	resource.WithHost(),
	resource.WithAttributes(semconv.ServiceName("spiffe-garage")))
if err != nil {
	panic(err)
}

tracerProvider := trace.NewTracerProvider(
	trace.WithBatcher(exp),
	trace.WithResource(res))
defer func() {
	if err := tracerProvider.Shutdown(ctx); err != nil {
		panic(err)
	}
}()
otel.SetTracerProvider(tracerProvider)

Before running the chosen subcommand, spiffe-garage sets up OpenTelemetry tracing. This will automatically send traces to the default OpenTelemetry HTTP endpoint, if something is listening. On my machines, that would be Grafana Alloy, which would send the traces to Tempo. Right now, I don't have the tracing collector set up on my Garage machines, because it introduces a cyclic dependency between spiffe-garage and Tempo.

package main

import (
	"context"
	"log/slog"
	"os"

	"github.com/alecthomas/kong"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	"go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.20.0"
)

var tracer = otel.Tracer("git.midna.dev/mjm/nix-config/packages/spiffe-tool/cmd/spiffe-garage")

<<CLI>>
<<main>>

spiffe-garage serve

The serve command runs the server that listens for incoming requests for Garage credentials.

type ServeCmd struct {
	AdminToken      []byte `name:"admin-token-file" type:"filecontent" default:"${creds_dir}/spiffe-garage.admin_token"`
	GarageAdminAddr string `default:"[::1]:3903"`
	TrustDomain     string `default:"home.mattmoriarity.com"`
	ConfigFile      string `name:"config" type:"existingfile"`

	garage         *garage.APIClient
	config         Config
	isShuttingDown atomic.Bool
}

The serve command accepts three possible flags, but they all have defaults that are suitable for production. An admin token for Garage is needed, as well as the address to connect to for the admin API. These are necessary to be able to manage Garage keys. The trust domain for SPIFFE is needed so the server can reject requests from other trust domains.

The non-flag fields on the structure will be initialized and used when the command is run.

type Config struct {
	Identities map[string]IdentityConfig `json:"identities"`
}

type IdentityConfig struct {
	Buckets map[string]*garage.ApiBucketKeyPerm `json:"buckets"`
}

The server uses a JSON configuration file to determine which identities are allowed to request credentials, and which buckets they are given access to. The Config type captures the structure of that config file.

func (c *ServeCmd) Run(ctx context.Context) error {
	<<get-socket-listener>>
	<<setup-signal-context>>
	<<setup-tls>>
	<<create-garage-client>>
	<<read-config-file>>
	<<serve-http>>
	<<wait>>
	<<shutdown>>
}

The serve command runs through a few setup steps before starting the HTTPS server, waiting to be terminated, and finally shutting down.

listeners, err := activation.Listeners()
if err != nil {
	return fmt.Errorf("getting socket listeners: %w", err)
}

if len(listeners) != 1 {
	return fmt.Errorf("incorrect number of listeners (expected 1, got %d)", len(listeners))
}
l := listeners[0]
defer l.Close()

spiffe-garage is expecting to be run with a socket passed from systemd, so this gets the listener for that socket file descriptor. The socket should be exposed to external HTTPS requests from clients that need credentials.

ctx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
defer stop()

This provides a context to use going forward which will be cancelled when the program receives an interrupt or terminate signal. It's an easy way to give the various concurrent tasks that are going to be started soon a way to know when to drop what they are doing so the program can exit.

source, err := workloadapi.NewX509Source(ctx)
if err != nil {
	return fmt.Errorf("creating x509 source: %w", err)
}
defer source.Close()

trustDomain := spiffeid.RequireTrustDomainFromString(c.TrustDomain)
tlsConfig := tlsconfig.MTLSServerConfig(source, source, tlsconfig.AuthorizeMemberOf(trustDomain))
tlsListen := tls.NewListener(l, tlsConfig)

To listening for incoming mTLS connections, spiffe-garage uses the Go SPIFFE SDK to set up a TLS config. It will connect to the SPIFFE workload API, request a certificate and trust bundle, and also configure the TLS verification to only accept client certificates that match the expected trust domain.

adminToken := strings.TrimSpace(string(c.AdminToken))
if adminToken == "" {
	return fmt.Errorf("admin token for garage is empty")
}

rCtx, stopRequests := context.WithCancel(context.Background())
rCtx = context.WithValue(rCtx, garage.ContextAccessToken, adminToken)
defer stopRequests()

gConfig := garage.NewConfiguration()
gConfig.Host = c.GarageAdminAddr
c.garage = garage.NewAPIClient(gConfig)

Next, the server will need an API client for the Garage admin API. It first checks if the admin token passed in is non-empty, and bails early if not.

The Garage client takes the token from a value stored in the context. So it creates a context, rCtx, that includes that value and will be used as the context when handling incoming requests.

Finally, it creates a new client while specifying the host to target, which is read from the flag passed to the command.

f, err := os.Open(c.ConfigFile)
if err != nil {
	return fmt.Errorf("reading config file: %w", err)
}
defer f.Close()

if err := json.NewDecoder(f).Decode(&c.config); err != nil {
	return fmt.Errorf("decoding config json: %w", err)
}

The server will create new Garage access keys on-demand for each successful request. To be useful, those keys need to be assigned permissions for the buckets they need to access. The rules for those permissions come from a JSON config file, which is read when the server starts.

srv := &http.Server{
	Handler:     c.Handler(),
	BaseContext: func(_ net.Listener) context.Context { return rCtx },
}
go func() {
	if err := srv.Serve(tlsListen); err != nil && err != http.ErrServerClosed {
		panic(err)
	}
}()

Now it's time to get the HTTPS server listening. The Handler() method will be covered later and defines the routes the server can handle. BaseContext is set to provide rCtx as the context associated with each request.

The server listens on a separate goroutine, because the main one will be responsible for waiting for a termination signal.

daemon.SdNotify(false, daemon.SdNotifyReady)
<-ctx.Done()

With the server listening, spiffe-garage is ready, and notifies systemd of this fact. Then it waits for the signal context to be done, which should only happen when a termination signal is received.

c.isShuttingDown.Store(true)
time.Sleep(20 * time.Second)

shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()

return srv.Shutdown(shutdownCtx)

spiffe-garage goes out of its way to avoid clients making requests unsuccessfully while it is trying to shutdown. When it's time, it sets a flag to indicate that the server is shutting down. This will cause the health check to start failing, but incoming requests for credentials will be unaffected. Then it waits 20 seconds so that the 15 second interval for the Consul health check has passed, so it can be reasonably assumed that Consul has noticed that this instance is unavailable, and clients shouldn't send new requests to it anymore.

At this point, it's safe to actually shutdown the server. It won't be able to accept incoming requests anymore, but that's okay because clients should already know this instance is unhealthy. Existing requests are given 15 seconds to finish, which should be more than enough time under healthy conditions.

func (c *ServeCmd) Handler() http.Handler {
	m := http.NewServeMux()
	m.Handle("GET /healthz", http.HandlerFunc(c.checkHealth))
	m.Handle("/creds", http.HandlerFunc(c.getCreds))
	return otelhttp.NewHandler(m, "Handler")
}

The HTTP handler for the server is pretty simple. The /healthz endpoint provides a health check to be used by Consul. The /creds endpoint is the one that will be called by clients in order to get credentials.

func (c *ServeCmd) getCreds(w http.ResponseWriter, r *http.Request) {
	ctx := r.Context()
	span := trace.SpanFromContext(ctx)

	<<get-spiffe-id>>
	<<get-matching-identity>>
	<<create-key>>
	<<allow-bucket-access>>
	<<send-cred-response>>
}

Getting credentials involves a series of steps:

• Get the SPIFFE ID from the client certificate
• Check for a matching identity in the server's configuration
• Create a new access key
• Grant the access key permission for the configured buckets
• Send the response containing the credentials for the new key

spiffeID, err := getSPIFFEIDFromCerts(ctx, r.TLS.PeerCertificates)
if err != nil {
	span.RecordError(err)
	span.SetStatus(codes.Error, err.Error())
	http.Error(w, fmt.Sprintf("error getting spiffe id from request: %v", err), http.StatusInternalServerError)
	return
}
span.SetAttributes(attribute.Stringer("spiffe.id", spiffeID))

A helper function is used to pull the SPIFFE ID out of the client certificate that the incoming request presented. If a SPIFFE ID can't be found for some reason, it's an error and credentials will not be provided. If one is found, it's set as an attribute in the tracing span.

func getSPIFFEIDFromCerts(ctx context.Context, certs []*x509.Certificate) (spiffeid.ID, error) {
	ctx, span := tracer.Start(ctx, "getSPIFFEIDFromCerts", trace.WithAttributes(attribute.Int("cert.count", len(certs))))
	defer span.End()

	if len(certs) == 0 {
		err := fmt.Errorf("no certs in https request")
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return spiffeid.ID{}, err
	}

	cert := certs[0]
	uris := cert.URIs
	span.SetAttributes(attribute.Int("cert.uri.count", len(uris)))
	if len(uris) != 1 {
		err := fmt.Errorf("wrong number of uris in certificate, expected 1, got %d", len(uris))
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return spiffeid.ID{}, err
	}

	uri := uris[0]
	span.SetAttributes(attribute.Stringer("cert.uri", uri))

	spiffeID, err := spiffeid.FromURI(uri)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return spiffeid.ID{}, fmt.Errorf("uri %q found in certificate is not a valid spiffe id: %w", uri, err)
	}

	span.SetAttributes(attribute.Stringer("spiffe.id", spiffeID))
	return spiffeID, nil
}

getSPIFFEIDFromCerts looks at a list of TLS client certificates and determines the SPIFFE ID from them. Only the first certificate in the list is examined: the client should not be presenting multiple certificates. That first certificate is expected to have a single URI SAN that contains the SPIFFE ID. If that URI is present and is a valid SPIFFE ID, it is returned without error. Otherwise, an appropriate error is returned.

ident, ok := c.config.Identities[spiffeID.String()]
if !ok {
	err := fmt.Errorf("no configured identity matching %q", spiffeID)
	span.RecordError(err)
	span.SetStatus(codes.Error, err.Error())
	http.Error(w, err.Error(), http.StatusForbidden)
	return
}

After getting the SPIFFE ID for the client making the request, it looks up the corresponding entry in the server's configuration. Requesting credentials for an identity that isn't configured will fail the request, because it doesn't make sense to issue credentials without a list of buckets to give access to.

expAt := time.Now().Add(1 * time.Hour)
keyInfo, _, err := c.garage.AccessKeyAPI.CreateKey(ctx).Body(garage.UpdateKeyRequestBody{
	Expiration: *garage.NewNullableTime(&expAt),
}).Execute()
if err != nil {
	span.RecordError(err)
	span.SetStatus(codes.Error, err.Error())
	http.Error(w, fmt.Sprintf("error creating new access key: %v", err), http.StatusInternalServerError)
	return
}

keyID := keyInfo.GetAccessKeyId()
secretKey := keyInfo.GetSecretAccessKey()

span.SetAttributes(attribute.String("garage.key_id", keyID))

Now that it has been determined that it makes sense to issue a new access key, said key is created. The key will expire within an hour, which is fine because the response will indicate this to the client, so the client will know to request a new one as that expiration approaches.

for name, bucket := range ident.Buckets {
	bucketInfo, _, err := c.garage.BucketAPI.GetBucketInfo(ctx).GlobalAlias(name).Execute()
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		http.Error(w, fmt.Sprintf("error getting id for %q bucket: %v", name, err), http.StatusInternalServerError)
		return
	}

	_, _, err = c.garage.PermissionAPI.AllowBucketKey(ctx).Body(*garage.NewBucketKeyPermChangeRequest(keyInfo.GetAccessKeyId(), bucketInfo.GetId(), *bucket)).Execute()
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		http.Error(w, fmt.Sprintf("error allowing access to %q bucket: %v", name, err), http.StatusInternalServerError)
		return
	}
}

Once the key has been created successfully, it is granted access to each bucket listed for the identity in the server's configuration. The configuration references buckets by their global alias, since those are the human-friendly name, but the API for altering the permissions requires the actual ID of the bucket, so that needs to be fetched.

resp := struct {
	Version         int
	AccessKeyId     string
	SecretAccessKey string
	Token           string
	Expiration      time.Time
}{
	1,
	keyID,
	secretKey,
	"",
	expAt,
}

out, err := json.Marshal(resp)
if err != nil {
	span.RecordError(err)
	span.SetStatus(codes.Error, err.Error())
	http.Error(w, fmt.Sprintf("error marshalling json response: %v", err), http.StatusInternalServerError)
	return
}

w.Header().Add("Content-Type", "application/json")
w.Write(out)

With the access key properly provisioned, the only thing left to do is send the information for it to the client. An anonymous struct type is used to serialize it to JSON. The format here is what AWS SDK credential providers expect to receive. Note that the response includes the expiration timestamp, so the SDK knows when to request new credentials to avoid interruption.

func (c *ServeCmd) checkHealth(w http.ResponseWriter, r *http.Request) {
	if c.isShuttingDown.Load() {
		http.Error(w, "Server is shutting down", http.StatusServiceUnavailable)
		return
	}

	_, err := c.garage.SpecialEndpointsAPI.Health(r.Context()).Execute()
	if err != nil {
		http.Error(w, fmt.Sprintf("Error checking Garage health: %v", err), http.StatusServiceUnavailable)
		return
	}

	w.WriteHeader(http.StatusOK)
	fmt.Fprint(w, "OK")
}

The other route that spiffe-garage handles is a health check. As mentioned earlier, if the server is shutting down, the health check fails while the server continues handling incoming requests. Otherwise, spiffe-garage is healthy if it can communicate with Garage successfully.

package main

import (
	"context"
	"crypto/tls"
	"crypto/x509"
	"encoding/json"
	"fmt"
	"net"
	"net/http"
	"os"
	"os/signal"
	"strings"
	"sync/atomic"
	"syscall"
	"time"

	"git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-golang"
	"github.com/coreos/go-systemd/v22/activation"
	"github.com/coreos/go-systemd/v22/daemon"
	"github.com/spiffe/go-spiffe/v2/spiffeid"
	"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
	"github.com/spiffe/go-spiffe/v2/workloadapi"
	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"
)

<<ServeCmd>>
<<ServeCmd.Run>>
<<ServeCmd.Handler>>
<<ServeCmd.getCreds>>
<<getSPIFFEIDFromCerts>>
<<ServeCmd.checkHealth>>
<<Config>>