type Host struct {
Name string
System string
DrvPath string
OutPath string
DeployConfig Config
RebootNeeded bool
sess *Session
log *slog.Logger
remoteRunner cmd.Runner
}
dippy has a type Host to represent a single host that it might need to interact with in some way. Hosts are created after performing a Nix evaluation of a deploy plan.
type Config struct {
TargetHost *string `json:"targetHost"`
TargetUser *string `json:"targetUser"`
AutoReboot bool `json:"rebootAutomatically"`
ConsulChecks []ConsulCheck `json:"consulChecks"`
}
type ConsulCheck struct {
Node string `json:"node"`
Name string `json:"name"`
}
The Config for a host is configured in Nix using the mjm.deploy.* options. When the deploy plan is evaluated, these settings are serialized to a JSON file, which dippy then reads and parses into the Config struct.
func NewHost(sess *Session, name string, system string, drvPath string, outPath string, deployConfig Config) *Host {
logger := slog.Default().WithGroup("host").With("name", name)
return &Host{
Name: name,
System: system,
DrvPath: drvPath,
OutPath: outPath,
DeployConfig: deployConfig,
sess: sess,
log: logger,
}
}
NewHost, as the name implies, creates a new Host. There isn't much special logic here. It creates a logger for the host that always includes its name, which is then used in later methods when actually performing actions on the host. It also stores the Session used to create it, which will be used to perform many of the operations described below.
func (h *Host) IsLocal() bool {
return h.DeployConfig.TargetHost == nil
}
func (h *Host) IsRemote() bool {
return !h.IsLocal()
}
A host with no target host configured is a local host: one that isn't intended to be deployed to remotely. Only my servers get deployed to remotely: workstations are manually updated from the machine itself.
In CI builds for pull requests, I want to build all hosts, including local ones, to be able to detect problems, so local hosts are included in the deploy plan like remote ones. But when it's time to perform an actual deploy, I want to skip these hosts. So being able to check whether a host is local or remote easily is useful.
var _ cmd.Runner = (*Host)(nil)
func (h *Host) Execute(ctx context.Context, c *cmd.Cmd) error {
runner, err := h.getRunner(ctx)
if err != nil {
return err
}
return runner.Execute(ctx, c)
}
Host implements the cmd.Runner interface from dippy's cmd package, meaning that a host is a thing that knows how to run arbitrary CLI commands. It does so by delegating to another runner, making this a simple convenience where code can do this:
cmd.Run(ctx, host, "echo", "hello world!")
instead of:
r, err := host.getRunner(ctx)
if err != nil {
...
}
cmd.Run(ctx, r, "echo", "hello world!")This will be used throughout the rest of the implementation of Host.
func (h *Host) getRunner(ctx context.Context) (cmd.Runner, error) {
<<Host.getRunner-local>>
<<Host.getRunner-cached-remote>>
<<Host.getRunner-new-remote>>
}
There are three possible cases for getting the command runner for a host:
//| id: Host.getRunner-local
if h.IsLocal() {
return h.sess, nil
}If the host is local, then the runner is just the host's deploy session, which will delegate to its command runner. This is expected to be a runner that executes locally on the machine running dippy.
//| id: Host.getRunner-cached-remote
if h.remoteRunner != nil {
h.log.DebugContext(ctx, "reusing existing remote runner")
return h.remoteRunner, nil
}For remote hosts, the runner is expected to operate over an SSH connection. The connection should ideally be reused across multiple commands, so once created it is stored on the host. So if it has already been created, the stored one is returned.
//| id: Host.getRunner-new-remote
h.log.DebugContext(ctx, "creating new remote runner")
runner, err := h.sess.RemoteRunner(*h.DeployConfig.TargetHost, *h.DeployConfig.TargetUser)
if err != nil {
return nil, fmt.Errorf("creating remote runner for %s: %w", h.Name, err)
}
h.remoteRunner = runner
return runner, nilIf no remote runner is stored already, then it needs to be created. This is done by calling the RemoteRunner function from the session, passing in the target host and user from the host's deployment config. This is expected to give a command runner that will connect to the host using SSH, but it is abstracted this way to support unit testing. Once created, the runner is stored on the host so that future commands will use the same runner and therefore the same SSH connection.
func (h *Host) Build(ctx context.Context) error {
l := h.log.With("drv_path", h.DrvPath)
l.InfoContext(ctx, "building host")
if err := h.sess.Realise(ctx, []string{h.DrvPath}); err != nil {
return fmt.Errorf("realising node %s: %w", h.Name, err)
}
l.InfoContext(ctx, "finished building host", "out_path", h.OutPath)
return nil
}
Building the toplevel for a host is a straightforward operation. Evaluating the host will produce the path to the derivation for it, so the Host already has that information. It just needs to ask Nix to realize that derivation.
This method only supports building a single host at a time, so it isn't usually used for commands that support operating on multiple hosts, like deploy and diff. For those, a similar method exists on Plan that will build every host in parallel, showing the output for all of them.
func (h *Host) Push(ctx context.Context) error {
if h.IsLocal() {
return nil
}
toUrl := fmt.Sprintf("ssh-ng://%s@%s", *h.DeployConfig.TargetUser, *h.DeployConfig.TargetHost)
l := h.log.With("out_path", h.OutPath, "target", toUrl)
l.InfoContext(ctx, "pushing system")
if err := h.sess.Copy(ctx, nix.CopyOptions{
Installables: []string{h.OutPath},
To: toUrl,
}); err != nil {
return fmt.Errorf("copying closure for %s: %w", h.Name, err)
}
l.InfoContext(ctx, "pushed system")
return nil
}
Push will copy the host's toplevel to the host itself, enabling further operations that require that it actually be present on the machine in the Nix store. It assumes that the toplevel was already built. This method does nothing if the host is local: no pushing is needed in that case, since the build already happened on the machine.
An ssh-ng URL is constructed from the deployment config for the host, and then a Nix copy is done to copy the toplevel to the machine. Nix will handle getting all of the dependencies copied as well, so once the operation is done, the toplevel and everything it needs to work will be present in the store on the remote machine.
func (h *Host) PushToAttic(ctx context.Context) error {
l := h.log.With("out_path", h.OutPath)
l.InfoContext(ctx, "pushing to attic cache")
if _, err := backoff.Retry(ctx, func() (any, error) {
if err := cmd.Run(ctx, h.sess, "attic", "push", "homelab-dippy:homelab", h.OutPath); err != nil {
return nil, fmt.Errorf("running attic push: %w", err)
}
return nil, nil
}, backoff.WithMaxTries(5), backoff.WithNotify(func(err error, d time.Duration) {
slog.WarnContext(ctx, "will retry failed operation", "error", err, "retry_in", d)
})); err != nil {
return err
}
l.InfoContext(ctx, "pushed to attic cache")
return nil
}
PushToAttic will copy the host's toplevel, and all of its dependencies, to my lab's attic cache. This cache is configured as a substituter in the Nix configuration of my machines, so this lets me avoid rebuilding things from source multiple times.
The push operation will sometimes fail intermittently, but usually succeeds upon a retry, so dippy does so automatically to avoid failing a diff or deploy for no good reason.
func (h *Host) Diff(ctx context.Context) ([]byte, error) {
h.log.InfoContext(ctx, "diffing against current system", "out_path", h.OutPath)
output, err := cmd.Output(ctx, h, path.Join(h.OutPath, "bin/nvd-json"), "diff", "/run/current-system", h.OutPath)
if err != nil {
return nil, fmt.Errorf("running nvd-json: %w", err)
}
return output, nil
}
Diff compares the packages installed in the new toplevel against those in the currently running system. It does so by running nvd-json, a tool I wrote for this purpose, on the system. nvd-json produces structured JSON that describes which packages have been added, removed, or changed in version. It also checks switch inhibitors to determine if a reboot is needed instead of switching in-place.
The output of nvd-json is returned from this method as raw bytes. It is the job of the caller to parse it as needed.
func (h *Host) DiffLocal(ctx context.Context) (bool, error) {
h.log.DebugContext(ctx, "diffing against current system", "out_path", h.OutPath)
output, err := cmd.Output(ctx, h, path.Join(h.OutPath, "bin/nvd-json"), "diff", "/run/current-system", h.OutPath)
if err != nil {
return false, fmt.Errorf("running nvd-json: %w", err)
}
var result struct {
Left string `json:"left"`
Right string `json:"right"`
Inhibitors []any `json:"inhibitors"`
}
if err := json.Unmarshal(output, &result); err != nil {
return false, fmt.Errorf("decoding nvd-json output: %w", err)
}
if result.Left == result.Right {
return false, nil
}
h.RebootNeeded = len(result.Inhibitors) > 0
if err := cmd.Run(ctx, h.sess, "nvd", "diff", "/run/current-system", h.OutPath); err != nil {
return false, fmt.Errorf("running nvd: %w", err)
}
return true, nil
}
DiffLocal performs a slightly different diff operation that is used when applying changes to the local host. It still runs nvd-json, but it is not interested in the full output. It uses the "left" and "right" fields to determine if the new toplevel is the same as the one the system is already running. If it is, then DiffLocal returns immediately with a result of false.
If the new toplevel is different than what is currently running, the RebootNeeded flag is updated on the host based on whether there are any switch inhibitors that changed. Then, dippy will run nvd to show the diff between the systems in the terminal. nvd is of course the inspiration for nvd-json: I wanted to show similar information in contexts besides the terminal. But for an interactive session at the terminal, I've so far opted not to reinvent the wheel.
DiffLocal then returns true to the caller to indicate that there are changes that can be applied.
func (h *Host) CheckRebootNeeded(ctx context.Context) error {
h.log.InfoContext(ctx, "checking if reboot is needed")
output, err := cmd.Output(ctx, h, path.Join(h.OutPath, "bin/nvd-json"), "reboot-check", h.OutPath)
if err != nil {
return fmt.Errorf("running nvd-json: %w", err)
}
var result struct {
RebootNeeded bool `json:"reboot_needed"`
}
if err := json.Unmarshal(output, &result); err != nil {
return fmt.Errorf("decoding nvd-json output: %w", err)
}
h.RebootNeeded = result.RebootNeeded
return nil
}
CheckRebootNeeded uses nvd-json to determine if the host can be switched to the new generation in-place or if it needs to be rebooted into the new generation. The check is based on a NixOS feature called switch inhibitors, which prevent using the "switch" action if particular things have changed from the current generation. nvd-json will check these inhibitors as well and report that a reboot is needed if any are changed.
The result will be stored on the host so it can then be used when deploying.
func (h *Host) Deploy(ctx context.Context, goal string) error {
h.log.InfoContext(ctx, "deploying")
<<Host.Deploy-apply>>
<<Host.Deploy-reboot>>
<<Host.Deploy-wait-until-healthy>>
h.log.InfoContext(ctx, "deployed")
return nil
}
Deploy applies the previously built and pushed changes to the remote host. This action happens in three steps: applying the changes, rebooting the host if needed, and waiting for the host to be healthy.
//| id: Host.Deploy-apply
if goal == "" {
goal = "switch"
if h.RebootNeeded {
goal = "boot"
}
}
if err := h.apply(ctx, goal); err != nil {
return fmt.Errorf("applying: %w", err)
}If no specific goal was requested and passed in to Deploy, then the goal needs to be chosen here. It will be "switch" unless it was determined that the host needs to be rebooted, in which case it will be "boot". Then the changes are applied using that goal.
const systemProfile = "/nix/var/nix/profiles/system"
func (h *Host) apply(ctx context.Context, goal string) error {
h.log.InfoContext(ctx, "setting system profile", "out_path", h.OutPath)
if err := cmd.Run(ctx, h, "run0", "nix-env", "--profile", systemProfile, "--set", h.OutPath); err != nil {
return fmt.Errorf("setting system profile: %w", err)
}
h.log.InfoContext(ctx, "activating system", "goal", goal)
if err := cmd.Run(ctx, h, "run0", path.Join(systemProfile, "bin/switch-to-configuration"), goal); err != nil {
return fmt.Errorf("running switch-to-configuration: %w", err)
}
return nil
}
Applying the changes is two steps. First, nix-env is used to set the system profile to point at the new toplevel. This creates a new generation in that profile. Once that is done, switch-to-configuration is called with the chosen goal, which applies the changes as appropriate.
This is exactly what nixos-rebuild does after building the system.
//| id: Host.Deploy-reboot
var rebootSkipped bool
if goal == "boot" {
if h.DeployConfig.AutoReboot {
if err := h.Reboot(ctx); err != nil {
return fmt.Errorf("rebooting: %w", err)
}
} else {
rebootSkipped = true
}
}If the boot goal is being used, whether by request or because it was detected to be necessary, the host needs to be rebooted. By default, dippy will reboot the host when deploying, and wait for it to come back up. Some hosts are configured to not automatically reboot, though, because doing so will interrupt the deploy process. These hosts will still have the boot goal applied, so the next reboot will be into the new generation, but dippy won't do that reboot automatically. Instead, I will have to do it by hand later.
//| id: Host.Deploy-wait-until-healthy
if !rebootSkipped {
if err := h.WaitUntilHealthy(ctx); err != nil {
return fmt.Errorf("waiting for %s to be healthy: %w", h.Name, err)
}
}Finally, if the changes were actually applied now, rather than being delayed until a later reboot, dippy will wait for the host to be healthy according to Consul before considering the deploy completed. The details of this are covered in a later section.
func (h *Host) ApplyLocal(ctx context.Context) error {
<<Host.ApplyLocal-choose-goal>>
if err := h.apply(ctx, goal); err != nil {
return fmt.Errorf("applying: %w", err)
}
if goal == "boot" {
fmt.Fprintln(os.Stderr, "Reboot to apply changes.")
}
return nil
}
ApplyLocal applies the previously built changes to the local host. Because it's an interactive process rather than a fully-automated one, it behaves a little differently than Deploy. First it needs to choose the goal to apply, and then it needs to actually apply those changes much like a deploy would. Then, if a reboot is needed, the user is informed of that, but dippy will not initiate it automatically.
//| id: Host.ApplyLocal-choose-goal
goal := "switch"
if h.RebootNeeded {
goal = "boot"
}
reader := bufio.NewReader(os.Stdin)
for {
fmt.Fprintf(os.Stderr, "Apply these changes with %s goal? ", goal)
input, err := reader.ReadString('\n')
if err != nil {
return fmt.Errorf("confirming apply: %v", err)
}
input = strings.ToLower(strings.TrimSpace(input))
if input == "y" || input == "yes" {
break
}
if input == "n" || input == "no" {
return nil
}
if input == "switch" || input == "boot" {
goal = input
h.log.InfoContext(ctx, "overriding goal", "goal", goal)
break
}
}First, a goal is chosen based on switch inhibitors, much like it would be in a deploy. Before applying, the user is asked to confirm if they would like to apply the changes. Before ApplyLocal, a diff of the changes should have been presented, so they would have that context to be able to decide whether to proceed.
If the user enters "y" or "yes", they will continue using the automatically chosen goal. If they enter "n" or "no", no changes will be applied. If they enter "switch" or "boot", then they will override the automatically chosen goal with the one they entered.
func (h *Host) Reboot(ctx context.Context) error {
h.log.InfoContext(ctx, "rebooting")
<<Host.Reboot-get-old-id>>
<<Host.Reboot-reboot>>
<<Host.Reboot-wait-loop>>
h.log.InfoContext(ctx, "rebooted")
return nil
}
Reboot tells the remote host to reboot and then waits for it to come back up. The waiting is important, as it could introduce unavailability to reboot a host and then continue deploying more hosts. Too many things might be down at once.
//| id: Host.Reboot-get-old-id
oldID, err := h.getBootID(ctx)
if err != nil {
return fmt.Errorf("getting original boot id: %w", err)
}
h.log.DebugContext(ctx, "got original boot id", "boot_id", oldID)Before rebooting, dippy gets the current boot ID from the host and stores it for later. This is how it will be able to know that the host has actually rebooted.
func (h *Host) getBootID(ctx context.Context) (string, error) {
ctx, cancel := context.WithTimeoutCause(ctx, 10*time.Second, fmt.Errorf("timeout checking boot ID"))
defer cancel()
output, err := cmd.Output(ctx, h, "cat", "/proc/sys/kernel/random/boot_id")
if err != nil {
return "", fmt.Errorf("getting boot id: %w", err)
}
return strings.TrimSpace(string(output)), nil
}
The boot ID is a random UUID that is accessible from /proc/sys/kernel/random/boot_id. It changes every boot, so it can be used to know if a host has rebooted between two points in time where the contents of the file was checked.
//| id: Host.Reboot-reboot
if err := cmd.Run(ctx, h, "run0", "reboot"); err != nil && !errors.Is(err, &ssh.ExitMissingError{}) {
return fmt.Errorf("initiating reboot: %w", err)
}The "run0 reboot" command is used to perform the reboot. If the command exits with an SSH error that there was no exit code, that is ignored, because often the SSH connection being used to execute the command is terminated once the reboot is started. That's fine and expected.
//| id: Host.Reboot-wait-loop
h.log.InfoContext(ctx, "waiting for reboot")
for {
newID, err := h.getBootID(ctx)
h.log.DebugContext(ctx, "check for new boot id", slog.Group("boot_id", "old", oldID, "new", newID), "err", err)
if err == nil && newID != oldID {
break
}
if err != nil {
h.remoteRunner = nil
}
time.Sleep(2 * time.Second)
}Now dippy waits in a loop where every two seconds, it tries to connect to the host and get its current boot ID. If it does so successfully and the new boot ID is different than the old one, then the host has successfully rebooted and the loop terminates.
The expectation is that in most iterations, checking the boot ID results in an error because the host is unreachable. That's not an issue and isn't reported. But if an error does occur, then the remote command runner that is cached on the host is cleared out, forcing a new connection to be made on the next iteration. This is important, as the previous connection might not be usable anymore. The loop could easily get stuck without a way to terminate if this wasn't done.
func (h *Host) WaitUntilHealthy(ctx context.Context) error {
if len(h.DeployConfig.ConsulChecks) == 0 {
return nil
}
<<Host.WaitUntilHealthy-setup>>
<<Host.WaitUntilHealthy-spawn-goroutines>>
<<Host.WaitUntilHealthy-wait>>
}
WaitUntilHealthy pauses the deploy to check if the host is healthy according to Consul checks. In its NixOS configuration, a host can designate which Consul services need to be healthy in order to advance the deploy. If it doesn't choose any, then this method returns right away.
//| id: Host.WaitUntilHealthy-setup
l := h.log.WithGroup("consul")
l.InfoContext(ctx, "waiting for healthy host", "checks", h.DeployConfig.ConsulChecks)
time.Sleep(5 * time.Second)
client, err := consulapi.NewClient(consulapi.DefaultNonPooledConfig())
if err != nil {
return fmt.Errorf("creating consul client: %w", err)
}If there are Consul services that need to be checked, then some information is logged and dippy waits five seconds. The reason for the wait is that there is a small race where the status of a service could be checked before its status is updated from what it was before the deploy restarted something. A small wait gives the Consul checks some extra time to catch up.
A shared Consul client is also created here that will be used throughout the rest of the method. It uses a non-pooled HTTP config because the default pooled one may leak idle connections if it's used temporarily like this one is.
//| id: Host.WaitUntilHealthy-spawn-goroutines
// group services to check by node, and then spin up a goroutine to loop to check each node
servicesByNode := make(map[string][]string)
for _, svc := range h.DeployConfig.ConsulChecks {
servicesByNode[svc.Node] = append(servicesByNode[svc.Node], svc.Name)
}
g, ctx := errgroup.WithContext(ctx)
for node, svcs := range servicesByNode {
g.Go(func() error {
<<Host.WaitUntilHealthy-watch-node>>
})
}Before I introduced microVMs to my lab, the next step was pretty simple: a loop to check for changes in node health. microVMs complicate this because they run their own Consul nodes, but are deployed as part of their host, not individually. When a host is deployed, waiting for it to be healthy means waiting for Consul checks across all of its microVMs as well.
The way dippy handles this is that the checks in the deployment config include both the service name and the node name. These checks are grouped into a list of services for each node, and then a goroutine is spawned for each node that has services to check.
//| id: Host.WaitUntilHealthy-watch-node
l := l.With("node", node)
var waitIndex uint64
var prevFailing []string
prevMissing := svcs
for {
<<Host.WaitUntilHealthy-query-checks>>
<<Host.WaitUntilHealthy-check-results>>
}Logging within each goroutine includes the name of the node being checked. Some variables used in the wait loop need to be set up beforehand. The wait index is used for querying Consul, so it will be explained shortly. The prevFailing and prevMissing variables are used to avoid repeating logging when nothing has changed. Each one is a list of service names, and the default state at the start of the loop is that all services being checked for the node are missing (not present in Consul).
//| id: Host.WaitUntilHealthy-query-checks
checks, err := backoff.Retry(ctx, func() (consulapi.HealthChecks, error) {
checks, meta, err := client.Health().Node(node, &consulapi.QueryOptions{WaitIndex: waitIndex})
if err != nil {
return nil, err
}
waitIndex = meta.LastIndex
return checks, nil
})
if err != nil {
return fmt.Errorf("checking node %q health in consul: %w", node, err)
}The first step for each iteration of the loop is to query the health of the node. This will return a list of all of the checks for services on that node. The wait index allows this to be done using long-polling: the request will only return a response when something has changed since the wait index that is passed in. Using this means we can get updates as soon as they happen without spamming the Consul agent with requests. Upon a successful response, we'll get a new last index, which we should use as the wait index for the next request.
It's possible for this request to fail intermittently during the deploy. In particular, deploys from a workstation access Consul through the ingress, so if that is being deployed, it may cause a temporary disruption. So this request is retried with backoff until it succeeds.
//| id: Host.WaitUntilHealthy-check-results
missing, failing := collectHealthResults(checks, svcs)
if !slices.Equal(missing, prevMissing) || !slices.Equal(failing, prevFailing) {
l.InfoContext(ctx, "health check results", "failing", failing, "missing", missing)
prevMissing = missing
prevFailing = failing
if len(missing) == 0 && len(failing) == 0 {
l.InfoContext(ctx, "node is healthy")
return nil
}
}The results from Consul are then gathered into two simple lists: one for services which are missing from Consul and one for services which are present in Consul but have failing checks. Both only include services that the host designated should be checked during the deploy. If either list has changed since the previous iteration of the loop, the new state is logged. If both lists are now empty, that means that all services to be checked are healthy, and the loop for this node can exit without error.
func collectHealthResults(checks consulapi.HealthChecks, svcs []string) (missing []string, failing []string) {
for _, svcName := range svcs {
var found, anyFailing bool
for _, check := range checks {
if check.ServiceName != svcName {
continue
}
found = true
if check.Status != consulapi.HealthPassing {
anyFailing = true
}
}
if !found {
missing = append(missing, svcName)
} else if anyFailing {
failing = append(failing, svcName)
}
}
return
}
The process for collecting the health results into the two lists shouldn't be too surprising. There might be more efficient ways to do it, but the n is small for these, so it's not worth it. For each service, it looks through all of the checks for any that belong to that service. It keeps track of whether any were found and also whether any were failing, and once it's looked at every check, it may add the service to one of the two lists.
//| id: Host.WaitUntilHealthy-wait
if err := g.Wait(); err != nil {
return err
}
h.log.InfoContext(ctx, "host is healthy")
return nilAfter spawning all of the goroutines for the different Consul nodes, dippy waits for them all to complete. If any of them fail, it's a failure of the whole operation, but hopefully with retries around the Consul request, that shouldn't generally happen. If there was no error, then all of the goroutines completed successfully, so every node is healthy and therefore the whole host can be considered healthy. The deploy can now proceed
//| file: packages/dippy/deploy/host.go package deploy import ( "bufio" "context" "encoding/json/v2" "errors" "fmt" "log/slog" "os" "path" "slices" "strings" "time" "git.midna.dev/mjm/nix-config/packages/dippy/cmd" "git.midna.dev/mjm/nix-config/packages/dippy/nix" "github.com/cenkalti/backoff/v7" consulapi "github.com/hashicorp/consul/api" "golang.org/x/crypto/ssh" "golang.org/x/sync/errgroup" ) <<Host>> <<Config>> <<NewHost>> <<Host.IsLocal>> <<Host.IsRemote>> <<Host.Execute>> <<Host.getRunner>> <<Host.Build>> <<Host.Push>> <<Host.PushToAttic>> <<Host.Diff>> <<Host.DiffLocal>> <<Host.CheckRebootNeeded>> <<Host.Deploy>> <<Host.apply>> <<Host.ApplyLocal>> <<Host.Reboot>> <<Host.getBootID>> <<Host.WaitUntilHealthy>> <<collectHealthResults>>
text/gemini;lang=en-USThis content has been proxied by September (UNKNO).