vm-proxy is a tool similar to socat to help proxy information between a host and a guest VM running on it.
func main() {
<<listen-to-src>>
<<spawn-next-command>>
<<systemd-ready>>
<<accept-loop>>
}
When run, vm-proxy mainly just listens for connections and accepts them in a loop.
src := os.Args[1]
l, err := listen(src)
if err != nil {
slog.Error("listening to source", "addr", src, "error", err)
os.Exit(1)
}
defer l.Close()The first argument to vm-proxy is the source address to listen on. Addresses can take three different forms:
All three of the styles of address are supported for both the source and destination. The details of how listening for each type of address works will be covered with the listen function later.
If there's an error listening, vm-proxy exits after logging the error. Otherwise, it defers closing the listener until the end of main and proceeds.
dst := os.Args[2]
rest := os.Args[3:]
if len(rest) > 0 {
startNextAndWait(rest)
}The destination address is passed as the second argument, but it won't be used until the accept loop later. Any arguments after the first two are interpreted as a command to run while the proxy server is running. This allows using vm-proxy in an execline sort of style. The details of this will be covered later.
daemon.SdNotify(false, daemon.SdNotifyReady)
slog.Info("listening for connections", "addr", l.Addr())Once the server is listening and the optional next command has been started, the proxy is ready. vm-proxy supports running with Type=notify in systemd, so it sends a notification if NOTIFY_SOCKET is set in the environment.
for {
s, err := l.Accept()
if err != nil {
slog.Error("accepting new connection", "error", err)
continue
}
go handleConn(s, dst)
}The accept loop is very simple: when a connection is available, it is accepted, and then a new goroutine is spawned to handle it. That goroutine will dial the destination and copy data from this connection to the destination connection, so the address of the destination must be passed along.
func listen(a string) (net.Listener, error) {
network, addr, found := strings.Cut(a, ":")
if !found {
return nil, fmt.Errorf("invalid address %q", a)
}
if network == "vsock" {
<<listen-vsock>>
}
os.Remove(addr)
return net.Listen("unix", addr)
}
To determine how to listen for connections, the address from the first argument to the command needs to be parsed. The network portion is separated from the rest, and if it's a Unix socket, then the listening is fairly straightforward. If there's an existing file at the socket path, it is removed, and then the Go "net" package is used to listen on that path. If a vsock address is given, the logic is a bit more involved:
cidOrPath, port, found := strings.Cut(addr, ":")
if !found {
return nil, fmt.Errorf("invalid address %q", a)
}VSOCK addresses for vm-proxy, as described above, come with two pieces of information separated by a colon, so these pieces are extracted from the address string.
var cid uint32
var err error
if cidOrPath == "" {
cid, err = vsock.ContextID()
if err != nil {
return nil, fmt.Errorf("getting local context ID: %w", err)
}
} else {
var ccid uint64
ccid, err = strconv.ParseUint(cidOrPath, 10, 32)
if err == nil {
cid = uint32(ccid)
}
}If the first portion of the address is a number, it will be interpreted as a context ID (CID), which is used to target a VSOCK connection at either the hypervisor, another process on the host, another process on the same machine, or a particular VM from the host. This portion can also be empty, in which case the local context ID will be inferred, allowing communication within the same machine.
After this block of code runs, either cid will be set to the desired CID, or err will contain an error from failing to parse the first portion of the address as a number.
if err == nil {
p, err := strconv.ParseUint(port, 10, 32)
if err != nil {
return nil, fmt.Errorf("invalid vsock address %q: %w", a, err)
}
return vsock.ListenContextID(cid, uint32(p), nil)
}If the first portion of the address was determined to be a CID and not a path, then the port number is parsed and a VSOCK listener is returned.
addr = cidOrPath + "_" + port
On the other hand, if the first portion was not a valid number, then it is interpreted as a file path. Some Hypervisors like Firecracker and cloud-hypervisor expose VSOCK connections on the host side through Unix domain sockets. When you start the VM, you provide a path for the socket. Then, a process on the host can listen on a Unix domain socket with that path followed by an underscore and the port number to listen on. When the guest dials a VSOCK port with the host CID, it will end up dialing this Unix domain socket.
On the listening side, this form of VSOCK address ends up being a shorthand for listening on a particular Unix socket. The port is appended to the socket path, and then the code falls through to the normal handling for Unix sockets described above.
func handleConn(s net.Conn, dst string) {
defer s.Close()
<<dial-destination>>
<<copy-src-to-dst>>
<<copy-dst-to-src>>
}
Handling a connection is simple at a high-level: first dial a new connection to the destination address for the proxy, and then read data from each end and write it to the other. Since back-and-forth communication is expected, each direction of copying needs to happen in parallel.
d, err := dial(dst)
if err != nil {
slog.Warn("connecting to destination", "addr", dst, "error", err)
return
}
defer d.Close()
slog.Info("beginning connection", "dest.addr", d.RemoteAddr().String(), "src.addr", s.RemoteAddr().String())The incoming source connection s is passed in to handleConn, but each incoming connection also needs its own outgoing connection d. The dial function will read the address passed in when the proxy is started and connect to it, returning the outgoing connection. Both s and d must be closed when this function exits.
func dial(a string) (net.Conn, error) {
network, addr, found := strings.Cut(a, ":")
if !found {
return nil, fmt.Errorf("invalid address %q", a)
}
var vsockPort string
if network == "vsock" {
<<dial-vsock>>
}
<<dial-unix>>
if vsockPort != "" {
<<connect-vsock>>
}
return c, nil
}
Dialing follows a somewhat similar structure to listening. The address is split into parts, VSOCK is handled a bit special, and then Unix sockets are handled. There is a little extra ceremony on this side, because the Firecracker style of doing VSOCK over UDS is handled differently in this direction, and requires a special handshake which will be described later.
cidOrPath, port, found := strings.Cut(addr, ":")
if !found {
return nil, fmt.Errorf("invalid address %q", a)
}
cid, err := strconv.ParseUint(cidOrPath, 10, 32)
if err == nil {
p, err := strconv.ParseUint(port, 10, 32)
if err != nil {
return nil, fmt.Errorf("invalid vsock address %s: %w", a, err)
}
return vsock.Dial(uint32(cid), uint32(p), nil)
}
addr = cidOrPath
vsockPort = portJust like when listening, for VSOCK addresses there is different handling if the first component is a CID or a path. There's no handling of an empty string for the CID here, but it could be added. If the CID parses as a valid number, then a normal VSOCK connection is made on the given CID and port. Otherwise, code will again fallthrough to the Unix socket code path. This time, the path does not get the port appended to it. Instead, the port is stored for later.
c, err := net.DialTimeout("unix", addr, 10*time.Second)
if err != nil {
return nil, err
}If the address was for a Unix socket or a VSOCK with a path, then a connection is made to the Unix socket at the path.
fmt.Fprintf(c, "CONNECT %s\n", vsockPort)
scan := bufio.NewScanner(c)
scan.Scan()
if !strings.HasPrefix(scan.Text(), "OK ") {
defer c.Close()
return nil, fmt.Errorf("unexpected response from vsock connect: %s", scan.Text())
}As mentioned above, a handshake is needed if this connection is actually for a host process connecting to a guest process listening on VSOCK. The host process must send a message of "CONNECT " followed by the port to connect to and then a newline. If connecting to the port succeeded, then the hypervisor will respond with a line that starts with "OK". If a different message is received, then it's treated as an error.
done := make(chan struct{})
go func() {
copyData(d, s)
done <- struct{}{}
}()The copyData function, which will be shown shortly, will block until the from stream is exhausted. Since copying needs to happen in two directions simultaneously, one of those directions must happen in its own goroutine, while the other can block the goroutine for handleConn. Arbitrarily, the copy from s to d is the one that will happen in a new goroutine. A channel is used to signal to the handleConn goroutine when this copy completes.
copyData(s, d)
<-done
slog.Info("ending connection", "dest.addr", d.RemoteAddr().String(), "src.addr", s.RemoteAddr().String())With the copy from s to d started, the other direction can happen directly in this goroutine. Once the copy from d to s completes, the handler waits for the other copy to also be complete before logging the end of the connection. This is the end of handleConn, so at this point the deferred statements from above will close both connections.
func copyData(to, from net.Conn) {
defer func() {
<<cleanup-copy-data>>
}()
<<get-to-raw-conn>>
<<copy-to-fd>>
}
If you know Go, you might think this copyData function isn't needed. io.Copy(to, from) should do the trick, right? In many situations, that's true, but here it ends up being slow because it gets implemented as a read/write loop, without taking advantage of any syscalls that might avoid copying data back-and-forth between kernel space and user space. While there are checks in the Go standard library to optimize io.Copy for various situations, none of the checks in Go cover vm-proxy's use cases. In particular, the fact that neither net.UnixConn nor vsock.Conn implement ReaderFrom or WriterTo defeat basically all automatic optimization here. Good performance is still attainable, but it requires jumping through some hoops.
Note that the optimizations mentioned here are only relevant for Linux. The other platforms Go supports either have less or different optimizations, and I'm only particularly concerned with running this on Linux, so I haven't looked very deeply into them.
toRaw, err := to.(syscall.Conn).SyscallConn()
if err != nil {
slog.Error("getting raw conn", "error", err)
return
}Both net.UnixConn and vsock.Conn implement the syscall.Conn interface, which allows getting a raw connection that can provide access to the underlying file descriptors for the connection. This is used to get the raw connection for the "to" side.
if err := toRaw.Write(func(fd uintptr) bool {
f := os.NewFile(fd, "writing")
io.Copy(f, from)
return true
}); err != nil {
slog.Error("copying data", "error", err)
}With the raw connection in hand, the Write method provides a file descriptor for the write side of the connection. The file descriptor is valid for at least the duration of the call of the function passed to Write. Within that function, the file descriptor can be inflated into a full os.File (the name passed to os.NewFile shouldn't matter, as there is not underlying file on disk, so it's just descriptive).
This is the magic: now instead of copying between two connection types that don't have optimized paths for copying, vm-proxy is now copying from a generic Reader to an os.File, which means it will use os.File's ReaderFrom implementation. That implementation is full of useful optimized paths, and one of them works for what vm-proxy needs.
First, it will check if the "from" side is actually another os.File, and will use the copy_file_range syscall if so. That one doesn't apply here, since the from side is still just a network connection. Even if it was turned into one in the same way the "to" side was, it couldn't use this path, since copy_file_range only works for regular files, which socket connections are not. So it will fall through to its next attempted optimization.
If the "from" side is a syscall.Conn and the connection is a stream type, not datagram, it will use the splice syscall to copy the data. As mentioned above, both relevant connection types implement syscall.Conn, so splicing is done here. The splice call allows copying data between the two file descriptors without copying the data between kernel and user space, which can produce a noticeable increase in bandwidth for the proxy.
closeRead(from) closeWrite(to) setDeadline(from, 10*time.Second) setDeadline(to, 10*time.Second)
After the copy is done, the appropriate sides of each connection are closed, and a ten second deadline is set on both connections. The deadline ensures that one side can't keep the connection open indefinitely: once one side has sent all of its data, the other has ten seconds to finish. As long as both sides are continuing to participate, there is no deadline and the data copying continues.
func startNextAndWait(args []string) {
<<start-command>>
<<wait-for-exit>>
}
vm-proxy supports an alternative behavior where instead of running indefinitely, it will spawn another command and run as long as that command runs. This essentially lets you wrap another command in one or more vm-proxy instances and run them all together without the need for a process supervisor.
cmd := exec.Command(args[0], args[1:]...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
slog.Error("starting next process", "argv", args, "error", err)
return
}The first step is of course to actually start the command. It inherits the three main file descriptors from vm-proxy. vm-proxy doesn't alter these, so this is passing them through from the parent process. The environment and working directory are also left unchanged.
go func() {
err := cmd.Wait()
if err == nil {
os.Exit(0)
} else {
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
os.Exit(exitErr.ExitCode())
} else {
os.Exit(1)
}
}
}()Once the command is started, a goroutine is used to wait for the command to exit in the background. Once it exits, vm-proxy will exit too. If the exit code of the command can be determined, vm-proxy will exit with that code as well.
main.go:
package main
import (
"bufio"
"errors"
"fmt"
"io"
"log/slog"
"net"
"os"
"os/exec"
"strconv"
"strings"
"syscall"
"time"
"github.com/coreos/go-systemd/v22/daemon"
"github.com/mdlayher/vsock"
)
<<main>>
<<listen>>
<<handleConn>>
<<dial>>
<<copyData>>
<<startNextAndWait>>
func closeRead(conn net.Conn) {
switch c := conn.(type) {
case *vsock.Conn:
_ = c.CloseRead()
case *net.UnixConn:
_ = c.CloseRead()
default:
_ = c.Close()
}
}
func closeWrite(conn net.Conn) {
switch c := conn.(type) {
case *vsock.Conn:
_ = c.CloseWrite()
case *net.UnixConn:
_ = c.CloseWrite()
default:
_ = c.Close()
}
}
func setDeadline(conn net.Conn, timeout time.Duration) {
_ = conn.SetDeadline(time.Now().Add(timeout))
}
text/gemini;lang=en-USThis content has been proxied by September (UNKNO).