RFD 26 Host Operating System & Hypervisor

We expect that customers will chiefly expect to use Linux and Windows guests in an Oxide Rack. At least initially, only operating systems that support UEFI booting will be supported.

Guests workloads will expect some minimum set of emulated deviced to be available in order to function properly. Beyond the normal x86 PC architecture devices (APIC, PIT, etc), we’ll need interfaces to block storage and network connectivity. The virtio device model has been popular and is (mostly) well supported by modern OSes. There are some questions around the quality of drivers available for Windows on that front.

For block storage, it would be advantageous to support API-driven hot insertion and hot removal of volumes from the guest. The PCI-centric device models (pci-virtio-block and NVMe) make this more challenging than something like virtio-scsi. It’s not clear to what extent such functionality is a requirement.

Most OSes utilize the x86 TSC for timekeeping, when it’s available. Windows is a notable standout here, in that it prefers the HPET. We may need to implement some portion of the Hyper-V paravirtualized device interface, so that Windows guests can use it (and the TSC) for time.

Under certain conditions, Linux guests are known to boot slowly as they wait for a slowly-filling entropy pool. It may be advantageous to provide a virtio-rng device for them to alleviate the dearth of boot time entropy.

Most access to the guest operating system will likely be through idiomatic remote access systems provided by the OS itself; e.g., SSH for Linux, and Remote Desktop Protocol (RDP) for Windows. As with a physical server, in the event that a guest is malfunctioning the user may want out-of-band access to the (virtual) console of the guest.

Console access may also be important when using operating system install media (e.g., ISO images) to create a virtual machine template image from scratch. In addition to console operation, the ability to "insert" an ISO file into an emulated CD-ROM drive or a raw disk image into an emulated USB drive may also be needed for this use case.

To make this work we’d need to emulate a frame buffer, and a workstation keyboard and mouse. Our guest firmware package will need support for providing those things to the guest OS. The emulated frame buffer would need to be able to be connected from the server to the remote user via some protocol like VNC or SPICE.

Nested virtualisation is not considered an essential feature at this time. While it may be useful for CI/CD workloads, and can be convenient for hypervisor development, it is challenging to emulate the underlying interfaces with flawless fidelity. That fact, combined with its dreadful performance for more production-facing workloads make it less attractive as a focus for the first product.

Live migration of virtual machines (commonly known as vMotion in VMware), where the state of a running VM is moved between physical machines without downtime or any other significant perceived impact, is considered a necessary feature of our product. It allows operators to migrate workload within the greater system to balance capacity and mitigate the impact of planned maintenance. The cost of software updates or hardware maintenance on a node are greatly reduced if the instances can be drained from it first.

Over the last 2+ years, a great deal of mistrust has been fostered over the ability (or lack thereof) of modern microprocessors to adequately isolate shared resources in the face of speculative execution and, more broadly, microarchitecture-focused attacks. Faced with these challenges, the OS must be flexible and provide the tools required (core-aware scheduling, vertical threads) in order to make a genuine attempt to mitigate those attacks.

Modern programming environments like Rust provide a high degree of memory safety, at least when used in their "safe" mode. As a result, a variety of common bugs in older software environments are generally avoided by construction. That said, bugs in the compiler or core libraries, or in "unsafe" code, can still result in exploitable defects. In addition, memory safety and even an advanced type system cannot completely preclude logic errors that exist in otherwise well-formed programs; e.g., a network service might incorrectly treat its input and allow a request to effect some unforeseen privilege escalation. As a result, we would like strong sandboxing facilities to allow us to balance the need for co-location of unrelated processes with the need for isolation between them.

As a minimum unit of isolation, the process model shared by all modern UNIX platforms affords us an isolated address space for a body of related code. Security critical components could conceivably be split into multiple separate processes which communicate through IPC mechanisms; e.g., a network server might handle incoming requests and forward instructions to a separate process for managing a virtual machine. Emulated guest requests for HTTP Metadata or DNS may be processed in yet another separate process. These processes should be able to opt out of certain privileged operations; e.g., the guest metadata service need not be able to fork new processes or read local files, while the virtual machine manager might not need to make any network requests at all. To make privilege separation between processes work well, we would need observable, robust IPC mechanisms with a reasonable performance profile.

Consider two different different classes of workload: first, a customer virtual machine; and second, a storage server program that forms part of our network block storage system. Each of these represent a different, but well-defined domain of security, functionality, and life cycle. These domains would benefit from being executed within an isolated partition where visibility of other domains is limited or completely obscured. From a life cycle perspective, the customer domain might be ephemeral, brought into existence by the control plane only while the VM is running on a particular server. In contrast, the network storage software is responsible for owning some amount of data, and will persist between server reboots.

In addition to limiting cross-domain visibility, we will likely want to use quotas or caps to limit the blast radius of a run away software component (e.g., one leaking memory) and provide more predictable performance for customer workloads. Classes of limit we may wish to consider include:

One level up from the process itself, the more abstract notion of a "service" may be composed of several related processes. A fault in any process may require a restart of all related processes. The supervisor will be responsible for restarting a software component that has failed, and may need to track and alarm on additional degraded states for software that appears to be failing persistently.

The supervisor should be able to track related processes, such as if a program creates a child process to handle some helper task or some set of requests. Child processes that form part of a unit should not be able to "escape" their relationship with other processes in a unit, even through acts like double forking.

The service supervisor will be another source of telemetry for the broader stack. Ideally it will report software faults with a similar set of structured information and through the same channels as faults in hardware devices.

It would help to have facilities in the operating system that enable robust deployment of the software artefacts that we layer on top. Some programs end up as a single broadly static binary, but others end up as a tree of files that must be deployed as a unit. A classical solution to the problem of deployment has been to use a software packaging system like dpkg/apt or rpm/yum. Management of those systems, especially at scale and with unattended operation, has proven challenging. Recent emerging approaches instead choose to build atomically deployable artefacts that comprise a prebuilt (and sometimes even immutable) file system image, containing all required and related files for a given service.

These modern approaches often make use of operating system facilities like containers that can serve both as a unit of isolation and as a unit of deployment. The deployable artefact is often much like an entire file system; e.g., we have heard that Google uses ext4 file system images mounted over an iSCSI transport from a central highly available storage service. Another previously implemented solution is to use ZFS send streams to reconstitute a child file system into an existing system pool.

As described earlier, at least some core internal workloads will need to live persistently on a particular server, and be started automatically as part of server boot. Such core services will need to exist in order to solve the chicken-and-egg problem of cold start of the rack.

It would be good to have a rich topology map within the server that covers all of the enumerable hardware, including:

Ideally, this topology information would be concentrated in one consistent interface, rather than a collection of disjoint ones. Dependable resource IDs could be derived from that for use on the local system and to properly annotate information that is collected at the cluster level.

It would be good to have a system that can track telemetry that may itself not rise to the level of an actionable fault, but where emerging trends may (in aggregate) represent a fault; e.g.,

As noted above, while some of these actions will be immediately actionable for the host OS, it is important that they also be made available to systems higher up in the stack so that reporting and decision-making at the cluster level will be possible.

The Fault Management Architecture (FMA) was introduced in Solaris 10 in 2005, and was subsequently inherited by illumos. It is a comprehensive system with several components, both in the kernel and in user mode, supporting the detection and reporting of both software and hardware faults. Events are correlated by diagnosis engines to produce a whole system view of fault conditions, with the potential for corrective action; e.g., activation of hot spare disks in a degraded ZFS pool, retirement of memory and CPUs, etc.

FMA provides kernel facilities for the management of faults in hardware devices; e.g.,

FMA stores informational telemetry (e.g., Ethernet link state changes) and hardware faults (e.g., correctable or uncorrectable DIMM errors) in a structured log. These log records are then processed by modules loaded by the fault management daemon (fmd(1M), FMDPRM). Some modules are diagnosis engines, and some serve other purposes such as forwarding telemetry records to remote hosts or fetching telemetry from subservient devices such as a service processor.

Coherent management of the health of the system requires a complete picture of the hardware topology. Modern computing systems are often composed of a complex tree or graph of buses and switches, with many interconnected devices cooperating to provide service. NVMe disks may be connected through a variety of PCI bridges, just as SAS and SATA disks may be attached via a complex topology of expanders or port multipliers. FMA provides a view of the hardware through libtopo, the topology library (a somewhat dated description appears in FMDPRM). The system is represented in different ways (e.g., logical topology and physical topology) via different trees or graphs of objects with a common property-based interface. Properties are exposed with serial and model numbers, and any other notable device-specific information such as link speed or disk capacity. Each device has a URL-like construct, the Fault Management Resource Identifier (FMRI) that uniquely identifies it, and is used in other parts of the stack as a way to concretely nominate particular devices.

FMA also provides a basis for the promotion of persistent soft faults into a hard fault using Soft Error Rate Discrimination (SERD). Broadly, one correctable memory error per week might be expected, but three per hour may be predictive of a fault. In addition to the general FM module interface, the Eversholt module and associated fault tree description language allow for the definition of various measurement engines and transformation of telemetry into diagnoses in a relatively declarative fashion.

The Linux ecosystem has a variety of point solutions to particular subsets of what FMA provides. Because the kernel project does not generally ship any associated daemons or libraries, they can only provide the portions of fault management that can or must be kernel-resident. One such subsystem is Error Detection And Correction (EDAC), a framework chiefly tasked with reporting memory errors and retiring pages. As a kernel-only component, this module does not seek to perform more in-depth diagnosis that might require, say, a persistent long term store of error telemetry, or the kind of classification that is best performed in a daemon. The focus of EDAC appears relatively narrow, and it is not clear that there is a broader fault management effort underway. Instead, the general state of the art outside DIMM faults appears to be liberal use of dmesg and the occasional non-fatal (!) oops for reporting serious errors.

Beyond the kernel, there are other projects like mcelog that seem to provide some user mode recording of more complex error conditions. The project is capable of decoding and logging the machine check exceptions from which it gets its name, as well as some set of memory and PCI errors. It even claims to support a level of predictive failure analysis. Platform support is not extremely broad; e.g., the project FAQ suggests that users of AMD Zen-era hardware should just stick to EDAC at least for now. The author of the project gave a talk in 2009 about a laudible goal of unified error reporting, but it seems that EDAC may perhaps have emerged as the plan of record and stalled other efforts.

On the topology front, there are a mixture of sources that one would need to tie together to provide a complete picture of the system. Some data is present in textual files within /proc and /sys, some of which appear to be documented as stable and some of which are treated as private implementation details that could presumably change with any new kernel release. Others are commands with textual (sometimes parseable) output; e.g., lshw, hwinfo, lspci from pciutils, lsusb from usbutils, dmidecode, lsblk from util-linux, hdparm, mii-tool from net-tools, ethtool, etc.

The illumos approach to fault management is by no means perfect. It no doubt has bugs, and will require work to adapt to the continuing evolution of computer hardware and software over time. With that in mind, it appears substantially more coherent and complete than the Linux ecosystem. The benefit of co-designed kernel and user mode components in one coherent whole is perhaps thrown into its sharpest relief with fault management, an area that requires participation right up and down the software stack.

Many software defects are difficult to reproduce, and inspection of the production system is required in order to determine the cause of the issue. In order to aid that exploration, debugging metadata must not be limited to special debug builds of the software. The production software must include enough metadata to enable runtime inspection of the state and execution of the system. Useful metadata for debugging can include:

It might seem like this is chiefly a concern at the application level, but our past experience has demonstrated how critical it is to extend this stance on complete visibility all the way down the stack. An application stack trace that stops at libc or some other system-provided library leaves us with a critical blind-spot in our exploration.

During the design of the system, there will be some set of metrics that we would like to collect that we can determine in advance. Examples of this class of metrics include:

For high-frequency events, and especially in modern vastly multi-core systems, efficient collection and aggregation of standing metrics like these is a critical part of making the system both high-performing and observable.

It would be convenient for the system to present as much static instrumentation as possible in a consistent, discoverable framework. When static metrics are presented in a self-describing structure, it can help with two broad use cases:

Static metrics are valuable when engineers can determine interesting values to collect and store in advance of a problem occurring. Unfortunately, many of the most pernicious software defects are not found or fixed by inspecting existing metrics.

Complex defects are often not easily reproducible; their presentation may depend on ambient and difficult to catalogue background activity that is unique to a specific customer site at a specific time. In order to chase these defects to root cause and make a fix, we must be able to ask questions about the dynamic state of the system and iteratively refine those questions as we hone in on the problem. New instrumentation should be able to be enabled on a running production system without updating or even restarting the software.

Human intervention in systems can lead to unforced errors, and even the most cautious engineers engaging in debugging exercises are not immune. Our instrumentation must be constructed to be safe to engage on a production system without fear of a crash, or data corruption, or serious and widely felt performance degradation. When not engaged, the machinery for instrumentation should have no impact on system operation or performance. In the limit, it should be safe to accidentally request the enabling of all possible system instrumentation without prolonged deleterious effect; it is better to report an error to the debugging engineer than to make the system unsafe.

Modern systems generally consist of a lot of software, performing operations at a dizzying rate. A naive firehose approach to system observability where post-processing is used to locate the subset of interesting events is unlikely to broadly useful. In order to have minimal impact on system performance, and to answer very specific questions, the instrumentation system should be able to efficiently and safely filter and aggregate relevant events in-situ; that is, to make the decision to record the event, and which data to record, at the time of the event.

Our hypervisor and other critical services, are likely to be implemented in a split between user processes and kernel modules. We will need instrumentation that can cleanly measure events that occur across more than one process, and which extend into the kernel.

In general, we expect to able to gather at least the following information:

In some cases it can also be desirable to allow instrumentation to take limited mutating actions against the system, such as invoke a data collector program once a particular sequence of trace events has been detected. In order to aid in the reproduction of races in multi-threaded software, it can be helpful to be able to inject small delays (hundreds of microseconds) at specific moments in order to widen a suspected race window. Facilities that perturb the system may present a trade-off in safety, and it’s possible we might want to be able to restrict these separately from general instrumentation in customer environments.

Defects are an inevitable part of shipping a software product. We should seek to delight our customers with a speedy resolution to reported issues, with as few repeated incidents as possible for each defect. As much as possible, we should aim to be able to fully debug an issue from the first reported incident. To enable us to do this, our system must record as much information as possible about the state of the system when the fault occurred.

To fully enable postmortem debugging, the system should provide rich facilities for the creation of core files in the face of serious faults. A core file will generally include the complete contents of process memory at the time of the fault, but it needs to include other information as well. A process has a number of open file descriptors that can represent a diverse set of resources beyond the process itself, and thus important context for debugging; e.g.,

The creation of a core file may have been discretionary; e.g., using a tool like gcore to create it on demand. If not discretionary, the system must record details about the cause of process termination; e.g., memory address and program counter for segmentation violations, or the source of a fatal signal if delivered from outside the process. We should also note the time, both in terms of observed wall clock time and monotonic seconds since boot. The more context we record, the more likely it is we will be able to debug the problem the first time it happens.

In addition to core files, we would like the system to be able to dump the entire contents of kernel memory, and optionally the memory of a specific selection of user processes, on panic of the system. Crash dumps provide a wealth of information and have been instrumental in debugging issues with kernel facilities such as device drivers and hypervisors. Crashes of an entire server are extremely disruptive, and are a class of fault that even advanced facilities like live migration are unable to paper over. We should seek to minimise the number of repeat incidents that take out an entire server.

While customers enjoy speedy resolution to the issues they report, they are presumably even happier with resolution of issues they did not have to discover themselves. For customers who are amenable to remote data collection by support organisations, we can use rich crash data collection facilities to provide an automated reporting service: core files and crash dumps can be reported to Oxide on creation for timely investigation.

As we engage with more customers and more deployed systems, managing the relationship between a specific core file and the software artefacts that produced it can be a challenge. It is important that core files include sufficient program text and metadata so that a debugger can make sense of them even if it is subsequently difficult to locate the exact input source code. Metadata may be in a format like DWARF format, or some other summarised format such as CTF, depending on binary size constraints and the toolchain that produced the binary.

Linux is demonstrably popular, and as such many of the open source components that we may choose for integration into the product are developed and tested first against Linux interfaces. Other platforms are often an after-thought for project maintainers, if they are even considered at all. Little needs to be said here: Linux distributions generally ship with a package set that contains sufficiently current versions of relevant open source software, and other software is generally easy to build; manufacturers are often directly involved in providing in-tree drivers for their hardware.

Building the product on top of illumos will mean we at Oxide are committing to some amount of engineering and advocacy work in the future to ensure continued support from relevant upstream projects. While the illumos community is smaller than the Linux community, we would by no means be the only participants.

Our early control plane database prototyping efforts have been underway using software packaged by the OmniOS project, along with software we have built ourselves. These bodies of software were relatively easy to build with only minor patches, including:

In addition to OmniOS, the pkgsrc project maintains cross-platform build recipes for a vast storehouse of open source software; some 20,000 packages are available for illumos systems. This project is a joint effort spear-headed by the NetBSD project.

The cornerstone of ensuring that modern software is easy to build is the continued availability of the runtimes and standard libraries that underpin that software. Support for illumos is in upstream stable versions, with regular CI/CD builds, of both the Rust and Go toolchains. Python, OpenJDK, and Node are also available. If a critical piece of software is not available natively and we cannot port it, it can likely be made to work within an emulation environment such as LX-branded zones or even bhyve.

With respect to drivers for hardware, it is difficult to predict how much work will be required before we have completed part selection. Some components will require custom driver work, such as the PCI-e interface for managing the switch. One notable example of a component where the illumos driver is actually maintained by the manufacturer themselves is Chelsio, who have had productive engagement with the illumos community in the past.