RFD 20
Host Bootstrap Software: Objectives
Sign in
committed

Host Bootstrap Software: Objectives

This RFD can be accessed by the following groups:[public]
RFD
20
Authors
Updated

Introduction

This document attempts to define requirements, goals, and non-goals for that portion of the software that executes on each server’s Host Processor cores from the time the cores complete their post-reset BIST until control is transferred to the host operating system. Software executing on other processors in the servers, whether or not part of the same package as the Host Processor(s), is excluded; however, interactions with these components are described as appropriate.

No implementation, whether using existing third-party software or otherwise, is proposed and no specific implementation should be assumed. Discussion of implementation options may be found in [rfd75] and [rfd241].

Note
Certain aspects of this material, most notably those related to trust extension and machine state during boot, have been superseded by [rfd241], [rfd215], and/or [rfd216]. While the objectives in this RFD are largely consistent with our implementation, readers are advised to consult that material for details.

Dependencies

Host Bootstrap Software (HBS) primarily consumes interfaces provided by the Service Processor (SP) and Host Processor (HP) and its peripherals, and primarily provides interfaces consumed by the Host Operating System (HOS). Because such interfaces are inherently common to multiple components, their definitions are extracted into separate documents effectively incorporated by reference into documents defining the producer(s) and consumer(s) of those interfaces.

HBS will interact with the HOS in accordance with a suitable protocol for trust extension and boot-time control handoff. Additionally, an implicit interface exists among HBS, machine components addressable by the HP, and HOS that relies on the configurable behaviour of those components that HBS may establish prior to handing off control. As an architectural principle, this configuration will be limited to the minimum necessary to accomplish these functions. These interfaces are described in greater detail in [rfd241] and [rfd215].

HP-HBS interfaces are largely defined by [amd64] and processor-specific documentation such as [amd-milan-ppr] and [amd-psp-boot]; however, those documents have not been written in a manner that separates clearly the processor architecture from the PC machine architecture, and a number of deviations from the specification are necessary to the construction of a modern machine suited for relevant applications. Where understood, this document will call out such deviations. A separate interface definition is not considered necessary because the interfaces provided by the Host Processor are not under our control in the current revision of the product.

HBS will interact with the Service Processor, described by [rfd8], which is expected to propagate boot policies from the rack controller and status/error information to the rack controller. The SP will also be responsible for upgrading the HBS and other system software to which the PSP and Host Processor require access prior to loading the HOS, and for extending trust to HBS in accordance with configured policy. HBS-SP interfaces are described in [rfd27] and subsequent refinements.

Definitions

The choice of the term "host bootstrap software" instead of "firmware" is a conscious one, for both technical and emotional reasons:

  1. Many components of the system, both visible and otherwise, contain or execute software. "Host" disambiguates the execution environment from others within the same server or rack. "Bootstrap" further disambiguates these functions from e.g. the host OS, hypervisor, or routines and data that may remain resident in memory after boot such as SMM support; the role of HBS in establishing such resident software and the general nature of that software are described here, but its architecture and implementation are not.

  2. "Software" is distinguished from "firmware" for the conceptual reason that, like the other software in our system, it can be modified and upgraded as part of a secure management strategy to correct defects, add features, or support unique customer requirements.

  3. "Software" is further distinguished from "firmware" in that the latter, especially but not uniquely in the PC market, carries negative connotations around quality, performance, code reuse, scope, manageability, observability, freedom, and security. Our reasoning here is similar to that in distinguishing our SP ([rfd8]) from a BMC.

Additional Oxide and industry-standard terminology and alphabet soup are defined in the [glossary]; universally understood terms such as PCIe are elided.

Purpose

The purpose of HBS is simple and straightforward. In accordance with owner-configured policy where available, HBS must:

  1. Load the HOS image from a storage device known to us in advance;

  2. Extend trust to that HOS if possible; and

  3. Transfer control to the HOS.

A more detailed discussion of these requirements is found below; we state here the high-level constraints that our implementation must satisfy, and our goals for it. It is important to understand that, in the PC architecture, these functions make up only a portion of the system firmware, which has many goals we do not share.

Constraints and Goals

  • Neither HBS nor the persistent state it creates are typically visible to user-level applications; however, it is required that HBS not take any action that breaks compatibility with correct application software except as explicitly defined for a compelling reason. Obvious exceptions aside, the expectation is that correct user-level (i.e., unprivileged) application software will produce the same results on an Oxide machine as on a PC with the same Host Processor(s) and an OS compatible with the Oxide HOS (if such exists).

  • An HBS image, along with any resident host software and data and software required for the operation of other processors (e.g., the PSP) must be freely redistributable, and the portions of it that are derived from first- or third-party open source software must be freely redistributable in source format. This is also the minimum requirement imposed by [ocposf] as of 1 March 2021 in order to use the Open Compute trademarks to market products; it is unlikely that this requirement is strong enough to unlock significant value, however, and neither AMD nor Intel appears to be on track to satisfy even the watered-down terms. Realistically, it is unlikely that either considers the ability to use the OCP trademarks sufficiently important to its business to alter behaviour.

  • A mechanism must be available to extend trust to the host OS image, and this mechanism must be controllable through a policy knob set via the SP. The range of possible policy options is likely to be similar to UEFI’s secure boot policies; however, we have somewhat different goals and requirements around this trust extension mechanism.

  • HBS shall be capable of loading software images from either of the following sources:

    • An M.2 NVMe device incorporated into the server at time of manufacture, or an authorized replacement therefor, provided that device has been initialised with data conforming to a specific documented pattern; or

    • The SP, via UART

    Booting from additional media types is a non-goal. Booting from arbitrary or external media will not be possible. This both simplifies HBS and reduces the attack surface.

  • Information regarding progress of the boot process must be available to the operator through some combination of the server console and status information propagated to the control plane software.

  • Boot failures must be communicated to the Service Processor for further propagation to the control plane software and operators, including where appropriate actionable details about the cause of the failure.

  • Transfer of control to the HOS must take no more time following power-on or warm reset than on commercially available PC machines of similar scale.

  • HBS shall support the following G- and S-states, as defined by ACPI:

    Table 1. Supported G- and S-States
    G-StateS-StateComment

    G0

    S0

    Normal execution

    G2

    S5

    Transient state between G3 and S0

    G3

    N/A

    Mechanical off, not reachable via host software action

    The following state transitions are supported: G0→G2, G2→G0, G3→G2. Transitions into G3 are invisible to host software and cannot be caused directly by it, but will occur if the power source or multiple power supplies have failed or been detached from the system or possibly under control of the SP or external logic; see [rfd81] for details. Note that S3 (suspend-to-RAM) and S4 (hibernate) are not supported by AMD’s family 0x17 and 0x19 enterprise processor line and are generally inappropriate to servers. Not implementing this functionality allows for dramatic simplification of HBS.

Basic Boot Sequence

The introduction of the Platform Security Processor (PSP, despite AMD’s recent attempt to rebrand it as the AMD Security Processor) brought the first significant change to the x86 bootstrap process since the 8086 was introduced by Intel in 1978. This description is a highly abbreviated version of the more detailed description of the bootstrap process found in [amd-psp-boot]. The PSP is an ARM Cortex-A5 located within the same package as the Zen cores, and is also referred to in some documentation as MP0. It has the highest level of access to all in-package resources and is the first core to come out of reset. When it does, the following sequence of operations takes place:

  1. The PSP, located within the same package as the Zen cores, loads a bootloader from mask ROM that reads the contents of one or two external ROMs (normally attached via SPI although [amd-milan-ppr] discusses LPC-attached storage as well). The external ROM contains additional firmware that executes on the PSP and other embedded cores, along with various metadata. This additional firmware, critically, initialises the onboard memory controller and configures DRAM. The processor does not support [g-car].

  2. The PSP loads the first portion (and, optionally, additional portions) of x86 software from the external ROM and places it in DRAM at a configurable address. It then writes values for rip, cs, and other critical registers into buffers normally used by the x86 cores for returning from CC6, and pokes them as if to resume from that state. Unfortunately, AMD did not see fit to record values of cr3 and descriptor tables that would start the core in 32-bit mode. This is described in [amd-psp-boot] §4.3.

  3. The OEM/IBV and AMD public keys incorporated into segments within the ROM are used to validate signatures on each of the loaded objects; the OEM key is used only to validate the first (and, optionally, subsequent) portions of software that will execute on the host. The AMD key is used to validate all others. A failure generally results in an error condition and boot failure.

  4. History is now replayed to bring the BSP into 32-bit protected mode.

  5. Boot software continues executing out of DRAM, optionally if desired loading additional software from ROMs or other sources if required.

Critically, the software used to initialise DRAM, for whatever reason the most highly guarded secrets of the entire computing world, no longer executes on the host processor. This should allow for the first time a 100% open source x86 HBS implementation. Of course, the source code to the software that runs on the PSP (and other internal processor cores) remains closed, and unfortunately no known mechanism exists for it to be signed by any key other than AMD’s. Additional background is available in [amd-agesa].

In this flow, the critical point of security policy enforcement is in the first stage of HBS, corresponding to the SEC phase of a UEFI implementation. It is here that we must decide whether and how to measure and validate software that will execute afterward, and whether to proceed if that validation fails. As in the UEFI SEC concept, this stub should be kept as small as possible to allow computer owners maximum freedom to replace bootstrap software as they see fit without needing to generate a key and have it signed by AMD.

Architectural Goals

[minnich20] asserts a LinuxBoot objective to "…​reduce complex, obscure firmware. Move functionality into kernel and userspace." We share this goal for HBS. Whenever possible, functionality is to be implemented in the HOS instead of HBS. Exceptions require compelling justification; i.e.,

  • Where required for HBS to perform its functions of loading, verifying, and transferring control to the HOS; or

  • To enforce security policies for which no other available mechanism adequately mitigates the identified threat; or

  • As mandated by hardware specifications not specific to the PC machine architecture.

The boundary between HBS and the HOS is arbitrary, and this architectural curiosity is discussed at some length in [rfd37].

Declarative Descriptors

[shuttleworth14] argues that "Declarative firmware that describes hardware linkages and dependencies but doesn’t include executable code is the best chance we have of real bottom-up security." While this is not sufficient, we agree with this architectural goal. Wholly consistent with and mutually supportive of our desire to move functionality into the HOS is the goal of eliminating as much as possible any resident executive not placed in memory and managed by the HOS. At best, such resident software is difficult and expensive to fix when bugs are found; at worst, it provides a ripe attack target and makes system behaviour impossible to reason about absent a high-speed logic analyser.

An example of the distinction between declarative and imperative software lies in the contrast between the Devicetree Standard that originated with OpenFirmware in the 1980s and the PC architecture’s ACPI tables stuffed with AML bytecode. To the extent that HBS must convey information to the HOS, it will be conveyed in a declarative manner rather than through executable software left resident in memory. The HOS is then free to subsume, copy, discard, or otherwise use this description as it sees fit. As discussed previously, it is our objective to limit this type of description to that which is unavoidable.

Other Goals

  • Transfer of control to the HOS should be extremely fast. For example, Coreboot can boot machines in several seconds that require minutes to boot via UEFI. As Coreboot typically loads a BIOS or UEFI payload that we will not require, even better performance should be possible, on par with Chromebooks using Coreboot+depthcharge.

  • All HBS software should be generally available open source and/or written by Oxide and made available under an open source license of our choosing. Note that this goal is broader than the redistribution requirement described above.

  • For the initial product, there is no requirement that the machine owner be able to boot software images not supplied by Oxide; however, the design should allow for mechanisms such as alternate signing keys or booting of unsigned/unverified images should those features become interesting later. These policy options are discussed further in [rfd27].

  • Control should be transferred to the HOS with the BSC and APs already in long mode with minimal additional assembly code required. The exact processor state will be defined in [rfd37] and subsequent refinements. A declarative enumeration of system state known to HBS and not otherwise readily obtainable by the HOS should be provided to the HOS via a documented standard format (ours or an existing one). This becomes mandatory to the extent that such state is non-discoverable (e.g., in a write-only register), variable (e.g., is not always fixed with respect to a discoverable hardware revision), and immutable (e.g., a write-once register).

Non-Goals

  • Compatibility with the PC machine architecture is not a goal, and it is not expected that any OS designed for that architecture will function without modification. Deviations from the PC architecture are expected to deliver value and will not be made solely for arbitrary or aesthetic reasons.

  • As a corollary, there is no requirement that HBS be able to boot anything other than software designed by Oxide to run on these systems. Normally this includes the HOS but potentially also diagnostic or other software if required to support manufacturing, field service, and other identified needs. Many if not all of these functions may be possible to implement as applications within the usual HOS, but we leave open the possibility of developing additional standalone test or diagnostic programs (such as automated burn-in tests and perhaps even interactive diagnostics for use by a repair depot) independent of the HOS if required. If such applications exist, HBS will boot them as and when directed by the SP. Host Operating Systems and other directly-loaded applications requiring a VGA-capable console device will never be supported.

  • Booting from arbitrary devices and/or filesystems is not a goal, and in fact will be disabled by policy if otherwise possible due to the likelihood of customer confusion as well as expanding the attack surface. In particular, storage devices used for customer/instance data in as described by [rfd60] could be accessed by HBS in the same manner as the HOS boot devices, and could therefore supply an HOS image; however, HBS must never access these devices for any reason.

  • Interactive booting will not be supported, except possibly in a debug mode not intended for direct customer use. Selection of boot device and trust-related policies will be established through the control plane (see [rfd49] and [rfd61]) and communicated via the SP; errors and diagnostic information will likewise be propagated to the operator via the SP. There will be no timeouts in the boot process waiting on input from a human.

  • AMD Rome and Milan CPUs do not support S3; however, even on Host Processors that otherwise support such S-states, HBS will not support sleep or resume from sleep. G- and S-states not described above will not be supported. Sleep states are for laptops.

Glossary

ACPI

Advanced Configuration and Power Interface, a legacy standard defining system and processor power states, a mixed declarative-imperative firmware interface via AML, and a variety of other platform software functionality. ACPI is discussed in detail below.

AGESA

AMD Generic Encapsulated Software Architecture, a hybrid library/framework intended for use by platform software architects. This artifact is discussed in detail below and in greater detail in [amd-agesa].

AML

ACPI Machine Language, a Turing-complete interpreted language used to supply and consume [g-acpi] tables. See the discussion of [g-acpi] below.

AP

Application Processor, defined by [amd64] as every processor core other than the BSC.

BSC

BootStrap Core, defined by [amd64] as the core designated to execute the vast majority of bootstrap software following a system reset. Also called the BSP in some software and by Intel.

BIST

Built-In Self-Test, a set of hardware tests run by a processor after reset is deasserted but before the first instruction is executed.

BMC

Baseboard Management Controller, a processor and adjacent logic present on a system board or occasionally a removable daughtercard that interposes on Host Processor and system bus functionality to provide S5 manageability, implement legacy devices such as VGA, and provide environmental monitoring and other functionality. See [rfd8] for a discussion of this legacy system.

CAR

Cache-as-RAM, a mechanism in which the processor’s onboard cache is configured to preclude writeback to DRAM or other eviction. This is used early in boot on Intel and older AMD processors to allow DRAM initialisation software the use of a (limited) stack, so that it can be written in a language other than assembly. Modern AMD processors do not support CAR.

Host Processor

The components of the main processor package(s) on the system board that provide customer-visible computation resources, in our design the x86 core complexes and logic visible to them. See [rfd12].

HOS

Host Operating System, the operating system running on the [g-host-processor](s). See [rfd26].

LPC

Low Pin Count, an interconnect used to carry legacy ISA signals to legacy peripherals (most commonly emulated) and occasionally for other purposes. LPC ROMs and TPMs are common, as are LPC slaves in [g-bmc]s attached to logic that emulates standard devices like 16550 UARTs.

SMM

System Management Mode, an x86 processor mode in which code set up by HBS or other software executes without HOS visibility. See [amd64] vol. 2 §10. SMM is entered via a System Management Interrupt (SMI), the means of triggering which are implementation-specific.

PSP

Platform Security Processor, an ARM Cortex-A5 located on the IOD within AMD Zen/Zen+/Zen2 family processors. This processor is responsible for initialising the SoC prior to starting the [g-host-processor] and provides fTPM, debug unlocking, and certain other services afterward.

Server

As defined in [RFD 24] §2.1. For software purposes, a server is the largest unit in an Oxide installation that presents a single system image.

SP

Service Processor. See [rfd8]. The interface between HBS and the SP is defined in [rfd27].

SPI

Serial Peripheral Interface, a de facto standard low-speed serial bus. x86 SoCs typically provide one or more SPI masters used to load system software such as HBS from SEEPROM (NOR flash) devices, though the bus has numerous other applications.

External References

Oxide RFDs

Standards and Specifications

  • [amd64] Advanced Micro Devices. AMD Architecture Programmer’s Manual. 2019.

  • [amd-agesa] Advanced Micro Devices. AMD Generic Encapsulated Software Architecture (AGESA™) Interface Specification (55483), revision 1.42. 2020. Distributed only under NDA.

  • [amd-milan-ppr] Advanced Micro Devices. Preliminary Processor Programming Reference (PPR) for AMD Family 19h Model 00h, Revision A0 Processors. Publication number 55898, revision 0.75. 2020. Distributed only under NDA.

  • [amd-psp-boot] Advanced Micro Devices. AMD Platform Security Processor BIOS Architecture Design Guide for AMD Family 17h Processors (55758), revision 1.09. 2020. Distributed only under NDA.

Other Sources