RFD 88 Chassis Management Responsibility Allocation

The SP will have full responsibility for running a thermal loop, in which temperature sensors and possibly other data sources will be polled and fan speeds adjusted. In addition, the SP may toggle simple throttling mechanisms such as THERMTRIP_L and APML-based memory bandwidth capping that are made available by the major onboard ASICs. The specific objectives (ΔT, temperature cap, etc.) of this function are outside the scope of this document; however, at a minimum, the SP will be responsible for maintaining all onboard device junction temperatures below the manufacturer’s absolute maximum rating. If this cannot be achieved, the SP will be responsible for shutting down the server’s or Sidecar’s main power rails. It is anticipated that if analogous functionality is required of the PSC’s SP, it will be limited to the PSC’s own thermal zone. The PSC’s SP may or may not have the ability to control whether each individual rectifier is supplying power (see [rfd121]); however, thermal management of the rectifiers is an autonomous function of the rectifiers themselves, not the PSC.

The SP may choose to set thresholds for alert delivery in sensors that support this, based on the thermal specifications of the monitored device, to rely on integrated fixed-threshold mechanisms provided by manufacturers, and/or its own internal thresholding driven by polled sensor data. This is an implementation detail of SP software to be determined based on available margin, detection latency requirements, anticipated speed of thermal excursions, number of data sources to be polled, polling frequency, and hardware resources available to perform polling and interrupt-driven external threshold detection. It is not the intent of this document to constrain implementation of the SP’s thermal loop in any way, only to enumerate the set of data sources to be made available for its use and any limitations on that use driven by the necessity of sharing those sources.

This task is allocated to the SP because it may be required to prevent damage to hardware even when the host or switch is not functioning properly, and when a defect in the host OS prevents host-based software from making forward progress. The design of Hubris, as described by [rfd41], lends itself well to rapid automatic recovery when defects or hardware errors prevent the SP from functioning as designed; while the host OS is also designed to recover automatically from detectable errors, not all errors and defects are detectable and recovery often takes sufficient time for a major thermal excursion to occur before proper control could be reestablished. Finally, as there is no other device on the Sidecar board designed for this purpose, parity suggests that allocating this function to the SP will simplify the Sidecar design by eliminating some of the environmental management discrete logic discussed in [rfd58] §4.1.2.

As a result of this allocation, SMBus sideband interfaces for all PCIe end devices (including those that are soldered to the baseboard) will be routed exclusively to the SP for temperature sensor reading. All additional discrete temperature sensors, such as those required for inlet air, will be routed to the SP for its exclusive use. PWM fan controllers and fan tachometers will be routed exclusively to the SP, as this forms the control portion of the thermal loop. The SP will also be responsible for fan presence and fault detection, since the host will not have direct access to these devices.

Serial Presence Detect (SPD) is a feature of DIMMs described in the [device-inventory] §1.2. The data stored in these is required by the memory controller (UMC) and associated phy microcontroller unit (PMU) to properly configure communication with the DIMMs during PSP boot. This requirement is architectural and imposed by AMD; it cannot be avoided in any reasonable way. There does not appear to be any means available to instruct the PSP to configure an I²C multi-master demux in order to access these devices; a mechanism is described by [amd-agesa] §4.1.7 for using a single-master mux that allows access by the host processor to multiple SPD banks, but not for describing the opposite mechanism for multi-master sharing of SPD bus segments.

In general, we assume that the DIMMs we use will incorporate the TSE2004 type part. Without this temperature sensor, it would be simple and obviously correct to assign responsibility for and attachment of these devices to the host processor. With it, however, the situation becomes more challenging: the on-DIMM sensor provides a more accurate indication of DIMM temperature than a separate sensor placed on the system board next to the connector. The SP is responsible for the thermal loop; while a modern DDR4 RDIMM is unlikely to be the first managed device to approach its maximum allowed temperature, we still want these sensor readings available. This gives us a few options:

We’ll discuss the merits of each of these options in turn, then conclude with a survey of existing designs known to us.

With one exception, all power supervision, sequencing, and conversion logic is the responsibility of the SP. All PMBus interfaces from power converters, onboard PMICs, and related devices, along with separate fault signals and associated discrete or integrated voltage, current, and power sensors will be routed to the SP (or to external logic under the SP’s control) for the purpose of monitoring and controlling power. The SP is also responsible for acting on excursions immediately, independent of fault diagnosis performed by the host or control plane software. That is not to imply that these errors needn’t be propagated to the control plane; they must, so that accurate diagnosis of transient or persistent faults can be made. The SP is not responsible for this diagnosis, only for the immediate response; latency associated with remote diagnosis is far too great to prevent damage in the case of power rails falling out of spec.

The SP does not play a direct role in regulation of the host processor’s VDDCR_CPU and VDDCR_SOC rails. Conversion and regulation of these rails is directed by the host processor itself via the SVI 2.0 protocol. Although it may be possible in principle for the SP to snoop or interpose on this channel between the host processor and the power regulators, doing so seems unnecessarily challenging and risky with no commensurate benefit. Instead, the SP may monitor and in a limited way control this subsystem via a separate out-of-band PMBus mechanism. In normal operation, the SP will function only as an observer; AMD’s architecture requires that the power conversion subsystem powering these rails act in accordance with the SVI 2.0 specification and the control commands issued by the processor. The SP is, however, ultimately responsible for ensuring that these rails are brought up or taken down in the appropriate sequence by enabling or disabling the power converters as required, and for shutting down these rails in the event of certain faults or power excursions that may be signalled by the host processor and/or the power converters and regulators.

On the PSC, power control functionality includes only the PSC’s own power domains; it is expected that the rectifiers and other shelf logic contain crowbars or other autonomous mechanisms for maintaining the power output to the bus bars within spec and shutting down if that cannot be achieved. Power monitoring on the PSC includes all power domains within the shelf that are visible to it, including the shelf output.

While the sideband signal routing discussed thus far may allow the SP to form at least a partial FRU inventory, we can see from [device-inventory] §1.4 and §2.2.2-3 that we cannot rely on sideband mechanisms to collect this information. Moreover, considerable error telemetry exists that is inherently available only to the host, including PCIe link state and protocol-specific transaction status. Absent a global interposer of the type contemplated by [rfd42], which is beyond our capabilities, there is no way for an out-of-band observer such as the SP to obtain this state directly. If we wished to aggregate this state at the SP, it would have to be collected and proxied by system software on the host processor. As we assume a diagnostic and recovery model driven by host software (with parts likely distributed within the larger control plane) rather than by the SP, we can eliminate software complexity by not implementing such a proxy.

The host will use in-band mechanisms to obtain FRUID and VPD from PCIe endpoint devices and DIMMs (SPD as noted previously), and will be responsible for collecting inventory of and error and performance telemetry from these devices. The managed devices included here are: M.2 and U.2 SSDs, NIC, the Tofino2 switch, and any AIC slots we may include in future designs. The host is best positioned by far to collect this data, which includes standard PCIe link and device status, AER, and device-specific status accessible only in-band. We do not specify here whether the host, higher-level control plane software, or both will be responsible for diagnosing faults from this telemetry stream and directing or taking appropriate action in response; the portions of the data flow diagrams included here that are downstream of the chassis management subsystem are intended only for representative purposes.

The host OS will be responsible for monitoring its own host processor (see [device-inventory] §1.1) using the large array of in-band mechanisms and for conveying this telemetry to the control plane. Should the host processor experience a catastrophic error, the SP may use the APML/SB-RMI interface to obtain error data and convey it to the control plane. As discussed previously, this interface is available on the same bus segment as the SB-TSI temperature monitoring endpoint, a function also assigned to the SP. This mechanism should be used only if the host processor is non-functional to avoid duplicate error reports.

The absence of a host processor is something of a special and unusual case: the onboard power sequencing logic (which may be implemented by the SP) should ensure that host power rails are not energised in the absence of a host processor, or in the presence of a mechanically but not electrically compatible host processor model. See [amd-ethanol] and [amd-daytona] for example implementations. Applying power to a Gimlet under these conditions indicates either a major fault or operator error, and this condition must be reported to the control plane by the SP.

The TBD services that fall under this architectural umbrella process mostly public data, such as identity information for DIMMs or SSDs. Other data falls into a gray area: power and thermal sensor data, error reports generated by hardware, fault diagnoses, and other device health data. This data is not by itself sensitive, though some customers may consider it so. The communication channels on Gimlet and Sidecar baseboards used for the collection of this data are not protected by cryptographically strong protocols, so this data can easily be spoofed, tampered with, and read by any attacker with intrusive physical access. Attack vectors requiring intrusive physical access are outside this product’s threat model; however, this method could be used to add hostile devices to the machine with access to these communication channels. This is discussed further in later sections; however, the lack of any truly sensitive data accessible in this manner makes this seem like a low risk possibility.

Of greater interest is high-resolution power dissipation data, which in rare instances has been used as a side channel for observing privileged processor state (see [cve-2020-8695]). Although most of our power monitoring infrastructure is not likely designed to provide sufficient resolution to enable this type of attack (in particular, power data is unlikely to be sampled by the SP at any greater frequency than 1 Hz), the PDN components used to supply the host processor are capable of advanced high-resolution monitoring that could conceivably be used for this purpose. Compromised SP software or a hostile I²C master on the same bus could conceivably be used to create an exfiltration path for other confidential data. This risk is not specific to the architecture this RFD proposes; no alternative allocation of responsibilities would mitigate it. However, this does suggest that owners of the hardware providing this service should assess this risk in greater detail. For our purposes, sampled power dissipation (and component voltage levels and current flows) is considered non-confidential. Likewise, thermal data, which makes a far less effective side channel and in any event is also likely to be sampled at low resolution, requires no special protection with respect to confidentiality.

The confidentiality requirements associated with error telemetry and device health are externally defined by the services that provide or process them. The service processor is not expected to process this data, most of which this RFD allocates to the host processor. Both the host processor and the SP are expected to employ datalinks to the control plane that provide strong protection of confidentiality and integrity; however, service owners must consider the risks associated with any type of error telemetry that may include customer data (e.g., cleartext sent to a self-encrypting storage device in a failed transaction). Local processing of this data on the host processor is out of scope.

Fault diagnoses are expected to be made either locally by system software running on the host or by control plane software. Although out of scope for this RFD, these diagnoses (current and historical) likely merit at least some protection. An attacker who can obtain these may be more easily able to introduce malicious hardware (for example, by sending a malicious replacement for a piece of hardware known to be faulty, or by impersonating a technician sent to replace a piece of faulty hardware). An attacker who can alter them could do the same, as well as causing various classes of denial of service attacks: against an individual machine, by incorrectly introducing a fault diagnosis of some critical component, or against an entire organisation by introducing numerous spurious diagnoses over a period of time, forcing operators to diagnose faults manually or replace large quantities of functional hardware. This type of attack could be economically damaging to both Oxide and the customer, depending upon its extent. Because these diagnoses are derived from a range of other telemetry, the ability to violate the integrity of these upstream sources could be used in a similar manner.

Availability is mostly out of scope here, because the part of the system that is of interest is inaccessible except by intrusive physical attacks that require removing the component from the rack with associated loss of power. The data that would normally be associated with these services do not exist and/or cannot be sampled in this condition, and control paths are unusable. The much greater loss of availability has already occurred. Altering the machine to prevent collection of data or use of control paths once it has been reinstalled is certainly possible, but if the objective is to cause denial of service this seems needlessly complicated. One exception may be anything that can cause a loss of thermal data, which could cause premature fan wear, overheating, and other misbehaviour. The risk of such attacks seems low, as does the impact, though an attacker could conceivably make the hot aisle hotter and cause a nuisance for on-site technicians. Loss of availability of power telemetry is only a nuisance; however, interference with signals required for sequencing or fault detection could lead to temporary or permanent failure of the hardware. None of these vulnerabilities are specific to the allocation of responsibilities discussed here.

The services that will eventually be defined elsewhere in accordance with the general architecture described here are primarily low level, with commodity hardware on one side and the control plane software on the other. Communication channels with the control plane in both local SP software and local host software, out of scope here, are assumed for our purposes to provide mutual authentication and integrity protection. In scope, then, is the fact that assigning an actor the responsibility for monitoring and controlling a particular data source or control point has two side effects that affect security:

Generally, the considerations here are not affected by which actor is assigned a particular chassis management responsibility. However, allocating a particular function to the SP or host processor does change somewhat the shape of the attack surface. Software running on the host processor is potentially vulnerable to guest escape attacks, initiated either by hostile users (e.g., a customer’s disgruntled employee) or by remotely-compromised guest software (a wide range of possible threat actors with arbitrary motives). Software running on the SP is not directly vulnerable to this threat; however, as our architecture assigns most of the low-level monitoring and control functions to the SP, it becomes exposed to a wider range of attacks initiated via the managed devices themselves. This type of attack generally relies on the ability to replace a managed device, or some component of one, with a hostile device. Such an attack may be carried out by a threat actor with invasive access to the machine and the opportunity to reinstall the component after tampering with it or by compromising the supply chain.

A slightly higher-risk vector involves the installation of an SSD containing a hostile VPD EEPROM and/or additional hostile I²C master(s). Although the impact of such an attack is limited by the segmentation of each VPD endpoint onto its own dedicated SMBus (see [device-inventory] §2.2), the opportunity still exists to confuse SP software with invalid data, lock up the bus and delay the SP’s ability to access temperature data, and possibly other I²C-based attacks on downstream infrastructure. If we instead allocated responsibility for SSD VPD reading exclusively to the host (whether in-band, out-of-band, or both), similar risks would exist with respect to the host processor’s SMBus peripheral and host software. Note that as the SP is not designed to protect the host from hostile PCIe endpoints, and the host necessarily accesses the SSDs using a PCIe link, the risks to the host associated with a hostile SSD are not specific to chassis management and cannot be mitigated by any architectural choice within the scope of this document. An interposer of the type proposed in [rfd42], likely incorporating additional logic and software, could mitigate these risks, but is beyond our ability to execute at this time. Such a device may be worth revisiting should capabilities increase or these risks be deemed excessive.

This is the only service that is actually described by this RFD, so we should consider its security properties in greater detail. Generally speaking, the relevant aspects have already been discussed elsewhere. This service is required by the host to boot and for DRAM to function properly. DIMMs are subject to hostile replacement by a technician with intrusive physical access or through a supply chain attack, and could incorporate I²C devices designed to attack the SP, host, or other adjacent devices (most notably via the DDR4 margining DACs if installed; see the next section). The SPD temperature sensors are also used by the SP’s thermal loop, so compromised devices could mislead the SP into wasting energy, wearing out fans prematurely, overheating the adjacent hot aisle, etc. These attacks are likely very low risk; a hostile DIMM offers far more exciting opportunities to compromise a wide range of data and processes on the host. In such an attack, the temperature sensor could conceivably be used as a very low bandwidth exfiltration vector. Also like the rest of the telemetry sources, a bogus SPD target could attack software elsewhere in the system that consumes its data, including control plane software and even front-end software. The complexity of such multi-phase, indirect attacks (not to mention the far more exciting prospects available to an attacker capable of satisfying the prerequisite) would seem to render these vectors low risk.

Corresponding attacks against the SPD proxy by the service processor make little sense, but such attacks may be possible by compromised SP software. Once again, an attacker capable of compromising the SP software seriously enough to undertake such an attack would almost certainly prefer other, more exciting targets. A simplistic attack, disabling the proxy, would cause the host to fail to boot; an attacker would need to reset the host processor to trigger this, so there is no obvious reason such an attacker would not simply hold the host processor in reset instead. A more interesting attack might involve providing subtly wrong DIMM identification data to the memory controller, inducing random spurious errors, reboots, and fault diagnoses. As with other tampering or spoofing attacks against telemetry sources, this type of attack could vary in scope from a minor nuisance limited to a small number of components in a single chassis to a major organisational denial of service attack involving costly downtime and unnecessary hardware replacement across an entire fleet. This is a general risk associated with any fault management system, and is specific to neither the proxy service nor its implementation on the SP.

We have several choices in how we approach DDR4 margining, which are not mutually exclusive. These choices are out of scope for this document and will not be made here; however, it is necessary to list them because some of them introduce potential attack vectors via components of the SPD proxy service discussed previously:

The LTC2629-1, as is customary among devices of this type, performs no authentication against the source of commands it receives via I²C. The address of each is also fixed by AMD’s firmware interface. By design, the DACs have the ability to source or sink enough current to completely override the static resistor network (if present) and set the corresponding Vref(CA) voltage level(s). Therefore, if the DACs are present, any master on this bus could trivially interfere with some or all of the host-DRAM interfaces, rendering memory inaccessible or causing errors at whatever rate the bus master chooses. At a minimum, therefore, the DACs constitute a denial-of-service vector if installed. The nature of the reference voltage mechanism makes permanent damage to hardware unlikely.

If we choose to perform margining during design validation (which seems only prudent), and we choose to use the firmware mechanism (currently AMD’s recommended path), we then face a tradeoff between adding substantial cost and introducing this vulnerability on the one hand; or depopulating them and losing the ability to verify margin on boards in the field (or in the factory, without a special build for sampling). Should we leave the DACs populated, available attack vectors would include:

While none of these attack vectors seem especially high risk, they may along with the substantial added cost tip the balance away from shipping production boards with DDR4 margining DACs populated.

There are a few recurring themes here:

Most of these attack vectors seem to be fairly low-risk, and the attacks would be complex to carry out. Nearly all attacks one could mount against these services would have the effect of denial of service, either locally or by inducing a widespread rash of false inventory data and fault diagnoses, which could be quite costly for Oxide and probably the customer. In a few instances hardware could be permanently damaged, though the risk of that outcome attributable to the choices made here is very low. Plausible threat actors and motivations are limited mainly by the likely goal of denying service, and are otherwise nonspecific.

Mitigations specific to these services that may be worthwhile to explore include:

Additional details are specific to the services governed but not described by this document and are necessarily out of scope.