RFD 605 Virtual Machine Identity and Attestation

Before an attestation can be appraised, its integrity, authenticity, and freshness must be verified. Attestations must be accompanied by a certificate chain that links the attestation signing key to a trusted PKI root. Establishing the authenticity of an attestation requires that we first verify that this cert chain is valid [rfc5280]. In most cases this cert chain will end with the last intermediate before the root signing key. The Oxide RoT signs attestations with a key certified by the Oxide platform identity PKI. RoT-Rs controlled by some other authority will use keys certified by their own PKI.

Once an attestor’s cert chain has been verified back to a trusted root we appraise the attestation’s integrity by verifying the signature over it. The leaf certificate in the chain is the attestation signing key (often called the [alias] key) and so we extract its public part to verify this signature. Attestations produced by the Oxide Platform RoT are signatures over the serialized measurement log concatenated with a caller provided nonce. Verifying this signature requires that the relying party reconstruct the message signed. By entangling a random nonce through the Qualifying Data, verifying the signature over the reconstructed message establishes both the freshness, and the integrity of the attested data.

After verifying the signature over the attestation, the measurements represented in the attestation must be appraised. To aid in the appraisal process for the Oxide platform RoT we generate [CoRIM] documents as part of our release process. A [CoRIM] document contains a collection of reference measurements that an appraiser can use to identify the measurements produced by an RoT.

Appraisal is the process used to ensure that the measurements from an attestation produced by a particular RoT-R meet some policy. How this policy manifests is at the discretion of the appraiser. At a minimum a reasonable appraiser will reject an attestation that includes measurements not present in the set of reference integrity measurements trusted by the appraiser.

We view appraisals based solely on reference measurements from a trusted authority as a minimum bar. With this data alone we can only deduce whether or not a platform is running the software intended by the manufacturer. When the source code for the relevant software is not available for review, this is often the most thorough analysis that we can provide.

If the source code is available (publicly or otherwise) and the build is [reproducible], a much deeper analysis is possible. Reproducible builds are intended to allow a relying party to reconstruct a release artifact exactly from source code. This crates a link associating a release of the source code with the artifact and the reference measurements from the [CoRIM]. A relying party that can analyze the source code to determine its trustworthiness is making a significantly more informed decision than one that must simply trust the assertions from a [CoRIM].

Reproducible builds can be difficult to get right for large / complex software. We have done significant work to make reproducing Hubris (RoT and SP) images trivial but additional work will be required for up-stack components. Attestations that include measurements of 3rd party software on the platform may not however provide the same level of transparency or reproducibility. We anticipate having to place significant trust in the reference measurements distributed for any proprietary software artifacts.

Capturing the digest of an executable is trivial when the executable is a file on disk. An instance in the rack is composed of many executable and configuration files. All of this data is contained within the boot disk image and so the digest of the boot disk image fits our definition. This is not a particularly elegant, or efficient solution, but it is does get us the information that we need.

Any number of instances can be launched from a single image so from the perspective of measured boot they are identical. While the digest of the boot disk acts as an identity for measured boot & attestation, the larger rack system has it’s own context. Instances are uniquely identified in the [Oxide-API] by UUIDs. When an instance is provisioned it is assigned a UUID that subsequent API calls use to address the resource. Our notion of identity must be linked to the instance UUID as a means to allow relying parties driving the rack through the API to associate the two.

Additional configuration information associated with an instance may be relevant to its identity as well. We will consider these on a case-by-case basis.

Each sled in an Oxide rack has an RoT implemented on an NXP LPC55 [rfd0007]. This MCU is programmed with an instance of Hubris [rfd0041] configured to act as both the root of trust for measurement (RoT-M / RTM) and the root of trust for reporting (RoT-R / RTR) [TCG-Glossary]. When a sled is manufactured it generates a platform identity (PlatformId) key internally. This key is certified by our platform identity PKI [rfd0303][rfd0387].

This RoT performs a DICE-like boot leveraging the LPC55 ROM’s DICE implementation. The RoT bootloader and early hubris bringup code capture measurements of the RoT software image. This is used to construct a key hierarchy certified by the PlatformId. Certificates for leaf nodes in this PKI include the digest of the RoT software image. This is the first measurement captured by the Oxide Platform RoT measured boot.

Once this RoT has booted fully the Attest task is launched. This task receives a key pair (the alias key in [DICE Layering Architecture]) and certificate chain rooted in the platform identity PKI. This task is responsible for measuring the SP [rfd0529] and providing an interface for the SP to report additional measurements (similar to TPM2_PCR_Extend). Future work will use this mechanism in the SP as a means to report a measurement of the [HBS] image.

The Attest task provides an attestation API not unlike the TPM2 [rfd497]. Through this API a challenger obtains the serialized measurement log, a certificate chain, and an attestation signed by the RoT. The certificate chain is used to verify the signature over the attestation which includes a digest of the serialized measurement log and a caller supplied nonce (to guarantee freshness).

In the new Turin line, AMD is providing roots of trust for both measurement and reporting in the PSP. The PSP provides two attestation mechanisms. The first is a Dice Protection Environment ([DPE]) that’s exposed to the host. This DPE collects a measurement of the first non-AMD / host software executed. This software is our [HBS], the same [HBS] we’re actively working to measure in the SP. Getting this measurement from two sources will allow us to correlate the information across two attestations. Enabling access to this DPE and integrating it into our measured boot implementation is a high priority and currently under development.

The PSP also hosts an [SPDM] responder that is intended to provide BMCs a mechanism for appraising the (AMD proprietary) code executed on the PSP. Due to this RoT reporting only measurements from AMD components and our inability to verify these measurements back to source code we believe integrating attestations from this RoT to be a lower priority.

The [helios] operating system (OS) is launched from the Host Phase 1 & 2 images. This OS supervises the execution of the Oxide control plane and is thus best suited to extend the measurement chain to include OS components like processes and zones. These measurements must be recorded by either an existing RoT or a new one dedicated to the task. The AMD DPE is intended as a building block for such an RoT and is much more flexible as a result. The currently recommended approach is to use the AMD DPE as the basis for a new RoT dedicated to the OS and residing within the Illumos kernel.

A subset of the measurements identified in Measurement Overview are marked as "dynamic". This is intended to convey per-instance configuration data that, if changed, will affect instance behavior. These must be handled with care such that an attestation describing the instance’s [TCB] is accurate and complete.

The propolis process is responsible for instance life cycle management and is best suited to collect a log describing the relevant configuration (boot disk, CPUID / device model etc, device state etc). Above all else, propolis must ensure that the measurement log it produces accurately describes the instance it manages.

The data that will be included in the measurement log produced by propolis is discussed in Identity. Elements like the instance UUID are communicated to propolis by the control plane as part of deploying instance. Capturing a digest of the boot disk is less trivial.

propolis will only report a digest for an instance boot disk if the disk is:

These restrictions are necessary in order for propolis to reliably report the boot disk digest with reasonable performance. The read-only requirement here enables propolis to calculate the digest of the boot disk in parallel to the instance booting. Until propolis has finished calculating this digest, it must respond to requests from the instance with an error indicating that the caller should try again.

The threats that necessitate these restrictions are discussed in Security Considerations.

When propolis boots an instance with a mutable boot disk, it can still report the UUID and other metadata accurately. Omitting the digest of the boot disk reduces the information available to appraisers, but the other metadata may still be useful. The implications of this choice for appraisers is discussed further in Security Considerations.

Unlike the Oxide or AMD RoTs, propolis does not have access to a key that it can used to sign attestations. Future work building on top of the AMD [DPE] may make this possible, but the urgency of our current situation requires a more immediate solution.

Our initial implementation is limited to attestations from the Oxide platform RoT. This RoT produces attestations that are a signature over the hubpack serialized measurement log concatenated with a caller supplied nonce as the Qualifying Data:

where hubpack(log_rot) is a function that produces the hubpack serialization of the RoT measurement log.

Acting as an RoT-R for each VM instance, propolis must record the digest of the instance boot disk & UUID. When the VM requests an attestation, propolis will serialize a JSON representation of the instance UUID & [boot disk] digest: JSON(log_vm). To bind these additional measurements to an attestation from the Oxide platform RoT, propolis will combine JSON(log_vm) with the qdata provided by the caller within the instance.

Assuming that the instance provides propolis with a 32 byte nonce then:

where: qdata_i is the qualifying data provided by the instance

propolis will then add its serialized measurement log to the qualifying data from the instance :

where:

propolis will then request an attestation from the Oxide RoT providing qdata_p as input. The Oxide platform RoT will respond to this challenge with an attestation:

The verification formula would then become:

Measured boot implementations are slowly gaining traction across the hardware ecosystem and our goal is to enable as many as is practical. We’ve already spoken to SSD vendors that claim to provide an SPDM responder as a means to enable attestation, but we’ve not yet been able to get details on how to enable or use this feature. As these products mature we will evaluate them for inclusion in our attestation and appraisal implementation.

How the VM communicates with the host to make these requests has significant ramifications for the scope and shape of this API and related ones. To date, the only communication between the hypervisor and the instance has been the existing NoCloud cloud-init static data ([rfd186]). Several use cases, including this one, motivate a richer channel:

These use cases all involve direct communication between the instance and the hypervisor. A second category of communication has the instance reaching network services that we may want to be accessible independent of the external network. These include NTP, DNS ([rfd21]), and even the Oxide API itself.

Finally, when looking at interface options we must also consider:

We consider three primary interfaces:

We start with the virtualized UART or other serial device. Whether this is the classic 16550-style UART or a virtio-console, the ultimate user interface is the same. The ability to communicate with a UART exists on most every operating system. In addition, because the UART exposes a device path (say in /dev), the instance can control which users and containers inside of it potentially have access.

The UART has two notable downsides. First, UARTs have no inherent framing: if you disconnect mid-operation and then reconnect, the other side cannot tell, and may still be processing an earlier request or response. This requires every client to implement the same framing mechanism. This downside is surmountable. Second, by default a UART can only be opened by a single process at a given time. While the framing format could allow for multiple in-flight requests, this increases the complexity of the framing mechanism.

From here, we turn to two related mechanisms: using IP based communication or vsocks. vsocks are similar to IP sockets: they have a source and destination port as well as source and destination client IDs, though for most use cases one can use the well known hypervisor client ID. By using connection-oriented sockets (SOCK_STREAM), we can easily handle multiple requests and deal with resumption. This does not eliminate the need for framing within the socket itself.

Next is the the question of how to deal with the guest’s ability to lock down the interface. In general there is no great way to do this, but there are two techniques, both requiring guest cooperation. The first is to require that requests can only be initiated from privileged ports. This requires that an application call bind() to bind to a local address. This is doable, but can quickly get complicated, as one can no longer rely on the operating system to assign ports automatically. That said, having the option is nice. The second approach is to inject a per-boot shared secret into the instance in a privileged form that is then required in these requests.

We don’t consider any of these approaches a requirement for our initial goal. The use of privileged ports is useful when we consider future cases such as an agent running in the instance that wants to receive event notifications.

Comparing the IP and vsock interfaces surfaces a few notable tradeoffs. IP sockets are much more common and have far more tooling, and other cloud providers have used IP-based sockets for similar purposes. These have led to numerous server-side request forgery issues ([AWS-SSRF]), though such challenges are not unique to IP-based sockets: if we opt for vsock, we still need to evaluate our APIs and designs against this class of threat, even if it may be considered harder to exploit.

The major difference between the IP and vsock interfaces is what they require from the guest and how network configuration fits in. Guests may be configured with IP interfaces supporting only IPv4, only IPv6, both, or neither. While many guests will end up with IP interfaces, it is possible to create a guest interface with none. It would be nice if access to metadata services and similar features still worked in that environment.

Today our environments have no notion of link-local address usage. Because public clouds don’t support broadcast or multicast traffic ([rfd9]), they never need true link-local semantics. Oxide, however, does have customers who care about multicast support ([rfd488]). It is unlikely that we’ll ever need a real use for link-local addresses in the product; a distinct advantage of vsocks, though, is that they do not interfere with traditional host-based networking or anything else one might want to do. Similarly, IPv6 defaults to assigning a link-local address to an interface while IPv4 does not. Because some of our longer-term goals want a guest to listen for events from the control plane, this makes the vsock interface slightly easier to use. We do not want to require a guest to listen on a specific address assigned from its VPC Subnet for this purpose; the guest should be free to use all of its ports however it desires.

Conversely, a major advantage of IP-based sockets is that they are easier to use and make requests against. Common utilities like curl and nc were designed for IP-based sockets and have no support for vsocks, which can make development and introspection harder. Windows also lacks an out-of-the-box driver for vsocks, though they are covered by the same driver package used for all virtio devices. Not all operating systems support this interface, which is a disadvantage.

The last question is ease of implementation. Both paths require a chunk of work. The IP route means developing a way to identify all of these requests in OPTE and route them to a zone for processing. While aspects of this logic are required to implement some of the services we’ve envisioned, such as internal NTP and DNS ([rfd21]), none of it exists yet. The vsock route requires adding a good deal of support to propolis. It’s hard to estimate the exact magnitude of either effort, but the propolis support seems likely to be the lower lift, and propolis requires this work regardless.

There are a lot of tradeoffs here. Our proposal is to go with vsocks, in part because of our time-to-market needs, because it stays out of the way of whatever the guest might want to do with networking, and because it enables future-looking paths that likewise avoid interfering with guest networking.

One important observation is that even if we select this path today, it can be part of the virtual machine definition in some form (e.g., part of the VM virtual platform definition [rfd505] or similar), allowing it to be changed and evolved over time.

Our choice of virtio-sockets (vsock) as the communication channel between the hypervisor and the instance means the protocol that we implement must work over a standard stream socket. The protocol that we build on top of this socket should be as simple as is practical. Efficiency is also important but the artifacts exchanged over this channel are relatively small.

To obtain an attestation the caller / instance must provide Qualifying Data to the VM Instance RoT. The data that it receives in response will include attestations, measurement logs, and cert chains from each relevant RoT on the platform. Our initial implementation will include the Oxide Platform RoT and the VM Instance RoT. The VM Instance RoT will not have a signing key and will instead entangle the measurements it collects with the attestation from the Oxide Platform RoT. The artifacts returned from the [VM Instance RoT] be limited to one cert chain, two measurement logs, and one attestation. The size of the cert chain will be an order of magnitude larger than the other artifacts, though it is still only single digit KiBs in size.

Binary protocols are generally more efficient than text but given the relatively small size of the data exchanged through this protocol efficiency is a secondary goal. Simplicity is far more important on this front and so we will encode all messages as JSON. Each message exchanged will be serialized as JSON with a new line terminating the message. This format is known as JSONL / [JSON-Lines].

The API exposed to an instance by the host must be able to capture attestations from the relevant RoTs. Initially this will include a single attestation from the existing Oxide platform RoT. Future work to add attestations from other RoTs (AMD, host OS etc) would ideally be backward compatible. Our initial release however will provide no such guarantees. Customers must be made aware of this limitation in our initial work and should plan for future changes.

Each RoT on the oxide platform must expose some interface for the platform to drive interactions. The API / interaction model that we layer over these interfaces and expose to VM instances must be compatible with these underlying APIs. As we build out this architecture over time it is possible that we’ll need to integrate an RoT with an incompatible / difficult API. At the time of writing however the RoTs that we know we will need to integrate with either support [SPDM] or the simple API exposed by the Oxide platform RoT.

Creating a binding between an instance’s UUID and a cryptographic key requires that each guest possess a key. This key will be as sensitive as the operations or access it authorizes. Misuse or loss of such a key can be damaging on a number of vectors, many of which depend on the resources protected by a relying party / access control mechanism. Cryptographic tokens like [CCID] devices and TPMs are commonly recommended as mitigations.

This problem is increasingly relevant for virtual machines, and virtual cryptographic tokens (vsmartcard , vTPM etc) are often provided in cloud environments as a mitigation. Oxide has not yet implemented such a mechanism for VMs due to relatively low customer interest. Our direction currently is to provide a binding between an instance’s identifying features (image digest, UUID) and a key generated and held by the client.

Though this is still a notional API we believe that the Oxide platform RoT will be the bottle neck in our initial implementation. As part of this work we’ve developed a mock implementation [vm-attest-proto] of the proposed [VM Instance RoT]. The tests in this implementation rely on an existing mock of the Oxide platform RoT. To support the collection of initial performance characteristics, the [attest-time] tool was developed to configure the mock VM instance RoT to talk to the Oxide platform RoT through the IPCC interface. This tool then makes repeated calls to one of the Oxide platform RoT commands emitting the duration as text.

Tests were carried out on gimlet hardware in one of our test environments. Data sets of 100000 samples were collected for each of the 3 functions in the API, on 4 different systems for comparison. The [m3vs] utility consumes these samples and calculates the min, max, mean, variance, and standard deviation from each. The results from each test follows:

The attest command was tested in the 4 gimelets attached to the dublin environment. The raw data can be obtained from [attest-time-dublin-data]. The following table provides a summary (values in microseconds / µs & rounded to the nerest whole unit):

The data above verifies a few expected characteristics:

The get_measurement_logs function is the fastest of the commands. This is expected for a few reasons:

The get_cert_chains function is the slowest of the commands. This is expected because cert chains aren’t small and it takes several messages to get the whole cert chain out of the RoT. This command is a good candidate for Optimization, likely though Caching.

Some of the less expected characteristics:

The extreme min & max for the attest function can most likely be attributed to the IPCC channel being busy. If we carve this data up into z-score bands we can see that all samples fall within 1 or 2 standard deviations from the mean. There are however a handfull of outliers in the 37th and 39th standard deviations from the mean. The code to do these calculations can be found in [sds-from-mean].

This appears to be an artifact of my test setup and the fact that remote access to these systems uses pilot to access the system console through the SP. My test script collected samples from the attest command first and so I believe several early samples are competing with my console traffic. I’m working to confirm this now.

The relatively high standard deviation in the get_cert_chains function is a bit surprising. This can probably be attributed to it being the only call that requires multiple messages exchanged with the RoT.

Our initial implementation will include no optimizations for calls through this API. Each call will cause the VM instance RoT to do some work and then make the required calls into the Oxide platform RoT. We do however believe that some optimizations may be necessary and so we explore some options here.

propolis must be capable of defending itself against misbehaving (intentionally or otherwise) VM instances. It must gracefully reject malformed requests from the associated instance while returning a sensible error message.

Denial of service through resource contention / exhaustion is another threat that we must consider. This can be mitigated through rate limiting use of the interface between propolis and the instance. propolis may rate limit this API in the future though our initial implementation will not. Customers should plan their use of this API to minimize contention (rate limiting within instances).

Our initial implementation will include an attestation from the Oxide Platform RoT. The measurements reflected in this attestation are limited to the RoT and SP software images. With this information we can identify when a platform has booted legitimate Oxide RoT and SP software.

There is however a lot of software executing between these components and the VM Instance RoT. Until our measurement chain contains the [TCB] for the instance, consumers of this API cannot reasonably asses their trustworthiness. This implies that these components must be trusted implicitly which is contrary to our goals. Between the initial release of this feature and when our measured boot implementation reflects all [TCB] elements the risk associated with this implicit trust must be assumed.

An example of such a risk is present in the VM Instance RoT. This RFD proposes that the propolis software take on the additional responsibility of generating a log structure describing the instance boot_disk (if it’s read-only), and its UUID, then binding this data to an attestation from the Oxide Platform RoT.

Assume for a moment that in doing this work we introduce a bug in propolis that causes it to misreport this information. Assume further that some time after release this bug is discovered, fixed, and a new release is made. Appraisers should, through the attestation appraisal process, be able to differentiate between old vulnerable releases and newer ones where the bug has been fixed. Without a measurement of propolis in the attestation, appraisers will be unable to do so. Threats like this can only be mitigated by continuing to fill in our measured boot implementation until it is complete.

For this system to behave correctly propolis must reliably report the data that it observes. There are a number of ways to configure which disk is used to boot an instance in the rack however and from the perspective of propolis, some of these configurations are less deterministic than others. To ensure that propolis VM Instance RoT is able to reliably report the digest of the disk booted by an instance we must impose some restrictions.

The API described in this RFD allows instances to obtain an attestation from the platform while binding some data held by the instance to it. The instance can use this API to challenge the platform for an attestation & appraise the results directly but the output of this process is of little use. By deploying an instance on to the rack, the instance has already been configured to place a high level of trust in the sleds that make up the rack (propolis controls it’s memory & execution).

This API can however be a building block that an instance can use to convince a remove challenger that it can be trusted. The construction of such a protocol is left as an exercise to the customer. Incorrect use of this API in building such a protocol is a risk for us and customers. This could result in the compromise of customer data, and reputational damage for Oxide.

Mitigating this threat requires that we describe, and possibly provide a reference implementation of a secure protocol that uses this API and the attestation data correctly. The customer driving this initial implementation has implemented a protocol that they believe is sufficiently secure for their use-case. Future work by Oxide should provide a reference implementation demonstrating proper use of this API in a remote attestation protocol.

Of specific concern are instances with mutable boot disks. The attestation produced by such instances will fail to report a digest of their boot disk. Without this information, the threat of impersonation and interposition attacks will always exist as the challenger cannot know wither or not some intermediate is proxying commands through a vulnerable instance. This can be partially mitigated if the challenger knows the expected UUID of the instance being appraised. We say this is a partial mitigation because this mitigation will fail if the instance assigned said UUID is in collusion with or can be manipulated into attesting to arbitrary data.