RFD 48 Control Plane Requirements

Automatically migrating storage resources in the case of apparent server failure: See the discussion in [_automatic_migration_of_resources], which describes several very different approaches. Which of these are customers likely to prefer?

Extra capacity vs. reduced availability: several sections below implicitly assume operators would prefer to keep extra capacity around (unused) than accept service outages. For example, the sections on control plane software update assume that operators have capacity available for live-migrating Instances to avoid an outage while a server is rebooted. Some operators might prefer to take the outage instead of having excess capacity. Are their customer requirements in this area? Can/should we encourage customers one way or the other? Should we provide a policy knob?

Broadly, the control plane must provide:

Some of the requirements (e.g., control over host power and console access for remote support) may have significant implications for both rack and control plane design, so it’s worth enumerating these in some detail.

Availability. Availability of the control plane refers to the property that requests to provision resources succeed when the underlying resources are available within the rack and requests to reconfigure or monitor resources succeed as long as they are well-formed. Unavailability refers to request failure due to hardware or software failure.

Durability. Along the same lines, resources created in the control plane are expected to be durable unless otherwise specified. That is, if the whole rack is powered off and on again ("cold start"), the system should converge to a point where all Instances, disks, and networking resources that were running before the power outage are available as they were from the user’s perspective before the event. Similarly, if a compute server is lost (either through graceful decommissioning or otherwise), it should be possible to resume service of resources that were running on that server (e.g., Instances, disks) on other servers in the rack. There may be additional constraints on how many servers can fail permanently before data is lost, but in no case should it be possible to permanently lose an Instance, disk, or other resource after the permanent failure of a single compute server.

Consistency. Generally, users can expect strong consistency for resources within some namespace. The bounds of the namespace for a particular resource may vary as described in [rfd24]. For example, if a user creates an Instance, another user with appropriate permissions should immediately see that Instance. In terms of CAP, the system is generally CP, with an emphasis on avoiding partitions through reliable software and hardware.

Scalability and performance. The API is designed with a scheme for naming and pagination that supports operating on arbitrarily large collections, so in principle it’s expected to support arbitrary numbers of most resources. In practice, the system is intended to support on the order of 100 servers in a rack and 10,000 VMs in a rack. While these numbers are unlikely to change drastically in the future, the long-term goal of providing a single view over multiple racks means the system will need to support much larger numbers of servers and other resources. To avoid catastrophic degradation in performance (to the point of unavailability) as the system is scaled, aggressive limits will be imposed on the numbers of most resources. Operators may choose to raise these limits but will be advised to test the system’s performance at the new scale.

Security. [rfd6] discusses control plane security in great detail.

Supportability and debuggability. Effective customer support includes rapidly diagnosing issues and releasing fixes with low-risk updates. To achieve this, all the software in the rack, including the control plane, must be built with supportability in mind, which means being able to collect enough information about failures to diagnose them from their first occurrence in the field as much as possible and being able to update software with low risk to the system. Details will be covered in an RFD to-be-named-later.

Our immediate goal is to build a control plane suitable for a single-rack deployment using a design that can be extended for multiple racks, AZs, and regions. [rfd24] discusses what it means to extend various API abstractions beyond a single rack. This is mostly beyond the scope of this RFD, but it’s worth mentioning a few key points here.

A key result of [rfd24] is that users (both developers and operators) will likely want to interact with multiple Oxide racks as a unit, whether that’s at the level of an availability zone, region, or fleet. To achieve this, we’ll eventually need to build API endpoints that aggregate resources provided by individual racks (e.g., "region API servers"), an interface for those endpoints to manage components within racks (either directly or via a rack-level API), and potentially other components as well. We could spend considerable time fleshing out what this looks like and even implement v1 of the rack using the same components that we’ll use to extend the control plane beyond one rack. However, we’ll defer as much of that work as possible, as it’s not strictly necessary for v1 and may well be invalidated by our experience building the single-rack control plane.

Because we expect that customers will want to be able to list Instances for multiple racks, we can infer that in a multi-rack system, the data associated with a given rack’s control plane must be replicated beyond that rack. Otherwise, an attempt to list Instances in the region when a single rack is offline would either fail (violating our availability requirement) or return incomplete results (violating our consistency requirement). This combination of requirements eliminates the approach of using a simple federation of autonomous racks.

There’s more on multi-rack support in the proposed architecture below.

As described above, the control plane must support automatic update of all of the Oxide-supported software components in the system, from the control plane itself to device firmware. For the control plane software, update mechanisms should use well-known best practices to maintain availability and minimize the risk associated with software updates:

Ideally, the software update process would be sufficiently reliable and non-impacting that operators could opt to have the system do it automatically without any input from them. In practice, operators are likely to want to control when updates happen to mitigate risk.

The software update mechanism must be aware of interfaces between loosely coupled components (i.e., components that are not upgraded as a unit) so that only compatible combinations of components are deployed. We should define policies for each of these interfaces that specifies what combinations of versions are supported (e.g., versions N-1, N, and N+1 in both directions).

For reference, here’s a review of several resources defined in the API, per [rfd4]:

For many API resources, there are necessarily multiple copies of the state. There is almost always some state in the data plane, and there almost certainly needs to be state stored elsewhere in the control plane as well. To take a few examples:

For each piece of state, we can ask several key questions to evaluate consistency, performance, and availability:

It’s worth keeping in mind that we can have different answers to these questions for different resources, and we can even have different answers for different data associated with a single resource.

Consider these properties of an Instance:

The id and name are important for being able to locate the rest of the metadata about an Instance. Names are also used to page through all of the Instances on the system. We almost certainly want these in a single database from which we can easily find a particular Instance or list a group of them having adjacent names, though this doesn’t mean the authoritative state has to be in that database.

The number-of-CPUs property is more interesting, so we’ll use that example to explore different approaches.

One possible principle would be to say that the running state (i.e., the state in the data plane) is the authoritative state of the system and everything in the control plane is essentially a cache. That means that if the user requested a change to the number of CPUs, we must change the actual running number of CPUs (i.e., the data plane state) synchronously with that request and the request can only complete successfully if we successfully update the data plane. Joyent’s Triton uses this approach, quite successfully, for many properties of VMs, and it has the advantage that changes made outside the control plane for whatever reason (intentionally or otherwise) can be made to work. However, this approach falls apart for any kind of distributed resource like most networking resources — see the example above about IP subnets.

Another possible principle would be to say that the control plane’s state is authoritative. Once changes are made there, they are propagated to the data plane. The question then becomes: are those changes propagated synchronously (such that failure to propagate them result in failing to make the change) or asynchronously? If synchronous, latency and availability may be slightly worse, since more systems must participate in order to complete a request successfully.

The major downsides of asynchronous propagation are that it allows the system to accumulate debt (in the form of future work), possibly without bound; and it introduces a divergence between the database state and the actual runtime state. What happens if the user requests more CPUs and the compute server is down? What state do they see in the API? What if they want to make some other change (or even another change to the number of CPUs) — is that based on the previous runtime state or the last state committed to the database?

If done carefully, asynchronous propagation has a major upside: it allows users to record their intent, putting the burden on the system to execute the requested operation and retry as needed to handle transient failures. The consistency questions can be resolved by first-classing this model: the API would expose a current state and an intended state. After all, users ultimately want to establish a particular state of the system, and the mechanics of the API operations required to do that (and handling the possible failures and the associated unwinding of changes) can be a massive effort. A model that lets users express intent and tells the system to make it reality can be a substantial improvement in user experience, as demonstrated by the success of tools like Terraform. In practice, though, building such a system is a considerable amount of work and makes for a more complex API and a more complex implementation. See the appendix below for details.

Some resources don’t correspond to infrastructure resources at all (like organizations, teams, users, ssh keys, etc.). The obvious choice is for these to live in the database, although some of these may also be associated with resources in foreign databases (e.g., LDAP) and we’ll need to better understand what’s authoritative and how differences are resolved.

Thinking about SSH keys as another example: in this case, a single API operation (adding an SSH key to a user’s account) may require propagation to many different compute servers, depending on how these keys are used. In that case, we likely would want to say that the keys are propagated asynchronously and establish a few guarantees: that the single copy in the control plane is authoritative and that propagation to the rest of the system will eventually complete. We’re essentially using asynchronous propagation here.

Whatever choice we make to ensure consistency of the control plane’s data, it’s conceivable that state could diverge. This can happen for a number of reasons:

The appropriate way to address divergence likely depends on the specific case: what data diverged, how, and why? Implementation and operational experience will likely inform policies here.

Where possible, we can prefer an approach that first-classes the idea that configuration divergence could happen. For example, imagine an update mechanism that functions by explicitly updating both a database and a cache. This has the property that if something else updates the database, the cache may be left stale indefinitely. Instead, the mechanism could work even in the normal case by just updating the database and one could either configure that in such a way that the cache is notified automatically or the cache could periodically poll for unreported changes. In this case, if the database were updated by some unexpected component, the system would find out quickly.

We can and should be rigorous in dealing with unexpected divergence — recording, debugging, and if necessary fixing cases that we didn’t expect.

Having concluded that the control plane will need some highly-available distributed database, there are a few obvious options:

Other options are possible as well.

Experience from various team members with a number of storage systems has demonstrated the value of systems that support logical replication as a first-class operation. When it’s possible to replicate chunks of the namespace to newly-created nodes (even if it takes a long time), a number of tasks are made much simpler:

A carefully designed replication mechanism can make all of these use cases possible with minimal downtime and low risk, preserving the option to rollback right up until a final switchover.