RFD 357 External DNS in the MVP

As mentioned in [rfd57], during initial setup, the operator will delegate a DNS domain to the Oxide system. The system will use this domain to provide DNS names for the external API and console and potentially other services. The information provided during initial setup includes:

In terms of the number N: we suggest 3-10. Addresses are sometimes valuable resources, but too few addresses limits our ability to scale DNS horizontally and to survive transient failure.

Ideally, the customer will have set up the glue records for delegation in their own DNS servers before the Oxide rack arrives on-site. Setting this up is part of the critical path for provisioning the first VM, but it’s not blocked on having the rack on-site.

The Oxide system will provide authoritative DNS servers for the delegated domain. We’ll call these external DNS servers (as in: serving DNS about the Oxide system to external consumers, not servers that are external to the system).

Each of the IP addresses should have a DNS server, but in principle these could overlap. This would allow the customer to give us, say, 10 IP addresses for future horizontal scalability. But we could attach these 10 addresses to only three DNS servers if we feel that’s all we need.

Through the external DNS servers, the system will provide one DNS name for each Silo: a DNS name to be used for both the console and external API. Following the pattern in RFD 21, each Silo would get a dns_name that defaults to the name. Then we would generate DNS names $silo_dns_name.sys.$delegated_domain. So if the customer delegates oxide.corp.globex.example and creates Silo eng, the console and API would both be served from eng.sys.oxide.corp.globex.example. These are the DNS names that end users would use to reach the console and external API. The "sys" subdomain distinguishes these names from any names that might be provided under RFD 21 so that cookies would never be accidentally shared across domains.

This DNS name would have A/AAAA records containing IP addresses on the customer network that are used for Nexus instances serving the console and API. Allocating and managing these is beyond the scope of this RFD, but the assumption here is that these come from an IP pool designated by the operator for this purpose. The details of exactly how these records are assembled in response to a given query are to-be-determined. The server may select a subset of A/AAAA records and/or may randomize the order of returned results. These are common (if only weakly effective) techniques for load balancing when using typical DNS clients.^[1] Eventually we will probably also want the ability to configure which Nexus instances are in the pool of externally-accessible ones so that we can isolate instances for debugging or to scale down with the ability to roll back (scale back up) quickly.

Note that the only DNS names provided here are per-Silo. There is no domain proposed for the Oxide system as a whole. That’s partly because a goal of Silos is to virtualize the Oxide system.

To support reverse DNS, we should also serve PTR records for the customer-facing IPv4 and IPv6 addresses used for Nexus and the console. This can likely be punted to post-MVP.

The set of external IP addresses for the external DNS servers is fixed by the operator configuration described above. If the operator gives us 5 specific addresses for DNS, then those are the only 5 addresses we can use, and we should use all of them (lest clients see failures because they picked an address we weren’t using). But this does not mean there can be only a fixed number of DNS servers or that they have to be updated in-place. We can still use standard blue-green-style deployments to update these servers. To do this, we’d deploy a new external DNS server, get it up-to-date and verify it’s working, then move one of the fixed external IPs from one of the existing DNS servers that we want to decommission from that server to the new one. This still allows a low-risk rollback by moving the IP back to the old server.

Confidentiality of DNS requests and responses can be provided by things like DNS over HTTPS and DNS over TLS. We don’t believe this is a particular goal for our customer base in the MVP timeframe and we do not intend to implement it now.

Authentication of DNS data can potentially be provided by the same mechanisms or DNSSEC. In the MVP, we’re only providing external DNS for services on HTTP/TLS. TLS already provides end-to-end authentication of the service. We don’t intend to do anything else to authenticate DNS in the MVP timeframe.

The system will provide APIs to manage the private keys and certificates used for customer-facing services (i.e., the public API and the console). This is deliberately a bit vague. See [_certificate_management] for discussion.

In DNS, delegation is the mechanism by which the owner of some DNS domain (who runs DNS servers for that domain) delegates to someone else (generally running separate servers) authority over some subdomain. Suppose the Globex company wants to deploy an Oxide system for their developer teams. Let’s say they already use corp.globex.example for internal infrastructure and they run authoritative DNS servers for corp.globex.example. They can delegate the subdomain oxide.corp.globex.example to the Oxide system. This means:

A delegated domain is often called a zone. But zones are not synonymous with levels of the hierarchy. For example, globex.example, might be one zone containing names corp.globex.example, www.globex.example, and even cypress-creek.corp.globex.example. Or these might all be separate zones. This is up to the administrator of globex.example and the people to whom they delegate these zones.

To set up delegation like this, an administrator for the parent zone corp.globex.example creates two kinds of records:

This is important for us because it means that if we want the Oxide system to have control over some domain, and if we expect the customer want to pick parent domain, then they need to delegate a domain to us. And that means some work on their side to configure their own nameserver in order to set up the rack.

We’ve generally been assuming that separate Silos will have separate API/console DNS names. This is probably not strictly necessary. Here are the (arguably vague) reasons why this is preferred:

Note that just because Silos have different DNS names does not mean they’d have different IP addresses or Nexus instances.

It seems likely that any of these could be made to work fine.

Whatever scheme we use for assigning DNS names to our API and console service must account for how we will obtain trusted TLS certificates (and the corresponding private keys) for these services.

Our external DNS servers must have high availability, which is to say that they must survive at least one and probably multiple failures. The current design uses self-contained servers, so it’s pretty straightforward to deploy quite a few of them.

DNS is a distributed A-P system (in the CAP sense). It’s explicitly eventually consistent, meaning that two clients might see different views of the system at the same time, even if an individual change can eventually be expected to be observed by all clients. This is extremely helpful for implementing the server because it means we don’t need to ensure that Nexus propagates changes to external DNS atomically. We just need to ensure that we keep attempting to update them until we succeed.

Unfortunately, this doesn’t mean DNS is highly available. In an ideal world, any DNS domain would have N + M identical but independent nameservers where N are required to serve peak load, and M of them could fail without any impact to the availability of either DNS itself or any of the services using it (e.g., an HTTP service under this domain). This can be made to work and we intend to deploy things this way for internal service discovery ([rfd248]). However, it relies on specific behaviors from DNS clients that are not widely implemented (e.g., that they query more than one server or retry requests on failure). Unfortunately, the safer assumption is that a client will do the least useful thing (e.g., if there are five NS records, it will choose only one; that one will be the one that’s not up right now; it will wait for several seconds in some synchronous context; and it will not retry the request on failure).

The upshot for us is that we should generally run a working DNS server at every IP address for which there’s an NS record and we should try to minimize the time during which service at those IP addresses is unavailable. To this end, we’ll want the ability to dynamically move these IPs between DNS servers, and we should never remove one of these IPs from a working server unless we’re about to add it to another working server. Instead, the update process should deploy a new server, get it working, then move an IP address from an old server. With luck, this will not be a significant cause of unavailability in the MVP. Future enhancements could use short TTLs and maybe quickly stop advertising nameservers that we believe are offline, though it’s not clear how that will work given that we don’t control the customer’s glue records.

In terms of scale, we might consider:

The volume of DNS data is likely to be quite small — on the order of a few records per Silo, of which we expect most customers to deploy only a handful in the MVP. We could easily reach thousands of Silos before this was remotely a consideration.

Thanks to DNS’s eventual consistency (see above), read load is often easy to scale horizontally by providing more nameservers or extending TTLs (with some cost in latency-to-update the DNS data). We haven’t rigorously quantified the expected load, but we expect it to be pretty modest. Even hundreds of queries per second would reflect quite a lot of concurrent end users. Experience suggests that that level is easily handleable by a single threaded server storing hundreds of hostnames. If we need to scale out, we can deploy more servers.

The update load is very light. Database updates only happen when an externally-visible service changes IP addresses, which we’d expect only to happen during updates and maybe in response to unexpected failures. The expected volume would be much less than 1 per second.

These numbers may change dramatically post-MVP when we support the [RFD 21] instance DNS features. (See [_out_of_scope_for_mvp].) The total DNS data is not likely to be huge and the read load remains horizontally scalable. The mechanism of propagating updates is discussed in [rfd367].

We assume that where possible, the control plane will want to use standard cloud deployment practices like immutable infrastructure and blue-green deploys. This means that when updating Nexus, we’d deploy new Nexus nodes, verify that they’re functioning as expected, then bring them into service by giving them an IP on the external network and put them into external DNS, then remove old nodes by doing the reverse. To achieve this, we need direct control over the DNS data (it’s modified multiple times in this example).

It also seems pretty likely that we’ll want to expose more services in the future (e.g., an object store). By having the customer delegate a zone to the Oxide system, we preserve the flexibility to add more DNS names or change the DNS names we use for existing services (provided we can do so in a compatible way).

Similarly, we’ll want to be able to update the DNS servers themselves using blue-green deploys. This can be done in the same way as above for Nexus, provided that we have the ability to attach (and detach) the specific externally-facing nameserver IPs to (and from) the DNS servers.

Note that the number of addresses in-service doesn’t have to match the number of servers. We could conceivably support 12 addresses (for future horizontal scale) that are attached to only 4 instances (3 on each server, if that’s all we need for scalability and availability).

Per [rfd169], the console uses session cookies to authenticate requests that originate in the user’s web browser. All requests made by the browser on behalf of the console should have the session cookie so that the request can be authenticated. But whether a given cookie is included in a browser request depends on the DNS domain being requested, the DNS domain of the page that issued the request, and the attributes of the cookie. See "define where cookies are sent" in this Mozilla guide for a more detailed description.

As an example, if we were to serve the console and API from separate DNS names like console.my-silo.example and api.my-silo.example, the default behavior would be that cookies set by the server at one of these domains would not be visible by the other. We could use the domain attribute to cause cookies to be sent to all subdomains under my-silo.example, which might be okay because we control those, but it’s a broader scope than needed.

For now, we plan to put the API and console at the same DNS name, so it’s okay if our cookies are strictly available only to the exact same hostname.

While not part of this RFD’s determinations, it seems useful to document our expectations around networking for the external DNS servers and API servers.

Recall that [RFD 21] talks about a few kinds of external IP addresses. These external IPs all work broadly the same way: they’re reserved from an underlying IP pool when allocated and assigned to an Instance by having Nexus reach out to boundary services (Dendrite) to point that IP at the corresponding component (by updating the Tofino state table). The difference is around when allocation, assignment, unassignment, and deallocation happen. Of interest here are Ephemeral IPs (which are allocated and assigned when an Instance starts and unassigned and deallocated when an Instance stops) and Floating IPs (which are allocated, assigned, unassigned, and deallocated by explicit API operations, which means the allocation is independent of the lifetime of any particular Instance).

The external DNS servers must be hosted on the specific external IP addresses that have been designated for that purpose, since any change needs to be coordinated with changes to the customer’s DNS configuration. As a result, these addresses will function like Floating IPs: reserved when they’re configured (i.e., during initial setup and again only if an operator changes them) and assigned dynamically to whichever external DNS server(s) are in-service right now.

In principle, we have more flexibility with the external API servers, in that the set of IPs on which we host the external API can change as long as DNS reflects the current set of IPs. As a result, we could use something more like an Ephemeral IP, where we allocate and assign it whenever we want to bring a Nexus instance into service for the external API, and then unassign and unallocate it when we want to remove that Nexus instance from service. This may well be fine. However, the poor behavior of many DNS clients (see [_assumptions]) means we may prefer to reuse addresses when possible, in which case we may prefer to treat these like Floating IPs.

Another idea is that we still have a DNS zone as described above, but the customer is responsible for operating the authoritative DNS servers. The Oxide system could still provide external DNS servers. But the intent would be that they’re only used as the source for AXFR/IXFR-style zone transfers to customer-operated DNS servers. For better and worse, this approach shifts responsibility for operating the DNS servers to the customer. (Better because it’s potentially less work for us to scale and operationalize this; worse because it may be a worse product experience.)

It’s conceivable that we avoid any responsibility for operating external DNS and instead require that customers own this completely. This would considerably complicate the customer experience since they’d have to apply DNS updates whenever the Oxide system wants to modify DNS (e.g., during Nexus software update or to mitigate unavailability (planned or otherwise)). It would also complicate the process of upgrading Nexus because we’d either have to rely on operator action in the middle of the update or give up the benefits of controlling DNS. Finally, we’d also have to give up on providing the [RFD 21] features around providing DNS names for things like instances. This seems untenable.

The only upside here is that it could involve minimal development work and the customer could have more control over DNS (e.g., TTLs or the specific DNS names used). The major downsides are potentially much worse customer experience, more development work for software update, and potentially riskier software updates.

There are logically two externally-facing services:

These are logically pretty separate components. For the console, Nexus mostly just serves static assets. [rfd223] discusses alternative architectures where the web console could even use a different runtime environment.

Above we propose one DNS name $silo pointing to a service that serves both the console and API. We could instead separate this into two separate DNS names: console.$silo and api.$silo. For now, they could resolve to the same set of IP addresses — the Nexus external IP addresses — and we wouldn’t have to change much about Nexus. The main advantage is that it would be easier in the future to separate these components, since outside of Nexus they’d be treated as separate components anyway. This in turn would allow us to scale them independently, for example.

The disadvantage: there’s some complexity to deal with, like: which service do the login endpoints come from? Which one are the login cookies set for? In other words: if the advantage of this is that we could more easily separate these later, that only works because we’re essentially doing the work to separate them now. When we think of this as taking on more work in the MVP timeframe so that we have less to do later, for the particular future where we might want to separate these services, it doesn’t sound so appealing.

The proposal above suggests using identical components for internal DNS (see [rfd248]) and external DNS (this RFD). But they’d be deployed as separate fleets. We could instead deploy a single fleet of servers that operate different domains on different addresses. (That is, we operate the internal domain for queries arriving on the internal network and we operate the external domain for queries arriving on the external network.)

Given that we’re already sharing the software component, there’s not much advantage to this. Instead, by separating the two sets of servers, we get the ability to scale these services independently, plus separate security and fault domains. This way, a denial of service or compromise of the external DNS servers will not necessarily affect the internal ones. This is important, since the external servers are at higher risk (being exposed directly to the customer) and the internal ones are absolutely critical to the availability of the control plane.