RFD 63 Network Architecture

In RFD 21 User Networking API we introduce a number of high level concepts and API entities such as VPCs, Internet Gateways, VPC Routing, VPC Peering, VPC Firewalls, and more.

The fundamental idea is that each VPC is built using an overlay network with its own virtual network identifier using Geneve. The implementation of VPC Subnets, VPC Routing Tables, VPC Firewalls, Internet Gateways, VPC Peering, the instance’s DHCP and more is done for the most part by the [sect-opte], which is a programmable software component that allows us to run a series of transformations on packets and uses the [sect-onds] to know what the set of rules to apply is.

Ephemeral and Floating IPs, along with outbound NAT and Internet gateways, are built leveraging a similar virtual network and [sect-opte]. In this case, a dedicated virtual network is reserved for use with the Boundary Services which takes care of translating data between the encapsulated format and a form appropriate for the underlying network.

Finally, pieces like DNS are implemented by the broader network services with some assistance from [sect-opte].

Notably, this network architecture, purposefully doesn’t solve the problem of trust. It leaves the network, broadly speaking as an entity that will forward most traffic and leaves it to the entities that are generating and receiving it to determine whether or not they should act upon it.

XXX There’s probably more

Presume the following example diagram of a pair of servers in the rack connected to each switch.

Here, let’s walk through what happens if SERV5 needs to send traffic to SERV0. Traffic is being sent from one of the service IPs in site:12_05::/64 to site:12_00::/64. The rack identifier and server each have 8 bits in their part of the IPv6 address, the distinction between the two is indicated with the _ (underscore) character. See the address breakdown in the above image. It’s also worth nothing that site here is an IPv6 block that’s only routable in the context of the Oxide environment itself. That is, we use [sect-bs] to reach the outside world that way it doesn’t matter if our customers are using IPv6, IPv4, or both (and hopefully not neither!).

If we take the previous example, let’s extend it and say we had another rack in the same site:

Here let’s discuss what would happen if SERV5 in rack 12 was trying to send to SERV1 in rack ab. The steps below are somewhat abbreviated. The switches in each rack are fully connected. That is each switch in rack 12 is connected to each switch in rack ab and vice versa, but the two switches with a rack are not connected.

Presume that an instance on the VPC Subnet represented by 192.168.1.0/24 is looking up the DNS name for another instance on its VPC Subnet named db whether this is through someone running an explicit command like dig or host or a side-effect of using a tool like ping, curl, or something else. The following steps would roughly happen:

Assume that we have two instances, 'test' and 'web' that are on two different servers, 'A' and 'B'. These are part of the same VPC and project. If one were to try and connect to an HTTP server between the two, then here’s an example of what would happen. Let’s assume someone on 'test' ran something like curl http://web/index.html.

The first step would be DNS resolution. For that, see [flow-dns] above. Let’s assume that has been completed and we got that the IP address of 'web' was 10.0.0.88. With that in hand, the following steps might occur:

A common sanity check that someone might do is pinging an external resource, e.g. ping 8.8.8.8. If we assume an instance in a default configuration, it will have an Internet Gateway configured that causes all traffic not in the VPC network to be subject to a NAT and reach the outside world. This flow leverages Boundary Services.

In this example, presume that a user has set up a floating IP with the address 1.2.3.4 and assigned it to their instance through the API previously. Presume for the moment that there is an HTTP server running on the instance the floating IP is mapped to. In our customer’s broader network, Boundary Services will be advertised as the routing target via bgp. With that in mind, here are the approximate steps that would occur:

Microsoft has a trio of networking papers that describe the design and evolution of their virtual networking features which are in the VL2, Ananta, and VFP papers.

The VL2 paper introduced (not for the first time) and popularized a scheme for what could be seen as overlay networks. It discusses the use of equal-cost multi-pathing and of limiting broadcast domain size by giving services a service address that is translated into the actual host that is running that service by a directory server.

This was used as a basis in Ananta where Microsoft put together a scheme for doing NAT and general non-terminating load balancing (that is load balancing that does not accept a connection and open a new one to a backend). This was brought to a head in the VFP paper, which describes the full network virtualization solution that Microsoft has.

There are a couple of highlights from their approaches and things that apply to us:

Google’s Andromeda paper provides an interesting discussion of the evolution of GCP’s network virtualization implementation and details into it. There is a lot of wisdom to draw from the evolution of Andromeda. Off the bat, there are similarities to what we see in [past-msft] and [past-fab]. There is a control plane that is distinct from the data plane. These two environments have different requirements and while they’re integrated the way and means that they scale and evolve is naturally different.

An important aspect that is focused upon in Andromeda is the ability to have agility. The ability to change out the software, transfer state, and do so in a small amount of time is needed not just for upgrade but also for live migration. An important takeaway is that whatever we design, we’re going to want to think carefully through how we can make iteration and the deployment path easy for customers. While deploying software on-premises is often more involved than in a managed environment, the more seamless we can make the experience, the more willing our customers will be to take and roll out upgrades.

An interesting consequence of this design is that it allows them to do migrations between arbitrary clusters, which is a useful ability.

One of the things that they highlight was the evolutionary path that led them to adopt what they call 'hoverboards'. A fundamental problem of the data plane that acts on flows and connections is how state is pushed out to it. The idea behind the 'hoverboard' is that being able to spin up VMs in the broader datacenter that can handle this general processing is a good idea and keeps the primary data path simple. Instead, based on specific usage, data flows are programmed back into the host.

An interesting thing they also point out is dedicated 'co-processor' threads which are there to do CPU intensive packet operations. The most notable of which is encryption, though apparently this only occurs when going out to the WAN or when transiting between clusters in GCP. One realization from that is that it suggests most traffic inside of a cluster is not encrypted.

The use of 'hoverboards' and the 'co-processors' is somewhat different from what we see in VFP from Microsoft. In VFP hot-path flows were programmed into specialized NIC hardware, where as here we have specialized flows actually being programmed in the host. Both approaches are useful and in fact, the Andromeda paper goes through a number of different discussions of other techniques one could use and how they ended up with this one.

From my perspective, we don’t necessarily have the spare capacity at our initial small rack-level scale to be able to dedicate specific VMs that we can provision based on demand. While the evolutions that both Microsoft and Google have made towards dedicated special-purpose hardware and being able to leverage more of the fleet, because we need to deliver an initial product and our initial scale will be small, we should choose to learn from both of them and that the initial designs did work for smaller scales.

A lot of the lessons that they’ve learned from this can help us as we try to look beyond the initial product and ask ourselves how do we build environments for customers that scale beyond just a handful of racks. What does it mean when even just the thought of 100 racks is a drop in the bucket. Like with the [past-msft] work, having these papers is an incredibly useful resource to draw upon, which can’t be understated.

In the beginning of 2014 and into delivery in early 2015, a scheme for doing network virtualization and providing a similar set of functionality was developed at Joyent, called 'fabrics'. While there is no canonical paper à la the items we’ve discussed above, the high-level view was an overlay / underlay scheme was developed that leveraged VXLAN as an encapsulation protocol and a custom scheme we developed which was a simple TCP protocol that went through a proxy to our production databases. A lot of the experiences (both positive and negative) are a major influence on the design proposed here and provide additional, useful background.

A major focus of the design was on reusing existing OS-level abstractions (such as virtual switches) and the creation of a new abstraction: the overlay device. The overlay device was a data link upon which additional virtual NICs could be created and took two specifications, an encapsulation plugin and a search plugin. The encapsulation plugin described how to transform the packet to put it on the wire. The search plugin operated in userland and when the kernel had an unknown destination, userland would be responsible for looking that up.

Like is described in the VL2 paper and alluded to in the Andromeda paper, there was a centralized database that had mappings. When things changed, we had an invalidation record that would be written out for each compute node that needed to take an action and it would take it. While there was an idea of pre-loading mappings, we always did so lazily. The one optimization that we had was that when we received an ARP or NDP request, we would not only inject the response, but also the mapping from the virtual MAC address to the compute node.

With those basics in mind, we can begin to talk about some of the things that worked well, some of the things that worked poorly, and which of those were due to implementation and which were due to more general architecture.

One thing that did make sense, and this is reflected through what we’ve seen written from Google and Microsoft, was the split between the data path and the control path. Effectively having the data path just do its thing (whether in-kernel or in userland) and having a separate entity push updates has been useful. This is a general observation folks have made and is part of the design of other open source tools like Open vSwitch. Fundamentally this split seems to be one that’s worth keeping.

That said, an important thing is keeping that actual database available and having the things that fetch from it be able to actually handle transient failures in a way that doesn’t really increase latency. This is only useful in so far as the broader system is actually available and tolerant to those failures.

Another major lesson was the choice to focus solely on L2 networks. While what the customers ultimately cared about and got in the API were L3 networks that they could control the IP addresses over, the actual implementation focused on L2 addressing. This had a couple of interesting side effects over the years:

With this in mind, we instead choose to focus on a design that really functions at L3 and provides us the flexibility that we need to implement the flows described earlier. The approach that Google has chosen by using an off-subnet route is actually very useful and one that we intend to use ourselves as discussed in RFD 21.

This leaves a few distinct advantages to us. Critically that the guest only ever has to be able to have an ARP address for the gateway, which will be fixed. There will still be lookups in the system, but they shouldn’t interfere with the ability of the guest to actually generate and send those packets.

Also, this construction basically means that we’re treating this as an L3 forwarding problem, which ties into what we’re actually trying to create which is routing, NATing, and making decisions about reachability based on IP level information.

One of the other interesting choices was how we tried to plug this into the kernel. There were a lot of benefits for us in terms of maintainability and getting things off the ground and leveraging things that already existed. At the time, the focus was on building different blocks that we could assemble into what we needed. While that was reasonable based on our business needs and constraints, it left some challenges. In particular, some aspects of performance were a little trickier due to some of the ways we plugged into the existing networking stack (effectively both as an application consuming and a driver producing).

Like Microsoft, Google, and certainly others have done, constructing a programmable engine that we can change on the fly or evolve and iterate on in a faster way is worthwhile. In addition, as we’ll talk about later on in [sect-opte], depending on how we build this, it’ll give us a lot of flexibility and the ability to more easily adapt and evolve the product.

A related lesson is just to be careful about where the abstractions are created and how you can see through them. While abstractions and layering are important and useful, eventually you also have to cut through them.

The physical network is built upon IPv6 and utilizes subnets to summarize collections of resources. Addresses are assigned from the IPv6 ULA range. For example:

Conceptually, you can see this in the following image:

Based on the above, you can see that each site can be summarized with an IPv6 /48. We’ve then broken this up such that we have eight bits dedicated to indicating the rack in a site and eight bits to indicate the server inside of the rack. This gives us approximately 256 racks in a single AZ and allows us to have up to 256 hosts in each rack.

RFD 76 discusses the range of hosts that could fit into a given rack given different configurations. In the densest configuration listed there, 1OU high x 1/3 wide, that would give us 78 servers in the rack. Even if that quantity were able to double, we would still have plenty of addresses available.

The limitation of around 256 racks would give us on the order of around 10-20 thousand servers that could be represented in a single /48 segment. Is it likely the case that this is a reasonable sizing for a Cell or perhaps even an Availability Zone. Though more research into the scalability and bandwidth constraint of such a network core needs to be done.

An important aspect of this scheme is the ease of summarizing resources. The more that we can summarize resources, the less routing table entries that we’ll need to program into the switch. Given our plans to maximize switch resources for Boundary Services, being able to make sure that as we expand the rack we don’t end up exploding our switch state is an important aspect of making sure we can design around [goal-3].

Having a specific set of IPv6 addresses map to a server may on the face appear to fly in the face of [goal-5].

As described earlier, each server is allocated an IPv6 /64 which it advertises for routing on the network. The process by which each sled acquires the /64 is described in more detail within RFD 259: Who’s On First.

Once provisioned to the sled, that /64 will then be used by the server itself and addresses from that /64 prefix can be allocated to various internal services that run on the control plane. Although each sled will be responsible for actually using the addresses, allocation will need to be split between different entities in the system such as the Rack Setup Service (RSS), Nexus, and the local node (driven primarily by sled agent). The details of the allocation scheme are left to other RFDs.

This addressing scheme implies a few important things:

The rationale for this approach are a few fold. The first is that we’re trying to reduce what we’re advertising in the routing protocol. By using addresses from the server, we minimize the number of entries that we have in the routing table and means that we can much more cleanly summarize a rack.

Another reason for this approach is that a number of aspects of the control plane design suggest that there will be a service discovery mechanism. Because Nexus will probably have some need to scale over time and discover what exists, the underlying addresses for most of that shouldn’t matter.

However, that does also mean that we will need some number of addresses for service discovery itself. If we assume that these may be Cell or Availability Zone-wide, then our plan is to take one of the rack /56 entries and reserve it for this purpose. In this particular case, we will support advertising the individual /64 entries in the routing protocol for the scope of the site. The hope is that the number of entries used for this will be small. One could imagine a handful for consensus such as 3, 5, or 7. This could potentially be augmented by anycast support

The above sections describe how we believe the control plane and network will operate between hosts. However, there are a few other cases that we need to consider, notably that of the service processors, the power shelf controller, and the problem of cold boot.

Each data link in the system will have an IPv6 interface on it that has a link-local IPv6 address. This will be true of the switch ports and each of the server’s ports. A server’s routing information will always suggest that for a server in the rack, the next hop will be one of the two switches and it will always advertise that the paths for its server /64 are over its link-local addresses. This coincides with the design of isolated L2 networks and reduces the need for ARP/NDP.

We’ve decided to not cross-connect the switches as part of our network design mostly because of the question of how much cross-sectional bandwidth should that use and to try and fit into various different evolutionary designs and other systems (originally in 2020 we were looking at using RIFT, which has transitioned to delay driven multipath as part of our further experimenting and usage. As a result, each server will need to have reachability information for at least the other servers in its rack pushed down. While we may be able to get away with summarizing portions of remote racks higher-up, we’ll have to work to ensure that we don’t need to have every server have to keep track of distinct paths to every other server if we can to maintain routing scalability.

The internal routing protocol that we use is discussed in RFD 196 Maghemite: The Oxide Routing Stack. This protocol and its needs are somewhat distinct from what we use to interface with a customer’s network (see [bs-adv]) which are more constrained due to the need to interface with our customer’s existing environments (see [goal-4]).

One requirement that we believe we should have is that this routing protocol should be able to operate in a mode where only cryptographically signed messages. We may require different chains of trust depending on what we are performing. For example, using a server-level secret for information within the rack that is trusted by the rack and then using a rack-level secret that is trusted by the Cell or Availability Zone when joining multiple racks together.

XXX This section was initially forgotten and needs to be fleshed out.

This section summarizes the order of resources that we will need to maintain in the switch for physical networking. This list may seem small, that is intentional, to allow for the majority of the resources to be used for Boundary Services.

From a routing table perspective a switch in the rack would need to have entries for:

We would need to keep track of the following order of MAC addresses divided into multiple disjoint L2 networks with corresponding VLANs within the primary switch:

While these might increase by a constant factor, the good news is that the total order of these resources should not put a major dent in the Tofino 2’s resources and allows us to leverage its functionality for more interesting purposes.

The main part of the solution space that we’re exploring are various packet encapsulation schemes. This section provides an introduction to them, their use, and associated trade offs.

For Oxide’s initial product implementation, I propose that we perform network encapsulation with Geneve. Concretely within a region (or perhaps an availability zone) we would allocate every customer VPC a Geneve ID as well as some for system purposes such as Boundary Services. This will be used in tandem with [sect-opte] and [sect-onds] to provide a full set of network services and network virtualization.

Most of the IP based protocols do not have this notion of a virtual ID, which is an important part of differentiating traffic. While it could be created with our own work or abusing an IPsec AH or ESP Security Association context, creating something new and fighting the protocol and others interpretations doesn’t buy us anything compared to a protocol with an explicit notion built-in.

The physical network will be leveraged as an underlay network. When a flow leaves a virtual machine, it will pass through the [sect-opte] and be destined towards the host that has the instance. Intermediate switches will not be required to be aware of this, only the compute nodes and Boundary Services.

With this in mind, it’s worth evaluating how this helps us meet our goals and why this solution was chosen over others.

One of the major impacts with the choice of having customers with their own IPv4 networks is that of [goal-5]. Modern network design focuses on reducing the scale of broadcast domains to improve the operational situation. One distinct advantage of using any overlay network based scheme is that it gives this flexibility at the cost of maintaining a database that has the ability to map between the virtualised network interface of an instance and the physical host it is on. However, that is well trodden ground and is employed in aspects of Microsoft’s VL2, Google’s Andromeda, Joyent’s Fabrics, and other OSS implementations.

Ultimately, the ability to put an instance on any host in an availability zone (or perhaps region) and is what allows us to meet [goal-5]. While there will be other aspects of the system that might limit this migration (e.g. storage), this flexibility is well worth the cost.

This ties into a broader benefit of this approach. Basically that of being able to separate out the physical network that is used for general communication of instance data and the control plane from that of the instance. The physical infrastructure needs to worry about about how to build itself in a scalable way and if the customer instances were directly on that physical infrastructure, that would ultimately devolve into very large L2 broadcast domains which can inhibit scalability.

Two goals that are related here are [goal-2] and [goal-3]. The use of the network encapsulation as part of the compute node’s network datapath (discussed in more detail in [sect-opte]) basically pushes out most of the heavy lifting of the network to a combination of the compute nodes and boundary services. This gives us a few useful properties. Once a packet leaves a given server, then it will be able to leverage the general availability characteristics of the physical network and its inherent reliability and scalability. There will be nothing specific that we need to do here by employing the use of an overlay network and performing network virtualization.

By fanning out the work of the network encapsulation and decapsulation to the compute nodes and boundary services, that improves our scalability odds. This doesn’t mean that we won’t hit scalability issues by doing this on the server at some point. This reality was pointed out in the Google Andromeda paper. Here at least, we are reducing the flow state table to the flows that are likely active on a given compute node based on the instances present, which will, on average, be a subset of all the instances in a project.

Importantly, this is often comparing itself to the more classic network approach of a pair of devices that provide a single service that every packet has to pass through.

As discussed in the background section, there is a lot of hardware NIC functionality that helps us with offloading [goal-1]. The ability to have the current suite of stateless offloads (checksums, TSO, etc.) and a foundation for doing LRO in software is useful. Notably, not all protocols have these accelerations available. While VXLAN does and some IP based encapsulation does as well, it is not often as well defined. While it is possible that we could use a different protocol and restrict ourselves to a smaller set of vendor cards, the evolutionary options provided by Geneve are more appealing.

In addition, most of the IP based options don’t provide a common way for using information from the inner-flow to influence equal-cost multi-pathing without teaching intermediate nodes about that. While there’s no reason we can’t, being able to leverage something that has it built-in is an advantage.

The high level vision that we propose incorporates a combination of ideas from both those articulated in RFD 9 Networking Considerations as well as aspects that we’ve learned from the different implementations discussed in [sect-past].

At the core of OPTE is actually an execution environment itself. The core idea is to be able to build a rust-based execution engine into which we can feed a series of instructions, rules, and transformations that should occur in a connection-oriented fashion. The main goal for the rust based execution engine and rule processing is a couple fold:

The hope is that with this engine we would be able to deliver different plugins that can operate in a layered fashion, similar to what Microsoft describes in VFP. With this we could build properties that allow us to perform actions like encapsulation, NAT rewriting and state tracking, determine whether a firewall allows or denies traffic, and more.

OPTE is made up of two major components:

This split is useful for a couple of reasons. The first is a separation of concerns. In essence, the dataplane part of it may be embedded into operating system kernels or some other semi-privileged state and we don’t want to bake in the question of how to actually find and track these updates or perform these functions. The means by which we talk to and communicate in the control plane will naturally want to evolve separately from the data plane and vice versa and aspects of these may also vary on the environment that we’re found in.

For a visual version of how this fits into the broader environment, see [img-compute].

While the full design and architecture need more thought, the rough idea here is that we can install a series of per-guest virtual NIC layers of transformations. Each layer will potentially look up existing state (for example, routing rules) or allocate state (firewall rules) and then may push on a set of transformations to apply once all of the layers are processed (e.g. NAT or Geneve encapsulation). Some examples of how OPTE fits into packet flows are: [flow-inst], [flow-nat], and [flow-extip].

Packets in the dataplane that require additional processing or are being implemented by a specific control plane service, such as DNS requests for the internal DNS server or instance metadata, will be passed into a bounded queue for processing by the control plane agent. The control plane agent will take care of any necessary rate-limiting, process the packet, and then inject a reply back into the guest network. Examples of these flows are: [flow-dns].

In addition, it will be responsible for the coordination of snapshotting, and freezing state for live migration as well as processing and coordinating with the control plane.

XXX This may want to be a top-level section that describes why we’ve chosen where to break out different components.

A critical aspect of OPTE is that it exists on each server. This has several implications and follows from our goals. While it is easier to maintain the overall set of routing and firewall rules to apply when there are fewer places with that information, that has an inherent challenge with scalability and availability. At the heart of the decision to push this out to the server are [goal-1], [goal-2], and [goal-3].

Ultimately, in this overall design and architecture, there are a few different places that we could do this processing. When we’re trying to think through this, the things we need to ask ourselves are:

With the proposed structure there are primarily two different extremes that we can start with to manage these questions of state, validation, and transformation:

For the sake of this discussion, when we talk about the 'network fabric' we’re referring to the switches and routers that forward traffic. The producers and consumers generally refer to the servers themselves and boundary services.

The proposed initial bias is to have the network fabric itself be simple and instead push that complexity on the server node. Fundamentally, this is because we believe this will give us a better foundation for creating a more scalable and rich implementation. There are a few reasons for this:

A fundamental challenge that we have with OPTE is bounding state. While, it’s tempting to try and pretend we don’t need to track state, to implement the features we’ve outlined in RFD 9 Networking Considerations and RFD 21 User Networking API we need some amount of per-flow state. However, this state is also the enemy. Many network operators have had to debug the problem of the firewall or NAT state table that is full, causing all subsequent connections to fail.

Fundamentally, in OPTE we have this tension between having limits to help protect the broader system, but the knowledge that when these limits are hit there are dire consequences for customer’s instances to be able to continue behaving in the ways that they expect. The other challenge is that sizing these limits is, difficult. As a starting point, we can draw from other cloud providers who document this information, but then the most important thing we need to do is actually provide end to end alerts and monitoring in the product for these types of facilities.

I propose that we use Virtio in our product as the initial interface for networking between the guest and the host. This does not suggest or require the use of Virtio between other parts of the product.

With respects to the goals that we laid out, Virtio does help us on a number of them. There is wide driver support in a number of operating systems, including, but not limited to:

This wide OS support is important. This is in support of [goal-4]. While some emulated devices have a wide degree of support, they are often in turn widely supported because they are older devices which means that they aren’t suitable for use with SR-IOV. Instead, if we look at things like SR-IOV devices, the support matrix is much more scattered. Even things like AWS’s ena and GCP’s gve have fairly limited platform support as compared to Virito, not that they don’t have other merits. An important assumption going into this is: Oxide does not have the resources to write a high quality networking device driver for multiple, different operating systems for our 1.0 product.

Virtio does support a number of offload features and allows us to negotiate support for IPv4 and IPv6 based checksums, segmentation offload, and large receive offload. For guests that are primarily doing HTTP-style traffic with larger assets, these are all features that dramatically improve the performance for the guest. If a guest is focused on the packets per second rates or focused on small packets, then this is less advantageous. Virtio also supports the ability to wire up multiple transmit and receive queues and put together something like receive side scaling through it. While it may not be as featured as some device drivers and not all guests necessarily take advantage of it, it does provide us a useful tool.

With that in mind, it’s worth talking through why Virtio over emulating devices. In general, because the OS support is equal, the main advantage we have with Virtio comes to the main advantages of para-virtualization over device emulation. Strictly, the number of reads and writes that will trap out is reduced and you have something that is designed knowing it is virtualised. This makes the corresponding device drivers simpler and easier to reason about in the guests.

In addition, because the commonly emulated devices are older, they often don’t always have support for things like TCP segmentation offload, large receive offload, or sometimes even things like IPv6 checksum offload. This all flies in the face of [goal-1]. If we had to choose between the two, there’s basically no reason not to use Virtio over an emulated device in this part of the space.

When we compare this to using an SR-IOV and virtual function style solution there are a couple of different considerations for us.

Based on the above trade offs, it seems like leveraging host-based Virtio is the right initial choice for us and makes more initial sense for us and directionally. While the virtual function interfaces of products on the market may be somewhat snappier, they create more complications in terms of managing those devices. In addition, even if we went down an SR-IOV path, we’d want to actually have a Virtio like interface for that, or some other wide-spread, vendor-independent device interface. As the only thing that we know for certain is that we will likely change and evolve the host interface and host side of things.

One of the principle jobs that we’ll have is advertising our network to the broader customer environment. While the customer will be giving us control of a specific range of IPv4 and IPv6 addresses, it will be our responsibility to advertise them to the broader customer network in the form of a routing protocol. In general, there are three major forms that this takes:

From initial conversations we’ve had with different customers we’ve come to the conclusion that BGP is probably the most important of these three options to support out of the gate. In the spirit of [goal-4], we should start with BGP and then prioritize the hardcoded static routes. As we are successful and need to interface with more customers, and have customer demand, we should look at implementing support in the boundary services for OSPF.

While the network protocol routing is the most important service that we need to implement in boundary services (ignoring the packet transformations), there are a couple of other services that we’ll need to consider and possibly add support for. These include:

The Oxide physical network treats itself as an isolated series of L3 IPv6 subnets which is done in service of our goals and gives us a solid foundation for building modern networks. In addition, it serves to isolate us from the varying implementations of our customer’s networks, their IP allocation schemes, and other details.

As we laid out in [sect-virt], we are currently planning on leveraging network encapsulation and virtualization in part to implement VPCs and in service of maintaining [goal-3] in the face of [goal-5]. To do this, the boundary services needs to be able to translate traffic between two formats: encapsulated traffic on the Oxide Physical Network and the customer’s broader network.

If you walk through the example flows [flow-nat] and [flow-extip] you will note that all instance related traffic is always encapsulated on the physical network, thus allowing the physical network to ignore the addressing that customers use.

To facilitate making this a reality, when boundary receives a packet from the broader customer network, it will need to be able to have enough information to encapsulate that and forward it to the correct host on the physical network. Conversely, when a packet comes from the Oxide physical network destined for the customer network, boundary services will simply need to decapsulate it and forward it onto the customer side.

To implement this we actually look, initially, to our choice of switch silicon — Tofino. One of the challenges that we have in the initial buy of the rack is to minimize the amount of compute overhead that we are using. Because, as we’ve discussed earlier in [opte-server] and elsewhere, we’re trying to minimize the actual router requirements for the physical network, this means that we have plenty of switch resources leftover exactly for this purpose. Therefore our plan is to initially rely on the switch itself for doing this transformation!

This all implies that boundary services needs all the state for the following:

Unfortunately, this means that when provisioning happens and ephemeral IPs, floating IPs, or NAT state are allocated, we need to be able to update boundary services with this information. However, the silver lining here is twofold:

From a physical perspective, operators have often the scraping of switches or leveraging SNMP to build dashboards where they can look at switch ports and understand questions like:

To that end, we believe that in the broader product we need to be able to visualize this data and make it actionable. See RFD 82 Motivations and Principles for the Design of Operator Facilities for a larger discussion of the rationale.

To be able to build this there are a couple of things that we need to be able to establish. The first of these is topology. To build up topology, we propose that the all of the servers and switches in the environment support the link layer discovery protocol (LLDP). This will allow us to exchange information between servers, switches, and other network equipment about what is connected to each switch. Note, that this information cannot be relied upon for correctness of the implementation, but rather it should be used to help us diagnose and understand problems. This is analogous to what is proposed in RFD 40 Physical Presence and Location: Requirements and Architecture for understanding the physical layout and composition of the rack.

While LLDP gives us the ability to discern and create topology, we also need to gather regular metrics and feed that into whatever broader solutions for metrics the rack ends up developing. We need regular metrics from our switches and all of the NICs in them. While the full list of metrics that we care about is deserving of its own RFD, at a high-level it’s about utilization and errors, with simple breakdowns based upon different dimensions such as packet size, unicast, multicast, and packet type such as IPV4, IPv6, TCP, UDP, etc.

When we consider the application team’s perspective they generally don’t care about the actual physical network itself, unless there’s a problem. Instead, they generally care about things that impact their application. Common situations here may be things like:

Much in the same way that the physical view wants to have topology information, we want to eventually be able to construct the same thing but for the application and answer the question of what is talking to what. Unlike the physical view, this is a much more fluid and dynamic construct and is in many ways related to the proposed future work of Firewall Flow Logging.

One possible way to imagine building this is to leverage the information that OPTE has in its firewall tables to build up these pictures. Because this may be driven by snapshots, it may not be the most accurate picture, but it is a useful starting point. The one nice thing about OPTE is that regardless of where it lives, it will have to deal with all data from the guest at some level, which allows us a picture of what connections exist.

Similarly, because OPTE does see all the data, we can consider how we evolve it to track things like data on specific flows and it could even allow us the ability to create the equivalent of netstat-style statistics without needing an agent in the guest.

In addition to the aspects that are mentioned above, the overall network will need to make sure that functionality like the following works:

The first thing that we need to concern ourselves is with the means by which multiple racks are able to physically connect to another and when connected, be able to communicate.

Communication is fairly straightforward. Because each rack has an explicit IPv6 summary (see [sect-phys] for more information) which is carved out of the broader /48, we know that we can assign each subsequent rack one of these and that this will represent everything behind that rack in the routing table. Ensuring that a new rack that is joining the broader cell or availability does not use an address that is already in use will require some coordination during installation; however, hopefully it can be limited to that point.

Because we can represent each rack as a summarized entry, the amount of routing state that we should need for each additional rack should hopefully be O(racks) and not be something that grows at a faster rate. The implicit constant there is still important, but the design should mean we have adequate switch resources for this.

The bigger challenge that we need to think about is connectivity. The way that we evolve switch connectivity between the Cell and availability zone will vary as the environment scales up and we figure out what the desired oversubscription ratios are. Given that we have plenty of additional ports on the switch for this (see RFD 58 Rack Switch), we have options for directly connecting a smaller number of racks or at some point connecting to a rack that had additional networking gear for building up larger CLOS-style topologies.

The biggest current requirement of this scheme is that all Oxide racks in the same Availability Zone are always connected to one another through Oxide-supplied gear. Put differently, to traverse between two servers in the same availability zone, no equipment from the broader customer network should be involved. While at the surface this may fly in the face of [goal-4], this has large implications for [goal-6] and some of the others.

Fundamentally, we are going to want to be able to leverage our own equipment and have guarantees on the operation and how packets flow between them. If customer equipment is in the middle of that, it’s going to make things much more problematic when we deal with management and understanding why problems have arisen and in particular, Investigations of Unknown Problems. Many of our customers teams are stratified between network management and server management and are oft to point fingers at one another. The more information we need from devices outside of our control to be able to accurately diagnose the problem, the harder it is going to be for us to resolve the problem with a good experience.

While we know that not all customers will have organizational boundaries that are quite as pathological as described above, when we think about what it means to be able to understand the relative bandwidth allocations in the network and perform QoS, it will become much harder for us to do that while crossing the customer’s broader network segments to talk between two racks.

This does come at a cost. It means that we will eventually need to develop more specialized networking equipment that plays the roles of aggregation and core switches in our customers environments today; however, this still seems like an area where we’ll be able to provide value and ultimately by tying that into the automation of the product, ideally be able to offer a better product to our customers.

An important consideration in all of this is the scalability of network services. There are a couple of different aspects for us to consider:

So, first we should address the scalability of the different user-services that are implemented by OPTE OPTE. In particular, this includes: VPCs, Routing, Firewalling, and aspects of NAT.

From a performance perspective, because we are fanning this out to each server, as additional racks are added, our resources to perform these actions should be able to scale with that environment. Because each compute node is doing its own routing and firewalling for customer traffic, that should be manageable.

However, as the environment grows, so will the amount of state that is required to perform this. To deal with this, we will need to impose limits on things like the number of firewall rules or the number of instances in a project or that can be shared together in peered VPCs. Ultimately we will need to pick certain values and figure out what will be manageable.

When we consider boundary services, this becomes somewhat trickier. While we initially are starting with boundary services being implemented by the rack switches themselves, when we end up in larger environments it will likely make sense to have additional boundary services devices that perform this. To make that scale we will want to have a way to shard this out to make each shard highly available. We should work to figure out what the approximate order of magnitude of entries we can keep in a group of coherent boundary service devices.

That said, if we do have a way to shard that out such as directing subsets of boundary services to different addresses, then this makes the scalability problem a bit more manageable and we should be able to increase the number of those to handle this, though the connectivity to do that can be challenging.

Finally there are a collection of different services that we have which are in essence a part of the larger control plane. These will need to be horizontally scalable once we get to a certain point so it’ll be important that we can design an evolutionary path for that from the beginning. Similarly, knowing that we can eventually shard some of those databases may be rather valuable.

A major area that we already know that customers want is load balancing. We can roughly break down load balancers into two camps based on whether or not they actually terminate the connection or not. There is a lot of literature on load balancing and software out there, whether it is related to Microsoft’s Ananta, Google’s Maglev, or more. While we describe a few approaches and ways this can fit together, this is by no means meant to be constraining future us, rather hopefully proving to future us that this isn’t impossible.

In any case, it wouldn’t be too hard to augment boundary services with additional general purpose compute resources. For example, one could imagine that for load balancing, boundary services would be able to forward this onto a bunch of intermediate nodes that would do the actual connection termination and health checking of backend instances before forwarding them on. If a connection-forwarding version was instead preferred, we could potentially put that hashing into boundary services itself and have the secondary compute be responsible for performing health checks and updating boundary services appropriately.

When traffic is leaving the instance, if we are not doing a connection-terminating load balancer, then we can employ a technique called 'direct server return' as discussed in the Ananta paper (and others) and have OPTE direct traffic back towards boundary services to be returned. However, if there is something doing termination, then we need to know how to send traffic back towards that. In an ideal world, that can be managed by OPTE as though sending to any other instance.

IPsec based VPNs and cross-region traffic share interesting properties in that they generally require that data is encrypted in schemes that provide both confidentiality and authentication. While IPsec does require specific mechanisms that we might not require in a cross-region setting, these are more similar than different.

In essence, when there’s a VPN on the scene there is some amount of routing that needs to go on, which fits into the existing models proposed by OPTE rather nicely.

The next challenge, which will be more of an issue for the IPsec VPNs is that we ultimately need something that is running IKE on a per-VPN basis and paying attention to routing rules. This will mean some amount of HA for this which will need to be designed carefully.

The larger challenge that we’ll have is the same for both the cross-region behaviors and IPsec VPNs, mainly the actual act of performing encryption. There are many models that we could employ here. On one hand, we could consider pushing this into OPTE. On the other hand, we may not have the resources to do that in either the NIC or in the host CPU and may want to instead have dedicated systems whose job it is to encrypt and forward. This is where aspects of Andromeda’s Hoverboard approach and careful application of hardware acceleration could be useful.

Firewall flow logging is something that we can build in concert with OPTE. As we discuss in the Application Observability section, to build up the notion of a customer’s topology is a similar problem to flow logging. There is both a need to be able to collect the data, but then for flow logging to be useful, one needs to be able to aggregate and amalgamate it.

The most useful thing here is that the foundation we’ve built has enough information for us to take care of the first step which is do we have the data. Building the ability to hoover it up at the necessary rate for this to be useful its a complicated feature in and of itself, but this is a useful starting point.

This feature refers to a guest’s device having the ability to have multiple IP addresses associated with a given interface. Implementing this on top of our existing foundation is fairly straightforward. It would require additional database state and support in the firewall API syntax; however, once that was present, all this should require is a slight modification of some of the OPTE processing modules.

Another thing to explore is the idea of support for anycast services. A great example of this is how Google uses it to implement their public DNS server 8.8.8.8. In this case, if multiple different availability zones, whether they’re in the same or different regions, are part of the same fleet, they may be able to have functionality where they can coordinate and nominate a specific public IP address this way.

Once that’s been done, then boundary services can advertise this on the broader customer’s network and if that is destined to be a public IP, then it will need to be advertised by that customer on the broader Internet. However, if it’s just offering an internal service that won’t be necessary.

With control over the internal network fabric and our IGP, it should also be possible to create internal anycast services. This may be a useful way to implement some parts of our HA services or control plane services, especially as the environment grows.