RFD 21 User Networking API

Let’s look at an example of a VPC that’s used to implement a blog that’s logically made up of three layers: a load balancer, HTTP/application servers, and a database tier:

Let’s look at the different components. The items listed below correspond to the values in the image.

To make this clearer, let’s work through some sample flows here. Assume that a user is trying to reach the blog over IPv4 and is doing an HTTP GET request. The HTTP request would first target the IPv4 floating IP 72.14.168.115. When that traffic reaches the Oxide network, the network will translate that into a request that is directed to the 'Load Balancer' instance, 10.169.10.30.

From here, the 'load balancer' would forward it to one of the two HTTP instances, 'HTTP 1' or 'HTTP 2' based on its internal policies. Assuming it chose 'HTTP 2', then it would send that HTTP request to the 'HTTP 2' instance, using either its IPv4 or IPv6 address. The application server will then break down the request and fulfill it. If as a part of it, it needs to access the database cluster, it’ll send a message to one of the databases, which will be facilitated by the 'VPC Router' (6). There may be firewalls in place that may further restrict communication between the instances and the VPC Subnets.

When the database replies, it will reply back to the 'HTTP 2' instance, whose traffic will be routed through the Router. The 'HTTP 2' instance will then reply to the 'Load Balancer'. When the 'Load Balancer' wants to reply to the traffic, it will go through the 'Internet Gateway' (7). The 'Internet Gateway' (7) is responsible for making sure that the reply has the right outgoing IP address (that of the floating IP) and making sure it goes out to the Internet. All traffic leaving the VPC must go through an 'Internet Gateway' (7).

Let’s consider another example, let’s say that 'Postgres 2' want to make a connection to the Internet just to ping 8.8.8.8. In this case, the traffic from 'Postgres 2' would first go to the Router (6). The rules in the router would direct it to the 'Internet Gateway' (7). A NAT session will be established for this communication and the rack will pick an IP address and port to use for the NAT translation from a rack-wide shared pool. After rewriting the packet, when the 'Internet Gateway' (7) replies it’ll rewrite the response it got back from 8.8.8.8 and direct it towards the IPv4 address of 'Postgres 2' 10.168.20.12. That will traverse the 'Router' (6) and then 'Postgres 2' will receive the response.

In addition to these examples described above there are a couple of other things to point out about the VPC:

A VPC is broken into a series of VPC Subnets. A VPC Subnet spans more than one Oxide rack and can operate in an entire availability zone, which is selected when the subnet is created. Each VPC Subnet has an associated IPv4 and IPv6 CIDR block. The IPv4 block must be allocated from one of the IPv4 private address ranges. The largest IPv4 subnet that can be created is /8 and the smallest is a /26 which allows for approximately 64 addresses. A VPC may have multiple VPC Subnets, each with distinct IPv4 address ranges, the only constraint being that VPC Subnets may not overlap. VPC Subnets also have IPv6 address ranges associated with them. These must be be Unique Local Addresses in the range fd00::/48.

When a VPC is created, a default VPC Subnet is created for you. This uses the IPv4 address range 172.30.0.0/22, and a random IPv6 Unique Local Address in the range fd00::/64. This /64 range is allocated out of the /48 prefix allocated to the VPC, which may be chosen at the time the VPC is created, or a random prefix will be assigned at that time.

When an instance is created, it is associated with a VPC Subnet and receives an address from that VPC Subnet. By default, the following communication rules are set up on a VPC Subnet:

When creating an instance, a user may specify an IP address from the subnet to use or allow the system to pick one. In addition, IP addresses in the network may be reserved through the API. A reserved address will not be allocated by the system automatically. It can only be used by explicitly requesting it when provisioning an instance. An instance may use both IPv4 and IPv6 addresses from the VPC Subnet or only a single one.

A number of addresses in a network are used by Oxide to provide rack services. For example, if the subnet were 192.168.1.0/24 and in IPv6 fd12:3456::/64, the following addresses would be reserved:

A VPC Router defines a series of rules that indicate where network traffic should be sent depending on its destination. A VPC Router is part of the underlying fabric of the VPC and is different from the routing table found inside of a guest (e.g. the information you get when one runs netstat -rn).

Every rule in a VPC Router contains the following:

There are a number of different targets. These targets include:

There are two types of routing tables:

In each VPC, there is a VPC-wide VPC System Router that is created when the VPC is created. Routes are automatically added to and removed from the System VPC Router. Routes cannot be added or removed directly from this table; however a few entries can be modified. The VPC System Router contains the following types of entries:

In addition to the VPC System Router, a VPC may contain a number of VPC Custom Routers. The VPC Custom Router is used to provide additional routes and override the behavior of the VPC System Router. Each VPC Subnet may have a single, optional VPC Customer Router associated with it. The same VPC Custom Router can be associated with multiple VPC Subnets.

An Internet Gateway provides instances in a VPC access to the Internet and acts in a similar fashion to a traditional source NAT that one would find in a home network. When a VPC is created, an Internet Gateway is created by default as well. The default Internet Gateway shows up in the VPC System Router by default. A subnet without an Internet Gateway cannot route to the Internet on its own.

It is possible to create additional Internet Gateways and to associate them with VPC Custom Routers. This allows you to be able to have no outbound access by default, but to allow certain subnets to have access.

Each Internet Gateway is associated with a pool of external addresses that may be shared with other VPCs and projects.

However, there are many cases where applications need to deal with remote services that allow and deny access based on the IP address. To deal with this, a number of Floating IP addresses may be allocated and associated with an Internet Gateway for traffic within a single availability zone. When using this mode, the system will not dynamically scale up the number of IP addresses associated with the NAT as that would otherwise defeat any IP address based filtering.

An additional use of this mode is to have no IPs associated with the Internet Gateway, causing all outbound traffic to be dropped. As described in the next section, an instance with a floating IP address will use that when making outbound connections. See the next section for more information.

See also additional design considerations.

By default, all networking for instances on a VPC is private to that VPC. This means that while instances can make requests to the Internet, they cannot receive inbound connections.

The system provides two different ways to allocate an external IP address:

When an instance has a Ephemeral or Floating IP address, it will not be visible inside of the guest. This means that it will not show up in tools like iconfig or ip addr. Instead, the Ephemeral or Floating IP address will act as a 1:1 NAT. Basically all traffic to and from that IP address will be forwarded to that particular instance and its primary interface. When the instance replies to that address, it will also be translated back into the external Ephemeral and Floating IP address.

To support an Ephemeral or floating IPv4 address, the instance must have a corresponding private IPv4 address on its primary interface. Similarly, to support an Ephemeral or Floating IPv6 address, the instance must have a corresponding IPv6 address.

To use an Ephemeral or Floating IP address, an instance must be in a subnet with an Internet Gateway. This helps maintain the simple rule that if an instance does not have an Internet Gateway, it cannot reach the Internet at all. The Internet Gateway will ensure that all traffic from that instance uses its Ephemeral or Floating IP when making new outbound connections and when replying to traffic. If an instance has more than one Floating IP address, then the Internet gateway will use all of the associated addresses when making outgoing connections.

Ephemeral and Floating IPs can be assigned and removed while the instance is running. An Ephemeral IP may be changed to a Floating IP through the API and a Floating IP can be changed back to an Ephemeral IP as well. Neither of these operations interrupt the connectivity of the IP address.

An IP pool is a collection of external addresses that are maintained by operators. There are separate pools for IPv4 and IPv6 addresses. A given pool may have more than one IP CIDR block inside of it. IP pools may be made available to all projects or they can be restricted to a specific project.

When allocating an Ephemeral IP, Floating IP, or an Internet Gateway, an IP Pool may be optionally specified, which will cause the corresponding IP address to come from those in the IP Pool. Note, an IP Pool is never used for addresses from VPC Subnets that are used by guests.

The Oxide API services provide DNS servers for resolving the names of instances to their underlying IP addresses. There are two different types of DNS servers that are provided:

A VPC Firewall is a tool that can be used to limit what instances can and cannot talk to other instances. Each VPC has its own independent VPC firewall. A VPC firewall is made up of a series of rules. There is one set of rules for incoming traffic and a second set of rules for outgoing traffic.

The VPC firewall is a stateful firewall. This means that when a connection is established due to allowed rules, there don’t need to be explicit rules going in the other direction. Consider the case where inbound traffic is denied, but outbound traffic is allowed. Then an instance makes a connection outbound, it still expects to receive some amount of inbound traffic. Because the firewall is stateful, an exception is made to allow that specific inbound reply back in.

Each firewall rule has the following attributes:

When specifying a target or a filter’s source or destination, it may be any of the values specified in Target Strings.

For filters, the use of explicit IP addresses and blocks is not recommended. Using tags, subnets, and instances, where possible will ensure that as an instance’s IP addresses and NICs change that nothing slips through firewall rules. It also allows most rules to target both IPv4 and IPv6 by default without having to think about whether the instance is using one or the other or updating a rule when that changes.

The priority of the rule dictates its evaluation order. Rules are sorted according to priority and rules with the lowest priority are evaluated first. The first rule that matches is used. Unlike routing, the prefix does not matter, only the priority. If there is both a matching allow rule and deny rule at the same priority then the deny rule takes priority.

While each VPC is independent and private, there are times that you want to be able to privately share traffic between two VPCs between different projects and organizations within a region without going out over public IP addresses.

For example, a company that provides a database as a service may have have a VPC or subnet per customer. In an ideal world, they would be able to peer that VPC or subnet into their customer’s private networks. An example of this is discussed in RFD 9 Networking Considerations.

To facilitate this, two VPCs can be peered together. When two VPCs are peered together, for each subnet that’s been peered, the following happens:

When a VPC peering relationship is established, subnet propagation can happen in two different modes:

The system will not allow two subnets from either end of a VPC to be shared if they overlap in their blocks. The use of IPv6 ULA /48 prefixes on a per-VPC basis should eliminate this possibility; however, it is easy to end up in this situation with IPv4 and the default subnet. Automatic mode cannot be enabled if there is existing overlap.

When a subnet is removed, the routes, firewall rules, and ability to make DNS requests about that subnet are removed. When a peered VPC relationship is removed, then all other information and rules about it are removed as well.

Currently, a VPC can only be peered to one other in this fashion. See also the discussion of future directions for VPC Peering and VPNs.

This section describes the behavior of an instance with respect to the network.

From here on, all APIs are rooted under the VPC, so there is an implicit /projects/{project_name}/vpcs/{vpc_name} at the start of every route.

See also: Future Looking APIs for Firewall Logging and Future Looking APIs: Multiple Firewalls.

See also: Future Looking APIs for VPNs.

See also: Future APIs for Load Balancers.

VPCs are represented by an object. When a project is created, a VPC is usually created automatically. Every VPC contains the following fields:

A VPC Subnet are represented as a JSON object with the following fields:

The VPC Route rules are represented as a JSON array of route objects. Here’s an example of a set of rules that might appear in a VPC Custom Router’s routes:

The body of each rule has the following fields:

In the destination member, the Target Strings have the following semantics:

In the target member, the Target Strings have the following semantics:

The VPC Firewall rules are represented as an array of JSON objects. Here’s an example of a set of rules:

Each rule has the following fields:

IPv6 addressing marks an opportunity for a major departure from how IPv4 is often treated in cloud deployments. Because IPv4 addresses are scarce and there is a limited set of both public and private addresses, all of the major cloud providers have virtualized IPv4 addresses that customers use for communication. In addition, they use a 1:1 NAT for mapping Internet-accessible IPv4 addresses to an internal address.

With IPv6, these same constraints aren’t necessarily required. AWS is using their large IPv6 address space allocations to give every VPC a public /56 which is split into explicit /64 allocations for a subnet. The implication is that for those IPv6 uses, there is no difference between whether an address is considered public or private. Rather, it is all about the firewall posture.

A related challenge with this model is the question of how does a floating IP work. Critically, AWS does not support Elastic IP for IPv6. This makes sense, given the design that they’re using. AWS likely is not doing any kind of virtualization games here, instead they’re just using normal routing and, ACLs, and firewalling.

While the current API proposes one solution to this, there is a related thing that we need to consider, which is how we map addresses to and from one location. The advantage of the current NAT approach is that it doesn’t require any adjustments in the guest. However, this means that it’s also not possible to have multiple external IP addresses and distinguish them. An alternative approach that would require more work in the guest would be to route those addresses to the guest directly. This relies on the use of addresses outside of the ULA space.

The current API design mimics IPv4 and allows IPv4 and IPv6 to have a similar experience. It also works on the assumption that customers will not have a large amount of publicly routed IPv6 space.

To try and help answer these questions, there is the IPv6 part of the customer questions section.

Currently, VPC Router destinations are mostly described as subnets. As we have other resources that might dynamically expand to more than one IP block (for example a default subnet with IPv4 and IPv6), it may make sense to allow destinations to be input that way. If we did this, that would mean that the destination would include the following:

The main things that this approach raises are questions around things like:

On the other hand, it would solve some potential problems:

I’m not sure there are many customer questions that I would ask to help us clarify what direction to proceed down. My inclination is probably to add support for it in the user API.

The scalability of an Internet Gateway and its design from a user perspective is an interesting challenge. Here are some of the goals and constraints that we have, some of which are in tension:

I don’t have concrete other API proposals; however, this is what has led to the current set of thoughts. However, as we evaluate what makes sense and what doesn’t, this may be useful to keep in mind. There are a bunch of trade offs between scalability by default and having a dedicated set of IP addresses.

The ability for an Internet Gateway to have no IP addresses associated with it feels like a bit of a kludge. While it does simplify the router / floating IP interaction in some ways, I wonder if it makes it more complicated at the same time. It’s not clear to me what percent of instances will want to have all outbound Internet access cordoned off, but also have a floating IP address associated with a single instance in a given subnet.

The current design requires that a router entry be present in either the system or a subnet’s VPC Custom Router for an Internet Gateway. The current view is that if there is no entry for an Internet Gateway the instance cannot route to the Internet. It’s not clear how much this matches what someone would naively expect or not.

We could consider having this bypass the Internet Gateway and associated policy altogether, though we still have to honor the VPC Router for other aspects (for example a deny rule for a CIDR block). Ultimately, I think it’s clearer that we do require it so that way thing are consistent. Even if it does introduce a few oddities for the Internet Gateway discussed above.

At the moment we’re only suggesting supporting the A, AAAA, and PTR records in our DNS systems. There are a few other record types we should look at:

The DNS scheme for records that was proposed is somewhat verbose. The following were the goals that went into this design:

The current scheme is designed as:

By using the organization, project, and VPC, we can avoid the issue of whether or not duplicates exist. While it’s easy to enforce that there are no duplicates for a single user, when we start to bridge multiple projects and VPCs together, that becomes much more difficult and cumbersome.

Currently, the defined name spaces are for instances and Floating IPs. These take the forms <instance>.<az>.inst and <name>.fip respectively. If we were to integrate load balancers or service names into DNS, then those could have their own scheme. For example we could see all of them as:

This currently suggests that we want to include everything as below the VPC for resources. That may be reasonable, given that this is the granularity for networking.

To reduce the friction of names, there are a few things we could do. For example, by default, we can set the DNS search domains in an instance to things like: inst.<vpc>.<project>.<org>.<suffix> or <vpc>.<project>.<org>.<suffix>. This will give us good ergonomics. In addition, because of things like VPC peering, being able to have longer names allows us to actually expose DNS names to all members of the peered VPC.

We will inevitably hit the point where the wrong IP address is assigned to an instance or an instance wants to have a network interface added or removed. These two are not the same, though have some related challenges.

Today most virtual network interfaces whether they are based on virtio or SR-IOV do not generally show up as a hot-pluggable device. Most networking device drivers have not been made to be hot-pluggable or designed around surprise removal, unless working with a physical USB to Ethernet adapter, which we would not want to emulate for numerous reasons. That means that the only good way to remove an interface from a guest is basically to reboot it.

To that end, GCE explicitly doesn’t allow changing the number of interfaces for an existing instance. Amazon does.

Adding and removing IP addresses from instances is much easier mechanically, but there is still coordination inside of the guest. Because most addresses are assigned via DHCP or statically, someone has to go and update that configuration or trigger a DHCP renewal from the instance. Depending on the amount of machinery that we add to guests, this can be made fairly automatic. Alternatively we can set lower lease times for DHCP, but that has its own trade offs.

While we should design the API in such a fashion that adding support for these is possible, we’ll need to think carefully about the implications of each of the paths and work through that in the design.

Currently, all networks default to an MTU of 1500 bytes. While this is the most common MTU across the current public Internet, having a higher internal MTU can reduce the overhead for reaching higher-throughputs. On the other hand, when different networks have different MTUs, that can cause a lot of hard to debug friction, especially due to challenges around Path MTU discovery.

Currently, we include the MTU of subnets and have a default MTU in the top-level VPC. We can make it easy to change these going forward if we’d like as we explore things with customers. Some questions to help us better understand this are in the MTU part of the customer questions section.

To help us understand the open IPv6 questions discussed in IPv6 Addressing, here are some questions we might ask customers:

While Oxide is not looking to be a public DNS registrar, there are other ideas that we can adopt that might make life easier for application writers. Especially in the face of not having a load balancer for version 1. These ideas were developed originally by Alex Wilson as part of the Joyent Triton Container Naming Service.

The more interesting thing that it did was to provide a bunch of DNS SRV records for services. Services could register an SRV record by pushing a metadata tag out. Once that was the case, CNS would advertise that as part of the SRV record. It would also do some amount of rudimentary health-checking by paying attention to whether the instance was up or down at an API level and if the host it was on was up or down. There was also metadata that an instance could add to temporarily remove itself from the SRV record.

While this requires cooperation, this could prove useful. As part of building out the Manta object store, SRV records where instances would register proved to be quite useful. Though the fact that instances had to explicitly register was important there for liveliness and could prove a reason that this doesn’t make sense.

A variant of the above is to allow a collection of addresses based on a tag. This would result in a number of A or AAAA answers depending on which instances had that particular tag.

The current design of our DNS record scheme does allow for expansion in this regard and wouldn’t make it too hard to include either approach.

Currently only a single IPv4 or IPv6 address can be assigned to an interface, though multiple interfaces can be assigned to an instance. In the future we should probably allow additional IP addresses to be assigned to an instance.

Today, we have a restriction on the number of VPCs that can be peered together. As we have customers build up system that exist across more than one region, we will want to be able to support peering together multiple different VPCs. The biggest gotchas are to understand what the routing and firewall rules between multiple disjoint things look like. For example you can imagine a love triangle of peered VPCs where A peers with both B and C, but B and C are not peered:

Working through the semantics of this or much more complicated relationships will be important to understand from an ergonomic perspective and also from the overlapping subnet perspective. It may make sense in such a world to allow a VPC Custom Router to pick how to win in the overlapping subnet problem. These are things that we should think about and make sure that the API is future proof for this all.

Based on discussion in RFD 9, we are currently not trying to integrate VPNs in the first version of the product. However, we should consider how they might look.

There are a couple of different types of VPNs that we need to consider that customers are looking for. There are a few different types of VPNs to consider:

While both of these are solvable in instances with software, the first case is one where we can really improve the experience with broader integration into the VPC Router and minimizing the setup issues. Realistically, both of these want different looking API structures. For the time being, we’ll only discuss the first category.

Today, all of the major providers utilize IPsec to create these site to site VPNs through a combination of dedicated hardware appliances form companies like Cisco and Juniper as well as scalable software services. BGP routes are exchanged over these links to allow for high-availability and to minimize the manual configuration that is required as networks are added and removed. While there are other technologies such as WireGuard and OpenVPN, we are focusing on IPsec due to the interoperability with AWS, GCP, Azure, and others.

When an IPsec VPN is created, the system will do the following:

This would fit into the API with the following high-level API summary:

Firewall flog logging is a feature that many clouds and switches have which basically operates as a series of flow logs that include what connections were active, statistics about the connection, and what kinds of connections were dropped.

Here are some considerations for the API:

This would probably cause us to add additional fields to the existing firewall rules objects or other things such as instances for collecting the flow logging.

This could have the following high-level API endpoints for us to consider. These don’t go into all the details that we would like, but at least give us a starting point for how it might fit into the broader API. At a high level one could create a firewall logging session and then mark certain rules with which of the logging instances they belong to.

In the current API design, we have a singular firewall resource that contains all the rules that are present in the VPC. The scope of these rules determines what sets of instances we target. There is a possible future where we would want disjoint collections of rules for a VPC. While I’m not sure where exactly this fits in, it’s worth evaluating how this impacts the API.

Today, in the vein of the global firewall we have a single entry point: /firewall/rules. This represents the system firewall. In a world of multiple disjoint firewalls, this could instead be represented as /firewalls/{firewall_name}/rules.

This means that if we were to make a transition, we would need to pick a name for the primary firewall, perhaps system and then we would need to reserve both system and rules for the names of firewalls and basically rewrite actions on /firewall/rules to /firewalls/system/rules.

An alternative approach would be to say there is an explicit name for the firewall object today and just always have it there, but no means of creating or deleting sets of rules. This would leave /firewalls as more of a general listing endpoint which has a single entry and then we’d have /firewalls/system/ and /firewalls/system/rules. The previously described aspects of firewall logging would all change in an analogous fashion.

I’m not really sure how we would apply these disjoint rule sets to different VMs, but if we came up with something, it would reduce the complexity of adding it.

Load balancing is a flagship feature of many clouds and one of the things that customers care quite a lot. RFD 9 Networking Considerations goes into some rationale as to why it’s not in version 1 of the product, but we can imagine it will be there in a subsequent version before too long.

While a whole separate discussion is needed to describe the actual features of load balancers that are worth having (L4, L7, TLS, health checking, forwarding vs. terminating, etc.) there are a few things that we want to tease out as part of the current API design.

From an API perspective, the load balancer pretty easily fits in as a project-scoped service, as it can potentially operate between VPCs. Here are examples of where it would fit in, with an implicit /projects/{project_name} on the endpoints. However, you could also replace the /net with a specific VPC scope of /vpcs/{vpc_name}. Note the APIs here are completely hypothetical and would need to change based on the service we actually want to create: