RFD 58 Rack Switch

The function of the SP control plane traffic is to allow for reliable low level control over network and compute node system board primitives and provide services (if necessary) needed to bootstrap the main network data plane. This traffic is traditionally referred to as out-of-band (OOB) management.

Given the limited processing capabilities of the SP when compared to the host CPU, this traffic has the following characteristics:

In order to facilitate this traffic the rack switch is expected to provide:

Given the critical nature of this traffic it is expected to be available first and disrupted last.

All remaining control plane traffic which does not fit the SP Control traffic criteria is considered to belong in the Host Control Traffic class. This includes traffic for higher level control plane primitives such as data replication for control plane state, API services, larger volume telemetry, etc. This traffic may cross the boundary between Oxide racks and availability zones. As such it is expected to have the following characteristics:

In order to facilitate this traffic the rack switch is expected to provide:

The configuration and state management of Host Control networking facilities is likely to depend on services provided by the SP control plane.

The Tomahawk3 (TH3, [tomahawk3-brief]) is Broadcom’s current high performance, full-featured, fixed-function switch ASIC, aimed at data center networks. It supports up to 64x200G, or 12.8Tb/s of traffic. Its feature list is long, including support for the VxLAN and Geneve encapsulation protocols.

We have performed a napkin design exercise to determine if this ASIC can be implemented in the way we would like to, using an external PCI Express link to a compute node and with support for attestation of its firmware. As far as we have been able to determine this is possible, but Broadcom has not been willing to provide feedback on this design without us purchasing a substantial block of engineering support hours. The napkin can be found here.

While this chipset would certainly meet our needs, our experience interacting with Broadcom makes it clear that while they are happy to sell us silicon, they have little interest in supporting us. In no uncertain terms have they suggested we stick with the tried and true, working with one of their design/manufacturing partners. While this is not necessarily bad advice given what they know about us and where we are in the process, this attitude does not match the expectations we have for the relationships with our primary technology partners.

The Barefoot Networks (recently acquired by Intel) Tofino 2 (TF2, [tofino2-brief]) ASIC is a very different device from the TH3. Where the TH3 is fixed-function, the Tofino ASICs are programmable using the P4 programming language [p4-wikipedia]. This allows for switch functions/features to be determined by the integrator (us) rather than the ASIC designer. It should be evident that this is hugely appealing to us, as it will allow us to enable/deliver features in future releases of our software stack. We could even go so far as to tailor switch/data plane features to specific (classes of) customers, when appropriate. With support for 64x200G this would mean the switch may be used in multiple design generations of our racks while providing the ability to adapt to changing customer requirements in the future.

It should be noted that Broadcom provides a device with similar capabilities in the form of the Trident4 [trident4-brief]. It was suggested however that the TH3 would be better suited to our needs and we were consequently not provided with documentation.

Where Broadcom has not made much of an attempt to win our business, the opposite is true of the Barefoot Networks (BN) team. They went out of their way to pitch Tofino during early meetings with Intel, positively surprising us in the process. They have consistently been prompt to answer our questions, provide presentations and demos of Tofino’s design details, development environment and have provided engineering support to help with a similar napkin design exercise to determine if Tofino can be made to work using an external PCI Express link to our compute node.

In addition to great support, BN has supplied us with the source code to pretty much everything we need in order to implement Tofino in our product. This includes the full driver stack, run time, compiler extensions to the open source P4 compiler and some working P4 application examples.

Tofino 2 is an appealing device for a next generation data center product like the Oxide rack. Its programmable nature makes that we can deliver functionality over time, tailor to the workloads of specific (classes of) customers through hardware acceleration of future networking applications. The port density and raw performance of the device provides for headroom, which will allow for reuse of the (majority of the) switch design in subsequent Oxide rack generations.

During the OCP Summit 2020 Intel has shown Tofino to be an integral part of their data center offerings. Combined with the support provided by the BN team to date and the fact that we were given the source code to everything we may need, we feel confident we will be able to develop a meaningful partnership in the coming years. As such it is plan of record to use Tofino 2 as the ASIC in our rack switch.

The dedicated network option is a tried and true method of implementing an out-of-band management network. This could be implemented by adding secondary commodity FE/GigE switch ASICs to the rack switch chassis, each of which is connected to every SP present in the rack. This would provide a redundant path for stage 1 control plane traffic.

The alternative to a discrete out-of-band management network is the use of NC-SI. This method uses the NIC in each compute node and effectively turns it into a small switch, providing a Fast Ethernet-like interface through which the SP can connect to the rack data plane. A control protocol is provided as part of the standard for the SP to manage the upstream ports to the data plane and the forwarding filter, allowing it to send/receive traffic.

When implemented using a PCI Express Add-in Card (AIC) NC-SI becomes a somewhat awkward thing which needs to channeled over SMBus. Fortunately the OCP Mezzanine specification provides a more sane method, allowing for a relatively straight forward RMII between the Ethernet MAC connected to the SP and the NIC. The appeal of this solution is that apart from the Mezzanine slot little infrastructure is required on the main board of the compute node and no additional hardware needs to be added to the rack to provide a the stage 1 control plane network. This traffic would simply be carried over separate VLANs within the data plane.

While the promise of no additional hardware is appealing, NC-SI implementations currently in the market have proven to be fragile, riddled with bugs in both the NIC firmware and the BMC client stack. The protocol state machine is complex and vendors have struggled getting the implementation in their firmware to work correctly. This has caused a somewhat negative feedback cycle, with adoption being slow due to problems, but the implementations at the same time not getting the miles and priority required to make them reliable.

Given the importance of this traffic it is of paramount importance we can get this to work reliable for NC-SI to be a viable option. As such the NIC vendor selection will have to focus and to offset risk we are aiming to do the following:

If done well, parts of the work for the evaluation rig can be reused in our control plane protocol stack.

Note for future readers: this section was written much later than the surrounding sections. It is for this reason you will find additional design aspects creep in. We felt it necessary however to include this section for our future selves as this reflects our thinking during the design of this part of the system and to remind ourselves why we decided against the use of RS-485 or a similar serial protocol.

Building an Ethernet-compatible management network has turned out to be harder than we expected, due to:

Given all of this, there are no off-the-shelf solutions that we could merely buy and deploy. If we’re having to assemble something from parts, why does it need to be Ethernet? Why can’t we do something apparently simpler, such as an RS485-style serial protocol, or even CAN? All our SP candidates have two or more UARTs and CAN MACs. We considered this — in many ways it’s very appealing. But there are a number of problems.

First, any communication protocol we designed would need to recreate much of Ethernet. We would need link-layer addressing, switching for peer-to-peer communications, error detection/correction, some amount of rate negotiation for upwards compatibility, provisions for flow control, etc. This is not a deal-breaker, but it is engineering effort, and by using an Ethernet MAC we get most of this for free. (Also, because we’re a tools company, you know we’d have to write our own logic analyzer plugins for our custom stuff; Ethernet analyzers already exist.)

Second, we can’t meet the above requirements by "just" deploying something like RS-485, because while RS-485 in an NRZ encoding is balanced in the differential sense, is not inherently DC-balanced. Our UARTs only do NRZ, and cannot (for example) scramble or Manchester-encode their output. This means it’s hard to AC couple, requiring external isolator devices or more complex coupling circuits. We could balance the signal using an external scrambler or encoder chip, but there aren’t fantastic options for such chips.

Third, the data rates on RS-485 and CAN are both in the neighborhood of 10Mb/s before overhead is taken into account. This is 1/10 the throughput of our proposed 100Mb/s Ethernet and could become a problem for management traffic throughput (particularly on CAN, where the overhead is comparatively high).

Fourth, because we cannot buy a 36+ port switch chip for a protocol of our own devising, the connections at the switch could easily become a bottleneck, magnifying the bandwidth problem by sharing the 10Mb/s across all nodes. (CAN, in particular, expects to be a shared medium rather than a switched network. RS-485 can work either way, but we’d need to build it ourselves.) Whereas the switch side of the SGMII-based solution is relatively straightforward.

Finally, while the v1 SP has a single Fast Ethernet MAC (100Mb/s), other candidate parts such as the NXP RT1170 will become available over the next year. We have the possibility of switching to a v2 SP with redundant MACs and higher management throughput. But these parts' UARTs and CAN MACs are the same speed as our current choice, so by relying on them, we don’t get any improvement — whereas with the SGMII-based approach, we can potentially slot in a v2-based sled and have it magically get more bandwidth, thanks to auto-negotiation. And so an Ethernet-based management network seems like the least-worst option for a number of reasons, and has potential for future expansion that could be valuable within the next few years.

After several rounds of questions with the NIC vendors the team has taken a hard looks at the responses. For most implementations we feel there are shortcomings in the way traffic is separated between the host CPU and attached SP. Furthermore the the implementations require shared ownership and control of the NIC between host CPU and SP and we do not feel confident this can be properly resolved. In response we have chosen to implement a dedicated management network in the rack switch and use blind mating of the additional cabling to overcome and improve on serviceability challenges. More details on this decision are available in [rfd89]. There are some details we would have to work out such as how the switch domains are joined together in order to allow traffic between host CPUs and SPs, but overall this solution is well understood and the individual components carry less inherent complexity.

Assuming the rack switch is provided with standby power as soon as it is mounted in the rack and power is applied to the bus bar, the following minimum set of signals are required to control the different system board functions:

In addition to the signals listed above AMD, Epyc host CPUs require some additional signals to enable hot-plug support. These signals, primarily used to implement hot-plug support for NVMe SSDs, are described in the PCI Express ExpressModule specification. Implementing the Rack Switch Device Interface using the same electrical specification would allow for a single hot-plug implementation on the compute node for these subsystems. This list of signals, used by the AMD host CPU to track and drive the power state of the device, is as follows:

For details on how these signals should be implemented to enable hotplug support, please refer to [rfd66] § 2.1 and [rfd129]. Please refer to [rfd95] § 6.2 for considerations on signal reference, isolation and back-driving.

The PCIe Express ExpressModule specification assumes module power is provided by the host system. It is for this reason the interface provides the controls listed above to enabled, disable and monitor the module power state. The rack switch does not meet this assumption since it is powered independently from the compute node. Wiring these signals into the rack switch power control function and letting the AMD hot-plug machinery drive this directly may cause (part of) the power to the switch ASICs and interface modules to be reset on hot re-attach.

Since it is desired for the switching services to remain available during re-attach (claimed to be supported by TF2) some amount of emulation of these signals will be necessary in order satisfy power-on logic in the hot-plug infrastructure on the host CPU and allow the OS and device driver to take control and manage TF2 re-initialization. This emulation will be provided by connecting the PRSNT#, PERST#, PWREN#, PWRFLT# and SMBus signals to the system Service Processor (SP). Software on the SP will then initiate and manage the hotplug procedure with the host CPU, depending on overall system state.

Tofino 2 devices are available in SKUs with 32+1, 48+1 and 64+1 Ethernet ports. The price for these devices is primarily driven by the amount of (enabled) programmable resources, the number of packet processing pipelines (2, 4) and the number of stages within these pipelines (12, 20), and not the number of Ethernet ports. While most customers are unlikely to utilize all 32 network ports, the cost reduction for buying a device with 48 ports is not significant (early pricing indicates a few hundred dollars per device at most). Similarly the design and BOM cost of providing the extra ports on the front of the chassis should be low, while having these extra ports available will mean that some customers will be able to connect more racks together without requiring additional switches implementing a wider spine for the network fabric. An example:

A customer would be able to connect up to 32 racks in a single cell using a full mesh, requiring 31 inter-rack links and leaving one link available per rack switch as uplink to the customer network/the Internet. Tofino 2 will allow the use of 200GBASE-SR4 optical transceivers once these are available, providing each rack pair with 2 x 200Gb/s of peak bandwidth and 2 x 40/100/200Gb/s per rack to the network and/or Internet. At 32 compute nodes per rack this would be an oversubscription ratio of 1:16 when all rack switches are operational while at 24 compute nodes per rack this ratio is reduced to 1:12. Depending on inter-rack bandwidth requirements of storage and application traffic this ratio may be acceptable.

For cells with less than 32 racks, the inter-rack bandwidth can be improved (half the number of racks means half as much oversubscription per rack pair), network cost can be reduced (because less transceivers are required) and with better network policies and better traffic steering capabilities asymmetrical links and network graphs of different shapes such as a grid or cube would be possible. All this can safely explored at a later time.

If a customer desires more capacity than can be provided by the 32 rack cell described above without investing in additional networking infrastructure, racks could be grouped into smaller cells (8, 16, 24 racks each) with higher oversubscription ratios between cells. This would require software support in order to keep applications and traffic local to cells as much as possible. Alternatively or in addition a customer could choose to deploy one or more separate AZs.

For customers requiring larger cells or cells with more cross-sectional bandwidth, our current rack switch design and cabled backplane can be extended to form a new network centric rack type. The cabled backplane should in this case significantly cut down the per link cost, cabling and management burden of interconnecting these spine switches, while the 32 network ports per rack switch would allow for 256+ 200Gb/s fabric ports per rack. At full port utilization this would provide a currently staggering (let’s revisit this in 10 years) 51.2Tb/s+ of cross-sectional bandwidth per network rack, allowing for cells of 200+ racks and 6400+ compute nodes.

The Ball Grid Array (BGA) pinout of TF2 devices with 64 ports match the backplane/network grouping, aiding board layout if ports can allocated to both the front and back of the rack switch. For devices with 48 ports the pins are split 32/16 with one quadrant not connected, while 32 port SKUs have all 32 ports allocated on one side of the device.

The primary data plane of the rack switch consists of the Tofino 2 ASIC connected to pluggable transceivers using an interface popularly known as QSFP28 and specified in [sff8665]. Each rack switch is expected to be connected using both types of cabling; the compute nodes in the rack will be connected using DAC (albeit with a blind-mating twist), while connectivity to the broader network will be provided using fiber optic transceivers and cabling.

Optical transceivers implemented as QSFP28 modules are active devices which require some amount of active control. Not unlike other high speed interfaces, the electrical interface to these modules can be divided in a low- and high speed portion. The low speed portion consists of an I²C bus and several strap pins for presence, power state as well as status LEDs, while the high speed portion contains the eight differential pairs used to send/receive Ethernet frames. The full interface is specified through a combination of [sff8665], [sff8636], [sff8679].

Of particular note is that while the module implements a standard I²C physical layer signaling scheme, the address of the module is fixed and used as a control word to indicate a read or write operation. This means in practice that QSFP28 modules can not share an I²C bus and instead need a bus segment per module.

[sff8636] specifies a fairly comprehensive register map containing the information needed to control a QSFP28 module. Of this information available there are three pieces of information which are of particular concern to us and will drive the design of the rack switch board:

The challenge managing the interface modules is that multiple entities require access to the information provided via the I²C interface. In particular, identification and status information is to be provided to the control plane, and is needed during network boot and module initialization. The power and thermal information is required at a regular interval by the board control function providing thermal management, while link state information is needed by the ASIC driver to manage the SerDes. The latter two functions may not be running on the same CPU which may lead to contention getting timely access to the information needed.

The implementation of the board infrastructure required to power and manage up to 64 interface modules can be done in one of several ways, each with different trade-offs. This section is an attempt to replicate the formula found in [rfd88] to enumerate and analyze these options in order to pick an implementation which meets requirements while adequately constraining complexity. For this discussion the principles outlined in [rfd88] § 2 very much apply and are recommended reading.

Without further ado, the following is a short list of possible options:

Option (1) carries the least amount of complexity in the hardware, but imposes limitations on other parts of the system. In particular, it provides no way for any logic on the system board to read interface module temperature data, making the host CPU and driver an active part in chassis thermal management. Given the thermal management reliability requirements and the latency of this data path this is not desirable. Of secondary concern is the use of I²C switches to provide each interface module with its own segment. These switches are prone to bus error states, requiring recovery logic in software. While seemingly sufficient at first glance, this option limits the way a rack switch can be managed.

The addition of Gemini to the rack switch as proposed in (2) significantly changes the system itself. It allows for board functions such as power sequencing and thermal management to be handled autonomously by the SP, but comes with the consequence of having Gemini be part of the boot sequence and having to maintain and manage what will most likely be a different build from the SP found on the system board of a compute node. With that said, having an SP on the system board will make for a more appropriate solution to managing the interface modules.

Option (2)(a) is similar to (1) in that it is an attempt to solve for the constraints using minimum hardware complexity. What makes (2)(a) strictly better compared to (1) is that the SP will assume control of the network interface modules, allowing it to regularly poll power and temperature data in addition to forwarding module presence, identification and status data to the control plane. This is in line with the principles found in [rfd88] § 2. Any data and LOS interrupts required by the host CPU and driver will need to be forwarded by the SP over the SMBus in the PCIe link, which may pose a bottle neck depending on what is required by the driver. Finally the SP has a limited number of I²C peripherals and IO pins available which means that one or more layers of I²C switches may be required to implement this option. This carries similar concerns over I²C bus errors as in (1).

Option (2)(b) is an attempt to address I²C bus reliability by replacing the I²C muxes with two FPGAs which each implement a SPI-to-QSFP28 bridge. With the SPI bus most likely being significantly higher bandwidth than the I²C segments, this will allow the SP to address up to 32 interface modules more or less in parallel addressing possible concerns around throughput, data and LOS interrupt latency and bus reliability. Because a higher bandwidth path now exists between the network interface modules and the SP, it can prepare/pack data more efficiently when sending it to the host CPU and driver over the PCIe SMBus. This solutions comes at a cost of adding two FPGAs to the system board and require soft logic to be developed and maintained for the SPI-to-QSFP28 bridge. Placing the FPGAs on both sides of the system board should make routing of signals easier, since serial to parallel fan-out happens closer to the interface modules.

Finally option (2)(c) trades technical complexity for the flexibility of having interface module data independently available with low latency on both the SP and host CPU/driver, allowing management tasks to be allocated where they most naturally fit. This flexibility comes at the price of having to implement PCIe on the FPGA and adding a PCIe switch between the host CPU and components on the rack switch system board. This latter part is not necessarily bad is it may help hot-plug and hot re-attach by allowing the SP more control over the PCIe link. And depending on the placement of this switch, it may replace a re-timer since the switch will perform the same function. (2)(c) does pose some tension with respect to [rfd88] § 2, in particular the principle of "Single point of execution". The FPGA X-to-QSFP28 bridge would need to implement some amount of arbitration in order to manage requests from both the SP and host CPU/driver and some care must be taken in both the software running in the SP and the driver not to work against each other.

The following is a list of design dimensions which need to be considered:

The requirement for a 10G port in order to connect to Tofino 2 combination with the number of ports required for compute nodes eliminates most switch ASICs readily available for the workgroup/industrial market. The following is the short list of implementation options:

The two major implementation options are quite different. This section is an attempt to capture and analyze these differences on a number of dimensions:

After discussion we have decided to adopt (2)(iv) as our PoR for the dedicated management network. The VCS7444-02 from Microchip, while more powerful than we need and arguably a bit pricy, is a solid choice for our use case and requires the least amount of PCB real estate in the rack switch. Given it is targeting the industrial Ethernet market and it being a relatively recent addition to the Microchip lineup it should meet the five to ten year life of our rack switch product.

Furthermore we feel that the use of SGMII as a signaling standard in our backplane for this network is an appropriate choice when considering risk of implementation and availability of parts in the future.

Finally we opt for a combination of a Microchip KSZ8463FRL and Microchip VSC8552 or VSC8562 to provide the required RMII interface to the SP and SGMII interfaces to the backplane for both the compute node and the PSC. A final selection between those two PHYs will depend on a requirement for IEE 1588 support. The KSZ8463FRL and VSC8552/VSC8562 will be connected using their 100FX SerDes, steering us clear of having to use transformers or use capacitive coupling of copper PHYs. This configuration is more involved (and expensive) than we would like, but since this implementation detail is limited to the compute node and PSC respectively, it can be improved in future product if/when more SP capable MCUs with dual GigE MACs and possibly native SGMII (fiber) support become available.

The use of all Microchip parts for this portion of our system is on purpose with the intend that if problems were to arise we only require the support from a single FAE team.

While the chassis and environmental conditions for the rack switch are somewhat different from the compute node, it is possible to reuse most of the design for the latter. This implies the use of the SP in order to implement the thermal control loop, but doing so is supported by the following:

The fans and fan controller used in the compute node can most likely be reused in the rack switch chassis. The PoR fan is a 48V 80mm model, which fits in the 2OU available to the compute node. Alternatively and if able a 92mm model may be possible, allowing for more efficient cooling at the cost of reusing this part. Depending on cooling capacity we may need more than the three dual rotor fans than used in the compute node, but if needed an additional fan controller can be added to accommodate these additional channels.

The Tofino 2 ASIC does not have features to allow more fine grained control over power consumption. Parts of the device (SerDes, redundant packet processing pipelines) may be put in a lower power state as a result of lower resource utilization, but it does not provide frequency throttling levers. In order to avoid device damage as a result of thermal runaway it will be wise to add a discrete failsafe in the form of a temperature sensor with an alarm function connected to the on-die diode, which can be programmed during system boot with a temperature threshold near the max junction temperature and raising an alert signal if tripped. This signal would be used as an input to the ASIC power rails sequencing logic, providing a last resort low latency emergency shutdown. With the fans not connected to these rails, this should allow the system to quickly regain positive environmental control in the chassis.

One of the goals of using 3OU for each sidecar was to enable the use of 80mm fans for cooling and to allow us to reuse the exact same fans that we’re currently slated to use in Gimlet, which are dual-rotor, counter-rotating 80x80x80mm fans ([rfd114]). As we believe fans are a component that has a reasonable failure rate due to mechanical wear-out ([rfd132]) and servicing the Sidecar chassis directly is quite complicated, we opted to make the rear fans designed as a toolless CRU.

In other words, someone should be able to insert and remove the fans in an independent fashion without having to power off, otherwise disrupt service to Sidecar, or perform any other actions (such as removing a backplane cable or a transceiver for some reason).

While hot-serviceable fans make it easier for a customer to perform the service and reduce the impact of it, this also means that there is a corresponding responsibility to finish the service action quickly. Missing one or two rotors of cooling may cause thermal degradation, the exact amount of which is still subject to thermal modeling. While Gimlet combines all fans into a single cold-serviceable unit ([rfd132]), because the fans are hot-serviceable, they should be independently serviceable to minimize the impact to Sidecar.

The front panel of Sidecar presents both a design challenge and opportunity. The front panel is full of QSFP transceivers and fiber optic cables. If a sidecar has to be serviced, this is one place where absent us doing something, it’s going to be challenging for customers to make sure that all the fibers and transceivers are returned to the same place that they started.

When it comes to networking, LEDs have been historically used to indicate both whether a link is up, the speed at which the link is up, and then whether or not there is activity on the link. Most existing switches have some number of LEDs per transceiver port to indicate this. We take a slightly different approach to LED management in an attempt to maintain actionability (see [rfd77] for the rationale).

As a result, we stick to using a single, monochrome LED for each individually serviced component in Sidecar. This LED is used to indicate component health and does not indicate activity. Explicitly we want to have the following LEDs: