RFD 363 Minibar

Oxide identified five specific manufacturing test capabilities Minibar needed to allow programming stations from being able to program, test, and lock sleds in a single step:

As Minibar’s feature set developed, Oxide realized that Minibar’s backplane breakout capabilities would also enable easier compute sled development and debugging in a lab environment. Users could use Minibar to connect commodity network equipment to the sled’s management network links, communicate with the sled’s Ignition target without having to use a Sidecar rack switch, and test sleds with different PCIe add-in cards. However, to support Socket SP5 compute sled development, Minibar needed two additional capabilities:

Oxide plans to leverage these capabilities to develop system software for future Ethernet ASICs and PCIe accelerator cards.

The programming station needs to test that the sled’s Ignition target is functioning properly and confirm that both Ignition LVDS links are functional up to the sled’s backplane interface. The easiest way to perform this test is to put an Ignition controller in Minibar.

Minibar’s FPGA implements two channels of an Ignition controller and connects to both of the sled’s Ignition LVDS links. The Ignition controller can be accessed over SPI from Minibar’s service processor.

Initially, Minibar’s Ignition controller will have the same functionality as Sidecar’s Ignition controller. Eventually, however, we intend for Minibar’s Ignition controller to be able to send arbitrary Ignition packets so we can fuzz the Ignition target. However, we do not currently have a roadmap for when this will be implemented.

Minibar breaks the sled’s PCIe Gen3/Gen4 x4 backplane interface out to a 75 W PCIe Gen3/Gen4 x16 PCIe slot as shown in [figure_pcie_breakout]. Although the PCIe interface still only operates in x4 mode, the x16 PCIe slot allows users to plug in full-size PCIe add-in cards.

Minibar routes the PCIe data lanes directly from the sled to the CEM connector; however, the PCIe auxiliary signals (PERST#, PRSNT#, and PWRFLT#) and the PCIe auxiliary I2C interface pass through Minibar’s FPGA. In normal operation, a PCIe Gen3 x4 Ethernet NIC is installed in Minibar’s PCIe slot. The PCIe Ethernet NIC acts as a PCIe endpoint for link testing (i.e., did the interface come up in Gen3 x4 mode at 8 GT/S?) and allows the programming station to connect to the sled’s host CPU over PCIe to load the host OS onto the internal M.2 SSDs. Minibar’s FPGA passes the PCIe auxiliary signals back and forth between the sled and the PCIe slot, and allows users to observe and control the PCIe auxiliary signals in real time for debugging.

The compute sled generates the 100 MHz PCIe reference clock and sends it to Minibar over the backplane interface. In the Oxide Rack, the PCIe reference clock is ac coupled and terminated at the clock generator in the rack switch. To keep the electrical interface as close to the Oxide Rack as possible, Minibar ac-couples and terminates the PCIe reference clock, then uses a PCIe Gen4 reference clock buffer to redrive a DC-coupled HSCL reference clock to the CEM slot.

[figure_management_network_block_diagram] illustrates Minibar’s management network switch archtecture. The two SGMII management network links from the sled and the two SGMII management network links from Minibar’s SP are routed to a VSC7448 switch, which uses a VSC8504 PHY over QSGMII to break out the three Ethernet links to three 100BASE-T ports - one for each of the sled’s two links, and one for Minibar’s SP. Like in Sidecar, the VSC7448 is responsible for isolating Minibar’s SP from the sled’s SP.

Using a VSC7448 switch makes the management network interface between Minibar and the sled as electrically similar to the Oxide Rack as possible and requires the smallest amount of software development, but it adds an extra $150 to Minibar’s cost per board. We believe the small cost increase per board is justified given Minibar’s small projected build quantities and the need for the management network architecture to work on the first spin.

Note that both of the SGMII links from Minibar’s SP are connected to the VSC7448, even though Minibar’s SP is only given a single BASE-T Ethernet port on the back panel. The decision to wire up both of the SP’s SGMII links was actually motivated by software - it allows Minibar’s SP to reuse the same KSZ8463/VSC8562 configuration as any other SP without us having to go in and disable one link. We can simply have the VSC7448 - which already needs an updated configuration - blackhole one of the ports. It’s also ea

The Minibar PCB wires the sled’s two 200G/100GBASE-KR4 Ethernet links in a loopback configuration as shown in [figure_ethernet_loopback]. If the host OS checks the T6 link status and both links are up at full speed, the sled’s BASE-KR4 Ethernet links are working correctly up to the sled’s backplane interface.

Minibar’s production tester configuration mimics a single cubby partition in an Oxide Rack, including the blind-mate backplane. Compute sleds slide into the Minibar cubby and lock in place with the same latching handle and hardware users would encounter in the Oxide Rack. The Minibar electronics are installed in the back of the cubby and connect to the compute sled using the same floating backplane cartridge and backplane cables used in the Oxide Rack. Instead of more backplane cables, Minibar’s rear panel has a power button, a power connector, three Ethernet ports, and a PCIe slot.

Minibar’s cubby and enclosure are constructed from extruded aluminum framing and 3D-printed plastic parts rather than a bulk sheet metal design to minimize design time, cost, mass, and tooling NRE. A fully-assembled minibar production tester measures 1.07 meter long, 305 mm wide, and 135 mm tall.

Driven by the need for a blind-mateable interface and the desire for a long-term tester solution, the signal and power interconnect between the Minibar tester PCBA and the compute sled PCBA occurs via patch cables. This supports a longer-term use of the tester PCBAs, as the Samtec Examax connectors used are only rated for 250 mating cycles. Without an intermediate cable to act as a sacrificial link in the interconnect chain which can be rotated or replaced as part of periodic tester maintenance, the tester PCBAs would conservatively only last long enough to build approximately 5 full racks, assuming 1-2 mating cycles per test. While it is unlikely that any lab-use versions would see this number of mating cycles, the desire for a singular PCBA and interconnect design drives this decision. Additionally, the use of a flexible mating mechanism allows flexibility in placement in the lab-use designs, which should allow for a design with a smaller overall footprint.

At over 1 meter long, Minibar manufacturing testers were too large to fit on the shelves in Oxide’s lab, so Minibar’s lab configuration, called Minibar Lite, packages the Minibar electronics in a 3D printed enclosure that mounts on top of the compute sled. Minibar Lite does not blind mate to the compute sled; instead, users manually install the power cable and backplane cables and secure them with 3D-printed retention clips. Additionally, the overall materials cost of this configuration is significantly less than the production version.

Minibar contains two physical controls to help prevent compute sleds from being damaged during programming and testing:

Minibar provides two physical controls to help prevent operators from coming into contact with the portions of the compute sled and Minibar electronics that operate at the bus voltage: