RFD 291 illumos GPIO Framework

In illumos, we often have three levels that we have thought about on x86 side traditionally:

The historical power of x86 has been that the success of the IBM PC eventually lead to the point that by the 1990s, even though the original PC was no longer there, everyone was building systems that were "compatible" with it. This meant that there was a common platform for x86 systems and therefore operating systems trying to operate on a new machine generally didn’t require a lot of porting effort (except for working around all the different firmware issues an errata). In contrast, in similar ARM systems (and really until sometime in the 2010s), most systems had a unique kernel for each ARM board. This eventually consolidated with things such as the atag effort, flattened device tree, and the 64-bit ARMv8-A platform adoption of ACPI + UEFI.

In this model though there’s a lot that electrically is different between these systems. And the promise of ACPI is just around discoverability of what’s there. Tools like that and SMBIOS actually are meant to encourage different pieces of hardware to be radically different. While say a computer built by Dell and one built by HP will contain support for the same CPU socket, DIMMS, and PCI Express devices, the actual way the machine is put together will vary wildly. Even within the same product family, this may be true.

On the SPARC side historically, there was another class called the 'Machine' or 'Implementation' which was meant to refer to a particular instantiation of a platform and go into all of the details and specifics of its implementation. While here we had a different unix kernel image for each machine and drivers that were specific to each machine, the machine did build knowledge about specific devices that were on the board around I²C, GPIOs, etc.

The oxide platform is much closer to SPARC in this regard (see [rfd215] for more specifics). We have the benefit of knowing exactly what our machines will look like and what is inside of it. This will help us out a bunch as we’ll get to later in the specifics of the proposal.

If you look at the embedded space, it is also the case that you may have a common "platform" in the sense of say a development board that brings a lot of pins to headers. However, the specifics of how things are used and what peripherals are mapped to what pins ultimately depends in those case on the end user of it itself. With something like a Raspberry Pi, while some pins are well defined, others are purposefully not so you can do other things.

So why all of this for a moment. The important thing here is that while the SoC defines the total number of GPIOs that it has, the programming interface, and related, the question of which GPIOs are in use, are safe to use, or being muxed with other peripherals requires not only intimate knowledge of the particular machine, but can vary from individual system to individual system.

This in turn provides a challenge for us that we’ll get to in our further explanation of our design, but we really can’t assume that toggling any GPIO arbitrarily or changing its mode is safe. This can actually be used to damage a system! The same GPIO on an AMD system could be harmlessly unused or on another system be used as an input and if you change it to an output, damage the pin and driver!

A different set of challenges is that the way GPIOs are configured and the different settings here actually vary substantially based on the type of hardware in question. To make this a little clearer, let’s discuss several different classes of devices. To help make this clear, we have a classic I²C based GPIO controller, AMD and Intel GPIOs, and then also using two ARM based microcontrollers. While illumos does not support or really try to target MMU-less devices, the microcontrollers help show the diversity of GPIO configurations.

Let’s look at what’s required to configure GPIOs for these different devices in turn:

As seen in the discussion of the different GPIO attributes, the design of GPIOs is very different. In addition, the way that interrupts can occur can vary substantially. On x86 these mechanisms often include things such as being able to route to an I/O APIC pin, route to the ACPI related SCI (system control interrupt), and sometimes to an NMI (non-maskable interrupt).

The question of what is usable is something that is ultimately dependent on the machine and more practically, any firmware that may be running on it. The main note from this is that we need to assume and should plan on the idea that interruptibility is going to be a much more complicated thing here.

In particular, when things like the I/O APIC on x86 are used or on other platforms where there are similar smaller numbers of banks available, then it becomes much harder to have well functioning interrupts and the design of such subsystems can become trickier. For example, on most non-Oxide x86 systems, the I/O APIC configuration is owned basically by the ACPI firmware. This leaves us with a few general thoughts:

As discussed in the goals section, there are two main concepts that we have:

A GPIO is the fundamental unit that hardware exposes that has a set of attributes associated with it. As seen in the background section, the actual set of attributes there are large and vary from device to device. Our focus here is on having a uniform way of describing attributes, but acknowledging that attributes are GPIO-specific.

We also want to introduce the idea of DPIO, which is a GPIO that has had its attributes constrained to specific values. At that point, if the GPIO is an output someone could control a single property: whether it is low or high. Similarly if the GPIO is an input, one could only read the current value. It’s important to note that a DPIO cannot exist on its own. It always is created by constraining a GPIO.

These two ideas — GPIOs and DPIOs — are tied together into the various abstractions that we want to create. It’s also worth noting that some of this inspiration comes from [humility] and [hubris]. There, a GPIO is enabled and configured with all of the relevant attributes and then it is easy to use in a way that doesn’t need to think about those attributes.

This effort will create a new kernel framework for managing and using GPIOs and create a private GPIO provider interface for device drivers. Our intent is that the kernel framework takes care of things like minor node management, character ops, interfacing with other illumos subsystems, and related.

Between a provider and the framework a GPIO is identified by a controller instance-specific ID, generally just an unsigned integer. From there, we would take the idea from MAC link properties, basically describing and defining attributes on a per-GPIO basis. The useful bits of the MAC property abstraction come from the idea of the three property entry points that basically allow one to say separately:

Here, we’d adopt something similar for attributes of GPIOs. While there may be an attribute that can be shared across all GPIOs by such as voltage or the human-name name of the GPIO, the vast majority of these are somewhat different and trying to create a global list will lead to a lot of duplication or trying to fit things that aren’t really the same into shapes. Instead, we suggest that most attributes are scoped to the driver. So if say we had say an AMD SP3 specific driver named for the sake of example sp3gpio, things like how pull up’s are configured, drive strengths, and related would be phrased as a property specific to that. If we look at some of the earlier examples this might result in properties like:

The driver would be responsible for describe what the valid values for these are. The form that these take is private to the driver and framework right now. They could be strings, integers that correspond to enumeration values, or a nested structure. We don’t expect users to interact directly with these right now. Instead users would specify strings that would be translated into the corresponding structure by code in userland that corresponds to each driver.

With this in mind, it’d probably make sense to use nvlist_t structures that are passed around internally with each key in the nvlist_t corresponding to an attribute. This also would provide a relatively easy way for us to pass all of the valid values. Simply fill out all possible values as an array of items in the nvlist_t.

The reason for this is we’d love for when someone lists attributes of a GPIO it becomes obvious what they all are ala dladm’s link properties. For example we could see something like:

The above is just an example of what this could be and this is all still up in the air and will be tbd based on what we actually figure out in the internals, but this suggests the main way that these attributes would come up.

Our intent here is introduce a gpioadm(8) command that will be driven by a corresponding library. While we intend to make the command more stable, at this time the library will be private. The gpioadm command will follow in the vein of other commands and have several sub-commands that relate to different aspects of this in a hierarchical fashion. The following describes the tree of commands. Intermediate items would simply list the other things that they can execute:

The intent here is that gpioadm controller list will list all the current controllers and some basic information about the controller itself and other attributes here.

The gpioadm gpio list would list all GPIOs in the system with the ability filter that list based on controller or other filters. The gpioadm gpio list is not intended to also list attributes. This is mostly just to make everything usable. For example, there are maybe on the order of 3-10 attributes per GPIO. And while for a device like the PCA9535 there’d only be 16 GPIOs worth of things to show (times attributes), when we get to say the SP3 use case, there are over 75 GPIOs. So suddenly this makes the list output a lot harder to use if there are a lot of attributes there.

The gpioadm gpio attr list would list the attributes of a single GPIO in a dladm linkprop style. The gpioadm gpio attr set would be able to set a number of attributes for a single GPIO. It’s important to call out that these all work on a single GPIO.

The gpioadm gpio define basically is used to transform a GPIO into a DPIO. This would have the side effect of creating a DPIO character device and allow the DPIO to be named. Once a GPIO has been defined as a DPIO, most of the attributes and related will be frozen until the DPIO is undefined. The gpioadm gpio undefine would be use to remove a dpio.

An idea that is less clear if it is useful right now is to also have a set of dpio subcommands that allow for listing items that are specifically DPIOs (gpioadm dpio list), reading the current value for DPIOs that are inputs (gpioadm dpio read), and changing the output value for DPIOs that are ouptuts (gpioadm dpio set). As we work through this, that’ll tell us what makes more sense here in terms of abstractions and what’ll be useful. Most of the features of reading and setting will be something that can be obtained through gpio attributes normally.

We’ve talked a bit about DPIOs, so it’s worth going into more detail about what these are. Specifically a DPIO is a character device that is managed by the kernel gpio framework. This would show up as /dev/gpio/:name. The name must be globally unique on the system. It has the following attributes:

An alternative approach to using read/write is to use ioctls to get and set this. The advantage here is that they give us a bit more control over interface evolution, but they can be more annoying to use.

While the existing controls are useful for ad-hoc configuration, we expect that DPIOs are how folks will generally interact with the GPIO subsystem for more persistent and programmatic consumption, at least initially. The fact that the creation of the DPIO constrains the GPIO is what makes it much easier to tie into other software and why we only really want to expose the more standard character device this way.

Persistence and platform configuration is an important thing that we need to consider. We want to give folks the ability to control and persist DPIOs and the corresponding set of attributes that we got there. However, a challenge is that this requires a degree of stability around the set of attributes and how we anticipate naming them that may or may not hold initially.

In theory the platform or machine code if it exists could establish DPIOs. While this can’t be done on i86pc automatically by the kernel, this can where the hardware is more constrained and known. In particular, for Gimlet, we would have the platform apply knowledge about what the set of DPIOs should be and have them come into effect early on. Whether this is something that calls into a platform-hook, or is a file that’s delivered to source this from, or something that the driver just knows initially, isn’t quite clear right now. Longer term, what we believe we should do is have the ability to save a set of DPIOs with their attributes and provide a means of replaying that.

One way that this could be done is that we could save all of the resulting nvlist_t attributes for a given GPIO and the corresponding things reqiured to turn it into a DPIO (e.g. the name of the DPIO, the underlying GPIO controller name and instance, etc.). This is something that the DPIO framework could read at some point ala /etc/path_to_inst or the persistent unit addresses.

Currently our intent is to defer this as someone who wants this could write a service with a series of gpioadm commands. This isn’t ideal, but does at least seem a reasonable starting point. We may opt to implement the kernel-based knowledge based on the machine (which only applies to the oxide platform).

As part of implementing this, our intent is to implement the following:

The first major question that we need to answer for ourselves is how dynamic does this need to be? A lot of this depends on the environment itself and how much information you know in advance.

At one extreme here is [hubris]. In particular, there’s an important aspect of hubris that answers this question in a rather nice way: the total set of applications that may exist and run on a device is known at build time. This means that one can, generally speaking, definitively know the total set of peripherals that will be used and the set of pins that they are on.

More or less, unless someone is using [humility] to manipulate things, this means that we can set up the I/O mux in a relatively static fashion and use that. This doesn’t mean that there isn’t any dynamicism in there. For example, in the case of Sidecar and Gimlet we actually have a single I²C controller adjust between multiple distinct sets of pins. However, from an external perspective, what is needed is all a 'known known' at build time. This is one of the key aspects of Hubris and allows for a lot of power simplifications in its design and implementation.

On the other hand, when we consider a traditional multi-process unixy system, this is the opposite of the case. We generally use a single set of packages and binaries to run on an entire class of systems and instead dynamically discover the set of hardware that is there either through self-describing buses ala PCI or through other means such as ACPI, UEFI, device tree, etc.

Here, the set of applications that can run on the system at run-time is pretty much independent (ignoring things like ISA, etc.) from the actual kernel and broader operating system. While there are lots of ways to try and reason about I/O pin muxing or to use knowledge about the specific application at hand and try and create a specific set of rules, there’s also no reason that you can’t have this truly be dynamic as well.

For example, consider a world where a pair of pins can be shifted between distinct GPIO, I²C, and SMBus controllers. In this case, you could take the dynamicism to the opposite extreme and basically say that the act of opening the controller or using it to drive a transaction on the bus is what switches around and activates the I/O mux itself. That is, there’s no reason that we have to set this in advance and can instead let the system actually drive and dictate that.

Because we are designing and thinking about this right now in terms of the broader more unixy systems where dynamicism exists, this is where a lot of our initial thoughts are turned towards.

Intertwined with the dynaicism is the matter of how policy can and should be expressed. Let’s discuss some of the different aspects of where policy comes from:

How exactly this policy should be structured and controller and how it should be implemented from an architectural perspective are a little unclear, but it does suggest that policy may be a discrete, but intertwined piece of the puzzle.

Put differently, while there are drivers and hardware that describes the general relationship between things, it won’t and shouldn’t be able to describe policy itself as that ties into the actual machine implementation and broader hardware platform design, particularly in worlds where you have to deal with platform firmware that may use these.

If you start to manually try to write out all the conflicts for even a simple SoC with very few alternate functions like an AMD Milan CPU, the rules start getting quite complicated to manually write out. From surveying other system design and implementations, an important detail comes out as implemented by the Linux pinmux subsystem (and possibly others): individual drivers should not try to separately ask the question of what overlaps or implement custom selection logic, instead there should be a single implementation of this.

Put differently, we should instead structure this information such that shared logic in the kernel (whether a nexus driver, library-like misc module, something like ksensor, etc.) should deal with figuring out what is overlapping and presenting things to user land. For example, imagine if a tool could spit out the following:

To help talk about this we can actually structure this information as a large series of data tables. Here are the following layers:

Let’s make this a little more concrete. Here’s a JSONish example (used for making this clear as opposed to saying this should be JSON) of structuring this data we’d do for the AMD SP3 UARTs:

With the above declarative declaration we would know that if we CY38 as EGPIO140 then we can’t use uart3 in its basic configuration or uart1 with hardware flow control, though we’d still be able to use a basic uart1. Similarly if someone asked to use uart0 with hardware flow control the corresponding uart2 and GPIOs would all be blocked out.

While the above with AMD only includes a single set of pins that can be used for each peripheral configuration, this is not the case on the H753 and other smaller devices.

While the overall I/O mux part of this RFD is still exploratory, it does suggest that the nexus and more generally the ndi has a role to play here (though this should not be treated as a commitment to that fact). Ideally individual drivers should not need to know about the specifics of the actual pins or other pieces here. Instead, they should just know that a certain set of configurations is available to them and be able to request and hold the system in a given configuration, say while open or while waiting for some other event to complete.

Carrying on our UART example, the parent (e.g. huashan or a similar FCH driver) is responsible for knowing that up to 4 UARTs exist on this system and the different set of configurations that may or may not exist. It would need to program what exists specific to the hardware and know that if we’re on an SP3 system it is these set of pins; however, on SP5 it is something else and if we’re on a client system the number and count of such devices changes again.

This has a complication with pad configuration as the way that pads should be set may vary between different peripherals and it’s not quite clear right now.

However, it’s worth noting that whatever is setting policy is somewhat disjoint from this part of the stack. That is, if you imagine the parent of the UART driver exists on many different AMD systems, whether an i86pc-based or oxide system. However, on i86pc there are different constraints that come from ACPI / UEFI that constrain whether that is even possible to use or not due to the fact that AML may itself tweak and change something into a GPIO as part of its implementation under the hood. That is, what is safe, is a harder question to answer there. Conversely, on an Oxide system we may have a very different policy because we actually know everything that is going on.

While none of this in itself a plan, it does point out several things we need to consider when we dig deeper.

In the short term, we are likely to stay focused on the specifics of Gimlet and then come back and fill this out as we have more experience and figure out what actually makes sense for muxing.