RFD 538 Webhook API

Events are categorized into event classes. These classes indicate the resource scope that an event relates to, and what event occurred. An event class is a string consisting of one or more segments, with multiple separated by . characters. Each segment describes the category of the event with increasing specificity, with the first segment being the most general. These classes form a hierarchy of event types. Webhook receivers subscribe to one or more event classes to determine what events they will be notified for.

Top-level categories in the event class hierarchy will represent broad categories of entities, followed by increasingly specific subcategories.

Each event that occurs within the system is uniquely identified by a UUID ([RFC9562]). These UUIDs are included in the event payload sent to webhook receivers. If an event results in notifications being delivered to multiple webhook receivers, the event UUID sent to each receiver will be the same. This may be used to correlate events across multiple receivers, and to de-duplicate repeated deliveries of the same event.

Webhook payloads are signed with a HMAC digest ([RFC2104]) using secret keys shared between the webhook receiver and the Oxide control plane to allow the receiver to verify the authenticity of the payload. In a creation request, a user submits a list of secrets they would like to have the webhook dispatcher use for signing payloads. Each secret will be assigned an ID generated by the system that is be unique within the webhook it belongs to.

Secrets cannot be changed once created, and can only be deleted. To rotate a secret, a user first creates a new secret, verifies that their application can validate the new signature, and then deletes the old secret. When a secret is deleted, the value is be erased and the secret is marked as deleted.

While users can ignore signature verification in their receivers, it is required that a webhook configuration always has at least one secret.

See here for the API endpoints used to create, view, and manage the secrets associated with a webhook receiver.

For a webhook receiver to receive an event, the user registering the receiver must have the viewer role for the resource the event relates to. Fault management alerts and other hardware events require the fleet.viewer role.

Currently, webhook receivers may only be created or modified by users with the fleet.admin role, and all webhook receivers operate with the fleet.viewer role. Webhook receivers with more restrictive access control roles may be implemented in future Oxide system software, as described here.

All webhook delivery requests are structured as described in this section.

If a webhook notification is not successfully delivered, the dispatcher will attempt to deliver it again, in order to ensure that the receiver receives the notification.

There is no guarantee about the order in which messages will be sent to a receiver. At any point in time, a dispatcher may be have multiple in progress requests to a receiver. Each request may succeed, fail, and retry independently, resulting in messages appearing out of order in the context of the receiver.

Webhook messages always include a timestamp header (indicating the time at which the message was sent) to allow the receiver to reconcile the order of receipt with the order of sends. This header only provides the order in which the messages were sent, and does not make any claims about the order of the underlying events. Most event payloads will additionally include timestamps in the message body describing the time at which the event was observed.

Liveness probes are HTTP requests sent by the webhook dispatcher to a receiver endpoint to determine whether it is available.

These requests do not represent notifications for actual events, and should not be handled as events by the receiver implementation. Instead, probe requests are used to determine whether the receiver endpoint is currently available and could receive an actual event notification if one were to be sent.

Liveness probes have a number of uses:

Liveness probes are particularly valuable when a webhook receiver subscribes to infrequent but high-priority events, such as alerts. When the system is operating normally, no alerts will be generated. If probes are not sent to a receiver endpoint while the rack is operating normally and no faults have been detected, the receiver endpoint could become unavailable at any time without being detected. Should a fault then be detected in the rack, the dispatcher will attempt to deliver an alert to the receiver, but if the receiver is unavailable, the alert will be lost and operators may be unaware of the fault.

Liveness probes are event delivery requests with the "probe" event class. A probe request may be sent to a webhook receiver using the POST /webhooks/{webhook}/probe API endpoint.

The receiver endpoint must respond with a HTTP 2xx status code to indicate that it is available. If the webhook dispatcher cannot connect to the receiver endpoint, the request times out, or the receiver responds with a 3xx, 4xx, or 5xx status code, the probe is considered to have failed, similarly to the failure criteria for event delivery.

If a receiver endpoint is unavailable, events dispatched to that receiver may not be received. The webhook dispatcher will retry failed delivery attempts up to two times, with a one-minute backoff before the first retry and a five-minute backoff before the second retry. However, if a receiver is unavailable for long enough to miss both retry attempts, the webhook dispatcher will not attempt to deliver that request again unless explicitly asked to by the receiver.

Therefore, it is recommended that receiver endpoints request that any failed deliveries be resent when they start up. This way, if the software implementing a receiver crashes and is restarted, it can trigger a new delivery attempt for any events that it may have missed due to the crash. The simplest way to do so is for the receiver to call the POST /webhooks/{webhook}/probe API endpoint with the ?resend=true query parameter to trigger a liveness probe request to itself. When such a probe request succeeds, the webhook dispatcher will then resend all events which have not been delivered successfully to that receiver.

Alternatively, if more precise control over which events are resent is required, the GET /webhooks/{webhook}/deliveries API endpoint provides a list of event delivery attempts to a webhook receiver. To list only failed delivery attempts, the query parameters ?failed=true&pending=false&delivered=false can be added to requests to this endpoint. The receiver may then request re-delivery of specific events using the POST /webhooks/{webhook}/deliveries/{event_id}/resend API endpoint. This is more complex than triggering probe requests, but it may be useful in situations where the receiver only wishes to resend a subset of failed events. For example, it may only request re-delivery of events that occurred within a particular time window, or it may only resend events that were not processed by a redundant receiver subscribed to the same event classes.

In addition to checking for failed delivery attempts on startup, receiver implementations may periodically attempt to resend failed deliveries while they are running. Even if a receiver endpoint has not crashed, event deliveries may have been missed due to network connectivity problems or other transient issues.

Liveness probes may be used to monitor the health of a receiver endpoint. An external system which periodically triggers liveness probe requests to a receiver using the POST /webhooks/{webhook}/probe API endpoint can detect situations where the receiver is unavailable and alert operators to the outage. This allows receiver failures to be detected and resolved proactively, reducing the likelihood of missing notifications for important events.

While sending requests directly to the receiver process can detect failures where the receiver is completely offline, it does not exercise communication between the Oxide control plane and the receiver. Outages may still occur when the receiver endpoint is online and capable of processing requests if it is not reachable by the Oxide rack’s webhook dispatcher. Calling the POST /webhooks/{webhook}/probe API endpoint requests that the webhook dispatcher send a request to the receiver, which will fail if the Oxide control plane cannot communicate with the receiver.

It may be valuable for a monitoring system to both trigger liveness probe requests from the webhook dispatcher using the POST /webhooks/{webhook}/probe endpoint and to send its own probes directly to the webhook receiver. This provides separate signals for whether the receiver endpoint is running at all, and for whether it is reachable by the Oxide control plane. Operators can then reason about whether an outage is due to the webhook receiver process not running at all, or due to a network partition between it and the webhook dispatcher.

If a receiver endpoint is unavailable, webhook events may be missed. Therefore, when webhooks are used to provide notifications of critical events such as alerts, operators are encouraged to run multiple independent webhook receiver endpoints subscribed to the same event classes. When multiple receiver endpoints are subscribed to the same event classes, the same events will be delivered to all receivers. Therefore, event UUIDs should be used to de-duplicate events that are delivered to multiple receivers, the same event UUID is used for each receiver that receives the event.

Tolerance for delays in event delivery due to receiver downtime is an important tradeoff to consider when selecting these mechanisms. When only single receiver endpoint is used, checking for failed deliveries when that receiver starts up will ensure that deliveries missed during receiver downtime are eventually processed. However, during the period of time when the receiver was unavailable, no events will have been processed. In contrast, if there are multiple redundant receivers, events will still be received immediately as long as at least one receiver is available. Therefore, if receiving events in a timely manner is important (e.g. for high-urgency fault-management alerts), consider the use of redundant receivers.

Nonetheless, there are some failure scenarios in which even redundant receivers could miss events. An outage may impact all redundant receivers, if they are running in the same failure domain, or if a network partition effects all communication to and from the Oxide rack. Therefore, even when redundant receiver endpoints are used, checking for failed deliveries periodically and on startup is still recommended if these classes of failures are a concern.

The Oxide webhook API permits multiple shared secrets to be configured for a single webhook receiver. If a receiver is configured with more than one secret, webhook requests will include an x-oxide-signature header for each secret. The value of these headers includes the secret’s ID in addition to the HMAC signature of the payload, allowing the receiver to determine which secret was used to generate that signature.

Multiple secrets may be used during secret key rotations in order to ensure that the webhook payload is always signed with a secret that can be verified by the receiver. When rotating secrets for a webhook receiver, first create the new secret using the POST /webhooks/{webhook}/secrets API endpoint, which returns the ID assigned to that secret. Then, the webhook receiver may be configured to verify signatures with the new secret. During this time, any webhook payload will be signed with both the new secret and the old secret, allowing it to be verified by the receiver regardless of whether it has yet been configured to accept the new secret. Finally, once the receiver is ready to accept the new secret, the old secret may be deleted using the DELETE /webhooks/{webhook}/secrets/{secret_id} API endpoint.

At present, webhooks are used to deliver relatively low-volume events (e.g. user alerts). In the future, as webhook events are emitted by additional subsystems, it may become necessary to implement mechanisms for rate-limiting the delivery of webhook events, in order to reduce the impact of webhook delivery on other control plane workloads. However, this is deemed out of scope for the MVP implementation.

An alternative interface based on HTTP long-polling [RFC6202] could be provided for event delivery.

In a long-polling-based interface, the webhook receiver would act as a client of an endpoint served by the Oxide control plane, rather than the Oxide control plane’s webhook dispatcher initiating requests to a receiver endpoint. When such a request is received by the control plane, the request is kept open until events not yet seen by the client are published to that receiver. When events are published, the Oxide control plane would complete the request with a response representing those events. The client would then send a subsequent request, indicating which events it has successfully received, and the process begins again. The acknowledgement of received events could be implemented via parameters on the long-polling request, or through a separate acknowledgement endpoint.

Unlike the traditional webhook notification mechanism described in this RFD, a long-polling mechanism has the webhook receiver acting as a client of the Oxide control plane. This removes the need to serve a webhook receiver endpoint and ensure it is accessible to the webhook delivery client. It also avoids the need to track receiver liveness in the webhook dispatcher, as the receiver is responsible for initiating all communication between the control plane and the receiver. However, the traditional request-based approach is more commonly used for webhook delivery, so the MVP implementation will focus on that mechanism rather than long-polling. In the future, long-polling may be added as an alternative interface for webhook delivery.

Currently, all webhook receivers are created by a user with fleet admin permissions, and operate with fleet viewer permissions. In the future, we would like to allow webhook receivers to operate with more restrictive permissions, and control what data is included in webhook payloads based on the webhook receiver’s roles. This would permit users with more restrictive permissions to create webhook receivers that only receive data that they are allowed to view.

For a webhook to receive an event, its identity must have permission to access the resource that the event relates to.

A webhook’s RBAC identity must have the viewer role on a resource for the webhook dispatcher to send it an event relating to that resource. Similar to subscriptions, if a webhook does not have any permissions configured, then it will not be sent any events. Roles also limit the data that a webhook is sent.

Delivery of webhook notifications cannot be guaranteed if the receiver endpoint is offline. Continuing to attempt delivery to an unreachable receiver can be costly. Therefore, it may be desirable for the webhook dispatcher to not attempt to deliver events to a receiver that has been unreachable for an extended period of time.

If delivery of an event fails permanently for a receiver (i.e., all retry attempts for that event are exhausted), the receiver could be marked as failed. Receivers could also marked as failed when a liveness probe request fails. When a receiver is marked as failed, the dispatcher will not attempt to deliver events to that receiver until it is marked as available again. This avoids exhausting the retry budget for events while the receiver is in an unavailable state, preventing those events from being delivered successfully.

Liveness probes are still sent to receivers that have been marked as failed. Once a liveness probe request to a failed receiver has succeeded, the receiver is marked as available again, and delivery of events will resume. We may also wish to provide an API endpoint to manually clear the failed status of a receiver.

Once receiver health is tracked by the fault management system, an alert could be generated when a receiver enters the failed state. Of course, a failed receiver endpoint cannot receive an alert indicating its own unavailability. However, if redundant receiver endpoints exist, other replicas could receive an alert indicating that one of their compatriots has gone offline.