"Distributed sagas" is an appealing way to build reliable systems that correctly track state within complex, multi-step processes and automatically unwind when they cannot complete successfully. Omicron has used them heavily to carry out critical operations like instance start, disk creation, and region replacement. Over the years, we’ve run into a mix of development-time issues and operational issues that have made sagas pretty painful in practice, often resulting in reduced system reliability. This RFD avoids weighing in on whether the benefits outweigh the problems, but seeks to document the challenges as well as alternative solutions (primarily: reconciler-pattern background tasks).
Introduction to distributed sagas
"Distributed sagas" is a design pattern where a complex distributed-systems flow is decomposed into simple actions and undo actions.[1] The constraints are simple to describe:
Both actions and their undo actions must be idempotent.
Actions are allowed to fail. Undo actions must never fail.
Undo actions must be commutative with their actions. If an undo action is executed, then the corresponding action is executed (presumably for the second time), that should result in the same behavior as if the undo action was done last.
The idea is that actions make API requests that change various systems. The saga pattern seeks to guarantee that:
before the saga executes, no changes have been made
while the saga is executing, some changes may have been made
when the saga completes, all of the external services will reflect that either:
all of the saga’s changes have been made, or
none of the saga’s changes have been made (treating "the change was undone" as equivalent to "the change was never made")
A saga execution coordinator (SEC) uses the rules of distributed sagas to execute the actions and undo actions such that the above properties hold. Importantly, the goal is to ensure that the final state of the world (including external services) reflects either all of the saga’s changes having been made or none of them have been made. That’s true even when the SEC itself crashes and restarts.
Catie McCaffrey gave the canonical talk on distributed sagas in 2017 using the example of a travel web site booking a trip for a user. The trip includes a flight, hotel, and rental car. The goal is to ensure that either all of these are booked or none of them are booked. The whole point of the saga here is to ensure that the system does not come to rest with the user having a flight and no hotel, or a hotel and no rental car, etc.[2]
Distributed sagas are very appealing. Consider the process of starting an instance in the Oxide system. This involves:
picking a sled and allocating resources for the instance on the sled
allocating IP addresses from a shared pool
programming various networking services so that the right routing rules and NAT rules are in place
starting the virtual machine itself
attaching disks
and more. If you imagine starting with straight-line code to do this, unwind on failure, and make it all crash-safe, that sounds hard! On the other hand, it sounds pretty easy to turn each of these operations into a small, idempotent action and corresponding undo action. Then you can let the SEC handle figuring out what undo steps need to be executed when. By doing it this way, either the instance is running and all allocations are accounted-for, or the instance is not running and no allocations exist for it. While the saga is running, intermediate states are possible, like an allocation with no instance. But eventually we expect this to converge on either "running with an allocation" or "not running with no allocation".
As we’ll see, there are cases where the saga can come to rest in an intermediate state. When that happens, the saga contract is violated. These violations could be as innocuous as leaked resources or as dangerous as multiple copies of an instance running simultaneously. Generally, one would expect that the blast radius of such a failure is limited to the resources related to the saga (e.g., the instance itself or its allocation on a sled).
Distributed sagas at Oxide
Oxide determined to use distributed sagas for various control plane operations in [rfd107]. We built a framework called Steno to run distributed sagas within Nexus (see control plane architecture docs). Today, sagas are used for a variety of operations:
instance create, delete, update, start, IP attach/detach
project create
VPC subnet create, update, and delete
various storage (disks, images, snapshots, volumes) create/delete
Crucible region replacement (various sagas)
In doing this, we had to address some gaps in distributed sagas as they’re widely described:
Lack of shared state between saga actions and undo actions. For example, an early step for the "instance start" saga is to pick a sled where the instance will run. A subsequent step uses this choice to make a request to the corresponding sled agent. This information has to be persisted (since a key function of sagas is to survive crashes).[3][4]
High availability. The distributed sagas talk doesn’t discuss dealing with permanent SEC failure. We store the persistent log in CockroachDB, so one could imagine a failover operation to another SEC. However, this has all the usual problems around identifying failure (distinguishing a partition from permanent failure). We’ve dealt with this by only doing these failovers when a Nexus instance (the SEC) has been expunged and we know it’s never going to run again.
We anticipated these gaps or ran into them immediately. This RFD is primarily about the less obvious problems that we’ve learned the hard way.
Practical problems with distributed sagas
Over years of deploying distributed sagas at Oxide, we’ve repeatedly discovered programming patterns that seem to work and maybe work most of the time but become major problems under unusual but important operational conditions. While going through these, it’s helpful to consider whether these are intrinsic problems or accidents of our design and whether these are reliably fixable or will remain ongoing risks and costs.
Summary of problems
Sagas are difficult to implement for a variety of reasons, most of them intrinsic to the concept of sagas.
People forget that indeterminate failures must be retried. This is easy to mess up, hard to verify, and failure results in corruption of the control plane state managed by the saga.
But permanent failures must not be retried, even when they look transient or indeterminate. This, too, is easy to mess up. In the short term, failure results in a hung saga. When the system is next updated, these sagas can fail in arbitrarily bad ways (if a MUPdate-based update is used) or the update can itself become stuck (if a self-service update is used).
Undo actions must never fail. This one could be mitigated through breaking changes to Steno, which are non-trivial but believed manageable.
Confusion over whether undo actions get run for failed actions
Static DAGs make it hard to respond to a changing environment
Worse than these development-time problems are these operational problems that we believe are intrinsic to distributed sagas:
Sagas don’t play well with any kind of upgrade. Across an upgrade, we either need to wait for sagas to complete (which means a single bug in any saga (or even a component used by a saga) can cause an update to be stuck in a way that requires support intervention) or abandon any running sagas.
Stuck and abandoned sagas cannot be automatically recovered. Abandonment of any saga should be assumed to require a support case and manual intervention to resolve.
There are a few reasons sagas either get abandoned today or could become abandoned with proposed changes:
A support person explicitly abandons a saga because there’s a bug resulting in the saga being hung.
A support person explicitly abandons a saga because attempting to run it is causing other damage (e.g., a Nexus crash loop).
The system implicitly abandons a saga because it is
Stuck. That’s what happens when an undo action has failed and the saga cannot proceed to complete or unwind on its own.Nexus is being updated to abandon sagas that cannot be recovered for non-transient reasons to avoid them being run in some future version of Nexus that can recover them.
MUPdate may be modified to abandon in-flight sagas to avoid trying to run them in a new version of Nexus. (More on this below.)
These are all bugs. But bugs are inevitable (or: perfect software is an aspiration, not reality). The problem here is that the blast radius of saga bugs is much larger than is reasonable:
Because of the tight coupling mentioned above, any kind of hung saga must result in self-service update either abandoning it (resulting in a support call) or blocking on it (meaning that the whole system update gets stuck behind even a small bug in a non-critical subcomponent).
Similarly, the MUPdate process must choose between:
abandoning a saga (resulting in more work for support)
rolling the dice to see if things happen to work (which is what we do today, and often works)
draining sagas (which would be non-trivial project and is not always possible anyway)
The rest of this section describes these problems in more detail.
People forget that indeterminate failures must be retried
Imagine writing a saga action that makes an HTTP request to another service, like a request to Sled Agent to start an instance. It’s easy to think you can tack ? to the end of the operation, causing the action to fail if the operation itself failed.[5] This is one of those things that will work almost all the time and lead to corruption in unusual but important cases.
Imagine that the error you got back was a network timeout. In this case, it’s impossible to know if the instance is now running on that sled. If the instance is running, and the action fails (because it just propagated the indeterminate error), the saga may unwind having left an instance running.
The same thing can happen with actions that insert or update database state. Timeout errors (and many other transient errors) mean you don’t know if the operation completed. Assuming failure can leave the database state inconsistent, violating the saga’s expected postcondition.
This issue, while intrinsic, wasn’t really discovered until late 2022. It’s discussed in steno#66 and [rfd419].
omicron#10166 is an example of this bug.
In writing this RFD, we found multiple instances of database queries executed in saga context that propagate all errors up with ?, without regard to whether the error means that the operation definitely did or didn’t succeed. There are likely many bugs lurking here.
But permanent failures must not be retried, even when they look transient or indeterminate
Having discovered and fixed the above issues by wrapping transient errors in infinite retry loops, we started running into a different problem: sagas would sometimes hang indefinitely because they were trying forever to contact an instance of a service that longer exists (and getting a transient-looking error, like a connection timeout). These infinite retry loops all need to also check that the instance they’re trying to reach still exists.
Related issues: omicron#4331, omicron#4259, omicron#5022.
Undo actions must never fail
As defined, distributed saga undo actions are not allowed to fail. If they fail, by definition, the system can neither run the saga to completion nor unwind it back to the beginning. The saga guarantees are violated. The system is left in a state where some of the side effects of the saga have happened, but not all of them. Think: space has been reserved on a sled for an instance and the instance is "starting", but the instance is not running, there’s no way to finish starting or stopping it, and there’s no way to release the allocation, either. (Put differently: if there were a way to fix these problems, that’s what the undo actions should have done.)
From this, we can conclude that undo actions, like actions, must retry any indeterminate errors (which means most transient errors). They must only return permanent errors when they know they can never succeed (e.g., the record they’re responsible for updating is gone altogether).
However, this consequence is not very obvious. As with regular actions, people write undo actions that simply propagate whatever error they got. When this happens, Steno puts the saga into a Stuck state. It can never move forward or backward again. It would take support intervention (likely with engineering help) to figure out how to repair such a problem, since it depends on which saga it was, which actions had succeeded, which undo actions had run, and which undo actions had succeeded.
Examples of undo actions that failed:
The good news here is that changes to the Steno interface could make it harder to make this mistake by accident. Unfortunately, this isn’t trivial. See steno#283 and omicron#5908.
Idempotency is harder than it sounds and hard to verify
Idempotency is like memory safety in unsafe languages like C or async cancellation in Rust, in that:
doing it correctly requires that programmers remember to follow a bunch of specific rules
the compiler cannot identify when these rules are being violated
it’s very hard to verify even with automated tests
very often, code that’s wrong appears to work
getting it wrong can lead to awful bugs of the "implicit, non-fatal failure" variety
Naturally then, we’ve seen issues where basic idempotence wasn’t upheld:
In omicron#3305, a saga node was creating another saga, which is not idempotent.
In omicron#6069, omicron#6071, and omicron#6097, saga actions were doing database
INSERTqueries that aren’t idempotent.In omicron#2094, it was noted that an action was generating UUIDs used in later steps. This isn’t idempotent.
Worse, what sagas require is something stronger than idempotence. Traditionally, idempotence means that "a subroutine with side effects is idempotent if multiple calls to the subroutine have the same effect on the system state as a single call". But that assumes both calls actually completed (or, equivalently, that they’re atomic). But saga actions can potentially start, do some work, crash, then start again and run to completion. Why’s that harder? Consider an action that takes two steps. Step 1 is idempotent, and if it detects that it’s been done, the action reports success. Step 2 does some additional step. A crash in the middle of the action means that step 1 will reliably be run correctly, but step 2 could sometimes be skipped (inadvertently). This may sound contrived, but omicron#5980 describes a situation very similar to this outside the context of sagas. (It’s a little different in that that one requires concurrent execution to break, not a crash, but the structure is very similar.)
Can we at least automatically test for idempotency? We do have automated tests that run individual saga actions multiple times to identify obvious problems. But they cannot detect problems that only result from partial execution of the action followed by complete execution.
Confusion over whether undo actions get run for failed actions
In the same neighborhood of "actions that do multiple things": developers have sometimes expected that if an action fails, then the undo action will get run. This is not how distributed sagas work. Undo actions are allowed to assume that the action itself has completed successfully.
We’ve often ended up working around this behavior by attaching undo steps to the previous action. See this comment from a saga definition. See also omicron#1466 for more, and why this is also problematic.
Static DAGs make it hard to respond to a changing environment
Saga actions make up a DAG that predetermines all the steps that have to happen as part of the saga. This decomposition into small steps is helpful for the reasons explained in [_overview_of_distributed_sagas]. But the predetermination is a problem in an environment that can change out from under the saga.
In practice, an operation might want to do something different for various reasons:
the user’s intended goal has changed (e.g., user issues "instance stop" during an "instance start" saga)
the system’s state has changed (e.g., a sled is expunged — or even just temporarily unavailable — while we’re trying to put an instance on it)
the software itself has changed (this is the upgrade case described below, where a new version of the system may require different steps to be taken to process the same operation)
Sagas don’t always have a great way to respond to these things. At best, they can proactively check for these conditions and bail out, relying on whatever component created them in the first place to re-evaluate what to do (and potentially start another saga).
Stuck and abandoned sagas cannot be automatically recovered
As described under Undo actions must never fail, sagas can become Stuck if an undo action fails. Sagas can also become abandoned, which means that they stop running at some arbitrary point in the middle. The end result is the same as when they become Stuck. The contract around the saga is violated. It neither runs to completion nor fully unwinds, but remains indefinitely in an intermediate state with resources allocated (or running). Abandonment can be explicit (the saga’s state is set to abandoned and Nexus never attempts to recover it any more) or implicit (something about the saga state prevents it from being recovered). Saga abandonment is discussed at length in [rfd555].
A saga becomes stuck or abandoned (explicitly or implicitly) when:
an undo action fails due to a permanent error (hopefully, these are always bugs like the ones mentioned above)
a saga becomes unrecoverable (hopefully, these are always bugs, like steno#462)
a support person explicitly abandons it, maybe because:
the saga is causing a panic loop in Nexus (due to a bug)
the saga is hung and cannot be made to complete (presumably a bug)
the saga is hung (or just taking a long time) and we want to proceed with a self-service update anyway or a MUPdate (and don’t want to risk the saga being re-executed after the MUPdate)
The problem with this situation is that there’s no automated recovery possible. It’s never safe for newer versions of Nexus to run sagas created by older versions, which means the problem cannot be repaired with a software update. The best we can do for this situation is omicron#10530 (create an "active problem" for it), expect support to get engaged, and craft a special-purpose remediation procedure for each distinct saga and failure mode that we find in this situation. While these are almost all bugs, they’ve happened many times and we cannot reasonably hope that they won’t keep happening. (See also omicron#7623 ("Sagas with extended retry loops may be undrainable during upgrade windows").)
Sagas don’t play well with any kind of upgrade
The problem with sagas and upgrade
[rfd289] describes how the nature of sagas makes them very tightly coupled to the current version of Nexus and the database schema. Consider again the "instance start" saga with roughly these steps:
pick a sled and allocate resources on the sled
allocate an IP addresses from a shared pool
program various networking services so that the right routing rules and NAT rules are in place
start the virtual machine itself
attach disks
The appeal of distributed sagas is that this complex process is broken up into small, manageable steps about which you can mostly reason locally. But the small, atomic steps involved in starting an instance change over time as we enhance the system to do more things (like adding local storage, which added several steps to the saga). In particular:
Necessary steps may be added to the saga (e.g., propagating some new piece of networking config somewhere).
Steps that used to be needed may become invalid (e.g., previously we may have issued a request to sled agent to, say, set up routing rules for the VPC, while a new version of the software instead pokes a background task to update the config using a different request type).
Data output from one action and consumed by another may change incompatibly (e.g., as we replace cases where sagas were looking up items by name (which is mutable) instead of id — see omicron#1536).
Assumptions made by actions about what previous actions did can become violated.
This creates a huge problem for upgrade: if a saga is running when the system is upgraded (however upgrade works), the new version of the software can’t be expected to successfully resume the saga. What would happen in these cases?
If a new version adds a new necessary step to this kind of saga, that won’t be reflected in the saga’s DAG (since that was constructed in the previous version), so that step simply won’t happen. The failure mode here can be arbitrarily bad. Examples:
If the new action is what establishes connectivity to the rest of the instance’s VPC, the instance will just be isolated.
If the new action is necessary to enforce a security constraint, that constraint will just be violated.
If the new action records accounting information to avoid the system running out of space, that simply won’t be stored and the system may wind up overprovisioning itself later.
If some step from the saga was removed, the saga simply won’t be recoverable since Steno has no idea what to do for the action whose implementation no longer exists.
If the data shared between actions changes incompatibly, the action that consumes it will likely fail, triggering an unwind. (Note that it’s possible that this action had already partially run previously, meaning this could additionally introduce the kind of corruption described above when people forget that indeterminate errors must be retried.)
If some action in the new code assumes something that was supposed to be done by a previous action (and that also changed in this upgrade), Nexus may crash on a blown assertion or other panic. This often leads to a crash loop, resulting in Nexus offline altogether, as seen in omicron#7361.
See also Alternative discarded: migrating sagas.
How self-service update deals with sagas
Per [rfd289], the self-service update process determined to say that all sagas must be drained from the system (i.e., run to completion or failure) before the upgrade. This way, Nexus never sees sagas created by older versions of itself.
This works fine on sunny days. But if even one saga becomes hung, the update process blocks indefinitely (until support can come in and fix the problem). A saga can hang because it didn’t account for the possibility that the component it’s talking to has been expunged or because the saga makes a request to a component that itself hangs (as seen in crucible#1945). It’s always a bug if this happens, but the degree to which it impacts the system is disproportionate. The only thing a hung saga should affect are the resources within its scope (e.g., the instance, disk, etc. that it operates on). Here, it blocks all of system update.
This problem cannot easily be mitigated. See omicron#7623. The only way to automatically unblock the update would be to abandon such sagas. But every abandoned saga requires support to investigate it and repair it. This might be an improvement for update, but it’s not a great answer to the underlying problem.
How MUPdate deals with sagas
It doesn’t. Every time we MUPdate a system, we’re rolling the dice: if there are sagas running when the rack is parked, anything could happen once the new Nexus comes up and resumes the saga. See the consequences mentioned above.
In practice, we do seem to have gotten pretty lucky, having done many MUPdates on customer systems without noticing a problem most of the time. We certainly have seen problems like panic loops resulting in Nexus offline altogether. And many of the consequences above could go unnoticed (which doesn’t make them unimportant, as in the case of a security issue).
We could imagine quiescing sagas during MUPdate-based updates as we do for self-service updates. However, this is tricky to do, and it wouldn’t cover MUPdates that are done to recover a broken control plane.
In general: if the saga itself is broken resulting in a hang, the only safe option is to abandon it, which creates the headaches described above.
Main alternative: background tasks / reconcilers
How background tasks can help
Background tasks (what [rfd373] introduced as "reliable persistent workflows" (RPWs)), which are very similar to the Kubernetes reconciler pattern, provide an alternative for some use cases. They’re designed to make various common anti-patterns hard to do by accident.
Background tasks are activated periodically and on-demand. Typically, each activation fetches the intended state of the world (usually from the database), fetches the current state of the world (from the database, inventory, or requests to other components), and then takes steps to make reality match what’s intended.
One way to implement a complex process like "instance start" as a background task would be something like:[6]
operations that change the state of the instance set some flag, like
state = Startingthe background task:
iterates instances needing attention (e.g.,
state = Starting) and for each one:determines the current state (e.g., fetches state from sled agent, queries for any allocation records, etc.)
picks one step to do next (e.g., if no allocation records are found, then do the allocation step)
If "unwinding" is necessary (maybe reflected by a separate intended_state = stopped), it works basically the same way: determine the next step and take it.
This is a very different programming style from sagas. Sagas lay out the long list of steps ahead of time and then execute them in order. With this pattern, the program just identifies one next step and does it. We rely on being invoked again later to take the next step (see continuation-style programming). Herein lies the advantage:
If the intended or actual state of the world changes from one activation to the next, the background task can make a different choice about what to do next than it would have expected at the start. For example, if we’re halfway through creating an instance and the sled we pick is expunged, we could tear that down and then pick another sled. A saga can’t do that. It has to unwind and then rely on something else to start a new saga that will pick another sled.
This approach is very amenable to the plan-execute pattern and exhaustive testing. For example: the background task can collect all the state relevant to its next choice and hand that to a pure function to decide what to do. This can be tested for all kinds of input using property-based tests, etc.
For developer experience: it’s often possible to factor this decision engine into its own package within Nexus, allowing for much faster iteration time (rather than rebuilding all of Nexus to test every change).
If the very process for doing whatever the task does needs to change across a software update, there’s nothing to "fix up". The underlying data is represented with a well-typed database schema, so any data transformations are handled by the existing schema update process (which includes good support for testing migrations). (One can still imagine cases that get missed. It’s important that each logical step actually check that it’s been done, rather than make assumptions like "if I did this step, I must also have done this earlier step" (which may not be true after an upgrade).) This significantly mitigates Sagas don’t play well with any kind of upgrade.
For this reason, bugs in the process can be fixed by a software update. As we said, this isn’t possible for sagas since updated software cannot be assumed to be able to run old sagas. But with a background task, if a bug causes it to get stuck doing something, we can fix that with a software update without needing anything like saga abandonment or the manual investigation and recovery that that entails. (This addresses Stuck and abandoned sagas cannot be automatically recovered.)
Since there’s no smaller abstraction (the "action") with properties that need to be upheld (like idempotence), a bunch of problems simply go away:
People forget that indeterminate failures must be retried is simply not a problem. (Idempotence of individual steps can still be important.)
Confusion over whether undo actions get run for failed actions goes away.
Undo actions must never fail goes away.
The underlying problem of a process not finishing can still happen. The blast radius is limited to what it should be: the resources affected by this operation. It doesn’t block upgrade and it doesn’t necessitate anything like abandonment.
Background tasks are stateless by design. They’re always supposed to re-fetch any state they need. This makes it harder for them to get "stuck" talking to a component that’s been expunged.
Example: Reconfigurator and Upgrade
Reconfigurator elected early on to use background tasks and not sagas primarily because it was determined that the system absolutely has to be able to respond to changing circumstances. We considered using a saga that lays out the several dozen or hundred steps of an upgrade, but if a sled failed permanently halfway through before it had been updated, the system would be in trouble. It can’t proceed, and it can’t rewrite the saga DAG partway through, and unwinding doesn’t make sense.
It took quite some time for the team to embrace the idea that the planner would only ever plan one next step, not lay out the whole path forward, but over time it’s clearly been better than laying out a whole set of steps ahead of time.
Challenges using background tasks to replace sagas
Some saga use cases may be amenable to rewrites as background tasks, but others might be tricky.
Among the biggest challenges is that sagas (sort of) guarantee that only one Nexus instance is operating on them at a time. More precisely, they guarantee that only one Nexus instance is operating on this saga at once. However, additional synchronization is necessary to ensure that there’s not another saga trying to do the same thing (e.g., two "instance start" sagas running for the same instance).
By contrast, background tasks run in all Nexus instances concurrently. You could try to establish mutual exclusion with a distributed lock/lease, except that they don’t work.[7] Reconfigurator and related systems deal with this problem by ensuring that each action that these background tasks take is safe to be executed concurrently (and even with an older version of the intended state). Mostly, this amounts to:
When making database queries, use transactions that check that the current state matches what the caller last read.
When making requests to other APIs (like sled agent and DNS servers):
Requests are essentially declarative. They specify a new set of configuration, not CRUD actions.
Each request includes a generation number.
Servers store the last-received configuration immediately, then try to apply it asynchronously.
Servers reject configurations with older generation numbers.
With these patterns, multiple Nexus instances can be carrying out Reconfigurator instructions and the system converges to the expected state without ever going backward, even when some Nexus instances are simultaneously carrying out old instructions.
Tooling to help write background tasks
One idea is to create a library or framework that allow background tasks to look like sagas. See Alternative considered: background task tooling to make it look like sagas for a discussion on this.
Other tooling might better leverage what background tasks are good at. Consider instead a framework where consumers implement:
trait SomeComplicatedProcess {
// See https://docs.rs/proptest/latest/proptest/strategy/trait.Strategy.html.
// `Strategy` makes it possible to define property-based tests that
// construct arbitrary input states and verify expected properties.
type State: proptest::strategy::Strategy;
type Actions;
fn next_state(&self, state: State) -> Actions;
fn simulate_actions(&self, state: &State, actions: Actions) -> State;
}This would define the guts of a "planner" (as in the plan-execute pattern). We could build general-purpose tooling for comprehensive property-based testing, a CLI for interactively exploring various states (similar to reconfigurator-cli), etc.
Alternative discarded: migrating sagas
In talking about how Sagas don’t play well with any kind of upgrade, it often comes up: can we write migrations that "fix up" any in-flight sagas? It doesn’t seem likely. The three examples above do not lend themselves to any kind of migration. For example: a saga that previously stored only the name of a disk that now needs to store the id would have to look up that id by name. But that might now refer to something else (the problem we were trying to solve in this example). Or it might fail altogether. And unwinding isn’t necessarily an option because undo actions have the same problems (their shape may have changed or they may rely on data that’s changed incompatibly).
There may be cases that could be migrated, but even then, the burden seems untenable:
Virtually any change to a saga action would require carefully considering the update impact — what would happen to previous sagas of the same type?
New versions of software would have to maintain translators for sagas created in many old versions of the software.
Each such translator would need to know how to fix up sagas' DAGs as well as the data stored for each node. These translators would need to know what to do for every possible intermediate state of the saga, which includes all possible combinations of outputs from every valid combination of completed actions.
This code is hard to test (because of the number of different configurations) and is very rarely executed at runtime, a combination that doesn’t lend itself to reliability.
This feels like snapshotting the heap of a running program, replacing the program text, and running some translators to translate the old heap’s objects into those that the new version of the program expects. It feels like the wrong answer because these structures are intrinsically so tightly coupled to the program itself.
This is also discussed in [rfd555] (under "Alternatives considered").
Examples
These examples are provided to illustrate the (very reasonable) coupling between the version of Nexus that creates a saga and the version of Nexus that executes it.
Example 1: Removed actions in disk_create
The action generating
created_diskwas split into two generatingcrucible_diskandlocal_storage_diskThe action generating
disk_runtimewas renamed to producecreated_disk. A second action also producescreated_disk.
These naming changes would break in-progress sagas. Any existing saga would have a "create_disk_record" action, but that action no longer exists because the new version has separate actions for creating the disk for Crucible, local storage, and read-only disks. To be clear, this change is perfectly fine — it’s just an example of the kinds of change we need to be able to make that would be much more complicated if we considered migrating old, in-progress sagas.
Example 2: Behavior change in region replacement
Between R12 M1 and R13, the region_snapshot_replacement_start saga was changed to add a few new actions. Previously, some key values were determined from the saga parameters. After the changes, they were computed by the newly-added actions. (For example, the rsrss_get_alloc_region_params action previously used params.request.old_snapshot_id, but was changed to use the snapshot_id produced by the new snapshot_and_region_id action, which didn’t get it from the saga parameters any more.)
A consequence of this is that in R13, a saga created in R12 would have successfully been recovered, but would have failed in these later actions when they failed to look up the outputs from a node that doesn’t actually exist in the saga (because it was created from R12 software). Fortunately, rsrss_get_alloc_region_params doesn’t make any changes to the system. If it did, it could also run into the idempotency issues described above. The function propagates the parameter lookup failure (as Steno’s examples and documentation recommend), but if a side effect had happened during an earlier invocation of the action, this side effect would be left undone.
Example 3: Behavior change in vpc_create
Also between R12 M1 and R13, the vpc_create saga was changed to add a new action that creates the "system" route for the new VPC’s subnet. This behavior replaced more ad hoc code that would "fill in missing routes after the fact if a nexus were to crash in an awkward spot".
However, a saga created in R12 that was in-flight during an upgrade and resumed in R13 would result in a VPC without such a route (because the saga didn’t have that action in it) and no code to create it. The created VPC would be forever missing the route and presumably be broken.
Alternative considered: background task tooling to make it look like sagas
One could imagine a library or framework that would allow consumers to define background tasks similarly to sagas today: you could define a DAG of states with transformations between them. For example, the "instance start" one could have a state for each discrete step in the saga as it exists today, but it would be an explicit Rust type (like an enum with a variant for each step) that’s stored with a strong database schema. By writing code that matches over the current state and writes a new state, one could know that all cases were handled. Since the state is stored with a strong database schema, changes to the states or their associated data would be expressed as schema migrations.
It’s possible something like this could be useful. But it might also re-introduce the same problems we’re trying to solve. One problem with sagas is their rigidity: they assume a set of steps will happen in a specific way and that nothing can invalidate that. Background tasks that re-evaluate the state of things and decide what to do are more flexible in responding to changes.
Determinations
Guidance for sagas in new work
New work should at the very least carefully consider using background tasks over sagas. When in doubt, prefer the background task. When sagas are selected, one must consider how the saga might need to be abandoned and how we’ll plan to repair such sagas.
Guidance for working with existing sagas
In working on existing sagas, developers must continue to be very mindful of the problems mentioned above:
Actions really must be idempotent (and ideally should be structured so that this is easy to verify and hard to break).
Undo actions really must never fail.
All error paths must check for indeterminate errors.
All paths that make API requests must check each time whether the thing they’re talking to has been expunged.
It’s assumed that if this action successfully made this request before and then crashed and hits this condition, there’s still nothing to undo because the only side effect was in this service that’s now expunged. If that’s not true (e.g., if the work in that component would have been migrated to a different component upon expungement), this operation cannot be correctly implemented with a saga. That’s because propagating the "this is expunged" error won’t trigger the undo action to run.
We could do some work to help with this:
build a helper for examining an
omicron_common::api::external::Errorand checking if the result could be indeterminate (would likely returntruefor all transient errors, though some of them are not indeterminate)build helpers for obtaining clients for various services that check whether those services have been expunged and return an appropriate error
change the type signature for undo actions to make it harder to accidentally fail from them (see steno#283).
build a Claude skill to review code for these problems
These aren’t likely to be enough to avoid continuing to hit these problems. They only make it a little easier to do the correct thing, not any harder to do the broken thing.
When working on these sagas, particularly when these requirements become burdensome, we should consider replacing them with background tasks.
Non-determinations
At this time, we’re not suggesting eliminating all sagas from the system.
We could consider:
Self-service update could automatically abandon sagas that have been running for too long. This should eventually lead to notifying operators via active problems (see omicron#10530) followed by support intervention. This creates a lot of work for ourselves, but not doing this involves the same amount of work while under the gun in the middle of a stuck update.
omicron#6804 Quiesce for MUPdate (but this is tricky work and not an investment in the update experience we want for customers)
Security Considerations
None known.
External References
[rfd107] Oxide Computer Co. RFD 107 Workflows Engine.
[rfd289] Oxide Computer Co. RFD 289 Steno Upgrade.
[rfd373] Oxide Computer Co. RFD 373 Reliable Persistent Workflows.
[rfd419] Oxide Computer Co. RFD 419 Only YOU can prevent unwinding sagas.
[rfd555] Oxide Computer Co. RFD 555 Addressing operational challenges with sagas.