RFD 177 Implementation of Data Storage

The Guest takes IO from Propolis and communicates that to the Upstairs, which sends to downstairs over the network, waits for response and reports the results back to Propolis.

The Guest and Upstairs are meant to keep as little state as possible, and require no additional storage apart from the data structures in RAM.

The downstairs stores blocks on physical storage. It is intentionally meant to perform as little of its own logic as it can get away with - all logic should be performed by the Upstairs.

A Downstairs will answer requests from any Upstairs that has resources allocated to the SSD this Downstairs operates on.

Each of the ten U.2 SSDs in a compute sled will be formatted as a single-device zpool with no redundancy.

A Crucible zone will be created for each SSD.

Each Crucible zone will have a delegated ZFS dataset (file system) within the pool on the SSDs where it will create and store regions:

Inside a Crucible zone you will have a single Crucible agent and one or more Downstairs processes:

The Crucible agent is responsible for listening for requests from Nexus. When a request arrives, the agent will create the Downstairs region and either start a new Downstairs process for this region, or update a running Downstairs process to answer requests for this new region.

The exact balance of how many Downstairs processes are started vs reused is still TBD, pending performance, fault tolerance, and resources required.

Most of the time, Guest VMs will interact with Volumes. Volumes present the same interface as the Upstairs (read, write, flush, etc) but where the blocks are read from and written to is different.

Volumes are composed of a list of subvolumes, along with an optional read only parent. The subvolumes are concatenated together end to end, but the read only parent always underlays starting at the beginning of the Volume.

For example, here is a Volume with four subvolumes and one read only parent:

The typical use of the read only parent will be to construct disks with an image underlay - creating a new disk out of an Ubuntu image for example. The term read only parent is used here but could easily be substituted with "ubuntu image". See [_relationship_between_volumes_and_api_objects] for more.

The Volume layer will take care of translating requests into the LBA to the appropriate sub volume or optional read only parent (more on this later). Without a read only parent, reads and writes to the Volume will be directed to the appropriate subvolume:

The example read or write range denoted by A interacts with two subvolumes, while the read or write range denoted by B only interacts with one.

If a read only parent is present, reads that occur to the LBA range of the read-only parent will read from both that parent and the subvolumes:

Crucible stores per-block integrity hashes, and those are used here to determine which block (from the read only parent or from the appropriate subvolume) to serve:

Writes will never be sent to the read only parent, only subvolumes. In this way, writes to subvolumes overlay blocks from the read only parent.

Reading from two locations like this incurs a read latency penalty as long as the read only parent is present. The intention is that a background task will be reading blocks from the read-only parent and writing those to the volume (creating an integrity hash in the appropriate subvolume), and once the read only parent’s whole LBA range has been copied the read only parent will be removed. See [_block_migration_task] for more information.

This interface is used by Nexus to allocate storage resources and assign them to a VM.

Workflow for creating a Virtual Volume:

Propolis will use the Guest to communicate with an Upstairs. Each Guest will implement basic read, write, and flush functionality, plus some Crucible specific functions.

Importantly, Propolis will handle all device emulation. Users will select NVMe, virtio, or some other device whose block requests will be fulfilled by Crucible, and Crucible will be none the wiser.

The actual work (encryption, network I/O) will occur on another set of threads Upstairs manages (e.g., a tokio runtime).

Completion will be through a callback Upstairs provided by Propolis at setup time.

Rate limiting can be done in both Propolis and the Upstairs. Upstairs will have more information about its internals that is not available to Propolis (i.e. if one of the three downstairs is having trouble keeping up). It would make sense to have a limit at Propolis. Some consideration was given to having the network switch implement bandwidth limits for storage traffic, but this may be too little too late, though could prevent a bad actor from a DOS attack on the network backplane.

When an Upstairs connects to a Downstairs, part of the negotiation that occurs will involve a process called "Activation". The Upstairs requests to be Active, and sends along a few parameters:

Between different connections, the Upstairs UUID remains the same, and the session UUID changes. This allows for future optimizations, such as handling different cases:

A Downstairs can be started in either read-write or read-only mode. A Downstairs launched in read-only mode will return an error for any operation that modifies on-disk state - typically, a Downstairs that is launched to serve data from a ZFS snapshot directory will be in read-only mode, and regular Downstairs will be in read-write mode.

If there is a mismatch in the Upstair’s requested mode (read-only or read-write) and the Downstair’s mode, the activation is rejected by the Downstairs. Similarly, if there is a mismatch in the encryption expectation, the activation will be rejected.

Each Upstairs must use a unique UUID, as this disambiguates the job IDs that arrive at the Downstairs. Generation number will be used to deactivate old connections so that at most one Upstairs with a certain UUID is connected and active.

An Upstairs activation will be accepted if there are no existing sessions with the same Upstairs UUID. If there is an existing session with the same Upstairs UUID, the Upstairs activation will be accepted if the gen number is higher than the current session’s gen number. In this case the current session will be forcibly deactivated by the Downstairs. This is intentionally a takeover, as opposed to more complicated procedures like leader election.

A read-only Downstairs supports multiple active read-only Upstairs connecting to it. Any Upstairs that tries to issue an operate that will change on-disk state will see an error. It will not support any read-write Upstairs connections.

A read-write Downstairs only supports one active read-write Upstairs connecting to it. It will not support any read-only Upstairs connections.

End to end data integrity protects from corruption between data in a buffer in a guest VM, and the data on physical storage. ZFS will provide us with protection for corruption of data at rest but we need to consider corruption of data after Crucible accepts it and before it lands on disk. Corruption can occur at any point: in Crucible memory, as it goes over the network, in the NICs, etc.

Note: Crucible can protect data once it lands in Crucibles buffers, but to really offer E2E protection, the OS running in the VM will have to do it itself. Crucible can only protect the data once it lands in Crucibles buffers.

Three mirrors gives Crucible the ability to compare for an corruption that happens after data leaves an upstairs, and per-block integrity hashes allow for determining if corruption has occured. Hashes are computed post-encryption by the Upstairs and are part of the protocol messages that are sent to the Downstairs:

As mentioned above, ZFS itself provides and corrects for corruption of data at rest on the underlying hardware.

xxHash was chosen as the hash algorithm:

The hash digest is a simple u64 value meaning the protocol messages are not made unnecessarily large by including these digests.

We plan to encrypt data when it arrives in the Upstairs and decrypt before we leave Upstairs - users of Guest will be unaware.

Data in transit between Upstairs and Downstairs will occur over TLS.

By encrypting before mirroring, reconstruction between mirrors will be simpler as the encrypted data should be the same on each mirror, and the keys will not be required to reconstruct.

Keys for data encryption will be provided from Nexus (it’s part of the material that an Upstairs will be initialized with).

AES-GCM-SIV-256 was chosen as the algorithm:

The nonce and tag associated with AES-GCM-SIV is referred to as an "encryption context" for a block, and is stored in the SQLite database for an extent.

Authenticated encryption was chosen to mitigate the scenario when an attacker has access to extents Downstairs - with unauthenticated encryption, that attacker can modify those blocks and the Upstairs will decrypt them, producing garbage data or something worse. With authenticated encryption, the Upstairs will be aware of this tampering, and will report reads of those blocks as an error instead of serving garbage data.

Integrity hashes for blocks with encryption contexts will include those encryption contexts as part of the input to the hash algorithm. This is to ensure end to end integrity of the encryption context as well.

The performance of AES-GCM-SIV is mostly the same as AES-XTS - as Josh pointed out, they’re both AES. Introducing per-block metadata storage incurred a small performance degradation, but this was seen with unencrypted blocks requiring integrity hash storage as well.

Key rotation will require rewriting of all data. If Volume 1 exists with key A, rekeying can be achieved by constructing a new Volume 2 with key B that uses Volume 1 as a read only parent:

The migration task that copies blocks from the read only parent to the Volume will then perform the re-encryption. In Propolis, Volume 2 can be swapped for Volume 1 at runtime with no impact to the VM.

Leaky bucket (as a meter) has been implemented to limit IOPs per second and bandwidth per second. Each Upstairs can be configured with an optional IOPs/s and bytes/s, and will limit appropriately.

Rate limiting can exist on several levels, at the downstairs, at the Upstairs, and Propoplis itself can control flow.

We need to select a good high bandwidth low latency data transfer in Rust With at least a 3x multiplier for writes, we should be sure we are efficiently sending data over the wire.

How the Upstairs and Downstairs authenticate each other is important to prevent unauthorized connections and interception.

Using TLS and certificates is the first choice for over the wire encryption. As the data itself will be encrypted, this will protect the metadata. Only using TLS will not perform authentication though: mutually authenticated TLS using client and server certificates could be selected but note that this only authenticates the stream between the Upstairs and Downstairs, not the requests themselves. mTLS also has a strong dependency on correct and synchronized clocks.

Authenticating each protocol message also incurs overhead and it’s important to select something lightweight.

Keys and/or certificates will always come from Nexus.

Using AES-GCM-SIV for Crucible’s authenticated encryption means that a nonce and tag for a block has to be stored alongside the encrypted data, and writing these three parts out separately introduces the potential for a crash to cause the on-disk state to be unreadable - if all three of the nonce + tag + data are not atomically written to disk, decryption will fail.

The combinations of nonce, tag, and data that are valid are:

Any other combination will not decrypt. If the Guest has not issued a flush, Crucible should guarantee that an Upstairs read after an interrupted Downstairs write will either return to the guest the data from the old block or the new block, and that any write interruption should not cause the data to become unreadable.

If the Guest has issued a flush, and Crucible has returned successfully from that flush, it should never serve the old block.

That above table isn’t the whole picture, because those writes are not atomic. Operating systems do not have to commit data to a disk after every write. The only guarantee is that data will be on a disk after an fsync. In the mode that we’ve configured it to use, SQLite will perform an fsync of the metadata db after every transaction. This leads to a scenario where between extent fsyncs writes to the extent files could be on disk:

*: unless disk firmware is lying or has a bug or is “consumer” grade or ?

Between fsyncs above, the data on disk could be any combination of v1, v2, or v3. The only guarantee provided by the OS that we can rely on is that returning ok from fsync means data has been durably written to disk.

Interruptions can occur at any point during the write, and at any point before fsync returns ok. Depending on the fsync behaviour of SQLite and the fsync behaviour of the downstairs, different issues may arise. Because SQLite fsyncs after every operation but extents are only fsynced after receiving a flush comment from Upstairs, a crash could cause data to fail decryption due to skew between the encryption context on disk and the data on disk:

This sort of error would go unnoticed until there was a power loss because the state in RAM is always ok.

I’m going to use “interrupted write” and “crash consistent” interchangeably here. Also, as before, “encryption context” means nonce and tag for a certain block.

We cannot be crash-consistent if we do not write out all of the nonce and tag and data atomically. We currently use extent based storage for the data and SQLite for the nonce and tag. This means we cannot overwrite any of those parts individually, and because we have separate storage for the data and encryption context we cannot overwrite these together atomically.

This means we have to safely append instead of overwriting.

On-disk data has to decrypt successfully at all times, or we lose data.

When performing a write, perform the following steps:

The only guarantee presented by the OS is that fsync needs to be called to ensure persistence:

Ensuring crash consistency means accounting for extent and SQLite fsync behaviour. Writes to the extent cannot be considered durable until fsync is called and returns successfully. We have to assume that any write to the extent between subsequent fsyncs may have made its way to disk, and from this assumption we have to store all encryption context rows between fsyncs.

For SQLite, we’re using write-ahead logging with the synchronous PRAGMA set to FULL. https://www.sqlite.org/wal.html states:

https://www.sqlite.org/pragma.html#pragma_synchronous states:

By default, PRAGMA synchronous seems to be 2, which is FULL. This is probably the “lowest” setting we should consider, given that the documentation states that NORMAL isn’t durable.

In the happy path, reads can return the extent data and the most recent encryption context. These will both come out of RAM. When a crash or power loss occurs, the downstairs will boot back up and have to contend with an unknown file state. The Upstairs will have to receive every encryption context and the data, and figure out which encryption context matches what is on disk.

Repairing an extent is coordinated by the upstairs, but the actual transfer of data happens downstairs to downstairs. Each downstairs runs a repair server and listens for repair requests from other downstairs. A downstairs with an extent that needs repair will be given the address:port of another downstairs that contains a good copy of that extent.

For example, if Downstairs_0 has extent 4 that needs repair. The Upstairs will tell Downstairs_0, you should get extent 4 from Downstairs_2. Downstairs_0 will then contact Downstairs_2 and request the file list, then each file required to repair extent 4. Once Downstairs_0 has all the files, it will replace the local copy with the new versions.