RFD 60 Storage Architecture Considerations

Customer data must be able to survive server and SSD failure. Accordingly North is responsible for sending data to multiple instances of South. Initially this might look like simple redundancy / mirroring (i.e. because economic efficiency isn’t a top-level goal); in the future this will look more like erasure coding (EC) / RAID.

When chunks / components of a volume become temporarily or permanently absent, we need mechanisms to track which chunks contain valid data, repair chunks after a transient absence, or reconstruct them in coordination with the control plane.

We want to ensure with extremely high confidence that data returned to a VM as the result of a read matches exactly the data previously written. While SSDs promise very low error rates, we do not want to rely on them to detect invalid data [disagreement welcome on this point if there are dissenting opinions]. This implies that we will store checksums that validate user data.

Legacy storage systems would store user data and checksums together. For example, rather than 512B sectors, they would use media with 520B sectors and store an 8 byte checksum alongside user data. This has some obvious gaps e.g. a sector that contains stale data still appears valid (phantom write), and a write sent to the wrong sector appears valid (misdirected write). Contemporary storage systems employ a Merkle tree of checksums in which the checksum is stored alongside the reference to the data rather than with the data itself. We’ll pursue the modern approach.

One could imagine either North or South owning this responsibility:

When data corruption is detected, it will be reported to a fault management facility—​likely part of the general control plane, beyond the scope of this document.

This may be one of the limited instances where we are content to have overlapping mechanisms between North and South. Both may have a role to play in end-to-end data integrity. In particular when scrubbing data we would ideally like South to be able to operate autonomously, or—​barring that—​to minimize the network traffic between North and South due to a scrub (e.g. with a command from North to South that includes a checksum that could be validated on South without transferring data across the network).

There are two general approaches regarding snapshots:

COW filesystems such as ZFS, Btrfs, and WAFL use the former: new data are written to new locations while old data may be retained for snapshots. EMC storage arrays, VMware vSAN, and others use the latter: overwritten data triggers a copy of old data to a disk or region dedicated for snapshot storage. In general, we prefer 1 as it is simpler to reason about and to operate (although 2 has the advantage of simpler capacity management for data vs. snapshot storage).

In either North or South (or a combination of the two) we will have metadata that describes all user data. A snapshot preserves a point-in-time, consistent representation of that metadata along with the data it refers to.

The control plane will own the snapshot policy: when snapshots are created and when they’re deleted. North will, at a minimum, need to coordinate top level snapshot actions. Either North will own the metadata and be responsible for taking snapshots or South will own the metadata and North will need to "freeze" writes and coordinate the various South chunks.

Users will administer storage performance with comprehensible knobs such as IOPS (which AWS EBS uses) and throughput. While traffic at any point in the system could be shaped, it makes the most sense to do it closest to the VM before operations have been aggregated or optimized deeper into the system.

While the primary, user-facing rate limiting will be in North. South will likely require some aspect of rate limiting to prevent against DoS or other cascading failures.

Compression could happen at either North or South, but we’ll want to compress before we encrypt. That said, compression may yield little to no benefit. It will be common for a VM instance to write in 4KB units. Even if the data were significantly compressible, we would need a 4KB sector on an SSD to store the data. In addition, compressing encrypted data may leave us exposed to CRIME attacks whereby we can leak the compressibility of plaintext data from which it can be possible to extract data. While we should consider it in our architecture, compression may be a post-v1.0 item depending on the efficacy we anticipate.

Encryption could similarly happen either in either component. If we want a different key for each volume (with a given VM granted access to particular keys) then encryption should happen in North (or even north of North, before we receive the block data). It may also be attractive to encrypt in North so data are encrypted end-to-end as it flows through the system.

Note that encryption requires additional metadata for each block such as additional entropy (e.g. salt, initialization vector (IV)), and a message authentication code (MAC — effectively a checksum to validate the ciphertext). We would ideally like to use existing, vetted components for encryption. Barring that, we’ll faithfully replicate the mechanism from a vetted component.

Storage is a finite resource. Some component of our storage system will need to keep track of what blocks are free or in use, and it will need to allocate new blocks intelligently. Much ink has been spilled regarding the best ways to allocate and manage blocks because it’s a critical determinant of performance especially as the system grows fragmented, full, or both.

A given SSD will host chunks that correspond to disparate volumes. Reasonable architectures could variously assign allocation as a responsibility of either North or South. In North, it would assume a relatively static division of SSDs in South, assigning large chunks in which North would manage the find-grained blocks. Alternatively, South could handle allocation, managing blocks for different North instances effectively side-by-side. This would allow for a more dynamic management of available capacity.

South is responsible for anything that requires direct access to hardware such as retiring devices. It will also collect telemetry relevant for fault management and feed it into a generic facility.

This approach implements South as a very simple layer, carving up each SSD into a number of fixed-sized chunks, each exposed over the network to a specific North (as dictated by the control plane). North would be implemented as a virtual block device layer (such as ZFS exposing a single zvol) using those network-visible, remote block devices as the backing store.

This moves almost all responsibility to North, leaving South to monitor disks, and route reads and writes. South would be responsible for aggregating fault telemetry (e.g. failed checksums) from North to diagnose increased failure rates in particular SSDs.

The control plane would be responsible for assigning chunks from various instances of South to various instances of North (within the same Cell). The number of chunks (read, the total capacity) would be dynamic, but slow moving. That is to say, the control plane could assign new chunks (or, potentially, remove chunks), but it would be a relatively uncommon event.

The amount of storage assigned to a given instance of North would be a function of several factors:

This approach implements South as a virtual block device layer (e.g. ZFS exposing zvol volumes). The virtual disk volume in North is statically divided into LBA ranges (e.g. in 1GB chunks); each range has 3 associated instances of South. North and South communicate over a network protocol for both block data and related metadata. North handles a write by checking its LBA and forwarding it to the appropriate South instances. Reads would be distributed to the various components (e.g. round-robin, or more intelligently based on observed performance).

This pushes most of the complexity South with the goal of making North a relatively simple layer. Absent a failure, North would forward operations to the appropriate South based simply on the LBA.

A simple approach would implement both North and South as ZFS. Southern ZFS would expose volumes over the network. Northern ZFS would build a pool on those volumes, and create a single volume for the VM.

Compared with the two previous approaches, this approach offers the SSD space sharing on South as with the Northern Mux, and it allows us to use—​rather than build—​complex logic to handle recovery from expected failures. It would also allow for a more flexible assignment of storage from South to North unlike the fixed chunk size of the previous two approaches.

A variant of ZFS on ZFS, this approach imagines replacing the block-based allocator in North’s ZFS with a protocol that delegates allocation to South. All filesystems need to determine a free LBA at which to store new data. For subsequent reads, this LBA effectively acts as a key used to retrieve the related value. One can imagine replacing the code that writes data and returns the LBA to the consumer with something that looks more like interactions with a key/value store, in this case a new, non-block-based interface shared by South.

Consider the ZFS IO pipeline for logical writes above. Rather than allocating a free LBA we would generate a known unique ID.

This would be relevant if the ZFS on ZFS approach was deemed of interest, but computing allocations on both North and South was determined to be too computationally costly.

The VM <⇒ North <⇒ South interfaces are extremely important and may constrain our future ability to innovate in follow-on versions. A common failure of hyperconverged storage is to leave islands of capacity and IOPS unused and inaccessible. For examples VMware’s VSAN product looks most similar to the "Southern Volume Manager" architecture where the minimum allocatable chunk is 255GB. In addition to this being quite large, it can mean that a given chunk can only deliver the IOPS of that one underlying device or the collection of devices on a single server.

While it may not be critical in version 1.0, we need to consider future customer demands where a large number of IOPS may be concentrated over a small number of GB (or vice versa). We need to keep this in mind when it comes to designing the various interfaces in particular to ensure we aren’t locked into fixed allocations.

Customer data may have very different size and performance attributes compared with the metadata we store to track data, verify checksums, encrypt, track snapshots, etc. While version 1.0 will have a single storage medium (TLC NAND), storage technologies are (again) bifurcating to QLC NAND and pmem (e.g. Optane, zSSD). It would be prudent to anticipate how we might take advantage of those different technologies, and—​in particular—​how they might be applied to data and metadata.

This was deemed to be the simplest of the options, but perhaps too simple. If viable, it would let us lean almost exclusively on the known quantity of ZFS while building relatively little and quite simple code in the data path. In particular the only novel software required of this approach would be a network block device to expose portions of SSDs on South to North where they would be consumed by ZFS. The prevailing concern is that this would be a significant departure from the design center for ZFS. ZFS was designed for spinning disks and adapted, nominally, for SSDs and SAN devices. The types of failures and aberrations we expect over our internal network (robust as it is) will differ from those of disks. Nodes (and therefore disks) will reboot and disappear unexpectedly; we weren’t sure how well ZFS would handle this or how challenging it would be to invest in improving ZFS in this regard.

We simulated this configuration in AWS with ZFS (on OmniOS), instances with locally attached disks, and iSCSI as the block protocol between North and South nodes. While there were likely many contributing factors, the performance of this arrangement was quite poor (~10%) compared with the performance of the instance-attached SSDs. In addition (and as anticipated) there was little resilience to failures e.g. when nodes were taken offline. Some of this was attributable to iSCSI, but after exploring the code and discussing with Matt Ahrens, it seemed like it would be quite challenging to incorporate into ZFS (more on this below).

While this architecture may have been the simplest option, it proved to be too simple. Further, it more closely resembles an extended, local-storage system rather than a clustered one.

This was an attractive option since, of the options under consideration, it most closely resembles that of other distributed storage systems (such as Ceph) while being significantly simpler both in ambition and design. An apprehension here was that it would involve quite a bit of delicate code in the data path that would need to be resilient to all types of failures and pathologies. In addition, we weren’t sure if we could deliver adequate performance.

To explore this we developed a simulator in which we can develop, refine, and test the algorithms that dictate the handling of all failure and success cases. The early results of this simulator have been promising.

While we could consider investing in ZFS to have it accommodate network-mirror-aware storage nodes, it seems like a similar effort to developing the Northern Mux albeit with a much more complex surround of ZFS making it harder to build, test, and maintain. Combined with the poor performance of the Southern Volume Manager, the Northern Mux looks very attractive: the work will be no more complex (and likely much less complex) with more latitude over the design while conforming more closely with the architectures of modern distributed storage systems.

We consulted with Matt Ahrens on these conclusions to make sure we weren’t making a rash judgement with regard to the capabilities of ZFS. He agreed with the selection of experiments, and the conclusions.

The steps from here will be to