RFD 125 Telemetry requirements and building blocks

Telemetry often describes events that happen in the system. These events can also be aggregated into metrics (specifically, counters) that describe how many times the event happened. Examples:

Both events and metrics have their advantages: events provide details that are often necessary to debug broken systems. Metrics are far more scalable and enable people to talk about trends and define SLOs. It’s possible to compute metrics based on an event stream: one could always compute myservice.nrpcs_done from a stream of the myservice.rpc_done events generated by the RPC server. You can’t go the other way around.

In a sense, most behaviors begin as events and the question is where the lossy conversion to metrics happens (if at all). At the application level, systems like OpenTelemetry (or OpenTracing) preserve the event representation all the way to query time. They collect an event stream from an application, store that, and compute any needed summary metrics. On the other hand, programs that expose Prometheus endpoints have computed summaries (and thrown out the event data) before the data leaves the program.

DTrace directly exposes events. (DTrace probes are essentially points in the system that can generate an event.) DTrace makes it very easy to summarize events (often by counting them) at the point of instrumentation. In this way, DTrace lets the user decide when to aggregate them.

Regarding data collection: we propose developing a common component (i.e., Rust crate) that we can use to define events in a program and expose them as a combination of DTrace probes, a raw event stream, or aggregated metrics. This behavior can be configurable based on the program and the event. This approach allows us to build rich instrumentation into applications even when the transmission or storage of that data would generally be prohibitive. We can adjust the policy based on the anticipated data volume or other details. This technique has been used in the past to enable extra verbose debug logging to an out-of-band channel without restarting a program. This facility could potentially be used (in some form) in applications, system libraries, or potentially the kernel.

What about data storage and querying? We’ll almost certainly want some kind of event-based debug tracing for the control plane. For most other use cases, it’s not clear that it matters whether a system is event-based or metric-based. It’s worth noting that the volume of some events may be so high that we could never expect to emit events from the system under normal operation (e.g., arrival of network packets).

What about gauges? The above discussion applies to events and counters. Gauges are different. These are quantities that must be sampled to obtain any value, and the sample doesn’t tell you anything about the value at any other time. Temperature, memory usage, and firmware revision might all be modeled as gauges. You might have data for these at specific points in time, but that doesn’t tell you anything about any other times. Gauges can be modeled by saying that the measurement itself is an event. That measurement might happen on some periodic basis or because some device determined that a threshold was exceeded (e.g., a temperature sensor exceeded a particular threshold).

In systems that store time series data, it’s common to associate key-value pairs with each item (whether a numeric quantity or an event). This can lead to an explosion in cardinality, the number of distinct key-value pairs in the system. Depending on the design of the system, this can lead to a significant increases in memory and storage used for the time series data. The CPU required to serve queries and query latency overall can be affected, too. Sometimes, once the cardinality has grown large (intentionally or otherwise), few mitigations are available to restore performance.

Many of our use cases require very predictable, real-time performance. We may define SLOs for API requests, particularly those used to render real-time graphs, that set expectations like 99% of queries will complete within a few hundred milliseconds. Other use cases (like alerting) don’t require quite so tight deadlines, but still demand good performance. At the other end of the spectrum, complex queries over hardware telemetry that are used for product iteration could be allowed to take days.

Physical resources used for time series collection, storage, and querying come from the pool of hardware made available to operators and developers. Efficiency is important to maximize value to the customer. Predictability is also important for the customer’s capacity planning. Systems that support schedules for re-aggregating data are particularly useful for historical storage.

illumos includes a subsystem called FMA, the Fault Management Architecture. This is described in some detail in [rfd26]. Both kernel and userland components of the system can generate events, which are expected to conform to a hierarchical schema. A daemon called fmd contains diagnosis engines that can subscribe to event streams, make diagnoses (which are other events), and potentially take actions like removing a device from service. FMA records these events in a persistent log.

With illumos as the host operating system, we’ll be using FMA at the very least for synchronous response to hardware failures and probably asynchronous response to explicit hardware failures when local decisions are possible (e.g., a disk has failed). What’s less clear is what FMA’s role is for asynchronous automated response to other events and as a primary store and query backend for data generated both by FMA and other event sources.

One possible approach is to say that the operating system will continue generating telemetry with FMA, fmd will still be used for synchronous and some asynchronous automatic response within the same system, that FMA’s events (or summarized metrics) are forwarded to a more distributed telemetry system for further analysis, canonical storage, and later querying by the rest of the system. (That system could even be implemented with fmd.)

Summary: FMA is event-based, does not support historical querying (except to dump the whole log), and performs vital functions beyond the scope of this RFD (e.g., taking components out of service). It’s not obviously intended to support a very high volume of events nor events not related to faults.

Prometheus is a widely-used system for collecting, storing, querying, and alerting for metric data. Its sweet spot is monitoring request rates and error counts from distributed service and alerting based on these values. The model is pretty simple: you define collectors, which "scrape" remote components (fetching metric data from them). You can use PromQL to query the stored data. You can configure alerts. You can also configure rules, which are essentially predefined queries that are evaluated as data arrives (generating a new metric).

There are a bunch of things that are too simple about this model:

Many of these deficiencies can be worked around by setting up additional Prometheus instances, some of which might pull from others. For example, you can achieve HA by having two completely independent Prometheus instances, although this presents issues such as deduplication of alerts. (That particular problem seems to generally be solved by systems like Thanos, which provide additional alerting tools that implement deduplication.) You can potentially scale horizontally by pulling different metrics into different instances, or by having tiers of Prometheus servers. (Tiers can also be used to reaggregate data with different time buckets or retention times.) But there’s no way to move metrics from one to another. This is a very manual process.

It’s also worth pointing out some major values clashes we’d likely have with the Prometheus community. There’s been a lot of controversy over the approximation and extrapolation behavior. The developers' interactions seem rather at odds with our own values of empathy, responsibility, and teamwork.

Thanos is a separate set of components for providing additional storage capabilities to Prometheus. The main benefits of Thanos are long-term (e.g., infinite) storage of metric data, with optional components of "compacting" data and providing consistent queries across data from multiple Prometheus instances.

There are two important components and a third group of components. The Sidecar component is used for local queries and to transfer data to a long-term storage solution, most commonly cloud buckets. The Compactor coalesces data files into larger blocks, among those already transfered to buckets. Additionally, the Compactor is used to reaggregate data into a new time resolution. Importantly, this does not remove the original resolution of data, but adds a new series.

There are additional components to Thanos, which provide things like a consistent query layer on top of the data collected by one or more Prometheus instances. However, it’s important to note that Thanos itself provides no tools to manage a highly-available Prometheus cluster. To the extent it’s required, availability guarantees would need to be provided by another component (or Oxide); Thanos is designed for storage and consistent querying of such data, but not management of such a cluster.

InfluxDB is a timeseries DB for collecting, storing, processing, querying and alerting on the basis of event or metric data. Data is organized into buckets (analogous to tables in a traditional DB), from which data can be queried or on which tasks can be run (scheduled queries). Data can be transformed in many ways, such as resampling, aggregation or other statistics. This processed data can be pushed into new buckets or used to generate alerts. In contrast to Prometheus, where data is pulled from remote components, Influx uses a push-based approach by default, but also supports pulling data via scrapers. (Interestingly, these are supposed to provide data in the Prometheus format, which suggests an intended interoperability.)

Influx has similar availability and scalability concerns as Prometheus. Clustered solutions are provided as paid services, with the open-source implemenation providing only a single server using the local filesystem for storage. Oxide would bear the responsibility of providing availability guarantees.

Pain points: In the process of evaluating InfluxDB, a number of small issues have cropped up. Any single point is minor, but they are worrying when taken together, and may indicate poor-quality code, rough edges, reliability or stability issues, or other problems. The main problems are documented here for reference.

ClickHouse is a column-oriented database management system designed for efficient online analytical processing (OLAP). While not strictly a timeseries database, it has been used as such due to its efficiency and architecture, and seems to stack up well in terms of performance. ClickHouse has a number of excellent features:

ClickHouse’s most attractive feature at this point is native support for clustering, without requiring a commercial license. It uses asynchronous multi-master replication to distribute data in an available and fault-tolerant way, and supports simultaneous sharding and replication. This is a standout feature, among the options thus far considered. (Only CrateDB also implements this.)

However, the native support for clustering highlights ClickHouse’s only obvious potential drawback: clustering requires Apache ZooKeeper, which has a reputation for high management overhead. While the server itself supports extremely tight control over resource utilization (e.g, memory usage) , ZooKeeper may be more difficult to constrain. ZooKeeper is not part of the data path — the documentation clearly states this, and this is borne out by experimentation (see below). Additionally, ZooKeeper’s reputation stems from the difficulty of managing it directly. Experimentation with a small ClickHouse cluster shows that ZK is largely invisible to the client, managed entirely by the nodes of the cluster.

Note: At this point, ClickHouse seems a likely development target. Its documentation is solid; the system is highly configurable and easily introspected; it supports clustering, TLS, authentication/authorization; and it can be used with existing client-side libraries via its MySQL interface (and a native Rust client exists). See below for details on experimentation and testing.

VictoriaMetrics is an open-source monitoring solution and timeseries database. Though not a fork, it is closely related to Prometheus, supporting the same query language and providing interoperability via the remote_write protocol. VM also integrates directly with Alertmanager, the alerting component that ships with Prometheus.

VM has several attractive features. It is fast and scalable, claiming to provide both higher throughput and lower resource utilization than Prometheus; VM fixes several long-standing issues with Prometheus’s query language (e.g., the rate function); it supports a wide range of wire protocols and ingestion methods (both push and pull). Its most attractive feature is native support for clustering.

The drawbacks of VM are largely around performance, resource utilization, and its general alignment with Oxide values. See Detailed experimentation for a more complete description of the former two points — misalignment with Oxide’s values is described here.

The most prominent features of the documentation page appear to be articles or talks celebrating VictoriaMetrics, "case studies" from users who’ve successfully deployed VM. While it can certainly be useful to see the system deployed in the wild, such links would be more appropriate as addenda rather than the first section.

One of VM’s key selling points is a better compression ratio for on-disk data, compared to Prometheus (and others). However, this investigation makes clear that this is at best misleading. VictoriaMetrics appears to round floating-point values, including timestamps, on insertion — so we get better compression, of course, but our data is wrong. NaN values are also ignored without warning, apparently on the assumption that these will be ignored when plotting or aggregating data in any case.

As another strike against VM’s rigor, its implementation of several key query-language functions appears flawed. The simplest example is the scalar function, which is designed to return the only element of an array, or a NaN if the array’s length is greater than 1. VM’s implementation can return a vector! This kind of type or category error arises in several situations, as the above blog post shows, and is indicative of a general lack of attention to rigor and detail. (Contrast this with ClickHouse, which is extremely precise throughout.)

For a more detailed analysis of the performance of the DB, please see below.

Elasticsearch is an open-source search and analytics engine, originally designed to support fast indexing and search of text documents. It is widely used as part of the ELK stack for collecting, analyzing and monitoring text-based log file data streams. Despite the heavy text-orientation, Elasticsearch supports a wide variety of data types. Fast text-search is enabled by inverted indices, while other data types are indexed by more appropriate structures (KD-trees for certain numeric data, for example).

Elasticsearch presents several significant advantages. It provides extremely fast search (at the storage cost of precomputed indices); its documentation is extensive and complete; availability, consistency, and clustering are part of the vanilla free and open version. There are notable downsides, however. It’s not clear how resource-intensive Elasticsearch is, though presumably storage costs will be incurred to pay for speedy search capabilities. (As a reference, Lucene, on which Elasticsearch is built, claims that text indices occupy 20-30% the size of the indexed text. While not trivial, that’s a relatively benign overhead. Note that Elaticsearch itself may entail additional overhead.) Finally, the Elasticsearch ecosystem runs on the JVM, which may mean other resource-utilization issues. On the other hand, one may strictly enforce memory limitations on the JVM at startup time.

Another downside is that, while Elasticsearch provides the core storage and search capabilities, a number of other useful features are either missing or only available in enterprise versions. For example, encrypting traffic via TLS requires an enterprise subscription. Similarly, automated alerting mechanisms are only available via subscription. (See here for the full feature-matrix.) These aspects would be built and maintained by Oxide.

CrateDB is an open-source SQL-compatible databased intended for large volumes of machine generated data, such as from IoT sensor networks. The database provides clustering, high-availability, replication and sharding, behind a Postgres-compatible SQL interface. The database is configurable, high performance, and supports both fixed and dynamic table schemas. CrateDB is compatible with the Postgres wire protocol, with a few exceptions, and supports a simple HTTP interface as well, which should make application development straightforward.

CrateDB also has a number of drawbacks. It is built on top of Lucene and Elasticsearch, and so presents similar drawbacks in terms of resource usage. There are also a number of features, such as user management and authentication, which are only available in the enterprise version.

Monarch is Google’s most recently published time series system. There are many appealing aspects to it, including the data model, distribution of queries, high availability, horizontal scalability, etc., but there’s no public implementation we can use. See [jc-monarch].

Other systems that we eliminated early:

As mentioned above, one approach for decoupling telemetry producers from consumers is to run a service that accepts telemetry from other components in the system and applies some sort of routing policy to other components. This is a reasonable operational definition of a message bus, which suggests investigating options in this space.

One possible such system that has been floated is NATS. This is a highly-available, secure, and flexible message bus, with multi-tenancy, availability, and performance as primary design goals. The vanilla system, however, does not fit our needs. There are a few minor points, but the most important disqualifying factor is that it makes at most once delivery guarantees. The documentation explicitly states that guaranteed delivery must be implemented in the application, via acknowledgements, sequence numbers, or similar mechanisms.

The distribution includes an additional binary, however, called the streaming server, which does provide different guarantees around message delivery. Messages may be persisted to durable storage which, along with what NATS calls "durable subscriptions", would support reliable message delivery between endpoints in the face of failure on either end. The NATS documentation makes clear in several sections that the delivery is guaranateed to be at-least-once. That is, NATS core provides at-most-once and NATS streaming provides at-least-once message semantics. Applications are responsible for using these building blocks to implement exactly-once semantics.

As a final note, while NATS core supports subscribing to topics with wildcards, NATS streaming does not support this. All subscriptions must be directed to an actual topic.

The conclusion here is that the NATS streaming server may be useful, but it’s not clear. It offers a number of features that we may or may not need (authentication, encryption, multi-tenancy, rate-limiting). It seems clear this is intended to be used to communicate between many parties over untrusted networks, where logical segregation of those communication streams is required. The documentation and quickstarts often reference systems like an IoT sensor mesh. These features don’t seem relevant for our goals. NATS will likely provide some useful building blocks as a basic message bus, but will undoubtedly require lots of application code in any case.

It’s not yet clear if this would be used for metric data in addition to event-based telemetry. If only for event-based telemetry, is this necessary? The systems discussed above primarily store metric-oriented data. FMA provides rich, detailed event-based telemetry based largely on hardware telemetry. Some of this will be used for synchronous automated response on the system where the data is generated. Much of this data can be modeled as metric data and exported off the system.

Still, it may be useful to expose the original event data to the control plane. One could imagine an NVME device producing a non-critical event (or a group of them) that causes a control-plane-level diagnosis engine to move blocks from a network volume off that device and onto another one elsewhere in the rack. This approach is particularly important for issues that require data from multiple systems to diagnose.

The combination of InfluxDB and a message-bus of some kind is attractive. This would support routing of messages, including duplication, to replicated databases. While building an available system out of the database and a bus is feasible, it’s worth mentioning that one already exists in the form of influx-spout. This includes multiple Influx instances on top of which a NATS message bus sits. The bus distributes data to the individual DBs, provides routing rules, filtering/downsampling, and more. This system hasn’t been evaluated rigorously, but we should definitely do so before deciding to reinvent this particular wheel.

The Tokio project has spawned a collection of crates under the tracing-rs heading. The ecosystem contains a core crate which defines the concept of a span, a duration of time within a program, and an event, which is a particular moment in time within a span. These are generally used to define an ordering of events, particularly in concurrent applications, and metadata associated with those spans and events. Developers generally instrument their code with macros similar to logging (such as event!() or span!()), which define when those spans are entered/exited or an event occurs. Metadata can be attached to both spans and events, making them quite rich, strongly-typed indicators of program state.

The most interesting thing for Oxide’s purposes is the Collect trait, which provides precise control over what happens when some instrumentation is triggered. Implementors can register their interest in a particular callsite, based on static information such as function name, line number, and more, or on the basis of dynamic, runtime information associated with the implementor. There are many crates which implement the Collector trait in various ways, such as those which emit log messages, send events to distributed tracing PaaS tools like honeycomb.io, and much more.

One could imagine Oxide implementing its telemetry on top of the tracing ecosystem. We could define collectors which include dynamically-configurable actions such as: sending data to new endpoints; emitting log messages; manipulating counters or other contextual data; or changing the "log level" of the handlers. The various handlers implementing the above can be easily composed using the related tracing-subscriber crate, for example. Being part of the Tokio stack, integration with async code can be done simply.

The details of the proposed crate(s) for implementing our telemetry system remain to be seen, however the tracing-rs ecosystem seems to provide a large amount of important functionality, as well as powerful and widely-used abstractions for building custom tracing toolkits.

Tests were run the following machine:

Numbers reported below focus on CPU and memory utilization. CPU is reported as a percentage, where 100% represents one logical core, for a theoretical maximum of 800% on this machine. Memory refers to the resident set size, unless otherwise noted.

Tests limiting memory resources, and the ClickHouse clustering tests, were run using Docker Desktop for Mac, version 20.10.5 (build 55c4c88). Note that this uses Apple’s hypervisor framework directly, there’s no other VM between the host OS and the container. Access to CPU was not restricted, and thus 100% represents a full logical core. It’s not clear from Docker’s documentation what measurement the memory utilization really refers to, but some articles indicate it measures RSS. The VictoriaMetrics Docker image used had ID victoriametrics/victoria-metrics@sha256:a9fbe1ef0a55a2b8d07fa8663dd7b034084a05e3aabaa31071dc3409789d4d6f, and the ClickHouse image had ID yandex/clickhouse-server@sha256:d359e6cff39486d2563cd16a307e0981ea6150760d7589b35e0e76ee7ba889ce. Both were running a LinuxKit kernel, version 4.19.121.

In all cases, the same client application was used — a Rust binary running directly on the host machine.

Tests were run with 10 projects, teams, and sleds, with either 10, 20, 50, or 100 VMs. 22 total measurements were generated, which are stored as 22 columns (in 4 tables) in the case of ClickHouse, or distinct timeseries in VictoriaMetrics. The total data size in these cases were: 1K - 100K rows per insertion period (10s) for ClickHouse, or 22K-2.2M total timeseries per insertion period for VictoriaMetrics. (The total amount of data is identical, only the storage details differed.) Data was inserted for one hour, during which time a range of queries was run manually, to keep the system busy and experiment with the query languages. Data types were very simple — a millisecond precision timestamp along with unsigned 32-bit integers for the remaining fields. The data we’ll store will almost certainly include other fixed-width types, as well as variable length data such as strings. These were chosen for simplicity, and to allow more direct comparison between the databases. Each uses faily complicated encoding and compression schemes, and I thought it most helpful to sidestep those details when comparing disk and memory usage.

ClickHouse's resource utilization was very predictable. It generally consumed about 10-50% of a CPU, and memory and disk utilization grew linearly as data was inserted. There were brief spikes of CPU utilization, up to about 75%, as queries were run and new data was inserted. This is likely due to CH’s background tasks for merging and compressing data, or in the case of some larger queries. All queries were snappy, even large (and pointless) joins across multiple tables. Using the system was subjectively "nice", as CH supports SQL, albeit with some oddities and lots of additional functions and features.

VictoriaMetrics: Resource utilization was overall similar to ClickHouse, though the pattern was markedly different. Immediately upon starting to insert data, CPU jumped to about 100-150% and memory usage to 400MiB. CPU then settled down over the first few minutes to a steady state of about 5-20% throughout the rest of the test. Memory inched up a bit higher than CH, to 500-600MiB in total by the end of the hour. Querying was also subjectively fast, and the compact expressions for historical queries are extremely nice.

The behavior for 20 and 50 VMs was largely the same (though scaled up) as the 10-VM case. To effectively stress the systems, they were run with strict resource limitations and 100 VMs in total. Containers for each database server were run using the official images available on Docker Hub, using the command docker run --memory 500000000, i.e., limited to 500MB of memory. Using 100 unique VMs to generate metrics results in 100K unique instances, which equates to 100K rows in 22 columns inserted per period for ClickHouse, or 2.2M unique time points per period in VictoriaMetrics.

ClickHouse effectively maxes out the memory of its container, hitting about 95% immediately and staying there. The CPU was not pegged, however, and the server remained responsive to client queries. Inserts appeared to return just as quickly (though this was not rigorously confirmed), and command-line queries to the DB were similarly snappy as the unconstrained case. Overall, CH performed extremely well here.

VictoriaMetrics did not fair as well. It also maxed out the resources of its environment immediately, however, inserts were obviously much slower — the metric generator client slowed to a crawl and within a minute or two started generating the following errors:

Though it is an educated guess, this exhaustion is likely a result of memory limitations. The Docker container was given access to all cores, and CPU utilization was well-below the theoretical maximum of 800% on this machine, while memory usage was above 97% for the duration of this short test.

Most surveyed solutions do not offer clustering, either at all or only as part of a paid plan. VictoriaMetrics and ClickHouse were the only exceptions, the main reason for comparing these two in detail.

VM’s clustering is fairly simplistic — users are required to manage the components of the cluster themselves, including the separate storage nodes, one or more vminsert instances to insert data (possibly behind a load-balancer), and one or more vmselect instances against which queries are run. Data is entirely replicated between the storage nodes, with no support for sharding. Multitenancy is supported.

ClickHouse’s clustering is more full-featured. Sharding and replication are independently controlled, with any level of either aspect supported. ZooKeeper is used to control data replication, but is not involved in non-modifying queries of the data. The set of servers involved in clustering, including which shards or replicas they manage, is defined in a static configuration file. However, the actual distribution of data is defined per table, so that one may support separate sharding or replication patterns for any table.

Given the drastic performance differences above, and the fact that ClickHouse is already the leading candidate, only ClickHouse was evaluated for its clustering behavior. Testing was done again using Docker containers, specifically, following the procedure described here, which used Docker Compose to orchestrate multiple servers. A single ZooKeeper container was run, along with 6 ClickHouse servers (3 shards, each with two replicas). The hits table as described in the official clustering guide was created, and the example dataset was broken into 1000 line chunks to test the behavior of incremental insertions into the cluster.

The setup of CH’s clustering is very flexible. In this instance, a local table was created on each server which contained a shard of the total data. Each of these tables used the ReplicatedMergeTree table engine to transparently replicate the data between the different servers. On top of this, a Distributed table was created — this contains no additional data, instead providing a view onto the local tables. Data can be inserted directly into the local tables, and will appear in the distributed table, or vice versa. (The former might be done if the client can better shard the data than the server, for example.) The advantage of the distributed table is that it can be created on each server, allowing them all to serve a consistent view of the data — no server is special, and clients may operate against any one of them.

In general, data inserted into the distributed table appears more or less immediately on any of the servers. All queries are consistent, fast, and correct. As a more interesting test, one of the Docker containers in the cluster was paused, to simulate a network partition. The cluster continued to perform without errors in this case, both for read-only queries and when inserting new data. Queries occasionally exhibited increased latency, though this was uncommon.

After unpausing the container, its CH server immediately rejoined the cluster and began receiving any new data. Queries against the newly-restarted server occasionally returned partial results (e.g., only 300 of 1000 new rows were available), though this was very quickly resolved (less than a second), and often not seen at all.

As a final stress test, the ZooKeeper node itself was paused. The behavior here was interesting (and extremely reassuring!). Read-only queries against any node in the cluster continued to perform well, returning results quickly and accurately. Inserting new data, however, resulted in an error:

Because CH cannot guarantee the consistency of the replicated table, it converts the table to read-only mode. Note that all queries, even those which required accessing multiple shards, continued to work. ZK is only used to consistently manage the cluster state when the underlying tables change, while the different servers communicate directly to serve read queries. Once the ZK container was unpaused, data could be inserted successfully without issue.

Overall, CH’s clustering is excellent. Though a bit tedious and error-prone to set up, the system performed extremely well: normal usage saw little to no noticeable client-side performance impact, and network partitions were gracefully handled with minimal impact on the user and zero impact on the data itself.

At this point, we’ve decided to move forward with ClickHouse as the underlying storage technology for our metric and telemetry data. Its performance is extremely impressive, even in highly constrained environments. It is extremely flexible, supporting the full SQL standard as well as a large number of extension and custom functions. The documentation is sometimes a bit opaque (clearly written by a non-native English speaker, for example), but it is extensive and detailed. And finally, though a bit difficult to set up, ClickHouse’s support for clustering and HA is extremely robust and flexible.

A number of outstanding questions remain, especially relating to access to telemetry data for both customers and Oxide. This section attempts to constrain possible answers to these questions, in the context of using ClickHouse as the underlying storage system. No determinations have been or will be made at this time — this section simply provides notes about important open questions and avenues for tackling them using ClickHouse.