Background
In [rfd53], we outlined our data storage needs for the Oxide control plane; in [rfd110], we outlined our rationale for selecting CockroachDB as that database. In selecting CockroachDB, our reservations were primarily around its licensing: when we made the decision, Cockroach Labs had already changed their licensing from Apache 2.0 to the BSL 1.1 in [cockroach-bsl] — with the following Additional Use Grant:
You may make use of the Licensed Work, provided that you may not use the Licensed Work for a Database Service.
A “Database Service” is a commercial offering that allows third parties (other than your employees and contractors) to access the functionality of the Licensed Work by creating tables whose schemas are controlled by such third parties.
As [rfd53] indicates, the control plane database is not used for a database service; this restriction prevents the BSL 1.1 from being an OSI-approved open source license, but did not feel otherwise prohibitively limiting. Beyond the BSL, CockroachDB used the (ill-named) Cockroach Community License for source-available features that require a commercial license to use. As [rfd110] explains, we made the determination to use the OSS build, which includes no CCL code.
CockroachDB becomes proprietary
On August 15th, 2024, Cockroach Labs announced in [cockroach-proprietary] that they would be changing to a strictly proprietary model: on November 18th, 2024, the BSL code will be relicensed to be source-available but otherwise proprietary (that is, requiring a license to use). This new license — which appears to be called the CockroachDB Software License — contains additional strictures beyond the CCL with respect to enforced telemetry, mandatory license keys, benchmarking restrictions, etc.
Whither CockroachDB?
CockroachDB has worked well for us to date; what are our options?
Replacing CockroachDB
As [rfd53] outlines, the space and performance requirements are modest — but durability, availability, and hands-off operability are essential. Replacing CockroachDB with something else is certainly not impossible, but it would also be a significant amount of work — and that’s assuming we have something obvious to replace it with. This option might have some long term tenability, but in the short term, it’s not practical.
Commercially-licensed CockroachDB ("CockroachDB Enterprise")
Oxide embeds software in a shipping product. While we are not opposed to commercial relationships with software suppliers, it is imperative that the software that runs on the Oxide rack itself be unencumbered with respect to any software licensing. CockroachDB appears to have annual licensing (rather than perpetual) and per-core licensing (rather than per site). This option is a non-starter.
Source-available CockroachDB ("CockroachDB Enterprise Free")
Following their shift to a strictly proprietary model, Cockroach Labs is offering a free version, available to Oxide if two requirements are met:
Less than $10M annual revenue
Mandatory enabled telemetry back to Cockroach Labs.
Both of these requirements are entirely unacceptable to Oxide; this option is a non-starter.
Apache 2.0-licensed CockroachDB 22.x
The BSL contains within it an automatic conversion to the Apache Public License 2.0. According to [cockroach-licensing], here is the schedule for automatic conversion to Apache 2.0 from [cockroach-licensing-faq]:[1]
As of this writing (August 2024), we are running CockroachDB 22.1.9; this converts to Apache 2.0 in May 2025.[2] While we are abiding by the BSL 1.1 constraint of 22.1.9, we would certainly prefer to be running Apache 2.0: not only is it OSI-approved, but it is also clear with respect to nuances like patent grants. This option assumes that we will self-support, but this is not in fact a change: since our evaluation, we have known that our use case (namely, embedded in a shipping physical product and on an illumos derivative) is very unusual — and when we made the decision to integrate CockroachDB, it was with the awareness that we would self-support. Indeed, in the four years since our determination, the most serious CockroachDB issue ([omicron-1146]) did, in fact, necessitate self-support — with the root-cause ultimately determined to be an OS issue ([illumos-15254]) as regaled in [oxf-s2e38].
Under this option, we will be able to upgrade to CockroachDB 22.2 (which will convert to Apache 2.0 no later than December 6th, 2025[3]), but we will not be able to upgrade beyond the 22.2.x series: Cockroach Labs has indicated that all subsequent patch releases of the 23.1 will be relicensed.
This has the obvious benefit that this does not necessitate changing our software at all (though it stunts the CockroachDB update plans outlined in [rfd469]), with the cost that we are self-supporting on an older release of the database. However, given the limited issues that we have seen, the challenge of updating the database, and our need to self-support anyway, these strictures do not feel unreasonable — especially given the impracticality and/or unacceptability of the alternatives.
Determinations
The immediate plan is to self-support on CochroachDB 22.1 and potentially
CockroachDB 22.2; we will not upgrade CockroachDB beyond 22.2. We have
several
patches that we are floating on CockroachDB; in the foreseeable future we will
seek to integrate these into a repository in the oxidecomputer GitHub
organization to allow others to run CockroachDB as we run it. This is not
intended to be a community fork (we have no current intent to accept outside
contributions); we will make decisions in this repository entirely around our
own needs. If a community fork emerges based on CockroachDB 22.x, we will
support it (and we will specifically seek to get our patches integrated), but
we may or may not adopt it ourselves: we are very risk averse with respect to
this database and we want to be careful about outsourcing any risk decisions to
any entity outside of Oxide.
External References
[cockroach-bsl] Cockroach Labs. Why we’re relicensing CockroachDB. 2019.
[cockroach-licensing-faq] Cockroach Labs. Licensing FAQs. 2019.
[cockroach-proprietary] Cockroach Labs. Evolving our self-hosted offering and license model. 2024.
[illumos-15254] %ymm registers not restored after signal handler
[omicron-1146] cockroachdb crashed in Go runtime during test run
[oxf-s2e38] Oxide Computer Company. Oxide and Friends, Season 2, Episode 38, A Debugging Odyssey.
[rfd53] Oxide Computer Company. RFD 53 Control plane data storage requirements.
[rfd110] Oxide Computer Company. RFD 110 CockroachDB for the control plane database
[rfd469] Oxide Computer Company. RFD 469 Stopgap CockroachDB Updates.