RFD 634 Git ref files for Dropshot versioned APIs

A versioned API is one where not all clients are expected to be at the same version, typically because an automated update is in progress. Unlike a lockstep API, which is stored as a single file, a versioned API is stored as a directory of files, one per version. Once a version of an API is integrated into main, it is considered blessed: an immutable source of truth.

For example, the Sled Agent API is a versioned API, since a newer version of Sled Agent might have older versions of Nexus and other system components as clients. The Dropshot API manager stores each version of the Sled Agent API as a separate JSON file, and also manages a latest symlink pointing to the latest version.

The [progenitor] client for Sled Agent refers to the latest symlink, so that when a new version of the Sled Agent API is added, the in-tree client automatically picks up the update.

A Git ref is a text file with a .gitref extension, containing a single line of the form <commit-hash>:<path-to-file> followed by a newline. (This usage of "Git ref" is unrelated to Git references such as branches and tags.)

An example Git ref is:

The <commit-hash> field is a full Git commit hash: 40 hexadecimal characters for SHA-1 repositories, or 64 for SHA-256. The <path-to-file> field is the path to a file at that commit, using forward slashes on all platforms, including Windows.

A Git ref can be dereferenced, or followed, by reading the file contents at the specified revision. The format is designed so it can directly be provided as an argument to commands like git show:

For commands that don’t accept arguments in a <commit>:<path> format similar to git show, standard Unix tools such as cut can be used to extract the components:

Alternatively: IFS=: read -r c p < file.gitref; jj file show -r "$c" "root:$p".

(The Jujutsu root: fileset resolves paths from the root of the repository, regardless of the current working directory.)

This RFD proposes that the Dropshot API manager optionally convert eligible API versions to Git ref files. This capability is called Git ref storage.

A versioned API document is stored as a Git ref if all of the following are true:

If an API should be stored as a Git ref, but is currently stored as a JSON file, then the Dropshot API manager will automatically convert the JSON to a Git ref, storing it as filename.json.gitref. The <commit-hash> refers to the commit that most recently added the file. (If a file is removed and later re-added, the commit that re-added it is used, not the original.)

If an API should be stored as JSON, but is currently stored as a Git ref, then the Dropshot API manager will automatically convert the Git ref back to a JSON file.

If both .json and .json.gitref are found for an API, the Dropshot API manager will delete whichever one is redundant, based on the above rules.

This RFD introduces an approach for referring to an old version of a file. The general strategy is inspired by [git-lfs], which stores pointer files with the format:

One key difference between Git LFS and Git ref files is that with the former, the referred-to files are stored on an external server. With Git ref files, they are stored within the same repository. The respective choices make sense because Git LFS is meant for files that are too large to comfortably fit in a Git repository. OpenAPI documents are small enough to store directly.

Alternative: keep current storage, tooling on top. We can choose not to change how documents are stored in source control, and instead write a utility to perform the appropriate diffs (and leave, e.g., a comment on the PR). That is simpler to implement, but it doesn’t solve the issue that API change diffs appear to be very large on GitHub. It also doesn’t restore the ability to perform blame operations in the GitHub UI.

Alternative: use Git LFS. There are two main downsides to using Git LFS: first, an external server is required to serve objects; and second, LFS files can only be diffed with some difficulty. These tradeoffs are reasonable for the kinds of large binary assets Git LFS is designed for, but not so much for OpenAPI documents.

Alternative: do not store Git ref files, always do history lookups. We could store only the latest version, and use git log on a directory to find old versions of APIs. However, that results in a significant legibility loss: making a list of old APIs requires looking at Git history. It also becomes unclear when support for an API version is dropped. With Git ref files, dropping support for an API version means deleting the corresponding .gitref file.

Alternative: don’t store Git ref files as regular files. There are several alternatives to storing Git ref files as regular files, such as Git notes. Regular files on disk are significantly more legible than these less common Git features.

Alternative: smudge/clean filters. We could have fully materialized files for each API version on disk, but when storing them in the repository, use a Git clean filter to convert them to Git ref files. Then, when updating to a revision, use a smudge filter to materialize them. This option has a few issues:

By being regular files with a .gitref suffix, Git ref files address all these concerns.

Alternative: lobby GitHub to turn on --find-copies-harder. Given the inherent computational complexity of --find-copies-harder and the general priorities of GitHub, a lobbying effort seems unlikely to succeed. The problem discussed in [why-no-git-copy-detection] below—that a copy might be detected against a different version than desired—also remains.

Alternative: convert API versions to Git ref files via a separate command. This alternative has the downside of inconsistent results when one side of a merge does conversions to Git ref files and the other side does not. Having a uniform, automatic strategy removes this possibility and reduces developer burden.

The Git ref format is chosen to make git show $(cat file.gitref) work without manipulation of the Git ref’s contents. This is the most natural format for Git ref files.

Alternative: a single manifest listing all prior versions. One option is to have, for example, an old-versions.json file listing out all prior versions. This is not unreasonable, but it would complicate git show usage. It also has a greater chance of resulting in merge conflicts, particularly around situations where one side of a merge adds multiple versions while the other side only adds one version. Separate Git ref files per API version address both these concerns.

Alternative: use the merge base. This RFD proposes storing the commit where a file was most recently introduced. An alternative is to store the merge base between the current branch and main. However, this leads to merge conflicts when multiple developers work on the same API but with different merge bases:

In this scenario, the merge base of feature1 and main is M2, and the merge base of feature2 and main is M3. If the Dropshot API manager stored Git ref files using the merge base, the Git ref files would have different contents, leading to a merge conflict. The choice of using the first commit when the file was most recently introduced produces results independent of developers' merge bases.

Alternative: use the first commit when the file was first introduced. This RFD specifies that we use the first commit when the file was most recently introduced. Another option is to use the first commit when the file was first introduced. For example, for an API version v1:

In this scenario, the RFD proposes storing commit C within the Git ref. The alternative approach would store commit A.

A core assumption behind API versioning is that blessed versions are immutable. Under this assumption, in the situation described above, both A and C are valid options for the Git ref. But we have to pick a consistent approach, and commit C is slightly better because it represents an unbroken history.

If, for some reason, this assumption is not upheld, then the hash in the file name will almost certainly be different, and C would be the only valid option.

Git ref storage for older versions introduces significant tradeoffs. On balance, the benefits are large and the drawbacks can be mitigated, but the tradeoffs are worth documenting and considering. While the change impacts the whole team, if we eventually decide the costs outweigh the benefits, it is reversible: the Dropshot API manager can convert Git ref files back to JSON files, and the original JSON contents remain in Git history.

At Oxide, we maintain [progenitor] to generate Rust HTTP clients from OpenAPI documents. With server-side-versioned APIs [rfd532], clients typically only need the latest version (accessed via the -latest.json symlink). With Git ref storage, the latest version is always available as a materialized JSON file, so this common case is unaffected.

In some cases, clients for older versions of APIs are required:

For now, the workaround is to disable Git ref storage for these APIs. But in the future, we can build into Progenitor the ability to dereference Git ref files (either as a fully integrated feature, or by enabling Progenitor to run scripts to fetch OpenAPI document contents).

In the rare scenario where Git history is rewritten with commands like [git-filter-repo], the Git ref files may become invalid. Should that need arise, we can build the required tooling to map Git ref files over to their new versions.