Oxide racks enforce quotas on resource usage for a variety of reasons. For example, quotas protect the community of users by preventing unforeseen spikes in usage. This way no single user can accidentally consume all the resources in a rack or fleet of racks, such that no other users can deploy resources. Although the API supports an arbitrary number of objects, operators or the system itself may place limits on the number of resources that users can create. These are known as quotas.
In [rfd43], we went over Identity and Access Management (IAM) policies. This RFD is going to be focused on Quotas. While IAM policies focus on who is allowed to access what, and are attached to different resources. Quota policies focus on the number of resources a project is allowed to consume. Although, different from [rfd43], the mechanics of attaching a policy to a project or organization are similar.
Quota policies can only be attached to an organization or project, not individual resources. This is where Quota policies differ from IAM policies.
Eventually, when users have more than one rack and different regions, you can imagine they might want to set different quota policies per region. This will be possible. Initially, we will set the policies "globally" (their first rack), from there once more regions are defined for the fleet, users can set specific policies per region. The API should allow for a different parameter to be set when accessing policies per region.
When a user goes to create a resource in a project, IAM will check if they are authorized to do so, quota policies are then checked to make sure the project has the quota to create the additional resource. If the project does not have enough quota we should return a specific error response in the API as covered in [RFD 4] error responses.
Policy inheritance
Please refer to [rfd44] to understand the hierarchy of resources.
Child resources of a parent resource automatically inherit the policies from the parent resource by default. This is the same behavior as [rfd43] as well.
For example, if an organization has a specific quota granted to it for various resources, projects in that organization will inherit those quotas.
For example, if an organization has a quota of 100 instances, all projects in the organization inherit the quota of 100 instances. This way you can set quota policies at the organization level that can be inherited by projects in that organization.
Regional policies versus global
Regional policies inherit from the global policy. Unless the quotas are explicitly modified for a region, the quota will be taken from the global quota policy.
Requesting more quota
When a project is out of quota for a particular resource and needs more, they will need to send an alert to their administrator. This should be easy someone to do from the UI, CLI, or API. Refer to [rfd55] for more on alerts.
The administrator (aka person with the IAM permission capable of changing quotas) can then modify the specific projects quota for that resource.
Not all projects have the same quotas. As your use of Oxide expands over time, your quotas may increase accordingly. If you expect a notable upcoming increase in usage, you can proactively request quota adjustments from the UI, CLI, or API.
Alerting when quota is about to run out
We should periodically check and alert project owners when their quota is about to run out. We can do this through the UI and through other alerting mechanisms as outlined in [rfd55].
Understanding VM, CPU, and IP address quotas
When planning your virtual machine (VM) instance needs, there are a number of quotas to consider that affect how many VM instances you can create.
VM quotas are managed at the regional level. VM instance, CPU, and Disk quotas can be consumed by any VM in the region, regardless of zone. For example, CPU quota is a regional quota, so there is a different limit and usage count for each region. To launch an 16 core instance in any zone in X region, you need enough quota for at least 16 CPUs in X region.
VPC and load balancing quotas are required to create firewalls, load balancers, subnets, and VPNs.
CPUs
CPU quota is the total number of virtual CPUs across all of your VM instances in a region. CPU quotas apply to running instances and instance reservations. Both standard and spot instances consume this quota. The differentiation between the two is covered in [RFD 4].
Virtual machine types
Quotas can be placed on machine types to limit the number of each specific machine type projects or organizations can deploy.
Quotas for spot instances
To use spot CPUs attached to spot VM instances, or to use disks attached to spot VM instances, you must have available quota in your project for the respective resource.
Disk quotas
This is the total size of disks that can be created in a region. Disks that are not attached to an instance still count against quota.
IP addresses
You must have enough public IP addresses for every VM that needs to be reachable from the public internet. Regional IP quota is for assigning IP addresses to VMs in that region.
In-use IP address. Includes both ephemeral and floating IP addresses that are currently being used by a resource.
Floating IP addresses that are not in use by a resource but are still reserved count against the quota.
Example policy
The syntax for quota policies is the following: "resource_type": integer. This
applies for all resources. There is an example below:
{
"ncpus": 96,
"nmemory": 256,
"ndisks": 2500,
"nvpcs": 100,
"nsubnets": 100,
"nrouters": 100,
"nroutes": 1000,
"nfirewallRules": 10000,
"nephemeralIPAddresses": 300,
"nfloatingIPAddresses": 20,
"nsnapshots": 5000,
"nimages": 5000,
"vms": {
"{type/size}": 10,
"{type/size}": 20,
}
}| 1 | You can add quotas per type of VM machine as well. |
External References
[rfd4] Oxide Computer Co. RFD 4: User Facing API. 2019.
[rfd43] Oxide Computer Co. RFD 43: Identity and Access Management. 2020.
[rfd44] Oxide Computer Co. RFD 44: Hierarchy and API for User Management and Organizations. 2020.
[rfd55] Oxide Computer Co. RFD 55: User Alerts. 2020.