RFD 358 Live Migration and Monotonic Time

In the product, sleds have an AMD Milan CPU. This means AMD support is our priority for testing and development. Still, we would like our implementations here to be general enough to support Intel CPUs as well.

To start, let’s walk through some examples of what the guest TSC would be in different circumstances, with the goal of understanding what the TSC offset should be programmed by the host as in each circumstance.

For these examples, to keep things simple, we have some of assumptions that don’t hold true in the real world, including:

To summarize, we have the following formula to calculate the TSC offset on a machine where the VM is booted:

And to calculate the offset on the destination machine in a live migration:

Recall that our formula for calculating the guest TSC is:

Let’s first look at how each of these values is represented on Intel and AMD systems.

Both SVM and VMX provide a mechanism to "scale" the TSC based on frequency. Before looking closer here, let’s review what the manuals say for each.

While described slghtly differently, both Intel and AMD allow for TSC scaling by allowing the system to program a fixed point number.

AMD’s documentation makes this more obvious by explicitly describing an MSR with an integer and fractional component, but Intel’s approach is similar.

The frequency ratio is thus represented as:

If freq_multiplier is a fixed point number, and SHIFT is the number of bits used for the fractional element, then we can refine our guest TSC and TSC offset equations from before to be:

We can calculate the TSC offset in a similar manner:

This is our formula for calculating the frequency multiplier:

where SHIFT is the number of bits of the fractional element of the multiplier.

Constraints

The size of the frequency multiplier is constrained by the architecture we are on. Those formats are:

Areas of flexibility

We have some choice in how we represent the frequencies of the guest and host, both in terms of units and integer type used to represent them.

(Today, the TSC frequency is a unsigned 64-bit integer representing the frequency in hz.)

Limitations

Regardless of the types we choose for the frequency, the possible limitations are the same. We can run into trouble if:

Our formula for calculating the TSC offset is:

Understanding where each calculation could run into underflow/overflow is helpful, but it doesn’t offer a concrete sense of what real life values might get us there.

A good starting point is the ratio between guest to host frequencies we will allow, as this is what’s used for scaling.

Let’s work through an example to see what happens. Suppose we allowed the maximum integer component ratio allowed by AMD: 255. How would that impact these calculations?

A ratio of 255 would give us the frequency multiplier:

This is a value using all 40 bits of the multiplier. Recall that our host scaling operation looks like:

Both initial_host_tsc and freq_multiplier have a max value of 64 bits. Assuming we can store the product in a 128-bit value, we only will overflow this value if the result after the shift is greater than 64-bits, as that’s the size of the offset. The maximum bits that can be used in the product of a binary multiplication is the sum of the number of bits needed for each operand; here, that’s int bits at a max for the multiplier, and 64 bits max for the TSC. In order to not overflow the scaled TSC, then, it can’t go above 56 bits.

The max scaled host TSC value would thus be 2^56 - 1 = 72057594037927935. At a speed of 1 GHz, that value would be reached in just over 2 years. But keep in mind that this is a scaled TSC, which in this example means that the guest is ticking 256x time faster, an absurd speedup.

Broadly, the tradeoff here is: Bits that are used for the ratio take away from bits that we can use to scale the host TSC. This means the scaled host TSC will overflow faster.

Clearly, allowing the max ratio is not reasonable (nor practical, as a frequency difference of 256x between CPUs is unheard of). Ideally, we want to pick a reasonable upper bound, ensuring that scaling the host TSC will not overflow for reasonable circumstances and move on.

The next question is what a reasonable ratio limit would be, knowing that each bit used for the int portion of the ratio takes away from how much the host TSC can be scaled, and assessing what reasonable assumptions we can make about how quickly CPU speeds will increase.

I would propose we use at least 4 bits, for a maximum allowed ratio of 15. From some cursory research, it’s clear that 2x is not a sufficient maximum, but 8x probably will be. For some context, the Linux KVM implementation allows a max of 31 (5 bits) for AMD, and 255 (8 bits) for Intel.

Here is a summary of the absolute maximum guest lifetime, meaning that depending on when it was booted, it might not be able to live this long, for different ratios and guest frequencies. A 2x speedup seems plausible, and 15x and 31x much less so. Keep in mind that the ratio implies the guest is running much faster, and presumably began its life on much faster hardware.

Note: to generate the values for the above table, I used the formula:

We only begin to approach worrying low lifetimes in the 31x ratio range, for ridiculous speed differences. 15x also seems sufficient for these purposes.

Because VMs in some circumanstances may live much longer than physical hosts, it seems more plausible that VMs will move to hosts with faster TSC frequencies than they have. So how does that impact us?

First, it’s notable that in a perfectly-modeled system, for any ratio that can’t be perfectly represented in a binary fraction (e.g., 0.5, 0.25, 0.75, etc…​), calculating the guest TSC will lead to some drift.

As an example using tsc-simulator, a tool I wrote to help me model some of these calculations, here is an example guest TSC over time for a guest of 1GHz, and a host of 3GHz, for a ratio of 1/3:

Some amount of drift is fine and expected in a real system, but since the TSC is used to adjust the wall clock time, too much drift can upset userspace time synchronization daemons such as ntp.

As a strawman, I considered ntp’s threshold for max frequency error it can correct, 500 ppm. In some initial modeling of how this drift might work, I ran the following test:

For pretty substantially small ratios, I didn’t see error beyond even 1 ppm. It’s possible this isn’t the correct way to model this, and I’m open to suggestions there. I suspect modeling error over a longer period of time, and through multiple migrations might be required.

Further, one bit that hasn’t been discussed yet is the passing of time during migration, in which we presumably will need to move the wall clock forward to account for migration time. Given that the TSC is used to update the wall clock as well, we will need to consider what reasonable error bounds should be there as well — such as moving the TSC forward some amount. That seems far more likely to run into issues.

Given that there are several open questions here, I don’t yet have a good proposal for how we should think about the fractional portion of the ratio. But I’d like to get some feedback on this RFD in parallel with other work I’m doing.