Timekeeping Virtualization for X86-Based Architectures

1. 개요

x86 platform, 특히 이를 virtualization하는 작업에서 가장 복잡한 부분 중 하나는 사용할 수 있는 timing device가 매우 많고 각 device의 emulation도 복잡하다는 점입니다.

Virtualization은 guest CPU가 통제할 수 없는 시간 multiplexing을 추가합니다. Guest가 실행되지 않는 구간에도 real time은 흐르므로 guest가 관찰하는 CPU time, interrupt와 wall clock 사이의 관계가 bare metal과 달라집니다.

이 문서는 먼저 timekeeping hardware를 설명한 뒤 발생하는 문제와 가능한 해결책을 제시하고, KVM guest 유형별 권고에 필요한 자료를 한곳에 모읍니다. 다른 곳에서 찾기 어려운 KVM 및 hardware virtualization 관련 시간 정보를 정리하는 것이 목적입니다.

.. SPDX-License-Identifier: GPL-2.0

======================================================
Timekeeping Virtualization for X86-Based Architectures
======================================================

:Author: Zachary Amsden <zamsden@redhat.com>
:Copyright: (c) 2010, Red Hat.  All rights reserved.

.. Contents

   1) Overview
   2) Timing Devices
   3) TSC Hardware
   4) Virtualization Problems

1. Overview
===========

One of the most complicated parts of the X86 platform, and specifically,
the virtualization of this platform is the plethora of timing devices available
and the complexity of emulating those devices.  In addition, virtualization of
time introduces a new set of challenges because it introduces a multiplexed
division of time beyond the control of the guest CPU.

First, we will describe the various timekeeping hardware available, then
present some of the problems which arise and solutions available, giving
specific recommendations for certain classes of KVM guests.

The purpose of this document is to collect data and information relevant to
timekeeping which may be difficult to find elsewhere, specifically,
information relevant to KVM and hardware-based virtualization.

2. Timing Devices

2. Timing device

먼저 기본 hardware timing device를 살펴봅니다. 각 device는 interrupt 방식, 접근 interface, 정밀도와 system-wide 또는 CPU-local 여부가 다릅니다.

TSC와 관련 KVM clock은 다른 timer보다 특별한 고려가 필요하므로 다음 장에서 별도로 자세히 다룹니다.

=================

First we discuss the basic hardware devices available.  TSC and the related
KVM clock are special enough to warrant a full exposition and are described in
the following section.

2.1. i8254 - PIT

2.1. i8254 PIT

PIT(Programmable Interrupt Timer)는 초기 PC timer 중 하나로 1.193182 MHz 고정 base clock과 세 channel을 갖습니다. 각 channel은 자체 counter와 mode를 가지며 periodic 또는 one-shot interrupt를 만들 수 있습니다.

원래 IBM PC에서 channel 1과 2는 일반 용도가 아니었고 각각 RAM refresh와 PC speaker 제어에 연결되었습니다. 현재는 PIT가 보통 emulated chipset에 통합되므로 별도 physical PIT는 쓰지 않습니다.

PIT는 I/O port `0x40-0x43`을 사용합니다. 16-bit counter는 I/O port에 한 byte 또는 여러 byte로 접근하며, command port `0x43`이 timer별 counter access 방식과 mode를 설정합니다.

여섯 mode가 있지만 모든 timer가 모든 mode를 지원하지는 않습니다. Mode 1과 5에 필요한 gate input이 실제로 연결된 것은 timer 2뿐이며, 그 gate는 port `61h` bit 0으로 제어합니다.

Mode 0은 software single timeout입니다. Gate가 high일 때 count down하고 0에 도달하면 output이 high가 됩니다. Timer 0과 1의 gate는 항상 high입니다.

Mode 1은 triggered one-shot입니다. 처음 output은 high이며 gate가 high로 바뀌면 output을 low로 두고 countdown을 시작합니다. 시작 후 gate가 내려가도 멈추지 않고 count 0에서 output이 다시 high가 됩니다.

Mode 2는 rate generator입니다. Countdown이 1에 도달하면 한 count 동안 output을 low로 만들고 다시 high로 바꾼 뒤 값을 reload해 자동 반복합니다. Gate가 low면 count를 멈추며 timer 2의 output이 low일 때 gate를 내리면 output이 즉시 high가 됩니다.

Mode 3은 high/low square wave를 만듭니다. Gate가 high일 때만 count하며 0에서 자동 reload하고 clock마다 두 번 감소시켜 설정 periodic rate에서 전체 high/low cycle을 만듭니다.

Mode 3의 짝수 count N은 high N/2, low N/2이고 홀수 N은 high (N+1)/2, low (N-1)/2입니다. Counter는 짝수 값만 latch하므로 read에서는 홀수를 관찰하지 않습니다. Timer 2가 square wave를 low-pass filtering해 sine과 비슷한 tone을 만드는 의도된 mode입니다.

Mode 4 software strobe는 programming과 counter load 뒤 count 0까지 output high를 유지하고 한 clock cycle만 low로 만든 뒤 high로 돌아옵니다. Reload하지 않으며 gate가 high일 때만 count합니다.

Mode 5 hardware strobe는 programming과 load 뒤 output high를 유지하다 gate rising edge에서 countdown을 시작합니다. 시작 뒤 gate가 낮아져도 멈추지 않고 0에서 한 cycle low pulse를 낸 뒤 high로 복귀하며 reload하지 않습니다.

PIT는 일반 binary count 외에 BCD count도 지원합니다. Port `0x43` command의 bits 7-4는 command, bits 3-1은 mode, bit 0은 binary(0) 또는 BCD(1)를 선택합니다. Mode encoding `11X`는 undefined입니다.

`1110` status latch 뒤 ports `0x40-0x42`를 읽으면 output pin, count load/expire 상태, read/write mode, timer mode와 binary/BCD 상태를 얻습니다.

----------------

One of the first timer devices available is the programmable interrupt timer,
or PIT.  The PIT has a fixed frequency 1.193182 MHz base clock and three
channels which can be programmed to deliver periodic or one-shot interrupts.
These three channels can be configured in different modes and have individual
counters.  Channel 1 and 2 were not available for general use in the original
IBM PC, and historically were connected to control RAM refresh and the PC
speaker.  Now the PIT is typically integrated as part of an emulated chipset
and a separate physical PIT is not used.

The PIT uses I/O ports 0x40 - 0x43.  Access to the 16-bit counters is done
using single or multiple byte access to the I/O ports.  There are 6 modes
available, but not all modes are available to all timers, as only timer 2
has a connected gate input, required for modes 1 and 5.  The gate line is
controlled by port 61h, bit 0, as illustrated in the following diagram::

  --------------             ----------------
  |            |           |                |
  |  1.1932 MHz|---------->| CLOCK      OUT | ---------> IRQ 0
  |    Clock   |   |       |                |
  --------------   |    +->| GATE  TIMER 0  |
                   |        ----------------
                   |
                   |        ----------------
                   |       |                |
                   |------>| CLOCK      OUT | ---------> 66.3 KHZ DRAM
                   |       |                |            (aka /dev/null)
                   |    +->| GATE  TIMER 1  |
                   |        ----------------
                   |
                   |        ----------------
                   |       |                |
                   |------>| CLOCK      OUT | ---------> Port 61h, bit 5
                           |                |      |
  Port 61h, bit 0 -------->| GATE  TIMER 2  |       \_.----   ____
                            ----------------         _|    )--|LPF|---Speaker
                                                    / *----   \___/
  Port 61h, bit 1 ---------------------------------/

The timer modes are now described.

Mode 0: Single Timeout.
 This is a one-shot software timeout that counts down
 when the gate is high (always true for timers 0 and 1).  When the count
 reaches zero, the output goes high.

Mode 1: Triggered One-shot.
 The output is initially set high.  When the gate
 line is set high, a countdown is initiated (which does not stop if the gate is
 lowered), during which the output is set low.  When the count reaches zero,
 the output goes high.

Mode 2: Rate Generator.
 The output is initially set high.  When the countdown
 reaches 1, the output goes low for one count and then returns high.  The value
 is reloaded and the countdown automatically resumes.  If the gate line goes
 low, the count is halted.  If the output is low when the gate is lowered, the
 output automatically goes high (this only affects timer 2).

Mode 3: Square Wave.
 This generates a high / low square wave.  The count
 determines the length of the pulse, which alternates between high and low
 when zero is reached.  The count only proceeds when gate is high and is
 automatically reloaded on reaching zero.  The count is decremented twice at
 each clock to generate a full high / low cycle at the full periodic rate.
 If the count is even, the clock remains high for N/2 counts and low for N/2
 counts; if the clock is odd, the clock is high for (N+1)/2 counts and low
 for (N-1)/2 counts.  Only even values are latched by the counter, so odd
 values are not observed when reading.  This is the intended mode for timer 2,
 which generates sine-like tones by low-pass filtering the square wave output.

Mode 4: Software Strobe.
 After programming this mode and loading the counter,
 the output remains high until the counter reaches zero.  Then the output
 goes low for 1 clock cycle and returns high.  The counter is not reloaded.
 Counting only occurs when gate is high.

Mode 5: Hardware Strobe.
 After programming and loading the counter, the
 output remains high.  When the gate is raised, a countdown is initiated
 (which does not stop if the gate is lowered).  When the counter reaches zero,
 the output goes low for 1 clock cycle and then returns high.  The counter is
 not reloaded.

In addition to normal binary counting, the PIT supports BCD counting.  The
command port, 0x43 is used to set the counter and mode for each of the three
timers.

PIT commands, issued to port 0x43, using the following bit encoding::

  Bit 7-4: Command (See table below)
  Bit 3-1: Mode (000 = Mode 0, 101 = Mode 5, 11X = undefined)
  Bit 0  : Binary (0) / BCD (1)

Command table::

  0000 - Latch Timer 0 count for port 0x40
	sample and hold the count to be read in port 0x40;
	additional commands ignored until counter is read;
	mode bits ignored.

  0001 - Set Timer 0 LSB mode for port 0x40
	set timer to read LSB only and force MSB to zero;
	mode bits set timer mode

  0010 - Set Timer 0 MSB mode for port 0x40
	set timer to read MSB only and force LSB to zero;
	mode bits set timer mode

  0011 - Set Timer 0 16-bit mode for port 0x40
	set timer to read / write LSB first, then MSB;
	mode bits set timer mode

  0100 - Latch Timer 1 count for port 0x41 - as described above
  0101 - Set Timer 1 LSB mode for port 0x41 - as described above
  0110 - Set Timer 1 MSB mode for port 0x41 - as described above
  0111 - Set Timer 1 16-bit mode for port 0x41 - as described above

  1000 - Latch Timer 2 count for port 0x42 - as described above
  1001 - Set Timer 2 LSB mode for port 0x42 - as described above
  1010 - Set Timer 2 MSB mode for port 0x42 - as described above
  1011 - Set Timer 2 16-bit mode for port 0x42 as described above

  1101 - General counter latch
	Latch combination of counters into corresponding ports
	Bit 3 = Counter 2
	Bit 2 = Counter 1
	Bit 1 = Counter 0
	Bit 0 = Unused

  1110 - Latch timer status
	Latch combination of counter mode into corresponding ports
	Bit 3 = Counter 2
	Bit 2 = Counter 1
	Bit 1 = Counter 0

	The output of ports 0x40-0x42 following this command will be:

	Bit 7 = Output pin
	Bit 6 = Count loaded (0 if timer has expired)
	Bit 5-4 = Read / Write mode
	    01 = MSB only
	    10 = LSB only
	    11 = LSB / MSB (16-bit)
	Bit 3-1 = Mode
	Bit 0 = Binary (0) / BCD mode (1)

2.2. RTC

2.2. RTC

초기 PC의 두 번째 timing device는 MC146818 real-time clock입니다. 원래 chip은 obsolete이며 현재는 system chipset이 emulation하고 때로는 HPET와 복잡한 IRQ routing을 조합합니다.

RTC는 index register로 읽을 CMOS byte를 선택하는 CMOS variable interface를 사용합니다. Index register가 하나뿐이므로 CMOS와 RTC read에는 lock 보호가 필요합니다.

`hwclock` 같은 userspace utility에 직접 RTC 접근을 허용하는 것은 위험합니다. Kernel의 CMOS read/write와 충돌해 memory 내용을 손상시킬 수 있습니다.

RTC interrupt는 보통 IRQ 8로 routing됩니다. Periodic timer, 하루 한 번 alarm, MC146818의 CMOS register update 완료 interrupt로 동작할 수 있으며 status register가 interrupt 종류를 표시합니다.

Battery 전원으로 system이 꺼져 있어도 current time field를 갱신합니다. Status register가 update-in-progress를 나타내는 동안에는 current time field를 읽지 않아야 합니다.

Clock은 32.768 kHz crystal을 사용하므로 RTC가 초를 세게 하려면 register A bits 6-4에 32 kHz divider를 program해야 합니다.

--------

The second device which was available in the original PC was the MC146818 real
time clock.  The original device is now obsolete, and usually emulated by the
system chipset, sometimes by an HPET and some frankenstein IRQ routing.

The RTC is accessed through CMOS variables, which uses an index register to
control which bytes are read.  Since there is only one index register, read
of the CMOS and read of the RTC require lock protection (in addition, it is
dangerous to allow userspace utilities such as hwclock to have direct RTC
access, as they could corrupt kernel reads and writes of CMOS memory).

The RTC generates an interrupt which is usually routed to IRQ 8.  The interrupt
can function as a periodic timer, an additional once a day alarm, and can issue
interrupts after an update of the CMOS registers by the MC146818 is complete.
The type of interrupt is signalled in the RTC status registers.

The RTC will update the current time fields by battery power even while the
system is off.  The current time fields should not be read while an update is
in progress, as indicated in the status register.

The clock uses a 32.768kHz crystal, so bits 6-4 of register A should be
programmed to a 32kHz divider if the RTC is to count seconds.

This is the RAM map originally used for the RTC/CMOS::

  Location    Size    Description
  ------------------------------------------
  00h         byte    Current second (BCD)
  01h         byte    Seconds alarm (BCD)
  02h         byte    Current minute (BCD)
  03h         byte    Minutes alarm (BCD)
  04h         byte    Current hour (BCD)
  05h         byte    Hours alarm (BCD)
  06h         byte    Current day of week (BCD)
  07h         byte    Current day of month (BCD)
  08h         byte    Current month (BCD)
  09h         byte    Current year (BCD)
  0Ah         byte    Register A
                       bit 7   = Update in progress
                       bit 6-4 = Divider for clock
                                  000 = 4.194 MHz
                                  001 = 1.049 MHz
                                  010 = 32 kHz
                                  10X = test modes
                                  110 = reset / disable
                                  111 = reset / disable
                       bit 3-0 = Rate selection for periodic interrupt
                                  000 = periodic timer disabled
                                  001 = 3.90625 uS
                                  010 = 7.8125 uS
                                  011 = .122070 mS
                                  100 = .244141 mS
                                     ...
                                 1101 = 125 mS
                                 1110 = 250 mS
                                 1111 = 500 mS
  0Bh         byte    Register B
                       bit 7   = Run (0) / Halt (1)
                       bit 6   = Periodic interrupt enable
                       bit 5   = Alarm interrupt enable
                       bit 4   = Update-ended interrupt enable
                       bit 3   = Square wave interrupt enable
                       bit 2   = BCD calendar (0) / Binary (1)
                       bit 1   = 12-hour mode (0) / 24-hour mode (1)
                       bit 0   = 0 (DST off) / 1 (DST enabled)
  OCh         byte    Register C (read only)
                       bit 7   = interrupt request flag (IRQF)
                       bit 6   = periodic interrupt flag (PF)
                       bit 5   = alarm interrupt flag (AF)
                       bit 4   = update interrupt flag (UF)
                       bit 3-0 = reserved
  ODh         byte    Register D (read only)
                       bit 7   = RTC has power
                       bit 6-0 = reserved
  32h         byte    Current century BCD (*)
  (*) location vendor specific and now determined from ACPI global tables

2.3. APIC

2.3. APIC timer

Pentium 이후 processor에는 APIC(Advanced Programmable Interrupt Controller)의 일부로 CPU마다 on-board timer가 있습니다. Memory-mapped register로 접근하며 IPI와 local timer interrupt를 포함해 각 CPU의 interrupt service를 제공합니다.

이론상 APIC은 local interrupt의 안전하고 안정적인 source지만 CPU-local MMIO hardware라는 특성 때문에 실제로 많은 bug와 glitch가 있었습니다. CPU errata와 workaround가 필요할 수 있습니다.

Virtualization에서는 workaround가 MMIO 추가 read overhead를 만들거나 계산 비용이 큰 추가 기능을 요구할 수 있습니다. APIC timer는 LVT(local vector timer) register로 program하고 one-shot 또는 periodic mode로 동작하며 programmable divider로 나눈 bus clock을 사용합니다.

---------

On Pentium and later processors, an on-board timer is available to each CPU
as part of the Advanced Programmable Interrupt Controller.  The APIC is
accessed through memory-mapped registers and provides interrupt service to each
CPU, used for IPIs and local timer interrupts.

Although in theory the APIC is a safe and stable source for local interrupts,
in practice, many bugs and glitches have occurred due to the special nature of
the APIC CPU-local memory-mapped hardware.  Beware that CPU errata may affect
the use of the APIC and that workarounds may be required.  In addition, some of
these workarounds pose unique constraints for virtualization - requiring either
extra overhead incurred from extra reads of memory-mapped I/O or additional
functionality that may be more computationally expensive to implement.

Since the APIC is documented quite well in the Intel and AMD manuals, we will
avoid repetition of the detail here.  It should be pointed out that the APIC
timer is programmed through the LVT (local vector timer) register, is capable
of one-shot or periodic operation, and is based on the bus clock divided down
by the programmable divider register.

2.4. HPET

2.4. HPET

HPET는 복잡한 device로 x86 PC의 PIT/RTC 지원을 대체하려는 목적으로 설계되었습니다. 실제 PC hardware는 legacy device emulation을 계속 제공하므로 완전 대체 여부는 불확실하고, legacy-free system은 hardware timer로 HPET만 제공할 수 있습니다.

HPET specification은 최소 세 hardware timer를 요구하지만 더 많은 timer를 구현할 자유를 줍니다. Fixed timer frequency를 강제하지 않고 frequency, error와 slew의 극한값만 제한합니다.

일반적으로 HPET는 PIT/RTC보다 정밀하고 system에 하나뿐이어서 local variation과 독립적인 time source로 권장됩니다. MMIO device이며 BIOS가 ACPI table로 존재를 알립니다.

HPET의 세부 specification은 다른 자료에 잘 문서화되어 있어 이 문서의 범위를 벗어납니다.

---------

HPET is quite complex, and was originally intended to replace the PIT / RTC
support of the X86 PC.  It remains to be seen whether that will be the case, as
the de facto standard of PC hardware is to emulate these older devices.  Some
systems designated as legacy free may support only the HPET as a hardware timer
device.

The HPET spec is rather loose and vague, requiring at least 3 hardware timers,
but allowing implementation freedom to support many more.  It also imposes no
fixed rate on the timer frequency, but does impose some extremal values on
frequency, error and slew.

In general, the HPET is recommended as a high precision (compared to PIT /RTC)
time source which is independent of local variation (as there is only one HPET
in any given system).  The HPET is also memory-mapped, and its presence is
indicated through ACPI tables by the BIOS.

Detailed specification of the HPET is beyond the current scope of this
document, as it is also very well documented elsewhere.

2.5. Offboard Timers

2.5. Offboard timer

Proprietary watchdog board나 흔한 `e1000` card처럼 자체 timing chip과 kernel 또는 userspace driver가 접근할 register를 가진 확장 card가 있습니다.

이런 timer를 Linux 등 OS kernel의 clocksource로 사용한 알려진 시도는 없고 합의된 platform 규칙을 따르지 않는 방식으로 여겨집니다. 제대로 virtualize하려면 추가 지원이 필요하지만 이를 사용하는 알려진 OS가 없어 현재 중요하게 다루지 않습니다.

--------------------

Several cards, both proprietary (watchdog boards) and commonplace (e1000) have
timing chips built into the cards which may have registers which are accessible
to kernel or user drivers.  To the author's knowledge, using these to generate
a clocksource for a Linux or other kernel has not yet been attempted and is in
general frowned upon as not playing by the agreed rules of the game.  Such a
timer device would require additional support to be virtualized properly and is
not considered important at this time as no known operating system does this.

3. TSC Hardware

3. TSC hardware

TSC(Time Stamp Counter)는 이론적으로 processor가 issue한 instruction cycle을 세어 시간을 측정하는 단순한 counter지만, 실제로는 여러 hardware 문제 때문에 가장 복잡한 timekeeping device입니다.

TSC는 내부적으로 64-bit MSR이며 `RDMSR`, `RDTSC`, 가능한 경우 `RDTSCP`로 읽습니다. 과거 hardware는 TSC write를 허용해도 low 32 bits만 쓸 수 있고 upper 32 bits를 clear하는 제한이 있었습니다.

Intel family 0Fh models 3·4·6 및 family 06h models e·f에서는 이 제한이 해제되어 64 bits 전체를 쓸 수 있습니다. AMD에서는 TSC MSR write 가능성이 architectural guarantee가 아닙니다.

CPL0에서는 TSC에 접근할 수 있고 CPL>0 software의 접근은 `CR4.TSD`로 조건부 제어합니다. TSD를 enable하면 CPL>0 TSC 접근이 disable됩니다.

일부 vendor의 `RDTSCP`는 TSC와 processor number에 대응하는 indicator를 atomic하게 함께 반환합니다. SMP에서 TSC가 동기화되지 않았을 때 processor별 offset array를 선택하는 index로 쓸 수 있으며 CPUID feature bit로 존재를 확인해야 합니다.

VMX와 SVM은 guest-visible TSC에 constant offset을 더하는 virtualization hardware field를 제공합니다. 더 새로운 구현은 scaling도 약속하지만 원문 작성 시점에는 널리 사용 가능하지 않습니다.

===============

The TSC or time stamp counter is relatively simple in theory; it counts
instruction cycles issued by the processor, which can be used as a measure of
time.  In practice, due to a number of problems, it is the most complicated
timekeeping device to use.

The TSC is represented internally as a 64-bit MSR which can be read with the
RDMSR, RDTSC, or RDTSCP (when available) instructions.  In the past, hardware
limitations made it possible to write the TSC, but generally on old hardware it
was only possible to write the low 32-bits of the 64-bit counter, and the upper
32-bits of the counter were cleared.  Now, however, on Intel processors family
0Fh, for models 3, 4 and 6, and family 06h, models e and f, this restriction
has been lifted and all 64-bits are writable.  On AMD systems, the ability to
write the TSC MSR is not an architectural guarantee.

The TSC is accessible from CPL-0 and conditionally, for CPL > 0 software by
means of the CR4.TSD bit, which when enabled, disables CPL > 0 TSC access.

Some vendors have implemented an additional instruction, RDTSCP, which returns
atomically not just the TSC, but an indicator which corresponds to the
processor number.  This can be used to index into an array of TSC variables to
determine offset information in SMP systems where TSCs are not synchronized.
The presence of this instruction must be determined by consulting CPUID feature
bits.

Both VMX and SVM provide extension fields in the virtualization hardware which
allows the guest visible TSC to be offset by a constant.  Newer implementations
promise to allow the TSC to additionally be scaled, but this hardware is not
yet widely available.

3.1. TSC synchronization

3.1. TSC synchronization

대부분의 구현에서 TSC는 CPU-local clock입니다. SMP에서 CPU power-on 시점이 다르면 CPU별 TSC 시작값도 다를 수 있으며 같은 die가 clock을 공유하는 경우가 많지만 항상 그런 것은 아닙니다.

BIOS, OS 또는 system software가 power-on 중 TSC를 다시 맞추려 할 수 있습니다. 그러나 64-bit 전체를 쓸 수 없는 hardware에서는 새 CPU의 TSC를 나머지 system과 동일하게 만드는 것이 불가능할 수 있습니다.

완벽한 동기화는 모든 값이 동일 clock에서 나올 때만 일반적으로 가능하므로 single-socket system이나 특별한 hardware 지원이 없는 system에서는 보장하기 어렵습니다.

------------------------

The TSC is a CPU-local clock in most implementations.  This means, on SMP
platforms, the TSCs of different CPUs may start at different times depending
on when the CPUs are powered on.  Generally, CPUs on the same die will share
the same clock, however, this is not always the case.

The BIOS may attempt to resynchronize the TSCs during the poweron process and
the operating system or other system software may attempt to do this as well.
Several hardware limitations make the problem worse - if it is not possible to
write the full 64-bits of the TSC, it may be impossible to match the TSC in
newly arriving CPUs to that of the rest of the system, resulting in
unsynchronized TSCs.  This may be done by BIOS or system software, but in
practice, getting a perfectly synchronized TSC will not be possible unless all
values are read from the same clock, which generally only is possible on single
socket systems or those with special hardware support.

3.2. TSC and CPU hotplug

3.2. TSC와 CPU hotplug

System boot 뒤 늦게 들어온 hotplug CPU의 TSC는 기존 CPU와 동기화되지 않을 수 있습니다. System software, BIOS 또는 SMM code가 값을 맞추려 해도 완벽한 일치는 보장되지 않습니다.

따라서 처음에는 synchronized였던 system도 CPU hotplug 뒤 작은 TSC synchronization defect가 OS와 virtualization environment에 노출되는 상태로 돌아갈 수 있습니다.

------------------------

As touched on already, CPUs which arrive later than the boot time of the system
may not have a TSC value that is synchronized with the rest of the system.
Either system software, BIOS, or SMM code may actually try to establish the TSC
to a value matching the rest of the system, but a perfect match is usually not
a guarantee.  This can have the effect of bringing a system from a state where
TSC is synchronized back to a state where TSC synchronization flaws, however
small, may be exposed to the OS and any virtualization environment.

3.3. TSC and multi-socket / NUMA

3.3. TSC와 multi-socket·NUMA

Multi-socket, 특히 큰 system은 system-wide clock 하나보다 socket별 clocksource를 가질 가능성이 큽니다. 서로 다른 crystal은 frequency가 완전히 일치하지 않고 온도와 전기적 변화로 CPU clock과 TSC가 시간에 따라 drift합니다.

Clock과 bus 설계에 따라 drift가 절대 오차로 제한될 수도 있고 계속 누적될 수도 있습니다. 큰 system은 EMI와 harmonic을 줄여 FCC 기준을 만족하려고 core별 clock을 의도적으로 slew하는 spread-spectrum clocking을 사용할 수도 있습니다.

이 때문에 NUMA 또는 multiple-socket system에서는 TSC가 계속 synchronized 상태를 유지한다고 신뢰하지 않는 것이 권장됩니다.

--------------------------------

Multi-socket systems, especially large multi-socket systems are likely to have
individual clocksources rather than a single, universally distributed clock.
Since these clocks are driven by different crystals, they will not have
perfectly matched frequency, and temperature and electrical variations will
cause the CPU clocks, and thus the TSCs to drift over time.  Depending on the
exact clock and bus design, the drift may or may not be fixed in absolute
error, and may accumulate over time.

In addition, very large systems may deliberately slew the clocks of individual
cores.  This technique, known as spread-spectrum clocking, reduces EMI at the
clock frequency and harmonics of it, which may be required to pass FCC
standards for telecommunications and computer equipment.

It is recommended not to trust the TSCs to remain synchronized on NUMA or
multiple socket systems for these reasons.

3.4. TSC and C-states

3.4. TSC와 C-state

C-state, 특히 C1E와 더 깊은 idle state에서는 TSC가 진행을 멈출 수 있습니다. CPU가 다시 실행되면 다른 CPU보다 뒤처진 TSC가 됩니다.

OS는 CPU와 chipset 식별 정보로 이런 processor를 탐지해 표시해야 합니다. 필요한 경우 알려진 external clocksource에 맞춰 TSC를 앞으로 보정할 수 있습니다.

---------------------

C-states, or idling states of the processor, especially C1E and deeper sleep
states may be problematic for TSC as well.  The TSC may stop advancing in such
a state, resulting in a TSC which is behind that of other CPUs when execution
is resumed.  Such CPUs must be detected and flagged by the operating system
based on CPU and chipset identifications.

The TSC in such a case may be corrected by catching it up to a known external
clocksource.

3.5. TSC frequency change / P-states

3.5. TSC frequency와 P-state

일부 CPU는 P-state로 frequency를 바꿉니다. TSC가 같은 비율을 유지할 수도 있고 CPU frequency와 함께 바뀔 수도 있으며 전환이 stagger 또는 slew되면 순간 TSC rate를 정확히 알 수 없고 범위만 알 수 있습니다.

이 경우 TSC는 stable time source가 아니므로 알려진 안정적인 external clock에 대해 calibration해야 시간 source로 사용할 수 있습니다.

TSC가 constant rate인지 P-state와 함께 scale하는지는 model-dependent이며 CPUID, chipset 또는 vendor-specific MSR field를 검사해야 합니다.

일부 vendor bug에서는 정상 실행 중 P-state 보정은 맞지만 halted CPU가 다른 processor의 cache miss를 처리하려고 일시적으로 P-state를 올릴 때 그 CPU의 TSC가 non-halted CPU보다 빨리 진행합니다. AMD Turion이 알려진 사례입니다.

------------------------------------

To make things slightly more interesting, some CPUs may change frequency.  They
may or may not run the TSC at the same rate, and because the frequency change
may be staggered or slewed, at some points in time, the TSC rate may not be
known other than falling within a range of values.  In this case, the TSC will
not be a stable time source, and must be calibrated against a known, stable,
external clock to be a usable source of time.

Whether the TSC runs at a constant rate or scales with the P-state is model
dependent and must be determined by inspecting CPUID, chipset or vendor
specific MSR fields.

In addition, some vendors have known bugs where the P-state is actually
compensated for properly during normal operation, but when the processor is
inactive, the P-state may be raised temporarily to service cache misses from
other processors.  In such cases, the TSC on halted CPUs could advance faster
than that of non-halted processors.  AMD Turion processors are known to have
this problem.

3.6. TSC and STPCLK / T-states

3.6. TSC와 STPCLK·T-state

Processor에 들어오는 external signal도 TSC를 멈출 수 있습니다. 보통 overheating을 막기 위한 thermal emergency power control에서 사용합니다.

이 조건이 발생했다는 사실을 탐지할 방법이 일반적으로 없으므로 TSC 신뢰성 평가와 virtualization time 보정이 더 어려워집니다.

------------------------------

External signals given to the processor may also have the effect of stopping
the TSC.  This is typically done for thermal emergency power control to prevent
an overheating condition, and typically, there is no way to detect that this
condition has happened.

3.7. TSC virtualization - VMX

3.7. VMX TSC virtualization

VMX는 `RDTSC`, `RDMSR`, `WRMSR`, `RDTSCP` instruction을 조건부 trap할 수 있어 어떤 방식으로든 TSC를 완전히 virtualize할 수 있습니다.

또한 host TSC를 passthrough하면서 VMCS의 `TSC_OFFSET` field를 더할 수 있습니다. VMCS field는 전용 VMX instruction으로 읽고 써야 합니다.

-----------------------------

VMX provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP
instructions, which is enough for full virtualization of TSC in any manner.  In
addition, VMX allows passing through the host TSC plus an additional TSC_OFFSET
field specified in the VMCS.  Special instructions must be used to read and
write the VMCS field.

3.8. TSC virtualization - SVM

3.8. SVM TSC virtualization

SVM도 `RDTSC`, `RDMSR`, `WRMSR`, `RDTSCP` instruction을 조건부 trap할 수 있으므로 TSC를 완전히 virtualize할 수 있습니다.

Passthrough 방식에서는 host TSC에 SVM control block이 지정한 offset field를 더해 guest-visible TSC를 만듭니다.

-----------------------------

SVM provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP
instructions, which is enough for full virtualization of TSC in any manner.  In
addition, SVM allows passing through the host TSC plus an additional offset
field specified in the SVM control block.

3.9. TSC feature bits in Linux

3.9. Linux TSC feature bit

Architecture가 명시적으로 보장하지 않는 한 TSC가 완벽한 synchronization을 유지한다고 보장할 수 없습니다. Multi-socket 또는 NUMA에서는 local consistency가 있어도 socket별로 독립적으로 진행할 수 있습니다.

Linux의 TSC attribute feature bit는 UP 또는 single-node system에서만 의미 있다고 받아들여야 합니다. 개별 capability가 cross-socket synchronization까지 자동으로 보장하지는 않습니다.

------------------------------

In summary, there is no way to guarantee the TSC remains in perfect
synchronization unless it is explicitly guaranteed by the architecture.  Even
if so, the TSCs in multi-sockets or NUMA systems may still run independently
despite being locally consistent.

The following feature bits are used by Linux to signal various TSC attributes,
but they can only be taken to be meaningful for UP or single node systems.

=========================	=======================================
X86_FEATURE_TSC			The TSC is available in hardware
X86_FEATURE_RDTSCP		The RDTSCP instruction is available
X86_FEATURE_CONSTANT_TSC	The TSC rate is unchanged with P-states
X86_FEATURE_NONSTOP_TSC		The TSC does not stop in C-states
X86_FEATURE_TSC_RELIABLE	TSC sync checks are skipped (VMware)
=========================	=======================================

4. Virtualization Problems

4. Virtualization의 시간 문제

Virtualization에서는 host와 여러 VM이 시간을 공유합니다. Virtual OS는 CPU를 100% 사용한다고 가정할 수 있지만 실제 vCPU는 언제든 preempt될 수 있습니다.

Guest가 interrupt source를 disable해도 disable되는 것은 virtual interrupt뿐이며 physical machine과 host scheduling은 계속 진행됩니다. Real time의 흐름, machine interrupt 주입과 관련 clocksource가 더 이상 완전히 synchronized되지 않습니다.

Bare metal에서도 BIOS의 SMM이 x86 cycle을 빼앗아 비슷한 문제를 만들 수 있지만 virtualization만큼 극단적이지는 않습니다. SMM의 유사성은 여러 보정책을 bare metal에도 적용할 근거가 됩니다.

==========================

Timekeeping is especially problematic for virtualization because a number of
challenges arise.  The most obvious problem is that time is now shared between
the host and, potentially, a number of virtual machines.  Thus the virtual
operating system does not run with 100% usage of the CPU, despite the fact that
it may very well make that assumption.  It may expect it to remain true to very
exacting bounds when interrupt sources are disabled, but in reality only its
virtual interrupt sources are disabled, and the machine may still be preempted
at any time.  This causes problems as the passage of real time, the injection
of machine interrupts and the associated clock sources are no longer completely
synchronized with real time.

This same problem can occur on native hardware to a degree, as SMM mode may
steal cycles from the naturally on X86 systems when SMM mode is used by the
BIOS, but not in such an extreme fashion.  However, the fact that SMM mode may
cause similar problems to virtualization makes it a good justification for
solving many of these problems on bare metal.

4.1. Interrupt clocking

4.1. Interrupt clocking

Legacy OS는 PIT 또는 RTC의 periodic interrupt 수를 세어 시간을 유지하는 경우가 많습니다. Host virtualization engine이 초당 필요한 interrupt 수를 전달하지 못하면 guest time이 뒤처집니다.

1000 Hz처럼 높은 interrupt rate를 선택하면 문제가 특히 심하며 많은 Linux guest의 기본값이어서 주의가 필요합니다.

첫 해결책은 wall clock이나 real time을 추적하는 별도 source가 있는 guest에서 tick 부족을 무시하는 것입니다. Interrupt count를 time의 유일한 기준으로 쓰지 않는다면 보정이 필요하지 않을 수 있습니다.

두 번째는 guest에 추가 interrupt를 주입해 effective rate를 올리는 interrupt slewing입니다. Host load 또는 guest lag가 너무 크면 따라잡기 어렵고 burst injection 같은 복잡성이 생깁니다.

세 번째는 guest가 lost tick을 인식해 내부 보정하는 방식입니다. 이론상 유망하지만 Linux 구현은 오류가 많았고 널리 쓰이는 system에 buggy variant가 배포되었습니다.

Windows는 내부 시간 유지에 periodic RTC clocking을 사용해 interrupt slewing이 필요합니다. 다만 원문이 편집 메모와 함께 약 18.2 Hz인지 묻는 낮은 rate를 언급하며 실제 문제는 아직 크지 않았다고 설명합니다.

-----------------------

One of the most immediate problems that occurs with legacy operating systems
is that the system timekeeping routines are often designed to keep track of
time by counting periodic interrupts.  These interrupts may come from the PIT
or the RTC, but the problem is the same: the host virtualization engine may not
be able to deliver the proper number of interrupts per second, and so guest
time may fall behind.  This is especially problematic if a high interrupt rate
is selected, such as 1000 HZ, which is unfortunately the default for many Linux
guests.

There are three approaches to solving this problem; first, it may be possible
to simply ignore it.  Guests which have a separate time source for tracking
'wall clock' or 'real time' may not need any adjustment of their interrupts to
maintain proper time.  If this is not sufficient, it may be necessary to inject
additional interrupts into the guest in order to increase the effective
interrupt rate.  This approach leads to complications in extreme conditions,
where host load or guest lag is too much to compensate for, and thus another
solution to the problem has risen: the guest may need to become aware of lost
ticks and compensate for them internally.  Although promising in theory, the
implementation of this policy in Linux has been extremely error prone, and a
number of buggy variants of lost tick compensation are distributed across
commonly used Linux systems.

Windows uses periodic RTC clocking as a means of keeping time internally, and
thus requires interrupt slewing to keep proper time.  It does use a low enough
rate (ed: is it 18.2 Hz?) however that it has not yet been a problem in
practice.

4.2. TSC sampling and serialization

4.2. TSC sampling과 serialization

CPU cycle counter는 가장 정밀한 time source지만 local이고 불안정하거나 동기화되지 않을 수 있습니다. 높은 정밀도 때문에 sampling delay와 instruction ordering 문제도 두드러집니다.

Counter를 읽은 순간 그 값은 이미 과거 값입니다. Superscalar CPU는 instruction을 out-of-order로 실행하므로 결과를 실제 사용 시점보다 미리 읽을 수도 있으며 이를 non-serialized execution이라고 합니다.

정확한 TSC 측정에는 `CPUID`나 MSR read 같은 serializing instruction으로 execution order를 강제해야 합니다.

Hardware virtualization에서 CPUID 자체가 trap-and-emulate될 수 있어 serialization은 성능 문제를 만듭니다. 완벽히 synchronized된 system에서도 다른 CPU가 볼 때 TSC가 뒤로 읽히는 결과를 막도록 구현에서 방어해야 할 수 있습니다.

-----------------------------------

As the highest precision time source available, the cycle counter of the CPU
has aroused much interest from developers.  As explained above, this timer has
many problems unique to its nature as a local, potentially unstable and
potentially unsynchronized source.  One issue which is not unique to the TSC,
but is highlighted because of its very precise nature is sampling delay.  By
definition, the counter, once read is already old.  However, it is also
possible for the counter to be read ahead of the actual use of the result.
This is a consequence of the superscalar execution of the instruction stream,
which may execute instructions out of order.  Such execution is called
non-serialized.  Forcing serialized execution is necessary for precise
measurement with the TSC, and requires a serializing instruction, such as CPUID
or an MSR read.

Since CPUID may actually be virtualized by a trap and emulate mechanism, this
serialization can pose a performance issue for hardware virtualization.  An
accurate time stamp counter reading may therefore not always be available, and
it may be necessary for an implementation to guard against "backwards" reads of
the TSC as seen from other CPUs, even in an otherwise perfectly synchronized
system.

4.3. Timespec aliasing

4.3. Timespec aliasing

TSC와 다른 time source를 비교할 때 serialization 부족은 aliasing 문제를 만듭니다. TSC 정밀도가 훨씬 높아 external clock이 같은 값 C를 유지하는 동안 여러 TSC 값이 가능합니다.

예를 들어 external clock C가 변하지 않는 동안 `(T, T+10)`을 읽을 수 있고 non-serialized read 때문에 실제 범위가 `(T-1 .. T+10)`처럼 흔들릴 수 있습니다.

External value로 calibration한 TSC 기반 시간은 단일 값이 아니라 유효 범위를 가질 수 있습니다. Recalibration 뒤 계산한 시간이 이전 calibration의 결과보다 뒤로 갈 수도 있습니다.

Linux kernel time은 이론상 고해상도 `timespec`으로 표현되지만 실제로는 jiffy rate 같은 더 큰 granularity로 진행하고 catch-up mode에서는 더 큰 step으로 뛸 수 있어 문제가 특히 큽니다.

따라서 `kvmclock`과 TSC virtualization처럼 TSC 계산에서 파생되는 값의 계산·recalibration에는 aliasing을 고려해야 합니다.

----------------------

Additionally, this lack of serialization from the TSC poses another challenge
when using results of the TSC when measured against another time source.  As
the TSC is much higher precision, many possible values of the TSC may be read
while another clock is still expressing the same value.

That is, you may read (T,T+10) while external clock C maintains the same value.
Due to non-serialized reads, you may actually end up with a range which
fluctuates - from (T-1.. T+10).  Thus, any time calculated from a TSC, but
calibrated against an external value may have a range of valid values.
Re-calibrating this computation may actually cause time, as computed after the
calibration, to go backwards, compared with time computed before the
calibration.

This problem is particularly pronounced with an internal time source in Linux,
the kernel time, which is expressed in the theoretically high resolution
timespec - but which advances in much larger granularity intervals, sometimes
at the rate of jiffies, and possibly in catchup modes, at a much larger step.

This aliasing requires care in the computation and recalibration of kvmclock
and any other values derived from TSC computation (such as TSC virtualization
itself).

4.4. Migration

4.4. Migration

VM migration은 두 방식으로 timekeeping 문제를 만듭니다. 첫째, migration 동안 interrupt를 전달할 수 없는 시간이 생겨 이후 guest time을 따라잡아야 합니다. 필요한 보정이 보통 NTP correction window 안의 작은 값이라 NTP가 어느 정도 도울 수 있습니다.

둘째, TSC 또는 raw bus clock을 노출한 HPET 기반 timer는 destination에서 다른 rate로 동작할 수 있어 hypervisor가 timer를 virtualize하고 보정해야 합니다.

더 빠른 machine으로 이동하면 passthrough TSC를 유지하기 어렵습니다. 빠른 clock을 그대로 보이면 guest time이 평소보다 빨리 진행할 수 있습니다. 느린 clock은 original rate까지 catch-up할 수 있어 상대적으로 덜 어렵습니다.

KVM clock은 guest가 TSC를 nanosecond 값으로 변환하도록 multiplier와 offset을 저장해 이 문제를 피합니다.

--------------

Migration of a virtual machine raises problems for timekeeping in two ways.
First, the migration itself may take time, during which interrupts cannot be
delivered, and after which, the guest time may need to be caught up.  NTP may
be able to help to some degree here, as the clock correction required is
typically small enough to fall in the NTP-correctable window.

An additional concern is that timers based off the TSC (or HPET, if the raw bus
clock is exposed) may now be running at different rates, requiring compensation
in some way in the hypervisor by virtualizing these timers.  In addition,
migrating to a faster machine may preclude the use of a passthrough TSC, as a
faster clock cannot be made visible to a guest without the potential of time
advancing faster than usual.  A slower clock is less of a problem, as it can
always be caught up to the original rate.  KVM clock avoids these problems by
simply storing multipliers and offsets against the TSC for the guest to convert
back into nanosecond resolution values.

4.5. Scheduling

4.5. Scheduling

OS scheduling은 정밀한 시간과 interrupt 발생을 기준으로 할 수 있어 virtualization의 지연과 편향에 영향을 받습니다. 이론상 효과가 random하고 고르게 분포해야 하지만 실제로는 guest device access, virtualization exit 원인과 context switch가 특정 workload에 편향을 줄 수 있습니다.

이 영향은 충분히 연구되지 않았습니다. 일부 구현은 VM이 실제로 실행된 CPU time을 보여 주는 paravirtualized scheduler clock을 제공해 보정합니다.

---------------

Since scheduling may be based on precise timing and firing of interrupts, the
scheduling algorithms of an operating system may be adversely affected by
virtualization.  In theory, the effect is random and should be universally
distributed, but in contrived as well as real scenarios (guest device access,
causes of virtualization exits, possible context switch), this may not always
be the case.  The effect of this has not been well studied.

In an attempt to work around this, several implementations have provided a
paravirtualized scheduler clock, which reveals the true amount of CPU time for
which a virtual machine has been running.

4.6. Watchdogs

4.6. Watchdog

Linux lock detector 같은 watchdog timer는 virtual timer interrupt 지연이나 real time 해석 오류 때문에 hardware virtualization 아래에서 잘못 발화할 수 있습니다.

대개 이런 warning은 spurious하여 무시할 수 있지만 환경에 따라 해당 detection을 disable해야 할 수도 있습니다. 실제 lockup 가능성과 virtualization 지연을 log와 scheduling 상태로 구분해야 합니다.

--------------

Watchdog timers, such as the lock detector in Linux may fire accidentally when
running under hardware virtualization due to timer interrupts being delayed or
misinterpretation of the passage of real time.  Usually, these warnings are
spurious and can be ignored, but in some circumstances it may be necessary to
disable such detection.

4.7. Delays and precision timing

4.7. Delay와 정밀 timing

Virtualized system에서는 정밀 timing과 정확한 delay가 불가능할 수 있습니다. Physical hardware를 제어하거나 느린 device I/O를 보상하려고 delay를 넣는 software가 영향을 받습니다.

Physical hardware control을 일반 virtualization에서 완전히 해결하려면 full real-time OS와 RT-aware virtualization platform이 필요하므로 보편적으로 해결할 수 없습니다.

느린 I/O 보상 delay는 성능 문제를 만들 수 있지만 큰 문제일 가능성은 낮고 많은 경우 configuration 또는 paravirtualization으로 제거할 수 있습니다.

--------------------------------

Precise timing and delays may not be possible in a virtualized system.  This
can happen if the system is controlling physical hardware, or issues delays to
compensate for slower I/O to and from devices.  The first issue is not solvable
in general for a virtualized system; hardware control software can't be
adequately virtualized without a full real-time operating system, which would
require an RT aware virtualization platform.

The second issue may cause performance problems, but this is unlikely to be a
significant issue.  In many cases these delays may be eliminated through
configuration or paravirtualization.

4.8. Covert channels and leaks

4.8. Covert channel과 정보 누출

완벽한 virtual time 구현이 아니라면 시간 정보는 필연적으로 host에 관한 정보를 guest에 누출합니다. Guest는 timing으로 hypervisor 존재를 추론하는 red-pill 계열 탐지를 할 수 있습니다.

CPU utilization 자체를 signalling channel로 사용하면 guest 사이 정보가 새어 나갈 수도 있습니다. 이를 막으려면 real time을 더 이상 그대로 추적하지 않는 완전히 isolated된 virtual time이 필요합니다.

완전 격리 시간은 일부 security 또는 QA context에서는 유용하지만 real-world deployment에는 일반적으로 권장되지 않습니다.

------------------------------

In addition to the above problems, time information will inevitably leak to the
guest about the host in anything but a perfect implementation of virtualized
time.  This may allow the guest to infer the presence of a hypervisor (as in a
red-pill type detection), and it may allow information to leak between guests
by using CPU utilization itself as a signalling channel.  Preventing such
problems would require completely isolated virtual time which may not track
real time any longer.  This may be useful in certain security or QA contexts,
but in general isn't recommended for real-world deployment scenarios.