Generic vCPU device interface

ARM64 PMUv3 control

Virtual CPU device도 다른 KVM device와 같은 `struct kvm_device_attr`을 사용해 `KVM_SET_DEVICE_ATTR`, `KVM_GET_DEVICE_ATTR`, `KVM_HAS_DEVICE_ATTR`을 받습니다. Group과 attribute는 architecture-specific이며 vCPU 전체의 설정과 control을 대상으로 합니다.

`KVM_ARM_VCPU_PMU_V3_IRQ`는 이 vCPU의 PMUv3 overflow interrupt number를 int pointer로 get/set합니다. PPI 또는 SPI일 수 있지만 모든 vCPU에서 interrupt type은 같아야 합니다. PPI면 number도 모두 같고 SPI면 vCPU마다 별도 number가 필요합니다.

`KVM_ARM_VCPU_PMU_V3_INIT`은 추가 parameter 없이 PMUv3를 initialize합니다. In-kernel virtual GIC를 쓸 때는 irqchip initialization 뒤에 호출해야 합니다. 이미 사용 중인 IRQ는 `-EEXIST`, GIC·PMU 미준비는 `-ENODEV`·`-ENXIO`, 중복 init은 `-EBUSY`입니다.

`KVM_ARM_VCPU_PMU_V3_FILTER`는 `kvm_pmu_event_filter`의 `base_event`, `nevents`, `action`으로 half-open event range `[base_event, base_event + nevents)`를 ALLOW 또는 DENY합니다.

여러 range를 등록할 수 있으며 ARMv8.0 event space는 10-bit, ARMv8.1 이후는 16-bit입니다. 같은 range에 반대 action을 등록해도 최초 global default는 바뀌지 않습니다. 예를 들어 첫 ALLOW [0:10) 뒤 같은 range를 DENY하면 전체 range가 disabled로 남습니다.

`SW_INCR` event 0은 hardware event를 count하지 않으므로 filter되지 않습니다. `CHAIN` 0x1E filter도 효과가 없고 cycle counter는 `CPU_CYCLES` 0x11로 filter할 수 있습니다.

Filter 설치는 PMUv3·GIC 준비가 필요하고 init 또는 vCPU 실행 뒤에는 `-EBUSY`, 잘못된 range는 `-EINVAL`입니다.

`KVM_ARM_VCPU_PMU_V3_SET_PMU`는 `/sys/devices` 또는 `/sys/bus/event_source`의 PMU `type` 값을 지정해 guest event emulation에 사용할 physical PMU를 선택합니다. Heterogeneous system에서 특히 유용하며 한 vCPU의 선택이 모든 vCPU에 적용됩니다.

KVM은 선택 PMU와 연결된 physical CPU로 vCPU를 자동 pin하지 않습니다. Userspace가 scheduling affinity를 관리해야 하며 지원하지 않는 CPU에서 실행하면 `KVM_RUN`이 `KVM_EXIT_FAIL_ENTRY`와 `KVM_EXIT_FAIL_ENTRY_CPU_UNSUPPORTED` reason 및 processor ID를 반환합니다.

PMU event filter가 이미 있으면 PMU를 선택할 수 없습니다. Init·vCPU run·filter 뒤 SET_PMU는 `-EBUSY`; PMU ID access 실패는 `-EFAULT`; PMU 미발견은 `-ENXIO`; allocation 실패는 `-ENOMEM`입니다.

`KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS`는 virtual PMU의 `PMCR_EL0.N` 최대값으로 구현 event-counter 수를 제한합니다. SET_PMU로 physical PMU를 명시적으로 선택한 뒤에만 가능하고 선택 PMU 범위를 벗어나면 `-EINVAL`입니다. 새 PMU를 선택하면 이 설정 효과는 취소됩니다.

.. SPDX-License-Identifier: GPL-2.0

======================
Generic vcpu interface
======================

The virtual cpu "device" also accepts the ioctls KVM_SET_DEVICE_ATTR,
KVM_GET_DEVICE_ATTR, and KVM_HAS_DEVICE_ATTR. The interface uses the same struct
kvm_device_attr as other devices, but targets VCPU-wide settings and controls.

The groups and attributes per virtual cpu, if any, are architecture specific.

1. GROUP: KVM_ARM_VCPU_PMU_V3_CTRL
==================================

:Architectures: ARM64

1.1. ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_IRQ
---------------------------------------

:Parameters: in kvm_device_attr.addr the address for PMU overflow interrupt is a
	     pointer to an int

Returns:

	 =======  ========================================================
	 -EBUSY   The PMU overflow interrupt is already set
	 -EFAULT  Error reading interrupt number
	 -ENXIO   PMUv3 not supported or the overflow interrupt not set
		  when attempting to get it
	 -ENODEV  KVM_ARM_VCPU_PMU_V3 feature missing from VCPU
	 -EINVAL  Invalid PMU overflow interrupt number supplied or
		  trying to set the IRQ number without using an in-kernel
		  irqchip.
	 =======  ========================================================

A value describing the PMUv3 (Performance Monitor Unit v3) overflow interrupt
number for this vcpu. This interrupt could be a PPI or SPI, but the interrupt
type must be same for each vcpu. As a PPI, the interrupt number is the same for
all vcpus, while as an SPI it must be a separate number per vcpu.

1.2 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_INIT
---------------------------------------

:Parameters: no additional parameter in kvm_device_attr.addr

Returns:

	 =======  ======================================================
	 -EEXIST  Interrupt number already used
	 -ENODEV  PMUv3 not supported or GIC not initialized
	 -ENXIO   PMUv3 not supported, missing VCPU feature or interrupt
		  number not set
	 -EBUSY   PMUv3 already initialized
	 =======  ======================================================

Request the initialization of the PMUv3.  If using the PMUv3 with an in-kernel
virtual GIC implementation, this must be done after initializing the in-kernel
irqchip.

1.3 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_FILTER
-----------------------------------------

:Parameters: in kvm_device_attr.addr the address for a PMU event filter is a
             pointer to a struct kvm_pmu_event_filter

:Returns:

	 =======  ======================================================
	 -ENODEV  PMUv3 not supported or GIC not initialized
	 -ENXIO   PMUv3 not properly configured or in-kernel irqchip not
	 	  configured as required prior to calling this attribute
	 -EBUSY   PMUv3 already initialized or a VCPU has already run
	 -EINVAL  Invalid filter range
	 =======  ======================================================

Request the installation of a PMU event filter described as follows::

    struct kvm_pmu_event_filter {
	    __u16	base_event;
	    __u16	nevents;

    #define KVM_PMU_EVENT_ALLOW	0
    #define KVM_PMU_EVENT_DENY	1

	    __u8	action;
	    __u8	pad[3];
    };

A filter range is defined as the range [@base_event, @base_event + @nevents),
together with an @action (KVM_PMU_EVENT_ALLOW or KVM_PMU_EVENT_DENY). The
first registered range defines the global policy (global ALLOW if the first
@action is DENY, global DENY if the first @action is ALLOW). Multiple ranges
can be programmed, and must fit within the event space defined by the PMU
architecture (10 bits on ARMv8.0, 16 bits from ARMv8.1 onwards).

Note: "Cancelling" a filter by registering the opposite action for the same
range doesn't change the default action. For example, installing an ALLOW
filter for event range [0:10) as the first filter and then applying a DENY
action for the same range will leave the whole range as disabled.

Restrictions: Event 0 (SW_INCR) is never filtered, as it doesn't count a
hardware event. Filtering event 0x1E (CHAIN) has no effect either, as it
isn't strictly speaking an event. Filtering the cycle counter is possible
using event 0x11 (CPU_CYCLES).

1.4 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_SET_PMU
------------------------------------------

:Parameters: in kvm_device_attr.addr the address to an int representing the PMU
             identifier.

:Returns:

	 =======  ====================================================
	 -EBUSY   PMUv3 already initialized, a VCPU has already run or
                  an event filter has already been set
	 -EFAULT  Error accessing the PMU identifier
	 -ENXIO   PMU not found
	 -ENODEV  PMUv3 not supported or GIC not initialized
	 -ENOMEM  Could not allocate memory
	 =======  ====================================================

Request that the VCPU uses the specified hardware PMU when creating guest events
for the purpose of PMU emulation. The PMU identifier can be read from the "type"
file for the desired PMU instance under /sys/devices (or, equivalent,
/sys/bus/even_source). This attribute is particularly useful on heterogeneous
systems where there are at least two CPU PMUs on the system. The PMU that is set
for one VCPU will be used by all the other VCPUs. It isn't possible to set a PMU
if a PMU event filter is already present.

Note that KVM will not make any attempts to run the VCPU on the physical CPUs
associated with the PMU specified by this attribute. This is entirely left to
userspace. However, attempting to run the VCPU on a physical CPU not supported
by the PMU will fail and KVM_RUN will return with
exit_reason = KVM_EXIT_FAIL_ENTRY and populate the fail_entry struct by setting
hardare_entry_failure_reason field to KVM_EXIT_FAIL_ENTRY_CPU_UNSUPPORTED and
the cpu field to the processor id.

1.5 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS
--------------------------------------------------

:Parameters: in kvm_device_attr.addr the address to an unsigned int
	     representing the maximum value taken by PMCR_EL0.N

:Returns:

	 =======  ====================================================
	 -EBUSY   PMUv3 already initialized, a VCPU has already run or
                  an event filter has already been set
	 -EFAULT  Error accessing the value pointed to by addr
	 -ENODEV  PMUv3 not supported or GIC not initialized
	 -EINVAL  No PMUv3 explicitly selected, or value of N out of
	 	  range
	 =======  ====================================================

Set the number of implemented event counters in the virtual PMU. This
mandates that a PMU has explicitly been selected via
KVM_ARM_VCPU_PMU_V3_SET_PMU, and will fail when no PMU has been
explicitly selected, or the number of counters is out of range for the
selected PMU. Selecting a new PMU cancels the effect of setting this
attribute.

ARM64 architected timer IRQ

`KVM_ARM_VCPU_TIMER_CTRL`은 in-kernel virtual GIC에 연결되는 architected timer interrupt를 설정합니다. 모든 timer IRQ는 PPI 범위 `16 <= intid < 32`여야 하며 잘못된 number는 `-EINVAL`, 어느 vCPU든 이미 실행됐으면 `-EBUSY`입니다.

서로 다른 timer에 같은 PPI를 설정하면 vCPU가 실행되지 않습니다. 한 vCPU에서 설정하면 그 시점에 생성된 모든 vCPU의 해당 timer 값이 바뀌고 다른 vCPU의 이전 설정도 덮어씁니다.

Userspace는 모든 vCPU를 만든 뒤 어느 vCPU도 실행하기 전에 최소 한 vCPU에서 timer interrupt number를 구성해야 합니다.

2. GROUP: KVM_ARM_VCPU_TIMER_CTRL
=================================

:Architectures: ARM64

2.1. ATTRIBUTES: KVM_ARM_VCPU_TIMER_IRQ_{VTIMER,PTIMER,HVTIMER,HPTIMER}
-----------------------------------------------------------------------

:Parameters: in kvm_device_attr.addr the address for the timer interrupt is a
	     pointer to an int

Returns:

	 =======  =================================
	 -EINVAL  Invalid timer interrupt number
	 -EBUSY   One or more VCPUs has already run
	 =======  =================================

A value describing the architected timer interrupt number when connected to an
in-kernel virtual GIC.  These must be a PPI (16 <= intid < 32).  Setting the
attribute overrides the default values (see below).

==============================  ==========================================
KVM_ARM_VCPU_TIMER_IRQ_VTIMER   The EL1 virtual timer intid (default: 27)
KVM_ARM_VCPU_TIMER_IRQ_PTIMER   The EL1 physical timer intid (default: 30)
KVM_ARM_VCPU_TIMER_IRQ_HVTIMER  The EL2 virtual timer intid (default: 28)
KVM_ARM_VCPU_TIMER_IRQ_HPTIMER  The EL2 physical timer intid (default: 26)
==============================  ==========================================

Setting the same PPI for different timers will prevent the VCPUs from running.
Setting the interrupt number on a VCPU configures all VCPUs created at that
time to use the number provided for a given timer, overwriting any previously
configured values on other VCPUs.  Userspace should configure the interrupt
numbers on at least one VCPU after creating all VCPUs and before running any
VCPUs.

.. _kvm_arm_vcpu_pvtime_ctrl:

ARM64 stolen-time IPA

`KVM_ARM_VCPU_PVTIME_IPA`는 해당 vCPU의 stolen-time structure base address를 64-bit 값으로 설정합니다. Address는 64-byte aligned이고 유효한 guest memory region 안에 있어야 합니다.

Structure layout과 guest mapping 규칙은 `Documentation/virt/kvm/arm/pvtime.rst`를 따릅니다.

3. GROUP: KVM_ARM_VCPU_PVTIME_CTRL
==================================

:Architectures: ARM64

3.1 ATTRIBUTE: KVM_ARM_VCPU_PVTIME_IPA
--------------------------------------

:Parameters: 64-bit base address

Returns:

	 =======  ======================================
	 -ENXIO   Stolen time not implemented
	 -EEXIST  Base address already set for this VCPU
	 -EINVAL  Base address not 64 byte aligned
	 =======  ======================================

Specifies the base address of the stolen time structure for this VCPU. The
base address must be 64 byte aligned and exist within a valid guest memory
region. See Documentation/virt/kvm/arm/pvtime.rst for more information
including the layout of the stolen time structure.

x86 TSC offset과 live migration

`KVM_VCPU_TSC_OFFSET`은 host TSC에 대한 guest의 unsigned 64-bit offset입니다. `guest_tsc = host_tsc + KVM_VCPU_TSC_OFFSET` 관계를 사용하며 parameter access 실패는 `-EFAULT`, 미지원은 `-ENXIO`입니다.

이 attribute를 사용하면 live migration 동안 VM이 pause된 시간까지 guest TSC가 이어지도록 destination의 vCPU별 offset을 조정할 수 있습니다.

`KVM_SET_CLOCK`은 기록 뒤 지난 시간만큼 kvmclock을 진행시킵니다. Source와 destination의 CLOCK_REALTIME이 동기화되지 않았거나 source pause와 destination step 4~7 사이가 길면 guest timeout 같은 문제가 생길 수 있습니다.

4. GROUP: KVM_VCPU_TSC_CTRL
===========================

:Architectures: x86

4.1 ATTRIBUTE: KVM_VCPU_TSC_OFFSET

:Parameters: 64-bit unsigned TSC offset

Returns:

	 ======= ======================================
	 -EFAULT Error reading/writing the provided
		 parameter address.
	 -ENXIO  Attribute not supported
	 ======= ======================================

Specifies the guest's TSC offset relative to the host's TSC. The guest's
TSC is then derived by the following equation:

  guest_tsc = host_tsc + KVM_VCPU_TSC_OFFSET

This attribute is useful to adjust the guest's TSC on live migration,
so that the TSC counts the time during which the VM was paused. The
following describes a possible algorithm to use for this purpose.

From the source VMM process:

1. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (tsc_src),
   kvmclock nanoseconds (guest_src), and host CLOCK_REALTIME nanoseconds
   (host_src).

2. Read the KVM_VCPU_TSC_OFFSET attribute for every vCPU to record the
   guest TSC offset (ofs_src[i]).

3. Invoke the KVM_GET_TSC_KHZ ioctl to record the frequency of the
   guest's TSC (freq).

From the destination VMM process:

4. Invoke the KVM_SET_CLOCK ioctl, providing the source nanoseconds from
   kvmclock (guest_src) and CLOCK_REALTIME (host_src) in their respective
   fields.  Ensure that the KVM_CLOCK_REALTIME flag is set in the provided
   structure.

   KVM will advance the VM's kvmclock to account for elapsed time since
   recording the clock values.  Note that this will cause problems in
   the guest (e.g., timeouts) unless CLOCK_REALTIME is synchronized
   between the source and destination, and a reasonably short time passes
   between the source pausing the VMs and the destination executing
   steps 4-7.

5. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (tsc_dest) and
   kvmclock nanoseconds (guest_dest).

6. Adjust the guest TSC offsets for every vCPU to account for (1) time
   elapsed since recording state and (2) difference in TSCs between the
   source and destination machine:

   ofs_dst[i] = ofs_src[i] -
     (guest_src - guest_dest) * freq +
     (tsc_src - tsc_dest)

   ("ofs[i] + tsc - guest * freq" is the guest TSC value corresponding to
   a time of 0 in kvmclock.  The above formula ensures that it is the
   same on the destination as it was on the source).

7. Write the KVM_VCPU_TSC_OFFSET attribute for every vCPU with the
   respective value derived in the previous step.