Hyper-V 게스트 VM hibernation

배경

Linux는 전력을 절약하기 위해 자체 hibernation을 지원합니다. Suspend-to-disk라고도 하며 memory image를 disk에 기록하고 hardware를 가능한 가장 낮은 power state로 전환합니다.

Resume할 때 hardware를 다시 시작하고 disk의 memory image를 복원해 중단했던 위치에서 실행을 계속합니다. 일반 동작은 `Documentation/admin-guide/pm/sleep-states.rst`의 Hibernation 절을 참조합니다.

보통 개인 laptop처럼 단일 사용자가 쓰는 장치에서 같은 hardware로 hibernate·resume하며, Linux kernel의 orchestration code는 hibernated 상태 동안 hardware 구성이 바뀌지 않았다고 가정합니다.

Linux 안에서는 `/sys/power/state`에 `disk`를 쓰거나 적절한 argument로 reboot system call을 호출해 hibernation을 시작합니다. 사용자 공간의 `systemctl hibernate` 같은 명령이 이를 감싸며 command line이나 lid-close event에 반응해 실행할 수 있습니다.

.. SPDX-License-Identifier: GPL-2.0

Hibernating Guest VMs
=====================

Background
----------
Linux supports the ability to hibernate itself in order to save power.
Hibernation is sometimes called suspend-to-disk, as it writes a memory
image to disk and puts the hardware into the lowest possible power
state. Upon resume from hibernation, the hardware is restarted and the
memory image is restored from disk so that it can resume execution
where it left off. See the "Hibernation" section of
Documentation/admin-guide/pm/sleep-states.rst.

Hibernation is usually done on devices with a single user, such as a
personal laptop. For example, the laptop goes into hibernation when
the cover is closed, and resumes when the cover is opened again.
Hibernation and resume happen on the same hardware, and Linux kernel
code orchestrating the hibernation steps assumes that the hardware
configuration is not changed while in the hibernated state.

Hibernation can be initiated within Linux by writing "disk" to
/sys/power/state or by invoking the reboot system call with the
appropriate arguments. This functionality may be wrapped by user space
commands such "systemctl hibernate" that are run directly from a
command line or in response to events such as the laptop lid closing.

게스트 VM hibernation 고려 사항

Hyper-V의 Linux guest도 hibernate할 수 있으며 이때 hardware는 Hyper-V가 VM에 제공하는 virtual hardware입니다. 대상 VM만 정지하고 다른 guest VM과 Hyper-V host는 계속 정상 실행됩니다. Windows host 자체 hibernation의 guest 영향은 이 문서 범위 밖입니다.

Virtual hardware는 hibernate와 resume 사이에 구성을 쉽게 바꿀 수 있어 physical system보다 복원이 어렵습니다. 같은 VM이어도 memory size를 변경하거나 virtual NIC, SCSI controller, assigned virtual PCI device를 추가·제거할 수 있습니다.

대부분의 변경은 resume 실패를 일으키지만 새 virtual NIC, SCSI controller 또는 vPCI device를 추가하는 경우는 동작해야 합니다.

Hibernated VM의 disk를 동일한 virtual hardware 구성을 가진 새 VM으로 옮길 수도 있습니다. 이런 이동 뒤에도 resume하는 것이 바람직하지만 별도 제약이 있으며 뒤의 다른 VM에서 복원 절이 설명합니다.

Considerations for Guest VM Hibernation
---------------------------------------
Linux guests on Hyper-V can also be hibernated, in which case the
hardware is the virtual hardware provided by Hyper-V to the guest VM.
Only the targeted guest VM is hibernated, while other guest VMs and
the underlying Hyper-V host continue to run normally. While the
underlying Windows Hyper-V and physical hardware on which it is
running might also be hibernated using hibernation functionality in
the Windows host, host hibernation and its impact on guest VMs is not
in scope for this documentation.

Resuming a hibernated guest VM can be more challenging than with
physical hardware because VMs make it very easy to change the hardware
configuration between the hibernation and resume. Even when the resume
is done on the same VM that hibernated, the memory size might be
changed, or virtual NICs or SCSI controllers might be added or
removed. Virtual PCI devices assigned to the VM might be added or
removed. Most such changes cause the resume steps to fail, though
adding a new virtual NIC, SCSI controller, or vPCI device should work.

Additional complexity can ensue because the disks of the hibernated VM
can be moved to another newly created VM that otherwise has the same
virtual hardware configuration. While it is desirable for resume from
hibernation to succeed after such a move, there are challenges. See
details on this scenario and its limitations in the "Resuming on a
Different VM" section below.

다른 Hyper-V host로 이동

Hyper-V는 VM을 다른 host로 이동할 수 있으며 VM Configuration Version으로 processor model과 Hyper-V version 호환성을 확인해 비호환 host로의 이동을 막으려 합니다.

Linux는 boot 때 host와 processor 차이를 감지해 적응하지만 hibernation image에서 실행을 재개할 때는 이 감지를 다시 하지 않습니다.

따라서 한 host에서 hibernate한 VM을 processor model이나 Hyper-V version이 다른 host에서 resume하면 image에 저장된 설정이 새 host와 맞지 않을 수 있습니다. Linux가 mismatch를 재검출하지 않으므로 undefined behavior나 실패가 생길 수 있습니다.

Hyper-V also provides ways to move a VM from one Hyper-V host to
another. Hyper-V tries to ensure processor model and Hyper-V version
compatibility using VM Configuration Versions, and prevents moves to
a host that isn't compatible. Linux adapts to host and processor
differences by detecting them at boot time, but such detection is not
done when resuming execution in the hibernation image. If a VM is
hibernated on one host, then resumed on a host with a different processor
model or Hyper-V version, settings recorded in the hibernation image
may not match the new host. Because Linux does not detect such
mismatches when resuming the hibernation image, undefined behavior
and failures could result.

게스트 VM hibernation 활성화

Hyper-V guest VM hibernation은 기본적으로 꺼져 있습니다. Hyper-V balloon driver가 제공하는 memory hot-add와 hibernation이 호환되지 않기 때문입니다.

Hot-add 뒤 hibernate하면 VM은 시작 때보다 많은 memory를 image에 담지만 resume 때 Hyper-V는 원래 할당량만 제공합니다. 이 memory size mismatch 때문에 복원이 실패합니다.

관리자는 Hyper-V가 guest에 제공하는 ACPI 구성에서 virtual S4 sleep state를 활성화해야 합니다. VM의 WMI property를 바꾸는 구체적인 단계는 문서 범위 밖입니다.

S4 활성화는 관리자가 hot-add보다 Linux hibernation을 우선한다는 표시로 취급되어 Linux Hyper-V balloon driver가 hot-add를 비활성화합니다. `/sys/power/disk`에 `platform` option이 있거나 `/sys/bus/vmbus/hibernation`에서 활성 상태를 확인할 수 있으며 관련 함수는 `hv_is_hibernation_supported()`입니다.

Linux는 x86에서 ACPI sleep state를 지원하지만 arm64에서는 지원하지 않으므로 Hyper-V arm64 guest에서는 hibernation을 사용할 수 없습니다.

Enabling Guest VM Hibernation
-----------------------------
Hibernation of a Hyper-V guest VM is disabled by default because
hibernation is incompatible with memory hot-add, as provided by the
Hyper-V balloon driver. If hot-add is used and the VM hibernates, it
hibernates with more memory than it started with. But when the VM
resumes from hibernation, Hyper-V gives the VM only the originally
assigned memory, and the memory size mismatch causes resume to fail.

To enable a Hyper-V VM for hibernation, the Hyper-V administrator must
enable the ACPI virtual S4 sleep state in the ACPI configuration that
Hyper-V provides to the guest VM. Such enablement is accomplished by
modifying a WMI property of the VM, the steps for which are outside
the scope of this documentation but are available on the web.
Enablement is treated as the indicator that the administrator
prioritizes Linux hibernation in the VM over hot-add, so the Hyper-V
balloon driver in Linux disables hot-add. Enablement is indicated if
the contents of /sys/power/disk contains "platform" as an option. The
enablement is also visible in /sys/bus/vmbus/hibernation. See function
hv_is_hibernation_supported().

Linux supports ACPI sleep states on x86, but not on arm64. So Linux
guest VM hibernation is not available on Hyper-V for arm64.

게스트 VM hibernation 시작

Guest는 일반 Linux 방식으로 `/sys/power/state`에 `disk`를 쓰거나 reboot system call을 호출해 스스로 hibernation을 시작할 수 있습니다.

Hyper-V Linux guest는 추가로 Shutdown integration service를 지원합니다. 관리자가 VM 밖에서 hibernate 명령을 보내면 Linux의 Hyper-V shutdown driver가 요청을 받고 `EVENT=hibernate` uevent를 전송합니다.

관련 함수는 `shutdown_onchannelcallback()`과 `send_hibernate_uevent()`입니다. VM에는 이 event를 처리해 hibernation을 시작하는 udev rule이 있어야 합니다.

Initiating Guest VM Hibernation
-------------------------------
Guest VMs can self-initiate hibernation using the standard Linux
methods of writing "disk" to /sys/power/state or the reboot system
call. As an additional layer, Linux guests on Hyper-V support the
"Shutdown" integration service, via which a Hyper-V administrator can
tell a Linux VM to hibernate using a command outside the VM. The
command generates a request to the Hyper-V shutdown driver in Linux,
which sends the uevent "EVENT=hibernate". See kernel functions
shutdown_onchannelcallback() and send_hibernate_uevent(). A udev rule
must be provided in the VM that handles this event and initiates
hibernation.

VMBus device 보존 모델

VMBus bus driver와 개별 VMBus device driver는 Linux의 hibernate·resume orchestration에서 호출되는 suspend·resume 함수를 구현합니다.

핵심 방식은 SCSI controller 같은 Linux device와 primary VMBus channel의 data structure를 그대로 남겨 hibernation image에 포함하는 것입니다. 이렇게 device state가 image를 거쳐 보존됩니다.

VM이 resume하면 Hyper-V가 device를 다시 offer하고, 이를 복원된 hibernation image에 이미 있는 data structure에 연결합니다.

Handling VMBus Devices During Hibernation & Resume
--------------------------------------------------
The VMBus bus driver, and the individual VMBus device drivers,
implement suspend and resume functions that are called as part of the
Linux orchestration of hibernation and of resuming from hibernation.
The overall approach is to leave in place the data structures for the
primary VMBus channels and their associated Linux devices, such as
SCSI controllers and others, so that they are captured in the
hibernation image. This approach allows any state associated with the
device to be persisted across the hibernation/resume. When the VM
resumes, the devices are re-offered by Hyper-V and are connected to
the data structures that already exist in the resumed hibernation
image.

GUID matching과 relid 갱신

VMBus device는 class GUID와 instance GUID로 식별됩니다. Device creation·deletion 설명은 `Documentation/virt/hyperv/vmbus.rst`를 참조합니다.

Resume 함수는 Hyper-V가 offer한 device의 class·instance GUID가 hibernate 시점의 device와 같다고 기대합니다. GUID가 같으면 복원 image의 primary channel data structure와 새 offer를 matching할 수 있습니다.

기존 primary channel과 맞지 않는 offer는 새로 추가된 device로 정상 처리합니다. 반대로 image의 primary channel과 일치하는 offer가 없으면 resume sequence가 10초 기다린 뒤 진행하지만 unmatched device는 복원 VM에서 오류를 일으킬 가능성이 큽니다.

VM configuration이 같아도 boot마다 새 offer의 `relid`는 바뀔 수 있습니다. VMBus bus driver resume 함수는 GUID로 channel을 matching한 뒤 변경된 relid를 갱신합니다.

VMBus devices are identified by class and instance GUID. (See section
"VMBus device creation/deletion" in
Documentation/virt/hyperv/vmbus.rst.) Upon resume from hibernation,
the resume functions expect that the devices offered by Hyper-V have
the same class/instance GUIDs as the devices present at the time of
hibernation. Having the same class/instance GUIDs allows the offered
devices to be matched to the primary VMBus channel data structures in
the memory of the now resumed hibernation image. If any devices are
offered that don't match primary VMBus channel data structures that
already exist, they are processed normally as newly added devices. If
primary VMBus channels that exist in the resumed hibernation image are
not matched with a device offered in the resumed VM, the resume
sequence waits for 10 seconds, then proceeds. But the unmatched device
is likely to cause errors in the resumed VM.

When resuming existing primary VMBus channels, the newly offered
relids might be different because relids can change on each VM boot,
even if the VM configuration hasn't changed. The VMBus bus driver
resume function matches the class/instance GUIDs, and updates the
relids in case they have changed.

Primary·sub-channel과 socket 처리

VMBus sub-channel은 hibernation image에 보존하지 않습니다. 각 device driver의 suspend 함수가 hibernate 전에 모든 sub-channel을 닫아야 합니다.

Sub-channel을 닫으면 Hyper-V가 `RESCIND_CHANNELOFFER` message를 보내고 Linux가 channel data structure를 해제해 흔적을 완전히 제거합니다.

Primary channel은 closed로 표시하고 ring buffer를 해제하지만 rescind message를 받지 않아 channel data structure는 남습니다. Resume 때 ring buffer를 다시 할당하고 기존 primary channel을 열며 Hyper-V와 통신해 sub-channel을 처음부터 다시 만듭니다.

Hyper-V socket의 Linux 쪽 endpoint는 hibernate 시 강제로 닫힙니다. Guest가 host endpoint를 강제로 닫을 수는 없지만 host가 이후 endpoint를 사용하면 오류가 납니다.

VMBus device는 `freeze`와 `poweroff` phase에서 같은 suspend 함수를, `thaw`와 `restore` phase에서 같은 resume 함수를 사용합니다. Phase 순서는 `Documentation/driver-api/pm/devices.rst`를 참조합니다.

VMBus sub-channels are not persisted in the hibernation image. Each
VMBus device driver's suspend function must close any sub-channels
prior to hibernation. Closing a sub-channel causes Hyper-V to send a
RESCIND_CHANNELOFFER message, which Linux processes by freeing the
channel data structures so that all vestiges of the sub-channel are
removed. By contrast, primary channels are marked closed and their
ring buffers are freed, but Hyper-V does not send a rescind message,
so the channel data structure continues to exist. Upon resume, the
device driver's resume function re-allocates the ring buffer and
re-opens the existing channel. It then communicates with Hyper-V to
re-open sub-channels from scratch.

The Linux ends of Hyper-V sockets are forced closed at the time of
hibernation. The guest can't force closing the host end of the socket,
but any host-side actions on the host end will produce an error.

VMBus devices use the same suspend function for the "freeze" and the
"poweroff" phases, and the same resume function for the "thaw" and
"restore" phases. See the "Entering Hibernation" section of
Documentation/driver-api/pm/devices.rst for the sequencing of the
phases.

상세 hibernation 순서

Linux PM subsystem은 사용자 공간 process를 freeze하고 hibernation image를 담을 memory를 할당합니다. `freeze` phase에서 각 VMBus device suspend 함수가 sub-channel을 제거하고 primary channel을 닫습니다.

이어서 VMBus bus suspend 함수가 Hyper-V socket channel을 닫고 host와의 top-level VMBus connection을 unload합니다. Linux PM은 non-boot CPU를 disable한 뒤 할당 memory에 image를 만들고 CPU를 다시 enable합니다. Image에는 closed primary channel 구조만 있고 sub-channel은 없습니다.

`thaw` phase에서 VMBus bus resume이 top-level connection을 재구성하고 device re-offer를 요청합니다. Primary offer의 relid를 갱신한 뒤 각 device resume 함수가 primary channel을 열고 필요한 sub-channel을 새로 만듭니다.

VMBus가 다시 동작하면 Linux PM이 memory의 hibernation image를 disk에 씁니다. `poweroff` phase에서 device·bus suspend를 반복해 channel과 top-level connection을 닫고, non-boot CPU를 disable한 뒤 ACPI S4로 진입합니다.

Detailed Hibernation Sequence
-----------------------------
1. The Linux power management (PM) subsystem prepares for
   hibernation by freezing user space processes and allocating
   memory to hold the hibernation image.
2. As part of the "freeze" phase, Linux PM calls the "suspend"
   function for each VMBus device in turn. As described above, this
   function removes sub-channels, and leaves the primary channel in
   a closed state.
3. Linux PM calls the "suspend" function for the VMBus bus, which
   closes any Hyper-V socket channels and unloads the top-level
   VMBus connection with the Hyper-V host.
4. Linux PM disables non-boot CPUs, creates the hibernation image in
   the previously allocated memory, then re-enables non-boot CPUs.
   The hibernation image contains the memory data structures for the
   closed primary channels, but no sub-channels.
5. As part of the "thaw" phase, Linux PM calls the "resume" function
   for the VMBus bus, which re-establishes the top-level VMBus
   connection and requests that Hyper-V re-offer the VMBus devices.
   As offers are received for the primary channels, the relids are
   updated as previously described.
6. Linux PM calls the "resume" function for each VMBus device. Each
   device re-opens its primary channel, and communicates with Hyper-V
   to re-establish sub-channels if appropriate. The sub-channels
   are re-created as new channels since they were previously removed
   entirely in Step 2.
7. With VMBus devices now working again, Linux PM writes the
   hibernation image from memory to disk.
8. Linux PM repeats Steps 2 and 3 above as part of the "poweroff"
   phase. VMBus channels are closed and the top-level VMBus
   connection is unloaded.
9. Linux PM disables non-boot CPUs, and then enters ACPI sleep state
   S4. Hibernation is now complete.

상세 resume 순서

Guest VM은 새 Linux OS instance로 boot합니다. 정상 boot 경로로 top-level VMBus connection을 만들고 synthetic device를 활성화합니다.

Linux PM hibernation code가 swap space에서 image를 찾아 memory로 읽습니다. Image가 없으면 그대로 normal boot가 됩니다.

Image가 있으면 fresh OS instance에서 `freeze` phase로 VMBus device를 shutdown하고 top-level connection을 unload합니다. Non-boot CPU를 disable하고 읽어 둔 hibernation image로 control을 넘긴 뒤 image 안에서 CPU를 다시 시작합니다.

`resume` phase는 hibernation sequence의 5·6단계를 반복합니다. Top-level VMBus를 다시 만들고 offer를 image의 primary channel과 matching해 relid를 갱신하며 device resume 함수가 primary를 열고 sub-channel을 재생성합니다.

마지막으로 Linux PM이 resume sequence를 끝내면 VM은 hibernation image의 상태에서 정상 실행합니다.

Detailed Resume Sequence
------------------------
1. The guest VM boots into a fresh Linux OS instance. During boot,
   the top-level VMBus connection is established, and synthetic
   devices are enabled. This happens via the normal paths that don't
   involve hibernation.
2. Linux PM hibernation code reads swap space is to find and read
   the hibernation image into memory. If there is no hibernation
   image, then this boot becomes a normal boot.
3. If this is a resume from hibernation, the "freeze" phase is used
   to shutdown VMBus devices and unload the top-level VMBus
   connection in the running fresh OS instance, just like Steps 2
   and 3 in the hibernation sequence.
4. Linux PM disables non-boot CPUs, and transfers control to the
   read-in hibernation image. In the now-running hibernation image,
   non-boot CPUs are restarted.
5. As part of the "resume" phase, Linux PM repeats Steps 5 and 6
   from the hibernation sequence. The top-level VMBus connection is
   re-established, and offers are received and matched to primary
   channels in the image. Relids are updated. VMBus device resume
   functions re-open primary channels and re-create sub-channels.
6. Linux PM exits the hibernation resume sequence and the VM is now
   running normally from the hibernation image.

KVP pseudo-device 예외

VMBus KVP device는 다른 Hyper-V pseudo-device와 다르게 동작합니다. KVP primary channel을 닫으면 Hyper-V가 rescind message를 보내 device의 모든 흔적이 제거되지만 곧 다시 offer해 새로 만듭니다.

이 제거·재생성은 hibernation의 `freeze` phase에서 일어나 image에는 재생성된 KVP device가 들어갑니다. Resume sequence의 fresh OS `freeze`에서도 비슷하게 동작합니다.

두 경우 모두 이후 top-level VMBus connection을 unload하면서 Hyper-V 쪽 device가 버려지므로 해가 없고 전체 절차는 계속 동작합니다.

Key-Value Pair (KVP) Pseudo-Device Anomalies
--------------------------------------------
The VMBus KVP device behaves differently from other pseudo-devices
offered by Hyper-V.  When the KVP primary channel is closed, Hyper-V
sends a rescind message, which causes all vestiges of the device to be
removed. But Hyper-V then re-offers the device, causing it to be newly
re-created. The removal and re-creation occurs during the "freeze"
phase of hibernation, so the hibernation image contains the re-created
KVP device. Similar behavior occurs during the "freeze" phase of the
resume sequence while still in the fresh OS instance. But in both
cases, the top-level VMBus connection is subsequently unloaded, which
causes the device to be discarded on the Hyper-V side. So no harm is
done and everything still works.

Virtual PCI와 DDA device

Virtual PCI device는 physical PCI device를 VM의 physical address space에 직접 mapping해 VM이 hardware와 직접 상호 작용하게 합니다. Hyper-V의 Discrete Device Assignment(DDA) device와 SR-IOV NIC Virtual Function(VF)이 포함되며 `Documentation/virt/hyperv/vpci.rst`를 참조합니다.

Hyper-V는 top-level VMBus connection이 만들어진 뒤 synthetic device처럼 DDA device를 guest에 offer합니다. DDA는 VM에 정적으로 할당되며 관리자가 구성을 바꾸지 않는 한 instance GUID가 변하지 않습니다.

Linux에서는 DDA device가 VMBus identity와 PCI identity를 모두 가진 virtual PCI device입니다. 따라서 hibernation은 먼저 VMBus device로 channel을 관리하고, 이어 native PCI driver의 hibernation 함수로 PCI device를 처리합니다.

Virtual PCI devices
-------------------
Virtual PCI devices are physical PCI devices that are mapped directly
into the VM's physical address space so the VM can interact directly
with the hardware. vPCI devices include those accessed via what Hyper-V
calls "Discrete Device Assignment" (DDA), as well as SR-IOV NIC
Virtual Functions (VF) devices. See Documentation/virt/hyperv/vpci.rst.

Hyper-V DDA devices are offered to guest VMs after the top-level VMBus
connection is established, just like VMBus synthetic devices. They are
statically assigned to the VM, and their instance GUIDs don't change
unless the Hyper-V administrator makes changes to the configuration.
DDA devices are represented in Linux as virtual PCI devices that have
a VMBus identity as well as a PCI identity. Consequently, Linux guest
hibernation first handles DDA devices as VMBus devices in order to
manage the VMBus channel. But then they are also handled as PCI
devices using the hibernation functions implemented by their native
PCI driver.

SR-IOV NIC VF 재제공

SR-IOV NIC VF도 VMBus identity와 PCI identity를 가지므로 전체적으로 DDA와 비슷하게 처리합니다.

차이는 VM initial boot 때 VF가 바로 offer되지 않는다는 점입니다. 먼저 VMBus synthetic NIC driver가 실행되어 VF를 받을 준비가 됐다고 Hyper-V에 알린 뒤 offer가 옵니다.

그러나 VM reboot 없이 VMBus connection만 unload·re-establish하는 hibernate·resume 경로에서는 VF가 initial boot 중 이미 VM의 일부가 됐으므로 synthetic NIC driver의 별도 개입 없이 재구성된 connection에서 바로 offer됩니다.

SR-IOV NIC VFs also have a VMBus identity as well as a PCI
identity, and overall are processed similarly to DDA devices. A
difference is that VFs are not offered to the VM during initial boot
of the VM. Instead, the VMBus synthetic NIC driver first starts
operating and communicates to Hyper-V that it is prepared to accept a
VF, and then the VF offer is made. However, the VMBus connection
might later be unloaded and then re-established without the VM being
rebooted, as happens in Steps 3 and 5 in the Detailed Hibernation
Sequence above and in the Detailed Resume Sequence. In such a case,
the VFs likely became part of the VM during initial boot, so when the
VMBus connection is re-established, the VFs are offered on the
re-established connection without intervention by the synthetic NIC driver.

UIO device 제한

Hyper-V UIO driver `uio_hv_generic.c`를 사용하면 VMBus device를 사용자 공간에 노출해 사용자 공간 driver가 제어할 수 있습니다.

하지만 VMBus UIO driver는 hibernation에 필요한 suspend·resume operation을 지원하지 않습니다. UIO를 사용하는 VMBus device가 있으면 VM hibernation은 실패하고 Linux는 계속 정상 실행합니다.

가장 흔한 사용 사례는 DPDK networking이지만 다른 용도로도 쓰입니다.

UIO Devices
-----------
A VMBus device can be exposed to user space using the Hyper-V UIO
driver (uio_hv_generic.c) so that a user space driver can control and
operate the device. However, the VMBus UIO driver does not support the
suspend and resume operations needed for hibernation. If a VMBus
device is configured to use the UIO driver, hibernating the VM fails
and Linux continues to run normally. The most common use of the Hyper-V
UIO driver is for DPDK networking, but there are other uses as well.

다른 VM에서 resume하는 Azure 시나리오

Azure public cloud에서는 hibernated customer VM이 저장된 configuration과 disk로만 존재하고 어느 Hyper-V host에도 실행 중인 VM이 남지 않을 수 있습니다.

Resume할 때 동일 configuration의 새 Hyper-V VM을 만들며 다른 host일 가능성이 큽니다. Linux kernel의 image resume 단계가 이 새 VM에서도 동작해야 합니다.

Disk와 내용은 보존되지만 disk controller와 synthetic device에 Hyper-V가 제공하는 VMBus instance GUID는 일반적으로 달라집니다. 그대로면 resume가 실패하므로 Azure가 여러 특별 조치를 적용합니다.

Resuming on a Different VM
--------------------------
This scenario occurs in the Azure public cloud in that a hibernated
customer VM only exists as saved configuration and disks -- the VM no
longer exists on any Hyper-V host. When the customer VM is resumed, a
new Hyper-V VM with identical configuration is created, likely on a
different Hyper-V host. That new Hyper-V VM becomes the resumed
customer VM, and the steps the Linux kernel takes to resume from the
hibernation image must work in that new VM.

While the disks and their contents are preserved from the original VM,
the Hyper-V-provided VMBus instance GUIDs of the disk controllers and
other synthetic devices would typically be different. The difference
would cause the resume from hibernation to fail, so several things are
done to solve this problem:

고정 GUID와 Azure SCSI controller

Single-instance만 지원하는 VMBus synthetic device에는 Hyper-V가 항상 같은 instance GUID를 할당합니다. Hyper-V mouse, shutdown pseudo-device, time-sync pseudo-device 등이 이에 해당하며 local Hyper-V와 Azure 모두 같습니다.

Synthetic SCSI controller는 VM에 여러 instance가 있을 수 있어 일반적으로 VM마다 GUID가 달라집니다. 하지만 Azure VM은 정확히 두 controller를 사용하고 Azure code가 정상 Hyper-V 동작을 override해 항상 같은 두 instance GUID를 할당합니다.

따라서 새 VM에서 customer VM을 resume해도 SCSI controller GUID가 일치하지만 이 보장은 local Hyper-V 설치에는 적용되지 않습니다.

* For VMBus synthetic devices that support only a single instance,
  Hyper-V always assigns the same instance GUIDs. For example, the
  Hyper-V mouse, the shutdown pseudo-device, the time sync pseudo
  device, etc., always have the same instance GUID, both for local
  Hyper-V installs as well as in the Azure cloud.

* VMBus synthetic SCSI controllers may have multiple instances in a
  VM, and in the general case instance GUIDs vary from VM to VM.
  However, Azure VMs always have exactly two synthetic SCSI
  controllers, and Azure code overrides the normal Hyper-V behavior
  so these controllers are always assigned the same two instance
  GUIDs. Consequently, when a customer VM is resumed on a newly
  created VM, the instance GUIDs match. But this guarantee does not
  hold for local Hyper-V installs.

Azure synthetic NIC GUID

VMBus synthetic NIC도 여러 instance가 가능해 일반 Hyper-V에서는 VM마다 GUID가 달라집니다.

Azure는 customer VM을 deallocate·hibernate한 뒤 새 VM에서 다시 구성해도 synthetic NIC instance GUID가 변하지 않도록 정상 Hyper-V 동작을 override합니다. SCSI controller와 마찬가지로 local Hyper-V에는 이 보장이 없습니다.

* Similarly, VMBus synthetic NICs may have multiple instances in a
  VM, and the instance GUIDs vary from VM to VM. Again, Azure code
  overrides the normal Hyper-V behavior so that the instance GUID
  of a synthetic NIC in a customer VM does not change, even if the
  customer VM is deallocated or hibernated, and then re-constituted
  on a newly created VM. As with SCSI controllers, this behavior
  does not hold for local Hyper-V installs.

Azure vPCI와 VF 제약

새 VM에서 resume할 때 vPCI device의 instance GUID는 같지 않습니다. 따라서 Azure는 NVMe controller나 GPU 같은 DDA device가 있는 VM의 hibernation을 지원하지 않습니다.

SR-IOV NIC VF는 hibernate 전에 Azure가 VM에서 제거해 image에 VF device가 남지 않게 합니다. Resume할 때 image의 VF와 matching하지 않고 새 VF를 만듭니다.

VF를 미리 제거해야 하므로 Azure VM hibernation은 Azure Portal이나 Azure CLI에서 외부적으로 시작해야 합니다. 이 경로가 Shutdown integration service를 통해 Linux에 요청합니다.

Azure VM 안에서 self-initiate하면 VF가 image에 남아 올바르게 resume되지 않습니다.

* vPCI devices do not have the same instance GUIDs when resuming
  from hibernation on a newly created VM. Consequently, Azure does
  not support hibernation for VMs that have DDA devices such as
  NVMe controllers or GPUs. For SR-IOV NIC VFs, Azure removes the
  VF from the VM before it hibernates so that the hibernation image
  does not contain a VF device. When the VM is resumed it
  instantiates a new VF, rather than trying to match against a VF
  that is present in the hibernation image. Because Azure must
  remove any VFs before initiating hibernation, Azure VM
  hibernation must be initiated externally from the Azure Portal or
  Azure CLI, which in turn uses the Shutdown integration service to
  tell Linux to do the hibernation. If hibernation is self-initiated
  within the Azure VM, VFs remain in the hibernation image, and are
  not resumed properly.

다른 VM resume 요약

Azure는 VF를 제거하고 새 VM에서도 VMBus device instance GUID가 일치하도록 특별 조치를 적용해 대부분의 general-purpose VM size에서 hibernation을 지원합니다.

Local Hyper-V에서도 비슷한 orchestration을 구현할 수 있지만 기본 제공되지 않으므로 custom scripting이 필요합니다.

In summary, Azure takes special actions to remove VFs and to ensure
that VMBus device instance GUIDs match on a new/different VM, allowing
hibernation to work for most general-purpose Azure VMs sizes. While
similar special actions could be taken when resuming on a different VM
on a local Hyper-V install, orchestrating such actions is not provided
out-of-the-box by local Hyper-V and so requires custom scripting.