Interaction of Suspend code (S3) with the CPU hotplug infrastructure

Suspend와 CPU hotplug 비교 범위

이 문서는 Suspend-to-RAM(S3)이 CPU hotplug infrastructure를 내부적으로 사용하는 방식과 일반 CPU hotplug의 차이·공통 코드를 설명합니다. Freezer, CPU hotplug, 관련 lock과 notification의 call path에 초점을 둡니다.

원문 diagram은 두 경로가 갈라지고 합쳐지는 지점만 나타내며, 일반 hotplug와 suspend가 서로 경쟁할 때의 실제 interleaving은 그리지 않습니다. 해당 race는 마지막 절에서 별도로 다룹니다.

====================================================================
Interaction of Suspend code (S3) with the CPU hotplug infrastructure
====================================================================

(C) 2011 - 2014 Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>


I. Differences between CPU hotplug and Suspend-to-RAM
======================================================

How does the regular CPU hotplug code differ from how the Suspend-to-RAM
infrastructure uses it internally? And where do they share common code?

Well, a picture is worth a thousand words... So ASCII art follows :-)

[This depicts the current design in the kernel, and focuses only on the
interactions involving the freezer and CPU hotplug and also tries to explain
the locking involved. It outlines the notifications involved as well.
But please note that here, only the call paths are illustrated, with the aim
of describing where they take different paths and where they share code.
What happens when regular CPU hotplug and Suspend-to-RAM race with each other
is not depicted here.]

Suspend call path

Suspend-resume의 큰 흐름은 task freeze, nonboot CPU disable, suspend, nonboot CPU enable, task thaw 순서입니다.

`/sys/power/state`에 `mem`을 쓰면 `system_transition_mutex`를 획득하고 `PM_SUSPEND_PREPARE` notification을 보낸 뒤 task를 freeze합니다. 이어 `freeze_secondary_cpus()`가 `cpu_add_remove_lock`을 잡고 현재 online CPU를 순회합니다.

각 nonboot CPU에는 공통 코드 `_cpu_down()`을 호출합니다. 이 함수는 CPU를 내리는 동안 `cpuhotplug.lock`을 잡고 notable event마다 등록 callback을 실행합니다. 완료된 CPU는 `frozen_cpus` mask에 기록합니다.

그 뒤 `cpu_hotplug_disabled`를 증가시켜 일반 CPU hotplug를 disable하고 `cpu_add_remove_lock`을 해제한 다음 실제 suspend 단계로 들어갑니다.

On a high level, the suspend-resume cycle goes like this::

  |Freeze| -> |Disable nonboot| -> |Do suspend| -> |Enable nonboot| -> |Thaw |
  |tasks |    |     cpus      |    |          |    |     cpus     |    |tasks|


More details follow::

                                Suspend call path
                                -----------------

                                  Write 'mem' to
                                /sys/power/state
                                    sysfs file
                                        |
                                        v
                               Acquire system_transition_mutex lock
                                        |
                                        v
                             Send PM_SUSPEND_PREPARE
                                   notifications
                                        |
                                        v
                                   Freeze tasks
                                        |
                                        |
                                        v
                              freeze_secondary_cpus()
                                   /* start */
                                        |
                                        v
                            Acquire cpu_add_remove_lock
                                        |
                                        v
                             Iterate over CURRENTLY
                                   online CPUs
                                        |
                                        |
                                        |                ----------
                                        v                          | L
             ======>               _cpu_down()                     |
            |              [This takes cpuhotplug.lock             |
  Common    |               before taking down the CPU             |
   code     |               and releases it when done]             | O
            |            While it is at it, notifications          |
            |            are sent when notable events occur,       |
             ======>     by running all registered callbacks.      |
                                        |                          | O
                                        |                          |
                                        |                          |
                                        v                          |
                            Note down these cpus in                | P
                                frozen_cpus mask         ----------
                                        |
                                        v
                           Disable regular cpu hotplug
                        by increasing cpu_hotplug_disabled
                                        |
                                        v
                            Release cpu_add_remove_lock
                                        |
                                        v
                       /* freeze_secondary_cpus() complete */
                                        |
                                        v
                                   Do suspend

Resume path와 전환 lock

Resume에서는 `thaw_secondary_cpus()`가 `cpu_add_remove_lock`을 획득하고 `cpu_hotplug_disabled`를 감소시켜 일반 hotplug를 다시 허용합니다. 그런 다음 `frozen_cpus` mask의 CPU마다 `_cpu_up()`을 호출하고 lock을 해제합니다.

이후 task를 thaw하고 `PM_POST_SUSPEND` notification을 보낸 뒤 `system_transition_mutex`를 해제합니다. 이 mutex는 suspend 시작 직전에 잡혀 suspend와 resume 전체 cycle이 끝날 때까지 유지됩니다.

Resuming back is likewise, with the counterparts being (in the order of
execution during resume):

* thaw_secondary_cpus() which involves::

   |  Acquire cpu_add_remove_lock
   |  Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
   |  Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop]
   |  Release cpu_add_remove_lock
   v

* thaw tasks
* send PM_POST_SUSPEND notifications
* Release system_transition_mutex lock.


It is to be noted here that the system_transition_mutex lock is acquired at the
very beginning, when we are just starting out to suspend, and then released only
after the entire cycle is complete (i.e., suspend + resume).

일반 CPU hotplug와 공통 코드

일반 CPU offline은 `/sys/devices/system/cpu/cpu*/online`에 0을 쓰면서 시작합니다. `cpu_down()`이 `cpu_add_remove_lock`을 잡고 `cpu_hotplug_disabled > 0`이면 정상적으로 빠져나옵니다. 허용되면 `_cpu_down()`이 `cpuhotplug.lock` 아래 CPU를 내리고 등록 callback을 실행한 뒤 바깥 lock을 해제합니다.

CPU online은 대응하는 `cpu_up()`과 `_cpu_up()` 경로를 사용합니다. 따라서 일반 hotplug와 suspend 경로는 `_cpu_down()` 및 `_cpu_up()`에서 합쳐집니다.

차이는 `tasks_frozen` 인수입니다. 일반 hotplug는 항상 0을 넘기지만 suspend에서는 nonboot CPU를 offline/online할 때 이미 task가 freeze되어 있으므로 1을 넘깁니다.

::



                          Regular CPU hotplug call path
                          -----------------------------

                                Write 0 (or 1) to
                       /sys/devices/system/cpu/cpu*/online
                                    sysfs file
                                        |
                                        |
                                        v
                                    cpu_down()
                                        |
                                        v
                           Acquire cpu_add_remove_lock
                                        |
                                        v
                          If cpu_hotplug_disabled > 0
                                return gracefully
                                        |
                                        |
                                        v
             ======>                _cpu_down()
            |              [This takes cpuhotplug.lock
  Common    |               before taking down the CPU
   code     |               and releases it when done]
            |            While it is at it, notifications
            |           are sent when notable events occur,
             ======>    by running all registered callbacks.
                                        |
                                        |
                                        v
                          Release cpu_add_remove_lock
                               [That's it!, for
                              regular CPU hotplug]



So, as can be seen from the two diagrams (the parts marked as "Common code"),
regular CPU hotplug and the suspend code path converge at the _cpu_down() and
_cpu_up() functions. They differ in the arguments passed to these functions,
in that during regular CPU hotplug, 0 is passed for the 'tasks_frozen'
argument. But during suspend, since the tasks are already frozen by the time
the non-boot CPUs are offlined or onlined, the _cpu_*() functions are called
with the 'tasks_frozen' argument set to 1.
[See below for some known issues regarding this.]

중요 파일과 microcode 문제

Freezer entry point는 `kernel/power/process.c`의 `freeze_processes()`와 `thaw_processes()`입니다. Suspend entry point는 `kernel/power/suspend.c`의 `suspend_prepare()`, `suspend_enter()`, `suspend_finish()`입니다. CPU hotplug 쪽은 `kernel/cpu.c`의 `cpu_[up|down]()`, `_cpu_[up|down]()`, `[disable|enable]_nonboot_cpus()`입니다.

CPU hotplug에서 중요한 추가 문제는 CPU microcode update입니다. Kernel은 `drivers/base/firmware_loader/main.c`의 `request_firmware()`로 userspace에 microcode image를 요청한다는 점을 전제로 다음 경우를 구분합니다.

Important files and functions/entry points:
-------------------------------------------

- kernel/power/process.c : freeze_processes(), thaw_processes()
- kernel/power/suspend.c : suspend_prepare(), suspend_enter(), suspend_finish()
- kernel/cpu.c: cpu_[up|down](), _cpu_[up|down](),
  [disable|enable]_nonboot_cpus()



II. What are the issues involved in CPU hotplug?
------------------------------------------------

There are some interesting situations involving CPU hotplug and microcode
update on the CPUs, as discussed below:

[Please bear in mind that the kernel requests the microcode images from
userspace, using the request_firmware() function defined in
drivers/base/firmware_loader/main.c]

동일하거나 다른 CPU의 microcode

모든 CPU가 동일하면 같은 microcode revision을 각각 적용합니다. x86의 `arch/x86/kernel/microcode_core.c`에 있는 `collect_cpu_info()`가 CPU type을 찾아 올바른 revision을 적용하도록 돕습니다.

Kernel은 아래의 이기종 CPU 경우를 처리하기 위해 모든 CPU가 같더라도 공통 image 하나만 유지하지 않습니다. 일부 CPU가 다르면 `collect_cpu_info()` 같은 함수로 type/model을 찾은 뒤 CPU마다 올바른 microcode image 사본을 유지합니다.

a. When all the CPUs are identical:

   This is the most common situation and it is quite straightforward: we want
   to apply the same microcode revision to each of the CPUs.
   To give an example of x86, the collect_cpu_info() function defined in
   arch/x86/kernel/microcode_core.c helps in discovering the type of the CPU
   and thereby in applying the correct microcode revision to it.
   But note that the kernel does not maintain a common microcode image for the
   all CPUs, in order to handle case 'b' described below.


b. When some of the CPUs are different than the rest:

   In this case since we probably need to apply different microcode revisions
   to different CPUs, the kernel maintains a copy of the correct microcode
   image for each CPU (after appropriate CPU type/model discovery using
   functions such as collect_cpu_info()).

Physical CPU 교체

CPU를 물리적으로 hot-unplug하고 다른 type의 CPU를 넣을 수 있는 일반 hotplug에서는 offline 시 `CPU_DEAD` notification을 받은 microcode driver callback이 해당 CPU의 kernel-side image 사본을 해제합니다.

새 CPU를 online하면 image가 없으므로 type/model discovery를 다시 수행하고 userspace에 적절한 image를 요청한 뒤 적용합니다.

x86에서는 hotplug event에 등록된 `mc_cpu_callback()`이 `microcode_update_cpu()`를 호출합니다. Valid image가 없으면 `microcode_resume_cpu()` 대신 `microcode_init_cpu()`로 들어가 discovery와 userspace image 획득을 다시 수행합니다.

c. When a CPU is physically hot-unplugged and a new (and possibly different
   type of) CPU is hot-plugged into the system:

   In the current design of the kernel, whenever a CPU is taken offline during
   a regular CPU hotplug operation, upon receiving the CPU_DEAD notification
   (which is sent by the CPU hotplug code), the microcode update driver's
   callback for that event reacts by freeing the kernel's copy of the
   microcode image for that CPU.

   Hence, when a new CPU is brought online, since the kernel finds that it
   doesn't have the microcode image, it does the CPU type/model discovery
   afresh and then requests the userspace for the appropriate microcode image
   for that CPU, which is subsequently applied.

   For example, in x86, the mc_cpu_callback() function (which is the microcode
   update driver's callback registered for CPU hotplug events) calls
   microcode_update_cpu() which would call microcode_init_cpu() in this case,
   instead of microcode_resume_cpu() when it finds that the kernel doesn't
   have a valid microcode image. This ensures that the CPU type/model
   discovery is performed and the right microcode is applied to the CPU after
   getting it from userspace.

Suspend와 hibernation의 microcode

CPU를 물리적으로 제거하지 않는 일반 offline에서는 CPU 전원을 완전히 끄지 않고 가능한 가장 낮은 C-state에 둘 뿐이므로 online 때 microcode를 다시 적용할 필요가 없습니다. 이는 suspend 뒤 resume에서 흔한 상황입니다.

Hibernation에서는 모든 CPU가 완전히 꺼지므로 restore할 때 모든 CPU에 microcode image를 다시 적용해야 합니다. Suspend/resume 또는 hibernate/restore 사이에 다른 type CPU를 물리적으로 교체하지 않는다는 가정을 사용합니다.

현재 kernel은 suspend/hibernate cycle의 CPU offline에서 `cpuhp_tasks_frozen`이 set되어 있으면 기존 microcode image를 해제하지 않습니다. Resume/restore에서 image가 이미 있으므로 type/model을 다시 찾거나 revision 적합성을 재검증하지 않고 곧바로 CPU에 적용합니다.

d. Handling microcode update during suspend/hibernate:

   Strictly speaking, during a CPU hotplug operation which does not involve
   physically removing or inserting CPUs, the CPUs are not actually powered
   off during a CPU offline. They are just put to the lowest C-states possible.
   Hence, in such a case, it is not really necessary to re-apply microcode
   when the CPUs are brought back online, since they wouldn't have lost the
   image during the CPU offline operation.

   This is the usual scenario encountered during a resume after a suspend.
   However, in the case of hibernation, since all the CPUs are completely
   powered off, during restore it becomes necessary to apply the microcode
   images to all the CPUs.

   [Note that we don't expect someone to physically pull out nodes and insert
   nodes with a different type of CPUs in-between a suspend-resume or a
   hibernate/restore cycle.]

   In the current design of the kernel however, during a CPU offline operation
   as part of the suspend/hibernate cycle (cpuhp_tasks_frozen is set),
   the existing copy of microcode image in the kernel is not freed up.
   And during the CPU online operations (during resume/restore), since the
   kernel finds that it already has copies of the microcode images for all the
   CPUs, it just applies them to the CPUs, avoiding any re-discovery of CPU
   type/model and the need for validating whether the microcode revisions are
   right for the CPUs or not (due to the above assumption that physical CPU
   hotplug will not be done in-between suspend/resume or hibernate/restore
   cycles).

알려진 CPU hotplug race

첫 번째 문제는 일반 CPU hotplug가 `_cpu_down()`과 `_cpu_up()`에 `tasks_frozen=0`을 항상 넘긴다는 점입니다. 동시에 suspend 같은 out-of-band event가 task를 freeze했을 수 있으므로 실제 상태와 달라지고, `cpuhp_tasks_frozen`을 검사하는 callback이 잘못된 경로를 실행할 수 있습니다.

두 번째 문제는 CPU hotplug stress와 suspend freezer의 race입니다. Freezing 시작 전에 userspace의 CPU online request가 kernel로 들어온 뒤 userspace가 freeze될 수 있습니다. CPU online이 아직 microcode update를 끝내지 않았다면 image를 얻으려고 freeze된 userspace를 `TASK_UNINTERRUPTIBLE` 상태로 기다립니다.

Freezer는 남은 task를 freeze하려 하지만 이 wait 때문에 CPU online hotplug task를 freeze하지 못합니다. 결국 task freezing이 실패하고 suspend가 중단됩니다.

III. Known problems
===================

Are there any known problems when regular CPU hotplug and suspend race
with each other?

Yes, they are listed below:

1. When invoking regular CPU hotplug, the 'tasks_frozen' argument passed to
   the _cpu_down() and _cpu_up() functions is *always* 0.
   This might not reflect the true current state of the system, since the
   tasks could have been frozen by an out-of-band event such as a suspend
   operation in progress. Hence, the cpuhp_tasks_frozen variable will not
   reflect the frozen state and the CPU hotplug callbacks which evaluate
   that variable might execute the wrong code path.

2. If a regular CPU hotplug stress test happens to race with the freezer due
   to a suspend operation in progress at the same time, then we could hit the
   situation described below:

    * A regular cpu online operation continues its journey from userspace
      into the kernel, since the freezing has not yet begun.
    * Then freezer gets to work and freezes userspace.
    * If cpu online has not yet completed the microcode update stuff by now,
      it will now start waiting on the frozen userspace in the
      TASK_UNINTERRUPTIBLE state, in order to get the microcode image.
    * Now the freezer continues and tries to freeze the remaining tasks. But
      due to this wait mentioned above, the freezer won't be able to freeze
      the cpu online hotplug task and hence freezing of tasks fails.

   As a result of this task freezing failure, the suspend operation gets
   aborted.