요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
==================
Memory Hot(Un)Plug
==================
This document describes generic Linux support for memory hot(un)plug with
a focus on System RAM, including ZONE_MOVABLE support.
.. contents:: :local:
Introduction
============
Memory hot(un)plug allows for increasing and decreasing the size of physical
memory available to a machine at runtime. In the simplest case, it consists of
physically plugging or unplugging a DIMM at runtime, coordinated with the
operating system.
Memory hot(un)plug is used for various purposes:
- The physical memory available to a machine can be adjusted at runtime, up- or
downgrading the memory capacity. This dynamic memory resizing, sometimes
referred to as "capacity on demand", is frequently used with virtual machines
and logical partitions.
- Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One
example is replacing failing memory modules.
- Reducing energy consumption either by physically unplugging memory modules or
by logically unplugging (parts of) memory modules from Linux.
Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also
used to expose persistent memory, other performance-differentiated memory and
reserved memory regions as ordinary system RAM to Linux.
Linux only supports memory hot(un)plug on selected 64 bit architectures, such as
x86_64, arm64, ppc64 and s390x.
Memory Hot(Un)Plug Granularity
------------------------------
Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the
physical memory address space into chunks of the same size: memory sections. The
size of a memory section is architecture dependent. For example, x86_64 uses
128 MiB and ppc64 uses 16 MiB.
Memory sections are combined into chunks referred to as "memory blocks". The
size of a memory block is architecture dependent and corresponds to the smallest
granularity that can be hot(un)plugged. The default size of a memory block is
the same as memory section size, unless an architecture specifies otherwise.
All memory blocks have the same size.
Phases of Memory Hotplug
------------------------
Memory hotplug consists of two phases:
(1) Adding the memory to Linux
(2) Onlining memory blocks
In the first phase, metadata, such as the memory map ("memmap") and page tables
for the direct mapping, is allocated and initialized, and memory blocks are
created; the latter also creates sysfs files for managing newly created memory
blocks.
In the second phase, added memory is exposed to the page allocator. After this
phase, the memory is visible in memory statistics, such as free and total
memory, of the system.
Phases of Memory Hotunplug
--------------------------
Memory hotunplug consists of two phases:
(1) Offlining memory blocks
(2) Removing the memory from Linux
In the first phase, memory is "hidden" from the page allocator again, for
example, by migrating busy memory to other memory locations and removing all
relevant free pages from the page allocator After this phase, the memory is no
longer visible in memory statistics of the system.
In the second phase, the memory blocks are removed and metadata is freed.
Memory Hotplug Notifications
============================
There are various ways how Linux is notified about memory hotplug events such
that it can start adding hotplugged memory. This description is limited to
systems that support ACPI; mechanisms specific to other firmware interfaces or
virtual machines are not described.
ACPI Notifications
------------------
Platforms that support ACPI, such as x86_64, can support memory hotplug
notifications via ACPI.
In general, a firmware supporting memory hotplug defines a memory class object
HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI
driver will hotplug the memory to Linux.
If the firmware supports hotplug of NUMA nodes, it defines an object _HID
"ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all
assigned memory devices are added to Linux by the ACPI driver.
Similarly, Linux can be notified about requests to hotunplug a memory device or
a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory
blocks, and, if successful, hotunplug the memory from Linux.
Manual Probing
--------------
On some architectures, the firmware may not be able to notify the operating
system about a memory hotplug event. Instead, the memory has to be manually
probed from user space.
The probe interface is located at::
/sys/devices/system/memory/probe
Only complete memory blocks can be probed. Individual memory blocks are probed
by providing the physical start address of the memory block::
% echo addr > /sys/devices/system/memory/probe
Which results in a memory block for the range [addr, addr + memory_block_size)
being created.
.. note::
Using the probe interface is discouraged as it is easy to crash the kernel,
because Linux cannot validate user input; this interface might be removed in
the future.
Onlining and Offlining Memory Blocks
====================================
After a memory block has been created, Linux has to be instructed to actually
make use of that memory: the memory block has to be "online".
Before a memory block can be removed, Linux has to stop using any memory part of
the memory block: the memory block has to be "offlined".
The Linux kernel can be configured to automatically online added memory blocks
and drivers automatically trigger offlining of memory blocks when trying
hotunplug of memory. Memory blocks can only be removed once offlining succeeded
and drivers may trigger offlining of memory blocks when attempting hotunplug of
memory.
Onlining Memory Blocks Manually
-------------------------------
If auto-onlining of memory blocks isn't enabled, user-space has to manually
trigger onlining of memory blocks. Often, udev rules are used to automate this
task in user space.
Onlining of a memory block can be triggered via::
% echo online > /sys/devices/system/memory/memoryXXX/state
Or alternatively::
% echo 1 > /sys/devices/system/memory/memoryXXX/online
The kernel will select the target zone automatically, depending on the
configured ``online_policy``.
One can explicitly request to associate an offline memory block with
ZONE_MOVABLE by::
% echo online_movable > /sys/devices/system/memory/memoryXXX/state
Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by::
% echo online_kernel > /sys/devices/system/memory/memoryXXX/state
In any case, if onlining succeeds, the state of the memory block is changed to
be "online". If it fails, the state of the memory block will remain unchanged
and the above commands will fail.
Onlining Memory Blocks Automatically
------------------------------------
The kernel can be configured to try auto-onlining of newly added memory blocks.
If this feature is disabled, the memory blocks will stay offline until
explicitly onlined from user space.
The configured auto-online behavior can be observed via::
% cat /sys/devices/system/memory/auto_online_blocks
Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
``online_movable`` to that file, like::
% echo online > /sys/devices/system/memory/auto_online_blocks
Similarly to manual onlining, with ``online`` the kernel will select the
target zone automatically, depending on the configured ``online_policy``.
Modifying the auto-online behavior will only affect all subsequently added
memory blocks only.
.. note::
In corner cases, auto-onlining can fail. The kernel won't retry. Note that
auto-onlining is not expected to fail in default configurations.
.. note::
DLPAR on ppc64 ignores the ``offline`` setting and will still online added
memory blocks; if onlining fails, memory blocks are removed again.
Offlining Memory Blocks
-----------------------
In the current implementation, Linux's memory offlining will try migrating all
movable pages off the affected memory block. As most kernel allocations, such as
page tables, are unmovable, page migration can fail and, therefore, inhibit
memory offlining from succeeding.
Having the memory provided by memory block managed by ZONE_MOVABLE significantly
increases memory offlining reliability; still, memory offlining can fail in
some corner cases.
Further, memory offlining might retry for a long time (or even forever), until
aborted by the user.
Offlining of a memory block can be triggered via::
% echo offline > /sys/devices/system/memory/memoryXXX/state
Or alternatively::
% echo 0 > /sys/devices/system/memory/memoryXXX/online
If offlining succeeds, the state of the memory block is changed to be "offline".
If it fails, the state of the memory block will remain unchanged and the above
commands will fail, for example, via::
bash: echo: write error: Device or resource busy
or via::
bash: echo: write error: Invalid argument
Observing the State of Memory Blocks
------------------------------------
The state (online/offline/going-offline) of a memory block can be observed
either via::
% cat /sys/devices/system/memory/memoryXXX/state
Or alternatively (1/0) via::
% cat /sys/devices/system/memory/memoryXXX/online
For an online memory block, the managing zone can be observed via::
% cat /sys/devices/system/memory/memoryXXX/valid_zones
Configuring Memory Hot(Un)Plug
==============================
There are various ways how system administrators can configure memory
hot(un)plug and interact with memory blocks, especially, to online them.
Memory Hot(Un)Plug Configuration via Sysfs
------------------------------------------
Some memory hot(un)plug properties can be configured or inspected via sysfs in::
/sys/devices/system/memory/
The following files are currently defined:
====================== =========================================================
``auto_online_blocks`` read-write: set or get the default state of new memory
blocks; configure auto-onlining.
The default value depends on the
CONFIG_MHP_DEFAULT_ONLINE_TYPE kernel configuration
options.
See the ``state`` property of memory blocks for details.
``block_size_bytes`` read-only: the size in bytes of a memory block.
``probe`` write-only: add (probe) selected memory blocks manually
from user space by supplying the physical start address.
Availability depends on the CONFIG_ARCH_MEMORY_PROBE
kernel configuration option.
``uevent`` read-write: generic udev file for device subsystems.
``crash_hotplug`` read-only: when changes to the system memory map
occur due to hot un/plug of memory, this file contains
'1' if the kernel updates the kdump capture kernel memory
map itself (via elfcorehdr and other relevant kexec
segments), or '0' if userspace must update the kdump
capture kernel memory map.
Availability depends on the CONFIG_MEMORY_HOTPLUG kernel
configuration option.
====================== =========================================================
.. note::
When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two
additional files ``hard_offline_page`` and ``soft_offline_page`` are available
to trigger hwpoisoning of pages, for example, for testing purposes. Note that
this functionality is not really related to memory hot(un)plug or actual
offlining of memory blocks.
Memory Block Configuration via Sysfs
------------------------------------
Each memory block is represented as a memory block device that can be
onlined or offlined. All memory blocks have their device information located in
sysfs. Each present memory block is listed under
``/sys/devices/system/memory`` as::
/sys/devices/system/memory/memoryXXX
where XXX is the memory block id; the number of digits is variable.
A present memory block indicates that some memory in the range is present;
however, a memory block might span memory holes. A memory block spanning memory
holes cannot be offlined.
For example, assume 1 GiB memory block size. A device for a memory starting at
0x100000000 is ``/sys/devices/system/memory/memory4``::
(0x100000000 / 1Gib = 4)
This device covers address range [0x100000000 ... 0x140000000)
The following files are currently defined:
=================== ============================================================
``online`` read-write: simplified interface to trigger onlining /
offlining and to observe the state of a memory block.
When onlining, the zone is selected automatically.
``phys_device`` read-only: legacy interface only ever used on s390x to
expose the covered storage increment.
``phys_index`` read-only: the memory block id (XXX).
``removable`` read-only: legacy interface that indicated whether a memory
block was likely to be offlineable or not. Nowadays, the
kernel return ``1`` if and only if it supports memory
offlining.
``state`` read-write: advanced interface to trigger onlining /
offlining and to observe the state of a memory block.
When writing, ``online``, ``offline``, ``online_kernel`` and
``online_movable`` are supported.
``online_movable`` specifies onlining to ZONE_MOVABLE.
``online_kernel`` specifies onlining to the default kernel
zone for the memory block, such as ZONE_NORMAL.
``online`` let's the kernel select the zone automatically.
When reading, ``online``, ``offline`` and ``going-offline``
may be returned.
``uevent`` read-write: generic uevent file for devices.
``valid_zones`` read-only: when a block is online, shows the zone it
belongs to; when a block is offline, shows what zone will
manage it when the block will be onlined.
For online memory blocks, ``DMA``, ``DMA32``, ``Normal``,
``Movable`` and ``none`` may be returned. ``none`` indicates
that memory provided by a memory block is managed by
multiple zones or spans multiple nodes; such memory blocks
cannot be offlined. ``Movable`` indicates ZONE_MOVABLE.
Other values indicate a kernel zone.
For offline memory blocks, the first column shows the
zone the kernel would select when onlining the memory block
right now without further specifying a zone.
Availability depends on the CONFIG_MEMORY_HOTREMOVE
kernel configuration option.
=================== ============================================================
.. note::
If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/
directories can also be accessed via symbolic links located in the
``/sys/devices/system/node/node*`` directories.
For example::
/sys/devices/system/node/node0/memory9 -> ../../memory/memory9
A backlink will also be created::
/sys/devices/system/memory/memory9/node0 -> ../../node/node0
Command Line Parameters
-----------------------
Some command line parameters affect memory hot(un)plug handling. The following
command line parameters are relevant:
======================== =======================================================
``memhp_default_state`` configure auto-onlining by essentially setting
``/sys/devices/system/memory/auto_online_blocks``.
``movable_node`` configure automatic zone selection in the kernel when
using the ``contig-zones`` online policy. When
set, the kernel will default to ZONE_MOVABLE when
onlining a memory block, unless other zones can be kept
contiguous.
======================== =======================================================
See Documentation/admin-guide/kernel-parameters.txt for a more generic
description of these command line parameters.
Module Parameters
------------------
Instead of additional command line parameters or sysfs files, the
``memory_hotplug`` subsystem now provides a dedicated namespace for module
parameters. Module parameters can be set via the command line by predicating
them with ``memory_hotplug.`` such as::
memory_hotplug.memmap_on_memory=1
and they can be observed (and some even modified at runtime) via::
/sys/module/memory_hotplug/parameters/
The following module parameters are currently defined:
================================ ===============================================
``memmap_on_memory`` read-write: Allocate memory for the memmap from
the added memory block itself. Even if enabled,
actual support depends on various other system
properties and should only be regarded as a
hint whether the behavior would be desired.
While allocating the memmap from the memory
block itself makes memory hotplug less likely
to fail and keeps the memmap on the same NUMA
node in any case, it can fragment physical
memory in a way that huge pages in bigger
granularity cannot be formed on hotplugged
memory.
With value "force" it could result in memory
wastage due to memmap size limitations. For
example, if the memmap for a memory block
requires 1 MiB, but the pageblock size is 2
MiB, 1 MiB of hotplugged memory will be wasted.
Note that there are still cases where the
feature cannot be enforced: for example, if the
memmap is smaller than a single page, or if the
architecture does not support the forced mode
in all configurations.
``online_policy`` read-write: Set the basic policy used for
automatic zone selection when onlining memory
blocks without specifying a target zone.
``contig-zones`` has been the kernel default
before this parameter was added. After an
online policy was configured and memory was
online, the policy should not be changed
anymore.
When set to ``contig-zones``, the kernel will
try keeping zones contiguous. If a memory block
intersects multiple zones or no zone, the
behavior depends on the ``movable_node`` kernel
command line parameter: default to ZONE_MOVABLE
if set, default to the applicable kernel zone
(usually ZONE_NORMAL) if not set.
When set to ``auto-movable``, the kernel will
try onlining memory blocks to ZONE_MOVABLE if
possible according to the configuration and
memory device details. With this policy, one
can avoid zone imbalances when eventually
hotplugging a lot of memory later and still
wanting to be able to hotunplug as much as
possible reliably, very desirable in
virtualized environments. This policy ignores
the ``movable_node`` kernel command line
parameter and isn't really applicable in
environments that require it (e.g., bare metal
with hotunpluggable nodes) where hotplugged
memory might be exposed via the
firmware-provided memory map early during boot
to the system instead of getting detected,
added and onlined later during boot (such as
done by virtio-mem or by some hypervisors
implementing emulated DIMMs). As one example, a
hotplugged DIMM will be onlined either
completely to ZONE_MOVABLE or completely to
ZONE_NORMAL, not a mixture.
As another example, as many memory blocks
belonging to a virtio-mem device will be
onlined to ZONE_MOVABLE as possible,
special-casing units of memory blocks that can
only get hotunplugged together. *This policy
does not protect from setups that are
problematic with ZONE_MOVABLE and does not
change the zone of memory blocks dynamically
after they were onlined.*
``auto_movable_ratio`` read-write: Set the maximum MOVABLE:KERNEL
memory ratio in % for the ``auto-movable``
online policy. Whether the ratio applies only
for the system across all NUMA nodes or also
per NUMA nodes depends on the
``auto_movable_numa_aware`` configuration.
All accounting is based on present memory pages
in the zones combined with accounting per
memory device. Memory dedicated to the CMA
allocator is accounted as MOVABLE, although
residing on one of the kernel zones. The
possible ratio depends on the actual workload.
The kernel default is "301" %, for example,
allowing for hotplugging 24 GiB to a 8 GiB VM
and automatically onlining all hotplugged
memory to ZONE_MOVABLE in many setups. The
additional 1% deals with some pages being not
present, for example, because of some firmware
allocations.
Note that ZONE_NORMAL memory provided by one
memory device does not allow for more
ZONE_MOVABLE memory for a different memory
device. As one example, onlining memory of a
hotplugged DIMM to ZONE_NORMAL will not allow
for another hotplugged DIMM to get onlined to
ZONE_MOVABLE automatically. In contrast, memory
hotplugged by a virtio-mem device that got
onlined to ZONE_NORMAL will allow for more
ZONE_MOVABLE memory within *the same*
virtio-mem device.
``auto_movable_numa_aware`` read-write: Configure whether the
``auto_movable_ratio`` in the ``auto-movable``
online policy also applies per NUMA
node in addition to the whole system across all
NUMA nodes. The kernel default is "Y".
Disabling NUMA awareness can be helpful when
dealing with NUMA nodes that should be
completely hotunpluggable, onlining the memory
completely to ZONE_MOVABLE automatically if
possible.
Parameter availability depends on CONFIG_NUMA.
================================ ===============================================
ZONE_MOVABLE
============
ZONE_MOVABLE is an important mechanism for more reliable memory offlining.
Further, having system RAM managed by ZONE_MOVABLE instead of one of the
kernel zones can increase the number of possible transparent huge pages and
dynamically allocated huge pages.
Most kernel allocations are unmovable. Important examples include the memory
map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations
can only be served from the kernel zones.
Most user space pages, such as anonymous memory, and page cache pages are
movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones.
Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable
allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is
absolutely no guarantee whether a memory block can be offlined successfully.
Zone Imbalances
---------------
Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance,
which can harm the system or degrade performance. As one example, the kernel
might crash because it runs out of free memory for unmovable allocations,
although there is still plenty of free memory left in ZONE_MOVABLE.
Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1
are definitely impossible due to the overhead for the memory map.
Actual safe zone ratios depend on the workload. Extreme cases, like excessive
long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all.
.. note::
CMA memory part of a kernel zone essentially behaves like memory in
ZONE_MOVABLE and similar considerations apply, especially when combining
CMA with ZONE_MOVABLE.
ZONE_MOVABLE Sizing Considerations
----------------------------------
We usually expect that a large portion of available system RAM will actually
be consumed by user space, either directly or indirectly via the page cache. In
the normal case, ZONE_MOVABLE can be used when allocating such pages just fine.
With that in mind, it makes sense that we can have a big portion of system RAM
managed by ZONE_MOVABLE. However, there are some things to consider when using
ZONE_MOVABLE, especially when fine-tuning zone ratios:
- Having a lot of offline memory blocks. Even offline memory blocks consume
memory for metadata and page tables in the direct map; having a lot of offline
memory blocks is not a typical case, though.
- Memory ballooning without balloon compaction is incompatible with
ZONE_MOVABLE. Only some implementations, such as virtio-balloon and
pseries CMM, fully support balloon compaction.
Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be
disabled. In that case, balloon inflation will only perform unmovable
allocations and silently create a zone imbalance, usually triggered by
inflation requests from the hypervisor.
- Gigantic pages are unmovable, resulting in user space consuming a
lot of unmovable memory.
- Huge pages are unmovable when an architectures does not support huge
page migration, resulting in a similar issue as with gigantic pages.
- Page tables are unmovable. Excessive swapping, mapping extremely large
files or ZONE_DEVICE memory can be problematic, although only really relevant
in corner cases. When we manage a lot of user space memory that has been
swapped out or is served from a file/persistent memory/... we still need a lot
of page tables to manage that memory once user space accessed that memory.
- In certain DAX configurations the memory map for the device memory will be
allocated from the kernel zones.
- KASAN can have a significant memory overhead, for example, consuming 1/8th of
the total system memory size as (unmovable) tracking metadata.
- Long-term pinning of pages. Techniques that rely on long-term pinnings
(especially, RDMA and vfio/mdev) are fundamentally problematic with
ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside
on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they
have to be migrated off that zone while pinning. Pinning a page can fail
even if there is plenty of free memory in ZONE_MOVABLE.
In addition, using ZONE_MOVABLE might make page pinning more expensive,
because of the page migration overhead.
By default, all the memory configured at boot time is managed by the kernel
zones and ZONE_MOVABLE is not used.
To enable ZONE_MOVABLE to include the memory present at boot and to control the
ratio between movable and kernel zones there are two command line options:
``kernelcore=`` and ``movablecore=``. See
Documentation/admin-guide/kernel-parameters.rst for their description.
Memory Offlining and ZONE_MOVABLE
---------------------------------
Even with ZONE_MOVABLE, there are some corner cases where offlining a memory
block might fail:
- Memory blocks with memory holes; this applies to memory blocks present during
boot and can apply to memory blocks hotplugged via the XEN balloon and the
Hyper-V balloon.
- Mixed NUMA nodes and mixed zones within a single memory block prevent memory
offlining; this applies to memory blocks present during boot only.
- Special memory blocks prevented by the system from getting offlined. Examples
include any memory available during boot on arm64 or memory blocks spanning
the crashkernel area on s390x; this usually applies to memory blocks present
during boot only.
- Memory blocks overlapping with CMA areas cannot be offlined, this applies to
memory blocks present during boot only.
- Concurrent activity that operates on the same physical memory area, such as
allocating gigantic pages, can result in temporary offlining failures.
- Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap
Optimization (HVO) is enabled.
Offlining code may be able to migrate huge page contents, but may not be able
to dissolve the source huge page because it fails allocating (unmovable) pages
for the vmemmap, because the system might not have free memory in the kernel
zones left.
Users that depend on memory offlining to succeed for movable zones should
carefully consider whether the memory savings gained from this feature are
worth the risk of possibly not being able to offline memory in certain
situations.
Further, when running into out of memory situations while migrating pages, or
when still encountering permanently unmovable pages within ZONE_MOVABLE
(-> BUG), memory offlining will keep retrying until it eventually succeeds.
When offlining is triggered from user space, the offlining context can be
terminated by sending a signal. A timeout based offlining can easily be
implemented via::
% timeout $TIMEOUT offline_block | failure_handling
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 개요
1-9이 문서는 System RAM과 ZONE_MOVABLE 지원을 중심으로 Linux의 일반적인 memory hotplug 및 hotunplug 기능을 설명합니다. 아래 내용은 원문의 local contents가 가리키는 구성·상태·정책·실패 조건을 순서대로 번역합니다.
소개와 사용 목적
10-37Memory hot(un)plug는 실행 중인 machine에서 사용할 수 있는 physical memory 크기를 늘리거나 줄입니다. 가장 단순한 형태는 operating system과 조율하면서 runtime에 DIMM을 물리적으로 꽂거나 빼는 것입니다.
| 목적 | 설명 |
|---|---|
| 동적 용량 | runtime에 physical memory를 증감하는 "capacity on demand"; virtual machine과 logical partition에서 흔히 사용 |
| 무중단 교체 | 고장 난 DIMM 또는 전체 NUMA node 같은 hardware를 downtime 없이 교체 |
| 전력 절감 | memory module을 물리적으로 제거하거나 일부를 Linux에서 논리적으로 분리 |
| System RAM 노출 | persistent memory, 성능 특성이 다른 memory, reserved region을 일반 System RAM으로 제공 |
Linux의 기본 memory hot(un)plug infrastructure는 이제 persistent memory, 성능 특성이 다른 memory와 reserved memory region을 일반 System RAM으로 노출하는 데도 쓰입니다. 지원 범위는 x86_64, arm64, ppc64, s390x 같은 일부 64-bit architecture로 한정됩니다.
Memory block 단위
38-52Linux memory hot(un)plug는 physical address space를 같은 크기의 memory section으로 나누는 SPARSEMEM model을 사용합니다. section 크기는 architecture별로 다르며, 예를 들어 x86_64는 128 MiB, ppc64는 16 MiB입니다.
여러 memory section은 memory block으로 결합됩니다. memory block 크기는 architecture가 hot(un)plug할 수 있는 최소 단위이며, 별도 지정이 없으면 memory section 크기와 같습니다. 한 system의 모든 memory block 크기는 동일합니다.
Memory hotplug 단계
53-69Memory hotplug는 (1) memory를 Linux에 추가하고 (2) memory block을 online하는 두 단계로 진행됩니다.
| 단계 | 처리 |
|---|---|
| add | memory map("memmap") 같은 metadata와 direct mapping용 page table을 할당·초기화하고 memory block 및 관리용 sysfs 파일을 생성 |
| online | 추가된 memory를 page allocator에 공개하여 system의 free·total memory 통계에 반영 |
Memory hotunplug 단계
70-84Memory hotunplug는 (1) memory block을 offline하고 (2) Linux에서 memory를 제거하는 두 단계입니다.
| 단계 | 처리 |
|---|---|
| offline | 사용 중인 memory를 다른 위치로 migrate하고 관련 free page를 page allocator에서 제거하여 memory 통계에서 숨김 |
| remove | memory block을 제거하고 metadata를 해제 |
Hotplug notification
85-110Linux가 hotplug된 memory를 추가하려면 event를 통지받아야 합니다. 이 절은 ACPI를 지원하는 system만 다루며 다른 firmware interface나 virtual machine 전용 mechanism은 설명하지 않습니다.
x86_64 같은 ACPI platform에서 firmware는 일반적으로 HID "PNP0C80"인 memory class object를 정의합니다. 새 memory device의 hotplug notification을 받으면 ACPI driver가 해당 memory를 Linux에 hotplug합니다.
NUMA node hotplug를 지원하는 firmware는 _HID "ACPI0004", "PNP0A05" 또는 "PNP0A06" object를 정의합니다. event가 오면 ACPI driver가 배정된 모든 memory device를 추가합니다. 반대로 memory device나 NUMA node의 hotunplug 요청도 ACPI로 받을 수 있고, driver는 관련 block을 모두 offline한 뒤 성공한 경우에만 Linux에서 memory를 제거합니다.
수동 probe
111-135일부 architecture의 firmware는 memory hotplug event를 OS에 통지하지 못하므로 user space에서 수동 probe해야 합니다. probe interface는 다음 위치입니다.
/sys/devices/system/memory/probe완전한 memory block만 probe할 수 있습니다. block의 physical 시작 주소 addr를 다음처럼 기록합니다.
% echo addr > /sys/devices/system/memory/probe그 결과 [addr, addr + memory_block_size) 범위의 memory block이 생성됩니다.
Linux는 user input을 검증할 수 없어 kernel crash를 일으키기 쉬우므로 probe interface 사용은 권장되지 않습니다. 이 interface는 앞으로 제거될 수 있습니다.
Online과 offline의 의미
136-150memory block을 만든 뒤 Linux가 실제로 사용하려면 block을 online해야 합니다. 반대로 제거하기 전에는 block에 속한 어떤 memory도 Linux가 사용하지 않도록 offline해야 합니다.
Kernel은 새 block을 자동 online하도록 구성할 수 있고, driver는 memory hotunplug를 시도할 때 자동으로 offline을 시작할 수 있습니다. memory block은 offlining에 성공한 뒤에만 제거할 수 있습니다.
수동 online
151-181auto-onlining이 꺼져 있으면 user space가 online을 직접 실행해야 하며, 보통 udev rule로 자동화합니다. 기본 state interface는 다음과 같습니다.
% echo online > /sys/devices/system/memory/memoryXXX/state단순 online interface에서는 1을 기록합니다.
% echo 1 > /sys/devices/system/memory/memoryXXX/onlinetarget zone을 지정하지 않으면 kernel이 설정된 online_policy에 따라 자동 선택합니다. ZONE_MOVABLE을 명시하려면 다음 명령을 사용합니다.
% echo online_movable > /sys/devices/system/memory/memoryXXX/state보통 ZONE_NORMAL인 kernel zone을 명시하려면 다음과 같이 합니다.
% echo online_kernel > /sys/devices/system/memory/memoryXXX/state성공하면 block state가 online으로 바뀝니다. 실패하면 상태는 그대로이며 명령도 실패로 끝납니다.
자동 online
182-213Kernel은 새로 추가한 memory block을 자동 online하도록 구성할 수 있습니다. 기능이 꺼져 있으면 user space가 명시적으로 online할 때까지 block은 offline에 머뭅니다. 현재 동작은 다음처럼 확인합니다.
% cat /sys/devices/system/memory/auto_online_blocksonline, online_kernel 또는 online_movable을 기록해 auto-onlining을 켤 수 있습니다.
% echo online > /sys/devices/system/memory/auto_online_blocksonline 값은 online_policy에 따라 target zone을 자동 선택합니다. 설정 변경은 이후에 추가되는 memory block에만 적용됩니다.
Corner case에서 auto-onlining이 실패할 수 있으며 kernel은 재시도하지 않습니다. 다만 기본 구성에서는 실패하지 않을 것으로 예상합니다.
ppc64의 DLPAR는 offline 설정을 무시하고 추가된 block을 계속 online합니다. online이 실패하면 그 block을 다시 제거합니다.
Memory block offline
214-246현재 구현은 대상 block에서 모든 movable page를 다른 곳으로 migrate합니다. page table을 포함한 대부분의 kernel allocation은 unmovable이므로 migration과 offlining이 실패할 수 있습니다.
block의 memory를 ZONE_MOVABLE에서 관리하면 성공 신뢰성이 크게 높아지지만 corner case 실패는 남습니다. offlining은 사용자가 중단할 때까지 매우 오래, 심하면 영원히 재시도할 수도 있습니다.
% echo offline > /sys/devices/system/memory/memoryXXX/state단순 interface에서는 0을 기록합니다.
% echo 0 > /sys/devices/system/memory/memoryXXX/online성공하면 state가 offline으로 바뀌고, 실패하면 상태가 유지되며 명령은 다음 오류 중 하나를 낼 수 있습니다.
bash: echo: write error: Device or resource busybash: echo: write error: Invalid argumentMemory block 상태 관찰
247-262memory block의 online, offline, going-offline 상태는 state에서 확인합니다.
% cat /sys/devices/system/memory/memoryXXX/state1 또는 0으로 보는 대체 interface는 online입니다.
% cat /sys/devices/system/memory/memoryXXX/onlineonline block을 관리하는 zone은 valid_zones로 확인합니다.
% cat /sys/devices/system/memory/memoryXXX/valid_zones전역 sysfs 구성
263-312System administrator는 여러 interface로 memory hot(un)plug를 구성하고 block, 특히 online 동작을 제어할 수 있습니다. 전역 속성은 다음 directory에 있습니다.
/sys/devices/system/memory/| 파일 | 접근 | 기능 |
|---|---|---|
| auto_online_blocks | read-write | 새 memory block의 기본 상태와 auto-onlining을 설정·조회. 기본값은 CONFIG_MHP_DEFAULT_ONLINE_TYPE에 따르며 자세한 값은 block의 state 속성과 동일 |
| block_size_bytes | read-only | memory block 크기를 byte 단위로 표시 |
| probe | write-only | physical 시작 주소를 받아 user space에서 block을 수동 probe. CONFIG_ARCH_MEMORY_PROBE에 의존 |
| uevent | read-write | device subsystem용 일반 udev 파일 |
| crash_hotplug | read-only | memory map 변경 시 kernel이 elfcorehdr와 관련 kexec segment를 포함한 kdump capture-kernel map을 갱신하면 1, user space가 갱신해야 하면 0. CONFIG_MEMORY_HOTPLUG에 의존 |
CONFIG_MEMORY_FAILURE가 켜져 있으면 test 목적의 page hwpoisoning을 실행하는 hard_offline_page와 soft_offline_page가 추가됩니다. 이 기능은 memory block의 실제 offlining이나 memory hot(un)plug 자체와 직접 관련되지는 않습니다.
Block별 sysfs 구성
313-395각 memory block은 online 또는 offline할 수 있는 block device로 표현되며 device 정보는 sysfs에 있습니다. 존재하는 block은 다음 경로에 나열됩니다.
/sys/devices/system/memory/memoryXXXXXX는 자릿수가 가변인 memory block id입니다. present block은 해당 범위에 일부 memory가 있음을 뜻하지만 memory hole을 포함할 수 있고, hole을 가로지르는 block은 offline할 수 없습니다.
예를 들어 block 크기가 1 GiB이고 시작 주소가 0x100000000이면 id는 다음 계산으로 4가 됩니다.
(0x100000000 / 1Gib = 4)따라서 /sys/devices/system/memory/memory4는 [0x100000000 ... 0x140000000) 주소 범위를 담당합니다.
| 파일 | 접근 | 기능 |
|---|---|---|
| online | read-write | online/offline을 실행하고 상태를 보는 단순 interface. online 시 zone은 자동 선택 |
| phys_device | read-only | s390x에서 covered storage increment를 노출하던 legacy interface |
| phys_index | read-only | memory block id인 XXX |
| removable | read-only | 과거 offline 가능성을 추정하던 legacy interface. 현재는 kernel이 memory offlining을 지원할 때만 1 |
| state | read-write | 고급 상태 interface. write: online, offline, online_kernel, online_movable. read: online, offline, going-offline |
| uevent | read-write | device용 일반 uevent 파일 |
| valid_zones | read-only | online block은 현재 zone, offline block은 지금 online할 때 선택될 zone을 표시. DMA, DMA32, Normal, Movable, none을 반환하며 CONFIG_MEMORY_HOTREMOVE에 의존 |
state에 online_movable을 쓰면 ZONE_MOVABLE, online_kernel을 쓰면 ZONE_NORMAL 같은 기본 kernel zone에 배치합니다. online은 자동 선택합니다. valid_zones의 Movable은 ZONE_MOVABLE이고 DMA, DMA32, Normal은 kernel zone입니다. none은 여러 zone이나 node에 걸쳐 있어 offline할 수 없음을 뜻합니다. offline block에서는 첫 column이 지금 zone을 지정하지 않고 online할 때 kernel이 고를 zone입니다.
CONFIG_NUMA가 켜져 있으면 memoryXXX directory를 /sys/devices/system/node/node* 아래 symbolic link로도 접근할 수 있습니다.
/sys/devices/system/node/node0/memory9 -> ../../memory/memory9memory block 쪽에는 NUMA node로 돌아가는 backlink도 생성됩니다.
/sys/devices/system/memory/memory9/node0 -> ../../node/node0Kernel command line
396-414memory hot(un)plug 처리에 영향을 주는 command line parameter는 다음 두 가지입니다.
| 매개변수 | 기능 |
|---|---|
| memhp_default_state | /sys/devices/system/memory/auto_online_blocks를 설정하는 방식으로 auto-onlining 구성 |
| movable_node | contig-zones policy에서 자동 zone 선택을 구성. 다른 zone의 연속성을 유지할 수 없으면 ZONE_MOVABLE을 기본 선택 |
더 일반적인 설명은 Documentation/admin-guide/kernel-parameters.txt를 참조합니다.
Module parameter
415-551memory_hotplug subsystem은 추가 command line option이나 sysfs file 대신 전용 module parameter namespace를 제공합니다. boot command line에서는 memory_hotplug. prefix를 붙입니다.
memory_hotplug.memmap_on_memory=1parameter는 다음 directory에서 확인하며 일부는 runtime에 바꿀 수 있습니다.
/sys/module/memory_hotplug/parameters/| 매개변수 | 접근 | 기능 |
|---|---|---|
| memmap_on_memory | read-write | 추가한 block 자체에서 memmap을 할당하도록 요청하는 hint. hotplug 실패 가능성과 NUMA 원격 배치를 줄이지만 큰 huge page를 만들 물리 메모리를 단편화할 수 있음. force는 pageblock 정렬 때문에 메모리를 낭비할 수 있으며 모든 구성에서 강제되지는 않음 |
| online_policy | read-write | target zone을 지정하지 않은 online의 기본 정책. contig-zones는 zone 연속성을 우선하고 movable_node에 따라 fallback하며, auto-movable은 장치 단위와 ratio를 고려해 가능한 block을 ZONE_MOVABLE로 배치 |
| auto_movable_ratio | read-write | auto-movable의 최대 MOVABLE:KERNEL 비율(%). 기본 301이며 present page, device별 회계, CMA를 MOVABLE로 계산. 서로 다른 장치의 ZONE_NORMAL은 다른 장치의 MOVABLE 허용량을 늘리지 않음 |
| auto_movable_numa_aware | read-write | auto_movable_ratio를 전체 system뿐 아니라 NUMA node별로도 적용할지 선택. 기본 Y이며 CONFIG_NUMA에 의존 |
memmap_on_memory를 사용하면 memmap이 같은 NUMA node에 남고 hotplug가 실패할 가능성이 낮아지지만, 예를 들어 memmap 1 MiB를 pageblock 2 MiB에 맞추면 1 MiB가 낭비됩니다. force라도 memmap이 한 page보다 작거나 architecture가 해당 구성을 지원하지 않으면 강제할 수 없습니다.
online_policy=contig-zones는 zone을 연속으로 유지하려 합니다. block이 여러 zone과 교차하거나 어느 zone에도 속하지 않으면 movable_node가 설정된 경우 ZONE_MOVABLE, 아니면 보통 ZONE_NORMAL인 kernel zone을 선택합니다. memory를 online한 뒤에는 policy를 바꾸지 않아야 합니다.
online_policy=auto-movable은 구성과 device 정보를 바탕으로 가능한 block을 ZONE_MOVABLE로 online하여 대량 hotplug 뒤의 zone imbalance를 피하면서 reliable hotunplug 범위를 넓히려는 virtualization용 정책입니다. movable_node는 무시하므로 firmware memory map으로 boot 초기에 hotpluggable node가 노출되는 bare-metal 구성에는 적합하지 않습니다.
hotplug DIMM은 전체가 ZONE_MOVABLE 또는 전체가 ZONE_NORMAL이 되며 둘을 섞지 않습니다. virtio-mem은 함께 hotunplug해야 하는 block unit을 특별 처리하면서 가능한 많은 block을 ZONE_MOVABLE에 둡니다. 이 정책은 ZONE_MOVABLE 자체가 문제인 구성을 보호하지 않으며, 한번 online한 block의 zone을 동적으로 바꾸지도 않습니다.
auto_movable_ratio는 auto-movable의 최대 MOVABLE:KERNEL 비율을 percent로 정합니다. 회계는 zone의 present page와 device별 수치를 결합하고 kernel zone 안의 CMA memory도 MOVABLE로 셉니다. 기본값 301%는 여러 구성에서 8 GiB VM에 24 GiB를 hotplug해 전부 ZONE_MOVABLE로 online할 수 있게 하며, 추가 1%는 firmware allocation 등으로 present가 아닌 page를 보정합니다.
한 memory device가 제공한 ZONE_NORMAL memory는 다른 device의 ZONE_MOVABLE 허용량을 늘리지 않습니다. 한 hotplug DIMM을 ZONE_NORMAL로 online해도 다른 DIMM의 자동 MOVABLE 배치를 허용하지 않습니다. 반면 virtio-mem device 내부에서 ZONE_NORMAL로 online된 memory는 같은 device의 ZONE_MOVABLE 허용량을 늘릴 수 있습니다.
auto_movable_numa_aware=Y가 기본이며 ratio를 system 전체뿐 아니라 NUMA node별로도 적용합니다. 완전히 hotunplug할 node를 다룰 때 이를 끄면 가능한 경우 node memory 전체를 ZONE_MOVABLE로 자동 online하는 데 도움이 됩니다. 이 parameter는 CONFIG_NUMA에 의존합니다.
ZONE_MOVABLE 기본 원리
552-570ZONE_MOVABLE은 memory offlining의 신뢰성을 높이는 핵심 mechanism입니다. System RAM을 kernel zone 대신 ZONE_MOVABLE에서 관리하면 만들 수 있는 transparent huge page와 동적 huge page 수가 늘어날 수도 있습니다.
memory map은 보통 전체 memory의 1/64을 차지하며 page table과 kmalloc()도 대표적인 unmovable kernel allocation입니다. 이들은 kernel zone에서만 제공할 수 있습니다. 반면 anonymous memory와 page cache 같은 대부분의 user-space page는 movable이어서 ZONE_MOVABLE과 kernel zone 양쪽에서 할당할 수 있습니다.
ZONE_MOVABLE은 movable allocation만 받아 unmovable allocation을 kernel zone에 제한합니다. ZONE_MOVABLE이 없으면 어떤 memory block의 offlining 성공도 보장할 수 없습니다.
Zone imbalance
571-590System RAM을 ZONE_MOVABLE에서 지나치게 많이 관리하는 상태를 zone imbalance라고 하며 system을 손상시키거나 성능을 낮출 수 있습니다. ZONE_MOVABLE에 free memory가 충분해도 unmovable allocation용 kernel-zone memory가 고갈되어 kernel이 crash할 수 있습니다.
일반적으로 MOVABLE:KERNEL 비율 3:1 또는 4:1까지는 괜찮지만 memory map overhead 때문에 63:1은 명백히 불가능합니다. 실제 안전 비율은 workload에 달려 있고, 과도한 long-term page pinning 같은 극단적 사례는 ZONE_MOVABLE을 전혀 감당하지 못할 수 있습니다.
kernel zone 안의 CMA memory는 본질적으로 ZONE_MOVABLE memory처럼 동작하므로, 특히 CMA와 ZONE_MOVABLE을 함께 쓸 때 같은 비율 검토가 필요합니다.
ZONE_MOVABLE 크기 산정
591-650보통 available System RAM의 큰 부분은 user space가 직접 사용하거나 page cache를 통해 간접 사용합니다. 이런 page는 일반적으로 ZONE_MOVABLE에서 문제없이 할당할 수 있으므로 상당한 RAM을 MOVABLE로 관리하는 것이 합리적입니다. 다만 zone ratio를 조정할 때 다음 항목을 검토해야 합니다.
| 검토 항목 | 영향 |
|---|---|
| 많은 offline block | offline 상태도 metadata와 direct-map page table을 소비하지만 일반적인 구성은 아님 |
| balloon compaction 없음 | ZONE_MOVABLE과 호환되지 않음. virtio-balloon과 pseries CMM 등 일부만 완전 지원하며 CONFIG_BALLOON_COMPACTION이 꺼지면 inflation이 unmovable allocation으로 zone imbalance를 만듦 |
| gigantic page | unmovable이므로 user space가 kernel-zone memory를 많이 소비 |
| huge page migration 없음 | architecture가 migration을 지원하지 않으면 huge page도 unmovable |
| page table | unmovable. 과도한 swapping, 매우 큰 file 또는 ZONE_DEVICE mapping은 corner case에서 kernel zone을 압박 |
| DAX | 일부 구성에서는 device memory의 memory map을 kernel zone에서 할당 |
| KASAN | tracking metadata가 total system memory의 1/8 같은 큰 unmovable overhead를 만들 수 있음 |
| 장기 page pinning | RDMA와 vfio/mdev는 ZONE_MOVABLE 및 offlining과 근본적으로 충돌. pin 전에 page를 zone 밖으로 migrate해야 하며 여유 MOVABLE memory가 있어도 실패하거나 비용이 증가할 수 있음 |
Long-term pinning은 pinned page를 ZONE_MOVABLE에 둘 수 없으므로 pin하는 동안 zone 밖으로 migrate해야 합니다. 그 결과 free MOVABLE memory가 충분해도 pin이 실패할 수 있고 migration overhead로 비용이 커집니다.
기본적으로 boot 때 구성된 모든 memory는 kernel zone에서 관리하며 ZONE_MOVABLE은 사용하지 않습니다. boot memory를 ZONE_MOVABLE에 포함하고 movable/kernel zone 비율을 제어하려면 kernelcore=와 movablecore= command line option을 사용합니다. 설명은 Documentation/admin-guide/kernel-parameters.rst를 참조합니다.
ZONE_MOVABLE에서도 남는 실패 조건
651-696ZONE_MOVABLE을 사용해도 memory block offlining이 실패하는 corner case가 남습니다.
| 조건 | 설명 |
|---|---|
| memory hole | boot memory block, XEN balloon 또는 Hyper-V balloon으로 hotplug한 block에 적용 가능 |
| 혼합 NUMA node·zone | 한 block에 여러 node 또는 zone이 섞이면 offline 불가. boot 당시 block에만 해당 |
| system 보호 block | arm64 boot memory, s390x crashkernel 영역을 가로지르는 block 등 |
| CMA overlap | CMA 영역과 겹치는 boot memory block은 offline 불가 |
| 동시 물리 메모리 작업 | gigantic page 할당처럼 같은 영역을 다루는 activity가 일시 실패를 유발 |
| huge page 해체 OOM | 특히 HVO 사용 시 source huge page의 vmemmap용 unmovable page를 kernel zone에서 할당하지 못할 수 있음 |
Offlining code가 huge page 내용은 migrate해도 source huge page를 해체하지 못할 수 있습니다. 특히 HugeTLB Vmemmap Optimization(HVO)이 켜진 상태에서 kernel zone에 vmemmap용 free unmovable page가 없으면 allocation이 실패합니다. movable zone의 offlining 성공에 의존하는 사용자는 HVO의 절감 이득과 offline 불가 위험을 함께 평가해야 합니다.
page migration 중 out-of-memory가 발생하거나 ZONE_MOVABLE 안에서 영구적인 unmovable page를 계속 발견하면 이는 BUG이며, memory offlining은 성공할 때까지 재시도합니다. User space에서 시작한 offlining context는 signal로 종료할 수 있고 timeout 기반 처리는 다음처럼 구현할 수 있습니다.
% timeout $TIMEOUT offline_block | failure_handling
운영 핵심
memory-hotplug.rst:1-696Memory hotplug는 물리 메모리를 Linux에 등록하는 add와 page allocator에 공개하는 online을 구분합니다. 제거 경로도 offline과 remove가 분리되며, 실제 운영에서는 block 단위와 zone 배치, unmovable allocation 때문에 생기는 실패 가능성을 함께 관리해야 합니다.