← Documents Documentation/admin-guide/cgroup-v1/hugetlb.rst GitHub 원문 ↗

Linux 6.18.37 · Administration / Cgroup v1

HugeTLB Controller

Hugepage-size별 fault·reservation limit, SIGBUS, shared charge와 offline zombie 동작을 설명합니다.

Source pathDocumentation/admin-guide/cgroup-v1/hugetlb.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

Control files

hugetlb.rst:1-65

Mount와 page-size별 accounting file을 정리합니다.

Fault and reservation

hugetlb.rst:66-113

Fault limit과 reservation limit의 차이를 설명합니다.

Shared and offline caveats

hugetlb.rst:114-139

Shared memory charge와 cgroup offline lifetime을 다룹니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 ==================
2 HugeTLB Controller
3 ==================
4
5 HugeTLB controller can be created by first mounting the cgroup filesystem.
6
7 # mount -t cgroup -o hugetlb none /sys/fs/cgroup
8
9 With the above step, the initial or the parent HugeTLB group becomes
10 visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in
11 the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.
12
13 New groups can be created under the parent group /sys/fs/cgroup::
14
15 # cd /sys/fs/cgroup
16 # mkdir g1
17 # echo $$ > g1/tasks
18
19 The above steps create a new group g1 and move the current shell
20 process (bash) into it.
21
22 Brief summary of control files::
23
24 hugetlb.<hugepagesize>.rsvd.limit_in_bytes # set/show limit of "hugepagesize" hugetlb reservations
25 hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes # show max "hugepagesize" hugetlb reservations and no-reserve faults
26 hugetlb.<hugepagesize>.rsvd.usage_in_bytes # show current reservations and no-reserve faults for "hugepagesize" hugetlb
27 hugetlb.<hugepagesize>.rsvd.failcnt # show the number of allocation failure due to HugeTLB reservation limit
28 hugetlb.<hugepagesize>.limit_in_bytes # set/show limit of "hugepagesize" hugetlb faults
29 hugetlb.<hugepagesize>.max_usage_in_bytes # show max "hugepagesize" hugetlb usage recorded
30 hugetlb.<hugepagesize>.usage_in_bytes # show current usage for "hugepagesize" hugetlb
31 hugetlb.<hugepagesize>.failcnt # show the number of allocation failure due to HugeTLB usage limit
32 hugetlb.<hugepagesize>.numa_stat # show the numa information of the hugetlb memory charged to this cgroup
33
34 For a system supporting three hugepage sizes (64k, 32M and 1G), the control
35 files include::
36
37 hugetlb.1GB.limit_in_bytes
38 hugetlb.1GB.max_usage_in_bytes
39 hugetlb.1GB.numa_stat
40 hugetlb.1GB.usage_in_bytes
41 hugetlb.1GB.failcnt
42 hugetlb.1GB.rsvd.limit_in_bytes
43 hugetlb.1GB.rsvd.max_usage_in_bytes
44 hugetlb.1GB.rsvd.usage_in_bytes
45 hugetlb.1GB.rsvd.failcnt
46 hugetlb.64KB.limit_in_bytes
47 hugetlb.64KB.max_usage_in_bytes
48 hugetlb.64KB.numa_stat
49 hugetlb.64KB.usage_in_bytes
50 hugetlb.64KB.failcnt
51 hugetlb.64KB.rsvd.limit_in_bytes
52 hugetlb.64KB.rsvd.max_usage_in_bytes
53 hugetlb.64KB.rsvd.usage_in_bytes
54 hugetlb.64KB.rsvd.failcnt
55 hugetlb.32MB.limit_in_bytes
56 hugetlb.32MB.max_usage_in_bytes
57 hugetlb.32MB.numa_stat
58 hugetlb.32MB.usage_in_bytes
59 hugetlb.32MB.failcnt
60 hugetlb.32MB.rsvd.limit_in_bytes
61 hugetlb.32MB.rsvd.max_usage_in_bytes
62 hugetlb.32MB.rsvd.usage_in_bytes
63 hugetlb.32MB.rsvd.failcnt
64
65
66 1. Page fault accounting
67
68 ::
69
70 hugetlb.<hugepagesize>.limit_in_bytes
71 hugetlb.<hugepagesize>.max_usage_in_bytes
72 hugetlb.<hugepagesize>.usage_in_bytes
73 hugetlb.<hugepagesize>.failcnt
74
75 The HugeTLB controller allows users to limit the HugeTLB usage (page fault) per
76 control group and enforces the limit during page fault. Since HugeTLB
77 doesn't support page reclaim, enforcing the limit at page fault time implies
78 that, the application will get SIGBUS signal if it tries to fault in HugeTLB
79 pages beyond its limit. Therefore the application needs to know exactly how many
80 HugeTLB pages it uses before hand, and the sysadmin needs to make sure that
81 there are enough available on the machine for all the users to avoid processes
82 getting SIGBUS.
83
84
85 2. Reservation accounting
86
87 ::
88
89 hugetlb.<hugepagesize>.rsvd.limit_in_bytes
90 hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes
91 hugetlb.<hugepagesize>.rsvd.usage_in_bytes
92 hugetlb.<hugepagesize>.rsvd.failcnt
93
94 The HugeTLB controller allows to limit the HugeTLB reservations per control
95 group and enforces the controller limit at reservation time and at the fault of
96 HugeTLB memory for which no reservation exists. Since reservation limits are
97 enforced at reservation time (on mmap or shget), reservation limits never causes
98 the application to get SIGBUS signal if the memory was reserved before hand. For
99 MAP_NORESERVE allocations, the reservation limit behaves the same as the fault
100 limit, enforcing memory usage at fault time and causing the application to
101 receive a SIGBUS if it's crossing its limit.
102
103 Reservation limits are superior to page fault limits described above, since
104 reservation limits are enforced at reservation time (on mmap or shget), and
105 never causes the application to get SIGBUS signal if the memory was reserved
106 before hand. This allows for easier fallback to alternatives such as
107 non-HugeTLB memory for example. In the case of page fault accounting, it's very
108 hard to avoid processes getting SIGBUS since the sysadmin needs precisely know
109 the HugeTLB usage of all the tasks in the system and make sure there is enough
110 pages to satisfy all requests. Avoiding tasks getting SIGBUS on overcommited
111 systems is practically impossible with page fault accounting.
112
113
114 3. Caveats with shared memory
115
116 For shared HugeTLB memory, both HugeTLB reservation and page faults are charged
117 to the first task that causes the memory to be reserved or faulted, and all
118 subsequent uses of this reserved or faulted memory is done without charging.
119
120 Shared HugeTLB memory is only uncharged when it is unreserved or deallocated.
121 This is usually when the HugeTLB file is deleted, and not when the task that
122 caused the reservation or fault has exited.
123
124
125 4. Caveats with HugeTLB cgroup offline.
126
127 When a HugeTLB cgroup goes offline with some reservations or faults still
128 charged to it, the behavior is as follows:
129
130 - The fault charges are charged to the parent HugeTLB cgroup (reparented),
131 - the reservation charges remain on the offline HugeTLB cgroup.
132
133 This means that if a HugeTLB cgroup gets offlined while there is still HugeTLB
134 reservations charged to it, that cgroup persists as a zombie until all HugeTLB
135 reservations are uncharged. HugeTLB reservations behave in this manner to match
136 the memory controller whose cgroups also persist as zombie until all charged
137 memory is uncharged. Also, the tracking of HugeTLB reservations is a bit more
138 complex compared to the tracking of HugeTLB faults, so it is significantly
139 harder to reparent reservations at offline time.
140

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

Mount와 page-size별 control files

1-65

HugeTLB controller는 cgroup filesystem을 `hugetlb` option으로 mount해 만듭니다. Initial parent group은 `/sys/fs/cgroup`에 나타나고 boot 시 system의 모든 task를 포함하며 `/sys/fs/cgroup/tasks`가 그 목록을 보여 줍니다.

HugeTLB controller can be created by first mounting the cgroup filesystem.

# mount -t cgroup -o hugetlb none /sys/fs/cgroup

With the above step, the initial or the parent HugeTLB group becomes
visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in
the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.

New groups can be created under the parent group /sys/fs/cgroup::

  # cd /sys/fs/cgroup
  # mkdir g1
  # echo $$ > g1/tasks

Parent 아래에 `g1` directory를 만들고 현재 shell PID를 `g1/tasks`에 쓰면 Bash가 새 HugeTLB group으로 이동합니다.

HugeTLB control-file families
File suffixPurpose
rsvd.limit_in_bytesReservation limit 설정·조회
rsvd.max_usage_in_bytesReservation과 no-reserve fault 최대치
rsvd.usage_in_bytesCurrent reservation과 no-reserve fault
rsvd.failcntReservation-limit allocation failure 수
limit_in_bytesFault-time HugeTLB usage limit
max_usage_in_bytesRecorded maximum usage
usage_in_bytesCurrent usage
failcntUsage-limit allocation failure 수
numa_statCgroup에 charge된 HugeTLB memory의 NUMA 정보

`<hugepagesize>`마다 reservation과 fault/usage accounting file이 생성됩니다.

Brief summary of control files::

 hugetlb.<hugepagesize>.rsvd.limit_in_bytes            # set/show limit of "hugepagesize" hugetlb reservations
 hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes        # show max "hugepagesize" hugetlb reservations and no-reserve faults
 hugetlb.<hugepagesize>.rsvd.usage_in_bytes            # show current reservations and no-reserve faults for "hugepagesize" hugetlb
 hugetlb.<hugepagesize>.rsvd.failcnt                   # show the number of allocation failure due to HugeTLB reservation limit
 hugetlb.<hugepagesize>.limit_in_bytes                 # set/show limit of "hugepagesize" hugetlb faults
 hugetlb.<hugepagesize>.max_usage_in_bytes             # show max "hugepagesize" hugetlb  usage recorded
 hugetlb.<hugepagesize>.usage_in_bytes                 # show current usage for "hugepagesize" hugetlb
 hugetlb.<hugepagesize>.failcnt                        # show the number of allocation failure due to HugeTLB usage limit
 hugetlb.<hugepagesize>.numa_stat                      # show the numa information of the hugetlb memory charged to this cgroup
Per-size interface expansion
64KB pageshugetlb.64KB.* files
32MB pageshugetlb.32MB.* files
1GB pageshugetlb.1GB.* files

지원하는 hugepage size마다 같은 file family가 독립적으로 생깁니다.

files include::

  hugetlb.1GB.limit_in_bytes
  hugetlb.1GB.max_usage_in_bytes
  hugetlb.1GB.numa_stat
  hugetlb.1GB.usage_in_bytes
  hugetlb.1GB.failcnt
  hugetlb.1GB.rsvd.limit_in_bytes
  hugetlb.1GB.rsvd.max_usage_in_bytes
  hugetlb.1GB.rsvd.usage_in_bytes
  hugetlb.1GB.rsvd.failcnt
  hugetlb.64KB.limit_in_bytes
  hugetlb.64KB.max_usage_in_bytes
  hugetlb.64KB.numa_stat
  hugetlb.64KB.usage_in_bytes
  hugetlb.64KB.failcnt
  hugetlb.64KB.rsvd.limit_in_bytes
  hugetlb.64KB.rsvd.max_usage_in_bytes
  hugetlb.64KB.rsvd.usage_in_bytes
  hugetlb.64KB.rsvd.failcnt
  hugetlb.32MB.limit_in_bytes
  hugetlb.32MB.max_usage_in_bytes
  hugetlb.32MB.numa_stat
  hugetlb.32MB.usage_in_bytes
  hugetlb.32MB.failcnt
  hugetlb.32MB.rsvd.limit_in_bytes
  hugetlb.32MB.rsvd.max_usage_in_bytes
  hugetlb.32MB.rsvd.usage_in_bytes
  hugetlb.32MB.rsvd.failcnt

Page-fault accounting

66-84 
1. Page fault accounting

::

  hugetlb.<hugepagesize>.limit_in_bytes
  hugetlb.<hugepagesize>.max_usage_in_bytes
  hugetlb.<hugepagesize>.usage_in_bytes
  hugetlb.<hugepagesize>.failcnt

Page-fault accounting은 control group별 HugeTLB usage를 제한하고 page fault 시 limit을 강제합니다. HugeTLB는 page reclaim을 지원하지 않으므로 limit을 넘는 page를 fault하려는 application은 `SIGBUS`를 받습니다.

Fault-limit enforcement
Application faults HugeTLB pageCharge against cgroup usageWould exceed limit_in_bytes?Allow and update usage or deliver SIGBUS

Reclaim 대신 fault를 실패시키므로 사전 capacity 계획이 필요합니다.

Application은 필요한 HugeTLB page 수를 미리 정확히 알아야 하고 sysadmin은 모든 user request를 만족할 page가 machine에 충분한지 보장해야 process가 SIGBUS를 받지 않습니다.

Reservation accounting과 MAP_NORESERVE

85-113 
2. Reservation accounting

::

  hugetlb.<hugepagesize>.rsvd.limit_in_bytes
  hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes
  hugetlb.<hugepagesize>.rsvd.usage_in_bytes
  hugetlb.<hugepagesize>.rsvd.failcnt

Reservation accounting은 control group별 HugeTLB reservation을 제한하며 reservation 시점과 reservation이 없는 HugeTLB memory fault 시 limit을 강제합니다. Reservation은 `mmap` 또는 `shget` 때 검사하므로 memory를 미리 reserve했다면 application은 이후 SIGBUS를 받지 않습니다.

Fault limit versus reservation limit
ModeEnforcement timeOver-limit outcome
Page-fault accountingFault timeSIGBUS; reclaim unavailable
Reserved mappingmmap or shget reservation timeReservation fails before use
MAP_NORESERVEFault timeFault-limit처럼 SIGBUS 가능

Limit을 검사하는 시점과 application fallback 가능성이 다릅니다.

Reservation-first allocation
Request reservation at mmap/shgetCheck rsvd.limit_in_bytesReservation succeedsLater HugeTLB fault is coveredNo limit-caused SIGBUS
Reservation rejectedApplication chooses fallbackFor example non-HugeTLB memory

사전 reservation은 HugeTLB가 부족할 때 non-HugeTLB memory 같은 대안을 선택할 기회를 줍니다.

`MAP_NORESERVE` allocation은 fault limit과 같은 방식으로 fault 시 usage를 검사해 limit을 넘으면 SIGBUS를 보냅니다. Overcommitted system에서 모든 task의 정확한 usage를 알아야 하는 page-fault accounting보다 reservation limit이 관리하기 쉽습니다.

Shared HugeTLB memory charge

114-124

Shared HugeTLB memory의 reservation과 page fault는 memory를 처음 reserve하거나 fault한 task에 charge됩니다. 이후 같은 reserved/faulted memory를 사용하는 task에는 추가 charge하지 않습니다.

Shared-memory charge lifetime
First task reserves or faults shared HugeTLB memoryCharge first task cgroupSubsequent users share without chargeHugeTLB file deleted or memory unreserved/deallocatedUncharge

최초 task 종료가 아니라 HugeTLB object 수명에 charge가 연결됩니다.

Shared HugeTLB memory는 unreserve되거나 deallocate될 때만 uncharge되며, 보통 HugeTLB file이 삭제되는 시점입니다. 최초 reservation/fault를 일으킨 task가 exit하는 시점은 아닙니다.

Cgroup offline과 zombie reservation

125-139

Reservation이나 fault charge가 남은 HugeTLB cgroup이 offline되면 fault charge는 parent HugeTLB cgroup으로 reparent되지만 reservation charge는 offline cgroup에 남습니다.

Offline charge behavior
ChargeOffline actionLifetime
Fault chargeReparent to parent HugeTLB cgroupParent tracks it
Reservation chargeRemain on offline cgroupUntil all reservations uncharged

Charge 유형별 offline 처리입니다.

Zombie HugeTLB cgroup
Cgroup goes offline with reservationsKeep reservation charges localCgroup persists as zombieHugeTLB files unreserved or deallocatedAll charges removedCgroup can disappear

남은 reservation 때문에 offline cgroup object가 계속 존재합니다.

이 behavior는 charged memory가 모두 uncharge될 때까지 zombie로 남는 memory controller와 일치합니다. HugeTLB reservation tracking은 fault tracking보다 복잡해 offline 시 reparent하기가 훨씬 어렵습니다.