요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
Core interface files and CPU controller
cgroup-v2.rst:743-1090Interface format convention, cgroup core file과 lifecycle control, CPU accounting·weight·bandwidth·pressure·utilization clamp·idle control을 설명합니다.
Memory controller
cgroup-v2.rst:1091-1956Memory protection·throttle·hard limit·reclaim·OOM, detailed statistics와 NUMA placement, swap·zswap·pressure, usage guideline과 charge ownership을 다룹니다.
IO controller
cgroup-v2.rst:1957-2353IO statistics, IOCOST model·QoS, weight·absolute limit·pressure, page-cache writeback attribution, latency protection과 I/O priority policy를 설명합니다.
PID and Cpuset controllers
cgroup-v2.rst:2354-2686Process-number hard limit와 events, hierarchical CPU·memory-node placement, exclusive CPU allocation과 local·remote·isolated partition state를 다룹니다.
Device, RDMA, DMEM, HugeTLB and Misc controllers
cgroup-v2.rst:2687-2915Cgroup BPF device access, RDMA와 device-memory accounting, HugeTLB limit·NUMA statistic, Misc scalar resource registration·limit·ownership을 설명합니다.
Other behavior and cgroup namespaces
cgroup-v2.rst:2916-3103Perf-event와 root process의 non-normative behavior, cgroup namespace root·path virtualization·migration·setns와 namespace-private mount를 설명합니다.
Kernel programming and v2 rationale
cgroup-v2.rst:3104-3407Filesystem writeback annotation API, 제거된 v1 core feature, multiple hierarchy·thread granularity·inner-node competition·interface 문제와 memory controller 개선 rationale를 정리합니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. _cgroup-v2:
================
Control Group v2
================
:Date: October, 2015
:Author: Tejun Heo <tj@kernel.org>
This is the authoritative documentation on the design, interface and
conventions of cgroup v2. It describes all userland-visible aspects
of cgroup including core and specific controller behaviors. All
future changes must be reflected in this document. Documentation for
v1 is available under :ref:`Documentation/admin-guide/cgroup-v1/index.rst <cgroup-v1>`.
.. CONTENTS
[Whenever any new section is added to this document, please also add
an entry here.]
1. Introduction
1-1. Terminology
1-2. What is cgroup?
2. Basic Operations
2-1. Mounting
2-2. Organizing Processes and Threads
2-2-1. Processes
2-2-2. Threads
2-3. [Un]populated Notification
2-4. Controlling Controllers
2-4-1. Availability
2-4-2. Enabling and Disabling
2-4-3. Top-down Constraint
2-4-4. No Internal Process Constraint
2-5. Delegation
2-5-1. Model of Delegation
2-5-2. Delegation Containment
2-6. Guidelines
2-6-1. Organize Once and Control
2-6-2. Avoid Name Collisions
3. Resource Distribution Models
3-1. Weights
3-2. Limits
3-3. Protections
3-4. Allocations
4. Interface Files
4-1. Format
4-2. Conventions
4-3. Core Interface Files
5. Controllers
5-1. CPU
5-1-1. CPU Interface Files
5-2. Memory
5-2-1. Memory Interface Files
5-2-2. Usage Guidelines
5-2-3. Memory Ownership
5-3. IO
5-3-1. IO Interface Files
5-3-2. Writeback
5-3-3. IO Latency
5-3-3-1. How IO Latency Throttling Works
5-3-3-2. IO Latency Interface Files
5-3-4. IO Priority
5-4. PID
5-4-1. PID Interface Files
5-5. Cpuset
5.5-1. Cpuset Interface Files
5-6. Device controller
5-7. RDMA
5-7-1. RDMA Interface Files
5-8. DMEM
5-8-1. DMEM Interface Files
5-9. HugeTLB
5.9-1. HugeTLB Interface Files
5-10. Misc
5.10-1 Misc Interface Files
5.10-2 Migration and Ownership
5-11. Others
5-11-1. perf_event
5-N. Non-normative information
5-N-1. CPU controller root cgroup process behaviour
5-N-2. IO controller root cgroup process behaviour
6. Namespace
6-1. Basics
6-2. The Root and Views
6-3. Migration and setns(2)
6-4. Interaction with Other Namespaces
P. Information on Kernel Programming
P-1. Filesystem Support for Writeback
D. Deprecated v1 Core Features
R. Issues with v1 and Rationales for v2
R-1. Multiple Hierarchies
R-2. Thread Granularity
R-3. Competition Between Inner Nodes and Threads
R-4. Other Interface Issues
R-5. Controller Issues and Remedies
R-5-1. Memory
Introduction
============
Terminology
-----------
"cgroup" stands for "control group" and is never capitalized. The
singular form is used to designate the whole feature and also as a
qualifier as in "cgroup controllers". When explicitly referring to
multiple individual control groups, the plural form "cgroups" is used.
What is cgroup?
---------------
cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.
cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.
cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.
Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
Basic Operations
================
Mounting
--------
Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2
hierarchy can be mounted with the following mount command::
# mount -t cgroup2 none $MOUNT_POINT
cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All
controllers which support v2 and are not bound to a v1 hierarchy are
automatically bound to the v2 hierarchy and show up at the root.
Controllers which are not in active use in the v2 hierarchy can be
bound to other hierarchies. This allows mixing v2 hierarchy with the
legacy v1 multiple hierarchies in a fully backward compatible way.
A controller can be moved across hierarchies only after the controller
is no longer referenced in its current hierarchy. Because per-cgroup
controller states are destroyed asynchronously and controllers may
have lingering references, a controller may not show up immediately on
the v2 hierarchy after the final umount of the previous hierarchy.
Similarly, a controller should be fully disabled to be moved out of
the unified hierarchy and it may take some time for the disabled
controller to become available for other hierarchies; furthermore, due
to inter-controller dependencies, other controllers may need to be
disabled too.
While useful for development and manual configurations, moving
controllers dynamically between the v2 and other hierarchies is
strongly discouraged for production use. It is recommended to decide
the hierarchies and controller associations before starting using the
controllers after system boot.
During transition to v2, system management software might still
automount the v1 cgroup filesystem and so hijack all controllers
during boot, before manual intervention is possible. To make testing
and experimenting easier, the kernel parameter cgroup_no_v1= allows
disabling controllers in v1 and make them always available in v2.
cgroup v2 currently supports the following mount options.
nsdelegate
Consider cgroup namespaces as delegation boundaries. This
option is system wide and can only be set on mount or modified
through remount from the init namespace. The mount option is
ignored on non-init namespace mounts. Please refer to the
Delegation section for details.
favordynmods
Reduce the latencies of dynamic cgroup modifications such as
task migrations and controller on/offs at the cost of making
hot path operations such as forks and exits more expensive.
The static usage pattern of creating a cgroup, enabling
controllers, and then seeding it with CLONE_INTO_CGROUP is
not affected by this option.
memory_localevents
Only populate memory.events with data for the current cgroup,
and not any subtrees. This is legacy behaviour, the default
behaviour without this option is to include subtree counts.
This option is system wide and can only be set on mount or
modified through remount from the init namespace. The mount
option is ignored on non-init namespace mounts.
memory_recursiveprot
Recursively apply memory.min and memory.low protection to
entire subtrees, without requiring explicit downward
propagation into leaf cgroups. This allows protecting entire
subtrees from one another, while retaining free competition
within those subtrees. This should have been the default
behavior but is a mount-option to avoid regressing setups
relying on the original semantics (e.g. specifying bogusly
high 'bypass' protection values at higher tree levels).
memory_hugetlb_accounting
Count HugeTLB memory usage towards the cgroup's overall
memory usage for the memory controller (for the purpose of
statistics reporting and memory protetion). This is a new
behavior that could regress existing setups, so it must be
explicitly opted in with this mount option.
A few caveats to keep in mind:
* There is no HugeTLB pool management involved in the memory
controller. The pre-allocated pool does not belong to anyone.
Specifically, when a new HugeTLB folio is allocated to
the pool, it is not accounted for from the perspective of the
memory controller. It is only charged to a cgroup when it is
actually used (for e.g at page fault time). Host memory
overcommit management has to consider this when configuring
hard limits. In general, HugeTLB pool management should be
done via other mechanisms (such as the HugeTLB controller).
* Failure to charge a HugeTLB folio to the memory controller
results in SIGBUS. This could happen even if the HugeTLB pool
still has pages available (but the cgroup limit is hit and
reclaim attempt fails).
* Charging HugeTLB memory towards the memory controller affects
memory protection and reclaim dynamics. Any userspace tuning
(of low, min limits for e.g) needs to take this into account.
* HugeTLB pages utilized while this option is not selected
will not be tracked by the memory controller (even if cgroup
v2 is remounted later on).
pids_localevents
The option restores v1-like behavior of pids.events:max, that is only
local (inside cgroup proper) fork failures are counted. Without this
option pids.events.max represents any pids.max enforcemnt across
cgroup's subtree.
Organizing Processes and Threads
--------------------------------
Processes
~~~~~~~~~
Initially, only the root cgroup exists to which all processes belong.
A child cgroup can be created by creating a sub-directory::
# mkdir $CGROUP_NAME
A given cgroup may have multiple child cgroups forming a tree
structure. Each cgroup has a read-writable interface file
"cgroup.procs". When read, it lists the PIDs of all processes which
belong to the cgroup one-per-line. The PIDs are not ordered and the
same PID may show up more than once if the process got moved to
another cgroup and then back or the PID got recycled while reading.
A process can be migrated into a cgroup by writing its PID to the
target cgroup's "cgroup.procs" file. Only one process can be migrated
on a single write(2) call. If a process is composed of multiple
threads, writing the PID of any thread migrates all threads of the
process.
When a process forks a child process, the new process is born into the
cgroup that the forking process belongs to at the time of the
operation. After exit, a process stays associated with the cgroup
that it belonged to at the time of exit until it's reaped; however, a
zombie process does not appear in "cgroup.procs" and thus can't be
moved to another cgroup.
A cgroup which doesn't have any children or live processes can be
destroyed by removing the directory. Note that a cgroup which doesn't
have any children and is associated only with zombie processes is
considered empty and can be removed::
# rmdir $CGROUP_NAME
"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy
cgroup is in use in the system, this file may contain multiple lines,
one for each hierarchy. The entry for cgroup v2 is always in the
format "0::$PATH"::
# cat /proc/842/cgroup
...
0::/test-cgroup/test-cgroup-nested
If the process becomes a zombie and the cgroup it was associated with
is removed subsequently, " (deleted)" is appended to the path::
# cat /proc/842/cgroup
...
0::/test-cgroup/test-cgroup-nested (deleted)
Threads
~~~~~~~
cgroup v2 supports thread granularity for a subset of controllers to
support use cases requiring hierarchical resource distribution across
the threads of a group of processes. By default, all threads of a
process belong to the same cgroup, which also serves as the resource
domain to host resource consumptions which are not specific to a
process or thread. The thread mode allows threads to be spread across
a subtree while still maintaining the common resource domain for them.
Controllers which support thread mode are called threaded controllers.
The ones which don't are called domain controllers.
Marking a cgroup threaded makes it join the resource domain of its
parent as a threaded cgroup. The parent may be another threaded
cgroup whose resource domain is further up in the hierarchy. The root
of a threaded subtree, that is, the nearest ancestor which is not
threaded, is called threaded domain or thread root interchangeably and
serves as the resource domain for the entire subtree.
Inside a threaded subtree, threads of a process can be put in
different cgroups and are not subject to the no internal process
constraint - threaded controllers can be enabled on non-leaf cgroups
whether they have threads in them or not.
As the threaded domain cgroup hosts all the domain resource
consumptions of the subtree, it is considered to have internal
resource consumptions whether there are processes in it or not and
can't have populated child cgroups which aren't threaded. Because the
root cgroup is not subject to no internal process constraint, it can
serve both as a threaded domain and a parent to domain cgroups.
The current operation mode or type of the cgroup is shown in the
"cgroup.type" file which indicates whether the cgroup is a normal
domain, a domain which is serving as the domain of a threaded subtree,
or a threaded cgroup.
On creation, a cgroup is always a domain cgroup and can be made
threaded by writing "threaded" to the "cgroup.type" file. The
operation is single direction::
# echo threaded > cgroup.type
Once threaded, the cgroup can't be made a domain again. To enable the
thread mode, the following conditions must be met.
- As the cgroup will join the parent's resource domain. The parent
must either be a valid (threaded) domain or a threaded cgroup.
- When the parent is an unthreaded domain, it must not have any domain
controllers enabled or populated domain children. The root is
exempt from this requirement.
Topology-wise, a cgroup can be in an invalid state. Please consider
the following topology::
A (threaded domain) - B (threaded) - C (domain, just created)
C is created as a domain but isn't connected to a parent which can
host child domains. C can't be used until it is turned into a
threaded cgroup. "cgroup.type" file will report "domain (invalid)" in
these cases. Operations which fail due to invalid topology use
EOPNOTSUPP as the errno.
A domain cgroup is turned into a threaded domain when one of its child
cgroup becomes threaded or threaded controllers are enabled in the
"cgroup.subtree_control" file while there are processes in the cgroup.
A threaded domain reverts to a normal domain when the conditions
clear.
When read, "cgroup.threads" contains the list of the thread IDs of all
threads in the cgroup. Except that the operations are per-thread
instead of per-process, "cgroup.threads" has the same format and
behaves the same way as "cgroup.procs". While "cgroup.threads" can be
written to in any cgroup, as it can only move threads inside the same
threaded domain, its operations are confined inside each threaded
subtree.
The threaded domain cgroup serves as the resource domain for the whole
subtree, and, while the threads can be scattered across the subtree,
all the processes are considered to be in the threaded domain cgroup.
"cgroup.procs" in a threaded domain cgroup contains the PIDs of all
processes in the subtree and is not readable in the subtree proper.
However, "cgroup.procs" can be written to from anywhere in the subtree
to migrate all threads of the matching process to the cgroup.
Only threaded controllers can be enabled in a threaded subtree. When
a threaded controller is enabled inside a threaded subtree, it only
accounts for and controls resource consumptions associated with the
threads in the cgroup and its descendants. All consumptions which
aren't tied to a specific thread belong to the threaded domain cgroup.
Because a threaded subtree is exempt from no internal process
constraint, a threaded controller must be able to handle competition
between threads in a non-leaf cgroup and its child cgroups. Each
threaded controller defines how such competitions are handled.
Currently, the following controllers are threaded and can be enabled
in a threaded cgroup::
- cpu
- cpuset
- perf_event
- pids
[Un]populated Notification
--------------------------
Each non-root cgroup has a "cgroup.events" file which contains
"populated" field indicating whether the cgroup's sub-hierarchy has
live processes in it. Its value is 0 if there is no live process in
the cgroup and its descendants; otherwise, 1. poll and [id]notify
events are triggered when the value changes. This can be used, for
example, to start a clean-up operation after all processes of a given
sub-hierarchy have exited. The populated state updates and
notifications are recursive. Consider the following sub-hierarchy
where the numbers in the parentheses represent the numbers of processes
in each cgroup::
A(4) - B(0) - C(1)
\ D(0)
A, B and C's "populated" fields would be 1 while D's 0. After the one
process in C exits, B and C's "populated" fields would flip to "0" and
file modified events will be generated on the "cgroup.events" files of
both cgroups.
Controlling Controllers
-----------------------
Availability
~~~~~~~~~~~~
A controller is available in a cgroup when it is supported by the kernel (i.e.,
compiled in, not disabled and not attached to a v1 hierarchy) and listed in the
"cgroup.controllers" file. Availability means the controller's interface files
are exposed in the cgroup’s directory, allowing the distribution of the target
resource to be observed or controlled within that cgroup.
Enabling and Disabling
~~~~~~~~~~~~~~~~~~~~~~
Each cgroup has a "cgroup.controllers" file which lists all
controllers available for the cgroup to enable::
# cat cgroup.controllers
cpu io memory
No controller is enabled by default. Controllers can be enabled and
disabled by writing to the "cgroup.subtree_control" file::
# echo "+cpu +memory -io" > cgroup.subtree_control
Only controllers which are listed in "cgroup.controllers" can be
enabled. When multiple operations are specified as above, either they
all succeed or fail. If multiple operations on the same controller
are specified, the last one is effective.
Enabling a controller in a cgroup indicates that the distribution of
the target resource across its immediate children will be controlled.
Consider the following sub-hierarchy. The enabled controllers are
listed in parentheses::
A(cpu,memory) - B(memory) - C()
\ D()
As A has "cpu" and "memory" enabled, A will control the distribution
of CPU cycles and memory to its children, in this case, B. As B has
"memory" enabled but not "CPU", C and D will compete freely on CPU
cycles but their division of memory available to B will be controlled.
As a controller regulates the distribution of the target resource to
the cgroup's children, enabling it creates the controller's interface
files in the child cgroups. In the above example, enabling "cpu" on B
would create the "cpu." prefixed controller interface files in C and
D. Likewise, disabling "memory" from B would remove the "memory."
prefixed controller interface files from C and D. This means that the
controller interface files - anything which doesn't start with
"cgroup." are owned by the parent rather than the cgroup itself.
Top-down Constraint
~~~~~~~~~~~~~~~~~~~
Resources are distributed top-down and a cgroup can further distribute
a resource only if the resource has been distributed to it from the
parent. This means that all non-root "cgroup.subtree_control" files
can only contain controllers which are enabled in the parent's
"cgroup.subtree_control" file. A controller can be enabled only if
the parent has the controller enabled and a controller can't be
disabled if one or more children have it enabled.
No Internal Process Constraint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Non-root cgroups can distribute domain resources to their children
only when they don't have any processes of their own. In other words,
only domain cgroups which don't contain any processes can have domain
controllers enabled in their "cgroup.subtree_control" files.
This guarantees that, when a domain controller is looking at the part
of the hierarchy which has it enabled, processes are always only on
the leaves. This rules out situations where child cgroups compete
against internal processes of the parent.
The root cgroup is exempt from this restriction. Root contains
processes and anonymous resource consumption which can't be associated
with any other cgroups and requires special treatment from most
controllers. How resource consumption in the root cgroup is governed
is up to each controller (for more information on this topic please
refer to the Non-normative information section in the Controllers
chapter).
Note that the restriction doesn't get in the way if there is no
enabled controller in the cgroup's "cgroup.subtree_control". This is
important as otherwise it wouldn't be possible to create children of a
populated cgroup. To control resource distribution of a cgroup, the
cgroup must create children and transfer all its processes to the
children before enabling controllers in its "cgroup.subtree_control"
file.
Delegation
----------
Model of Delegation
~~~~~~~~~~~~~~~~~~~
A cgroup can be delegated in two ways. First, to a less privileged
user by granting write access of the directory and its "cgroup.procs",
"cgroup.threads" and "cgroup.subtree_control" files to the user.
Second, if the "nsdelegate" mount option is set, automatically to a
cgroup namespace on namespace creation.
Because the resource control interface files in a given directory
control the distribution of the parent's resources, the delegatee
shouldn't be allowed to write to them. For the first method, this is
achieved by not granting access to these files. For the second, files
outside the namespace should be hidden from the delegatee by the means
of at least mount namespacing, and the kernel rejects writes to all
files on a namespace root from inside the cgroup namespace, except for
those files listed in "/sys/kernel/cgroup/delegate" (including
"cgroup.procs", "cgroup.threads", "cgroup.subtree_control", etc.).
The end results are equivalent for both delegation types. Once
delegated, the user can build sub-hierarchy under the directory,
organize processes inside it as it sees fit and further distribute the
resources it received from the parent. The limits and other settings
of all resource controllers are hierarchical and regardless of what
happens in the delegated sub-hierarchy, nothing can escape the
resource restrictions imposed by the parent.
Currently, cgroup doesn't impose any restrictions on the number of
cgroups in or nesting depth of a delegated sub-hierarchy; however,
this may be limited explicitly in the future.
Delegation Containment
~~~~~~~~~~~~~~~~~~~~~~
A delegated sub-hierarchy is contained in the sense that processes
can't be moved into or out of the sub-hierarchy by the delegatee.
For delegations to a less privileged user, this is achieved by
requiring the following conditions for a process with a non-root euid
to migrate a target process into a cgroup by writing its PID to the
"cgroup.procs" file.
- The writer must have write access to the "cgroup.procs" file.
- The writer must have write access to the "cgroup.procs" file of the
common ancestor of the source and destination cgroups.
The above two constraints ensure that while a delegatee may migrate
processes around freely in the delegated sub-hierarchy it can't pull
in from or push out to outside the sub-hierarchy.
For an example, let's assume cgroups C0 and C1 have been delegated to
user U0 who created C00, C01 under C0 and C10 under C1 as follows and
all processes under C0 and C1 belong to U0::
~~~~~~~~~~~~~ - C0 - C00
~ cgroup ~ \ C01
~ hierarchy ~
~~~~~~~~~~~~~ - C1 - C10
Let's also say U0 wants to write the PID of a process which is
currently in C10 into "C00/cgroup.procs". U0 has write access to the
file; however, the common ancestor of the source cgroup C10 and the
destination cgroup C00 is above the points of delegation and U0 would
not have write access to its "cgroup.procs" files and thus the write
will be denied with -EACCES.
For delegations to namespaces, containment is achieved by requiring
that both the source and destination cgroups are reachable from the
namespace of the process which is attempting the migration. If either
is not reachable, the migration is rejected with -ENOENT.
Guidelines
----------
Organize Once and Control
~~~~~~~~~~~~~~~~~~~~~~~~~
Migrating a process across cgroups is a relatively expensive operation
and stateful resources such as memory are not moved together with the
process. This is an explicit design decision as there often exist
inherent trade-offs between migration and various hot paths in terms
of synchronization cost.
As such, migrating processes across cgroups frequently as a means to
apply different resource restrictions is discouraged. A workload
should be assigned to a cgroup according to the system's logical and
resource structure once on start-up. Dynamic adjustments to resource
distribution can be made by changing controller configuration through
the interface files.
Avoid Name Collisions
~~~~~~~~~~~~~~~~~~~~~
Interface files for a cgroup and its children cgroups occupy the same
directory and it is possible to create children cgroups which collide
with interface files.
All cgroup core interface files are prefixed with "cgroup." and each
controller's interface files are prefixed with the controller name and
a dot. A controller's name is composed of lower case alphabets and
'_'s but never begins with an '_' so it can be used as the prefix
character for collision avoidance. Also, interface file names won't
start or end with terms which are often used in categorizing workloads
such as job, service, slice, unit or workload.
cgroup doesn't do anything to prevent name collisions and it's the
user's responsibility to avoid them.
Resource Distribution Models
============================
cgroup controllers implement several resource distribution schemes
depending on the resource type and expected use cases. This section
describes major schemes in use along with their expected behaviors.
Weights
-------
A parent's resource is distributed by adding up the weights of all
active children and giving each the fraction matching the ratio of its
weight against the sum. As only children which can make use of the
resource at the moment participate in the distribution, this is
work-conserving. Due to the dynamic nature, this model is usually
used for stateless resources.
All weights are in the range [1, 10000] with the default at 100. This
allows symmetric multiplicative biases in both directions at fine
enough granularity while staying in the intuitive range.
As long as the weight is in range, all configuration combinations are
valid and there is no reason to reject configuration changes or
process migrations.
"cpu.weight" proportionally distributes CPU cycles to active children
and is an example of this type.
.. _cgroupv2-limits-distributor:
Limits
------
A child can only consume up to the configured amount of the resource.
Limits can be over-committed - the sum of the limits of children can
exceed the amount of resource available to the parent.
Limits are in the range [0, max] and defaults to "max", which is noop.
As limits can be over-committed, all configuration combinations are
valid and there is no reason to reject configuration changes or
process migrations.
"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume
on an IO device and is an example of this type.
.. _cgroupv2-protections-distributor:
Protections
-----------
A cgroup is protected up to the configured amount of the resource
as long as the usages of all its ancestors are under their
protected levels. Protections can be hard guarantees or best effort
soft boundaries. Protections can also be over-committed in which case
only up to the amount available to the parent is protected among
children.
Protections are in the range [0, max] and defaults to 0, which is
noop.
As protections can be over-committed, all configuration combinations
are valid and there is no reason to reject configuration changes or
process migrations.
"memory.low" implements best-effort memory protection and is an
example of this type.
Allocations
-----------
A cgroup is exclusively allocated a certain amount of a finite
resource. Allocations can't be over-committed - the sum of the
allocations of children can not exceed the amount of resource
available to the parent.
Allocations are in the range [0, max] and defaults to 0, which is no
resource.
As allocations can't be over-committed, some configuration
combinations are invalid and should be rejected. Also, if the
resource is mandatory for execution of processes, process migrations
may be rejected.
"cpu.rt.max" hard-allocates realtime slices and is an example of this
type.
Interface Files
===============
Format
------
All interface files should be in one of the following formats whenever
possible::
New-line separated values
(when only one value can be written at once)
VAL0\n
VAL1\n
...
Space separated values
(when read-only or multiple values can be written at once)
VAL0 VAL1 ...\n
Flat keyed
KEY0 VAL0\n
KEY1 VAL1\n
...
Nested keyed
KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
...
For a writable file, the format for writing should generally match
reading; however, controllers may allow omitting later fields or
implement restricted shortcuts for most common use cases.
For both flat and nested keyed files, only the values for a single key
can be written at a time. For nested keyed files, the sub key pairs
may be specified in any order and not all pairs have to be specified.
Conventions
-----------
- Settings for a single feature should be contained in a single file.
- The root cgroup should be exempt from resource control and thus
shouldn't have resource control interface files.
- The default time unit is microseconds. If a different unit is ever
used, an explicit unit suffix must be present.
- A parts-per quantity should use a percentage decimal with at least
two digit fractional part - e.g. 13.40.
- If a controller implements weight based resource distribution, its
interface file should be named "weight" and have the range [1,
10000] with 100 as the default. The values are chosen to allow
enough and symmetric bias in both directions while keeping it
intuitive (the default is 100%).
- If a controller implements an absolute resource guarantee and/or
limit, the interface files should be named "min" and "max"
respectively. If a controller implements best effort resource
guarantee and/or limit, the interface files should be named "low"
and "high" respectively.
In the above four control files, the special token "max" should be
used to represent upward infinity for both reading and writing.
- If a setting has a configurable default value and keyed specific
overrides, the default entry should be keyed with "default" and
appear as the first entry in the file.
The default value can be updated by writing either "default $VAL" or
"$VAL".
When writing to update a specific override, "default" can be used as
the value to indicate removal of the override. Override entries
with "default" as the value must not appear when read.
For example, a setting which is keyed by major:minor device numbers
with integer values may look like the following::
# cat cgroup-example-interface-file
default 150
8:0 300
The default value can be updated by::
# echo 125 > cgroup-example-interface-file
or::
# echo "default 125" > cgroup-example-interface-file
An override can be set by::
# echo "8:16 170" > cgroup-example-interface-file
and cleared by::
# echo "8:0 default" > cgroup-example-interface-file
# cat cgroup-example-interface-file
default 125
8:16 170
- For events which are not very high frequency, an interface file
"events" should be created which lists event key value pairs.
Whenever a notifiable event happens, file modified event should be
generated on the file.
Core Interface Files
--------------------
All cgroup core files are prefixed with "cgroup."
cgroup.type
A read-write single value file which exists on non-root
cgroups.
When read, it indicates the current type of the cgroup, which
can be one of the following values.
- "domain" : A normal valid domain cgroup.
- "domain threaded" : A threaded domain cgroup which is
serving as the root of a threaded subtree.
- "domain invalid" : A cgroup which is in an invalid state.
It can't be populated or have controllers enabled. It may
be allowed to become a threaded cgroup.
- "threaded" : A threaded cgroup which is a member of a
threaded subtree.
A cgroup can be turned into a threaded cgroup by writing
"threaded" to this file.
cgroup.procs
A read-write new-line separated values file which exists on
all cgroups.
When read, it lists the PIDs of all processes which belong to
the cgroup one-per-line. The PIDs are not ordered and the
same PID may show up more than once if the process got moved
to another cgroup and then back or the PID got recycled while
reading.
A PID can be written to migrate the process associated with
the PID to the cgroup. The writer should match all of the
following conditions.
- It must have write access to the "cgroup.procs" file.
- It must have write access to the "cgroup.procs" file of the
common ancestor of the source and destination cgroups.
When delegating a sub-hierarchy, write access to this file
should be granted along with the containing directory.
In a threaded cgroup, reading this file fails with EOPNOTSUPP
as all the processes belong to the thread root. Writing is
supported and moves every thread of the process to the cgroup.
cgroup.threads
A read-write new-line separated values file which exists on
all cgroups.
When read, it lists the TIDs of all threads which belong to
the cgroup one-per-line. The TIDs are not ordered and the
same TID may show up more than once if the thread got moved to
another cgroup and then back or the TID got recycled while
reading.
A TID can be written to migrate the thread associated with the
TID to the cgroup. The writer should match all of the
following conditions.
- It must have write access to the "cgroup.threads" file.
- The cgroup that the thread is currently in must be in the
same resource domain as the destination cgroup.
- It must have write access to the "cgroup.procs" file of the
common ancestor of the source and destination cgroups.
When delegating a sub-hierarchy, write access to this file
should be granted along with the containing directory.
cgroup.controllers
A read-only space separated values file which exists on all
cgroups.
It shows space separated list of all controllers available to
the cgroup. The controllers are not ordered.
cgroup.subtree_control
A read-write space separated values file which exists on all
cgroups. Starts out empty.
When read, it shows space separated list of the controllers
which are enabled to control resource distribution from the
cgroup to its children.
Space separated list of controllers prefixed with '+' or '-'
can be written to enable or disable controllers. A controller
name prefixed with '+' enables the controller and '-'
disables. If a controller appears more than once on the list,
the last one is effective. When multiple enable and disable
operations are specified, either all succeed or all fail.
cgroup.events
A read-only flat-keyed file which exists on non-root cgroups.
The following entries are defined. Unless specified
otherwise, a value change in this file generates a file
modified event.
populated
1 if the cgroup or its descendants contains any live
processes; otherwise, 0.
frozen
1 if the cgroup is frozen; otherwise, 0.
cgroup.max.descendants
A read-write single value files. The default is "max".
Maximum allowed number of descent cgroups.
If the actual number of descendants is equal or larger,
an attempt to create a new cgroup in the hierarchy will fail.
cgroup.max.depth
A read-write single value files. The default is "max".
Maximum allowed descent depth below the current cgroup.
If the actual descent depth is equal or larger,
an attempt to create a new child cgroup will fail.
cgroup.stat
A read-only flat-keyed file with the following entries:
nr_descendants
Total number of visible descendant cgroups.
nr_dying_descendants
Total number of dying descendant cgroups. A cgroup becomes
dying after being deleted by a user. The cgroup will remain
in dying state for some time undefined time (which can depend
on system load) before being completely destroyed.
A process can't enter a dying cgroup under any circumstances,
a dying cgroup can't revive.
A dying cgroup can consume system resources not exceeding
limits, which were active at the moment of cgroup deletion.
nr_subsys_<cgroup_subsys>
Total number of live cgroup subsystems (e.g memory
cgroup) at and beneath the current cgroup.
nr_dying_subsys_<cgroup_subsys>
Total number of dying cgroup subsystems (e.g. memory
cgroup) at and beneath the current cgroup.
cgroup.stat.local
A read-only flat-keyed file which exists in non-root cgroups.
The following entry is defined:
frozen_usec
Cumulative time that this cgroup has spent between freezing and
thawing, regardless of whether by self or ancestor groups.
NB: (not) reaching "frozen" state is not accounted here.
Using the following ASCII representation of a cgroup's freezer
state, ::
1 _____
frozen 0 __/ \__
ab cd
the duration being measured is the span between a and c.
cgroup.freeze
A read-write single value file which exists on non-root cgroups.
Allowed values are "0" and "1". The default is "0".
Writing "1" to the file causes freezing of the cgroup and all
descendant cgroups. This means that all belonging processes will
be stopped and will not run until the cgroup will be explicitly
unfrozen. Freezing of the cgroup may take some time; when this action
is completed, the "frozen" value in the cgroup.events control file
will be updated to "1" and the corresponding notification will be
issued.
A cgroup can be frozen either by its own settings, or by settings
of any ancestor cgroups. If any of ancestor cgroups is frozen, the
cgroup will remain frozen.
Processes in the frozen cgroup can be killed by a fatal signal.
They also can enter and leave a frozen cgroup: either by an explicit
move by a user, or if freezing of the cgroup races with fork().
If a process is moved to a frozen cgroup, it stops. If a process is
moved out of a frozen cgroup, it becomes running.
Frozen status of a cgroup doesn't affect any cgroup tree operations:
it's possible to delete a frozen (and empty) cgroup, as well as
create new sub-cgroups.
cgroup.kill
A write-only single value file which exists in non-root cgroups.
The only allowed value is "1".
Writing "1" to the file causes the cgroup and all descendant cgroups to
be killed. This means that all processes located in the affected cgroup
tree will be killed via SIGKILL.
Killing a cgroup tree will deal with concurrent forks appropriately and
is protected against migrations.
In a threaded cgroup, writing this file fails with EOPNOTSUPP as
killing cgroups is a process directed operation, i.e. it affects
the whole thread-group.
cgroup.pressure
A read-write single value file that allowed values are "0" and "1".
The default is "1".
Writing "0" to the file will disable the cgroup PSI accounting.
Writing "1" to the file will re-enable the cgroup PSI accounting.
This control attribute is not hierarchical, so disable or enable PSI
accounting in a cgroup does not affect PSI accounting in descendants
and doesn't need pass enablement via ancestors from root.
The reason this control attribute exists is that PSI accounts stalls for
each cgroup separately and aggregates it at each level of the hierarchy.
This may cause non-negligible overhead for some workloads when under
deep level of the hierarchy, in which case this control attribute can
be used to disable PSI accounting in the non-leaf cgroups.
irq.pressure
A read-write nested-keyed file.
Shows pressure stall information for IRQ/SOFTIRQ. See
:ref:`Documentation/accounting/psi.rst <psi>` for details.
Controllers
===========
.. _cgroup-v2-cpu:
CPU
---
The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.
In all the above models, cycles distribution is defined only on a temporal
base and it does not account for the frequency at which tasks are executed.
The (optional) utilization clamping support allows to hint the schedutil
cpufreq governor about the minimum desired frequency which should always be
provided by a CPU, as well as the maximum desired frequency, which should not
be exceeded by a CPU.
WARNING: cgroup2 cpu controller doesn't yet support the (bandwidth) control of
realtime processes. For a kernel built with the CONFIG_RT_GROUP_SCHED option
enabled for group scheduling of realtime processes, the cpu controller can only
be enabled when all RT processes are in the root cgroup. Be aware that system
management software may already have placed RT processes into non-root cgroups
during the system boot process, and these processes may need to be moved to the
root cgroup before the cpu controller can be enabled with a
CONFIG_RT_GROUP_SCHED enabled kernel.
With CONFIG_RT_GROUP_SCHED disabled, this limitation does not apply and some of
the interface files either affect realtime processes or account for them. See
the following section for details. Only the cpu controller is affected by
CONFIG_RT_GROUP_SCHED. Other controllers can be used for the resource control of
realtime processes irrespective of CONFIG_RT_GROUP_SCHED.
CPU Interface Files
~~~~~~~~~~~~~~~~~~~
The interaction of a process with the cpu controller depends on its scheduling
policy and the underlying scheduler. From the point of view of the cpu controller,
processes can be categorized as follows:
* Processes under the fair-class scheduler
* Processes under a BPF scheduler with the ``cgroup_set_weight`` callback
* Everything else: ``SCHED_{FIFO,RR,DEADLINE}`` and processes under a BPF scheduler
without the ``cgroup_set_weight`` callback
For details on when a process is under the fair-class scheduler or a BPF scheduler,
check out :ref:`Documentation/scheduler/sched-ext.rst <sched-ext>`.
For each of the following interface files, the above categories
will be referred to. All time durations are in microseconds.
cpu.stat
A read-only flat-keyed file.
This file exists whether the controller is enabled or not.
It always reports the following three stats, which account for all the
processes in the cgroup:
- usage_usec
- user_usec
- system_usec
and the following five when the controller is enabled, which account for
only the processes under the fair-class scheduler:
- nr_periods
- nr_throttled
- throttled_usec
- nr_bursts
- burst_usec
cpu.weight
A read-write single value file which exists on non-root
cgroups. The default is "100".
For non idle groups (cpu.idle = 0), the weight is in the
range [1, 10000].
If the cgroup has been configured to be SCHED_IDLE (cpu.idle = 1),
then the weight will show as a 0.
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_weight`` callback depending on what the
callback actually does.
cpu.weight.nice
A read-write single value file which exists on non-root
cgroups. The default is "0".
The nice value is in the range [-20, 19].
This interface file is an alternative interface for
"cpu.weight" and allows reading and setting weight using the
same values used by nice(2). Because the range is smaller and
granularity is coarser for the nice values, the read value is
the closest approximation of the current weight.
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_weight`` callback depending on what the
callback actually does.
cpu.max
A read-write two value file which exists on non-root cgroups.
The default is "max 100000".
The maximum bandwidth limit. It's in the following format::
$MAX $PERIOD
which indicates that the group may consume up to $MAX in each
$PERIOD duration. "max" for $MAX indicates no limit. If only
one number is written, $MAX is updated.
This file affects only processes under the fair-class scheduler.
cpu.max.burst
A read-write single value file which exists on non-root
cgroups. The default is "0".
The burst in the range [0, $MAX].
This file affects only processes under the fair-class scheduler.
cpu.pressure
A read-write nested-keyed file.
Shows pressure stall information for CPU. See
:ref:`Documentation/accounting/psi.rst <psi>` for details.
This file accounts for all the processes in the cgroup.
cpu.uclamp.min
A read-write single value file which exists on non-root cgroups.
The default is "0", i.e. no utilization boosting.
The requested minimum utilization (protection) as a percentage
rational number, e.g. 12.34 for 12.34%.
This interface allows reading and setting minimum utilization clamp
values similar to the sched_setattr(2). This minimum utilization
value is used to clamp the task specific minimum utilization clamp,
including those of realtime processes.
The requested minimum utilization (protection) is always capped by
the current value for the maximum utilization (limit), i.e.
`cpu.uclamp.max`.
This file affects all the processes in the cgroup.
cpu.uclamp.max
A read-write single value file which exists on non-root cgroups.
The default is "max". i.e. no utilization capping
The requested maximum utilization (limit) as a percentage rational
number, e.g. 98.76 for 98.76%.
This interface allows reading and setting maximum utilization clamp
values similar to the sched_setattr(2). This maximum utilization
value is used to clamp the task specific maximum utilization clamp,
including those of realtime processes.
This file affects all the processes in the cgroup.
cpu.idle
A read-write single value file which exists on non-root cgroups.
The default is 0.
This is the cgroup analog of the per-task SCHED_IDLE sched policy.
Setting this value to a 1 will make the scheduling policy of the
cgroup SCHED_IDLE. The threads inside the cgroup will retain their
own relative priorities, but the cgroup itself will be treated as
very low priority relative to its peers.
This file affects only processes under the fair-class scheduler.
Memory
------
The "memory" controller regulates distribution of memory. Memory is
stateful and implements both limit and protection models. Due to the
intertwining between memory usage and reclaim pressure and the
stateful nature of memory, the distribution model is relatively
complex.
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent. Currently, the
following types of memory usages are tracked.
- Userland memory - page cache and anonymous memory.
- Kernel data structures such as dentries and inodes.
- TCP socket buffers.
The above list may expand in the future for better coverage.
Memory Interface Files
~~~~~~~~~~~~~~~~~~~~~~
All memory amounts are in bytes. If a value which is not aligned to
PAGE_SIZE is written, the value may be rounded up to the closest
PAGE_SIZE multiple when read back.
memory.current
A read-only single value file which exists on non-root
cgroups.
The total amount of memory currently being used by the cgroup
and its descendants.
memory.min
A read-write single value file which exists on non-root
cgroups. The default is "0".
Hard memory protection. If the memory usage of a cgroup
is within its effective min boundary, the cgroup's memory
won't be reclaimed under any conditions. If there is no
unprotected reclaimable memory available, OOM killer
is invoked. Above the effective min boundary (or
effective low boundary if it is higher), pages are reclaimed
proportionally to the overage, reducing reclaim pressure for
smaller overages.
Effective min boundary is limited by memory.min values of
all ancestor cgroups. If there is memory.min overcommitment
(child cgroup or cgroups are requiring more protected memory
than parent will allow), then each child cgroup will get
the part of parent's protection proportional to its
actual memory usage below memory.min.
Putting more memory than generally available under this
protection is discouraged and may lead to constant OOMs.
If a memory cgroup is not populated with processes,
its memory.min is ignored.
memory.low
A read-write single value file which exists on non-root
cgroups. The default is "0".
Best-effort memory protection. If the memory usage of a
cgroup is within its effective low boundary, the cgroup's
memory won't be reclaimed unless there is no reclaimable
memory available in unprotected cgroups.
Above the effective low boundary (or
effective min boundary if it is higher), pages are reclaimed
proportionally to the overage, reducing reclaim pressure for
smaller overages.
Effective low boundary is limited by memory.low values of
all ancestor cgroups. If there is memory.low overcommitment
(child cgroup or cgroups are requiring more protected memory
than parent will allow), then each child cgroup will get
the part of parent's protection proportional to its
actual memory usage below memory.low.
Putting more memory than generally available under this
protection is discouraged.
memory.high
A read-write single value file which exists on non-root
cgroups. The default is "max".
Memory usage throttle limit. If a cgroup's usage goes
over the high boundary, the processes of the cgroup are
throttled and put under heavy reclaim pressure.
Going over the high limit never invokes the OOM killer and
under extreme conditions the limit may be breached. The high
limit should be used in scenarios where an external process
monitors the limited cgroup to alleviate heavy reclaim
pressure.
If memory.high is opened with O_NONBLOCK then the synchronous
reclaim is bypassed. This is useful for admin processes that
need to dynamically adjust the job's memory limits without
expending their own CPU resources on memory reclamation. The
job will trigger the reclaim and/or get throttled on its
next charge request.
Please note that with O_NONBLOCK, there is a chance that the
target memory cgroup may take indefinite amount of time to
reduce usage below the limit due to delayed charge request or
busy-hitting its memory to slow down reclaim.
memory.max
A read-write single value file which exists on non-root
cgroups. The default is "max".
Memory usage hard limit. This is the main mechanism to limit
memory usage of a cgroup. If a cgroup's memory usage reaches
this limit and can't be reduced, the OOM killer is invoked in
the cgroup. Under certain circumstances, the usage may go
over the limit temporarily.
In default configuration regular 0-order allocations always
succeed unless OOM killer chooses current task as a victim.
Some kinds of allocations don't invoke the OOM killer.
Caller could retry them differently, return into userspace
as -ENOMEM or silently ignore in cases like disk readahead.
If memory.max is opened with O_NONBLOCK, then the synchronous
reclaim and oom-kill are bypassed. This is useful for admin
processes that need to dynamically adjust the job's memory limits
without expending their own CPU resources on memory reclamation.
The job will trigger the reclaim and/or oom-kill on its next
charge request.
Please note that with O_NONBLOCK, there is a chance that the
target memory cgroup may take indefinite amount of time to
reduce usage below the limit due to delayed charge request or
busy-hitting its memory to slow down reclaim.
memory.reclaim
A write-only nested-keyed file which exists for all cgroups.
This is a simple interface to trigger memory reclaim in the
target cgroup.
Example::
echo "1G" > memory.reclaim
Please note that the kernel can over or under reclaim from
the target cgroup. If less bytes are reclaimed than the
specified amount, -EAGAIN is returned.
Please note that the proactive reclaim (triggered by this
interface) is not meant to indicate memory pressure on the
memory cgroup. Therefore socket memory balancing triggered by
the memory reclaim normally is not exercised in this case.
This means that the networking layer will not adapt based on
reclaim induced by memory.reclaim.
The following nested keys are defined.
========== ================================
swappiness Swappiness value to reclaim with
========== ================================
Specifying a swappiness value instructs the kernel to perform
the reclaim with that swappiness value. Note that this has the
same semantics as vm.swappiness applied to memcg reclaim with
all the existing limitations and potential future extensions.
The valid range for swappiness is [0-200, max], setting
swappiness=max exclusively reclaims anonymous memory.
memory.peak
A read-write single value file which exists on non-root cgroups.
The max memory usage recorded for the cgroup and its descendants since
either the creation of the cgroup or the most recent reset for that FD.
A write of any non-empty string to this file resets it to the
current memory usage for subsequent reads through the same
file descriptor.
memory.oom.group
A read-write single value file which exists on non-root
cgroups. The default value is "0".
Determines whether the cgroup should be treated as
an indivisible workload by the OOM killer. If set,
all tasks belonging to the cgroup or to its descendants
(if the memory cgroup is not a leaf cgroup) are killed
together or not at all. This can be used to avoid
partial kills to guarantee workload integrity.
Tasks with the OOM protection (oom_score_adj set to -1000)
are treated as an exception and are never killed.
If the OOM killer is invoked in a cgroup, it's not going
to kill any tasks outside of this cgroup, regardless
memory.oom.group values of ancestor cgroups.
memory.events
A read-only flat-keyed file which exists on non-root cgroups.
The following entries are defined. Unless specified
otherwise, a value change in this file generates a file
modified event.
Note that all fields in this file are hierarchical and the
file modified event can be generated due to an event down the
hierarchy. For the local events at the cgroup level see
memory.events.local.
low
The number of times the cgroup is reclaimed due to
high memory pressure even though its usage is under
the low boundary. This usually indicates that the low
boundary is over-committed.
high
The number of times processes of the cgroup are
throttled and routed to perform direct memory reclaim
because the high memory boundary was exceeded. For a
cgroup whose memory usage is capped by the high limit
rather than global memory pressure, this event's
occurrences are expected.
max
The number of times the cgroup's memory usage was
about to go over the max boundary. If direct reclaim
fails to bring it down, the cgroup goes to OOM state.
oom
The number of time the cgroup's memory usage was
reached the limit and allocation was about to fail.
This event is not raised if the OOM killer is not
considered as an option, e.g. for failed high-order
allocations or if caller asked to not retry attempts.
oom_kill
The number of processes belonging to this cgroup
killed by any kind of OOM killer.
oom_group_kill
The number of times a group OOM has occurred.
memory.events.local
Similar to memory.events but the fields in the file are local
to the cgroup i.e. not hierarchical. The file modified event
generated on this file reflects only the local events.
memory.stat
A read-only flat-keyed file which exists on non-root cgroups.
This breaks down the cgroup's memory footprint into different
types of memory, type-specific details, and other information
on the state and past events of the memory management system.
All memory amounts are in bytes.
The entries are ordered to be human readable, and new entries
can show up in the middle. Don't rely on items remaining in a
fixed position; use the keys to look up specific values!
If the entry has no per-node counter (or not show in the
memory.numa_stat). We use 'npn' (non-per-node) as the tag
to indicate that it will not show in the memory.numa_stat.
anon
Amount of memory used in anonymous mappings such as
brk(), sbrk(), and mmap(MAP_ANONYMOUS). Note that
some kernel configurations might account complete larger
allocations (e.g., THP) if only some, but not all the
memory of such an allocation is mapped anymore.
file
Amount of memory used to cache filesystem data,
including tmpfs and shared memory.
kernel (npn)
Amount of total kernel memory, including
(kernel_stack, pagetables, percpu, vmalloc, slab) in
addition to other kernel memory use cases.
kernel_stack
Amount of memory allocated to kernel stacks.
pagetables
Amount of memory allocated for page tables.
sec_pagetables
Amount of memory allocated for secondary page tables,
this currently includes KVM mmu allocations on x86
and arm64 and IOMMU page tables.
percpu (npn)
Amount of memory used for storing per-cpu kernel
data structures.
sock (npn)
Amount of memory used in network transmission buffers
vmalloc (npn)
Amount of memory used for vmap backed memory.
shmem
Amount of cached filesystem data that is swap-backed,
such as tmpfs, shm segments, shared anonymous mmap()s
zswap
Amount of memory consumed by the zswap compression backend.
zswapped
Amount of application memory swapped out to zswap.
file_mapped
Amount of cached filesystem data mapped with mmap(). Note
that some kernel configurations might account complete
larger allocations (e.g., THP) if only some, but not
not all the memory of such an allocation is mapped.
file_dirty
Amount of cached filesystem data that was modified but
not yet written back to disk
file_writeback
Amount of cached filesystem data that was modified and
is currently being written back to disk
swapcached
Amount of swap cached in memory. The swapcache is accounted
against both memory and swap usage.
anon_thp
Amount of memory used in anonymous mappings backed by
transparent hugepages
file_thp
Amount of cached filesystem data backed by transparent
hugepages
shmem_thp
Amount of shm, tmpfs, shared anonymous mmap()s backed by
transparent hugepages
inactive_anon, active_anon, inactive_file, active_file, unevictable
Amount of memory, swap-backed and filesystem-backed,
on the internal memory management lists used by the
page reclaim algorithm.
As these represent internal list state (eg. shmem pages are on anon
memory management lists), inactive_foo + active_foo may not be equal to
the value for the foo counter, since the foo counter is type-based, not
list-based.
slab_reclaimable
Part of "slab" that might be reclaimed, such as
dentries and inodes.
slab_unreclaimable
Part of "slab" that cannot be reclaimed on memory
pressure.
slab (npn)
Amount of memory used for storing in-kernel data
structures.
workingset_refault_anon
Number of refaults of previously evicted anonymous pages.
workingset_refault_file
Number of refaults of previously evicted file pages.
workingset_activate_anon
Number of refaulted anonymous pages that were immediately
activated.
workingset_activate_file
Number of refaulted file pages that were immediately activated.
workingset_restore_anon
Number of restored anonymous pages which have been detected as
an active workingset before they got reclaimed.
workingset_restore_file
Number of restored file pages which have been detected as an
active workingset before they got reclaimed.
workingset_nodereclaim
Number of times a shadow node has been reclaimed
pswpin (npn)
Number of pages swapped into memory
pswpout (npn)
Number of pages swapped out of memory
pgscan (npn)
Amount of scanned pages (in an inactive LRU list)
pgsteal (npn)
Amount of reclaimed pages
pgscan_kswapd (npn)
Amount of scanned pages by kswapd (in an inactive LRU list)
pgscan_direct (npn)
Amount of scanned pages directly (in an inactive LRU list)
pgscan_khugepaged (npn)
Amount of scanned pages by khugepaged (in an inactive LRU list)
pgscan_proactive (npn)
Amount of scanned pages proactively (in an inactive LRU list)
pgsteal_kswapd (npn)
Amount of reclaimed pages by kswapd
pgsteal_direct (npn)
Amount of reclaimed pages directly
pgsteal_khugepaged (npn)
Amount of reclaimed pages by khugepaged
pgsteal_proactive (npn)
Amount of reclaimed pages proactively
pgfault (npn)
Total number of page faults incurred
pgmajfault (npn)
Number of major page faults incurred
pgrefill (npn)
Amount of scanned pages (in an active LRU list)
pgactivate (npn)
Amount of pages moved to the active LRU list
pgdeactivate (npn)
Amount of pages moved to the inactive LRU list
pglazyfree (npn)
Amount of pages postponed to be freed under memory pressure
pglazyfreed (npn)
Amount of reclaimed lazyfree pages
swpin_zero
Number of pages swapped into memory and filled with zero, where I/O
was optimized out because the page content was detected to be zero
during swapout.
swpout_zero
Number of zero-filled pages swapped out with I/O skipped due to the
content being detected as zero.
zswpin
Number of pages moved in to memory from zswap.
zswpout
Number of pages moved out of memory to zswap.
zswpwb
Number of pages written from zswap to swap.
thp_fault_alloc (npn)
Number of transparent hugepages which were allocated to satisfy
a page fault. This counter is not present when CONFIG_TRANSPARENT_HUGEPAGE
is not set.
thp_collapse_alloc (npn)
Number of transparent hugepages which were allocated to allow
collapsing an existing range of pages. This counter is not
present when CONFIG_TRANSPARENT_HUGEPAGE is not set.
thp_swpout (npn)
Number of transparent hugepages which are swapout in one piece
without splitting.
thp_swpout_fallback (npn)
Number of transparent hugepages which were split before swapout.
Usually because failed to allocate some continuous swap space
for the huge page.
numa_pages_migrated (npn)
Number of pages migrated by NUMA balancing.
numa_pte_updates (npn)
Number of pages whose page table entries are modified by
NUMA balancing to produce NUMA hinting faults on access.
numa_hint_faults (npn)
Number of NUMA hinting faults.
pgdemote_kswapd
Number of pages demoted by kswapd.
pgdemote_direct
Number of pages demoted directly.
pgdemote_khugepaged
Number of pages demoted by khugepaged.
pgdemote_proactive
Number of pages demoted by proactively.
hugetlb
Amount of memory used by hugetlb pages. This metric only shows
up if hugetlb usage is accounted for in memory.current (i.e.
cgroup is mounted with the memory_hugetlb_accounting option).
memory.numa_stat
A read-only nested-keyed file which exists on non-root cgroups.
This breaks down the cgroup's memory footprint into different
types of memory, type-specific details, and other information
per node on the state of the memory management system.
This is useful for providing visibility into the NUMA locality
information within an memcg since the pages are allowed to be
allocated from any physical node. One of the use case is evaluating
application performance by combining this information with the
application's CPU allocation.
All memory amounts are in bytes.
The output format of memory.numa_stat is::
type N0=<bytes in node 0> N1=<bytes in node 1> ...
The entries are ordered to be human readable, and new entries
can show up in the middle. Don't rely on items remaining in a
fixed position; use the keys to look up specific values!
The entries can refer to the memory.stat.
memory.swap.current
A read-only single value file which exists on non-root
cgroups.
The total amount of swap currently being used by the cgroup
and its descendants.
memory.swap.high
A read-write single value file which exists on non-root
cgroups. The default is "max".
Swap usage throttle limit. If a cgroup's swap usage exceeds
this limit, all its further allocations will be throttled to
allow userspace to implement custom out-of-memory procedures.
This limit marks a point of no return for the cgroup. It is NOT
designed to manage the amount of swapping a workload does
during regular operation. Compare to memory.swap.max, which
prohibits swapping past a set amount, but lets the cgroup
continue unimpeded as long as other memory can be reclaimed.
Healthy workloads are not expected to reach this limit.
memory.swap.peak
A read-write single value file which exists on non-root cgroups.
The max swap usage recorded for the cgroup and its descendants since
the creation of the cgroup or the most recent reset for that FD.
A write of any non-empty string to this file resets it to the
current memory usage for subsequent reads through the same
file descriptor.
memory.swap.max
A read-write single value file which exists on non-root
cgroups. The default is "max".
Swap usage hard limit. If a cgroup's swap usage reaches this
limit, anonymous memory of the cgroup will not be swapped out.
memory.swap.events
A read-only flat-keyed file which exists on non-root cgroups.
The following entries are defined. Unless specified
otherwise, a value change in this file generates a file
modified event.
high
The number of times the cgroup's swap usage was over
the high threshold.
max
The number of times the cgroup's swap usage was about
to go over the max boundary and swap allocation
failed.
fail
The number of times swap allocation failed either
because of running out of swap system-wide or max
limit.
When reduced under the current usage, the existing swap
entries are reclaimed gradually and the swap usage may stay
higher than the limit for an extended period of time. This
reduces the impact on the workload and memory management.
memory.zswap.current
A read-only single value file which exists on non-root
cgroups.
The total amount of memory consumed by the zswap compression
backend.
memory.zswap.max
A read-write single value file which exists on non-root
cgroups. The default is "max".
Zswap usage hard limit. If a cgroup's zswap pool reaches this
limit, it will refuse to take any more stores before existing
entries fault back in or are written out to disk.
memory.zswap.writeback
A read-write single value file. The default value is "1".
Note that this setting is hierarchical, i.e. the writeback would be
implicitly disabled for child cgroups if the upper hierarchy
does so.
When this is set to 0, all swapping attempts to swapping devices
are disabled. This included both zswap writebacks, and swapping due
to zswap store failures. If the zswap store failures are recurring
(for e.g if the pages are incompressible), users can observe
reclaim inefficiency after disabling writeback (because the same
pages might be rejected again and again).
Note that this is subtly different from setting memory.swap.max to
0, as it still allows for pages to be written to the zswap pool.
This setting has no effect if zswap is disabled, and swapping
is allowed unless memory.swap.max is set to 0.
memory.pressure
A read-only nested-keyed file.
Shows pressure stall information for memory. See
:ref:`Documentation/accounting/psi.rst <psi>` for details.
Usage Guidelines
~~~~~~~~~~~~~~~~
"memory.high" is the main mechanism to control memory usage.
Over-committing on high limit (sum of high limits > available memory)
and letting global memory pressure to distribute memory according to
usage is a viable strategy.
Because breach of the high limit doesn't trigger the OOM killer but
throttles the offending cgroup, a management agent has ample
opportunities to monitor and take appropriate actions such as granting
more memory or terminating the workload.
Determining whether a cgroup has enough memory is not trivial as
memory usage doesn't indicate whether the workload can benefit from
more memory. For example, a workload which writes data received from
network to a file can use all available memory but can also operate as
performant with a small amount of memory. A measure of memory
pressure - how much the workload is being impacted due to lack of
memory - is necessary to determine whether a workload needs more
memory; unfortunately, memory pressure monitoring mechanism isn't
implemented yet.
Memory Ownership
~~~~~~~~~~~~~~~~
A memory area is charged to the cgroup which instantiated it and stays
charged to the cgroup until the area is released. Migrating a process
to a different cgroup doesn't move the memory usages that it
instantiated while in the previous cgroup to the new cgroup.
A memory area may be used by processes belonging to different cgroups.
To which cgroup the area will be charged is in-deterministic; however,
over time, the memory area is likely to end up in a cgroup which has
enough memory allowance to avoid high reclaim pressure.
If a cgroup sweeps a considerable amount of memory which is expected
to be accessed repeatedly by other cgroups, it may make sense to use
POSIX_FADV_DONTNEED to relinquish the ownership of memory areas
belonging to the affected files to ensure correct memory ownership.
IO
--
The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
IO Interface Files
~~~~~~~~~~~~~~~~~~
io.stat
A read-only nested-keyed file.
Lines are keyed by $MAJ:$MIN device numbers and not ordered.
The following nested keys are defined.
====== =====================
rbytes Bytes read
wbytes Bytes written
rios Number of read IOs
wios Number of write IOs
dbytes Bytes discarded
dios Number of discard IOs
====== =====================
An example read output follows::
8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0
8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021
io.cost.qos
A read-write nested-keyed file which exists only on the root
cgroup.
This file configures the Quality of Service of the IO cost
model based controller (CONFIG_BLK_CGROUP_IOCOST) which
currently implements "io.weight" proportional control. Lines
are keyed by $MAJ:$MIN device numbers and not ordered. The
line for a given device is populated on the first write for
the device on "io.cost.qos" or "io.cost.model". The following
nested keys are defined.
====== =====================================
enable Weight-based control enable
ctrl "auto" or "user"
rpct Read latency percentile [0, 100]
rlat Read latency threshold
wpct Write latency percentile [0, 100]
wlat Write latency threshold
min Minimum scaling percentage [1, 10000]
max Maximum scaling percentage [1, 10000]
====== =====================================
The controller is disabled by default and can be enabled by
setting "enable" to 1. "rpct" and "wpct" parameters default
to zero and the controller uses internal device saturation
state to adjust the overall IO rate between "min" and "max".
When a better control quality is needed, latency QoS
parameters can be configured. For example::
8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0
shows that on sdb, the controller is enabled, will consider
the device saturated if the 95th percentile of read completion
latencies is above 75ms or write 150ms, and adjust the overall
IO issue rate between 50% and 150% accordingly.
The lower the saturation point, the better the latency QoS at
the cost of aggregate bandwidth. The narrower the allowed
adjustment range between "min" and "max", the more conformant
to the cost model the IO behavior. Note that the IO issue
base rate may be far off from 100% and setting "min" and "max"
blindly can lead to a significant loss of device capacity or
control quality. "min" and "max" are useful for regulating
devices which show wide temporary behavior changes - e.g. a
ssd which accepts writes at the line speed for a while and
then completely stalls for multiple seconds.
When "ctrl" is "auto", the parameters are controlled by the
kernel and may change automatically. Setting "ctrl" to "user"
or setting any of the percentile and latency parameters puts
it into "user" mode and disables the automatic changes. The
automatic mode can be restored by setting "ctrl" to "auto".
io.cost.model
A read-write nested-keyed file which exists only on the root
cgroup.
This file configures the cost model of the IO cost model based
controller (CONFIG_BLK_CGROUP_IOCOST) which currently
implements "io.weight" proportional control. Lines are keyed
by $MAJ:$MIN device numbers and not ordered. The line for a
given device is populated on the first write for the device on
"io.cost.qos" or "io.cost.model". The following nested keys
are defined.
===== ================================
ctrl "auto" or "user"
model The cost model in use - "linear"
===== ================================
When "ctrl" is "auto", the kernel may change all parameters
dynamically. When "ctrl" is set to "user" or any other
parameters are written to, "ctrl" become "user" and the
automatic changes are disabled.
When "model" is "linear", the following model parameters are
defined.
============= ========================================
[r|w]bps The maximum sequential IO throughput
[r|w]seqiops The maximum 4k sequential IOs per second
[r|w]randiops The maximum 4k random IOs per second
============= ========================================
From the above, the builtin linear model determines the base
costs of a sequential and random IO and the cost coefficient
for the IO size. While simple, this model can cover most
common device classes acceptably.
The IO cost model isn't expected to be accurate in absolute
sense and is scaled to the device behavior dynamically.
If needed, tools/cgroup/iocost_coef_gen.py can be used to
generate device-specific coefficients.
io.weight
A read-write flat-keyed file which exists on non-root cgroups.
The default is "default 100".
The first line is the default weight applied to devices
without specific override. The rest are overrides keyed by
$MAJ:$MIN device numbers and not ordered. The weights are in
the range [1, 10000] and specifies the relative amount IO time
the cgroup can use in relation to its siblings.
The default weight can be updated by writing either "default
$WEIGHT" or simply "$WEIGHT". Overrides can be set by writing
"$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default".
An example read output follows::
default 100
8:16 200
8:0 50
io.max
A read-write nested-keyed file which exists on non-root
cgroups.
BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN
device numbers and not ordered. The following nested keys are
defined.
===== ==================================
rbps Max read bytes per second
wbps Max write bytes per second
riops Max read IO operations per second
wiops Max write IO operations per second
===== ==================================
When writing, any number of nested key-value pairs can be
specified in any order. "max" can be specified as the value
to remove a specific limit. If the same key is specified
multiple times, the outcome is undefined.
BPS and IOPS are measured in each IO direction and IOs are
delayed if limit is reached. Temporary bursts are allowed.
Setting read limit at 2M BPS and write at 120 IOPS for 8:16::
echo "8:16 rbps=2097152 wiops=120" > io.max
Reading returns the following::
8:16 rbps=2097152 wbps=max riops=max wiops=120
Write IOPS limit can be removed by writing the following::
echo "8:16 wiops=max" > io.max
Reading now returns the following::
8:16 rbps=2097152 wbps=max riops=max wiops=max
io.pressure
A read-only nested-keyed file.
Shows pressure stall information for IO. See
:ref:`Documentation/accounting/psi.rst <psi>` for details.
Writeback
~~~~~~~~~
Page cache is dirtied through buffered writes and shared mmaps and
written asynchronously to the backing filesystem by the writeback
mechanism. Writeback sits between the memory and IO domains and
regulates the proportion of dirty memory by balancing dirtying and
write IOs.
The io controller, in conjunction with the memory controller,
implements control of page cache writeback IOs. The memory controller
defines the memory domain that dirty memory ratio is calculated and
maintained for and the io controller defines the io domain which
writes out dirty pages for the memory domain. Both system-wide and
per-cgroup dirty memory states are examined and the more restrictive
of the two is enforced.
cgroup writeback requires explicit support from the underlying
filesystem. Currently, cgroup writeback is implemented on ext2, ext4,
btrfs, f2fs, and xfs. On other filesystems, all writeback IOs are
attributed to the root cgroup.
There are inherent differences in memory and writeback management
which affects how cgroup ownership is tracked. Memory is tracked per
page while writeback per inode. For the purpose of writeback, an
inode is assigned to a cgroup and all IO requests to write dirty pages
from the inode are attributed to that cgroup.
As cgroup ownership for memory is tracked per page, there can be pages
which are associated with different cgroups than the one the inode is
associated with. These are called foreign pages. The writeback
constantly keeps track of foreign pages and, if a particular foreign
cgroup becomes the majority over a certain period of time, switches
the ownership of the inode to that cgroup.
While this model is enough for most use cases where a given inode is
mostly dirtied by a single cgroup even when the main writing cgroup
changes over time, use cases where multiple cgroups write to a single
inode simultaneously are not supported well. In such circumstances, a
significant portion of IOs are likely to be attributed incorrectly.
As memory controller assigns page ownership on the first use and
doesn't update it until the page is released, even if writeback
strictly follows page ownership, multiple cgroups dirtying overlapping
areas wouldn't work as expected. It's recommended to avoid such usage
patterns.
The sysctl knobs which affect writeback behavior are applied to cgroup
writeback as follows.
vm.dirty_background_ratio, vm.dirty_ratio
These ratios apply the same to cgroup writeback with the
amount of available memory capped by limits imposed by the
memory controller and system-wide clean memory.
vm.dirty_background_bytes, vm.dirty_bytes
For cgroup writeback, this is calculated into ratio against
total available memory and applied the same way as
vm.dirty[_background]_ratio.
IO Latency
~~~~~~~~~~
This is a cgroup v2 controller for IO workload protection. You provide a group
with a latency target, and if the average latency exceeds that target the
controller will throttle any peers that have a lower latency target than the
protected workload.
The limits are only applied at the peer level in the hierarchy. This means that
in the diagram below, only groups A, B, and C will influence each other, and
groups D and F will influence each other. Group G will influence nobody::
[root]
/ | \
A B C
/ \ |
D F G
So the ideal way to configure this is to set io.latency in groups A, B, and C.
Generally you do not want to set a value lower than the latency your device
supports. Experiment to find the value that works best for your workload.
Start at higher than the expected latency for your device and watch the
avg_lat value in io.stat for your workload group to get an idea of the
latency you see during normal operation. Use the avg_lat value as a basis for
your real setting, setting at 10-15% higher than the value in io.stat.
How IO Latency Throttling Works
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
io.latency is work conserving; so as long as everybody is meeting their latency
target the controller doesn't do anything. Once a group starts missing its
target it begins throttling any peer group that has a higher target than itself.
This throttling takes 2 forms:
- Queue depth throttling. This is the number of outstanding IO's a group is
allowed to have. We will clamp down relatively quickly, starting at no limit
and going all the way down to 1 IO at a time.
- Artificial delay induction. There are certain types of IO that cannot be
throttled without possibly adversely affecting higher priority groups. This
includes swapping and metadata IO. These types of IO are allowed to occur
normally, however they are "charged" to the originating group. If the
originating group is being throttled you will see the use_delay and delay
fields in io.stat increase. The delay value is how many microseconds that are
being added to any process that runs in this group. Because this number can
grow quite large if there is a lot of swapping or metadata IO occurring we
limit the individual delay events to 1 second at a time.
Once the victimized group starts meeting its latency target again it will start
unthrottling any peer groups that were throttled previously. If the victimized
group simply stops doing IO the global counter will unthrottle appropriately.
IO Latency Interface Files
~~~~~~~~~~~~~~~~~~~~~~~~~~
io.latency
This takes a similar format as the other controllers.
"MAJOR:MINOR target=<target time in microseconds>"
io.stat
If the controller is enabled you will see extra stats in io.stat in
addition to the normal ones.
depth
This is the current queue depth for the group.
avg_lat
This is an exponential moving average with a decay rate of 1/exp
bound by the sampling interval. The decay rate interval can be
calculated by multiplying the win value in io.stat by the
corresponding number of samples based on the win value.
win
The sampling window size in milliseconds. This is the minimum
duration of time between evaluation events. Windows only elapse
with IO activity. Idle periods extend the most recent window.
IO Priority
~~~~~~~~~~~
A single attribute controls the behavior of the I/O priority cgroup policy,
namely the io.prio.class attribute. The following values are accepted for
that attribute:
no-change
Do not modify the I/O priority class.
promote-to-rt
For requests that have a non-RT I/O priority class, change it into RT.
Also change the priority level of these requests to 4. Do not modify
the I/O priority of requests that have priority class RT.
restrict-to-be
For requests that do not have an I/O priority class or that have I/O
priority class RT, change it into BE. Also change the priority level
of these requests to 0. Do not modify the I/O priority class of
requests that have priority class IDLE.
idle
Change the I/O priority class of all requests into IDLE, the lowest
I/O priority class.
none-to-rt
Deprecated. Just an alias for promote-to-rt.
The following numerical values are associated with the I/O priority policies:
+----------------+---+
| no-change | 0 |
+----------------+---+
| promote-to-rt | 1 |
+----------------+---+
| restrict-to-be | 2 |
+----------------+---+
| idle | 3 |
+----------------+---+
The numerical value that corresponds to each I/O priority class is as follows:
+-------------------------------+---+
| IOPRIO_CLASS_NONE | 0 |
+-------------------------------+---+
| IOPRIO_CLASS_RT (real-time) | 1 |
+-------------------------------+---+
| IOPRIO_CLASS_BE (best effort) | 2 |
+-------------------------------+---+
| IOPRIO_CLASS_IDLE | 3 |
+-------------------------------+---+
The algorithm to set the I/O priority class for a request is as follows:
- If I/O priority class policy is promote-to-rt, change the request I/O
priority class to IOPRIO_CLASS_RT and change the request I/O priority
level to 4.
- If I/O priority class policy is not promote-to-rt, translate the I/O priority
class policy into a number, then change the request I/O priority class
into the maximum of the I/O priority class policy number and the numerical
I/O priority class.
PID
---
The process number controller is used to allow a cgroup to stop any
new tasks from being fork()'d or clone()'d after a specified limit is
reached.
The number of tasks in a cgroup can be exhausted in ways which other
controllers cannot prevent, thus warranting its own controller. For
example, a fork bomb is likely to exhaust the number of tasks before
hitting memory restrictions.
Note that PIDs used in this controller refer to TIDs, process IDs as
used by the kernel.
PID Interface Files
~~~~~~~~~~~~~~~~~~~
pids.max
A read-write single value file which exists on non-root
cgroups. The default is "max".
Hard limit of number of processes.
pids.current
A read-only single value file which exists on non-root cgroups.
The number of processes currently in the cgroup and its
descendants.
pids.peak
A read-only single value file which exists on non-root cgroups.
The maximum value that the number of processes in the cgroup and its
descendants has ever reached.
pids.events
A read-only flat-keyed file which exists on non-root cgroups. Unless
specified otherwise, a value change in this file generates a file
modified event. The following entries are defined.
max
The number of times the cgroup's total number of processes hit the pids.max
limit (see also pids_localevents).
pids.events.local
Similar to pids.events but the fields in the file are local
to the cgroup i.e. not hierarchical. The file modified event
generated on this file reflects only the local events.
Organisational operations are not blocked by cgroup policies, so it is
possible to have pids.current > pids.max. This can be done by either
setting the limit to be smaller than pids.current, or attaching enough
processes to the cgroup such that pids.current is larger than
pids.max. However, it is not possible to violate a cgroup PID policy
through fork() or clone(). These will return -EAGAIN if the creation
of a new process would cause a cgroup policy to be violated.
Cpuset
------
The "cpuset" controller provides a mechanism for constraining
the CPU and memory node placement of tasks to only the resources
specified in the cpuset interface files in a task's current cgroup.
This is especially valuable on large NUMA systems where placing jobs
on properly sized subsets of the systems with careful processor and
memory placement to reduce cross-node memory access and contention
can improve overall system performance.
The "cpuset" controller is hierarchical. That means the controller
cannot use CPUs or memory nodes not allowed in its parent.
Cpuset Interface Files
~~~~~~~~~~~~~~~~~~~~~~
cpuset.cpus
A read-write multiple values file which exists on non-root
cpuset-enabled cgroups.
It lists the requested CPUs to be used by tasks within this
cgroup. The actual list of CPUs to be granted, however, is
subjected to constraints imposed by its parent and can differ
from the requested CPUs.
The CPU numbers are comma-separated numbers or ranges.
For example::
# cat cpuset.cpus
0-4,6,8-10
An empty value indicates that the cgroup is using the same
setting as the nearest cgroup ancestor with a non-empty
"cpuset.cpus" or all the available CPUs if none is found.
The value of "cpuset.cpus" stays constant until the next update
and won't be affected by any CPU hotplug events.
cpuset.cpus.effective
A read-only multiple values file which exists on all
cpuset-enabled cgroups.
It lists the onlined CPUs that are actually granted to this
cgroup by its parent. These CPUs are allowed to be used by
tasks within the current cgroup.
If "cpuset.cpus" is empty, the "cpuset.cpus.effective" file shows
all the CPUs from the parent cgroup that can be available to
be used by this cgroup. Otherwise, it should be a subset of
"cpuset.cpus" unless none of the CPUs listed in "cpuset.cpus"
can be granted. In this case, it will be treated just like an
empty "cpuset.cpus".
Its value will be affected by CPU hotplug events.
cpuset.mems
A read-write multiple values file which exists on non-root
cpuset-enabled cgroups.
It lists the requested memory nodes to be used by tasks within
this cgroup. The actual list of memory nodes granted, however,
is subjected to constraints imposed by its parent and can differ
from the requested memory nodes.
The memory node numbers are comma-separated numbers or ranges.
For example::
# cat cpuset.mems
0-1,3
An empty value indicates that the cgroup is using the same
setting as the nearest cgroup ancestor with a non-empty
"cpuset.mems" or all the available memory nodes if none
is found.
The value of "cpuset.mems" stays constant until the next update
and won't be affected by any memory nodes hotplug events.
Setting a non-empty value to "cpuset.mems" causes memory of
tasks within the cgroup to be migrated to the designated nodes if
they are currently using memory outside of the designated nodes.
There is a cost for this memory migration. The migration
may not be complete and some memory pages may be left behind.
So it is recommended that "cpuset.mems" should be set properly
before spawning new tasks into the cpuset. Even if there is
a need to change "cpuset.mems" with active tasks, it shouldn't
be done frequently.
cpuset.mems.effective
A read-only multiple values file which exists on all
cpuset-enabled cgroups.
It lists the onlined memory nodes that are actually granted to
this cgroup by its parent. These memory nodes are allowed to
be used by tasks within the current cgroup.
If "cpuset.mems" is empty, it shows all the memory nodes from the
parent cgroup that will be available to be used by this cgroup.
Otherwise, it should be a subset of "cpuset.mems" unless none of
the memory nodes listed in "cpuset.mems" can be granted. In this
case, it will be treated just like an empty "cpuset.mems".
Its value will be affected by memory nodes hotplug events.
cpuset.cpus.exclusive
A read-write multiple values file which exists on non-root
cpuset-enabled cgroups.
It lists all the exclusive CPUs that are allowed to be used
to create a new cpuset partition. Its value is not used
unless the cgroup becomes a valid partition root. See the
"cpuset.cpus.partition" section below for a description of what
a cpuset partition is.
When the cgroup becomes a partition root, the actual exclusive
CPUs that are allocated to that partition are listed in
"cpuset.cpus.exclusive.effective" which may be different
from "cpuset.cpus.exclusive". If "cpuset.cpus.exclusive"
has previously been set, "cpuset.cpus.exclusive.effective"
is always a subset of it.
Users can manually set it to a value that is different from
"cpuset.cpus". One constraint in setting it is that the list of
CPUs must be exclusive with respect to "cpuset.cpus.exclusive"
and "cpuset.cpus.exclusive.effective" of its siblings. Another
constraint is that it cannot be a superset of "cpuset.cpus"
of its sibling in order to leave at least one CPU available to
that sibling when the exclusive CPUs are taken away.
For a parent cgroup, any one of its exclusive CPUs can only
be distributed to at most one of its child cgroups. Having an
exclusive CPU appearing in two or more of its child cgroups is
not allowed (the exclusivity rule). A value that violates the
exclusivity rule will be rejected with a write error.
The root cgroup is a partition root and all its available CPUs
are in its exclusive CPU set.
cpuset.cpus.exclusive.effective
A read-only multiple values file which exists on all non-root
cpuset-enabled cgroups.
This file shows the effective set of exclusive CPUs that
can be used to create a partition root. The content
of this file will always be a subset of its parent's
"cpuset.cpus.exclusive.effective" if its parent is not the root
cgroup. It will also be a subset of "cpuset.cpus.exclusive"
if it is set. If "cpuset.cpus.exclusive" is not set, it is
treated to have an implicit value of "cpuset.cpus" in the
formation of local partition.
cpuset.cpus.isolated
A read-only and root cgroup only multiple values file.
This file shows the set of all isolated CPUs used in existing
isolated partitions. It will be empty if no isolated partition
is created.
cpuset.cpus.partition
A read-write single value file which exists on non-root
cpuset-enabled cgroups. This flag is owned by the parent cgroup
and is not delegatable.
It accepts only the following input values when written to.
========== =====================================
"member" Non-root member of a partition
"root" Partition root
"isolated" Partition root without load balancing
========== =====================================
A cpuset partition is a collection of cpuset-enabled cgroups with
a partition root at the top of the hierarchy and its descendants
except those that are separate partition roots themselves and
their descendants. A partition has exclusive access to the
set of exclusive CPUs allocated to it. Other cgroups outside
of that partition cannot use any CPUs in that set.
There are two types of partitions - local and remote. A local
partition is one whose parent cgroup is also a valid partition
root. A remote partition is one whose parent cgroup is not a
valid partition root itself. Writing to "cpuset.cpus.exclusive"
is optional for the creation of a local partition as its
"cpuset.cpus.exclusive" file will assume an implicit value that
is the same as "cpuset.cpus" if it is not set. Writing the
proper "cpuset.cpus.exclusive" values down the cgroup hierarchy
before the target partition root is mandatory for the creation
of a remote partition.
Currently, a remote partition cannot be created under a local
partition. All the ancestors of a remote partition root except
the root cgroup cannot be a partition root.
The root cgroup is always a partition root and its state cannot
be changed. All other non-root cgroups start out as "member".
When set to "root", the current cgroup is the root of a new
partition or scheduling domain. The set of exclusive CPUs is
determined by the value of its "cpuset.cpus.exclusive.effective".
When set to "isolated", the CPUs in that partition will be in
an isolated state without any load balancing from the scheduler
and excluded from the unbound workqueues. Tasks placed in such
a partition with multiple CPUs should be carefully distributed
and bound to each of the individual CPUs for optimal performance.
A partition root ("root" or "isolated") can be in one of the
two possible states - valid or invalid. An invalid partition
root is in a degraded state where some state information may
be retained, but behaves more like a "member".
All possible state transitions among "member", "root" and
"isolated" are allowed.
On read, the "cpuset.cpus.partition" file can show the following
values.
============================= =====================================
"member" Non-root member of a partition
"root" Partition root
"isolated" Partition root without load balancing
"root invalid (<reason>)" Invalid partition root
"isolated invalid (<reason>)" Invalid isolated partition root
============================= =====================================
In the case of an invalid partition root, a descriptive string on
why the partition is invalid is included within parentheses.
For a local partition root to be valid, the following conditions
must be met.
1) The parent cgroup is a valid partition root.
2) The "cpuset.cpus.exclusive.effective" file cannot be empty,
though it may contain offline CPUs.
3) The "cpuset.cpus.effective" cannot be empty unless there is
no task associated with this partition.
For a remote partition root to be valid, all the above conditions
except the first one must be met.
External events like hotplug or changes to "cpuset.cpus" or
"cpuset.cpus.exclusive" can cause a valid partition root to
become invalid and vice versa. Note that a task cannot be
moved to a cgroup with empty "cpuset.cpus.effective".
A valid non-root parent partition may distribute out all its CPUs
to its child local partitions when there is no task associated
with it.
Care must be taken to change a valid partition root to "member"
as all its child local partitions, if present, will become
invalid causing disruption to tasks running in those child
partitions. These inactivated partitions could be recovered if
their parent is switched back to a partition root with a proper
value in "cpuset.cpus" or "cpuset.cpus.exclusive".
Poll and inotify events are triggered whenever the state of
"cpuset.cpus.partition" changes. That includes changes caused
by write to "cpuset.cpus.partition", cpu hotplug or other
changes that modify the validity status of the partition.
This will allow user space agents to monitor unexpected changes
to "cpuset.cpus.partition" without the need to do continuous
polling.
A user can pre-configure certain CPUs to an isolated state
with load balancing disabled at boot time with the "isolcpus"
kernel boot command line option. If those CPUs are to be put
into a partition, they have to be used in an isolated partition.
Device controller
-----------------
Device controller manages access to device files. It includes both
creation of new device files (using mknod), and access to the
existing device files.
Cgroup v2 device controller has no interface files and is implemented
on top of cgroup BPF. To control access to device files, a user may
create bpf programs of type BPF_PROG_TYPE_CGROUP_DEVICE and attach
them to cgroups with BPF_CGROUP_DEVICE flag. On an attempt to access a
device file, corresponding BPF programs will be executed, and depending
on the return value the attempt will succeed or fail with -EPERM.
A BPF_PROG_TYPE_CGROUP_DEVICE program takes a pointer to the
bpf_cgroup_dev_ctx structure, which describes the device access attempt:
access type (mknod/read/write) and device (type, major and minor numbers).
If the program returns 0, the attempt fails with -EPERM, otherwise it
succeeds.
An example of BPF_PROG_TYPE_CGROUP_DEVICE program may be found in
tools/testing/selftests/bpf/progs/dev_cgroup.c in the kernel source tree.
RDMA
----
The "rdma" controller regulates the distribution and accounting of
RDMA resources.
RDMA Interface Files
~~~~~~~~~~~~~~~~~~~~
rdma.max
A readwrite nested-keyed file that exists for all the cgroups
except root that describes current configured resource limit
for a RDMA/IB device.
Lines are keyed by device name and are not ordered.
Each line contains space separated resource name and its configured
limit that can be distributed.
The following nested keys are defined.
========== =============================
hca_handle Maximum number of HCA Handles
hca_object Maximum number of HCA Objects
========== =============================
An example for mlx4 and ocrdma device follows::
mlx4_0 hca_handle=2 hca_object=2000
ocrdma1 hca_handle=3 hca_object=max
rdma.current
A read-only file that describes current resource usage.
It exists for all the cgroup except root.
An example for mlx4 and ocrdma device follows::
mlx4_0 hca_handle=1 hca_object=20
ocrdma1 hca_handle=1 hca_object=23
DMEM
----
The "dmem" controller regulates the distribution and accounting of
device memory regions. Because each memory region may have its own page size,
which does not have to be equal to the system page size, the units are always bytes.
DMEM Interface Files
~~~~~~~~~~~~~~~~~~~~
dmem.max, dmem.min, dmem.low
A readwrite nested-keyed file that exists for all the cgroups
except root that describes current configured resource limit
for a region.
An example for xe follows::
drm/0000:03:00.0/vram0 1073741824
drm/0000:03:00.0/stolen max
The semantics are the same as for the memory cgroup controller, and are
calculated in the same way.
dmem.capacity
A read-only file that describes maximum region capacity.
It only exists on the root cgroup. Not all memory can be
allocated by cgroups, as the kernel reserves some for
internal use.
An example for xe follows::
drm/0000:03:00.0/vram0 8514437120
drm/0000:03:00.0/stolen 67108864
dmem.current
A read-only file that describes current resource usage.
It exists for all the cgroup except root.
An example for xe follows::
drm/0000:03:00.0/vram0 12550144
drm/0000:03:00.0/stolen 8650752
HugeTLB
-------
The HugeTLB controller allows to limit the HugeTLB usage per control group and
enforces the controller limit during page fault.
HugeTLB Interface Files
~~~~~~~~~~~~~~~~~~~~~~~
hugetlb.<hugepagesize>.current
Show current usage for "hugepagesize" hugetlb. It exists for all
the cgroup except root.
hugetlb.<hugepagesize>.max
Set/show the hard limit of "hugepagesize" hugetlb usage.
The default value is "max". It exists for all the cgroup except root.
hugetlb.<hugepagesize>.events
A read-only flat-keyed file which exists on non-root cgroups.
max
The number of allocation failure due to HugeTLB limit
hugetlb.<hugepagesize>.events.local
Similar to hugetlb.<hugepagesize>.events but the fields in the file
are local to the cgroup i.e. not hierarchical. The file modified event
generated on this file reflects only the local events.
hugetlb.<hugepagesize>.numa_stat
Similar to memory.numa_stat, it shows the numa information of the
hugetlb pages of <hugepagesize> in this cgroup. Only active in
use hugetlb pages are included. The per-node values are in bytes.
Misc
----
The Miscellaneous cgroup provides the resource limiting and tracking
mechanism for the scalar resources which cannot be abstracted like the other
cgroup resources. Controller is enabled by the CONFIG_CGROUP_MISC config
option.
A resource can be added to the controller via enum misc_res_type{} in the
include/linux/misc_cgroup.h file and the corresponding name via misc_res_name[]
in the kernel/cgroup/misc.c file. Provider of the resource must set its
capacity prior to using the resource by calling misc_cg_set_capacity().
Once a capacity is set then the resource usage can be updated using charge and
uncharge APIs. All of the APIs to interact with misc controller are in
include/linux/misc_cgroup.h.
Misc Interface Files
~~~~~~~~~~~~~~~~~~~~
Miscellaneous controller provides 3 interface files. If two misc resources (res_a and res_b) are registered then:
misc.capacity
A read-only flat-keyed file shown only in the root cgroup. It shows
miscellaneous scalar resources available on the platform along with
their quantities::
$ cat misc.capacity
res_a 50
res_b 10
misc.current
A read-only flat-keyed file shown in the all cgroups. It shows
the current usage of the resources in the cgroup and its children.::
$ cat misc.current
res_a 3
res_b 0
misc.peak
A read-only flat-keyed file shown in all cgroups. It shows the
historical maximum usage of the resources in the cgroup and its
children.::
$ cat misc.peak
res_a 10
res_b 8
misc.max
A read-write flat-keyed file shown in the non root cgroups. Allowed
maximum usage of the resources in the cgroup and its children.::
$ cat misc.max
res_a max
res_b 4
Limit can be set by::
# echo res_a 1 > misc.max
Limit can be set to max by::
# echo res_a max > misc.max
Limits can be set higher than the capacity value in the misc.capacity
file.
misc.events
A read-only flat-keyed file which exists on non-root cgroups. The
following entries are defined. Unless specified otherwise, a value
change in this file generates a file modified event. All fields in
this file are hierarchical.
max
The number of times the cgroup's resource usage was
about to go over the max boundary.
misc.events.local
Similar to misc.events but the fields in the file are local to the
cgroup i.e. not hierarchical. The file modified event generated on
this file reflects only the local events.
Migration and Ownership
~~~~~~~~~~~~~~~~~~~~~~~
A miscellaneous scalar resource is charged to the cgroup in which it is used
first, and stays charged to that cgroup until that resource is freed. Migrating
a process to a different cgroup does not move the charge to the destination
cgroup where the process has moved.
Others
------
perf_event
~~~~~~~~~~
perf_event controller, if not mounted on a legacy hierarchy, is
automatically enabled on the v2 hierarchy so that perf events can
always be filtered by cgroup v2 path. The controller can still be
moved to a legacy hierarchy after v2 hierarchy is populated.
Non-normative information
-------------------------
This section contains information that isn't considered to be a part of
the stable kernel API and so is subject to change.
CPU controller root cgroup process behaviour
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When distributing CPU cycles in the root cgroup each thread in this
cgroup is treated as if it was hosted in a separate child cgroup of the
root cgroup. This child cgroup weight is dependent on its thread nice
level.
For details of this mapping see sched_prio_to_weight array in
kernel/sched/core.c file (values from this array should be scaled
appropriately so the neutral - nice 0 - value is 100 instead of 1024).
IO controller root cgroup process behaviour
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Root cgroup processes are hosted in an implicit leaf child node.
When distributing IO resources this implicit child node is taken into
account as if it was a normal child cgroup of the root cgroup with a
weight value of 200.
Namespace
=========
Basics
------
cgroup namespace provides a mechanism to virtualize the view of the
"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone
flag can be used with clone(2) and unshare(2) to create a new cgroup
namespace. The process running inside the cgroup namespace will have
its "/proc/$PID/cgroup" output restricted to cgroupns root. The
cgroupns root is the cgroup of the process at the time of creation of
the cgroup namespace.
Without cgroup namespace, the "/proc/$PID/cgroup" file shows the
complete path of the cgroup of a process. In a container setup where
a set of cgroups and namespaces are intended to isolate processes the
"/proc/$PID/cgroup" file may leak potential system level information
to the isolated processes. For example::
# cat /proc/self/cgroup
0::/batchjobs/container_id1
The path '/batchjobs/container_id1' can be considered as system-data
and undesirable to expose to the isolated processes. cgroup namespace
can be used to restrict visibility of this path. For example, before
creating a cgroup namespace, one would see::
# ls -l /proc/self/ns/cgroup
lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835]
# cat /proc/self/cgroup
0::/batchjobs/container_id1
After unsharing a new namespace, the view changes::
# ls -l /proc/self/ns/cgroup
lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183]
# cat /proc/self/cgroup
0::/
When some thread from a multi-threaded process unshares its cgroup
namespace, the new cgroupns gets applied to the entire process (all
the threads). This is natural for the v2 hierarchy; however, for the
legacy hierarchies, this may be unexpected.
A cgroup namespace is alive as long as there are processes inside or
mounts pinning it. When the last usage goes away, the cgroup
namespace is destroyed. The cgroupns root and the actual cgroups
remain.
The Root and Views
------------------
The 'cgroupns root' for a cgroup namespace is the cgroup in which the
process calling unshare(2) is running. For example, if a process in
/batchjobs/container_id1 cgroup calls unshare, cgroup
/batchjobs/container_id1 becomes the cgroupns root. For the
init_cgroup_ns, this is the real root ('/') cgroup.
The cgroupns root cgroup does not change even if the namespace creator
process later moves to a different cgroup::
# ~/unshare -c # unshare cgroupns in some cgroup
# cat /proc/self/cgroup
0::/
# mkdir sub_cgrp_1
# echo 0 > sub_cgrp_1/cgroup.procs
# cat /proc/self/cgroup
0::/sub_cgrp_1
Each process gets its namespace-specific view of "/proc/$PID/cgroup"
Processes running inside the cgroup namespace will be able to see
cgroup paths (in /proc/self/cgroup) only inside their root cgroup.
From within an unshared cgroupns::
# sleep 100000 &
[1] 7353
# echo 7353 > sub_cgrp_1/cgroup.procs
# cat /proc/7353/cgroup
0::/sub_cgrp_1
From the initial cgroup namespace, the real cgroup path will be
visible::
$ cat /proc/7353/cgroup
0::/batchjobs/container_id1/sub_cgrp_1
From a sibling cgroup namespace (that is, a namespace rooted at a
different cgroup), the cgroup path relative to its own cgroup
namespace root will be shown. For instance, if PID 7353's cgroup
namespace root is at '/batchjobs/container_id2', then it will see::
# cat /proc/7353/cgroup
0::/../container_id2/sub_cgrp_1
Note that the relative path always starts with '/' to indicate that
its relative to the cgroup namespace root of the caller.
Migration and setns(2)
----------------------
Processes inside a cgroup namespace can move into and out of the
namespace root if they have proper access to external cgroups. For
example, from inside a namespace with cgroupns root at
/batchjobs/container_id1, and assuming that the global hierarchy is
still accessible inside cgroupns::
# cat /proc/7353/cgroup
0::/sub_cgrp_1
# echo 7353 > batchjobs/container_id2/cgroup.procs
# cat /proc/7353/cgroup
0::/../container_id2
Note that this kind of setup is not encouraged. A task inside cgroup
namespace should only be exposed to its own cgroupns hierarchy.
setns(2) to another cgroup namespace is allowed when:
(a) the process has CAP_SYS_ADMIN against its current user namespace
(b) the process has CAP_SYS_ADMIN against the target cgroup
namespace's userns
No implicit cgroup changes happen with attaching to another cgroup
namespace. It is expected that the someone moves the attaching
process under the target cgroup namespace root.
Interaction with Other Namespaces
---------------------------------
Namespace specific cgroup hierarchy can be mounted by a process
running inside a non-init cgroup namespace::
# mount -t cgroup2 none $MOUNT_POINT
This will mount the unified cgroup hierarchy with cgroupns root as the
filesystem root. The process needs CAP_SYS_ADMIN against its user and
mount namespaces.
The virtualization of /proc/self/cgroup file combined with restricting
the view of cgroup hierarchy by namespace-private cgroupfs mount
provides a properly isolated cgroup view inside the container.
Information on Kernel Programming
=================================
This section contains kernel programming information in the areas
where interacting with cgroup is necessary. cgroup core and
controllers are not covered.
Filesystem Support for Writeback
--------------------------------
A filesystem can support cgroup writeback by updating
address_space_operations->writepages() to annotate bio's using the
following two functions.
wbc_init_bio(@wbc, @bio)
Should be called for each bio carrying writeback data and
associates the bio with the inode's owner cgroup and the
corresponding request queue. This must be called after
a queue (device) has been associated with the bio and
before submission.
wbc_account_cgroup_owner(@wbc, @folio, @bytes)
Should be called for each data segment being written out.
While this function doesn't care exactly when it's called
during the writeback session, it's the easiest and most
natural to call it as data segments are added to a bio.
With writeback bio's annotated, cgroup support can be enabled per
super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for
selective disabling of cgroup writeback support which is helpful when
certain filesystem features, e.g. journaled data mode, are
incompatible.
wbc_init_bio() binds the specified bio to its cgroup. Depending on
the configuration, the bio may be executed at a lower priority and if
the writeback session is holding shared resources, e.g. a journal
entry, may lead to priority inversion. There is no one easy solution
for the problem. Filesystems can try to work around specific problem
cases by skipping wbc_init_bio() and using bio_associate_blkg()
directly.
Deprecated v1 Core Features
===========================
- Multiple hierarchies including named ones are not supported.
- All v1 mount options are not supported.
- The "tasks" file is removed and "cgroup.procs" is not sorted.
- "cgroup.clone_children" is removed.
- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" or
"cgroup.stat" files at the root instead.
Issues with v1 and Rationales for v2
====================================
Multiple Hierarchies
--------------------
cgroup v1 allowed an arbitrary number of hierarchies and each
hierarchy could host any number of controllers. While this seemed to
provide a high level of flexibility, it wasn't useful in practice.
For example, as there is only one instance of each controller, utility
type controllers such as freezer which can be useful in all
hierarchies could only be used in one. The issue is exacerbated by
the fact that controllers couldn't be moved to another hierarchy once
hierarchies were populated. Another issue was that all controllers
bound to a hierarchy were forced to have exactly the same view of the
hierarchy. It wasn't possible to vary the granularity depending on
the specific controller.
In practice, these issues heavily limited which controllers could be
put on the same hierarchy and most configurations resorted to putting
each controller on its own hierarchy. Only closely related ones, such
as the cpu and cpuacct controllers, made sense to be put on the same
hierarchy. This often meant that userland ended up managing multiple
similar hierarchies repeating the same steps on each hierarchy
whenever a hierarchy management operation was necessary.
Furthermore, support for multiple hierarchies came at a steep cost.
It greatly complicated cgroup core implementation but more importantly
the support for multiple hierarchies restricted how cgroup could be
used in general and what controllers was able to do.
There was no limit on how many hierarchies there might be, which meant
that a thread's cgroup membership couldn't be described in finite
length. The key might contain any number of entries and was unlimited
in length, which made it highly awkward to manipulate and led to
addition of controllers which existed only to identify membership,
which in turn exacerbated the original problem of proliferating number
of hierarchies.
Also, as a controller couldn't have any expectation regarding the
topologies of hierarchies other controllers might be on, each
controller had to assume that all other controllers were attached to
completely orthogonal hierarchies. This made it impossible, or at
least very cumbersome, for controllers to cooperate with each other.
In most use cases, putting controllers on hierarchies which are
completely orthogonal to each other isn't necessary. What usually is
called for is the ability to have differing levels of granularity
depending on the specific controller. In other words, hierarchy may
be collapsed from leaf towards root when viewed from specific
controllers. For example, a given configuration might not care about
how memory is distributed beyond a certain level while still wanting
to control how CPU cycles are distributed.
Thread Granularity
------------------
cgroup v1 allowed threads of a process to belong to different cgroups.
This didn't make sense for some controllers and those controllers
ended up implementing different ways to ignore such situations but
much more importantly it blurred the line between API exposed to
individual applications and system management interface.
Generally, in-process knowledge is available only to the process
itself; thus, unlike service-level organization of processes,
categorizing threads of a process requires active participation from
the application which owns the target process.
cgroup v1 had an ambiguously defined delegation model which got abused
in combination with thread granularity. cgroups were delegated to
individual applications so that they can create and manage their own
sub-hierarchies and control resource distributions along them. This
effectively raised cgroup to the status of a syscall-like API exposed
to lay programs.
First of all, cgroup has a fundamentally inadequate interface to be
exposed this way. For a process to access its own knobs, it has to
extract the path on the target hierarchy from /proc/self/cgroup,
construct the path by appending the name of the knob to the path, open
and then read and/or write to it. This is not only extremely clunky
and unusual but also inherently racy. There is no conventional way to
define transaction across the required steps and nothing can guarantee
that the process would actually be operating on its own sub-hierarchy.
cgroup controllers implemented a number of knobs which would never be
accepted as public APIs because they were just adding control knobs to
system-management pseudo filesystem. cgroup ended up with interface
knobs which were not properly abstracted or refined and directly
revealed kernel internal details. These knobs got exposed to
individual applications through the ill-defined delegation mechanism
effectively abusing cgroup as a shortcut to implementing public APIs
without going through the required scrutiny.
This was painful for both userland and kernel. Userland ended up with
misbehaving and poorly abstracted interfaces and kernel exposing and
locked into constructs inadvertently.
Competition Between Inner Nodes and Threads
-------------------------------------------
cgroup v1 allowed threads to be in any cgroups which created an
interesting problem where threads belonging to a parent cgroup and its
children cgroups competed for resources. This was nasty as two
different types of entities competed and there was no obvious way to
settle it. Different controllers did different things.
The cpu controller considered threads and cgroups as equivalents and
mapped nice levels to cgroup weights. This worked for some cases but
fell flat when children wanted to be allocated specific ratios of CPU
cycles and the number of internal threads fluctuated - the ratios
constantly changed as the number of competing entities fluctuated.
There also were other issues. The mapping from nice level to weight
wasn't obvious or universal, and there were various other knobs which
simply weren't available for threads.
The io controller implicitly created a hidden leaf node for each
cgroup to host the threads. The hidden leaf had its own copies of all
the knobs with ``leaf_`` prefixed. While this allowed equivalent
control over internal threads, it was with serious drawbacks. It
always added an extra layer of nesting which wouldn't be necessary
otherwise, made the interface messy and significantly complicated the
implementation.
The memory controller didn't have a way to control what happened
between internal tasks and child cgroups and the behavior was not
clearly defined. There were attempts to add ad-hoc behaviors and
knobs to tailor the behavior to specific workloads which would have
led to problems extremely difficult to resolve in the long term.
Multiple controllers struggled with internal tasks and came up with
different ways to deal with it; unfortunately, all the approaches were
severely flawed and, furthermore, the widely different behaviors
made cgroup as a whole highly inconsistent.
This clearly is a problem which needs to be addressed from cgroup core
in a uniform way.
Other Interface Issues
----------------------
cgroup v1 grew without oversight and developed a large number of
idiosyncrasies and inconsistencies. One issue on the cgroup core side
was how an empty cgroup was notified - a userland helper binary was
forked and executed for each event. The event delivery wasn't
recursive or delegatable. The limitations of the mechanism also led
to in-kernel event delivery filtering mechanism further complicating
the interface.
Controller interfaces were problematic too. An extreme example is
controllers completely ignoring hierarchical organization and treating
all cgroups as if they were all located directly under the root
cgroup. Some controllers exposed a large amount of inconsistent
implementation details to userland.
There also was no consistency across controllers. When a new cgroup
was created, some controllers defaulted to not imposing extra
restrictions while others disallowed any resource usage until
explicitly configured. Configuration knobs for the same type of
control used widely differing naming schemes and formats. Statistics
and information knobs were named arbitrarily and used different
formats and units even in the same controller.
cgroup v2 establishes common conventions where appropriate and updates
controllers so that they expose minimal and consistent interfaces.
Controller Issues and Remedies
------------------------------
Memory
~~~~~~
The original lower boundary, the soft limit, is defined as a limit
that is per default unset. As a result, the set of cgroups that
global reclaim prefers is opt-in, rather than opt-out. The costs for
optimizing these mostly negative lookups are so high that the
implementation, despite its enormous size, does not even provide the
basic desirable behavior. First off, the soft limit has no
hierarchical meaning. All configured groups are organized in a global
rbtree and treated like equal peers, regardless where they are located
in the hierarchy. This makes subtree delegation impossible. Second,
the soft limit reclaim pass is so aggressive that it not just
introduces high allocation latencies into the system, but also impacts
system performance due to overreclaim, to the point where the feature
becomes self-defeating.
The memory.low boundary on the other hand is a top-down allocated
reserve. A cgroup enjoys reclaim protection when it's within its
effective low, which makes delegation of subtrees possible. It also
enjoys having reclaim pressure proportional to its overage when
above its effective low.
The original high boundary, the hard limit, is defined as a strict
limit that can not budge, even if the OOM killer has to be called.
But this generally goes against the goal of making the most out of the
available memory. The memory consumption of workloads varies during
runtime, and that requires users to overcommit. But doing that with a
strict upper limit requires either a fairly accurate prediction of the
working set size or adding slack to the limit. Since working set size
estimation is hard and error prone, and getting it wrong results in
OOM kills, most users tend to err on the side of a looser limit and
end up wasting precious resources.
The memory.high boundary on the other hand can be set much more
conservatively. When hit, it throttles allocations by forcing them
into direct reclaim to work off the excess, but it never invokes the
OOM killer. As a result, a high boundary that is chosen too
aggressively will not terminate the processes, but instead it will
lead to gradual performance degradation. The user can monitor this
and make corrections until the minimal memory footprint that still
gives acceptable performance is found.
In extreme cases, with many concurrent allocations and a complete
breakdown of reclaim progress within the group, the high boundary can
be exceeded. But even then it's mostly better to satisfy the
allocation from the slack available in other groups or the rest of the
system than killing the group. Otherwise, memory.max is there to
limit this type of spillover and ultimately contain buggy or even
malicious applications.
Setting the original memory.limit_in_bytes below the current usage was
subject to a race condition, where concurrent charges could cause the
limit setting to fail. memory.max on the other hand will first set the
limit to prevent new charges, and then reclaim and OOM kill until the
new limit is met - or the task writing to memory.max is killed.
The combined memory+swap accounting and limiting is replaced by real
control over swap space.
The main argument for a combined memory+swap facility in the original
cgroup design was that global or parental pressure would always be
able to swap all anonymous memory of a child group, regardless of the
child's own (possibly untrusted) configuration. However, untrusted
groups can sabotage swapping by other means - such as referencing its
anonymous memory in a tight loop - and an admin can not assume full
swappability when overcommitting untrusted jobs.
For trusted jobs, on the other hand, a combined counter is not an
intuitive userspace interface, and it flies in the face of the idea
that cgroup controllers should account and limit specific physical
resources. Swap space is a resource like all others in the system,
and that's why unified hierarchy allows distributing it separately.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서의 지위와 전체 목차
1-99이 문서는 2015년 10월 Tejun Heo <tj@kernel.org>가 작성한 cgroup v2의 설계, interface와 convention에 관한 authoritative documentation입니다. Core와 controller별 동작을 포함해 userland에서 볼 수 있는 cgroup의 모든 측면을 설명하며, 앞으로의 모든 변경 사항도 이 문서에 반영해야 합니다.
Cgroup v1 문서는 `Documentation/admin-guide/cgroup-v1/index.rst`의 `cgroup-v1` reference에서 볼 수 있습니다. 이 문서에 새 section을 추가할 때에는 아래 contents에도 반드시 entry를 추가해야 합니다.
원문의 전체 목차를 번호와 계층을 유지해 한국어로 정리했습니다.
용어와 cgroup의 계층 모델
100-142`cgroup`은 `control group`을 뜻하며 절대로 대문자로 쓰지 않습니다. 단수형은 기능 전체를 가리킬 때와 `cgroup controllers`처럼 qualifier로 쓸 때 사용합니다. 여러 개의 개별 control group을 명시적으로 가리킬 때에는 복수형 `cgroups`를 사용합니다.
Cgroup은 process를 계층적으로 조직하고, 그 hierarchy를 따라 system resource를 제어 가능하고 설정 가능한 방식으로 분배하는 mechanism입니다.
Cgroup은 크게 core와 controllers 두 부분으로 구성됩니다. Cgroup core는 process의 계층적 조직을 주로 담당합니다. Cgroup controller는 대개 특정 system resource를 hierarchy를 따라 분배하지만, resource 분배 이외의 목적을 제공하는 utility controller도 있습니다.
Cgroup 기능을 이루는 두 축입니다.
Cgroups는 tree structure를 이루며 system의 모든 process는 정확히 하나의 cgroup에만 속합니다. 한 process의 모든 thread도 같은 cgroup에 속합니다. Process가 생성될 때에는 생성 시점의 parent process가 속한 cgroup에 배치됩니다.
Process를 다른 cgroup으로 migrate할 수 있지만, 이 migration은 이미 존재하는 descendant process에는 영향을 주지 않습니다.
정해진 structural constraint를 따르면 controller를 cgroup별로 선택해 enable하거나 disable할 수 있습니다. 모든 controller 동작은 hierarchical합니다. 어떤 cgroup에서 controller를 enable하면 그 cgroup을 포함한 sub-hierarchy의 cgroup에 속한 모든 process에 영향을 줍니다.
Nested cgroup에서 controller를 enable하면 resource distribution을 항상 더 제한합니다. Root에 더 가까운 곳에서 설정한 restriction을 더 먼 descendant가 override할 수 없습니다.
Ancestor 정책은 descendant로 내려갈수록 유지되거나 더 엄격해질 수만 있습니다.
단일 hierarchy mount와 controller binding
143-183V1과 달리 cgroup v2에는 hierarchy가 하나만 있습니다. 다음 명령으로 cgroup v2 hierarchy를 mount할 수 있습니다.
# mount -t cgroup2 none $MOUNT_POINT`cgroup2` filesystem의 magic number는 `0x63677270`(`"cgrp"`)입니다. V2를 지원하고 v1 hierarchy에 bind되지 않은 모든 controller는 자동으로 v2 hierarchy에 bind되어 root에 나타납니다.
V2 hierarchy에서 active하게 사용하지 않는 controller는 다른 hierarchy에 bind할 수 있습니다. 따라서 v2 hierarchy와 legacy v1 multiple hierarchy를 완전한 backward compatibility를 유지하며 함께 사용할 수 있습니다.
Controller는 현재 hierarchy에서 더 이상 reference되지 않을 때에만 hierarchy 사이를 이동할 수 있습니다. Per-cgroup controller state는 asynchronously 파괴되고 lingering reference가 남을 수 있으므로, 이전 hierarchy를 마지막으로 unmount한 직후에도 controller가 v2 hierarchy에 즉시 나타나지 않을 수 있습니다.
마찬가지로 unified hierarchy 밖으로 옮길 controller는 완전히 disable해야 하며, disable된 controller가 다른 hierarchy에서 사용 가능해질 때까지 시간이 걸릴 수 있습니다. Controller 사이의 dependency 때문에 다른 controller까지 disable해야 할 수도 있습니다.
Controller state와 reference가 완전히 정리되어야 다른 hierarchy에서 사용할 수 있습니다.
V2와 다른 hierarchy 사이에서 controller를 동적으로 옮기는 기능은 development와 manual configuration에는 유용하지만 production에서는 강하게 권장하지 않습니다. System boot 뒤 controller 사용을 시작하기 전에 hierarchy와 controller association을 결정하는 것이 좋습니다.
V2로 전환하는 동안 system-management software가 boot 과정에서 v1 cgroup filesystem을 자동 mount해 수동 개입 전에 모든 controller를 가져갈 수 있습니다. Test와 experiment를 쉽게 하려면 kernel parameter `cgroup_no_v1=`로 v1의 controller를 disable해 항상 v2에서 사용할 수 있게 할 수 있습니다.
Cgroup v2 mount options
184-255Cgroup v2는 현재 다음 mount option을 지원합니다.
각 option이 바꾸는 동작과 적용 범위입니다.
`memory_recursiveprot`의 동작은 원래 default여야 했지만, 상위 tree level에 부정확하게 큰 `bypass` protection 값을 두는 등 기존 semantics에 의존하는 설정을 regress하지 않도록 mount option으로 제공됩니다.
HugeTLB를 memory controller accounting에 넣을 때 고려할 네 가지 사항입니다.
Pool allocation과 실제 cgroup charge는 서로 다른 시점에 일어납니다.
Process 조직과 migration lifecycle
256-310처음에는 모든 process가 속한 root cgroup만 존재합니다. Subdirectory를 만들면 child cgroup을 생성할 수 있습니다.
# mkdir $CGROUP_NAME하나의 cgroup은 여러 child cgroup을 가져 tree structure를 이룰 수 있습니다. 각 cgroup에는 read/write 가능한 `cgroup.procs` interface file이 있습니다. 이 file을 읽으면 해당 cgroup에 속한 모든 process PID가 한 줄에 하나씩 표시됩니다.
PID는 정렬되지 않습니다. 읽는 동안 process가 다른 cgroup으로 이동했다가 돌아오거나 PID가 recycle되면 같은 PID가 두 번 이상 나타날 수 있습니다.
Target cgroup의 `cgroup.procs`에 PID를 쓰면 process를 그 cgroup으로 migrate합니다. 한 번의 `write(2)` call로는 process 하나만 이동할 수 있습니다. Process가 여러 thread로 구성되어 있으면 어느 thread의 PID를 쓰더라도 process의 모든 thread가 함께 이동합니다.
Process가 child를 fork하면 새 process는 fork 시점에 parent가 속한 cgroup에서 태어납니다. Exit한 process는 reap될 때까지 exit 당시 cgroup과 연결되어 있습니다. 다만 zombie process는 `cgroup.procs`에 나타나지 않으므로 다른 cgroup으로 옮길 수 없습니다.
생성, migration, exit와 reap 단계에서 cgroup membership이 어떻게 유지되는지 보여 줍니다.
Child cgroup과 live process가 없는 cgroup은 directory를 제거해 파괴할 수 있습니다. Child가 없고 zombie process만 연결된 cgroup도 empty로 간주하므로 제거할 수 있습니다.
# rmdir $CGROUP_NAME`/proc/$PID/cgroup`은 process의 cgroup membership을 나열합니다. System에서 legacy cgroup도 사용하면 hierarchy마다 한 줄씩 여러 entry가 있을 수 있습니다. Cgroup v2 entry는 항상 `0::$PATH` 형식입니다.
# cat /proc/842/cgroup
...
0::/test-cgroup/test-cgroup-nestedProcess가 zombie가 된 뒤 그 process와 연결된 cgroup이 제거되면 path 뒤에 ` (deleted)`가 붙습니다.
# cat /proc/842/cgroup
...
0::/test-cgroup/test-cgroup-nested (deleted)Process membership을 조작하거나 확인하는 주요 interface입니다.
Thread mode와 threaded domain topology
311-380Cgroup v2는 process group의 thread 사이에 resource를 계층적으로 분배해야 하는 use case를 위해 일부 controller에서 thread granularity를 지원합니다. Default로 한 process의 모든 thread는 같은 cgroup에 속하며, 이 cgroup은 process나 thread에 특정되지 않은 resource consumption을 수용하는 resource domain이기도 합니다.
Thread mode를 사용하면 thread를 subtree 여러 곳에 나누어 놓으면서도 공통 resource domain을 유지할 수 있습니다. Thread mode를 지원하는 controller는 `threaded controller`, 지원하지 않는 controller는 `domain controller`라고 합니다.
Thread-aware resource 분배에 등장하는 용어입니다.
Cgroup을 threaded로 표시하면 threaded cgroup으로서 parent의 resource domain에 합류합니다. Parent도 threaded일 수 있으며, 이 경우 실제 resource domain은 hierarchy 더 위에 있습니다. Threaded subtree의 root, 즉 threaded가 아닌 가장 가까운 ancestor를 `threaded domain` 또는 `thread root`라고 하며 전체 subtree의 resource domain 역할을 합니다.
여러 threaded cgroup이 가장 가까운 non-threaded ancestor의 domain을 공유합니다.
Threaded subtree 안에서는 한 process의 thread를 서로 다른 cgroup에 둘 수 있으며 no-internal-process constraint를 적용받지 않습니다. 따라서 non-leaf cgroup에 thread가 있든 없든 threaded controller를 enable할 수 있습니다.
Threaded domain cgroup은 subtree의 모든 domain resource consumption을 수용하므로 실제 process가 없어도 internal consumption이 있는 것으로 간주됩니다. 그 결과 threaded가 아닌 populated child cgroup을 가질 수 없습니다. Root cgroup은 no-internal-process constraint를 받지 않으므로 threaded domain인 동시에 domain cgroup의 parent가 될 수 있습니다.
현재 cgroup의 operation mode 또는 type은 `cgroup.type`에 나타납니다. 이 file은 normal domain, threaded subtree의 domain 역할을 하는 domain, threaded cgroup 가운데 어느 상태인지 보여 줍니다.
Thread topology에서 관찰할 수 있는 type과 상태입니다.
Cgroup은 생성될 때 항상 domain이며 `cgroup.type`에 `threaded`를 쓰면 threaded로 바꿀 수 있습니다. 이 변경은 한 방향뿐이므로 한 번 threaded가 되면 다시 domain으로 바꿀 수 없습니다.
# echo threaded > cgroup.typeThread mode를 enable하려면 cgroup이 parent resource domain에 합류할 수 있어야 합니다. Parent는 valid (threaded) domain이거나 threaded cgroup이어야 합니다. Parent가 unthreaded domain이면 domain controller가 enable되어 있지 않고 populated domain child도 없어야 합니다. Root는 이 요구 사항에서 제외됩니다.
Topology 관점에서는 invalid cgroup이 생길 수 있습니다. 원문의 예에서 `A`는 threaded domain, `B`는 threaded이고, 그 아래 새로 생성된 `C`는 아직 domain입니다.
새 child는 domain으로 태어나지만 threaded parent는 domain child를 수용할 수 없습니다.
`C`는 domain child를 수용할 수 있는 parent에 연결되어 있지 않으므로 threaded cgroup으로 바꾸기 전까지 사용할 수 없습니다. 이 경우 `cgroup.type`은 `domain (invalid)`를 보고하고 invalid topology로 실패한 operation은 errno `EOPNOTSUPP`를 사용합니다.
Child 중 하나가 threaded가 되거나, cgroup 안에 process가 있는 상태에서 `cgroup.subtree_control`에 threaded controller를 enable하면 domain cgroup은 threaded domain으로 바뀝니다. 이 조건이 사라지면 threaded domain은 normal domain으로 돌아갑니다.
Thread interface와 resource accounting
381-416`cgroup.threads`를 읽으면 cgroup의 모든 thread ID가 나옵니다. Operation 단위가 process 대신 thread라는 점을 제외하면 `cgroup.procs`와 형식과 동작이 같습니다.
`cgroup.threads`는 어느 cgroup에서나 쓸 수 있지만 같은 threaded domain 안에서만 thread를 옮길 수 있으므로 각 threaded subtree 내부로 operation 범위가 제한됩니다.
Threaded domain cgroup이 전체 subtree의 resource domain 역할을 합니다. Thread가 subtree 여러 곳에 흩어져 있어도 모든 process는 threaded domain cgroup에 있는 것으로 간주됩니다.
Threaded subtree 안에서 두 migration interface의 의미입니다.
Threaded domain의 `cgroup.procs`에는 subtree의 모든 process PID가 들어 있고 subtree proper에서는 읽을 수 없습니다. 하지만 subtree 어디에서나 `cgroup.procs`에 쓸 수 있으며 matching process의 모든 thread를 해당 cgroup으로 migrate합니다.
Threaded subtree에서는 threaded controller만 enable할 수 있습니다. Threaded controller를 enable하면 그 cgroup과 descendant thread에 연결된 resource consumption만 account하고 control합니다. 특정 thread에 연결되지 않은 consumption은 모두 threaded domain cgroup에 속합니다.
Thread-specific consumption과 domain consumption의 owner가 갈립니다.
Threaded subtree는 no-internal-process constraint에서 제외되므로 threaded controller는 non-leaf cgroup의 thread와 child cgroup 사이의 경쟁을 처리할 수 있어야 합니다. 각 threaded controller가 이 경쟁을 처리하는 방식을 정의합니다.
현재 threaded cgroup에서 enable할 수 있는 threaded controller는 `cpu`, `cpuset`, `perf_event`, `pids`입니다.
현재 thread mode를 지원하는 controller 목록입니다.
Recursive populated notification
417-439모든 non-root cgroup에는 sub-hierarchy에 live process가 있는지를 나타내는 `populated` field가 든 `cgroup.events` file이 있습니다. 해당 cgroup과 descendant에 live process가 없으면 값은 `0`, 하나라도 있으면 `1`입니다.
값이 바뀌면 `poll`과 `[id]notify` event가 trigger됩니다. 이를 이용하면 특정 sub-hierarchy의 모든 process가 exit한 뒤 cleanup operation을 시작할 수 있습니다. Populated 상태 update와 notification은 recursive합니다.
원문의 A-B-C/D hierarchy와 각 cgroup의 process count·populated 값을 구조화했습니다.
C의 마지막 process가 exit하면 ancestor B에도 상태 변화가 전달됩니다.
예제에서 `A(4) - B(0) - C(1)`이고 B 아래에 `D(0)`도 있습니다. A, B, C의 `populated`는 `1`, D는 `0`입니다. C의 process 하나가 exit하면 B와 C의 값이 `0`으로 바뀌고 두 cgroup의 `cgroup.events`에 file-modified event가 발생합니다.
Controller availability와 enable·disable
440-493Controller가 kernel에서 지원되고, 즉 compile되어 있으며 disable되지 않았고 v1 hierarchy에 attach되지 않았으며, `cgroup.controllers`에 나열되어 있으면 해당 cgroup에서 available합니다.
Available하다는 것은 controller interface file이 cgroup directory에 노출되어 그 cgroup 안에서 target resource distribution을 관찰하거나 제어할 수 있다는 뜻입니다.
네 조건이 모두 만족되어야 cgroup에서 controller를 enable할 수 있습니다.
각 cgroup의 `cgroup.controllers`는 해당 cgroup에서 enable할 수 있는 available controller를 모두 나열합니다.
# cat cgroup.controllers
cpu io memoryDefault로 enable된 controller는 없습니다. `cgroup.subtree_control`에 `+controller` 또는 `-controller`를 써서 enable하거나 disable합니다.
# echo "+cpu +memory -io" > cgroup.subtree_control`cgroup.controllers`에 나열된 controller만 enable할 수 있습니다. 한 write에 여러 operation을 지정하면 모두 성공하거나 모두 실패합니다. 같은 controller에 여러 operation을 지정하면 마지막 operation이 유효합니다.
Cgroup에서 controller를 enable한다는 것은 immediate child 사이의 target resource distribution을 그 cgroup이 control한다는 뜻입니다.
원문의 A-B-C/D hierarchy에서 CPU와 memory 경쟁 범위를 구조화했습니다.
예제 hierarchy에서 A는 `cpu`와 `memory`를 enable해 child B에 주는 CPU cycle과 memory를 control합니다. B는 `memory`만 enable하므로 C와 D는 CPU cycle을 자유롭게 경쟁하지만 B에 주어진 memory를 나누는 방식은 control됩니다.
Controller가 resource를 child에 분배하므로 controller를 enable하면 child cgroup에 해당 controller interface file이 생성됩니다. B에서 `cpu`를 enable하면 C와 D에 `cpu.` prefix file이 생기고, B에서 `memory`를 disable하면 C와 D의 `memory.` prefix file이 제거됩니다.
따라서 `cgroup.`으로 시작하지 않는 controller interface file은 그 file이 보이는 cgroup 자체가 아니라 parent가 소유합니다.
Parent의 subtree_control operation이 child directory를 바꿉니다.
Top-down과 no-internal-process constraint
494-535Resource는 top-down으로 분배됩니다. Cgroup은 parent에게서 분배받은 resource만 child에 다시 분배할 수 있습니다.
Parent가 enable한 controller만 descendant가 이어서 enable할 수 있습니다.
그러므로 root가 아닌 `cgroup.subtree_control`에는 parent의 `cgroup.subtree_control`에서 enable한 controller만 들어갈 수 있습니다. Parent가 enable한 controller만 child가 enable할 수 있고, child 하나 이상이 enable한 controller는 parent가 disable할 수 없습니다.
Non-root cgroup은 자기 process가 하나도 없을 때에만 domain resource를 child에 분배할 수 있습니다. 즉 process가 없는 domain cgroup만 `cgroup.subtree_control`에 domain controller를 enable할 수 있습니다.
이 규칙은 domain controller가 enable된 hierarchy 영역에서 process가 항상 leaf에만 있도록 보장합니다. 따라서 child cgroup과 parent 내부 process가 서로 경쟁하는 상황을 배제합니다.
Domain controller를 child에 분배할 때의 규칙과 예외입니다.
Root cgroup은 이 restriction에서 제외됩니다. Root에는 다른 cgroup에 연결할 수 없는 process와 anonymous resource consumption이 있어 대부분 controller의 특별 처리가 필요합니다. Root resource consumption을 어떻게 제어할지는 각 controller가 정하며 Controllers 장의 Non-normative information 절에서 더 설명합니다.
Cgroup의 `cgroup.subtree_control`에 enable된 controller가 없으면 이 restriction은 방해가 되지 않습니다. 그렇지 않으면 populated cgroup 아래에 child를 만드는 것 자체가 불가능하기 때문에 중요한 예외입니다.
Cgroup의 resource distribution을 control하려면 child를 만들고 자기 process를 모두 child로 옮긴 다음 `cgroup.subtree_control`에 controller를 enable해야 합니다.
Internal process를 leaf child로 옮긴 뒤에야 domain controller를 enable합니다.
Delegation 방식과 권한 경계
536-570Cgroup은 두 가지 방식으로 delegate할 수 있습니다. 첫째, directory와 그 안의 `cgroup.procs`, `cgroup.threads`, `cgroup.subtree_control` file에 대한 write access를 less-privileged user에게 부여합니다. 둘째, `nsdelegate` mount option이 설정되어 있으면 cgroup namespace를 만들 때 자동으로 delegate합니다.
명시적 file permission과 namespace 생성 기반 위임은 최종적으로 같은 권한 모델을 만듭니다.
어떤 directory의 resource-control interface file은 parent resource의 distribution을 제어하므로 delegatee가 이 file에 write할 수 없어야 합니다. 첫 번째 방식에서는 해당 file access를 주지 않아 이를 보장합니다.
두 번째 방식에서는 최소한 mount namespacing으로 namespace 밖 file을 delegatee에게 숨겨야 합니다. Kernel은 cgroup namespace 안에서 namespace root의 모든 file write를 거부하지만, `/sys/kernel/cgroup/delegate`에 나열된 `cgroup.procs`, `cgroup.threads`, `cgroup.subtree_control` 등의 file은 예외입니다.
Delegatee는 받은 resource를 subtree 안에서만 재분배하며 parent restriction을 벗어날 수 없습니다.
두 delegation type의 최종 결과는 같습니다. 위임이 끝나면 user는 directory 아래에 sub-hierarchy를 만들고 process를 원하는 방식으로 조직하며 parent에게 받은 resource를 더 세분해 분배할 수 있습니다.
모든 resource controller의 limit과 다른 setting은 hierarchical합니다. Delegated sub-hierarchy 안에서 어떤 일을 하더라도 parent가 부과한 resource restriction을 벗어날 수 없습니다.
현재 cgroup은 delegated sub-hierarchy의 cgroup 수나 nesting depth를 제한하지 않습니다. 다만 향후 명시적 limit이 추가될 수 있습니다.
Delegated subtree의 migration containment
571-612Delegated sub-hierarchy는 delegatee가 process를 subtree 밖에서 안으로 끌어오거나 안에서 밖으로 내보낼 수 없다는 의미에서 containment됩니다.
Less-privileged user에게 delegate한 경우 non-root euid process가 target PID를 `cgroup.procs`에 써서 migrate하려면 다음 두 조건을 만족해야 합니다.
Writer는 destination file뿐 아니라 source와 destination의 common ancestor에도 권한이 있어야 합니다.
첫째 writer는 destination의 `cgroup.procs`에 write access가 있어야 합니다. 둘째 source cgroup과 destination cgroup의 common ancestor에 있는 `cgroup.procs`에도 write access가 있어야 합니다.
이 두 constraint 덕분에 delegatee는 delegated sub-hierarchy 안에서 process를 자유롭게 옮길 수 있지만 외부 process를 끌어오거나 내부 process를 외부로 밀어낼 수 없습니다.
예제에서는 user `U0`에게 cgroup `C0`과 `C1`을 delegate했습니다. U0는 C0 아래에 `C00`, `C01`을 만들고 C1 아래에 `C10`을 만들었으며, C0와 C1 아래의 모든 process는 U0 소유입니다.
원문의 ASCII hierarchy와 C10에서 C00으로 이동하려는 경로를 구조화했습니다.
U0가 현재 C10에 있는 process PID를 `C00/cgroup.procs`에 쓰려 한다고 가정합니다. U0는 destination file에는 write access가 있지만 C10과 C00의 common ancestor가 두 delegation point보다 위에 있고 그 ancestor의 `cgroup.procs`에는 write access가 없습니다. 따라서 write는 `-EACCES`로 거부됩니다.
Namespace delegation에서는 migration을 시도하는 process의 namespace에서 source와 destination cgroup이 모두 reachable해야 containment가 보장됩니다. 둘 중 하나라도 reachable하지 않으면 migration은 `-ENOENT`로 거부됩니다.
Delegation 방식별 boundary 위반의 검사와 error입니다.
한 번 조직하고 이름 충돌 피하기
613-651Process를 cgroup 사이에서 migrate하는 operation은 비교적 비싸고 memory 같은 stateful resource가 process와 함께 이동하지 않습니다. Migration과 여러 hot path 사이에는 synchronization cost에 관한 본질적 trade-off가 자주 존재하므로 이는 명시적인 design decision입니다.
따라서 서로 다른 resource restriction을 적용하려고 process를 cgroup 사이에서 자주 migrate하는 방식은 권장하지 않습니다. Workload는 시작할 때 system의 logical structure와 resource structure에 따라 한 번 cgroup에 배치해야 합니다.
Resource distribution을 동적으로 조정해야 할 때에는 process를 옮기는 대신 interface file을 통해 controller configuration을 바꿉니다.
Workload placement와 runtime tuning을 분리하는 권장 pattern입니다.
Cgroup interface file과 child cgroup은 같은 directory를 사용하므로 interface file과 충돌하는 이름의 child cgroup을 만들 수 있습니다.
Core와 controller prefix 및 workload-category 단어 회피 규칙입니다.
모든 cgroup core interface file은 `cgroup.` prefix를 사용하고 각 controller interface는 controller name과 dot를 prefix로 사용합니다. Controller name은 lowercase alphabet과 `_`로 구성되지만 `_`로 시작하지 않으므로 `_`를 collision-avoidance prefix character로 사용할 수 있습니다.
또한 interface filename은 workload를 분류할 때 자주 쓰는 `job`, `service`, `slice`, `unit`, `workload` 같은 term으로 시작하거나 끝나지 않습니다.
Cgroup 자체는 name collision을 막지 않으므로 이를 피하는 책임은 user에게 있습니다.
두 guideline이 방지하려는 비용과 충돌입니다.
Resource distribution model과 Weights
652-681Cgroup controller는 resource type과 예상 use case에 따라 여러 resource distribution scheme을 구현합니다. 이 절은 주요 scheme과 기대 동작을 설명합니다.
Weight model에서는 active child의 weight를 모두 더하고, 각 child의 weight가 합계에서 차지하는 비율만큼 parent resource를 나눕니다. 현재 resource를 사용할 수 있는 child만 분배에 참여하므로 work-conserving입니다.
참여 child가 동적으로 바뀌는 특성 때문에 이 model은 보통 stateless resource에 사용합니다.
Active child만 합계와 비율 계산에 참여합니다.
모든 weight 범위는 `[1, 10000]`이고 default는 `100`입니다. 직관적인 범위를 유지하면서도 양쪽 방향의 대칭적 multiplicative bias를 충분히 세밀하게 표현할 수 있습니다.
Weight가 범위 안에 있는 한 모든 configuration combination이 valid하므로 configuration change나 process migration을 거부할 이유가 없습니다.
`cpu.weight`는 active child에 CPU cycle을 비례 분배하는 대표 예입니다.
Weight scheme의 범위, default와 대표 interface입니다.
Limits와 Protections
682-722Limit model에서 child는 설정된 양까지만 resource를 사용할 수 있습니다. Child limit의 합이 parent가 사용할 수 있는 resource보다 커도 되므로 limit은 overcommit할 수 있습니다.
Limit 범위는 `[0, max]`이고 default `max`는 아무 제한도 하지 않는 noop입니다. Overcommit할 수 있으므로 모든 configuration combination이 valid하며 configuration change나 process migration을 거부할 이유가 없습니다.
`io.max`는 cgroup이 IO device에서 소비할 수 있는 maximum BPS 및/또는 IOPS를 제한하는 예입니다.
Protection model에서는 모든 ancestor usage가 각자의 protected level 아래에 있는 동안 설정된 resource 양까지 cgroup을 보호합니다. Protection은 hard guarantee일 수도 있고 best-effort soft boundary일 수도 있습니다.
Protection도 overcommit할 수 있으며, 이 경우 child 사이에서 parent에게 실제 available한 양까지만 보호됩니다.
Cgroup protection은 모든 ancestor가 자기 protected level 아래에 있을 때 유효합니다.
Protection 범위는 `[0, max]`이고 default `0`은 noop입니다. Overcommit이 허용되므로 모든 configuration combination이 valid하고 configuration change나 process migration을 거부할 이유가 없습니다.
`memory.low`는 best-effort memory protection을 구현하는 대표 예입니다.
두 model은 모두 overcommit을 허용하지만 설정의 의미와 default가 다릅니다.
Exclusive Allocations
723-742Allocation model에서는 finite resource의 일정량을 cgroup에 exclusive하게 할당합니다. Child allocation 합이 parent에게 available한 resource를 넘을 수 없으므로 allocation은 overcommit할 수 없습니다.
Allocation 범위는 `[0, max]`이고 default `0`은 resource가 하나도 할당되지 않았다는 뜻입니다.
Overcommit할 수 없으므로 일부 configuration combination은 invalid하며 거부해야 합니다. 해당 resource가 process execution에 필수라면 process migration도 거부될 수 있습니다.
`cpu.rt.max`는 realtime slice를 hard allocation하는 대표 예입니다.
Exclusive finite-resource assignment의 validation 특성입니다.
Child allocation 합계를 parent available amount와 비교합니다.
Interface file format
743-784가능한 경우 모든 cgroup interface file은 다음 형식 중 하나를 사용해야 합니다.
New-line separated values
(when only one value can be written at once)
VAL0\n
VAL1\n
...
Space separated values
(when read-only or multiple values can be written at once)
VAL0 VAL1 ...\n
Flat keyed
KEY0 VAL0\n
KEY1 VAL1\n
...
Nested keyed
KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
...원문의 네 형식을 write cardinality와 record 구조로 정리했습니다.
Writable file은 일반적으로 read와 write 형식이 같아야 합니다. 다만 controller는 뒤쪽 field 생략을 허용하거나 가장 흔한 use case를 위한 제한된 shortcut을 구현할 수 있습니다.
Flat-keyed와 nested-keyed file 모두 한 번에 key 하나의 value만 쓸 수 있습니다. Nested-keyed file에서는 sub-key pair 순서를 자유롭게 지정할 수 있고 모든 pair를 다 쓸 필요도 없습니다.
Interface naming과 value convention
785-856단일 feature의 setting은 file 하나에 담아야 합니다. Root cgroup은 resource control에서 제외되어야 하므로 resource-control interface file을 가져서는 안 됩니다.
Default time unit은 microsecond입니다. 다른 unit을 사용한다면 반드시 명시적 unit suffix를 붙여야 합니다. Parts-per quantity는 소수부가 최소 두 자리인 percentage decimal, 예를 들어 `13.40`을 사용해야 합니다.
Time, ratio, weight와 guarantee/limit naming 규칙입니다.
Weight-based distribution을 구현한 controller는 interface file 이름을 `weight`로 하고 범위 `[1, 10000]`, default `100`을 사용해야 합니다. Default를 100%로 보는 직관적인 범위 안에서 양방향으로 충분하고 대칭적인 bias를 허용하기 위한 값입니다.
Absolute resource guarantee와 limit은 각각 `min`, `max`, best-effort guarantee와 limit은 각각 `low`, `high`라는 file 이름을 사용해야 합니다. 이 네 control file에서 위쪽 무한대를 나타낼 때에는 read와 write 모두 special token `max`를 사용합니다.
Configurable default와 key별 override가 있는 setting은 default entry의 key를 `default`로 지정하고 file의 첫 entry에 놓아야 합니다. Default value는 `default $VAL` 또는 `$VAL`을 써서 update할 수 있습니다.
특정 override를 update할 때 value로 `default`를 쓰면 해당 override를 제거한다는 뜻입니다. 읽을 때에는 value가 `default`인 override entry가 나타나서는 안 됩니다.
Default update, device override 생성과 제거가 file readback에 반영되는 방식입니다.
Major:minor device number를 key로 하고 integer value를 저장하는 예제는 처음에 default `150`, device `8:0` override `300`을 보여 줍니다.
# cat cgroup-example-interface-file
default 150
8:0 300다음 두 write는 모두 default를 `125`로 update합니다.
# echo 125 > cgroup-example-interface-file # echo "default 125" > cgroup-example-interface-fileDevice `8:16`의 override를 `170`으로 설정할 수 있습니다.
# echo "8:16 170" > cgroup-example-interface-file`8:0` override를 `default` write로 제거한 뒤 file에는 default `125`와 `8:16 170`만 남습니다.
# echo "8:0 default" > cgroup-example-interface-file
# cat cgroup-example-interface-file
default 125
8:16 170빈도가 매우 높지 않은 event에는 event key-value pair를 나열하는 `events` interface file을 만들어야 합니다. Notify 가능한 event가 발생할 때마다 이 file에 file-modified event를 생성해야 합니다.
Interface convention이 사용자에게 제공하는 일관성입니다.
Counter update와 userspace notification의 순서입니다.
cgroup.type과 cgroup.procs
857-909모든 cgroup core file에는 `cgroup.` prefix가 붙습니다.
`cgroup.type`은 non-root cgroup에 존재하는 read/write single-value file입니다. 읽으면 cgroup의 current type을 보고합니다.
Core interface가 보고하는 네 cgroup type입니다.
`cgroup.type`에 `threaded`를 쓰면 cgroup을 threaded cgroup으로 바꿀 수 있습니다.
`cgroup.procs`는 모든 cgroup에 존재하는 read/write new-line-separated-values file입니다. 읽으면 cgroup에 속한 process PID를 한 줄에 하나씩 나열합니다. PID는 정렬되지 않으며 migration 왕복이나 read 중 PID recycling 때문에 같은 PID가 여러 번 나올 수 있습니다.
PID를 쓰면 해당 process를 cgroup으로 migrate합니다. Writer는 `cgroup.procs` file과 source/destination common ancestor의 `cgroup.procs` file 모두에 write access가 있어야 합니다.
Read, write와 delegation 시의 동작입니다.
Sub-hierarchy를 delegate할 때에는 containing directory와 함께 이 file의 write access도 부여해야 합니다.
Threaded cgroup에서는 모든 process가 thread root에 속하므로 `cgroup.procs` read가 `EOPNOTSUPP`로 실패합니다. Write는 지원되며 process의 모든 thread를 해당 cgroup으로 이동합니다.
Thread migration과 controller set
910-956`cgroup.threads`는 모든 cgroup에 있는 read/write new-line-separated-values file입니다. 읽으면 cgroup에 속한 thread TID를 한 줄에 하나씩 나열합니다. TID는 정렬되지 않고 migration 왕복이나 read 중 TID recycling으로 같은 TID가 여러 번 나올 수 있습니다.
TID를 쓰면 해당 thread를 cgroup으로 migrate합니다. Writer는 `cgroup.threads`에 write access가 있어야 하고, source와 destination cgroup이 같은 resource domain에 있어야 하며, 두 cgroup의 common ancestor `cgroup.procs`에도 write access가 있어야 합니다.
Sub-hierarchy를 delegate할 때에는 containing directory와 함께 `cgroup.threads` write access도 부여해야 합니다.
세 core file의 access mode, format과 핵심 의미입니다.
`cgroup.controllers`는 모든 cgroup에 있는 read-only space-separated-values file이며, 해당 cgroup에서 available한 controller를 순서 보장 없이 나열합니다.
`cgroup.subtree_control`은 모든 cgroup에 존재하는 read/write space-separated-values file이며 처음에는 비어 있습니다. 읽으면 cgroup에서 child로 resource를 분배하도록 enable된 controller를 나열합니다.
`+` 또는 `-` prefix가 붙은 controller 목록을 쓰면 enable 또는 disable합니다. 같은 controller가 여러 번 나오면 마지막 항목이 유효합니다. 한 write에 여러 enable/disable operation이 있으면 모두 성공하거나 모두 실패합니다.
Controller list parsing과 commit 규칙입니다.
Events, descendant limit와 cgroup.stat
957-1008`cgroup.events`는 non-root cgroup에 존재하는 read-only flat-keyed file입니다. 별도 설명이 없으면 이 file의 value가 바뀔 때 file-modified event가 생성됩니다.
Live process와 freezer 상태를 알리는 field입니다.
`cgroup.max.descendants`는 read/write single-value file이고 default는 `max`입니다. 허용할 descendant cgroup의 maximum number를 정합니다. 실제 descendant 수가 이 값 이상이면 hierarchy에서 새 cgroup 생성이 실패합니다.
`cgroup.max.depth`도 read/write single-value file이고 default `max`입니다. Current cgroup 아래에서 허용할 maximum descendant depth를 정합니다. 실제 depth가 값 이상이면 새 child cgroup 생성이 실패합니다.
Descendant count와 depth limit의 실패 조건입니다.
`cgroup.stat`은 read-only flat-keyed file입니다. `nr_descendants`는 visible descendant cgroup의 total number입니다.
`nr_dying_descendants`는 dying descendant cgroup의 total number입니다. User가 cgroup을 삭제하면 dying 상태가 되고 system load 등에 따라 정해지지 않은 시간 동안 이 상태에 머문 뒤 완전히 파괴됩니다.
어떤 상황에서도 process는 dying cgroup에 들어갈 수 없고 dying cgroup은 revive할 수 없습니다. Dying cgroup은 삭제 시점에 active하던 limit을 넘지 않는 범위에서 system resource를 소비할 수 있습니다.
`nr_subsys_<cgroup_subsys>`는 current cgroup과 그 아래의 live cgroup subsystem, 예를 들어 memory cgroup의 total number입니다. `nr_dying_subsys_<cgroup_subsys>`는 같은 범위의 dying cgroup subsystem total입니다.
Hierarchy와 subsystem lifecycle counter입니다.
Local frozen time과 cgroup.freeze
1009-1052`cgroup.stat.local`은 non-root cgroup에 존재하는 read-only flat-keyed file입니다. `frozen_usec`는 self 또는 ancestor group에 의해 freeze되었는지와 관계없이 이 cgroup이 freezing 시작부터 thawing까지 보낸 누적 시간입니다.
실제로 `frozen` 상태에 도달했는지 또는 도달하지 못했는지는 이 counter에서 따로 account하지 않습니다. 원문의 freezer-state ASCII representation에서 측정 시간은 point `a`부터 `c`까지입니다. 즉 frozen bit가 올라가는 `b`보다 앞에서 시작하고 내려가는 `d`보다 앞에서 끝납니다.
원문의 파형을 state transition과 측정 구간으로 재구성했습니다.
`cgroup.freeze`는 non-root cgroup의 read/write single-value file입니다. 허용값은 `0`과 `1`, default는 `0`입니다.
`1`을 쓰면 cgroup과 모든 descendant를 freeze합니다. 모든 process가 정지해 명시적으로 unfreeze할 때까지 실행되지 않습니다. Freeze 완료에는 시간이 걸릴 수 있으며, 완료되면 `cgroup.events`의 `frozen`이 `1`로 update되고 notification이 발생합니다.
Cgroup은 자기 setting 또는 어느 ancestor setting으로도 freeze될 수 있습니다. Ancestor 중 하나라도 frozen이면 해당 cgroup은 계속 frozen 상태입니다.
Request, completion notification과 ancestor dominance를 보여 줍니다.
Frozen cgroup의 process도 fatal signal로 kill할 수 있습니다. User의 명시적 move 또는 freeze와 `fork()`의 race 때문에 frozen cgroup에 들어가거나 나올 수도 있습니다. Frozen cgroup으로 이동한 process는 정지하고 밖으로 이동한 process는 실행 상태가 됩니다.
Cgroup의 frozen status는 cgroup tree operation에 영향을 주지 않습니다. Frozen 상태라도 empty cgroup을 삭제하거나 새 sub-cgroup을 만들 수 있습니다.
Freeze가 process와 tree operation에 미치는 영향입니다.
cgroup.kill과 PSI pressure
1053-1090`cgroup.kill`은 non-root cgroup에 있는 write-only single-value file이며 허용값은 `1`뿐입니다.
`1`을 쓰면 해당 cgroup과 모든 descendant cgroup을 kill합니다. 영향받는 cgroup tree의 모든 process에 `SIGKILL`을 전달합니다. Cgroup tree kill은 concurrent fork를 올바르게 처리하고 migration으로부터 보호됩니다.
Threaded cgroup에서 이 file에 쓰면 `EOPNOTSUPP`로 실패합니다. Cgroup kill은 process-directed operation이므로 thread-group 전체에 영향을 주기 때문입니다.
Concurrent fork와 migration을 고려하며 process tree 전체를 종료합니다.
`cgroup.pressure`는 read/write single-value file이며 허용값은 `0`과 `1`, default는 `1`입니다. `0`을 쓰면 cgroup PSI accounting을 disable하고 `1`을 쓰면 다시 enable합니다.
이 control attribute는 hierarchical하지 않습니다. 한 cgroup에서 PSI accounting을 disable하거나 enable해도 descendant PSI에는 영향을 주지 않으며 root에서 ancestor를 따라 enablement를 전달할 필요도 없습니다.
PSI는 cgroup마다 stall을 별도로 account하고 hierarchy 각 level에서 aggregate하므로 deep hierarchy의 일부 workload에는 무시할 수 없는 overhead가 생길 수 있습니다. 이때 non-leaf cgroup의 PSI accounting을 disable하는 데 `cgroup.pressure`를 사용할 수 있습니다.
각 cgroup의 pressure accounting switch는 descendant와 독립적입니다.
`irq.pressure`는 read/write nested-keyed file이며 IRQ/SOFTIRQ의 pressure stall information을 보여 줍니다. 자세한 내용은 `Documentation/accounting/psi.rst`의 `psi` reference를 참조하십시오.
마지막 세 core interface의 access, default와 핵심 동작입니다.
CPU controller model과 realtime 제약
1091-1126`cpu` controller는 CPU cycle distribution을 조절합니다. Normal scheduling policy에는 weight model과 absolute bandwidth limit model을 구현하고, realtime scheduling policy에는 absolute bandwidth allocation model을 구현합니다.
Scheduling policy별 CPU controller distribution model입니다.
모든 model에서 cycle distribution은 시간만을 기준으로 정의되고 task가 실행되는 frequency는 account하지 않습니다.
Optional utilization-clamping support는 `schedutil` cpufreq governor에 CPU가 항상 제공해야 할 minimum desired frequency와 넘지 않아야 할 maximum desired frequency hint를 전달합니다.
주의: cgroup2 CPU controller는 아직 realtime process의 bandwidth control을 지원하지 않습니다. Realtime process group scheduling용 `CONFIG_RT_GROUP_SCHED` option이 enable된 kernel에서는 모든 RT process가 root cgroup에 있을 때에만 CPU controller를 enable할 수 있습니다.
System-management software가 boot 과정에서 RT process를 이미 non-root cgroup에 배치했을 수 있습니다. 이 경우 `CONFIG_RT_GROUP_SCHED` kernel에서 CPU controller를 enable하기 전에 해당 process를 root cgroup으로 이동해야 합니다.
Realtime process placement를 검사하고 필요한 migration을 수행합니다.
`CONFIG_RT_GROUP_SCHED`가 disable되어 있으면 이 limitation은 적용되지 않고 일부 interface file은 realtime process에 영향을 주거나 이를 account합니다. 자세한 범위는 다음 CPU Interface Files 절에 나옵니다.
`CONFIG_RT_GROUP_SCHED`의 영향을 받는 것은 CPU controller뿐입니다. 다른 controller는 이 option과 관계없이 realtime process resource control에 사용할 수 있습니다.
Build option이 controller 사용에 미치는 범위입니다.
CPU scheduler category, stat과 weight
1127-1194Process와 CPU controller의 상호작용은 scheduling policy와 underlying scheduler에 따라 다릅니다. CPU controller 관점에서 process는 세 category로 나뉩니다.
이후 interface 설명에서 사용하는 세 scheduler category입니다.
Fair-class 또는 BPF scheduler 적용 시점의 자세한 내용은 `Documentation/scheduler/sched-ext.rst`의 `sched-ext` reference를 참조하십시오. 다음 interface 설명은 위 category를 사용하며 모든 time duration unit은 microsecond입니다.
`cpu.stat`은 read-only flat-keyed file이며 controller enable 여부와 관계없이 존재합니다.
항상 보고되는 전체-process counter와 controller enable 시 fair-class 전용 counter입니다.
`cpu.weight`는 non-root cgroup에 있는 read/write single-value file이고 default는 `100`입니다. Non-idle group(`cpu.idle = 0`)의 weight 범위는 `[1, 10000]`입니다. Cgroup이 `SCHED_IDLE`(`cpu.idle = 1`)로 설정되면 weight는 `0`으로 표시됩니다.
이 file은 fair-class scheduler process와 `cgroup_set_weight` callback을 가진 BPF scheduler process에만 영향을 줍니다. BPF 쪽 실제 영향은 callback 구현에 따라 달라집니다.
`cpu.weight.nice`는 non-root cgroup의 read/write single-value file이고 default `0`, nice 범위는 `[-20, 19]`입니다.
이는 `cpu.weight`의 alternative interface로 `nice(2)`와 같은 value를 사용해 weight를 읽고 설정하게 합니다. Nice range가 더 작고 granularity가 거칠기 때문에 read value는 current weight에 가장 가까운 approximation입니다.
`cpu.weight.nice`도 fair-class와 `cgroup_set_weight` callback이 있는 BPF scheduler process에만 영향을 주며 BPF 동작은 callback 구현에 달려 있습니다.
Native weight와 nice-compatible interface를 비교합니다.
CPU bandwidth, burst와 pressure
1195-1224`cpu.max`는 non-root cgroup의 read/write two-value file이고 default는 `max 100000`입니다. Maximum bandwidth limit은 다음 형식입니다.
$MAX $PERIODGroup은 각 `$PERIOD` duration마다 최대 `$MAX`만큼 사용할 수 있습니다. `$MAX`의 `max`는 limit이 없다는 뜻입니다. 숫자 하나만 쓰면 `$MAX`만 update합니다. 이 file은 fair-class scheduler process에만 영향을 줍니다.
Period마다 quota를 소비하고 다음 period에 다시 사용할 수 있습니다.
`cpu.max.burst`는 non-root cgroup의 read/write single-value file이며 default `0`, burst 범위는 `[0, $MAX]`입니다. Fair-class scheduler process에만 영향을 줍니다.
`cpu.pressure`는 read/write nested-keyed file로 CPU pressure stall information을 보여 줍니다. 자세한 내용은 `Documentation/accounting/psi.rst`의 `psi` reference를 참조하십시오. 이 file은 cgroup의 모든 process를 account합니다.
Bandwidth 제한과 pressure accounting의 대상 범위가 다릅니다.
Utilization clamp와 cgroup SCHED_IDLE
1225-1268`cpu.uclamp.min`은 non-root cgroup의 read/write single-value file입니다. Default `0`은 utilization boosting이 없다는 뜻입니다. Requested minimum utilization(protection)은 `12.34`처럼 percentage rational number로 표현합니다.
이 interface는 `sched_setattr(2)`와 비슷하게 minimum utilization clamp를 읽고 설정합니다. 이 값은 realtime process를 포함한 task-specific minimum utilization clamp를 제한하는 데 사용됩니다.
Requested minimum utilization은 항상 current maximum utilization limit인 `cpu.uclamp.max`로 cap됩니다. `cpu.uclamp.min`은 cgroup의 모든 process에 영향을 줍니다.
`cpu.uclamp.max`도 non-root cgroup의 read/write single-value file이며 default `max`는 utilization capping이 없다는 뜻입니다. Requested maximum utilization(limit)은 `98.76`처럼 percentage rational number로 표현합니다.
이 interface는 `sched_setattr(2)`와 비슷하게 maximum utilization clamp를 읽고 설정하며 realtime process를 포함한 task-specific maximum clamp를 제한합니다. Cgroup의 모든 process에 영향을 줍니다.
Cgroup clamp가 task-specific request와 결합되고 minimum은 maximum을 넘지 못합니다.
`cpu.idle`은 non-root cgroup의 read/write single-value file이고 default는 `0`입니다. 이는 per-task `SCHED_IDLE` scheduling policy의 cgroup analog입니다.
`1`로 설정하면 cgroup scheduling policy가 `SCHED_IDLE`이 됩니다. Cgroup 내부 thread는 서로 간의 relative priority를 유지하지만 cgroup 자체는 peer보다 매우 낮은 priority로 취급됩니다. Fair-class scheduler process에만 영향을 줍니다.
세 file의 default, 표현과 process 범위입니다.
Stateful Memory controller 개요
1269-1291`memory` controller는 memory distribution을 조절합니다. Memory는 stateful resource이며 limit model과 protection model을 모두 구현합니다.
Memory usage와 reclaim pressure가 서로 얽혀 있고 memory 자체가 stateful이므로 distribution model은 비교적 복잡합니다.
사용량을 추적하고 protection·limit·reclaim을 함께 적용하는 관계입니다.
완전히 빈틈없는 coverage는 아니지만 given cgroup의 주요 memory usage를 모두 추적하여 total memory consumption을 합리적인 수준까지 account하고 control합니다.
현재 추적하는 memory usage type은 userland page cache와 anonymous memory, dentry와 inode 같은 kernel data structure, TCP socket buffer입니다. 더 나은 coverage를 위해 이 목록은 향후 늘어날 수 있습니다.
현재 memory controller accounting 대상입니다.
memory.current, min과 low protection
1292-1354모든 memory amount unit은 byte입니다. `PAGE_SIZE`에 align되지 않은 값을 쓰면 readback에서 가장 가까운 `PAGE_SIZE` multiple로 올림될 수 있습니다.
`memory.current`는 non-root cgroup의 read-only single-value file로, 해당 cgroup과 descendant가 현재 사용하는 total memory amount를 보여 줍니다.
`memory.min`은 non-root cgroup의 read/write single-value file이고 default는 `0`입니다. 이는 hard memory protection입니다.
Cgroup usage가 effective min boundary 안에 있으면 어떤 조건에서도 그 memory를 reclaim하지 않습니다. Unprotected reclaimable memory가 없으면 OOM killer를 invoke합니다.
Effective min boundary, 또는 더 높다면 effective low boundary보다 위의 page는 boundary 초과량에 비례해 reclaim됩니다. 초과량이 작을수록 reclaim pressure도 줄어듭니다.
Effective min boundary는 모든 ancestor cgroup의 `memory.min`으로 제한됩니다. Child가 parent 허용량보다 더 많은 protected memory를 요구해 overcommit되면, 각 child는 자기 `memory.min` 아래 actual usage 비율에 따라 parent protection 일부를 받습니다.
일반적으로 available한 양보다 많은 memory를 이 protection 아래 두는 것은 권장하지 않으며 constant OOM을 일으킬 수 있습니다. Process가 populate하지 않은 memory cgroup의 `memory.min`은 무시됩니다.
`memory.low`는 non-root cgroup의 read/write single-value file이고 default `0`이며 best-effort memory protection을 제공합니다.
Usage가 effective low boundary 안에 있으면 unprotected cgroup에 reclaimable memory가 전혀 없는 경우에만 이 cgroup의 memory를 reclaim합니다. Effective low boundary, 또는 더 높다면 effective min boundary보다 위의 page는 초과량에 비례해 reclaim합니다.
Effective low boundary도 모든 ancestor의 `memory.low`로 제한됩니다. Overcommit이면 child의 `memory.low` 아래 actual usage 비율에 따라 parent protection을 나눕니다. 일반 available memory보다 큰 protection을 설정하는 것은 권장하지 않습니다.
Hard min과 best-effort low의 reclaim·OOM 차이입니다.
Ancestor cap과 overcommit 시 proportional sharing을 적용합니다.
memory.high와 memory.max
1355-1409`memory.high`는 non-root cgroup의 read/write single-value file이며 default는 `max`입니다. Memory usage throttle limit으로, usage가 high boundary를 넘으면 cgroup process를 throttle하고 heavy reclaim pressure에 둡니다.
High limit 초과는 OOM killer를 invoke하지 않으며 extreme condition에서는 limit을 넘은 상태가 허용될 수 있습니다. 이 limit은 external process가 제한된 cgroup을 monitor하면서 heavy reclaim pressure를 완화하는 scenario에 사용해야 합니다.
`memory.high`를 `O_NONBLOCK`으로 open하면 synchronous reclaim을 bypass합니다. Admin process가 자기 CPU resource를 memory reclaim에 쓰지 않고 job memory limit을 동적으로 조정할 때 유용합니다. Job의 다음 charge request가 reclaim을 trigger하거나 throttle됩니다.
`O_NONBLOCK`에서는 charge request가 지연되거나 target이 memory를 계속 바쁘게 touch해 reclaim이 느려지면 usage가 limit 아래로 내려가는 데 무한히 오래 걸릴 수 있습니다.
`memory.max`는 non-root cgroup의 read/write single-value file이며 default `max`입니다. Cgroup memory usage를 제한하는 주 mechanism인 hard limit입니다.
Usage가 limit에 닿고 줄일 수 없으면 해당 cgroup에서 OOM killer를 invoke합니다. 특정 상황에서는 usage가 일시적으로 limit을 넘을 수 있습니다.
Default configuration에서 일반 0-order allocation은 OOM killer가 current task를 victim으로 선택하지 않는 한 항상 성공합니다. 일부 allocation은 OOM killer를 invoke하지 않으며 caller가 다른 방식으로 retry하거나 `-ENOMEM`을 userspace에 반환하거나 disk readahead처럼 조용히 무시할 수 있습니다.
`memory.max`를 `O_NONBLOCK`으로 open하면 synchronous reclaim과 OOM kill을 모두 bypass합니다. Admin process 대신 job의 다음 charge request가 reclaim 또는 OOM kill을 trigger합니다. Charge가 지연되거나 reclaim이 느리면 usage 감소가 무기한 늦어질 수 있습니다.
High와 max boundary의 enforcement와 OOM behavior입니다.
Admin write에서 synchronous work를 피하고 target job의 next charge로 넘깁니다.
Proactive reclaim과 peak usage
1410-1454`memory.reclaim`은 모든 cgroup에 있는 write-only nested-keyed file로 target cgroup의 memory reclaim을 trigger하는 단순 interface입니다.
echo "1G" > memory.reclaimKernel은 지정한 양보다 많이 또는 적게 reclaim할 수 있습니다. Reclaim한 byte가 지정량보다 적으면 `-EAGAIN`을 반환합니다.
이 interface가 trigger하는 proactive reclaim은 memory cgroup에 memory pressure가 있다는 신호가 아닙니다. 따라서 일반 memory reclaim에서 동작하는 socket-memory balancing은 보통 수행하지 않으며 networking layer도 `memory.reclaim`에 의한 reclaim을 보고 적응하지 않습니다.
Nested key `swappiness`는 reclaim에 사용할 swappiness value를 지정합니다. `vm.swappiness`를 memcg reclaim에 적용한 것과 같은 semantics와 기존 limitation 및 향후 extension을 가집니다.
Valid range는 `[0-200, max]`이고 `swappiness=max`는 anonymous memory만 exclusive하게 reclaim합니다.
Requested byte와 actual reclaim 결과 및 networking exception입니다.
Nested key의 범위와 special value입니다.
`memory.peak`는 non-root cgroup에 있는 read/write single-value file입니다. Cgroup 생성 또는 해당 FD에서 가장 최근 reset 이후 cgroup과 descendant가 기록한 maximum memory usage입니다.
비어 있지 않은 어떤 문자열이든 이 file에 쓰면 같은 file descriptor를 통한 이후 read에 대해 current memory usage로 reset됩니다.
Reset은 file descriptor별 read state에 적용됩니다.
OOM group과 memory event counters
1455-1522`memory.oom.group`은 non-root cgroup의 read/write single-value file이며 default `0`입니다. 설정하면 OOM killer가 cgroup을 indivisible workload로 취급합니다.
해당 cgroup과, leaf가 아니라면 descendant에 속한 모든 task를 함께 kill하거나 전혀 kill하지 않습니다. Partial kill을 방지해 workload integrity를 보장할 수 있습니다.
`oom_score_adj`가 `-1000`인 OOM-protected task는 예외로 절대 kill하지 않습니다. 어떤 cgroup에서 OOM killer가 invoke되면 ancestor의 `memory.oom.group` 값과 관계없이 그 cgroup 밖 task를 kill하지 않습니다.
Workload integrity와 protected-task exception을 함께 적용합니다.
`memory.events`는 non-root cgroup의 read-only flat-keyed file입니다. 별도 설명이 없으면 value change가 file-modified event를 생성합니다.
모든 field는 hierarchical하므로 hierarchy 아래의 event 때문에 current file notification이 발생할 수 있습니다. Cgroup level의 local event는 `memory.events.local`을 사용합니다.
Protection, throttle, limit과 OOM event의 의미입니다.
`low`는 low boundary 아래인데도 high memory pressure로 reclaim된 횟수이며 대개 low boundary overcommit을 뜻합니다.
`high`는 high boundary 초과로 process가 throttle되고 direct memory reclaim을 수행한 횟수입니다. Global pressure가 아니라 high limit이 usage를 cap하는 cgroup에서는 이 event가 발생하는 것이 정상입니다.
`max`는 usage가 max boundary를 넘으려 한 횟수이며 direct reclaim으로 줄이지 못하면 OOM state로 갑니다. `oom`은 usage가 limit에 닿아 allocation이 실패하기 직전이었던 횟수입니다.
High-order allocation failure나 caller가 retry하지 말라고 요청한 경우처럼 OOM killer를 option으로 고려하지 않으면 `oom` event는 발생하지 않습니다.
`oom_kill`은 어떤 종류의 OOM killer든 이 cgroup 소속 process를 kill한 수이고 `oom_group_kill`은 group OOM이 발생한 횟수입니다.
`memory.events.local`은 `memory.events`와 비슷하지만 field가 hierarchical하지 않고 해당 cgroup local event만 담습니다. 이 file의 modified event도 local event만 반영합니다.
두 event file의 propagation 범위입니다.
memory.stat footprint, LRU와 slab counters
1523-1638`memory.stat`은 non-root cgroup에 존재하는 read-only flat-keyed file입니다. Cgroup memory footprint를 memory type, type별 detail, memory-management system의 current state와 past event로 분해합니다. 모든 memory amount unit은 byte입니다.
Entry는 사람이 읽기 좋은 순서로 배치되지만 새 entry가 중간에 추가될 수 있습니다. Item의 fixed position에 의존하지 말고 반드시 key로 specific value를 찾으십시오.
Per-node counter가 없거나 `memory.numa_stat`에 나타나지 않는 entry에는 `npn`(non-per-node) tag를 붙입니다.
Anonymous, file과 kernel memory의 기본 분류입니다.
Swap-backed cache, zswap와 file mapping/writeback 상태입니다.
Anonymous, file과 shmem의 THP-backed usage입니다.
`inactive_anon`, `active_anon`, `inactive_file`, `active_file`, `unevictable`은 page-reclaim algorithm이 사용하는 internal memory-management list에 있는 swap-backed 및 filesystem-backed memory amount입니다.
이 값은 type이 아니라 internal list state를 나타냅니다. 예를 들어 shmem page도 anon management list에 있으므로 `inactive_foo + active_foo`가 type-based `foo` counter와 같지 않을 수 있습니다.
Internal reclaim list와 kernel object cache 상태입니다.
Workingset, reclaim와 page-event counters
1639-1718Evicted page의 refault·activation·restore와 shadow-node reclaim입니다.
Swap 이동과 inactive LRU scan/reclaim 주체를 구분합니다.
Scanner별 pgscan과 successful reclaim pgsteal counter의 관계입니다.
Page fault와 active/inactive LRU 이동 및 lazyfree입니다.
Zero swap, zswap, THP, NUMA와 demotion
1719-1783Zero detection으로 I/O를 생략한 swap과 zswap 이동 counter입니다.
`CONFIG_TRANSPARENT_HUGEPAGE`와 swapout 방식에 따른 counter입니다.
NUMA hinting과 reclaim path별 page demotion을 구분합니다.
`hugetlb`는 HugeTLB page가 사용하는 memory amount입니다. Cgroup을 `memory_hugetlb_accounting` option으로 mount해 HugeTLB usage가 `memory.current`에 account되는 경우에만 이 metric이 나타납니다.
Mount option이 memory.current accounting과 memory.stat visibility를 함께 제어합니다.
NUMA locality와 swap interfaces
1784-1873`memory.numa_stat`은 non-root cgroup의 read-only nested-keyed file입니다. Cgroup memory footprint를 memory type과 type별 detail로 나누고 memory-management state를 node별로 보여 줍니다.
Memcg page는 어느 physical node에서든 allocate될 수 있으므로 이 file은 memcg 안의 NUMA locality를 관찰하는 데 유용합니다. Application CPU allocation과 이 정보를 결합해 application performance를 평가하는 것이 use case 중 하나입니다.
모든 memory amount unit은 byte이며 output format은 다음과 같습니다.
type N0=<bytes in node 0> N1=<bytes in node 1> ...Entry는 사람이 읽기 좋은 순서지만 중간에 새 entry가 추가될 수 있으므로 fixed position에 의존하지 말고 key로 찾으십시오. Entry는 `memory.stat` counter를 참조할 수 있습니다.
Node별 memory placement와 CPU placement를 함께 보아 workload locality를 평가합니다.
`memory.swap.current`는 non-root cgroup의 read-only single-value file로 cgroup과 descendant가 현재 사용하는 total swap amount를 보여 줍니다.
`memory.swap.high`는 non-root cgroup의 read/write single-value file이며 default `max`입니다. Swap usage가 이 throttle limit을 넘으면 userspace가 custom OOM procedure를 구현할 수 있도록 이후 모든 allocation을 throttle합니다.
이 limit은 cgroup의 point of no return을 표시하며 정상 operation 중 workload swapping amount를 관리하려는 용도가 아닙니다. Healthy workload는 이 limit에 닿지 않아야 합니다.
`memory.swap.peak`는 non-root cgroup의 read/write single-value file입니다. Cgroup 생성 또는 해당 FD의 latest reset 이후 cgroup과 descendant가 기록한 maximum swap usage입니다. Non-empty string을 쓰면 같은 FD의 이후 read 기준을 current memory usage로 reset합니다.
Reset은 file 전체의 global state가 아니라 write에 사용한 file descriptor의 이후 read 기준만 바꿉니다.
`memory.swap.max`는 non-root cgroup의 read/write single-value file이며 default `max`입니다. Swap usage hard limit으로, usage가 이 값에 닿으면 cgroup의 anonymous memory를 더 이상 swap out하지 않습니다.
Current, throttle, peak와 hard limit의 목적입니다.
`memory.swap.events`는 non-root cgroup의 read-only flat-keyed file입니다. 별도 설명이 없으면 value change가 file-modified event를 생성합니다.
Swap high, max와 allocation failure를 구분합니다.
Limit을 current usage 아래로 낮추면 existing swap entry를 점진적으로 reclaim하므로 usage가 오랫동안 limit보다 높게 유지될 수 있습니다. 이는 workload와 memory management에 미치는 충격을 줄입니다.
즉시 강제 회수하지 않고 existing entry를 서서히 줄입니다.
Zswap limit·writeback과 memory pressure
1874-1913`memory.zswap.current`는 non-root cgroup의 read-only single-value file로 zswap compression backend가 소비하는 total memory amount를 보여 줍니다.
`memory.zswap.max`는 non-root cgroup의 read/write single-value file이며 default `max`입니다. Zswap usage hard limit으로, pool이 limit에 닿으면 existing entry가 fault back되거나 disk로 write될 때까지 새 store를 거부합니다.
`memory.zswap.writeback`은 read/write single-value file이고 default `1`입니다. 이 setting은 hierarchical하므로 upper hierarchy에서 writeback을 disable하면 child cgroup도 암묵적으로 disable됩니다.
`0`으로 설정하면 zswap writeback과 zswap store failure 때문에 발생하는 swapping을 포함해 swapping device로 향하는 모든 swap attempt를 disable합니다.
Page가 incompressible해서 zswap store failure가 반복되면 writeback을 disable한 뒤 같은 page가 계속 거부되어 reclaim inefficiency가 나타날 수 있습니다.
Ancestor setting이 descendant의 disk writeback 가능성을 제거합니다.
이는 `memory.swap.max=0`과 미묘하게 다릅니다. Writeback을 disable해도 page를 zswap pool에는 쓸 수 있습니다. Zswap이 disable되어 있으면 이 setting은 효과가 없고, `memory.swap.max=0`이 아닌 한 일반 swapping은 허용됩니다.
Compressed pool 사용과 backing swap-device 사용을 분리합니다.
`memory.pressure`는 read-only nested-keyed file로 memory pressure stall information을 보여 줍니다. 자세한 내용은 `Documentation/accounting/psi.rst`의 `psi` reference를 참조하십시오.
Memory 후반 interface의 access와 default입니다.
Memory usage control guideline
1914-1937`memory.high`는 memory usage를 control하는 주 mechanism입니다. High limit 합계가 available memory보다 큰 overcommit을 허용하고 global memory pressure가 usage에 따라 memory를 분배하도록 두는 것도 viable strategy입니다.
High limit breach는 OOM killer를 trigger하지 않고 offending cgroup을 throttle합니다. 따라서 management agent는 상태를 monitor하면서 memory를 더 주거나 workload를 terminate하는 적절한 action을 취할 기회가 충분합니다.
High limit overcommit과 external agent의 remediation loop입니다.
Memory usage만으로 cgroup에 memory가 충분한지 판단하기는 쉽지 않습니다. Usage는 workload가 memory를 더 받으면 이익을 보는지 알려 주지 않기 때문입니다.
예를 들어 network에서 받은 data를 file에 쓰는 workload는 available memory를 모두 사용할 수 있지만 적은 memory로도 충분한 performance를 낼 수 있습니다.
Workload가 memory 부족으로 얼마나 영향받는지를 나타내는 memory pressure measure가 있어야 더 많은 memory가 필요한지 판단할 수 있습니다. 하지만 이 문서가 설명하는 memory-pressure monitoring mechanism은 아직 구현되지 않았습니다.
Usage 자체와 pressure signal의 정보 차이입니다.
Memory area ownership과 migration
1938-1956Memory area는 이를 instantiate한 cgroup에 charge되고 area가 release될 때까지 그 cgroup에 계속 charge됩니다. Process를 다른 cgroup으로 migrate해도 이전 cgroup에 있을 때 instantiate한 memory usage는 새 cgroup으로 이동하지 않습니다.
Process location과 memory-area owner는 독립적으로 유지됩니다.
하나의 memory area를 서로 다른 cgroup의 process가 사용할 수 있습니다. 이때 어느 cgroup에 charge될지는 deterministic하지 않습니다. 다만 시간이 지나면 high reclaim pressure를 피할 만큼 memory allowance가 충분한 cgroup에 area가 귀속될 가능성이 큽니다.
여러 cgroup이 공유하는 area는 reclaim과 재사용을 거치며 여유가 있는 owner 쪽으로 수렴할 수 있습니다.
어떤 cgroup이 상당한 memory를 sweep하고 그 memory를 다른 cgroup이 반복 access할 것으로 예상된다면 `POSIX_FADV_DONTNEED`를 사용해 해당 file의 memory-area ownership을 relinquish하는 것이 올바른 ownership을 보장하는 데 도움이 될 수 있습니다.
File cache를 대량으로 먼저 touch한 cgroup이 ownership을 놓아야 하는 상황입니다.
IO controller와 io.stat
1957-1989`io` controller는 IO resource의 distribution을 조절합니다. Weight 기반 distribution과 absolute bandwidth 또는 IOPS limit distribution을 모두 구현하지만, weight 기반 distribution은 `cfq-iosched`를 사용할 때만 제공되며 `blk-mq` device에서는 어느 방식도 제공되지 않습니다.
원문이 설명하는 controller 방식과 availability constraint입니다.
`io.stat`은 read-only nested-keyed file입니다. 각 line은 `$MAJ:$MIN` device number를 key로 사용하며 순서는 정해져 있지 않습니다.
Device별 read, write와 discard activity를 byte와 operation 수로 나눕니다.
`io.stat` read output 예시는 다음과 같습니다.
8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0
8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021IOCOST Quality of Service
1990-2044`io.cost.qos`는 root cgroup에만 존재하는 read/write nested-keyed file입니다. 현재 `io.weight` proportional control을 구현하는 IO cost model 기반 controller(`CONFIG_BLK_CGROUP_IOCOST`)의 Quality of Service를 설정합니다.
Line은 `$MAJ:$MIN` device number를 key로 사용하며 순서가 없습니다. 특정 device의 line은 그 device에 대해 `io.cost.qos` 또는 `io.cost.model`에 처음 write할 때 채워집니다.
Controller enable, mode, latency QoS와 issue-rate scaling range를 구성합니다.
Controller는 default로 disable되어 있으며 `enable=1`로 enable합니다. `rpct`와 `wpct`의 default는 0이고, 이때 controller는 internal device saturation state를 사용해 overall IO rate를 `min`과 `max` 사이에서 조절합니다.
더 나은 control quality가 필요하면 latency QoS parameter를 설정할 수 있습니다. 다음 예시는 sdb에 해당하는 device `8:16` 설정입니다.
8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0이 설정은 controller를 enable하고, read completion latency의 95th percentile이 75 ms를 넘거나 write latency의 95th percentile이 150 ms를 넘으면 device가 saturated되었다고 판단합니다. 그 결과 overall IO issue rate를 50%에서 150% 사이에서 조절합니다.
Percentile latency로 saturation을 판단하고 허용 범위에서 issue rate를 조절합니다.
Saturation point를 낮출수록 aggregate bandwidth를 희생하는 대신 latency QoS가 좋아집니다. `min`과 `max` 사이 adjustment range를 좁힐수록 IO behavior가 cost model에 더 잘 맞습니다.
IO issue base rate는 100%와 크게 다를 수 있으므로 `min`과 `max`를 근거 없이 설정하면 device capacity 또는 control quality를 크게 잃을 수 있습니다. 이 두 값은 잠시 line speed로 write를 받다가 수초 동안 완전히 stall하는 SSD처럼 temporary behavior change 폭이 큰 device를 조절하는 데 유용합니다.
Threshold와 scaling range가 latency, bandwidth와 model conformity에 미치는 영향입니다.
`ctrl=auto`이면 kernel이 parameter를 control하며 자동으로 바꿀 수 있습니다. `ctrl=user`를 설정하거나 percentile·latency parameter 중 하나라도 설정하면 `user` mode로 전환되어 automatic change가 disable됩니다. `ctrl=auto`를 쓰면 automatic mode로 복원됩니다.
Parameter write가 automatic tuning을 user-managed tuning으로 전환합니다.
IOCOST linear cost model
2045-2086`io.cost.model`은 root cgroup에만 존재하는 read/write nested-keyed file입니다. 현재 `io.weight` proportional control을 구현하는 IO cost model 기반 controller(`CONFIG_BLK_CGROUP_IOCOST`)의 cost model을 설정합니다.
Line은 순서 없는 `$MAJ:$MIN` device key로 구성됩니다. 특정 device line은 `io.cost.qos` 또는 `io.cost.model`에 그 device를 처음 write할 때 생성됩니다.
Model control mode와 사용 중인 model type입니다.
`ctrl=auto`이면 kernel이 모든 parameter를 dynamic하게 바꿀 수 있습니다. `ctrl=user`를 설정하거나 다른 parameter를 write하면 `ctrl`이 `user`가 되고 automatic change가 disable됩니다.
`[r|w]`는 read와 write 각각에 대응하며 4 KiB IO 특성을 사용합니다.
Builtin linear model은 이 값들로 sequential·random IO의 base cost와 IO size에 대한 cost coefficient를 결정합니다. 단순한 model이지만 대부분의 일반적인 device class를 충분히 다룰 수 있습니다.
IO cost model은 absolute하게 정확할 것으로 기대하지 않으며 device behavior에 맞춰 dynamic하게 scale됩니다. 필요하면 `tools/cgroup/iocost_coef_gen.py`로 device-specific coefficient를 생성할 수 있습니다.
Measured device limits에서 IO cost를 만들고 runtime behavior에 맞춰 scale합니다.
Device별 proportional IO weight
2087-2106`io.weight`는 non-root cgroup에 존재하는 read/write flat-keyed file이며 default는 `default 100`입니다.
첫 line은 specific override가 없는 device에 적용할 default weight입니다. 나머지 line은 순서 없는 `$MAJ:$MIN` device override입니다. Weight 범위는 [1, 10000]이며 sibling과 비교해 cgroup이 사용할 수 있는 relative IO time을 지정합니다.
Default와 device-specific override를 설정하고 해제하는 형식입니다.
`io.weight` read output 예시는 다음과 같습니다.
default 100
8:16 200
8:0 50Device override가 있으면 우선 사용하고, 없으면 default를 sibling-relative share로 적용합니다.
Absolute IO limit와 pressure
2107-2152`io.max`는 non-root cgroup에 존재하는 read/write nested-keyed file입니다. BPS와 IOPS 기반 IO limit을 설정하며 line은 순서 없는 `$MAJ:$MIN` device key를 사용합니다.
Read와 write 방향별 byte-rate·operation-rate ceiling입니다.
Write할 때 nested key-value pair를 원하는 수만큼 임의 순서로 지정할 수 있습니다. 특정 limit을 제거하려면 value로 `max`를 지정합니다. 같은 key를 여러 번 지정한 결과는 undefined입니다.
BPS와 IOPS는 IO direction별로 측정하며 limit에 도달하면 IO를 delay합니다. Temporary burst는 허용됩니다.
Device `8:16`의 read limit을 2 MiB/s BPS로, write limit을 120 IOPS로 설정합니다.
echo "8:16 rbps=2097152 wiops=120" > io.maxRead하면 다음과 같이 설정하지 않은 limit은 `max`로 나타납니다.
8:16 rbps=2097152 wbps=max riops=max wiops=120Write IOPS limit은 다음 write로 제거할 수 있습니다.
echo "8:16 wiops=max" > io.max이후 read output은 다음과 같습니다.
8:16 rbps=2097152 wbps=max riops=max wiops=maxDirection별 rate를 측정하고 burst를 허용하면서 초과 IO를 지연합니다.
`io.pressure`는 read-only nested-keyed file로 IO pressure stall information을 보여 줍니다. 자세한 내용은 `Documentation/accounting/psi.rst`의 `psi` reference를 참조하십시오.
Page-cache writeback domain과 attribution
2153-2186Page cache는 buffered write와 shared mmap을 통해 dirty 상태가 되고, writeback mechanism이 이를 backing filesystem에 asynchronous하게 씁니다. Writeback은 memory domain과 IO domain 사이에서 dirtying과 write IO의 균형을 맞춰 dirty memory 비율을 조절합니다.
Dirtying 속도와 backing IO를 조정해 dirty-memory proportion을 유지합니다.
`io` controller는 memory controller와 함께 page-cache writeback IO를 control합니다. Memory controller는 dirty memory ratio를 계산하고 유지할 memory domain을 정의하고, IO controller는 그 memory domain의 dirty page를 write out할 IO domain을 정의합니다.
System-wide dirty-memory state와 per-cgroup state를 모두 검사하며 둘 중 더 restrictive한 조건을 enforce합니다.
Memory accounting 범위와 write attribution 범위를 서로 다른 controller가 담당합니다.
Cgroup writeback에는 underlying filesystem의 explicit support가 필요합니다. 현재 `ext2`, `ext4`, `btrfs`, `f2fs`, `xfs`가 지원합니다. 그 밖의 filesystem에서는 모든 writeback IO가 root cgroup에 귀속됩니다.
Filesystem support 여부가 per-cgroup attribution 가능성을 결정합니다.
Memory와 writeback management는 ownership 추적 단위가 본질적으로 다릅니다. Memory는 page별로 추적하지만 writeback은 inode별로 추적합니다. Writeback에서는 inode 하나를 cgroup에 assign하고, 그 inode의 dirty page를 쓰는 모든 IO request를 해당 cgroup에 귀속합니다.
Page ownership cgroup이 inode ownership cgroup과 다른 page를 `foreign pages`라고 합니다. Writeback은 foreign page를 계속 추적하고 특정 foreign cgroup이 일정 기간 majority가 되면 inode ownership을 그 cgroup으로 전환합니다.
Per-page memory ownership과 per-inode writeback ownership의 차이를 majority tracking으로 보정합니다.
Concurrent writer limitation과 dirty sysctl
2187-2212한 inode를 주로 하나의 cgroup이 dirty하는 일반적인 use case에서는 main writer cgroup이 시간에 따라 바뀌더라도 이 model로 충분합니다. 그러나 여러 cgroup이 하나의 inode를 동시에 write하는 use case는 잘 지원되지 않아 IO의 상당 부분이 잘못 귀속될 수 있습니다.
Memory controller는 첫 사용 시 page ownership을 정한 뒤 page가 release될 때까지 갱신하지 않습니다. 따라서 writeback이 page ownership을 엄격히 따르더라도 여러 cgroup이 겹치는 영역을 dirty하는 패턴은 예상대로 동작하지 않습니다. 이런 usage pattern은 피하는 것이 좋습니다.
Writer pattern에 따른 inode-based attribution의 정확도입니다.
Writeback behavior에 영향을 주는 sysctl knob는 cgroup writeback에 다음과 같이 적용됩니다.
System knob를 cgroup별 available-memory 범위에 맞춰 적용합니다.
IO Latency protection hierarchy
2213-2239IO Latency는 IO workload를 보호하는 cgroup v2 controller입니다. Group에 latency target을 지정하면 average latency가 target을 넘을 때 controller가 protected workload보다 낮은 priority, 즉 더 높은 latency target을 가진 peer를 throttle합니다.
Limit은 hierarchy에서 peer level에만 적용됩니다. 원문 구조에서 A·B·C는 서로에게 영향을 주고, A 아래의 D·F도 서로에게 영향을 줍니다. C 아래에 sibling이 없는 G는 어느 group에도 영향을 주지 않습니다.
원문의 ASCII hierarchy를 sibling influence group과 parent-child 관계가 보이도록 재구성했습니다.
따라서 이상적인 구성은 A, B, C에 `io.latency`를 설정하는 것입니다. 일반적으로 device가 지원하는 latency보다 낮은 값을 설정해서는 안 되며 workload에 맞는 값을 experiment로 찾아야 합니다.
먼저 device의 expected latency보다 높은 값으로 시작하고 workload group의 `io.stat`에서 `avg_lat`를 관찰해 normal operation latency를 파악합니다. 실제 target은 이 `avg_lat`를 기준으로 10~15% 높게 설정합니다.
Device capability와 observed workload latency에서 안전한 target을 찾는 절차입니다.
IO latency throttling과 recovery
2240-2265`io.latency`는 work-conserving 방식입니다. 모든 group이 latency target을 만족하는 동안 controller는 개입하지 않습니다. 한 group이 target을 놓치기 시작하면 그 group보다 높은 target을 가진 peer group을 throttle합니다.
Target miss가 생길 때에만 lower-priority peer의 IO를 제한합니다.
Throttle은 다음 두 형태로 이루어집니다.
일반 IO와 직접 제한하기 어려운 IO를 서로 다른 방식으로 제어합니다.
Queue depth throttling은 group이 보유할 수 있는 outstanding IO 수를 제한합니다. No limit에서 시작해 비교적 빠르게 줄이며 최종적으로 한 번에 1 IO까지 clamp할 수 있습니다.
Swapping과 metadata IO처럼 throttle하면 higher-priority group에 악영향을 줄 수 있는 IO는 정상적으로 수행하도록 허용하지만 originating group에 charge합니다. 해당 group이 throttle 중이면 `io.stat`의 `use_delay`와 `delay`가 증가합니다.
`delay`는 이 group에서 실행되는 각 process에 추가되는 microsecond 수입니다. Swapping 또는 metadata IO가 많으면 값이 크게 늘 수 있으므로 individual delay event는 한 번에 1초로 제한합니다.
피해를 받은 group이 다시 latency target을 만족하면 이전에 throttle된 peer를 unthrottle하기 시작합니다. 피해 group이 IO를 완전히 멈춰도 global counter가 그에 맞게 unthrottle합니다.
Protected workload가 회복하거나 idle해지면 peer restriction을 해제합니다.
IO Latency interface files
2266-2291`io.latency`는 다른 controller와 비슷한 형식을 사용하며 device와 target time을 microsecond 단위로 지정합니다.
"MAJOR:MINOR target=<target time in microseconds>"Controller가 enable되면 `io.stat`에 일반 statistic과 함께 아래 field가 추가됩니다.
현재 queue restriction, smoothed latency와 evaluation window를 노출합니다.
`avg_lat`의 decay-rate interval은 `io.stat`의 `win` 값에 해당 `win` 값에 따른 sample 수를 곱해 계산할 수 있습니다.
`win` window는 IO activity가 있을 때만 경과합니다. Idle period는 가장 최근 window를 연장합니다.
I/O priority cgroup policies
2292-2319I/O priority cgroup policy 동작은 단일 attribute인 `io.prio.class`가 control합니다. 이 attribute가 받는 값은 다음과 같습니다.
Request의 기존 I/O priority class를 유지하거나 RT, BE, IDLE 방향으로 제한합니다.
I/O priority numeric mapping과 변환
2320-2353I/O priority policy에 연결되는 numerical value는 다음과 같습니다. Deprecated alias인 `none-to-rt`는 `promote-to-rt`와 같은 동작을 가리킵니다.
원문의 ASCII 표를 동일한 mapping의 구조화 표로 옮겼습니다.
각 I/O priority class의 numerical value는 다음과 같습니다.
Number가 커질수록 RT에서 BE, IDLE 방향으로 제한됩니다.
Request의 I/O priority class를 설정하는 algorithm은 다음과 같습니다. Policy가 `promote-to-rt`이면 request class를 `IOPRIO_CLASS_RT`로 바꾸고 request I/O priority level을 4로 변경합니다.
Policy가 `promote-to-rt`가 아니면 policy를 number로 변환한 뒤, policy number와 현재 numerical I/O priority class 중 maximum으로 request class를 바꿉니다.
RT promotion은 별도로 처리하고 나머지는 numeric maximum으로 priority ceiling을 적용합니다.
PID controller와 task exhaustion
2354-2369Process number controller는 지정된 limit에 도달한 뒤 cgroup에서 새 task가 `fork()` 또는 `clone()`되는 것을 막을 수 있게 합니다.
Cgroup의 task 수는 다른 controller가 막을 수 없는 방식으로 고갈될 수 있으므로 전용 controller가 필요합니다. 예를 들어 fork bomb은 memory restriction에 닿기 전에 task 수를 먼저 고갈시킬 가능성이 큽니다.
이 controller에서 사용하는 PID는 kernel이 process ID로 사용하는 TID를 뜻합니다.
Memory limit보다 먼저 고갈될 수 있는 task identifier를 별도 hard limit으로 보호합니다.
PID interface files와 policy enforcement
2370-2413Non-root cgroup의 process hard limit, usage, peak와 event를 제공합니다.
`pids.events`는 별도 설명이 없으면 value change 시 file-modified event를 생성합니다. `max` entry는 cgroup의 total process 수가 `pids.max` limit에 도달한 횟수이며 `pids_localevents`도 참조해야 합니다.
`pids.events.local`은 `pids.events`와 비슷하지만 field가 hierarchical하지 않고 현재 cgroup에 local합니다. 이 file에서 발생하는 file-modified event도 local event만 반영합니다.
Cgroup policy는 organisational operation을 막지 않으므로 `pids.current > pids.max`가 될 수 있습니다. Current보다 작은 limit을 설정하거나 충분히 많은 process를 cgroup에 attach하면 이런 상태가 됩니다.
그러나 `fork()`나 `clone()`으로 PID policy를 위반할 수는 없습니다. 새 process 생성이 cgroup policy를 위반하면 이 call들은 `-EAGAIN`을 반환합니다.
Organisation 변경은 over-limit 상태를 만들 수 있지만 새 task 생성은 hard limit을 넘지 못합니다.
Cpuset placement과 hierarchy
2414-2428`cpuset` controller는 task의 current cgroup에 있는 cpuset interface file이 지정한 resource만 사용하도록 task의 CPU와 memory-node placement를 제한합니다.
이는 large NUMA system에서 특히 유용합니다. Job을 적절한 크기의 system subset에 배치하고 processor·memory placement를 신중히 정해 cross-node memory access와 contention을 줄이면 전체 system performance를 높일 수 있습니다.
`cpuset` controller는 hierarchical하므로 parent가 허용하지 않은 CPU 또는 memory node를 사용할 수 없습니다.
Requested placement는 parent allowance와 online resource에 의해 실제 grant로 좁혀집니다.
Requested CPUs와 effective CPUs
2429-2470`cpuset.cpus`는 non-root cpuset-enabled cgroup에 존재하는 read/write multiple-values file입니다. Cgroup 안 task가 사용할 requested CPU를 나열하지만 실제 grant 목록은 parent constraint를 받으므로 requested 목록과 다를 수 있습니다.
CPU number는 comma-separated number 또는 range로 표현합니다.
# cat cpuset.cpus
0-4,6,8-10Empty value이면 가장 가까운 non-empty `cpuset.cpus` ancestor의 setting을 사용합니다. 그런 ancestor가 없으면 available CPU 전체를 사용합니다.
`cpuset.cpus` 값은 다음 update까지 일정하게 유지되며 CPU hotplug event의 영향을 받지 않습니다.
`cpuset.cpus.effective`는 모든 cpuset-enabled cgroup에 존재하는 read-only multiple-values file입니다. Parent가 실제 grant한 online CPU를 나열하며 current cgroup task가 사용할 수 있는 CPU입니다.
`cpuset.cpus`가 empty이면 `cpuset.cpus.effective`는 parent cgroup에서 이 cgroup에 available한 모든 CPU를 보여 줍니다. Empty가 아니면 보통 `cpuset.cpus`의 subset입니다.
Requested 목록의 CPU를 하나도 grant할 수 없으면 empty `cpuset.cpus`와 같은 방식으로 처리합니다. `cpuset.cpus.effective`는 CPU hotplug event의 영향을 받습니다.
Configuration state와 runtime grant state의 차이입니다.
Requested set, ancestor fallback와 parent constraint에서 effective set을 계산합니다.
Requested memory nodes와 migration
2471-2520`cpuset.mems`는 non-root cpuset-enabled cgroup에 존재하는 read/write multiple-values file입니다. Cgroup task가 사용할 requested memory node를 나열하지만 실제 grant 목록은 parent constraint에 따라 다를 수 있습니다.
Memory-node number는 comma-separated number 또는 range로 표현합니다.
# cat cpuset.mems
0-1,3Empty value이면 가장 가까운 non-empty `cpuset.mems` ancestor의 setting을 사용하며, 그런 ancestor가 없으면 available memory node 전체를 사용합니다.
`cpuset.mems` 값은 다음 update까지 일정하고 memory-node hotplug event의 영향을 받지 않습니다.
`cpuset.mems`에 non-empty value를 설정하면 cgroup task가 지정 범위 밖의 memory를 사용하고 있을 때 그 memory를 지정 node로 migrate합니다. Migration에는 cost가 들고 완전하지 않을 수 있어 일부 memory page가 남을 수 있습니다.
따라서 새 task를 cpuset에 spawn하기 전에 `cpuset.mems`를 올바르게 설정하는 것이 좋습니다. Active task가 있는 상태에서 바꿔야 하더라도 자주 변경해서는 안 됩니다.
Active workload의 node set을 바꾸면 best-effort page migration이 발생합니다.
`cpuset.mems.effective`는 모든 cpuset-enabled cgroup에 존재하는 read-only multiple-values file입니다. Parent가 실제 grant한 online memory node를 나열하며 current cgroup task가 사용할 수 있습니다.
`cpuset.mems`가 empty이면 parent cgroup에서 available한 모든 memory node를 보여 줍니다. Non-empty이면 보통 `cpuset.mems`의 subset이며, 요청 node를 하나도 grant할 수 없으면 empty request처럼 처리합니다.
`cpuset.mems.effective` 값은 memory-node hotplug event의 영향을 받습니다.
Configured NUMA-node request와 runtime online grant를 구분합니다.
Parent constraint와 online state가 effective memory nodes를 결정합니다.
Exclusive CPU request와 sibling rule
2521-2554`cpuset.cpus.exclusive`는 non-root cpuset-enabled cgroup에 존재하는 read/write multiple-values file입니다. 새 cpuset partition을 만드는 데 사용할 수 있는 exclusive CPU를 나열하며, cgroup이 valid partition root가 되기 전에는 이 값이 사용되지 않습니다.
Cgroup이 partition root가 되면 실제로 partition에 allocate된 exclusive CPU는 `cpuset.cpus.exclusive.effective`에 나타나며 requested `cpuset.cpus.exclusive`와 다를 수 있습니다. Requested 값이 설정되어 있다면 effective 값은 항상 그 subset입니다.
Partition 후보 CPU와 실제 partition allocation을 구분합니다.
사용자는 `cpuset.cpus`와 다른 값을 수동 설정할 수 있습니다. 다만 sibling의 `cpuset.cpus.exclusive`와 `cpuset.cpus.exclusive.effective`에 대해 CPU 목록이 exclusive해야 합니다.
또한 exclusive CPU를 떼어 간 뒤에도 sibling에 최소 한 CPU가 남도록, 이 목록은 sibling의 `cpuset.cpus`를 전부 포함하는 superset이 될 수 없습니다.
Parent cgroup의 exclusive CPU 하나는 child cgroup 하나에만 distribute할 수 있습니다. 같은 exclusive CPU가 둘 이상의 child에 나타나는 것은 exclusivity rule 위반이며 해당 write는 error로 reject됩니다.
Requested set은 sibling overlap과 CPU starvation을 모두 피해야 합니다.
Root cgroup은 partition root이며 available CPU 전체가 root의 exclusive CPU set에 포함됩니다.
Effective exclusive와 isolated CPU 집계
2555-2574`cpuset.cpus.exclusive.effective`는 모든 non-root cpuset-enabled cgroup에 존재하는 read-only multiple-values file입니다. Partition root를 만들 때 사용할 수 있는 effective exclusive CPU set을 보여 줍니다.
Parent가 root가 아니면 이 값은 parent의 `cpuset.cpus.exclusive.effective` subset입니다. Local `cpuset.cpus.exclusive`가 설정되어 있으면 그 subset이기도 합니다.
Local partition을 만들 때 `cpuset.cpus.exclusive`가 설정되지 않았다면 `cpuset.cpus`와 같은 implicit value가 있는 것으로 처리합니다.
`cpuset.cpus.isolated`는 root cgroup에만 존재하는 read-only multiple-values file입니다. Existing isolated partition에서 사용하는 모든 isolated CPU set을 보여 주며 isolated partition이 없으면 empty입니다.
Partition formation용 grant와 system-wide isolated aggregate를 제공합니다.
Cpuset partition과 local·remote 형성
2575-2609`cpuset.cpus.partition`은 non-root cpuset-enabled cgroup에 존재하는 read/write single-value file입니다. 이 flag는 parent cgroup이 소유하며 delegatable하지 않습니다.
Write 시 허용되는 세 state입니다.
Cpuset partition은 hierarchy 맨 위의 partition root와 그 descendant인 cpuset-enabled cgroup의 collection입니다. 별도 partition root와 그 descendant는 이 collection에서 제외됩니다.
Partition은 allocate된 exclusive CPU set에 exclusive access합니다. Partition 밖의 다른 cgroup은 이 set의 CPU를 사용할 수 없습니다.
Nested partition root는 parent partition collection에서 새 독립 경계를 만듭니다.
Partition에는 local과 remote 두 type이 있습니다. Local partition은 parent cgroup도 valid partition root인 경우이고, remote partition은 parent가 valid partition root가 아닌 경우입니다.
Parent state와 exclusive CPU 사전 구성 요구가 다릅니다.
현재 remote partition은 local partition 아래에 만들 수 없습니다. Remote partition root의 ancestor 중 root cgroup을 제외한 어느 것도 partition root일 수 없습니다.
Partition state와 isolated operation
2610-2643Root cgroup은 항상 partition root이며 state를 바꿀 수 없습니다. 그 밖의 모든 non-root cgroup은 `member`로 시작합니다.
`root`로 설정하면 current cgroup이 새 partition 또는 scheduling domain의 root가 됩니다. Exclusive CPU set은 `cpuset.cpus.exclusive.effective` 값으로 결정됩니다.
`isolated`로 설정하면 partition CPU는 scheduler load balancing이 없는 isolated state가 되고 unbound workqueue에서도 제외됩니다. 여러 CPU를 가진 isolated partition의 task는 optimal performance를 위해 각 CPU에 신중히 distribute하고 bind해야 합니다.
Member, balanced root와 isolated root의 runtime behavior입니다.
Partition root(`root` 또는 `isolated`)는 valid 또는 invalid일 수 있습니다. Invalid root는 일부 state information을 유지할 수 있지만 `member`처럼 동작하는 degraded state입니다.
`member`, `root`, `isolated` 사이의 모든 state transition이 허용됩니다.
Writable state는 서로 자유롭게 전환되지만 root 계열은 별도로 validity를 가집니다.
`cpuset.cpus.partition`을 read하면 `member`, `root`, `isolated`, `root invalid (<reason>)`, `isolated invalid (<reason>)` 중 하나가 나타납니다. Invalid partition root에는 invalid 이유를 설명하는 string이 parentheses 안에 포함됩니다.
Configured mode와 validity reason을 한 값에 함께 표시합니다.
Partition validity, monitoring과 recovery
2644-2686Local partition root가 valid하려면 다음 조건을 모두 만족해야 합니다.
Remote root는 local 조건에서 valid-parent requirement만 제외합니다.
Hotplug, `cpuset.cpus` 또는 `cpuset.cpus.exclusive` 변경 같은 external event는 valid partition root를 invalid로 만들거나 반대로 invalid root를 valid로 만들 수 있습니다. `cpuset.cpus.effective`가 empty인 cgroup으로는 task를 이동할 수 없습니다.
Resource availability와 configuration change가 partition validity를 재평가합니다.
Valid non-root parent partition에 associated task가 없으면 CPU 전체를 child local partition에 distribute할 수 있습니다.
Valid partition root를 `member`로 바꾸면 child local partition이 모두 invalid가 되어 그 안에서 실행 중인 task에 disruption을 줄 수 있으므로 주의해야 합니다.
비활성화된 child partition은 parent를 다시 partition root로 바꾸고 `cpuset.cpus` 또는 `cpuset.cpus.exclusive`를 올바르게 설정하면 recover할 수 있습니다.
Parent mode 변경은 local child validity에 연쇄적으로 영향을 줍니다.
`cpuset.cpus.partition` state가 바뀔 때마다 poll과 inotify event가 trigger됩니다. 직접 write, CPU hotplug 또는 validity status를 바꾸는 다른 변경이 모두 포함되므로 userspace agent는 continuous polling 없이 unexpected change를 monitor할 수 있습니다.
여러 원인의 state change를 event-driven userspace monitoring에 연결합니다.
`isolcpus` kernel boot command-line option으로 일부 CPU를 boot 시 load balancing이 disable된 isolated state로 미리 구성할 수 있습니다. 이 CPU를 partition에 넣으려면 isolated partition에서 사용해야 합니다.
Cgroup BPF 기반 device access control
2687-2710Device controller는 device file access를 관리합니다. `mknod`를 사용한 새 device file 생성과 existing device file access를 모두 포함합니다.
Cgroup v2 device controller에는 interface file이 없으며 cgroup BPF 위에 구현됩니다. Device file access를 control하려면 `BPF_PROG_TYPE_CGROUP_DEVICE` type의 BPF program을 만들고 `BPF_CGROUP_DEVICE` flag로 cgroup에 attach할 수 있습니다.
Device file access를 시도하면 해당 BPF program이 실행됩니다. Return value가 0이면 attempt가 `-EPERM`으로 실패하고, 0이 아니면 성공합니다.
Device operation마다 attached cgroup BPF program이 허용 여부를 결정합니다.
`BPF_PROG_TYPE_CGROUP_DEVICE` program은 device access attempt를 설명하는 `bpf_cgroup_dev_ctx` structure pointer를 받습니다. Context에는 access type(`mknod`/`read`/`write`)과 device type·major·minor number가 들어 있습니다.
BPF policy가 판단할 device access context입니다.
Example `BPF_PROG_TYPE_CGROUP_DEVICE` program은 kernel source tree의 `tools/testing/selftests/bpf/progs/dev_cgroup.c`에서 볼 수 있습니다.
RDMA resource limit와 accounting
2711-2749`rdma` controller는 RDMA resource의 distribution과 accounting을 조절합니다.
`rdma.max`는 root를 제외한 모든 cgroup에 존재하는 read/write nested-keyed file로 RDMA/IB device에 설정된 current resource limit을 설명합니다. Line은 순서 없는 device-name key이며, space-separated resource name과 distribute 가능한 configured limit을 포함합니다.
RDMA device별 HCA handle과 object 수를 제한합니다.
`mlx4`와 `ocrdma` device의 `rdma.max` 예시는 다음과 같습니다.
mlx4_0 hca_handle=2 hca_object=2000
ocrdma1 hca_handle=3 hca_object=max`rdma.current`는 current resource usage를 설명하는 read-only file이며 root를 제외한 모든 cgroup에 존재합니다.
mlx4_0 hca_handle=1 hca_object=20
ocrdma1 hca_handle=1 hca_object=23Configured ceiling과 observed usage를 같은 device/resource key로 비교합니다.
Device memory region accounting
2750-2792`dmem` controller는 device memory region의 distribution과 accounting을 조절합니다. Region마다 system page size와 같지 않은 고유 page size를 사용할 수 있으므로 unit은 항상 byte입니다.
`dmem.max`, `dmem.min`, `dmem.low`는 root를 제외한 모든 cgroup에 존재하는 read/write nested-keyed file로 region별 configured resource limit을 설명합니다.
`xe` device 예시는 다음과 같습니다.
drm/0000:03:00.0/vram0 1073741824
drm/0000:03:00.0/stolen max이 interface의 semantics와 계산 방식은 memory cgroup controller의 `max`, `min`, `low`와 같습니다.
`dmem.capacity`는 maximum region capacity를 보여 주는 read-only file이며 root cgroup에만 존재합니다. Kernel이 internal use를 위해 일부 memory를 reserve하므로 모든 capacity를 cgroup이 allocate할 수 있는 것은 아닙니다.
drm/0000:03:00.0/vram0 8514437120
drm/0000:03:00.0/stolen 67108864`dmem.current`는 current resource usage를 보여 주는 read-only file이며 root를 제외한 모든 cgroup에 존재합니다.
drm/0000:03:00.0/vram0 12550144
drm/0000:03:00.0/stolen 8650752Region limit, platform capacity와 current cgroup usage의 scope입니다.
Region-specific page size 차이를 숨기기 위해 모든 interface가 byte로 통일됩니다.
HugeTLB limit와 NUMA statistics
2793-2825HugeTLB controller는 control group별 HugeTLB usage를 제한하며 page fault 중 controller limit을 enforce합니다.
Huge page size별 usage, hard limit, event와 NUMA placement를 제공합니다.
`hugetlb.<hugepagesize>.events`의 `max` entry는 HugeTLB limit 때문에 allocation이 실패한 횟수입니다.
`hugetlb.<hugepagesize>.events.local`은 hierarchical하지 않고 current cgroup에 local합니다. 이 file의 file-modified event도 local event만 반영합니다.
`hugetlb.<hugepagesize>.numa_stat`은 `memory.numa_stat`과 비슷하게 이 cgroup의 `<hugepagesize>` HugeTLB page에 대한 NUMA information을 보여 줍니다. Active하게 사용 중인 page만 포함하며 per-node value unit은 byte입니다.
Huge page allocation과 page fault 경로에서 hard limit failure를 event로 기록합니다.
Misc scalar resource registration
2826-2842Miscellaneous cgroup은 다른 cgroup resource처럼 abstract할 수 없는 scalar resource를 limit하고 track하는 mechanism을 제공합니다. Controller는 `CONFIG_CGROUP_MISC` config option으로 enable됩니다.
Resource는 `include/linux/misc_cgroup.h`의 `enum misc_res_type{}`에 추가하고 대응 name은 `kernel/cgroup/misc.c`의 `misc_res_name[]`에 추가합니다.
Resource provider는 resource를 사용하기 전에 `misc_cg_set_capacity()`를 call해 capacity를 설정해야 합니다. Capacity가 설정되면 charge와 uncharge API로 usage를 갱신할 수 있습니다.
Misc controller와 상호작용하는 모든 API는 `include/linux/misc_cgroup.h`에 있습니다.
새 scalar resource를 등록하고 capacity를 설정한 뒤 usage를 account합니다.
Resource type, name와 controller API의 source location입니다.
Misc capacity, usage, limit와 events
2843-2907원문은 Miscellaneous controller가 3개 interface file을 제공한다고 서술한 뒤, 두 resource `res_a`, `res_b`가 등록된 경우의 다음 interface들을 설명합니다.
`misc.capacity`는 root cgroup에만 표시되는 read-only flat-keyed file로 platform에서 available한 miscellaneous scalar resource와 quantity를 보여 줍니다.
$ cat misc.capacity
res_a 50
res_b 10`misc.current`는 모든 cgroup에 표시되는 read-only flat-keyed file로 cgroup과 child의 current resource usage를 보여 줍니다.
$ cat misc.current
res_a 3
res_b 0`misc.peak`는 모든 cgroup에 표시되는 read-only flat-keyed file로 cgroup과 child가 기록한 historical maximum resource usage를 보여 줍니다.
$ cat misc.peak
res_a 10
res_b 8`misc.max`는 non-root cgroup에 표시되는 read/write flat-keyed file로 cgroup과 child의 allowed maximum usage를 설정합니다.
$ cat misc.max
res_a max
res_b 4Limit은 다음과 같이 specific value로 설정할 수 있습니다.
# echo res_a 1 > misc.max다음과 같이 `max`로 되돌릴 수도 있습니다.
# echo res_a max > misc.max`misc.max` limit은 `misc.capacity`에 표시된 capacity value보다 높게 설정할 수 있습니다.
Platform capacity, hierarchical usage·peak, configurable ceiling을 구분합니다.
`misc.events`는 non-root cgroup의 read-only flat-keyed file입니다. 별도 설명이 없으면 value change가 file-modified event를 생성하고 모든 field는 hierarchical합니다. `max`는 resource usage가 max boundary를 넘으려 한 횟수입니다.
`misc.events.local`은 `misc.events`와 비슷하지만 field가 current cgroup에 local하고 hierarchical하지 않습니다. 이 file의 file-modified event도 local event만 반영합니다.
Resource ceiling attempt를 hierarchy와 local view로 각각 노출합니다.
Misc resource charge ownership
2908-2915Miscellaneous scalar resource는 처음 사용된 cgroup에 charge되며 resource가 free될 때까지 그 cgroup에 계속 charge됩니다. Process를 다른 cgroup으로 migrate해도 charge는 process가 이동한 destination cgroup으로 옮겨지지 않습니다.
Process membership과 resource charge ownership은 migration 이후에도 분리되어 유지됩니다.
Other controller: perf_event
2916-2927`perf_event` controller가 legacy hierarchy에 mount되어 있지 않으면 v2 hierarchy에서 automatic하게 enable됩니다. 따라서 perf event를 항상 cgroup v2 path로 filter할 수 있습니다.
V2 hierarchy가 populated된 뒤에도 controller를 legacy hierarchy로 이동할 수 있습니다.
Legacy mount 여부에 따른 automatic v2 enable과 이후 이동 가능성을 나타냅니다.
Root cgroup CPU·IO process behavior
2928-2956이 section은 stable kernel API의 일부로 간주되지 않는 non-normative information을 담으므로 변경될 수 있습니다.
Root cgroup에서 CPU cycle을 distribute할 때 각 thread는 root의 별도 child cgroup에 host된 것처럼 처리됩니다. 이 implicit child weight는 thread nice level에 따라 달라집니다.
Mapping detail은 `kernel/sched/core.c`의 `sched_prio_to_weight` array를 참조하십시오. 이 array 값은 neutral `nice 0` 값이 1024 대신 100이 되도록 적절히 scale해야 합니다.
Root cgroup process는 IO distribution에서 implicit leaf child node에 host됩니다. 이 node는 weight 200인 일반 root child cgroup처럼 resource distribution에 포함됩니다.
Root 안의 직접 process를 controller별 implicit child entity로 취급합니다.
Cgroup namespace basics와 lifetime
2957-3006Cgroup namespace는 `/proc/$PID/cgroup` file과 cgroup mount view를 virtualize하는 mechanism입니다. `CLONE_NEWCGROUP` clone flag를 `clone(2)`와 `unshare(2)`에 사용해 새 cgroup namespace를 만들 수 있습니다.
Namespace 안 process의 `/proc/$PID/cgroup` output은 cgroupns root로 제한됩니다. Cgroupns root는 cgroup namespace를 만들 당시 process가 속한 cgroup입니다.
Creation 시점의 cgroup이 새 namespace의 stable root view가 됩니다.
Cgroup namespace가 없으면 `/proc/$PID/cgroup`은 process cgroup의 complete path를 보여 줍니다. Container처럼 cgroup과 namespace로 process를 isolate하는 환경에서는 이 path가 system-level information을 leak할 수 있습니다.
예를 들어 다음 path는 isolated process에 노출하기 곤란한 system data일 수 있습니다.
# cat /proc/self/cgroup
0::/batchjobs/container_id1새 cgroup namespace를 만들기 전에는 namespace inode와 complete cgroup path가 다음과 같이 보입니다.
# ls -l /proc/self/ns/cgroup
lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835]
# cat /proc/self/cgroup
0::/batchjobs/container_id1새 namespace를 `unshare`한 뒤에는 namespace inode가 바뀌고 cgroup view의 root가 `/`로 보입니다.
# ls -l /proc/self/ns/cgroup
lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183]
# cat /proc/self/cgroup
0::/Namespace 도입 전후 container가 보는 cgroup path 범위입니다.
Multi-threaded process의 어느 thread가 cgroup namespace를 unshare하면 새 cgroupns는 process 전체, 즉 모든 thread에 적용됩니다. 이는 v2 hierarchy에서는 자연스럽지만 legacy hierarchy에서는 예상 밖일 수 있습니다.
Cgroup namespace 안에 process가 있거나 mount가 namespace를 pin하는 동안 namespace는 유지됩니다. 마지막 usage가 사라지면 namespace만 destroy되고 cgroupns root와 actual cgroup은 남습니다.
Process와 mount reference가 namespace object의 lifetime을 결정합니다.
Namespace root와 caller-specific views
3007-3057Cgroup namespace의 `cgroupns root`는 `unshare(2)`를 call한 process가 실행 중인 cgroup입니다. 예를 들어 `/batchjobs/container_id1`의 process가 unshare하면 그 cgroup이 root가 됩니다. `init_cgroup_ns`에서는 real root(`/`)가 root입니다.
Namespace creator process가 나중에 다른 cgroup으로 이동해도 cgroupns root는 바뀌지 않습니다.
# ~/unshare -c # unshare cgroupns in some cgroup
# cat /proc/self/cgroup
0::/
# mkdir sub_cgrp_1
# echo 0 > sub_cgrp_1/cgroup.procs
# cat /proc/self/cgroup
0::/sub_cgrp_1Creator 이동은 process-relative path만 바꾸고 namespace root identity는 바꾸지 않습니다.
각 process는 `/proc/$PID/cgroup`에 대해 namespace-specific view를 얻습니다. Cgroup namespace 안 process는 자신의 root cgroup 내부 path만 볼 수 있습니다.
Unshared cgroup namespace 안에서 child process를 `sub_cgrp_1`으로 이동하면 다음과 같이 namespace-relative path가 보입니다.
# sleep 100000 &
[1] 7353
# echo 7353 > sub_cgrp_1/cgroup.procs
# cat /proc/7353/cgroup
0::/sub_cgrp_1Initial cgroup namespace에서는 같은 process의 real cgroup path가 보입니다.
$ cat /proc/7353/cgroup
0::/batchjobs/container_id1/sub_cgrp_1다른 cgroup에 root를 둔 sibling cgroup namespace에서는 caller 자신의 cgroup namespace root에 상대적인 path를 보여 줍니다. 원문 예시는 다음과 같습니다.
# cat /proc/7353/cgroup
0::/../container_id2/sub_cgrp_1Relative path는 caller의 cgroup namespace root에 상대적임을 나타내기 위해 항상 `/`로 시작합니다.
같은 PID도 reader의 cgroup namespace에 따라 다른 path로 표시됩니다.
Namespace 밖 migration과 setns(2)
3058-3086Cgroup namespace 안 process는 external cgroup에 proper access가 있으면 namespace root 안팎으로 이동할 수 있습니다. 다음 예시는 cgroupns root가 `/batchjobs/container_id1`이고 global hierarchy가 namespace 안에서도 accessible한 경우입니다.
# cat /proc/7353/cgroup
0::/sub_cgrp_1
# echo 7353 > batchjobs/container_id2/cgroup.procs
# cat /proc/7353/cgroup
0::/../container_id2이런 setup은 권장되지 않습니다. Cgroup namespace 안 task에는 자신의 cgroupns hierarchy만 노출해야 합니다.
Global hierarchy 노출은 namespace root 밖 이동과 상대 path 노출을 가능하게 합니다.
다른 cgroup namespace로 `setns(2)`하려면 process가 current user namespace에 대해 `CAP_SYS_ADMIN`을 가져야 하고 target cgroup namespace의 userns에 대해서도 `CAP_SYS_ADMIN`을 가져야 합니다.
두 user-namespace 관점의 capability가 모두 필요합니다.
다른 cgroup namespace에 attach해도 implicit cgroup change는 일어나지 않습니다. Attach한 process는 별도로 target cgroup namespace root 아래로 이동시켜야 합니다.
Namespace-private cgroupfs mount
3087-3103Non-init cgroup namespace 안에서 실행하는 process는 namespace-specific cgroup hierarchy를 mount할 수 있습니다.
# mount -t cgroup2 none $MOUNT_POINT이 mount는 cgroupns root를 filesystem root로 삼아 unified cgroup hierarchy를 mount합니다. Process는 자신의 user namespace와 mount namespace에 대해 `CAP_SYS_ADMIN`이 필요합니다.
`/proc/self/cgroup` virtualization과 namespace-private cgroupfs mount로 cgroup hierarchy view를 제한하면 container 안에 제대로 isolate된 cgroup view를 제공할 수 있습니다.
Proc path virtualization과 private filesystem root restriction을 결합합니다.
Kernel programming information scope
3104-3111이 section은 cgroup과 interaction해야 하는 영역의 kernel programming information을 담습니다. Cgroup core와 controller 자체의 구현은 다루지 않습니다.
Filesystem cgroup writeback support
3112-3146Filesystem은 `address_space_operations->writepages()`를 update해 다음 두 function으로 bio를 annotate하면 cgroup writeback을 지원할 수 있습니다.
Bio-level owner binding과 data-segment-level ownership accounting을 나눕니다.
`wbc_init_bio()`의 필수 ordering과 segment accounting 위치입니다.
Writeback bio가 annotate되면 `super_block`의 `->s_iflags`에 `SB_I_CGROUPWB`를 설정해 cgroup support를 enable할 수 있습니다. Journaled data mode처럼 특정 filesystem feature가 incompatible할 때 cgroup writeback support를 선택적으로 disable하는 데 유용합니다.
`wbc_init_bio()`는 specified bio를 cgroup에 bind합니다. Configuration에 따라 bio가 lower priority로 실행될 수 있고, writeback session이 journal entry 같은 shared resource를 hold하고 있으면 priority inversion이 발생할 수 있습니다.
이 문제에는 하나의 쉬운 solution이 없습니다. Filesystem은 특정 문제 case에서 `wbc_init_bio()`를 생략하고 `bio_associate_blkg()`를 직접 사용해 우회할 수 있습니다.
Shared resource를 잡은 lower-priority bio가 higher-priority work를 막을 수 있습니다.
Deprecated v1 core features
3147-3161Cgroup v2에서는 다음 v1 core feature를 지원하지 않거나 제거했습니다.
V2에서 제거된 hierarchy, mount와 core interface behavior입니다.
V1 multiple-hierarchy 문제와 v2 rationale
3162-3217Cgroup v1은 arbitrary number의 hierarchy를 허용했고 각 hierarchy가 임의 수의 controller를 host할 수 있었습니다. 겉보기에는 매우 flexible했지만 실제로는 유용하지 않았습니다.
각 controller instance는 하나뿐이므로 모든 hierarchy에서 유용한 freezer 같은 utility controller도 한 hierarchy에서만 사용할 수 있었습니다. Hierarchy가 populated되면 controller를 다른 hierarchy로 이동할 수도 없었습니다.
한 hierarchy에 bind된 모든 controller는 정확히 같은 hierarchy view를 강제받았고 controller별로 granularity를 다르게 할 수 없었습니다.
표면적 flexibility가 실제 controller placement와 operation을 제한한 이유입니다.
결국 같은 hierarchy에 둘 수 있는 controller 조합이 크게 제한되어 대부분의 configuration은 controller마다 별도 hierarchy를 사용했습니다. `cpu`와 `cpuacct`처럼 밀접한 controller만 함께 두는 것이 자연스러웠습니다.
Userland는 비슷한 hierarchy 여러 개를 관리하며 hierarchy operation마다 같은 step을 반복해야 했습니다. Multiple hierarchy support는 cgroup core implementation을 크게 복잡하게 만들었고 cgroup의 일반적 사용 방식과 controller 기능까지 제한했습니다.
Controller별 분리와 membership 식별용 controller가 hierarchy 수를 더 늘렸습니다.
Hierarchy 수에 limit이 없어서 thread의 cgroup membership을 finite length로 표현할 수 없었습니다. Key는 임의 수의 entry를 가져 길이 제한이 없었고 조작하기 매우 불편했습니다. Membership 식별만 하는 controller까지 추가되어 hierarchy 증가 문제를 더 악화시켰습니다.
Controller는 다른 controller hierarchy의 topology를 예상할 수 없어서 모두가 완전히 orthogonal hierarchy에 있다고 가정해야 했습니다. 이 때문에 controller cooperation이 불가능하거나 매우 번거로웠습니다.
대부분의 use case에서 완전히 orthogonal한 hierarchy는 필요하지 않습니다. 실제 요구는 controller별 granularity 차이이며, 특정 controller view에서는 leaf에서 root 방향으로 hierarchy를 collapse할 수 있어야 합니다.
예를 들어 어떤 level 아래 memory distribution은 구분하지 않으면서 CPU cycle distribution은 계속 세밀하게 control할 수 있어야 합니다.
V2가 필요한 핵심은 별도 hierarchy가 아니라 controller view별 subtree collapse입니다.
V1 thread granularity와 delegation 문제
3218-3261Cgroup v1은 한 process의 thread가 서로 다른 cgroup에 속하도록 허용했습니다. 일부 controller에는 이 구성이 의미가 없어 각자 무시하는 방식을 구현했고, 더 중요하게는 individual application에 노출하는 API와 system-management interface의 경계를 흐렸습니다.
일반적으로 in-process knowledge는 process 자신만 가지고 있습니다. 따라서 service-level process organization과 달리 process thread를 분류하려면 target process를 소유한 application의 active participation이 필요합니다.
System manager가 알 수 있는 service 경계와 application만 아는 in-process 경계를 구분합니다.
V1의 모호한 delegation model은 thread granularity와 결합되어 남용되었습니다. Individual application에 cgroup을 delegate해 자체 sub-hierarchy를 만들고 관리하며 resource distribution을 control하게 했고, 결과적으로 cgroup을 일반 program에 노출되는 syscall-like API 수준으로 끌어올렸습니다.
하지만 cgroup interface는 이 용도로 노출하기에 근본적으로 부적절했습니다. Process가 자신의 knob에 접근하려면 `/proc/self/cgroup`에서 target hierarchy path를 추출하고, knob name을 append해 path를 만들고, file을 open한 뒤 read/write해야 했습니다.
이 절차는 매우 clunky하고 이례적일 뿐 아니라 본질적으로 racy합니다. 여러 step을 하나의 transaction으로 정의할 conventional 방법이 없고 process가 실제로 자신의 sub-hierarchy를 조작한다는 보장도 없습니다.
Application이 own cgroup knob를 찾는 multi-step path construction에는 atomicity가 없습니다.
Controller는 public API였다면 받아들여지지 않았을 knob를 system-management pseudo filesystem에 추가했습니다. 제대로 abstract하거나 refine하지 않은 채 kernel internal detail을 직접 노출한 knob가 ill-defined delegation을 통해 application에 전달되어, 필요한 scrutiny 없이 public API를 만드는 shortcut처럼 cgroup이 악용되었습니다.
그 결과 userland는 잘못 동작하고 poor하게 abstract된 interface를 떠안았고, kernel은 의도치 않게 노출한 construct에 호환성 제약으로 묶였습니다.
System-management knob의 application API화가 양쪽에 장기 호환성 부담을 남겼습니다.
Inner node와 thread competition
3262-3302Cgroup v1은 thread를 어느 cgroup에도 둘 수 있어 parent cgroup의 thread와 child cgroup이 resource를 두고 경쟁하는 문제가 생겼습니다. 서로 다른 type의 entity가 경쟁했으며 명확한 해결 방식이 없어 controller마다 다르게 처리했습니다.
CPU, IO와 memory controller가 parent task와 child cgroup competition을 서로 다르게 처리했습니다.
CPU controller는 thread와 cgroup을 equivalent하게 보고 nice level을 cgroup weight로 map했습니다. 일부 case에서는 동작했지만 child가 특정 CPU-cycle ratio를 원하고 internal thread 수가 변하면 경쟁 entity 수에 따라 ratio가 계속 바뀌었습니다.
Nice-to-weight mapping도 obvious하거나 universal하지 않았고 thread에는 사용할 수 없는 다른 knob도 많았습니다.
IO controller는 thread를 host할 hidden leaf node를 각 cgroup에 implicit하게 만들고 모든 knob의 `leaf_` prefix copy를 제공했습니다. Internal thread를 동등하게 control할 수 있었지만 불필요한 nesting layer, messy interface와 큰 implementation complexity를 초래했습니다.
Memory controller는 internal task와 child cgroup 사이 동작을 control할 방법이 없고 behavior도 명확하지 않았습니다. 특정 workload에 맞춘 ad-hoc behavior와 knob를 추가하려는 시도는 장기적으로 해결하기 매우 어려운 문제로 이어질 수 있었습니다.
여러 controller가 internal task를 각기 다른 심각한 결함이 있는 방식으로 다뤄 cgroup 전체가 매우 inconsistent해졌습니다. 이 문제는 cgroup core가 uniform하게 해결해야 합니다.
Controller별 workaround가 interface와 semantics 분화를 키웠습니다.
V1 interface inconsistency와 v2 convention
3303-3331Cgroup v1은 oversight 없이 성장하며 많은 idiosyncrasy와 inconsistency를 만들었습니다.
Cgroup core의 한 문제는 empty cgroup notification이었습니다. Event마다 userland helper binary를 fork하고 execute했으며 event delivery는 recursive하지도 delegatable하지도 않았습니다. 이 mechanism의 한계 때문에 in-kernel event-delivery filtering까지 추가되어 interface가 더 복잡해졌습니다.
Controller interface도 문제가 있었습니다. 극단적으로 hierarchy organization을 완전히 무시하고 모든 cgroup을 root 바로 아래에 있는 것처럼 취급하는 controller가 있었으며, 일부는 inconsistent한 implementation detail을 userland에 대량 노출했습니다.
Controller 사이 consistency도 없었습니다. 새 cgroup 생성 시 일부 controller는 추가 restriction을 두지 않았지만 다른 controller는 explicit configuration 전에는 resource usage를 모두 금지했습니다.
같은 type의 control도 configuration knob name과 format이 크게 달랐고 statistic·information knob는 arbitrary한 name과 서로 다른 format·unit을 사용했으며 같은 controller 안에서도 일관되지 않았습니다.
Core notification부터 controller defaults와 statistics까지 공통 convention이 없었습니다.
Cgroup v2는 적절한 영역에 common convention을 정립하고 controller가 minimal하고 consistent한 interface를 expose하도록 update합니다.
공통 convention과 최소 interface가 controller 간 예측 가능성을 회복합니다.
Memory controller issues와 v2 remedies
3332-3407Original lower boundary인 soft limit은 default로 unset이므로 global reclaim이 선호하는 cgroup set이 opt-out이 아니라 opt-in입니다. 대부분 negative lookup을 optimize하는 cost가 너무 커서 implementation은 매우 큰데도 기본적으로 바람직한 behavior조차 제공하지 못합니다.
Soft limit에는 hierarchical meaning이 없습니다. Configured group은 hierarchy 위치와 무관하게 global rbtree에 equal peer로 organize되어 subtree delegation이 불가능합니다.
Soft-limit reclaim pass는 지나치게 aggressive하여 high allocation latency와 overreclaim에 따른 system-performance 저하를 일으켜 feature 자체의 목적을 무너뜨립니다.
반면 `memory.low` boundary는 top-down으로 allocate되는 reserve입니다. Cgroup은 effective low 안에 있을 때 reclaim protection을 받아 subtree delegation이 가능하고, effective low를 넘으면 overage에 비례한 reclaim pressure를 받습니다.
V1 soft limit과 v2 memory.low의 hierarchy·reclaim semantics 차이입니다.
Original high boundary인 hard limit은 OOM killer를 call해야 하더라도 움직일 수 없는 strict limit입니다. 이는 available memory를 최대한 활용하려는 목표와 대체로 충돌합니다.
Runtime workload memory consumption은 변하므로 overcommit이 필요하지만 strict upper limit에서는 working-set size를 매우 정확히 예측하거나 slack을 추가해야 합니다. Working-set estimation은 어렵고 error-prone하며 틀리면 OOM kill이 발생하므로 사용자는 보통 느슨한 limit을 택해 귀중한 resource를 낭비합니다.
반면 `memory.high`는 훨씬 conservative하게 설정할 수 있습니다. Boundary에 닿으면 allocation을 direct reclaim으로 보내 excess를 줄이도록 throttle하지만 OOM killer는 call하지 않습니다.
따라서 너무 aggressive한 high boundary는 process를 terminate하지 않고 gradual performance degradation을 만듭니다. User는 이를 monitor하고 acceptable performance를 유지하는 minimal memory footprint를 찾을 때까지 조정할 수 있습니다.
OOM 대신 관찰 가능한 performance degradation으로 working set을 조정합니다.
Concurrent allocation이 많고 group 안 reclaim progress가 완전히 무너지면 high boundary를 넘을 수 있습니다. 이 경우에도 group을 kill하기보다 다른 group 또는 system의 slack에서 allocation을 만족시키는 편이 대체로 낫습니다.
`memory.max`는 이런 spillover를 제한하고 buggy하거나 malicious한 application을 최종적으로 contain합니다.
Adaptive throttle과 ultimate containment를 별도 boundary로 제공합니다.
Original `memory.limit_in_bytes`를 current usage 아래로 설정하면 concurrent charge 때문에 limit-setting이 실패하는 race condition이 있었습니다.
반면 `memory.max`는 먼저 limit을 설정해 new charge를 막고, 새 limit을 만족할 때까지 reclaim과 OOM kill을 수행합니다. 이 과정은 limit이 충족되거나 `memory.max`에 write한 task가 kill될 때 끝납니다.
V2는 먼저 ceiling을 게시해 concurrent charge race를 닫은 뒤 usage를 낮춥니다.
Combined memory+swap accounting과 limiting은 swap space에 대한 실제 control로 대체되었습니다.
Original design에서 combined facility의 핵심 논리는 global 또는 parental pressure가 child의 신뢰할 수 없는 configuration과 무관하게 anonymous memory 전체를 swap할 수 있어야 한다는 것이었습니다.
그러나 untrusted group은 anonymous memory를 tight loop에서 계속 reference하는 등 다른 방식으로 swapping을 방해할 수 있습니다. 따라서 admin은 untrusted job을 overcommit할 때 완전한 swappability를 가정할 수 없습니다.
Trusted job에 combined counter는 intuitive한 userspace interface가 아니며 cgroup controller가 specific physical resource를 account하고 limit해야 한다는 원칙에도 어긋납니다. Swap space도 system의 다른 resource와 같으므로 unified hierarchy는 이를 별도로 distribute합니다.
Combined counter의 신뢰 가정과 v2의 physical-resource separation을 비교합니다.
Hierarchy model, operations and delegation
cgroup-v2.rst:1-742Cgroup v2의 용어, mount와 process·thread 조직, controller enable constraint, delegation containment, 권장 hierarchy organization과 resource distribution model을 정리합니다.