요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
.. SPDX-License-Identifier: GPL-2.0
2
.. include:: <isonum.txt>
4
.. |struct cpuidle_state| replace:: :c:type:`struct cpuidle_state <cpuidle_state>`
5
.. |cpufreq| replace:: :doc:`CPU Performance Scaling <cpufreq>`
7
========================
8
CPU Idle Time Management
9
========================
11
:Copyright: |copy| 2018 Intel Corporation
13
:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
16
Concepts
17
========
19
Modern processors are generally able to enter states in which the execution of
20
a program is suspended and instructions belonging to it are not fetched from
21
memory or executed. Those states are the *idle* states of the processor.
23
Since part of the processor hardware is not used in idle states, entering them
24
generally allows power drawn by the processor to be reduced and, in consequence,
25
it is an opportunity to save energy.
27
CPU idle time management is an energy-efficiency feature concerned about using
28
the idle states of processors for this purpose.
30
Logical CPUs
31
------------
33
CPU idle time management operates on CPUs as seen by the *CPU scheduler* (that
34
is the part of the kernel responsible for the distribution of computational
35
work in the system). In its view, CPUs are *logical* units. That is, they need
36
not be separate physical entities and may just be interfaces appearing to
37
software as individual single-core processors. In other words, a CPU is an
38
entity which appears to be fetching instructions that belong to one sequence
39
(program) from memory and executing them, but it need not work this way
40
physically. Generally, three different cases can be consider here.
42
First, if the whole processor can only follow one sequence of instructions (one
43
program) at a time, it is a CPU. In that case, if the hardware is asked to
44
enter an idle state, that applies to the processor as a whole.
46
Second, if the processor is multi-core, each core in it is able to follow at
47
least one program at a time. The cores need not be entirely independent of each
48
other (for example, they may share caches), but still most of the time they
49
work physically in parallel with each other, so if each of them executes only
50
one program, those programs run mostly independently of each other at the same
51
time. The entire cores are CPUs in that case and if the hardware is asked to
52
enter an idle state, that applies to the core that asked for it in the first
53
place, but it also may apply to a larger unit (say a "package" or a "cluster")
54
that the core belongs to (in fact, it may apply to an entire hierarchy of larger
55
units containing the core). Namely, if all of the cores in the larger unit
56
except for one have been put into idle states at the "core level" and the
57
remaining core asks the processor to enter an idle state, that may trigger it
58
to put the whole larger unit into an idle state which also will affect the
59
other cores in that unit.
61
Finally, each core in a multi-core processor may be able to follow more than one
62
program in the same time frame (that is, each core may be able to fetch
63
instructions from multiple locations in memory and execute them in the same time
64
frame, but not necessarily entirely in parallel with each other). In that case
65
the cores present themselves to software as "bundles" each consisting of
66
multiple individual single-core "processors", referred to as *hardware threads*
67
(or hyper-threads specifically on Intel hardware), that each can follow one
68
sequence of instructions. Then, the hardware threads are CPUs from the CPU idle
69
time management perspective and if the processor is asked to enter an idle state
70
by one of them, the hardware thread (or CPU) that asked for it is stopped, but
71
nothing more happens, unless all of the other hardware threads within the same
72
core also have asked the processor to enter an idle state. In that situation,
73
the core may be put into an idle state individually or a larger unit containing
74
it may be put into an idle state as a whole (if the other cores within the
75
larger unit are in idle states already).
77
Idle CPUs
78
---------
80
Logical CPUs, simply referred to as "CPUs" in what follows, are regarded as
81
*idle* by the Linux kernel when there are no tasks to run on them except for the
82
special "idle" task.
84
Tasks are the CPU scheduler's representation of work. Each task consists of a
85
sequence of instructions to execute, or code, data to be manipulated while
86
running that code, and some context information that needs to be loaded into the
87
processor every time the task's code is run by a CPU. The CPU scheduler
88
distributes work by assigning tasks to run to the CPUs present in the system.
90
Tasks can be in various states. In particular, they are *runnable* if there are
91
no specific conditions preventing their code from being run by a CPU as long as
92
there is a CPU available for that (for example, they are not waiting for any
93
events to occur or similar). When a task becomes runnable, the CPU scheduler
94
assigns it to one of the available CPUs to run and if there are no more runnable
95
tasks assigned to it, the CPU will load the given task's context and run its
96
code (from the instruction following the last one executed so far, possibly by
97
another CPU). [If there are multiple runnable tasks assigned to one CPU
98
simultaneously, they will be subject to prioritization and time sharing in order
99
to allow them to make some progress over time.]
101
The special "idle" task becomes runnable if there are no other runnable tasks
102
assigned to the given CPU and the CPU is then regarded as idle. In other words,
103
in Linux idle CPUs run the code of the "idle" task called *the idle loop*. That
104
code may cause the processor to be put into one of its idle states, if they are
105
supported, in order to save energy, but if the processor does not support any
106
idle states, or there is not enough time to spend in an idle state before the
107
next wakeup event, or there are strict latency constraints preventing any of the
108
available idle states from being used, the CPU will simply execute more or less
109
useless instructions in a loop until it is assigned a new task to run.
112
.. _idle-loop:
114
The Idle Loop
115
=============
117
The idle loop code takes two major steps in every iteration of it. First, it
118
calls into a code module referred to as the *governor* that belongs to the CPU
119
idle time management subsystem called ``CPUIdle`` to select an idle state for
120
the CPU to ask the hardware to enter. Second, it invokes another code module
121
from the ``CPUIdle`` subsystem, called the *driver*, to actually ask the
122
processor hardware to enter the idle state selected by the governor.
124
The role of the governor is to find an idle state most suitable for the
125
conditions at hand. For this purpose, idle states that the hardware can be
126
asked to enter by logical CPUs are represented in an abstract way independent of
127
the platform or the processor architecture and organized in a one-dimensional
128
(linear) array. That array has to be prepared and supplied by the ``CPUIdle``
129
driver matching the platform the kernel is running on at the initialization
130
time. This allows ``CPUIdle`` governors to be independent of the underlying
131
hardware and to work with any platforms that the Linux kernel can run on.
133
Each idle state present in that array is characterized by two parameters to be
134
taken into account by the governor, the *target residency* and the (worst-case)
135
*exit latency*. The target residency is the minimum time the hardware must
136
spend in the given state, including the time needed to enter it (which may be
137
substantial), in order to save more energy than it would save by entering one of
138
the shallower idle states instead. [The "depth" of an idle state roughly
139
corresponds to the power drawn by the processor in that state.] The exit
140
latency, in turn, is the maximum time it will take a CPU asking the processor
141
hardware to enter an idle state to start executing the first instruction after a
142
wakeup from that state. Note that in general the exit latency also must cover
143
the time needed to enter the given state in case the wakeup occurs when the
144
hardware is entering it and it must be entered completely to be exited in an
145
ordered manner.
147
There are two types of information that can influence the governor's decisions.
148
First of all, the governor knows the time until the closest timer event. That
149
time is known exactly, because the kernel programs timers and it knows exactly
150
when they will trigger, and it is the maximum time the hardware that the given
151
CPU depends on can spend in an idle state, including the time necessary to enter
152
and exit it. However, the CPU may be woken up by a non-timer event at any time
153
(in particular, before the closest timer triggers) and it generally is not known
154
when that may happen. The governor can only see how much time the CPU actually
155
was idle after it has been woken up (that time will be referred to as the *idle
156
duration* from now on) and it can use that information somehow along with the
157
time until the closest timer to estimate the idle duration in future. How the
158
governor uses that information depends on what algorithm is implemented by it
159
and that is the primary reason for having more than one governor in the
160
``CPUIdle`` subsystem.
162
There are four ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_,
163
``ladder`` and ``haltpoll``. Which of them is used by default depends on the
164
configuration of the kernel and in particular on whether or not the scheduler
165
tick can be `stopped by the idle loop <idle-cpus-and-tick_>`_. Available
166
governors can be read from the :file:`available_governors`, and the governor
167
can be changed at runtime. The name of the ``CPUIdle`` governor currently
168
used by the kernel can be read from the :file:`current_governor_ro` or
169
:file:`current_governor` file under :file:`/sys/devices/system/cpu/cpuidle/`
170
in ``sysfs``.
172
Which ``CPUIdle`` driver is used, on the other hand, usually depends on the
173
platform the kernel is running on, but there are platforms with more than one
174
matching driver. For example, there are two drivers that can work with the
175
majority of Intel platforms, ``intel_idle`` and ``acpi_idle``, one with
176
hardcoded idle states information and the other able to read that information
177
from the system's ACPI tables, respectively. Still, even in those cases, the
178
driver chosen at the system initialization time cannot be replaced later, so the
179
decision on which one of them to use has to be made early (on Intel platforms
180
the ``acpi_idle`` driver will be used if ``intel_idle`` is disabled for some
181
reason or if it does not recognize the processor). The name of the ``CPUIdle``
182
driver currently used by the kernel can be read from the :file:`current_driver`
183
file under :file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``.
186
.. _idle-cpus-and-tick:
188
Idle CPUs and The Scheduler Tick
189
================================
191
The scheduler tick is a timer that triggers periodically in order to implement
192
the time sharing strategy of the CPU scheduler. Of course, if there are
193
multiple runnable tasks assigned to one CPU at the same time, the only way to
194
allow them to make reasonable progress in a given time frame is to make them
195
share the available CPU time. Namely, in rough approximation, each task is
196
given a slice of the CPU time to run its code, subject to the scheduling class,
197
prioritization and so on and when that time slice is used up, the CPU should be
198
switched over to running (the code of) another task. The currently running task
199
may not want to give the CPU away voluntarily, however, and the scheduler tick
200
is there to make the switch happen regardless. That is not the only role of the
201
tick, but it is the primary reason for using it.
203
The scheduler tick is problematic from the CPU idle time management perspective,
204
because it triggers periodically and relatively often (depending on the kernel
205
configuration, the length of the tick period is between 1 ms and 10 ms).
206
Thus, if the tick is allowed to trigger on idle CPUs, it will not make sense
207
for them to ask the hardware to enter idle states with target residencies above
208
the tick period length. Moreover, in that case the idle duration of any CPU
209
will never exceed the tick period length and the energy used for entering and
210
exiting idle states due to the tick wakeups on idle CPUs will be wasted.
212
Fortunately, it is not really necessary to allow the tick to trigger on idle
213
CPUs, because (by definition) they have no tasks to run except for the special
214
"idle" one. In other words, from the CPU scheduler perspective, the only user
215
of the CPU time on them is the idle loop. Since the time of an idle CPU need
216
not be shared between multiple runnable tasks, the primary reason for using the
217
tick goes away if the given CPU is idle. Consequently, it is possible to stop
218
the scheduler tick entirely on idle CPUs in principle, even though that may not
219
always be worth the effort.
221
Whether or not it makes sense to stop the scheduler tick in the idle loop
222
depends on what is expected by the governor. First, if there is another
223
(non-tick) timer due to trigger within the tick range, stopping the tick clearly
224
would be a waste of time, even though the timer hardware may not need to be
225
reprogrammed in that case. Second, if the governor is expecting a non-timer
226
wakeup within the tick range, stopping the tick is not necessary and it may even
227
be harmful. Namely, in that case the governor will select an idle state with
228
the target residency within the time until the expected wakeup, so that state is
229
going to be relatively shallow. The governor really cannot select a deep idle
230
state then, as that would contradict its own expectation of a wakeup in short
231
order. Now, if the wakeup really occurs shortly, stopping the tick would be a
232
waste of time and in this case the timer hardware would need to be reprogrammed,
233
which is expensive. On the other hand, if the tick is stopped and the wakeup
234
does not occur any time soon, the hardware may spend indefinite amount of time
235
in the shallow idle state selected by the governor, which will be a waste of
236
energy. Hence, if the governor is expecting a wakeup of any kind within the
237
tick range, it is better to allow the tick trigger. Otherwise, however, the
238
governor will select a relatively deep idle state, so the tick should be stopped
239
so that it does not wake up the CPU too early.
241
In any case, the governor knows what it is expecting and the decision on whether
242
or not to stop the scheduler tick belongs to it. Still, if the tick has been
243
stopped already (in one of the previous iterations of the loop), it is better
244
to leave it as is and the governor needs to take that into account.
246
The kernel can be configured to disable stopping the scheduler tick in the idle
247
loop altogether. That can be done through the build-time configuration of it
248
(by unsetting the ``CONFIG_NO_HZ_IDLE`` configuration option) or by passing
249
``nohz=off`` to it in the command line. In both cases, as the stopping of the
250
scheduler tick is disabled, the governor's decisions regarding it are simply
251
ignored by the idle loop code and the tick is never stopped.
253
The systems that run kernels configured to allow the scheduler tick to be
254
stopped on idle CPUs are referred to as *tickless* systems and they are
255
generally regarded as more energy-efficient than the systems running kernels in
256
which the tick cannot be stopped. If the given system is tickless, it will use
257
the ``menu`` governor by default and if it is not tickless, the default
258
``CPUIdle`` governor on it will be ``ladder``.
261
.. _menu-gov:
263
The ``menu`` Governor
264
=====================
266
The ``menu`` governor is the default ``CPUIdle`` governor for tickless systems.
267
It is quite complex, but the basic principle of its design is straightforward.
268
Namely, when invoked to select an idle state for a CPU (i.e. an idle state that
269
the CPU will ask the processor hardware to enter), it attempts to predict the
270
idle duration and uses the predicted value for idle state selection.
272
It first uses a simple pattern recognition algorithm to obtain a preliminary
273
idle duration prediction. Namely, it saves the last 8 observed idle duration
274
values and, when predicting the idle duration next time, it computes the average
275
and variance of them. If the variance is small (smaller than 400 square
276
milliseconds) or it is small relative to the average (the average is greater
277
that 6 times the standard deviation), the average is regarded as the "typical
278
interval" value. Otherwise, either the longest or the shortest (depending on
279
which one is farther from the average) of the saved observed idle duration
280
values is discarded and the computation is repeated for the remaining ones.
282
Again, if the variance of them is small (in the above sense), the average is
283
taken as the "typical interval" value and so on, until either the "typical
284
interval" is determined or too many data points are disregarded. In the latter
285
case, if the size of the set of data points still under consideration is
286
sufficiently large, the next idle duration is not likely to be above the largest
287
idle duration value still in that set, so that value is taken as the predicted
288
next idle duration. Finally, if the set of data points still under
289
consideration is too small, no prediction is made.
291
If the preliminary prediction of the next idle duration computed this way is
292
long enough, the governor obtains the time until the closest timer event with
293
the assumption that the scheduler tick will be stopped. That time, referred to
294
as the *sleep length* in what follows, is the upper bound on the time before the
295
next CPU wakeup. It is used to determine the sleep length range, which in turn
296
is needed to get the sleep length correction factor.
298
The ``menu`` governor maintains an array containing several correction factor
299
values that correspond to different sleep length ranges organized so that each
300
range represented in the array is approximately 10 times wider than the previous
301
one.
303
The correction factor for the given sleep length range (determined before
304
selecting the idle state for the CPU) is updated after the CPU has been woken
305
up and the closer the sleep length is to the observed idle duration, the closer
306
to 1 the correction factor becomes (it must fall between 0 and 1 inclusive).
307
The sleep length is multiplied by the correction factor for the range that it
308
falls into to obtain an approximation of the predicted idle duration that is
309
compared to the "typical interval" determined previously and the minimum of
310
the two is taken as the final idle duration prediction.
312
If the "typical interval" value is small, which means that the CPU is likely
313
to be woken up soon enough, the sleep length computation is skipped as it may
314
be costly and the idle duration is simply predicted to equal the "typical
315
interval" value.
317
Now, the governor is ready to walk the list of idle states and choose one of
318
them. For this purpose, it compares the target residency of each state with
319
the predicted idle duration and the exit latency of it with the with the latency
320
limit coming from the power management quality of service, or `PM QoS <cpu-pm-qos_>`_,
321
framework. It selects the state with the target residency closest to the predicted
322
idle duration, but still below it, and exit latency that does not exceed the
323
limit.
325
In the final step the governor may still need to refine the idle state selection
326
if it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_. That
327
happens if the idle duration predicted by it is less than the tick period and
328
the tick has not been stopped already (in a previous iteration of the idle
329
loop). Then, the sleep length used in the previous computations may not reflect
330
the real time until the closest timer event and if it really is greater than
331
that time, the governor may need to select a shallower state with a suitable
332
target residency.
335
.. _teo-gov:
337
The Timer Events Oriented (TEO) Governor
338
========================================
340
The timer events oriented (TEO) governor is an alternative ``CPUIdle`` governor
341
for tickless systems. It follows the same basic strategy as the ``menu`` `one
342
<menu-gov_>`_: it always tries to find the deepest idle state suitable for the
343
given conditions. However, it applies a different approach to that problem.
345
.. kernel-doc:: drivers/cpuidle/governors/teo.c
346
:doc: teo-description
348
.. _idle-states-representation:
350
Representation of Idle States
351
=============================
353
For the CPU idle time management purposes all of the physical idle states
354
supported by the processor have to be represented as a one-dimensional array of
355
|struct cpuidle_state| objects each allowing an individual (logical) CPU to ask
356
the processor hardware to enter an idle state of certain properties. If there
357
is a hierarchy of units in the processor, one |struct cpuidle_state| object can
358
cover a combination of idle states supported by the units at different levels of
359
the hierarchy. In that case, the `target residency and exit latency parameters
360
of it <idle-loop_>`_, must reflect the properties of the idle state at the
361
deepest level (i.e. the idle state of the unit containing all of the other
362
units).
364
For example, take a processor with two cores in a larger unit referred to as
365
a "module" and suppose that asking the hardware to enter a specific idle state
366
(say "X") at the "core" level by one core will trigger the module to try to
367
enter a specific idle state of its own (say "MX") if the other core is in idle
368
state "X" already. In other words, asking for idle state "X" at the "core"
369
level gives the hardware a license to go as deep as to idle state "MX" at the
370
"module" level, but there is no guarantee that this is going to happen (the core
371
asking for idle state "X" may just end up in that state by itself instead).
372
Then, the target residency of the |struct cpuidle_state| object representing
373
idle state "X" must reflect the minimum time to spend in idle state "MX" of
374
the module (including the time needed to enter it), because that is the minimum
375
time the CPU needs to be idle to save any energy in case the hardware enters
376
that state. Analogously, the exit latency parameter of that object must cover
377
the exit time of idle state "MX" of the module (and usually its entry time too),
378
because that is the maximum delay between a wakeup signal and the time the CPU
379
will start to execute the first new instruction (assuming that both cores in the
380
module will always be ready to execute instructions as soon as the module
381
becomes operational as a whole).
383
There are processors without direct coordination between different levels of the
384
hierarchy of units inside them, however. In those cases asking for an idle
385
state at the "core" level does not automatically affect the "module" level, for
386
example, in any way and the ``CPUIdle`` driver is responsible for the entire
387
handling of the hierarchy. Then, the definition of the idle state objects is
388
entirely up to the driver, but still the physical properties of the idle state
389
that the processor hardware finally goes into must always follow the parameters
390
used by the governor for idle state selection (for instance, the actual exit
391
latency of that idle state must not exceed the exit latency parameter of the
392
idle state object selected by the governor).
394
In addition to the target residency and exit latency idle state parameters
395
discussed above, the objects representing idle states each contain a few other
396
parameters describing the idle state and a pointer to the function to run in
397
order to ask the hardware to enter that state. Also, for each
398
|struct cpuidle_state| object, there is a corresponding
399
:c:type:`struct cpuidle_state_usage <cpuidle_state_usage>` one containing usage
400
statistics of the given idle state. That information is exposed by the kernel
401
via ``sysfs``.
403
For each CPU in the system, there is a :file:`/sys/devices/system/cpu/cpu<N>/cpuidle/`
404
directory in ``sysfs``, where the number ``<N>`` is assigned to the given
405
CPU at the initialization time. That directory contains a set of subdirectories
406
called :file:`state0`, :file:`state1` and so on, up to the number of idle state
407
objects defined for the given CPU minus one. Each of these directories
408
corresponds to one idle state object and the larger the number in its name, the
409
deeper the (effective) idle state represented by it. Each of them contains
410
a number of files (attributes) representing the properties of the idle state
411
object corresponding to it, as follows:
413
``above``
414
Total number of times this idle state had been asked for, but the
415
observed idle duration was certainly too short to match its target
416
residency.
418
``below``
419
Total number of times this idle state had been asked for, but certainly
420
a deeper idle state would have been a better match for the observed idle
421
duration.
423
``desc``
424
Description of the idle state.
426
``disable``
427
Whether or not this idle state is disabled.
429
``default_status``
430
The default status of this state, "enabled" or "disabled".
432
``latency``
433
Exit latency of the idle state in microseconds.
435
``name``
436
Name of the idle state.
438
``power``
439
Power drawn by hardware in this idle state in milliwatts (if specified,
440
0 otherwise).
442
``residency``
443
Target residency of the idle state in microseconds.
445
``time``
446
Total time spent in this idle state by the given CPU (as measured by the
447
kernel) in microseconds.
449
``usage``
450
Total number of times the hardware has been asked by the given CPU to
451
enter this idle state.
453
``rejected``
454
Total number of times a request to enter this idle state on the given
455
CPU was rejected.
457
The :file:`desc` and :file:`name` files both contain strings. The difference
458
between them is that the name is expected to be more concise, while the
459
description may be longer and it may contain white space or special characters.
460
The other files listed above contain integer numbers.
462
The :file:`disable` attribute is the only writeable one. If it contains 1, the
463
given idle state is disabled for this particular CPU, which means that the
464
governor will never select it for this particular CPU and the ``CPUIdle``
465
driver will never ask the hardware to enter it for that CPU as a result.
466
However, disabling an idle state for one CPU does not prevent it from being
467
asked for by the other CPUs, so it must be disabled for all of them in order to
468
never be asked for by any of them. [Note that, due to the way the ``ladder``
469
governor is implemented, disabling an idle state prevents that governor from
470
selecting any idle states deeper than the disabled one too.]
472
If the :file:`disable` attribute contains 0, the given idle state is enabled for
473
this particular CPU, but it still may be disabled for some or all of the other
474
CPUs in the system at the same time. Writing 1 to it causes the idle state to
475
be disabled for this particular CPU and writing 0 to it allows the governor to
476
take it into consideration for the given CPU and the driver to ask for it,
477
unless that state was disabled globally in the driver (in which case it cannot
478
be used at all).
480
The :file:`power` attribute is not defined very well, especially for idle state
481
objects representing combinations of idle states at different levels of the
482
hierarchy of units in the processor, and it generally is hard to obtain idle
483
state power numbers for complex hardware, so :file:`power` often contains 0 (not
484
available) and if it contains a nonzero number, that number may not be very
485
accurate and it should not be relied on for anything meaningful.
487
The number in the :file:`time` file generally may be greater than the total time
488
really spent by the given CPU in the given idle state, because it is measured by
489
the kernel and it may not cover the cases in which the hardware refused to enter
490
this idle state and entered a shallower one instead of it (or even it did not
491
enter any idle state at all). The kernel can only measure the time span between
492
asking the hardware to enter an idle state and the subsequent wakeup of the CPU
493
and it cannot say what really happened in the meantime at the hardware level.
494
Moreover, if the idle state object in question represents a combination of idle
495
states at different levels of the hierarchy of units in the processor,
496
the kernel can never say how deep the hardware went down the hierarchy in any
497
particular case. For these reasons, the only reliable way to find out how
498
much time has been spent by the hardware in different idle states supported by
499
it is to use idle state residency counters in the hardware, if available.
501
Generally, an interrupt received when trying to enter an idle state causes the
502
idle state entry request to be rejected, in which case the ``CPUIdle`` driver
503
may return an error code to indicate that this was the case. The :file:`usage`
504
and :file:`rejected` files report the number of times the given idle state
505
was entered successfully or rejected, respectively.
507
.. _cpu-pm-qos:
509
Power Management Quality of Service for CPUs
510
============================================
512
The power management quality of service (PM QoS) framework in the Linux kernel
513
allows kernel code and user space processes to set constraints on various
514
energy-efficiency features of the kernel to prevent performance from dropping
515
below a required level.
517
CPU idle time management can be affected by PM QoS in two ways, through the
518
global CPU latency limit and through the resume latency constraints for
519
individual CPUs. Kernel code (e.g. device drivers) can set both of them with
520
the help of special internal interfaces provided by the PM QoS framework. User
521
space can modify the former by opening the :file:`cpu_dma_latency` special
522
device file under :file:`/dev/` and writing a binary value (interpreted as a
523
signed 32-bit integer) to it. In turn, the resume latency constraint for a CPU
524
can be modified from user space by writing a string (representing a signed
525
32-bit integer) to the :file:`power/pm_qos_resume_latency_us` file under
526
:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs``, where the CPU number
527
``<N>`` is allocated at the system initialization time. Negative values
528
will be rejected in both cases and, also in both cases, the written integer
529
number will be interpreted as a requested PM QoS constraint in microseconds.
531
The requested value is not automatically applied as a new constraint, however,
532
as it may be less restrictive (greater in this particular case) than another
533
constraint previously requested by someone else. For this reason, the PM QoS
534
framework maintains a list of requests that have been made so far for the
535
global CPU latency limit and for each individual CPU, aggregates them and
536
applies the effective (minimum in this particular case) value as the new
537
constraint.
539
In fact, opening the :file:`cpu_dma_latency` special device file causes a new
540
PM QoS request to be created and added to a global priority list of CPU latency
541
limit requests and the file descriptor coming from the "open" operation
542
represents that request. If that file descriptor is then used for writing, the
543
number written to it will be associated with the PM QoS request represented by
544
it as a new requested limit value. Next, the priority list mechanism will be
545
used to determine the new effective value of the entire list of requests and
546
that effective value will be set as a new CPU latency limit. Thus requesting a
547
new limit value will only change the real limit if the effective "list" value is
548
affected by it, which is the case if it is the minimum of the requested values
549
in the list.
551
The process holding a file descriptor obtained by opening the
552
:file:`cpu_dma_latency` special device file controls the PM QoS request
553
associated with that file descriptor, but it controls this particular PM QoS
554
request only.
556
Closing the :file:`cpu_dma_latency` special device file or, more precisely, the
557
file descriptor obtained while opening it, causes the PM QoS request associated
558
with that file descriptor to be removed from the global priority list of CPU
559
latency limit requests and destroyed. If that happens, the priority list
560
mechanism will be used again, to determine the new effective value for the whole
561
list and that value will become the new limit.
563
In turn, for each CPU there is one resume latency PM QoS request associated with
564
the :file:`power/pm_qos_resume_latency_us` file under
565
:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes
566
this single PM QoS request to be updated regardless of which user space
567
process does that. In other words, this PM QoS request is shared by the entire
568
user space, so access to the file associated with it needs to be arbitrated
569
to avoid confusion. [Arguably, the only legitimate use of this mechanism in
570
practice is to pin a process to the CPU in question and let it use the
571
``sysfs`` interface to control the resume latency constraint for it.] It is
572
still only a request, however. It is an entry in a priority list used to
573
determine the effective value to be set as the resume latency constraint for the
574
CPU in question every time the list of requests is updated this way or another
575
(there may be other requests coming from kernel code in that list).
577
CPU idle time governors are expected to regard the minimum of the global
578
(effective) CPU latency limit and the effective resume latency constraint for
579
the given CPU as the upper limit for the exit latency of the idle states that
580
they are allowed to select for that CPU. They should never select any idle
581
states with exit latency beyond that limit.
584
Idle States Control Via Kernel Command Line
585
===========================================
587
In addition to the ``sysfs`` interface allowing individual idle states to be
588
`disabled for individual CPUs <idle-states-representation_>`_, there are kernel
589
command line parameters affecting CPU idle time management.
591
The ``cpuidle.off=1`` kernel command line option can be used to disable the
592
CPU idle time management entirely. It does not prevent the idle loop from
593
running on idle CPUs, but it prevents the CPU idle time governors and drivers
594
from being invoked. If it is added to the kernel command line, the idle loop
595
will ask the hardware to enter idle states on idle CPUs via the CPU architecture
596
support code that is expected to provide a default mechanism for this purpose.
597
That default mechanism usually is the least common denominator for all of the
598
processors implementing the architecture (i.e. CPU instruction set) in question,
599
however, so it is rather crude and not very energy-efficient. For this reason,
600
it is not recommended for production use.
602
The ``cpuidle.governor=`` kernel command line switch allows the ``CPUIdle``
603
governor to use to be specified. It has to be appended with a string matching
604
the name of an available governor (e.g. ``cpuidle.governor=menu``) and that
605
governor will be used instead of the default one. It is possible to force
606
the ``menu`` governor to be used on the systems that use the ``ladder`` governor
607
by default this way, for example.
609
The other kernel command line parameters controlling CPU idle time management
610
described below are only relevant for the *x86* architecture and references
611
to ``intel_idle`` affect Intel processors only.
613
The *x86* architecture support code recognizes three kernel command line
614
options related to CPU idle time management: ``idle=poll``, ``idle=halt``,
615
and ``idle=nomwait``. The first two of them disable the ``acpi_idle`` and
616
``intel_idle`` drivers altogether, which effectively causes the entire
617
``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the
618
architecture support code to deal with idle CPUs. How it does that depends on
619
which of the two parameters is added to the kernel command line. In the
620
``idle=halt`` case, the architecture support code will use the ``HLT``
621
instruction of the CPUs (which, as a rule, suspends the execution of the program
622
and causes the hardware to attempt to enter the shallowest available idle state)
623
for this purpose, and if ``idle=poll`` is used, idle CPUs will execute a
624
more or less "lightweight" sequence of instructions in a tight loop. [Note
625
that using ``idle=poll`` is somewhat drastic in many cases, as preventing idle
626
CPUs from saving almost any energy at all may not be the only effect of it.
627
For example, on Intel hardware it effectively prevents CPUs from using
628
P-states (see |cpufreq|) that require any number of CPUs in a package to be
629
idle, so it very well may hurt single-thread computations performance as well as
630
energy-efficiency. Thus using it for performance reasons may not be a good idea
631
at all.]
633
The ``idle=nomwait`` option prevents the use of ``MWAIT`` instruction of
634
the CPU to enter idle states. When this option is used, the ``acpi_idle``
635
driver will use the ``HLT`` instruction instead of ``MWAIT``. On systems
636
running Intel processors, this option disables the ``intel_idle`` driver
637
and forces the use of the ``acpi_idle`` driver instead. Note that in either
638
case, ``acpi_idle`` driver will function only if all the information needed
639
by it is in the system's ACPI tables.
641
In addition to the architecture-level kernel command line options affecting CPU
642
idle time management, there are parameters affecting individual ``CPUIdle``
643
drivers that can be passed to them via the kernel command line. Specifically,
644
the ``intel_idle.max_cstate=<n>`` and ``processor.max_cstate=<n>`` parameters,
645
where ``<n>`` is an idle state index also used in the name of the given
646
state's directory in ``sysfs`` (see
647
`Representation of Idle States <idle-states-representation_>`_), causes the
648
``intel_idle`` and ``acpi_idle`` drivers, respectively, to discard all of the
649
idle states deeper than idle state ``<n>``. In that case, they will never ask
650
for any of those idle states or expose them to the governor. [The behavior of
651
the two drivers is different for ``<n>`` equal to ``0``. Adding
652
``intel_idle.max_cstate=0`` to the kernel command line disables the
653
``intel_idle`` driver and allows ``acpi_idle`` to be used, whereas
654
``processor.max_cstate=0`` is equivalent to ``processor.max_cstate=1``.
655
Also, the ``acpi_idle`` driver is part of the ``processor`` kernel module that
656
can be loaded separately and ``max_cstate=<n>`` can be passed to it as a module
657
parameter when it is loaded.]
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 정보
1-15이 문서는 `SPDX-License-Identifier: GPL-2.0`을 따르고 2018 Intel Corporation 저작물이며, 작성자는 Rafael J. Wysocki `<rafael.j.wysocki@intel.com>`입니다.
`struct cpuidle_state`와 CPU Performance Scaling 문서에 대한 교차 참조를 사용합니다.
Idle state와 logical CPU
16-76현대 processor는 program 실행을 멈추고 instruction fetch와 execution을 중단하는 idle state에 들어갈 수 있습니다. 사용하지 않는 hardware 부분의 power draw를 줄여 energy를 절약하며 CPU idle time management는 이 기능을 활용합니다.
CPUIdle은 scheduler가 보는 logical CPU 단위로 동작합니다. Logical CPU는 물리 부품과 일치하지 않을 수 있고 software에는 instruction sequence 하나를 실행하는 single-core processor처럼 보이는 interface입니다.
| 형태 | Idle 제어 단위 |
|---|---|
| Single-core | Processor 전체가 logical CPU 하나 |
| Multicore | Core마다 CPU 하나; package/cluster idle로 확장 가능 |
| Hardware threads | Core 안 thread마다 CPU 하나; 모두 idle이어야 core가 더 깊게 진입 가능 |
Single-core processor는 전체가 한 CPU입니다. Multicore에서는 각 core가 CPU이며 한 core의 요청이 조건에 따라 package나 cluster 전체의 idle을 trigger할 수 있습니다. 더 큰 unit의 다른 core가 이미 core-level idle이어야 마지막 core 요청이 상위 unit을 idle로 만들 수 있습니다.
한 core가 여러 program을 같은 시간대에 실행하면 hardware thread, Intel에서는 hyper-thread가 각각 logical CPU입니다. Thread 하나의 idle 요청은 그 thread만 멈추며, 같은 core의 모든 thread가 idle을 요청해야 core 또는 상위 unit이 더 깊은 idle state로 갈 수 있습니다.
Linux에서 idle CPU
77-111Linux는 special idle task 외에 실행할 task가 없는 logical CPU를 idle로 봅니다. Task는 실행할 instruction, 조작할 data와 실행 때 processor에 load할 context를 묶은 scheduler의 work 표현입니다.
Runnable task는 event 대기 같은 실행 방해 조건이 없고 CPU만 있으면 실행할 수 있습니다. Scheduler가 CPU에 할당하며 여러 task가 동시에 runnable이면 priority와 time sharing으로 진행시킵니다.
다른 runnable task가 없으면 idle task가 runnable해지고 CPU는 idle loop를 실행합니다. 가능한 idle state와 충분한 시간, 허용 latency가 있으면 hardware를 idle로 넣고, 그렇지 않으면 새 task가 올 때까지 사실상 쓸모없는 instruction loop를 실행합니다.
Idle loop, governor와 driver
112-185Idle loop의 각 iteration은 두 단계입니다. CPUIdle governor가 들어갈 state를 선택하고 CPUIdle driver가 processor hardware에 실제 진입을 요청합니다.
Platform driver는 logical CPU가 요청할 수 있는 state를 platform-independent 1차원 배열로 제공합니다. 덕분에 governor는 architecture와 hardware에 독립적입니다.
| Parameter | 의미 |
|---|---|
| Target residency | 더 얕은 상태보다 energy를 절약하려면 진입 시간을 포함해 머물러야 할 최소 시간 |
| Exit latency | Wakeup부터 첫 instruction 실행까지의 최악 시간; 필요한 경우 entry 시간도 포함 |
Governor는 가장 가까운 timer event까지 정확한 시간을 압니다. 이는 entry/exit를 포함해 hardware가 idle일 수 있는 최대 시간입니다. 하지만 non-timer wakeup은 언제든 올 수 있어 실제 idle duration은 wakeup 뒤에만 알 수 있습니다. Governor마다 과거 duration과 timer를 결합하는 algorithm이 달라 여러 governor가 존재합니다.
| Governor | 역할 |
|---|---|
| `menu` | Tickless 기본; 과거 duration과 timer로 다음 idle duration 예측 |
| `TEO` | Tickless 대안; timer event 중심으로 가장 깊은 적합 상태 선택 |
| `ladder` | Tick을 멈출 수 없는 system의 기본 |
| `haltpoll` | Polling 기반 governor |
사용 가능 governor는 `/sys/devices/system/cpu/cpuidle/available_governors`, 현재 값은 `current_governor_ro` 또는 `current_governor`에서 확인하고 runtime에 바꿀 수 있습니다.
Driver는 platform에 따라 정해지고 초기화 후 교체할 수 없습니다. Intel platform은 hardcoded state의 `intel_idle`과 ACPI table을 읽는 `acpi_idle`이 있으며, `intel_idle`이 꺼졌거나 processor를 인식하지 못하면 `acpi_idle`을 씁니다. 현재 driver는 `current_driver`에서 확인합니다.
Idle CPU와 scheduler tick
186-260Scheduler tick은 runnable task가 CPU time slice를 공유하도록 주기적으로 강제 전환하는 timer입니다. Idle CPU에는 idle task만 있어 이 주된 목적이 사라집니다.
Tick 주기는 kernel config에 따라 1-10 ms여서 idle CPU를 자주 깨웁니다. Tick보다 target residency가 긴 state는 의미가 없고 entry/exit energy도 낭비됩니다.
다만 tick range 안에 다른 timer나 예상 non-timer wakeup이 있으면 tick을 멈추는 비용이 낭비일 수 있습니다. Governor가 얕은 state를 골랐는데 wakeup이 늦으면 그 state에 너무 오래 머물러 energy를 낭비합니다. 반대로 가까운 wakeup을 예상하지 않으면 깊은 state를 선택하고 tick을 멈춰야 합니다.
따라서 tick 정지 결정은 governor가 하며 이미 멈춰 있으면 그대로 두는 편이 낫습니다. `CONFIG_NO_HZ_IDLE`을 끄거나 `nohz=off`를 주면 idle-loop tick 정지를 완전히 금지해 governor 결정을 무시합니다.
Idle CPU에서 tick을 멈출 수 있는 system을 tickless라 하며 일반적으로 더 energy-efficient합니다. Tickless 기본 governor는 `menu`, 그렇지 않은 system은 `ladder`입니다.
TEO governor
335-350Timer Events Oriented(TEO)는 tickless system용 대안 governor입니다. `menu`와 마찬가지로 조건에 맞는 가장 깊은 idle state를 찾지만 다른 방식으로 문제를 풉니다.
상세 algorithm은 `drivers/cpuidle/governors/teo.c`의 kernel-doc `teo-description`에서 가져옵니다.
Idle state 표현과 sysfs
351-506Physical idle state는 logical CPU가 특정 성질의 state 진입을 요청할 수 있는 `struct cpuidle_state` 1차원 배열로 표현합니다. Hierarchy가 있으면 한 object가 여러 level의 state 조합을 나타낼 수 있고 target residency와 exit latency는 가장 깊은 level의 성질을 반영해야 합니다.
예를 들어 두 core module에서 한 core의 X 요청이 다른 core도 X일 때 module MX를 허용한다면, X object의 target residency는 MX에서 energy를 절약할 최소 시간이고 exit latency는 MX에서 CPU가 instruction을 재개할 최악 시간을 포함해야 합니다.
Level 사이 hardware coordination이 없으면 driver가 hierarchy 전체를 처리하고 object 정의도 driver 책임입니다. 어떤 경우든 실제 hardware state의 exit latency 같은 물리 특성은 governor가 선택에 사용한 parameter를 넘으면 안 됩니다.
각 object에는 parameter와 entry 함수 pointer가 있고 대응하는 `struct cpuidle_state_usage`가 통계를 담습니다. CPU마다 `/sys/devices/system/cpu/cpu<N>/cpuidle/stateX/`가 있으며 숫자가 클수록 effective state가 깊습니다.
| Attribute | 의미 |
|---|---|
| `above` | 선택했지만 실제 idle duration이 target residency보다 확실히 짧았던 횟수 |
| `below` | 더 깊은 상태가 실제 duration에 더 적합했을 횟수 |
| `desc` | Idle state 설명 |
| `disable` | CPU별 비활성 상태; 유일한 writeable attribute |
| `default_status` | 기본 `enabled` 또는 `disabled` |
| `latency` | Exit latency(us) |
| `name` | 간결한 idle state 이름 |
| `power` | Power(mW), 미지정이면 0 |
| `residency` | Target residency(us) |
| `time` | Kernel이 측정한 누적 idle 시간(us) |
| `usage` / `rejected` | 성공 요청 횟수 / 거부 횟수 |
`desc`는 길고 whitespace/special character를 포함할 수 있고 `name`은 간결합니다. `disable`만 writeable하며 CPU별로 `1`이면 governor/driver가 그 state를 쓰지 않습니다. 모든 CPU에서 막으려면 모두 비활성화해야 하고 `ladder`는 한 state를 끄면 더 깊은 state도 선택하지 못합니다.
`0`을 쓰면 해당 CPU에서 고려할 수 있지만 driver가 global disable한 state는 사용할 수 없습니다. `power`는 복잡한 hierarchy에서 정의와 측정이 어려워 흔히 0이고 nonzero여도 정확하지 않을 수 있습니다.
`time`은 요청부터 wakeup까지 kernel이 잰 값이라 hardware가 더 얕은 state로 갔거나 거부한 경우도 구분하지 못해 실제 residency보다 클 수 있습니다. 신뢰할 수 있는 측정은 hardware residency counter를 써야 합니다.
진입 중 interrupt가 오면 요청이 거부될 수 있고 driver가 error를 반환합니다. `usage`와 `rejected`가 각각 성공과 거부 횟수를 보고합니다.
CPU PM QoS latency 제약
507-583PM QoS framework는 kernel과 userspace가 energy-efficiency 기능에 constraint를 걸어 필요한 성능 아래로 떨어지지 않게 합니다. CPUIdle에는 global CPU latency limit와 CPU별 resume latency가 영향을 줍니다.
| 범위 | Userspace interface |
|---|---|
| Global | `/dev/cpu_dma_latency`; file descriptor별 request |
| CPU별 | `cpu<N>/power/pm_qos_resume_latency_us`; userspace가 공유하는 request 하나 |
`/dev/cpu_dma_latency`에는 signed 32-bit binary 값을, `/sys/devices/system/cpu/cpu<N>/power/pm_qos_resume_latency_us`에는 signed 32-bit 문자열을 씁니다. 음수는 거부되고 단위는 microsecond입니다.
여러 request 중 더 제한적인 최솟값이 effective constraint입니다. `cpu_dma_latency`를 open하면 global priority list에 request가 생기고 file descriptor가 이를 나타냅니다. Write가 값을 갱신하고 close가 request를 제거한 뒤 list 최솟값을 다시 계산합니다.
CPU별 sysfs file은 userspace 전체가 공유하는 request 하나를 갱신하므로 접근 중재가 필요합니다. 실용적인 사용법은 process를 해당 CPU에 pin하고 그 CPU의 resume latency를 제어하는 것입니다. Kernel request도 같은 priority list에 들어갈 수 있습니다.
Governor는 global effective limit와 CPU별 effective resume latency 중 작은 값을 허용 exit latency 상한으로 사용하며 이를 넘는 state를 선택하면 안 됩니다.
Kernel command line 제어
584-657CPU별 sysfs `disable` 외에도 boot parameter로 CPU idle time management를 제어할 수 있습니다.
| Parameter | 효과 |
|---|---|
| `cpuidle.off=1` | CPUIdle governor/driver를 끄고 architecture 기본 idle 사용 |
| `cpuidle.governor=menu` | 사용할 governor 지정 |
| `nohz=off` | Idle loop의 scheduler tick 정지 금지 |
| `idle=halt` | X86에서 CPUIdle을 끄고 `HLT` 사용 |
| `idle=poll` | X86에서 tight polling; energy와 single-thread 성능에 악영향 가능 |
| `idle=nomwait` | `MWAIT` 금지, `acpi_idle`의 `HLT` 사용 |
| `intel_idle.max_cstate=<n>` | Intel idle state `<n>`보다 깊은 상태 제거 |
| `processor.max_cstate=<n>` | ACPI idle state `<n>`보다 깊은 상태 제거 |
`cpuidle.off=1`은 idle loop 자체는 남기지만 CPUIdle governor와 driver 호출을 막습니다. Architecture 공통 기본 mechanism을 사용하므로 거칠고 energy-efficient하지 않아 production에는 권장하지 않습니다.
`cpuidle.governor=`에는 사용 가능한 governor 이름을 줍니다. 이후 `idle=` option은 X86 전용이며 `intel_idle` 언급은 Intel processor에만 해당합니다.
`idle=halt`와 `idle=poll`은 `acpi_idle`과 `intel_idle`을 모두 끕니다. Polling은 거의 모든 idle energy 절약을 막고 package 내 idle CPU가 필요한 P-state도 막아 single-thread 성능까지 해칠 수 있습니다.
`idle=nomwait`는 `MWAIT` 대신 `HLT`를 쓰게 하며 Intel에서는 `intel_idle`을 끄고 `acpi_idle`을 강제합니다. ACPI table에 필요한 정보가 모두 있어야 동작합니다.
`intel_idle.max_cstate=<n>`과 `processor.max_cstate=<n>`은 각각 더 깊은 state를 driver에서 버립니다. `intel_idle.max_cstate=0`은 intel_idle을 끄고 acpi_idle을 허용하지만 `processor.max_cstate=0`은 `1`과 같습니다. `acpi_idle`은 별도 load 가능한 `processor` module 일부라 `max_cstate=<n>`을 module parameter로도 줄 수 있습니다.
운영 요약
cpuidle.rst:1-657CPUIdle governor는 예상 idle duration과 PM QoS latency 안에서 state를 고르고 driver가 hardware 진입을 수행합니다. Tick 정지, state disable, latency request와 boot parameter가 실제 선택을 함께 제한합니다.