요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
Weights and statistics
blkio-controller.rst:94-249BFQ weight override, latency·queue·debug·recursive statistics를 정리합니다.
Throttle and reset
blkio-controller.rst:250-301BPS·IOPS read/write limit, throttling 통계와 reset interface를 다룹니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
===================
Block IO Controller
===================
Overview
========
cgroup subsys "blkio" implements the block io controller. There seems to be
a need of various kinds of IO control policies (like proportional BW, max BW)
both at leaf nodes as well as at intermediate nodes in a storage hierarchy.
Plan is to use the same cgroup based management interface for blkio controller
and based on user options switch IO policies in the background.
One IO control policy is throttling policy which can be used to
specify upper IO rate limits on devices. This policy is implemented in
generic block layer and can be used on leaf nodes as well as higher
level logical devices like device mapper.
HOWTO
=====
Throttling/Upper Limit policy
-----------------------------
Enable Block IO controller::
CONFIG_BLK_CGROUP=y
Enable throttling in block layer::
CONFIG_BLK_DEV_THROTTLING=y
Mount blkio controller (see cgroups.txt, Why are cgroups needed?)::
mount -t cgroup -o blkio none /sys/fs/cgroup/blkio
Specify a bandwidth rate on particular device for root group. The format
for policy is "<major>:<minor> <bytes_per_second>"::
echo "8:16 1048576" > /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device
This will put a limit of 1MB/second on reads happening for root group
on device having major/minor number 8:16.
Run dd to read a file and see if rate is throttled to 1MB/s or not::
# dd iflag=direct if=/mnt/common/zerofile of=/dev/null bs=4K count=1024
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/s
Limits for writes can be put using blkio.throttle.write_bps_device file.
Hierarchical Cgroups
====================
Throttling implements hierarchy support; however,
throttling's hierarchy support is enabled iff "sane_behavior" is
enabled from cgroup side, which currently is a development option and
not publicly available.
If somebody created a hierarchy like as follows::
root
/ \
test1 test2
|
test3
Throttling with "sane_behavior" will handle the
hierarchy correctly. For throttling, all limits apply
to the whole subtree while all statistics are local to the IOs
directly generated by tasks in that cgroup.
Throttling without "sane_behavior" enabled from cgroup side will
practically treat all groups at same level as if it looks like the
following::
pivot
/ / \ \
root test1 test2 test3
Various user visible config options
===================================
CONFIG_BLK_CGROUP
Block IO controller.
CONFIG_BFQ_CGROUP_DEBUG
Debug help. Right now some additional stats file show up in cgroup
if this option is enabled.
CONFIG_BLK_DEV_THROTTLING
Enable block device throttling support in block layer.
Details of cgroup files
=======================
Proportional weight policy files
--------------------------------
blkio.bfq.weight
Specifies per cgroup weight. This is default weight of the group
on all the devices until and unless overridden by per device rule
(see `blkio.bfq.weight_device` below).
Currently allowed range of weights is from 1 to 1000. For more details,
see Documentation/block/bfq-iosched.rst.
blkio.bfq.weight_device
Specifies per cgroup per device weights, overriding the default group
weight. For more details, see Documentation/block/bfq-iosched.rst.
Following is the format::
# echo dev_maj:dev_minor weight > blkio.bfq.weight_device
Configure weight=300 on /dev/sdb (8:16) in this cgroup::
# echo 8:16 300 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:16 300
Configure weight=500 on /dev/sda (8:0) in this cgroup::
# echo 8:0 500 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:0 500
8:16 300
Remove specific weight for /dev/sda in this cgroup::
# echo 8:0 0 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:16 300
blkio.time
Disk time allocated to cgroup per device in milliseconds. First
two fields specify the major and minor number of the device and
third field specifies the disk time allocated to group in
milliseconds.
blkio.sectors
Number of sectors transferred to/from disk by the group. First
two fields specify the major and minor number of the device and
third field specifies the number of sectors transferred by the
group to/from the device.
blkio.io_service_bytes
Number of bytes transferred to/from the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of bytes.
blkio.io_serviced
Number of IOs (bio) issued to the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of IOs.
blkio.io_service_time
Total amount of time between request dispatch and request completion
for the IOs done by this cgroup. This is in nanoseconds to make it
meaningful for flash devices too. For devices with queue depth of 1,
this time represents the actual service time. When queue_depth > 1,
that is no longer true as requests may be served out of order. This
may cause the service time for a given IO to include the service time
of multiple IOs when served out of order which may result in total
io_service_time > actual time elapsed. This time is further divided by
the type of operation - read or write, sync or async. First two fields
specify the major and minor number of the device, third field
specifies the operation type and the fourth field specifies the
io_service_time in ns.
blkio.io_wait_time
Total amount of time the IOs for this cgroup spent waiting in the
scheduler queues for service. This can be greater than the total time
elapsed since it is cumulative io_wait_time for all IOs. It is not a
measure of total time the cgroup spent waiting but rather a measure of
the wait_time for its individual IOs. For devices with queue_depth > 1
this metric does not include the time spent waiting for service once
the IO is dispatched to the device but till it actually gets serviced
(there might be a time lag here due to re-ordering of requests by the
device). This is in nanoseconds to make it meaningful for flash
devices too. This time is further divided by the type of operation -
read or write, sync or async. First two fields specify the major and
minor number of the device, third field specifies the operation type
and the fourth field specifies the io_wait_time in ns.
blkio.io_merged
Total number of bios/requests merged into requests belonging to this
cgroup. This is further divided by the type of operation - read or
write, sync or async.
blkio.io_queued
Total number of requests queued up at any given instant for this
cgroup. This is further divided by the type of operation - read or
write, sync or async.
blkio.avg_queue_size
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
The average queue size for this cgroup over the entire time of this
cgroup's existence. Queue size samples are taken each time one of the
queues of this cgroup gets a timeslice.
blkio.group_wait_time
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
This is the amount of time the cgroup had to wait since it became busy
(i.e., went from 0 to 1 request queued) to get a timeslice for one of
its queues. This is different from the io_wait_time which is the
cumulative total of the amount of time spent by each IO in that cgroup
waiting in the scheduler queue. This is in nanoseconds. If this is
read when the cgroup is in a waiting (for timeslice) state, the stat
will only report the group_wait_time accumulated till the last time it
got a timeslice and will not include the current delta.
blkio.empty_time
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
This is the amount of time a cgroup spends without any pending
requests when not being served, i.e., it does not include any time
spent idling for one of the queues of the cgroup. This is in
nanoseconds. If this is read when the cgroup is in an empty state,
the stat will only report the empty_time accumulated till the last
time it had a pending request and will not include the current delta.
blkio.idle_time
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
This is the amount of time spent by the IO scheduler idling for a
given cgroup in anticipation of a better request than the existing ones
from other queues/cgroups. This is in nanoseconds. If this is read
when the cgroup is in an idling state, the stat will only report the
idle_time accumulated till the last idle period and will not include
the current delta.
blkio.dequeue
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This
gives the statistics about how many a times a group was dequeued
from service tree of the device. First two fields specify the major
and minor number of the device and third field specifies the number
of times a group was dequeued from a particular device.
blkio.*_recursive
Recursive version of various stats. These files show the
same information as their non-recursive counterparts but
include stats from all the descendant cgroups.
Throttling/Upper limit policy files
-----------------------------------
blkio.throttle.read_bps_device
Specifies upper limit on READ rate from the device. IO rate is
specified in bytes per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.read_bps_device
blkio.throttle.write_bps_device
Specifies upper limit on WRITE rate to the device. IO rate is
specified in bytes per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.write_bps_device
blkio.throttle.read_iops_device
Specifies upper limit on READ rate from the device. IO rate is
specified in IO per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.read_iops_device
blkio.throttle.write_iops_device
Specifies upper limit on WRITE rate to the device. IO rate is
specified in io per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.write_iops_device
Note: If both BW and IOPS rules are specified for a device, then IO is
subjected to both the constraints.
blkio.throttle.io_serviced
Number of IOs (bio) issued to the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of IOs.
blkio.throttle.io_service_bytes
Number of bytes transferred to/from the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of bytes.
Common files among various policies
-----------------------------------
blkio.reset_stats
Writing an int to this file will result in resetting all the stats
for that cgroup.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
개요와 upper-limit HOWTO
1-51Cgroup subsystem `blkio`는 block I/O controller를 구현합니다. Storage hierarchy의 leaf node와 intermediate node 모두에서 proportional bandwidth, maximum bandwidth 등 여러 I/O control policy가 필요합니다. 계획은 동일한 cgroup management interface를 사용하고 user option에 따라 내부 I/O policy를 전환하는 것입니다.
Throttling policy는 device별 upper I/O rate limit을 지정합니다. Generic block layer에서 구현되므로 leaf node뿐 아니라 device mapper 같은 higher-level logical device에도 사용할 수 있습니다.
Controller가 같은 cgroup interface 뒤에서 제공하는 정책 범위입니다.
Upper-limit policy를 사용하려면 `CONFIG_BLK_CGROUP=y`로 Block I/O controller를, `CONFIG_BLK_DEV_THROTTLING=y`로 block-layer throttling을 enable하고 blkio controller를 mount합니다.
Throttling/Upper Limit policy
-----------------------------
Enable Block IO controller::
CONFIG_BLK_CGROUP=y
Enable throttling in block layer::
CONFIG_BLK_DEV_THROTTLING=y
Mount blkio controller (see cgroups.txt, Why are cgroups needed?)::
mount -t cgroup -o blkio none /sys/fs/cgroup/blkioRoot group의 특정 device에 read bandwidth를 지정하는 형식은 `<major>:<minor> <bytes_per_second>`입니다. 예제는 device 8:16에 초당 1 MiB read limit을 설정합니다.
Specify a bandwidth rate on particular device for root group. The format
for policy is "<major>:<minor> <bytes_per_second>"::
echo "8:16 1048576" > /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device
This will put a limit of 1MB/second on reads happening for root group
on device having major/minor number 8:16.Direct I/O `dd` 예제는 4 MiB를 약 4초에 읽어 실제 1.0 MB/s로 throttle되는지 확인합니다.
Run dd to read a file and see if rate is throttled to 1MB/s or not::
# dd iflag=direct if=/mnt/common/zerofile of=/dev/null bs=4K count=1024
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/sWrite limit은 `blkio.throttle.write_bps_device` file로 설정합니다.
Kernel option부터 device rule 검증까지의 절차입니다.
Hierarchy 동작과 Kconfig option
52-93Throttling은 hierarchy를 지원하지만 cgroup 쪽 `sane_behavior`가 enable된 경우에만 활성화됩니다. 이는 현재 development option이며 public하게 사용할 수 없습니다.
원문 예제 hierarchy는 root 아래 test1·test2가 있고 test1 아래 test3이 있습니다. `sane_behavior`를 사용하면 모든 limit은 전체 subtree에 적용되고, 각 통계는 해당 cgroup task가 직접 만든 I/O에 대해서만 local하게 집계됩니다.
Parent limit은 subtree 전체에 적용되지만 local statistics는 직접 생성 I/O만 셉니다.
`sane_behavior`가 없으면 throttling은 사실상 모든 group을 같은 level로 취급합니다. 즉 pivot 아래 root, test1, test2, test3가 flat sibling처럼 동작합니다.
원래 parent-child 관계를 잃고 모든 group이 pivot의 sibling처럼 취급됩니다.
Controller·debug·throttling을 enable하는 Kconfig입니다.
BFQ weight와 기본 통계
94-163`blkio.bfq.weight`는 모든 device에 적용할 cgroup default weight를 지정합니다. Device별 rule이 `blkio.bfq.weight_device`에서 override할 때까지 사용합니다. 허용 범위는 1-1000이며 자세한 내용은 `Documentation/block/bfq-iosched.rst`를 참고합니다.
`blkio.bfq.weight_device`는 cgroup·device별 weight를 지정해 default group weight를 덮습니다. 형식은 `dev_maj:dev_minor weight`입니다. 예제는 `/dev/sdb`(8:16)에 300, `/dev/sda`(8:0)에 500을 설정하고, weight 0을 써 `/dev/sda`의 specific rule을 제거합니다.
blkio.bfq.weight_device
Specifies per cgroup per device weights, overriding the default group
weight. For more details, see Documentation/block/bfq-iosched.rst.
Following is the format::
# echo dev_maj:dev_minor weight > blkio.bfq.weight_device
Configure weight=300 on /dev/sdb (8:16) in this cgroup::
# echo 8:16 300 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:16 300
Configure weight=500 on /dev/sda (8:0) in this cgroup::
# echo 8:0 500 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:0 500
8:16 300
Remove specific weight for /dev/sda in this cgroup::
# echo 8:0 0 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:16 300Default와 per-device override의 관계입니다.
`blkio.time`은 device별 cgroup에 할당한 disk time을 millisecond로 표시합니다. 첫 두 field는 major/minor, 세 번째는 allocated disk time입니다. `blkio.sectors`는 group이 device와 주고받은 sector 수를 같은 세-field 구조로 표시합니다.
`blkio.io_service_bytes`는 group이 disk와 주고받은 byte 수를 read/write, sync/async operation type별로 나눕니다. Major, minor, operation type, byte count의 네 field입니다.
`blkio.io_serviced`는 group이 disk에 발행한 I/O(bio) 수를 read/write, sync/async별로 나누며 major, minor, operation type, I/O count의 네 field입니다.
Device identity 뒤에 이어지는 측정값과 단위입니다.
Service·wait·merge·queue 통계
164-202`blkio.io_service_time`은 이 cgroup I/O가 dispatch된 때부터 완료될 때까지의 총 시간을 nanosecond로 집계합니다. Flash device에도 의미 있게 하기 위한 단위입니다. Queue depth 1에서는 실제 service time을 나타냅니다.
Queue depth가 1보다 크면 request가 순서와 다르게 service될 수 있어 특정 I/O service time에 여러 I/O의 시간이 포함될 수 있습니다. 그 결과 total `io_service_time`이 실제 elapsed time보다 클 수 있습니다. Read/write, sync/async별로 나누며 네 번째 field가 ns 값입니다.
`blkio.io_wait_time`은 cgroup의 개별 I/O가 scheduler queue에서 service를 기다린 시간을 누적합니다. 모든 I/O의 합이므로 실제 elapsed time보다 클 수 있으며 cgroup 전체가 기다린 wall-clock time은 아닙니다.
Queue depth가 1보다 큰 device에서는 dispatch 뒤 device reordering 때문에 실제 service까지 기다린 시간을 포함하지 않습니다. 이 통계도 nanosecond이며 read/write, sync/async별 네 번째 field에 기록됩니다.
`blkio.io_merged`는 이 cgroup request로 merge된 bio/request 총수를 operation type별로 표시합니다. `blkio.io_queued`는 어느 순간 이 cgroup에 queue된 request 총수를 operation type별로 표시합니다.
Queue-depth와 누적 방식이 해석에 미치는 영향을 함께 표시합니다.
BFQ debug 통계와 recursive file
203-249다음 debug 통계는 `CONFIG_BFQ_CGROUP_DEBUG=y`일 때만 enable됩니다. `blkio.avg_queue_size`는 cgroup 생존 기간 전체의 average queue size이며, cgroup queue 중 하나가 timeslice를 받을 때 sample을 취합니다.
`blkio.group_wait_time`은 cgroup이 busy(queued request 0에서 1)해진 때부터 queue 하나가 timeslice를 받을 때까지 기다린 nanosecond입니다. 개별 I/O scheduler wait의 누적인 `io_wait_time`과 다릅니다. 현재 waiting 상태에서 읽으면 마지막 timeslice까지 누적값만 보고하고 현재 delta는 제외합니다.
`blkio.empty_time`은 service 중이 아닐 때 pending request 없이 보낸 nanosecond이며 cgroup queue를 위해 idling한 시간은 제외합니다. 현재 empty 상태에서 읽으면 마지막 pending request까지 누적값만 보고하고 현재 delta는 제외합니다.
`blkio.idle_time`은 다른 queue/cgroup의 기존 request보다 나은 request를 기대하며 I/O scheduler가 해당 cgroup을 위해 idle한 nanosecond입니다. 현재 idling 상태에서 읽으면 마지막 idle period까지 누적값만 보고하고 현재 delta는 제외합니다.
`blkio.dequeue`는 group이 device service tree에서 dequeue된 횟수입니다. 첫 두 field는 major/minor, 세 번째는 해당 device에서 dequeue된 count입니다.
Group scheduling state를 분석하는 추가 통계입니다.
`blkio.*_recursive` file은 각 non-recursive 통계와 같은 정보를 보여 주되 descendant cgroup의 통계도 포함합니다.
기본 file은 local I/O, recursive variant는 descendant까지 합산합니다.
Throttling upper-limit file
250-296`blkio.throttle.read_bps_device`와 `write_bps_device`는 device별 read/write upper rate를 bytes per second로 지정합니다. `read_iops_device`와 `write_iops_device`는 device별 read/write upper rate를 I/O per second로 지정합니다.
Throttling/Upper limit policy files
-----------------------------------
blkio.throttle.read_bps_device
Specifies upper limit on READ rate from the device. IO rate is
specified in bytes per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.read_bps_device
blkio.throttle.write_bps_device
Specifies upper limit on WRITE rate to the device. IO rate is
specified in bytes per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.write_bps_device
blkio.throttle.read_iops_device
Specifies upper limit on READ rate from the device. IO rate is
specified in IO per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.read_iops_device
blkio.throttle.write_iops_device
Specifies upper limit on WRITE rate to the device. IO rate is
specified in io per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.write_iops_deviceDirection과 rate unit별 네 upper-limit interface입니다.
한 device에 bandwidth와 IOPS rule을 모두 지정하면 I/O는 두 constraint를 모두 만족해야 합니다.
한 request는 구성된 모든 upper limit을 통과해야 합니다.
`blkio.throttle.io_serviced`는 group이 disk에 발행한 I/O(bio) 수를 read/write, sync/async별로 나눕니다. Major, minor, operation type, I/O count의 네 field입니다.
`blkio.throttle.io_service_bytes`는 group이 disk와 주고받은 byte 수를 read/write, sync/async별로 나눕니다. Major, minor, operation type, byte count의 네 field입니다.
Cgroup 통계 reset
297-301`blkio.reset_stats`에 integer를 쓰면 해당 cgroup의 모든 statistics를 reset합니다.
Policy 설정은 유지하면서 cgroup 통계 누적값을 지웁니다.
Policies and hierarchy
blkio-controller.rst:1-93Block-layer upper limit 설정과 sane_behavior 유무에 따른 hierarchy 차이를 설명합니다.