User Interface for Resource Control feature (resctrl)

Resctrl 기능과 마운트

Resctrl은 CPU의 공유 cache와 memory bandwidth를 할당하고 관찰하는 Resource Control 사용자 인터페이스다. Intel은 이를 Intel Resource Director Technology(Intel RDT), AMD는 AMD Platform Quality of Service(AMD QoS)라고 부른다.

기능은 `CONFIG_X86_CPU_RESCTRL`과 x86 `/proc/cpuinfo` flag로 활성 여부를 확인한다. 그러나 새 기능을 모두 cpuinfo flag로 추가하면 사람이 읽기 어려우므로, userspace가 resctrl의 `info` 디렉터리에서 확인할 수 있는 기능은 새 flag 추가를 피해야 한다.

mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl

RDT 기능은 서로 직교하므로 시스템은 monitoring만, control만, 또는 둘 다 지원할 수 있다. Cache pseudo-locking은 cache control을 이용해 데이터를 cache에 pin하는 독특한 사용법이다.

할당 또는 monitoring 중 하나만 있어도 마운트는 성공하지만, 실제 시스템이 지원하는 파일과 디렉터리만 생성된다.

.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

=====================================================
User Interface for Resource Control feature (resctrl)
=====================================================

:Copyright: |copy| 2016 Intel Corporation
:Authors: - Fenghua Yu <fenghua.yu@intel.com>
          - Tony Luck <tony.luck@intel.com>
          - Vikas Shivappa <vikas.shivappa@intel.com>


Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).

This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
flag bits:

===============================================        ================================
RDT (Resource Director Technology) Allocation        "rdt_a"
CAT (Cache Allocation Technology)                "cat_l3", "cat_l2"
CDP (Code and Data Prioritization)                "cdp_l3", "cdp_l2"
CQM (Cache QoS Monitoring)                        "cqm_llc", "cqm_occup_llc"
MBM (Memory Bandwidth Monitoring)                "cqm_mbm_total", "cqm_mbm_local"
MBA (Memory Bandwidth Allocation)                "mba"
SMBA (Slow Memory Bandwidth Allocation)         ""
BMEC (Bandwidth Monitoring Event Configuration) ""
ABMC (Assignable Bandwidth Monitoring Counters) ""
===============================================        ================================

Historically, new features were made visible by default in /proc/cpuinfo. This
resulted in the feature flags becoming hard to parse by humans. Adding a new
flag to /proc/cpuinfo should be avoided if user space can obtain information
about the feature from resctrl's info directory.

To use the feature mount the file system::

 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl

mount options are:

"cdp":
        Enable code/data prioritization in L3 cache allocations.
"cdpl2":
        Enable code/data prioritization in L2 cache allocations.
"mba_MBps":
        Enable the MBA Software Controller(mba_sc) to specify MBA
        bandwidth in MiBps
"debug":
        Make debug files accessible. Available debug files are annotated with
        "Available only with debug option".

L2 and L3 CDP are controlled separately.

RDT features are orthogonal. A particular system may support only
monitoring, only control, or both monitoring and control.  Cache
pseudo-locking is a unique way of using cache control to "pin" or
"lock" data in the cache. Details can be found in
"Cache Pseudo-Locking".


The mount succeeds if either of allocation or monitoring is present, but
only those files and directories supported by the system will be created.
For more details on the behavior of the interface during monitoring
and allocation, see the "Resource alloc and monitor groups" section.

`info` 할당 리소스 파일

`info` 디렉터리는 활성화된 resource 정보를 담고 resource마다 이름이 같은 하위 디렉터리를 만든다.

`bit_usage`는 resource instance마다 capacity bit 사용 상태를 문자로 주석 처리해 보여 준다. 모든 할당 뒤에도 `0`이 있으면 resource가 낭비되고 있다는 신호다.

요청 값이 hardware step 사이에 있으면 다음 사용 가능한 제어 step으로 반올림된다.

Info directory
==============

The 'info' directory contains information about the enabled
resources. Each resource has its own subdirectory. The subdirectory
names reflect the resource names.

Each subdirectory contains the following files with respect to
allocation:

Cache resource(L3/L2)  subdirectory contains the following files
related to allocation:

"num_closids":
                The number of CLOSIDs which are valid for this
                resource. The kernel uses the smallest number of
                CLOSIDs of all enabled resources as limit.
"cbm_mask":
                The bitmask which is valid for this resource.
                This mask is equivalent to 100%.
"min_cbm_bits":
                The minimum number of consecutive bits which
                must be set when writing a mask.

"shareable_bits":
                Bitmask of shareable resource with other executing
                entities (e.g. I/O). User can use this when
                setting up exclusive cache partitions. Note that
                some platforms support devices that have their
                own settings for cache use which can over-ride
                these bits.
"bit_usage":
                Annotated capacity bitmasks showing how all
                instances of the resource are used. The legend is:

                        "0":
                              Corresponding region is unused. When the system's
                              resources have been allocated and a "0" is found
                              in "bit_usage" it is a sign that resources are
                              wasted.

                        "H":
                              Corresponding region is used by hardware only
                              but available for software use. If a resource
                              has bits set in "shareable_bits" but not all
                              of these bits appear in the resource groups'
                              schematas then the bits appearing in
                              "shareable_bits" but no resource group will
                              be marked as "H".
                        "X":
                              Corresponding region is available for sharing and
                              used by hardware and software. These are the
                              bits that appear in "shareable_bits" as
                              well as a resource group's allocation.
                        "S":
                              Corresponding region is used by software
                              and available for sharing.
                        "E":
                              Corresponding region is used exclusively by
                              one resource group. No sharing allowed.
                        "P":
                              Corresponding region is pseudo-locked. No
                              sharing allowed.
"sparse_masks":
                Indicates if non-contiguous 1s value in CBM is supported.

                        "0":
                              Only contiguous 1s value in CBM is supported.
                        "1":
                              Non-contiguous 1s value in CBM is supported.

Memory bandwidth(MB) subdirectory contains the following files
with respect to allocation:

"min_bandwidth":
                The minimum memory bandwidth percentage which
                user can request.

"bandwidth_gran":
                The granularity in which the memory bandwidth
                percentage is allocated. The allocated
                b/w percentage is rounded off to the next
                control step available on the hardware. The

`L3_MON`과 BMEC event 구성

RDT monitoring이 있으면 `info/L3_MON` 디렉터리가 생긴다. `num_rmids`는 생성 가능한 `CTRL_MON + MON` group 수의 상한인 RMID 개수다. `mon_features`는 `llc_occupancy`, `mbm_total_bytes`, `mbm_local_bytes` 같은 활성 monitoring event를 나열한다.

BMEC를 지원하면 `mbm_total_bytes_config`와 `mbm_local_bytes_config`가 추가된다. 이 read/write 파일은 domain별 event mask를 저장하며 같은 domain의 모든 CPU에 영향을 준다.

어느 하나의 event 구성을 바꾸면 해당 domain에서 두 event의 모든 RMID bandwidth counter가 초기화된다. 각 RMID의 다음 첫 read는 `Unavailable`, 이후 read는 유효값을 반환한다.

기본 `mbm_total_bytes_config`는 모든 event를 세는 `0x7f`, `mbm_local_bytes_config`는 local memory event를 세는 `0x15`다. 출력 형식은 `domain=mask`를 세미콜론으로 연결한다.

# 현재 구성
cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config

# domain 0에서 read만 집계: bits 0,1,4,5 = 0x33
echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config

# domain 0,1에서 slow memory read 집계: bits 4,5 = 0x30
echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config

부분 write는 지정한 domain만 바꾸며 나머지 domain mask는 기존 값을 유지한다.

                available bandwidth control steps are:
                min_bandwidth + N * bandwidth_gran.

"delay_linear":
                Indicates if the delay scale is linear or
                non-linear. This field is purely informational
                only.

"thread_throttle_mode":
                Indicator on Intel systems of how tasks running on threads
                of a physical core are throttled in cases where they
                request different memory bandwidth percentages:

                "max":
                        the smallest percentage is applied
                        to all threads
                "per-thread":
                        bandwidth percentages are directly applied to
                        the threads running on the core

If RDT monitoring is available there will be an "L3_MON" directory
with the following files:

"num_rmids":
                The number of RMIDs available. This is the
                upper bound for how many "CTRL_MON" + "MON"
                groups can be created.

"mon_features":
                Lists the monitoring events if
                monitoring is enabled for the resource.
                Example::

                        # cat /sys/fs/resctrl/info/L3_MON/mon_features
                        llc_occupancy
                        mbm_total_bytes
                        mbm_local_bytes

                If the system supports Bandwidth Monitoring Event
                Configuration (BMEC), then the bandwidth events will
                be configurable. The output will be::

                        # cat /sys/fs/resctrl/info/L3_MON/mon_features
                        llc_occupancy
                        mbm_total_bytes
                        mbm_total_bytes_config
                        mbm_local_bytes
                        mbm_local_bytes_config

"mbm_total_bytes_config", "mbm_local_bytes_config":
        Read/write files containing the configuration for the mbm_total_bytes
        and mbm_local_bytes events, respectively, when the Bandwidth
        Monitoring Event Configuration (BMEC) feature is supported.
        The event configuration settings are domain specific and affect
        all the CPUs in the domain. When either event configuration is
        changed, the bandwidth counters for all RMIDs of both events
        (mbm_total_bytes as well as mbm_local_bytes) are cleared for that
        domain. The next read for every RMID will report "Unavailable"
        and subsequent reads will report the valid value.

        Following are the types of events supported:

        ====    ========================================================
        Bits    Description
        ====    ========================================================
        6       Dirty Victims from the QOS domain to all types of memory
        5       Reads to slow memory in the non-local NUMA domain
        4       Reads to slow memory in the local NUMA domain
        3       Non-temporal writes to non-local NUMA domain
        2       Non-temporal writes to local NUMA domain
        1       Reads to memory in the non-local NUMA domain
        0       Reads to memory in the local NUMA domain
        ====    ========================================================

        By default, the mbm_total_bytes configuration is set to 0x7f to count
        all the event types and the mbm_local_bytes configuration is set to
        0x15 to count all the local memory events.

        Examples:

        * To view the current configuration::
          ::

            # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
            0=0x7f;1=0x7f;2=0x7f;3=0x7f

            # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
            0=0x15;1=0x15;3=0x15;4=0x15

        * To change the mbm_total_bytes to count only reads on domain 0,
          the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary

MBM counter assignment와 event filter

`mbm_assign_mode`는 지원되는 counter 할당 mode를 보여 주며 대괄호가 현재 mode를 표시한다. Mode 전환 때 MBM event와 연결된 counter가 reset될 수 있다.

`mbm_event`에서는 hardware counter가 붙어 있을 때만 event가 누적된다. 각 group의 `mbm_L3_assignments`로 event별 할당을 정하고 `num_mbm_cntrs`에서 최대 counter 수를 확인한다. 할당하지 않으면 read 결과는 `Unassigned`다.

이 mode는 group 수가 hardware counter보다 많은 AMD 플랫폼에 유리하다. ABMC capability가 있는 AMD에서는 기본 활성화돼 RMID가 CPU에서 사용 중이 아니어도 counter 연결을 유지한다. AMD에서 지원된다면 hardware 재할당에 따른 reset·오해 가능한 값·`Unavailable`을 피하려고 `mbm_event`를 권장한다.

echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs

`num_mbm_cntrs`는 domain별 전체 counter 수, `available_mbm_cntrs`는 현재 할당 가능한 counter 수다. 예시는 각 L3 domain에 최대 32개, 그중 30개가 사용 가능함을 보여 준다.

`event_configs`는 할당 가능한 MBM event마다 하위 디렉터리를 가진다. 기본 event는 `mbm_local_bytes`, `mbm_total_bytes`이며, `mbm_event` mode에서만 접근 가능한 `event_filter`가 집계할 transaction을 정한다.

`event_filter`를 읽으면 쉼표로 연결한 현재 구성, 쓰면 새 transaction 집합을 지정한다.

`mbm_assign_on_mkdir`는 monitor group을 `mkdir`할 때 RMID·MBM event 쌍에 counter를 자동 할당할지 정한다. Boot와 `default`에서 `mbm_event`로 전환할 때 기본값은 `1`; `0`은 자동 할당 비활성이다.

`max_threshold_occupancy`는 사용했던 LLC occupancy counter를 재사용 가능하다고 볼 최대 byte 값이다.

`info/last_cmd_status`는 디렉터리 생성이나 control 파일 write마다 reset된다. 성공하면 `ok`, 실패하면 file operation의 errno보다 자세한 이유를 제공한다. 예를 들어 비연속 CBM `f7` write는 `mask f7 has non-consecutive 1-bits`를 남긴다.

          (in hexadecimal 0x33):
          ::

            # echo  "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config

            # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
            0=0x33;1=0x7f;2=0x7f;3=0x7f

        * To change the mbm_local_bytes to count all the slow memory reads on
          domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b
          in binary (in hexadecimal 0x30):
          ::

            # echo  "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config

            # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
            0=0x30;1=0x30;3=0x15;4=0x15

"mbm_assign_mode":
        The supported counter assignment modes. The enclosed brackets indicate which mode
        is enabled. The MBM events associated with counters may reset when "mbm_assign_mode"
        is changed.
        ::

          # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
          [mbm_event]
          default

        "mbm_event":

        mbm_event mode allows users to assign a hardware counter to an RMID, event
        pair and monitor the bandwidth usage as long as it is assigned. The hardware
        continues to track the assigned counter until it is explicitly unassigned by
        the user. Each event within a resctrl group can be assigned independently.

        In this mode, a monitoring event can only accumulate data while it is backed
        by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON
        group to specify which of the events should have a counter assigned. The number
        of counters available is described in the "num_mbm_cntrs" file. Changing the
        mode may cause all counters on the resource to reset.

        Moving to mbm_event counter assignment mode requires users to assign the counters
        to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.

        The mode is beneficial for AMD platforms that support more CTRL_MON
        and MON groups than available hardware counters. By default, this
        feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
        Monitoring Counters) capability, ensuring counters remain assigned even
        when the corresponding RMID is not actively used by any processor.

        "default":

        In default mode, resctrl assumes there is a hardware counter for each
        event within every CTRL_MON and MON group. On AMD platforms, it is
        recommended to use the mbm_event mode, if supported, to prevent reset of MBM
        events between reads resulting from hardware re-allocating counters. This can
        result in misleading values or display "Unavailable" if no counter is assigned
        to the event.

        * To enable "mbm_event" counter assignment mode:
          ::

            # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

        * To enable "default" monitoring mode:
          ::

            # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

"num_mbm_cntrs":
        The maximum number of counters (total of available and assigned counters) in
        each domain when the system supports mbm_event mode.

        For example, on a system with maximum of 32 memory bandwidth monitoring
        counters in each of its L3 domains:
        ::

          # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
          0=32;1=32

"available_mbm_cntrs":
        The number of counters available for assignment in each domain when mbm_event
        mode is enabled on the system.

        For example, on a system with 30 available [hardware] assignable counters
        in each of its L3 domains:
        ::

          # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
          0=30;1=30

"event_configs":
        Directory that exists when "mbm_event" counter assignment mode is supported.
        Contains a sub-directory for each MBM event that can be assigned to a counter.

        Two MBM events are supported by default: mbm_local_bytes and mbm_total_bytes.
        Each MBM event's sub-directory contains a file named "event_filter" that is
        used to view and modify which memory transactions the MBM event is configured
        with. The file is accessible only when "mbm_event" counter assignment mode is
        enabled.

        List of memory transaction types supported:

        ==========================  ========================================================
        Name                            Description
        ==========================  ========================================================
        dirty_victim_writes_all     Dirty Victims from the QOS domain to all types of memory
        remote_reads_slow_memory    Reads to slow memory in the non-local NUMA domain
        local_reads_slow_memory     Reads to slow memory in the local NUMA domain
        remote_non_temporal_writes  Non-temporal writes to non-local NUMA domain
        local_non_temporal_writes   Non-temporal writes to local NUMA domain
        remote_reads                Reads to memory in the non-local NUMA domain
        local_reads                 Reads to memory in the local NUMA domain
        ==========================  ========================================================

        For example::

          # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
          local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,
          local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all

          # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
          local_reads,local_non_temporal_writes,local_reads_slow_memory

        Modify the event configuration by writing to the "event_filter" file within
        the "event_configs" directory. The read/write "event_filter" file contains the
        configuration of the event that reflects which memory transactions are counted by it.

        For example::

          # echo "local_reads, local_non_temporal_writes" >
            /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter

          # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
           local_reads,local_non_temporal_writes

"mbm_assign_on_mkdir":
        Exists when "mbm_event" counter assignment mode is supported. Accessible
        only when "mbm_event" counter assignment mode is enabled.

        Determines if a counter will automatically be assigned to an RMID, MBM event
        pair when its associated monitor group is created via mkdir. Enabled by default
        on boot, also when switched from "default" mode to "mbm_event" counter assignment
        mode. Users can disable this capability by writing to the interface.

        "0":
                Auto assignment is disabled.
        "1":
                Auto assignment is enabled.

        Example::

          # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
          # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
          0

"max_threshold_occupancy":
                Read/write file provides the largest value (in
                bytes) at which a previously used LLC_occupancy
                counter can be considered for re-use.

Finally, in the top level of the "info" directory there is a file
named "last_cmd_status". This is reset with every "command" issued
via the file system (making new directories or writing to any of the
control files). If the command was successful, it will read as "ok".
If the command failed, it will provide more information that can be
conveyed in the error returns from file operations. E.g.
::

        # echo L3:0=f7 > schemata
        bash: echo: write error: Invalid argument
        # cat info/last_cmd_status
        mask f7 has non-consecutive 1-bits

CTRL_MON·MON group과 파일

Resource group은 resctrl 디렉터리로 표현한다. 마운트 직후 root인 default group은 시스템의 모든 task와 CPU를 소유하고 모든 resource를 전부 사용할 수 있다.

Control 기능이 있으면 root 아래에 resource 양을 달리하는 디렉터리를 만들 수 있다. Root와 이 top-level 디렉터리를 `CTRL_MON` group이라 한다. Monitoring이 있으면 각 CTRL_MON의 `mon_groups` 아래에 그 조상 group의 task subset을 관찰하는 `MON` group을 만든다.

Group 디렉터리를 제거하면 소유 task와 CPU가 부모로 이동한다. CTRL_MON을 제거하면 아래 MON도 모두 자동 제거된다. CPU를 monitor하는 MON은 이동할 수 없지만 task만 monitor하는 MON은 monitoring data와 task를 유지한 채 새 CTRL_MON 부모로 옮겨 allocation을 바꿀 수 있다. 그 밖에는 단순 rename만 허용한다.

Task를 CTRL_MON에 쓰면 이전 CTRL_MON과 모든 MON에서 제거된다. MON에 쓰려면 이미 그 MON의 부모 CTRL_MON에 속해야 하며 이전 MON에서는 제거된다. 여러 task 할당 중 실패는 `last_cmd_status`에 기록된다.

Pseudo-locked region은 `mode`에 `pseudo-locksetup`을 쓴 뒤 `schemata`에 cache 영역을 쓰면 만들며, 성공하면 mode가 자동으로 `pseudo-locked`가 된다.

Monitoring이 활성화되면 `mon_data`가 L3 domain과 event별 파일을 가진다. 두 domain이면 `mon_L3_00`, `mon_L3_01` 아래에 `llc_occupancy`, `mbm_total_bytes`, `mbm_local_bytes` 등이 생긴다. MON에서는 해당 group task 값, CTRL_MON에서는 자체 task와 모든 하위 MON task의 합을 보여 준다.

SNC에서는 `mon_L3_XX` 아래 node별 `mon_sub_L3_YY`가 추가된다. `mbm_event` mode에서 MON event에 counter가 없으면 `Unassigned`; CTRL_MON은 자체와 모든 관련 MON 어디에도 할당이 없을 때 `Unassigned`다. `mon_hw_id`는 debug 전용 RMID다.

`mbm_L3_assignments`는 `mbm_event` 지원 시 group의 counter 상태를 `<Event>:<Domain ID>=<state>` 형식으로 표시한다. Domain `*`는 write 때 모든 domain, `_`는 미할당, `e`는 exclusive 할당이다.

cat /sys/fs/resctrl/mbm_L3_assignments
echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments

`mba_MBps` 마운트 시 CTRL_MON의 `mba_MBps_event`는 software feedback loop 입력 event를 보여 준다. `mon_features`가 지원하는 bandwidth event 이름을 쓰면 입력을 바꾼다.

Resource alloc and monitor groups
=================================

Resource groups are represented as directories in the resctrl file
system.  The default group is the root directory which, immediately
after mounting, owns all the tasks and cpus in the system and can make
full use of all resources.

On a system with RDT control features additional directories can be
created in the root directory that specify different amounts of each
resource (see "schemata" below). The root and these additional top level
directories are referred to as "CTRL_MON" groups below.

On a system with RDT monitoring the root directory and other top level
directories contain a directory named "mon_groups" in which additional
directories can be created to monitor subsets of tasks in the CTRL_MON
group that is their ancestor. These are called "MON" groups in the rest
of this document.

Removing a directory will move all tasks and cpus owned by the group it
represents to the parent. Removing one of the created CTRL_MON groups
will automatically remove all MON groups below it.

Moving MON group directories to a new parent CTRL_MON group is supported
for the purpose of changing the resource allocations of a MON group
without impacting its monitoring data or assigned tasks. This operation
is not allowed for MON groups which monitor CPUs. No other move
operation is currently allowed other than simply renaming a CTRL_MON or
MON group.

All groups contain the following files:

"tasks":
        Reading this file shows the list of all tasks that belong to
        this group. Writing a task id to the file will add a task to the
        group. Multiple tasks can be added by separating the task ids
        with commas. Tasks will be assigned sequentially. Multiple
        failures are not supported. A single failure encountered while
        attempting to assign a task will cause the operation to abort and
        already added tasks before the failure will remain in the group.
        Failures will be logged to /sys/fs/resctrl/info/last_cmd_status.

        If the group is a CTRL_MON group the task is removed from
        whichever previous CTRL_MON group owned the task and also from
        any MON group that owned the task. If the group is a MON group,
        then the task must already belong to the CTRL_MON parent of this
        group. The task is removed from any previous MON group.


"cpus":
        Reading this file shows a bitmask of the logical CPUs owned by
        this group. Writing a mask to this file will add and remove
        CPUs to/from this group. As with the tasks file a hierarchy is
        maintained where MON groups may only include CPUs owned by the
        parent CTRL_MON group.
        When the resource group is in pseudo-locked mode this file will
        only be readable, reflecting the CPUs associated with the
        pseudo-locked region.


"cpus_list":
        Just like "cpus", only using ranges of CPUs instead of bitmasks.


When control is enabled all CTRL_MON groups will also contain:

"schemata":
        A list of all the resources available to this group.
        Each resource has its own line and format - see below for details.

"size":
        Mirrors the display of the "schemata" file to display the size in
        bytes of each allocation instead of the bits representing the
        allocation.

"mode":
        The "mode" of the resource group dictates the sharing of its
        allocations. A "shareable" resource group allows sharing of its
        allocations while an "exclusive" resource group does not. A
        cache pseudo-locked region is created by first writing
        "pseudo-locksetup" to the "mode" file before writing the cache
        pseudo-locked region's schemata to the resource group's "schemata"
        file. On successful pseudo-locked region creation the mode will
        automatically change to "pseudo-locked".

"ctrl_hw_id":
        Available only with debug option. The identifier used by hardware
        for the control group. On x86 this is the CLOSID.

When monitoring is enabled all MON groups will also contain:

"mon_data":
        This contains a set of files organized by L3 domain and by
        RDT event. E.g. on a system with two L3 domains there will
        be subdirectories "mon_L3_00" and "mon_L3_01".        Each of these
        directories have one file per event (e.g. "llc_occupancy",
        "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
        files provide a read out of the current value of the event for
        all tasks in the group. In CTRL_MON groups these files provide
        the sum for all tasks in the CTRL_MON group and all tasks in
        MON groups. Please see example section for more details on usage.
        On systems with Sub-NUMA Cluster (SNC) enabled there are extra
        directories for each node (located within the "mon_L3_XX" directory
        for the L3 cache they occupy). These are named "mon_sub_L3_YY"
        where "YY" is the node number.

        When the 'mbm_event' counter assignment mode is enabled, reading
        an MBM event of a MON group returns 'Unassigned' if no hardware
        counter is assigned to it. For CTRL_MON groups, 'Unassigned' is
        returned if the MBM event does not have an assigned counter in the
        CTRL_MON group nor in any of its associated MON groups.

"mon_hw_id":
        Available only with debug option. The identifier used by hardware
        for the monitor group. On x86 this is the RMID.

When monitoring is enabled all MON groups may also contain:

"mbm_L3_assignments":
        Exists when "mbm_event" counter assignment mode is supported and lists the
        counter assignment states of the group.

        The assignment list is displayed in the following format:

        <Event>:<Domain ID>=<Assignment state>;<Domain ID>=<Assignment state>

        Event: A valid MBM event in the
               /sys/fs/resctrl/info/L3_MON/event_configs directory.

        Domain ID: A valid domain ID. When writing, '*' applies the changes
                   to all the domains.

        Assignment states:

        _ : No counter assigned.

        e : Counter assigned exclusively.

        Example:

        To display the counter assignment states for the default group.
        ::

         # cd /sys/fs/resctrl
         # cat /sys/fs/resctrl/mbm_L3_assignments
           mbm_total_bytes:0=e;1=e
           mbm_local_bytes:0=e;1=e

        Assignments can be modified by writing to the interface.

        Examples:

        To unassign the counter associated with the mbm_total_bytes event on domain 0:
        ::

         # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
         # cat /sys/fs/resctrl/mbm_L3_assignments
           mbm_total_bytes:0=_;1=e
           mbm_local_bytes:0=e;1=e

        To unassign the counter associated with the mbm_total_bytes event on all the domains:
        ::

         # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
         # cat /sys/fs/resctrl/mbm_L3_assignments
           mbm_total_bytes:0=_;1=_
           mbm_local_bytes:0=e;1=e

        To assign a counter associated with the mbm_total_bytes event on all domains in
        exclusive mode:
        ::

         # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
         # cat /sys/fs/resctrl/mbm_L3_assignments
           mbm_total_bytes:0=e;1=e
           mbm_local_bytes:0=e;1=e

When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:

"mba_MBps_event":
        Reading this file shows which memory bandwidth event is used
        as input to the software feedback loop that keeps memory bandwidth
        below the value specified in the schemata file. Writing the
        name of one of the supported memory bandwidth events found in
        /sys/fs/resctrl/info/L3_MON/mon_features changes the input
        event.

Allocation·monitoring 선택 규칙

두 규칙 모두 명시적인 task membership을 먼저 보고, default task에 한해 현재 실행 CPU의 group을 보며, 마지막에 root default로 돌아간다.

Resource allocation rules
-------------------------

When a task is running the following rules define which resources are
available to it:

1) If the task is a member of a non-default group, then the schemata
   for that group is used.

2) Else if the task belongs to the default group, but is running on a
   CPU that is assigned to some specific group, then the schemata for the
   CPU's group is used.

3) Otherwise the schemata for the default group is used.

Resource monitoring rules
-------------------------
1) If a task is a member of a MON group, or non-default CTRL_MON group
   then RDT events for the task will be reported in that group.

2) If a task is a member of the default CTRL_MON group, but is running
   on a CPU that is assigned to some specific group, then the RDT events
   for the task will be reported in that group.

3) Otherwise RDT events for the task will be reported in the root level
   "mon_data" group.

Occupancy·RMID limbo·Cache ID·CBM

Task를 다른 group으로 옮겨도 새 cache allocation에만 영향을 준다. 옛 group에서 3MB occupancy를 보이던 task를 옮긴 직후에는 옛 group이 여전히 3MB, 새 group은 0일 수 있다. 기존 cache line을 다시 접근해도 hardware counter는 갱신되지 않으며, eviction과 새 load가 진행되면서 옛 값은 내려가고 새 값은 올라간다.

Cache allocation control도 마찬가지다. 더 작은 partition으로 옮겨도 기존 line을 강제 퇴출하지 않으므로 process가 옛 partition의 line을 계속 사용할 수 있다.

Hardware는 control group을 CLOSID, monitoring group을 RMID로 식별한다. 개수가 제한돼 CTRL_MON 생성은 CLOSID 또는 RMID 부족으로, MON 생성은 RMID 부족으로 실패할 수 있다.

해제한 RMID가 이전 사용자의 cache line에 아직 tag돼 있으면 즉시 재사용할 수 없다. Limbo 목록에서 occupancy가 내려가는지 확인하며, 재사용 불가능한 limbo RMID가 많으면 `mkdir`가 `-EBUSY`를 반환할 수 있다. `max_threshold_occupancy`가 재사용 가능한 occupancy byte 한계를 정한다.

`mon_llc_occupancy_limbo` tracepoint는 즉시 할당할 수 없는 RMID subset의 정확한 byte occupancy를 제공한다. 매초 출력된다고 보장할 수 없고 빈 monitor group 생성을 시도해 update를 강제해야 할 수 있으며 group 생성 실패 때만 출력될 수도 있다.

Schemata의 각 행은 resource 이름과 시스템의 각 instance에 적용할 값을 담는다.

Cache 공유 단위는 socket·core라는 가정 대신 `Cache ID`로 식별한다. 같은 cache level에서 시스템 전체에 유일하지만 연속 번호라고 보장하지 않는다. Logical CPU별 ID는 `/sys/devices/system/cpu/cpu*/cache/index*/id`에서 찾는다.

Cache allocation은 CBM bitmask로 표현한다. 최대 mask는 CPU model과 cache level마다 다르며 CPUID와 `info/{resource}/cbm_mask`에서 확인한다. 일부 Intel hardware는 1 bit가 연속해야 하므로 4-bit mask에서 `0x3`, `0x6`, `0xC`는 유효하지만 `0x5`, `0x9`, `0xA`는 유효하지 않다. `sparse_masks`로 비연속 지원을 확인한다.

20-bit mask에서는 bit 하나가 cache 용량 5%다. 네 등분 mask는 `0x1f`, `0x3e0`, `0x7c00`, `0xf8000`이다.

Notes on cache occupancy monitoring and control
===============================================
When moving a task from one group to another you should remember that
this only affects *new* cache allocations by the task. E.g. you may have
a task in a monitor group showing 3 MB of cache occupancy. If you move
to a new group and immediately check the occupancy of the old and new
groups you will likely see that the old group is still showing 3 MB and
the new group zero. When the task accesses locations still in cache from
before the move, the h/w does not update any counters. On a busy system
you will likely see the occupancy in the old group go down as cache lines
are evicted and re-used while the occupancy in the new group rises as
the task accesses memory and loads into the cache are counted based on
membership in the new group.

The same applies to cache allocation control. Moving a task to a group
with a smaller cache partition will not evict any cache lines. The
process may continue to use them from the old partition.

Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
to identify a control group and a monitoring group respectively. Each of
the resource groups are mapped to these IDs based on the kind of group. The
number of CLOSid and RMID are limited by the hardware and hence the creation of
a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID
and creation of "MON" group may fail if we run out of RMIDs.

max_threshold_occupancy - generic concepts
------------------------------------------

Note that an RMID once freed may not be immediately available for use as
the RMID is still tagged the cache lines of the previous user of RMID.
Hence such RMIDs are placed on limbo list and checked back if the cache
occupancy has gone down. If there is a time when system has a lot of
limbo RMIDs but which are not ready to be used, user may see an -EBUSY
during mkdir.

max_threshold_occupancy is a user configurable value to determine the
occupancy at which an RMID can be freed.

The mon_llc_occupancy_limbo tracepoint gives the precise occupancy in bytes
for a subset of RMID that are not immediately available for allocation.
This can't be relied on to produce output every second, it may be necessary
to attempt to create an empty monitor group to force an update. Output may
only be produced if creation of a control or monitor group fails.

Schemata files - general concepts
---------------------------------
Each line in the file describes one resource. The line starts with
the name of the resource, followed by specific values to be applied
in each of the instances of that resource on the system.

Cache IDs
---------
On current generation systems there is one L3 cache per socket and L2
caches are generally just shared by the hyperthreads on a core, but this
isn't an architectural requirement. We could have multiple separate L3
caches on a socket, multiple cores could share an L2 cache. So instead
of using "socket" or "core" to define the set of logical cpus sharing
a resource we use a "Cache ID". At a given cache level this will be a
unique number across the whole system (but it isn't guaranteed to be a
contiguous sequence, there may be gaps).  To find the ID for each logical
CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id

Cache Bit Masks (CBM)
---------------------
For cache resources we describe the portion of the cache that is available
for allocation using a bitmask. The maximum value of the mask is defined
by each cpu model (and may be different for different cache levels). It
is found using CPUID, but is also provided in the "info" directory of
the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware
requires that these masks have all the '1' bits in a contiguous block. So
0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks
if non-contiguous 1s value is supported. On a system with a 20-bit mask
each bit represents 5% of the capacity of the cache. You could partition
the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.

SNC와 Memory bandwidth 의미

SNC mode에서는 같은 L3 cache를 공유하고 NUMA distance도 더 작게 보고될 수 있어 Linux가 일반 NUMA node 사이보다 Sub-NUMA node 사이 task를 더 적극적으로 load balance할 수 있다.

각 `mon_L3_XX` top-level monitoring 파일은 같은 L3를 공유하는 모든 SNC node의 합이다. 특정 node CPU에 task를 bind했다면 `mon_sub_L3_YY`의 `llc_occupancy`, `mbm_total_bytes`, `mbm_local_bytes`에서 node-local 값을 읽는다.

Memory bandwidth allocation과 L3 CBM은 여전히 L3 cache level의 모든 SNC node에 적용된다. 다만 bit 하나가 나타내는 L3 용량은 L3당 SNC node 수로 나뉜다. 100MB·10-bit cache는 보통 bit당 10MB지만 SNC node 둘이면 bit당 5MB다.

MBA 기본 인터페이스는 전체 memory bandwidth의 백분율을 지정한다. 최소값은 `info/MB/min_bandwidth`, granularity는 `info/MB/bandwidth_gran`이며 step은 `min_bw + N * bw_gran`이다.

일부 Intel SKU의 throttling은 core 단위다. 같은 core thread가 서로 다른 값을 요청하면 `thread_throttle_mode`에 따라 낮은 값이 둘 모두에 적용될 수 있다.

MBA는 core별인데 MBM은 package level일 수 있어 제어 효과 해석이 헷갈릴 수 있다. 24-core package에서 core당 L2 외부 bandwidth가 10GBps, L3 외부가 100GBps이면 20 thread가 각각 50%로 5GBps를 써 이미 L3 100GBps를 채운다. 백분율을 높여도 실제 bandwidth는 늘지 않는다.

같은 10%라도 thread 하나는 최대 10GBps, 네 thread는 최대 40GBps를 쓸 수 있다. Group이 더 많은 core를 사용하면 지정 백분율이 같아도 실제 bandwidth가 달라진다.

이를 완화하려고 `mba_MBps`와 software controller `mba_sc`가 MBM counter의 실제 값을 읽어 percentage를 조정해 `actual bandwidth < user specified bandwidth`를 유지한다.

Notes on Sub-NUMA Cluster mode
==============================
When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
nodes much more readily than between regular NUMA nodes since the CPUs
on Sub-NUMA nodes share the same L3 cache and the system may report
the NUMA distance between Sub-NUMA nodes with a lower value than used
for regular NUMA nodes.

The top-level monitoring files in each "mon_L3_XX" directory provide
the sum of data across all SNC nodes sharing an L3 cache instance.
Users who bind tasks to the CPUs of a specific Sub-NUMA node can read
the "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes" in the
"mon_sub_L3_YY" directories to get node local data.

Memory bandwidth allocation is still performed at the L3 cache
level. I.e. throttling controls are applied to all SNC nodes.

L3 cache allocation bitmaps also apply to all SNC nodes. But note that
the amount of L3 cache represented by each bit is divided by the number
of SNC nodes per L3 cache. E.g. with a 100MB cache on a system with 10-bit
allocation masks each bit normally represents 10MB. With SNC mode enabled
with two SNC nodes per L3 cache, each bit only represents 5MB.

Memory bandwidth Allocation and monitoring
==========================================

For Memory bandwidth resource, by default the user controls the resource
by indicating the percentage of total memory bandwidth.

The minimum bandwidth percentage value for each cpu model is predefined
and can be looked up through "info/MB/min_bandwidth". The bandwidth
granularity that is allocated is also dependent on the cpu model and can
be looked up at "info/MB/bandwidth_gran". The available bandwidth
control steps are: min_bw + N * bw_gran. Intermediate values are rounded
to the next control step available on the hardware.

The bandwidth throttling is a core specific mechanism on some of Intel
SKUs. Using a high bandwidth and a low bandwidth setting on two threads
sharing a core may result in both threads being throttled to use the
low bandwidth (see "thread_throttle_mode").

The fact that Memory bandwidth allocation(MBA) may be a core
specific mechanism where as memory bandwidth monitoring(MBM) is done at
the package level may lead to confusion when users try to apply control
via the MBA and then monitor the bandwidth to see if the controls are
effective. Below are such scenarios:

1. User may *not* see increase in actual bandwidth when percentage
   values are increased:

This can occur when aggregate L2 external bandwidth is more than L3
external bandwidth. Consider an SKL SKU with 24 cores on a package and
where L2 external  is 10GBps (hence aggregate L2 external bandwidth is
240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20
threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3
bandwidth of 100GBps although the percentage value specified is only 50%
<< 100%. Hence increasing the bandwidth percentage will not yield any
more bandwidth. This is because although the L2 external bandwidth still
has capacity, the L3 external bandwidth is fully used. Also note that
this would be dependent on number of cores the benchmark is run on.

2. Same bandwidth percentage may mean different actual bandwidth
   depending on # of threads:

For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4
thread, with 10% bandwidth' can consume up to 10GBps and 40GBps although
they have same percentage bandwidth of 10%. This is simply because as
threads start using more cores in an rdtgroup, the actual bandwidth may
increase or vary although user specified bandwidth percentage is same.

In order to mitigate this and make the interface more user friendly,
resctrl added support for specifying the bandwidth in MiBps as well.  The
kernel underneath would use a software feedback mechanism or a "Software
Controller(mba_sc)" which reads the actual bandwidth using MBM counters
and adjust the memory bandwidth percentages to ensure::

        "actual bandwidth < user specified bandwidth".

By default, the schemata would take the bandwidth percentage values
where as user can switch to the "MBA software controller" mode using
a mount option 'mba_MBps'. The schemata format is specified in the below
sections.

Schemata 형식과 AMD SMBA

Memory bandwidth domain은 L3 cache다. AMD SMBA는 CXL.memory만 slow memory 장치로 지원하며 여러 장치가 있으면 모든 slow source를 묶어 전체에 한도를 적용한다. SMBA capability와 실제 slow memory 장치 존재는 독립적이므로 장치가 없으면 설정해도 성능에 영향이 없다.

Schemata를 읽으면 모든 resource의 모든 domain 상태가 나오고, 쓸 때는 바꿀 값만 지정하면 된다. 예를 들어 `L3DATA:2=3c0`만 쓰면 domain 2 data mask만 바뀌고 나머지 L3DATA와 L3CODE는 유지된다.

cat schemata
echo "L3DATA:2=3c0;" > schemata

# AMD: 1/8 GB/s 단위, cache id 1에 2GB/s = 16
echo "MB:1=16" > schemata

# AMD SMBA: cache id 1에 8GB/s = 64
echo "SMBA:1=64" > schemata

AMD schemata의 bandwidth는 1/8GB/s 배수다. 따라서 `MB:1=16`은 cache ID 1에 2GB/s, `SMBA:1=64`는 slow memory에 8GB/s 한도를 뜻한다.

L3 schemata file details (code and data prioritization disabled)
----------------------------------------------------------------
With CDP disabled the L3 schemata format is::

        L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

L3 schemata file details (CDP enabled via mount option to resctrl)
------------------------------------------------------------------
When CDP is enabled L3 control is split into two separate resources
so you can specify independent masks for code and data like this::

        L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
        L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

L2 schemata file details
------------------------
CDP is supported at L2 using the 'cdpl2' mount option. The schemata
format is either::

        L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

or

        L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
        L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...


Memory bandwidth Allocation (default mode)
------------------------------------------

Memory b/w domain is L3 cache.
::

        MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...

Memory bandwidth Allocation specified in MiBps
----------------------------------------------

Memory bandwidth domain is L3 cache.
::

        MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;...

Slow Memory Bandwidth Allocation (SMBA)
---------------------------------------
AMD hardware supports Slow Memory Bandwidth Allocation (SMBA).
CXL.memory is the only supported "slow" memory device. With the
support of SMBA, the hardware enables bandwidth allocation on
the slow memory devices. If there are multiple such devices in
the system, the throttling logic groups all the slow sources
together and applies the limit on them as a whole.

The presence of SMBA (with CXL.memory) is independent of slow memory
devices presence. If there are no such devices on the system, then
configuring SMBA will have no impact on the performance of the system.

The bandwidth domain for slow memory is L3 cache. Its schemata file
is formatted as:
::

        SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...

Reading/writing the schemata file
---------------------------------
Reading the schemata file will show the state of all resources
on all domains. When writing you only need to specify those values
which you wish to change.  E.g.
::

  # cat schemata
  L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
  L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
  # echo "L3DATA:2=3c0;" > schemata
  # cat schemata
  L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
  L3CODE:0=fffff;1=fffff;2=fffff;3=fffff

Reading/writing the schemata file (on AMD systems)
--------------------------------------------------
Reading the schemata file will show the current bandwidth limit on all
domains. The allocated resources are in multiples of one eighth GB/s.
When writing to the file, you need to specify what cache id you wish to
configure the bandwidth limit.

For example, to allocate 2GB/s limit on the first cache id:

::

  # cat schemata
    MB:0=2048;1=2048;2=2048;3=2048
    L3:0=ffff;1=ffff;2=ffff;3=ffff

  # echo "MB:1=16" > schemata
  # cat schemata
    MB:0=2048;1=  16;2=2048;3=2048
    L3:0=ffff;1=ffff;2=ffff;3=ffff

Reading/writing the schemata file (on AMD systems) with SMBA feature
--------------------------------------------------------------------
Reading and writing the schemata file is the same as without SMBA in
above section.

For example, to allocate 8GB/s limit on the first cache id:

::

  # cat schemata
    SMBA:0=2048;1=2048;2=2048;3=2048
      MB:0=2048;1=2048;2=2048;3=2048
      L3:0=ffff;1=ffff;2=ffff;3=ffff

  # echo "SMBA:1=64" > schemata
  # cat schemata
    SMBA:0=2048;1=  64;2=2048;3=2048
      MB:0=2048;1=2048;2=2048;3=2048
      L3:0=ffff;1=ffff;2=ffff;3=ffff

Cache pseudo-locking 원리

CAT는 application이 채울 수 있는 cache 공간을 제한한다. CPU는 현재 할당 영역 밖에 미리 들어간 데이터도 cache hit이면 읽고 쓸 수 있다. Pseudo-locking은 어느 application도 채울 수 없는 cache 부분에 데이터를 preload하고 이후 cache hit로만 제공해 평균 read latency가 낮은 memory 영역을 userspace에 mapping한다.

사용자가 pseudo-lock할 영역의 schemata와 함께 요청하면 먼저 일치하는 CBM의 새 CAT allocation `CLOSNEW`를 만든다. 이 영역은 현재 어느 CLOS와도 겹치면 안 되고 region이 존재하는 동안 미래에도 겹침을 허용하지 않는다.

Pseudo-locked CBM은 이후 어떤 CAT allocation에도 나타나지 않아 보호된다. 어느 CLOS에서 실행하는 application도 cache hit를 통해 이 memory에 접근할 수 있다.

Pseudo-locking은 cache 잔류 확률을 높일 뿐 배치를 보장하지 않는다. `INVD`, `WBINVD`, `CLFLUSH` 같은 명령은 데이터를 evict할 수 있고 C-state는 cache를 축소하거나 끌 수 있다. Region 생성 때 더 깊은 C-state를 자동 제한한다.

Application은 region이 있는 cache와 연결된 core 또는 그 subset에 affinity를 둬야 한다. 초기 `mmap()` 때 sanity check로 잘못된 affinity를 거부하지만 이후에는 강제하지 않으므로 application이 올바른 core에 계속 머물 책임이 있다.

두 단계 중 관리자는 cache 일부를 전용으로 할당하고 같은 크기 memory를 적재해 character device로 노출한다. Userspace application은 둘째 단계에서 그 device를 `mmap()`한다.

Cache Pseudo-Locking
====================
CAT enables a user to specify the amount of cache space that an
application can fill. Cache pseudo-locking builds on the fact that a
CPU can still read and write data pre-allocated outside its current
allocated area on a cache hit. With cache pseudo-locking, data can be
preloaded into a reserved portion of cache that no application can
fill, and from that point on will only serve cache hits. The cache
pseudo-locked memory is made accessible to user space where an
application can map it into its virtual address space and thus have
a region of memory with reduced average read latency.

The creation of a cache pseudo-locked region is triggered by a request
from the user to do so that is accompanied by a schemata of the region
to be pseudo-locked. The cache pseudo-locked region is created as follows:

- Create a CAT allocation CLOSNEW with a CBM matching the schemata
  from the user of the cache region that will contain the pseudo-locked
  memory. This region must not overlap with any current CAT allocation/CLOS
  on the system and no future overlap with this cache region is allowed
  while the pseudo-locked region exists.
- Create a contiguous region of memory of the same size as the cache
  region.
- Flush the cache, disable hardware prefetchers, disable preemption.
- Make CLOSNEW the active CLOS and touch the allocated memory to load
  it into the cache.
- Set the previous CLOS as active.
- At this point the closid CLOSNEW can be released - the cache
  pseudo-locked region is protected as long as its CBM does not appear in
  any CAT allocation. Even though the cache pseudo-locked region will from
  this point on not appear in any CBM of any CLOS an application running with
  any CLOS will be able to access the memory in the pseudo-locked region since
  the region continues to serve cache hits.
- The contiguous region of memory loaded into the cache is exposed to
  user-space as a character device.

Cache pseudo-locking increases the probability that data will remain
in the cache via carefully configuring the CAT feature and controlling
application behavior. There is no guarantee that data is placed in
cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
“locked” data from cache. Power management C-states may shrink or
power off cache. Deeper C-states will automatically be restricted on
pseudo-locked region creation.

It is required that an application using a pseudo-locked region runs
with affinity to the cores (or a subset of the cores) associated
with the cache on which the pseudo-locked region resides. A sanity check
within the code will not allow an application to map pseudo-locked memory
unless it runs with affinity to cores associated with the cache on which the
pseudo-locked region resides. The sanity check is only done during the
initial mmap() handling, there is no enforcement afterwards and the
application self needs to ensure it remains affine to the correct cores.

Pseudo-locking is accomplished in two stages:

1) During the first stage the system administrator allocates a portion
   of cache that should be dedicated to pseudo-locking. At this time an
   equivalent portion of memory is allocated, loaded into allocated
   cache portion, and exposed as a character device.
2) During the second stage a user-space application maps (mmap()) the
   pseudo-locked memory into its address space.

Pseudo-locking 생성·debug 인터페이스

Pseudo-locked region 생성은 `/sys/fs/resctrl`에 resource group 디렉터리를 만들고, `mode`에 `pseudo-locksetup`을 쓴 다음, `bit_usage`에서 모두 미사용인 bit로 `schemata`를 쓰는 세 단계다.

성공하면 mode가 `pseudo-locked`로 바뀌고 `/dev/pseudo_lock`에 group과 같은 이름의 character device가 생긴다. Userspace는 이 device를 `mmap()`해 region에 접근한다.

`CONFIG_DEBUG_FS`가 활성화되면 `/sys/kernel/debug/resctrl`에 pseudo-lock debug 인터페이스가 기본 제공된다. Kernel은 임의 memory 위치가 cache에 있는지 직접 검사할 방법이 없어 tracing으로 residency를 측정한다.

Latency test는 hardware prefetcher와 preemption을 끄고 32-byte stride로 region을 순회하며 cache hit/miss의 대체 시각화도 제공한다. L2/L3 측정은 platform의 model-specific precision counter가 있을 때 사용한다.

Region 생성 시 `/sys/kernel/debug/resctrl/<newdir>/pseudo_lock_measure`가 생긴다. 측정 전에 관련 tracepoint를 활성화해야 모든 결과가 tracing infrastructure에 기록된다.

Cache Pseudo-Locking Interface
------------------------------
A pseudo-locked region is created using the resctrl interface as follows:

1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
2) Change the new resource group's mode to "pseudo-locksetup" by writing
   "pseudo-locksetup" to the "mode" file.
3) Write the schemata of the pseudo-locked region to the "schemata" file. All
   bits within the schemata should be "unused" according to the "bit_usage"
   file.

On successful pseudo-locked region creation the "mode" file will contain
"pseudo-locked" and a new character device with the same name as the resource
group will exist in /dev/pseudo_lock. This character device can be mmap()'ed
by user space in order to obtain access to the pseudo-locked memory region.

An example of cache pseudo-locked region creation and usage can be found below.

Cache Pseudo-Locking Debugging Interface
----------------------------------------
The pseudo-locking debugging interface is enabled by default (if
CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl.

There is no explicit way for the kernel to test if a provided memory
location is present in the cache. The pseudo-locking debugging interface uses
the tracing infrastructure to provide two ways to measure cache residency of
the pseudo-locked region:

1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data
   from these measurements are best visualized using a hist trigger (see
   example below). In this test the pseudo-locked region is traversed at
   a stride of 32 bytes while hardware prefetchers and preemption
   are disabled. This also provides a substitute visualization of cache
   hits and misses.
2) Cache hit and miss measurements using model specific precision counters if
   available. Depending on the levels of cache on the system the pseudo_lock_l2
   and pseudo_lock_l3 tracepoints are available.

When a pseudo-locked region is created a new debugfs directory is created for
it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single
write-only file, pseudo_lock_measure, is present in this directory. The
measurement of the pseudo-locked region depends on the number written to this
debugfs file:

1:
     writing "1" to the pseudo_lock_measure file will trigger the latency
     measurement captured in the pseudo_lock_mem_latency tracepoint. See
     example below.
2:
     writing "2" to the pseudo_lock_measure file will trigger the L2 cache
     residency (cache hits and misses) measurement captured in the
     pseudo_lock_l2 tracepoint. See example below.
3:
     writing "3" to the pseudo_lock_measure file will trigger the L3 cache
     residency (cache hits and misses) measurement captured in the
     pseudo_lock_l3 tracepoint.

All measurements are recorded with the tracing infrastructure. This requires
the relevant tracepoints to be enabled before the measurement is triggered.

Pseudo-lock latency·hit/miss 예제

`newlock` region의 latency는 trace를 비우고 `pseudo_lock_mem_latency`에 `hist:keys=latency` trigger를 설정한 뒤 event를 활성화하고 `pseudo_lock_measure`에 1을 쓰는 순서로 측정한다. 이후 event를 끄고 histogram을 읽는다.

:> /sys/kernel/tracing/trace
echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist

예시 histogram은 총 8192 hit를 9개 latency bucket으로 나누며 38 cycle 3484회, 40 cycle 3204회가 대부분이고 456 cycle 1회 같은 긴 지연도 보여 준다.

L2 cache hit/miss는 `pseudo_lock_l2` tracepoint를 활성화하고 measure 파일에 2를 쓴 뒤 trace를 읽는다. 예시는 `hits=4097 miss=0`을 기록한다. L3에서는 대응 tracepoint와 값 3을 사용한다.

:> /sys/kernel/tracing/trace
echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
cat /sys/kernel/tracing/trace

Example of latency debugging interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example a pseudo-locked region named "newlock" was created. Here is
how we can measure the latency in cycles of reading from this region and
visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS
is set::

  # :> /sys/kernel/tracing/trace
  # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
  # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
  # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
  # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
  # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist

  # event histogram
  #
  # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active]
  #

  { latency:        456 } hitcount:          1
  { latency:         50 } hitcount:         83
  { latency:         36 } hitcount:         96
  { latency:         44 } hitcount:        174
  { latency:         48 } hitcount:        195
  { latency:         46 } hitcount:        262
  { latency:         42 } hitcount:        693
  { latency:         40 } hitcount:       3204
  { latency:         38 } hitcount:       3484

  Totals:
      Hits: 8192
      Entries: 9
    Dropped: 0

Example of cache hits/misses debugging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example a pseudo-locked region named "newlock" was created on the L2
cache of a platform. Here is how we can obtain details of the cache hits
and misses using the platform's precision counters.
::

  # :> /sys/kernel/tracing/trace
  # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
  # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
  # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
  # cat /sys/kernel/tracing/trace

  # tracer: nop
  #
  #                              _-----=> irqs-off
  #                             / _----=> need-resched
  #                            | / _---=> hardirq/softirq
  #                            || / _--=> preempt-depth
  #                            ||| /     delay
  #           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
  #              | |       |   ||||       |         |
  pseudo_lock_mea-1672  [002] ....  3132.860500: pseudo_lock_l2: hits=4097 miss=0

RDT allocation 예제

예제 1은 L3 mask가 4-bit인 2-socket 시스템에서 `p0`와 `p1`을 만든다. `p0`은 cache ID 0의 하위 50% mask `3`, ID 1의 상위 50% mask `c`; `p1`은 둘 다 하위 50% `3`을 사용한다. 두 group의 MBA는 socket마다 50%다.

mkdir p0 p1
echo "L3:0=3;1=c\nMB:0=50;1=50" > p0/schemata
echo "L3:0=3;1=3\nMB:0=50;1=50" > p1/schemata

Default group은 `L3:0=f;1=f`로 전체 cache를 유지한다. Memory bandwidth mask는 cache mask처럼 overlap 위치를 지정하지 않고 group이 사용할 수 있는 최대치만 정한다. `mba_sc`에서는 percentage 대신 socket 0에 1024MB, socket 1에 500MB처럼 절대값을 쓴다.

예제 2는 20-bit mask인 2-socket 시스템에서 PID 1234와 5678 실시간 task에 socket 0 L3의 각 25%를 독점적으로 준다. 먼저 default를 `L3:0=3ff;1=fffff`, `MB:0=50;1=100`으로 줄인다.

`p0`에는 `f8000`, `p1`에는 `7c00`을 주고 task ID를 `tasks`에 쓴 뒤 `taskset -cp`로 전용 CPU에 고정한다. MBA를 함께 쓰면 각 task group에 socket 0의 20%를 요청한다.

예제 3은 single-socket에서 core 4-7의 실시간 task와 core 0-3의 일반 workload를 분리한다. Task별 연결 대신 `p0/cpus`에 mask `F0`을 써 kernel과 해당 CPU의 task 모두 L3 상위 50% `ffc00`과 bandwidth 50%를 공유하게 한다.

예제 4는 8-bit L2 instance 둘에서 각 25%를 쓰는 exclusive group을 만든다. Default가 `ff` 전체를 쓰는 상태에서 `p0` mask `03`을 exclusive로 바꾸면 `schemata overlaps`로 실패한다.

echo 'L2:0=0xfc;1=0xfc' > schemata
echo exclusive > p0/mode
cat info/L2/bit_usage
# 0=SSSSSSEE;1=SSSSSSEE

Default를 `fc`로 줄인 뒤 `p0`을 exclusive로 만들면 성공하고 size는 instance마다 262144 byte다. 새 shareable `p1`은 exclusive `03`과 겹치지 않는 `fc`를 상속해 786432 byte를 얻는다. `bit_usage`는 `SSSSSSEE`로 공유·독점 구역을 표시한다. `p1`을 `01`로 바꿔 exclusive 영역과 겹치게 하면 `overlaps with exclusive group`으로 거부한다.

Examples for RDT allocation usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1) Example 1

On a two socket machine (one L3 cache per socket) with just four bits
for cache bit masks, minimum b/w of 10% with a memory bandwidth
granularity of 10%.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0 p1
  # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
  # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata

The default resource group is unmodified, so we have access to all parts
of all caches (its schemata file reads "L3:0=f;1=f").

Tasks that are under the control of group "p0" may only allocate from the
"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
Tasks in group "p1" use the "lower" 50% of cache on both sockets.

Similarly, tasks that are under the control of group "p0" may use a
maximum memory b/w of 50% on socket0 and 50% on socket 1.
Tasks in group "p1" may also use 50% memory b/w on both sockets.
Note that unlike cache masks, memory b/w cannot specify whether these
allocations can overlap or not. The allocations specifies the maximum
b/w that the group may be able to use and the system admin can configure
the b/w accordingly.

If resctrl is using the software controller (mba_sc) then user can enter the
max b/w in MB rather than the percentage values.
::

  # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata
  # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata

In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w
of 1024MB where as on socket 1 they would use 500MB.

2) Example 2

Again two sockets, but this time with a more realistic 20-bit mask.

Two real time tasks pid=1234 running on processor 0 and pid=5678 running on
processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
neighbors, each of the two real-time tasks exclusively occupies one quarter
of L3 cache on socket 0.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

First we reset the schemata for the default group so that the "upper"
50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
ordinary tasks::

  # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata

Next we make a resource group for our first real time task and give
it access to the "top" 25% of the cache on socket 0.
::

  # mkdir p0
  # echo "L3:0=f8000;1=fffff" > p0/schemata

Finally we move our first real time task into this resource group. We
also use taskset(1) to ensure the task always runs on a dedicated CPU
on socket 0. Most uses of resource groups will also constrain which
processors tasks run on.
::

  # echo 1234 > p0/tasks
  # taskset -cp 1 1234

Ditto for the second real time task (with the remaining 25% of cache)::

  # mkdir p1
  # echo "L3:0=7c00;1=fffff" > p1/schemata
  # echo 5678 > p1/tasks
  # taskset -cp 2 5678

For the same 2 socket system with memory b/w resource and CAT L3 the
schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
10):

For our first real time task this would request 20% memory b/w on socket 0.
::

  # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata

For our second real time task this would request an other 20% memory b/w
on socket 0.
::

  # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata

3) Example 3

A single socket system which has real-time tasks running on core 4-7 and
non real-time workload assigned to core 0-3. The real-time tasks share text
and data, so a per task association is not required and due to interaction
with the kernel it's desired that the kernel on these cores shares L3 with
the tasks.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

First we reset the schemata for the default group so that the "upper"
50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
cannot be used by ordinary tasks::

  # echo "L3:0=3ff\nMB:0=50" > schemata

Next we make a resource group for our real time cores and give it access
to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
socket 0.
::

  # mkdir p0
  # echo "L3:0=ffc00\nMB:0=50" > p0/schemata

Finally we move core 4-7 over to the new group and make sure that the
kernel and the tasks running there get 50% of the cache. They should
also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
siblings and only the real time threads are scheduled on the cores 4-7.
::

  # echo F0 > p0/cpus

4) Example 4

The resource groups in previous examples were all in the default "shareable"
mode allowing sharing of their cache allocations. If one resource group
configures a cache allocation then nothing prevents another resource group
to overlap with that allocation.

In this example a new exclusive resource group will be created on a L2 CAT
system with two L2 cache instances that can be configured with an 8-bit
capacity bitmask. The new exclusive resource group will be configured to use
25% of each cache instance.
::

  # mount -t resctrl resctrl /sys/fs/resctrl/
  # cd /sys/fs/resctrl

First, we observe that the default group is configured to allocate to all L2
cache::

  # cat schemata
  L2:0=ff;1=ff

We could attempt to create the new resource group at this point, but it will
fail because of the overlap with the schemata of the default group::

  # mkdir p0
  # echo 'L2:0=0x3;1=0x3' > p0/schemata
  # cat p0/mode
  shareable
  # echo exclusive > p0/mode
  -sh: echo: write error: Invalid argument
  # cat info/last_cmd_status
  schemata overlaps

To ensure that there is no overlap with another resource group the default
resource group's schemata has to change, making it possible for the new
resource group to become exclusive.
::

  # echo 'L2:0=0xfc;1=0xfc' > schemata
  # echo exclusive > p0/mode
  # grep . p0/*
  p0/cpus:0
  p0/mode:exclusive
  p0/schemata:L2:0=03;1=03
  p0/size:L2:0=262144;1=262144

A new resource group will on creation not overlap with an exclusive resource
group::

  # mkdir p1
  # grep . p1/*
  p1/cpus:0
  p1/mode:shareable
  p1/schemata:L2:0=fc;1=fc
  p1/size:L2:0=786432;1=786432

The bit_usage will reflect how the cache is used::

  # cat info/L2/bit_usage
  0=SSSSSSEE;1=SSSSSSEE

A resource group cannot be forced to overlap with an exclusive resource group::

  # echo 'L2:0=0x1;1=0x1' > p1/schemata
  -sh: echo: write error: Invalid argument
  # cat info/last_cmd_status
  overlaps with exclusive group

Cache pseudo-locking 전체 예제

예제는 cache ID 1의 L2에서 CBM `0x3`을 잠그고 `/dev/pseudo_lock/newlock`으로 노출한다. 먼저 `bit_usage`가 모두 `S`인지 확인하고 default schemata를 `L2:1=0xfc`로 줄여 하위 두 bit를 `0`으로 비운다.

cat info/L2/bit_usage
echo 'L2:1=0xfc' > schemata
mkdir newlock
echo pseudo-locksetup > newlock/mode
echo 'L2:1=0x3' > newlock/schemata

성공하면 `newlock/mode`는 `pseudo-locked`, cache ID 1 bit usage는 `SSSSSSPP`가 되며 character device가 생긴다.

Userspace 예제는 `_GNU_SOURCE`와 `sched.h`, `sys/mman.h` 등을 사용한다. Hard-coded CPU 2를 `CPU_SET`과 `sched_setaffinity()`로 지정해 region cache에 연결된 core에 고정한다.

그 뒤 `/dev/pseudo_lock/newlock`을 `O_RDWR`로 열고 system page size 한 page를 `PROT_READ | PROT_WRITE`, `MAP_SHARED`로 mmap한다. Application은 반환된 `mapping`으로 pseudo-locked memory를 사용하고 `munmap()`과 `close()`로 정리한다.

각 system call 실패는 `perror()` 뒤 `EXIT_FAILURE`로 종료하며, 정상 종료는 mapping 해제와 device close 뒤 `EXIT_SUCCESS`다.

Example of Cache Pseudo-Locking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
region is exposed at /dev/pseudo_lock/newlock that can be provided to
application for argument to mmap().
::

  # mount -t resctrl resctrl /sys/fs/resctrl/
  # cd /sys/fs/resctrl

Ensure that there are bits available that can be pseudo-locked, since only
unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
removed from the default resource group's schemata::

  # cat info/L2/bit_usage
  0=SSSSSSSS;1=SSSSSSSS
  # echo 'L2:1=0xfc' > schemata
  # cat info/L2/bit_usage
  0=SSSSSSSS;1=SSSSSS00

Create a new resource group that will be associated with the pseudo-locked
region, indicate that it will be used for a pseudo-locked region, and
configure the requested pseudo-locked region capacity bitmask::

  # mkdir newlock
  # echo pseudo-locksetup > newlock/mode
  # echo 'L2:1=0x3' > newlock/schemata

On success the resource group's mode will change to pseudo-locked, the
bit_usage will reflect the pseudo-locked region, and the character device
exposing the pseudo-locked region will exist::

  # cat newlock/mode
  pseudo-locked
  # cat info/L2/bit_usage
  0=SSSSSSSS;1=SSSSSSPP
  # ls -l /dev/pseudo_lock/newlock
  crw------- 1 root root 243, 0 Apr  3 05:01 /dev/pseudo_lock/newlock

::

  /*
  * Example code to access one page of pseudo-locked cache region
  * from user space.
  */
  #define _GNU_SOURCE
  #include <fcntl.h>
  #include <sched.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <unistd.h>
  #include <sys/mman.h>

  /*
  * It is required that the application runs with affinity to only
  * cores associated with the pseudo-locked region. Here the cpu
  * is hardcoded for convenience of example.
  */
  static int cpuid = 2;

  int main(int argc, char *argv[])
  {
    cpu_set_t cpuset;
    long page_size;
    void *mapping;
    int dev_fd;
    int ret;

    page_size = sysconf(_SC_PAGESIZE);

    CPU_ZERO(&cpuset);
    CPU_SET(cpuid, &cpuset);
    ret = sched_setaffinity(0, sizeof(cpuset), &cpuset);
    if (ret < 0) {
      perror("sched_setaffinity");
      exit(EXIT_FAILURE);
    }

    dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR);
    if (dev_fd < 0) {
      perror("open");
      exit(EXIT_FAILURE);
    }

    mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
            dev_fd, 0);
    if (mapping == MAP_FAILED) {
      perror("mmap");
      close(dev_fd);
      exit(EXIT_FAILURE);
    }

    /* Application interacts with pseudo-locked memory @mapping */

    ret = munmap(mapping, page_size);
    if (ret < 0) {
      perror("munmap");
      close(dev_fd);
      exit(EXIT_FAILURE);
    }

    close(dev_fd);
    exit(EXIT_SUCCESS);
  }

애플리케이션 사이 advisory locking

여러 resctrl 파일의 read/write로 구성된 작업은 원자적이어야 한다. 예를 들어 exclusive L3 예약은 모든 CBM 또는 `bit_usage`를 읽고, 어디에도 쓰이지 않는 연속 bit를 찾고, 새 디렉터리를 만들고, 그 bit를 새 `schemata`에 쓰는 네 단계다.

두 application이 동시에 실행하면 같은 bit를 골라 exclusive가 아니라 공유 예약을 만들 수 있다. 이를 막으려면 libc와 shell에서 사용할 수 있는 `flock`으로 `/sys/fs/resctrl` 자체를 잠근다.

flock -s /sys/fs/resctrl/ find /sys/fs/resctrl
flock /sys/fs/resctrl/ ./create-dir.sh

Shell 예제는 shared lock으로 원자적 구조 snapshot을 읽고, exclusive lock 아래 script에서 mask 계산·directory 생성·schemata write를 수행한다.

C 예제는 `/sys/fs/resctrl`을 `O_DIRECTORY`로 열고 `flock(fd, LOCK_SH)`, `flock(fd, LOCK_EX)`, `flock(fd, LOCK_UN)`을 각각 shared 획득, exclusive 획득, 해제 helper로 감싼다. Read-only 작업과 read/write 작업에 맞는 lock을 선택한다.

이는 kernel이 강제하는 transaction lock이 아니라 협력 application 사이의 advisory protocol이므로 모든 관리 도구가 같은 규약을 지켜야 한다.

Locking between applications
----------------------------

Certain operations on the resctrl filesystem, composed of read/writes
to/from multiple files, must be atomic.

As an example, the allocation of an exclusive reservation of L3 cache
involves:

  1. Read the cbmmasks from each directory or the per-resource "bit_usage"
  2. Find a contiguous set of bits in the global CBM bitmask that is clear
     in any of the directory cbmmasks
  3. Create a new directory
  4. Set the bits found in step 2 to the new directory "schemata" file

If two applications attempt to allocate space concurrently then they can
end up allocating the same bits so the reservations are shared instead of
exclusive.

To coordinate atomic operations on the resctrlfs and to avoid the problem
above, the following locking procedure is recommended:

Locking is based on flock, which is available in libc and also as a shell
script command

Write lock:

 A) Take flock(LOCK_EX) on /sys/fs/resctrl
 B) Read/write the directory structure.
 C) funlock

Read lock:

 A) Take flock(LOCK_SH) on /sys/fs/resctrl
 B) If success read the directory structure.
 C) funlock

Example with bash::

  # Atomically read directory structure
  $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl

  # Read directory contents and create new subdirectory

  $ cat create-dir.sh
  find /sys/fs/resctrl/ > output.txt
  mask = function-of(output.txt)
  mkdir /sys/fs/resctrl/newres/
  echo mask > /sys/fs/resctrl/newres/schemata

  $ flock /sys/fs/resctrl/ ./create-dir.sh

Example with C::

  /*
  * Example code do take advisory locks
  * before accessing resctrl filesystem
  */
  #include <sys/file.h>
  #include <stdlib.h>

  void resctrl_take_shared_lock(int fd)
  {
    int ret;

    /* take shared lock on resctrl filesystem */
    ret = flock(fd, LOCK_SH);
    if (ret) {
      perror("flock");
      exit(-1);
    }
  }

  void resctrl_take_exclusive_lock(int fd)
  {
    int ret;

    /* release lock on resctrl filesystem */
    ret = flock(fd, LOCK_EX);
    if (ret) {
      perror("flock");
      exit(-1);
    }
  }

  void resctrl_release_lock(int fd)
  {
    int ret;

    /* take shared lock on resctrl filesystem */
    ret = flock(fd, LOCK_UN);
    if (ret) {
      perror("flock");
      exit(-1);
    }
  }

  void main(void)
  {
    int fd, ret;

    fd = open("/sys/fs/resctrl", O_DIRECTORY);
    if (fd == -1) {
      perror("open");
      exit(-1);
    }
    resctrl_take_shared_lock(fd);
    /* code to read directory contents */
    resctrl_release_lock(fd);

    resctrl_take_exclusive_lock(fd);
    /* code to read and write directory contents */
    resctrl_release_lock(fd);
  }

RDT monitoring 예제

Event 파일, 예를 들어 `mon_data/mon_L3_00/llc_occupancy`를 읽으면 해당 MON 또는 CTRL_MON group의 현재 LLC occupancy snapshot을 byte로 얻는다.

예제 1은 2-socket 4-bit CBM에서 `p0`과 `p1`을 만들고 PID 5678·5679를 `p1`에 넣는다. `p1/mon_groups` 아래 `m11`, `m12`를 만들어 각 task를 분리하면 domain별 occupancy를 따로 읽을 수 있다. 부모 `p1/mon_data`는 두 MON을 포함한 합계 31234000을 보여 준다.

mkdir p1/mon_groups/m11 p1/mon_groups/m12
echo 5678 > p1/mon_groups/m11/tasks
echo 5679 > p1/mon_groups/m12/tasks
cat p1/mon_groups/m11/mon_data/mon_L3_00/llc_occupancy
cat p1/mon_data/mon_L3_00/llc_occupancy

예제 2는 group 생성 시 RMID가 할당된다는 점을 이용한다. 현재 shell PID `$$`를 `p1/tasks`에 넣은 뒤 `<cmd>`를 실행하면 command가 생성되는 순간부터 관찰된다.

예제 3은 CAT 없이 CQM만 있는 HSW 같은 시스템에서도 resctrl이 마운트됨을 보여 준다. CTRL_MON은 만들 수 없지만 root `mon_groups` 아래 `m01`, `m02`를 만들어 kernel thread를 포함한 task를 관찰하고 allocation 전 cache footprint를 profiling할 수 있다. 예시 수치는 workload가 주로 domain 0에서 동작함을 보여 준다.

예제 4는 single-socket의 실시간 task가 실행되는 CPU 4-7을 `p1/cpus`에 mask `f0`으로 넣고 `p1/mon_data/mon_L3_00/llc_occupancy`에서 이 CPU group의 occupancy를 읽는다.

Examples for RDT Monitoring along with allocation usage
=======================================================
Reading monitored data
----------------------
Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would
show the current snapshot of LLC occupancy of the corresponding MON
group or CTRL_MON group.


Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group)
------------------------------------------------------------------------
On a two socket machine (one L3 cache per socket) with just four bits
for cache bit masks::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0 p1
  # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
  # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
  # echo 5678 > p1/tasks
  # echo 5679 > p1/tasks

The default resource group is unmodified, so we have access to all parts
of all caches (its schemata file reads "L3:0=f;1=f").

Tasks that are under the control of group "p0" may only allocate from the
"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
Tasks in group "p1" use the "lower" 50% of cache on both sockets.

Create monitor groups and assign a subset of tasks to each monitor group.
::

  # cd /sys/fs/resctrl/p1/mon_groups
  # mkdir m11 m12
  # echo 5678 > m11/tasks
  # echo 5679 > m12/tasks

fetch data (data shown in bytes)
::

  # cat m11/mon_data/mon_L3_00/llc_occupancy
  16234000
  # cat m11/mon_data/mon_L3_01/llc_occupancy
  14789000
  # cat m12/mon_data/mon_L3_00/llc_occupancy
  16789000

The parent ctrl_mon group shows the aggregated data.
::

  # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
  31234000

Example 2 (Monitor a task from its creation)
--------------------------------------------
On a two socket machine (one L3 cache per socket)::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0 p1

An RMID is allocated to the group once its created and hence the <cmd>
below is monitored from its creation.
::

  # echo $$ > /sys/fs/resctrl/p1/tasks
  # <cmd>

Fetch the data::

  # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
  31789000

Example 3 (Monitor without CAT support or before creating CAT groups)
---------------------------------------------------------------------

Assume a system like HSW has only CQM and no CAT support. In this case
the resctrl will still mount but cannot create CTRL_MON directories.
But user can create different MON groups within the root group thereby
able to monitor all tasks including kernel threads.

This can also be used to profile jobs cache size footprint before being
able to allocate them to different allocation groups.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir mon_groups/m01
  # mkdir mon_groups/m02

  # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks
  # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks

Monitor the groups separately and also get per domain data. From the
below its apparent that the tasks are mostly doing work on
domain(socket) 0.
::

  # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy
  31234000
  # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy
  34555
  # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy
  31234000
  # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy
  32789


Example 4 (Monitor real time tasks)
-----------------------------------

A single socket system which has real time tasks running on cores 4-7
and non real time tasks on other cpus. We want to monitor the cache
occupancy of the real time threads on these cores.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p1

Move the cpus 4-7 over to p1::

  # echo f0 > p1/cpus

View the llc occupancy snapshot::

  # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy
  11234000

`mbm_assign_mode` 운용 예제

먼저 `info/L3_MON/mbm_assign_mode`에서 `[mbm_event]`가 표시되는지 확인하고, `num_mbm_cntrs`와 `available_mbm_cntrs`에서 domain별 최대·가용 counter를 읽는다. 예시는 32개 중 30개가 할당 가능하다.

Root group의 `mbm_L3_assignments`는 `mbm_total_bytes:0=e;1=e`처럼 event와 domain별 exclusive 할당을 보여 준다. `0=_`는 domain 0 해제, `*=_`는 모든 domain 해제, `*=e`는 모든 domain exclusive 할당이다.

할당 여부와 관계없이 event file 읽는 방법은 같다. 예제는 두 domain의 total과 local byte 값을 읽는다.

각 event의 `event_filter`에서 transaction 구성을 확인하고 `mbm_local_bytes` filter에 `remote_reads` 같은 항목을 추가할 수 있다. 구성을 바꾼 직후 첫 read는 counter reset 때문에 `Unavailable`, 다음 read는 현재 값을 반환할 수 있다.

필요하면 `mbm_assign_mode`에 `default`를 써 돌아간다. Mode 전환은 모든 resctrl group의 MBM counter와 event를 reset할 수 있다. 마지막에는 resctrl을 언마운트한다.

cat info/L3_MON/mbm_assign_mode
cat info/L3_MON/num_mbm_cntrs
cat info/L3_MON/available_mbm_cntrs
echo "mbm_total_bytes:0=_" > mbm_L3_assignments
echo "mbm_total_bytes:*=e" > mbm_L3_assignments
echo "default" > info/L3_MON/mbm_assign_mode
umount /sys/fs/resctrl/

Examples on working with mbm_assign_mode
========================================

a. Check if MBM counter assignment mode is supported.
::

  # mount -t resctrl resctrl /sys/fs/resctrl/

  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
  [mbm_event]
  default

The "mbm_event" mode is detected and enabled.

b. Check how many assignable counters are supported.
::

  # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
  0=32;1=32

c. Check how many assignable counters are available for assignment in each domain.
::

  # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
  0=30;1=30

d. To list the default group's assign states.
::

  # cat /sys/fs/resctrl/mbm_L3_assignments
  mbm_total_bytes:0=e;1=e
  mbm_local_bytes:0=e;1=e

e.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
::

  # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
  # cat /sys/fs/resctrl/mbm_L3_assignments
  mbm_total_bytes:0=_;1=e
  mbm_local_bytes:0=e;1=e

f. To unassign the counter associated with the mbm_total_bytes event on all domains.
::

  # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
  # cat /sys/fs/resctrl/mbm_L3_assignment
  mbm_total_bytes:0=_;1=_
  mbm_local_bytes:0=e;1=e

g. To assign a counter associated with the mbm_total_bytes event on all domains in
exclusive mode.
::

  # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
  # cat /sys/fs/resctrl/mbm_L3_assignments
  mbm_total_bytes:0=e;1=e
  mbm_local_bytes:0=e;1=e

h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
no change in reading the events with the assignment.
::

  # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
  779247936
  # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_total_bytes
  562324232
  # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
  212122123
  # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
  121212144

i. Check the event configurations.
::

  # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
  local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,
  local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all

  # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
  local_reads,local_non_temporal_writes,local_reads_slow_memory

j. Change the event configuration for mbm_local_bytes.
::

  # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" >
  /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter

  # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
  local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads

k. Now read the local events again. The first read may come back with "Unavailable"
status. The subsequent read of mbm_local_bytes will display the current value.
::

  # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
  Unavailable
  # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
  2252323
  # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
  Unavailable
  # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
  1566565

l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
done using the following command. Note that switching the mbm_assign_mode may reset all
the MBM counters (and thus all MBM events) of all the resctrl groups.
::

  # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
  mbm_event
  [default]

m. Unmount the resctrl filesystem.
::

  # umount /sys/fs/resctrl/

Intel MBM counter errata 보정

Skylake server의 SKX99와 Broadwell server의 BDF102 errata 때문에 Intel MBM counter가 특정 RMID에서 system memory bandwidth를 잘못 보고할 수 있다. Logical core에 할당된 RMID 기준 metric을 보고하는 `IA32_QM_CTR` register는 MSR `0xC8E`다.

이 문제로 실제 system bandwidth와 보고값이 일치하지 않을 수 있다. RMID가 표의 threshold보다 크면 MBM total과 local 값에 해당 correction factor를 곱해 보정한다.

원문은 Intel Xeon Scalable Family specification update의 SKX99, Xeon E5-2600 v4 update의 BDF102, 2세대 Xeon Scalable용 Intel RDT reference manual 링크를 추가 근거로 제공한다.

Intel RDT Errata
================

Intel MBM Counters May Report System Memory Bandwidth Incorrectly
-----------------------------------------------------------------

Errata SKX99 for Skylake server and BDF102 for Broadwell server.

Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics
according to the assigned Resource Monitor ID (RMID) for that logical
core. The IA32_QM_CTR register (MSR 0xC8E), used to report these
metrics, may report incorrect system bandwidth for certain RMID values.

Implication: Due to the errata, system memory bandwidth may not match
what is reported.

Workaround: MBM total and local readings are corrected according to the
following correction factor table:

+---------------+---------------+---------------+-----------------+
|core count        |rmid count        |rmid threshold        |correction factor|
+---------------+---------------+---------------+-----------------+
|1                |8                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|2                |16                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|3                |24                |15                |0.969650          |
+---------------+---------------+---------------+-----------------+
|4                |32                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|6                |48                |31                |0.969650          |
+---------------+---------------+---------------+-----------------+
|7                |56                |47                |1.142857          |
+---------------+---------------+---------------+-----------------+
|8                |64                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|9                |72                |63                |1.185115          |
+---------------+---------------+---------------+-----------------+
|10                |80                |63                |1.066553          |
+---------------+---------------+---------------+-----------------+
|11                |88                |79                |1.454545          |
+---------------+---------------+---------------+-----------------+
|12                |96                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|13                |104                |95                |1.230769          |
+---------------+---------------+---------------+-----------------+
|14                |112                |95                |1.142857          |
+---------------+---------------+---------------+-----------------+
|15                |120                |95                |1.066667          |
+---------------+---------------+---------------+-----------------+
|16                |128                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|17                |136                |127                |1.254863          |
+---------------+---------------+---------------+-----------------+
|18                |144                |127                |1.185255          |
+---------------+---------------+---------------+-----------------+
|19                |152                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|20                |160                |127                |1.066667          |
+---------------+---------------+---------------+-----------------+
|21                |168                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|22                |176                |159                |1.454334          |
+---------------+---------------+---------------+-----------------+
|23                |184                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|24                |192                |127                |0.969744          |
+---------------+---------------+---------------+-----------------+
|25                |200                |191                |1.280246          |
+---------------+---------------+---------------+-----------------+
|26                |208                |191                |1.230921          |
+---------------+---------------+---------------+-----------------+
|27                |216                |0                |1.000000          |
+---------------+---------------+---------------+-----------------+
|28                |224                |191                |1.143118          |
+---------------+---------------+---------------+-----------------+

If rmid > rmid threshold, MBM total and local values should be multiplied
by the correction factor.

See:

1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html

2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf

3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html

for further information.