Scrub Control

ECC memory scrubbing과 공통 제어

DRAM 용량과 비용이 증가하면서 메모리 subsystem reliability가 중요한 문제가 되었습니다. 손상된 데이터가 큰 비용이나 치명적 장애를 일으킬 수 있는 시스템에서 메모리 오류는 서버와 workload crash를 유발하는 주요 hardware failure입니다.

Memory scrubbing은 ECC engine이 메모리 미디어의 각 위치를 읽고 필요한 경우 오류를 교정한 뒤 같은 위치에 수정된 데이터를 다시 쓰는 기능입니다.

구성 가능한 속도로 DIMM을 scrub하면 correctable error가 uncorrectable error로 발전할 가능성을 줄입니다. 할당되지 않은 page에서 발견한 uncorrected error를 격리해 application이나 OS가 할당받지 못하게 하고, software와 hardware가 오류를 만날 가능성을 낮춥니다.

Scrub에서 얻은 failure 통계는 나중에 Post Package Repair나 sparing 같은 memory repair 기술을 사용할지 결정하는 자료가 됩니다.

Scrubbing에는 DRAM이 idle할 때 실행하는 background 또는 patrol scrub과 특정 주소 범위·메모리 영역을 대상으로 하는 on-demand scrub이 있습니다. CXL patrol scrub, CXL DDR5 ECS, ACPI RAS2, ACPI NVDIMM ARS, software scrubber 등 제어 방식이 서로 달라 userspace tool을 위한 표준 ABI가 필요합니다.

Generic EDAC scrub control은 다양한 hardware scrubber의 관리 기능을 통일된 함수와 sysfs control interface로 추상화합니다.

.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.2-no-invariants-or-later

=============
Scrub Control
=============

Copyright (c) 2024-2025 HiSilicon Limited.

:Author:   Shiju Jose <shiju.jose@huawei.com>
:License:  The GNU Free Documentation License, Version 1.2 without
           Invariant Sections, Front-Cover Texts nor Back-Cover Texts.
           (dual licensed under the GPL v2)

- Written for: 6.15

Introduction
------------

Increasing DRAM size and cost have made memory subsystem reliability an
important concern. These modules are used where potentially corrupted data
could cause expensive or fatal issues. Memory errors are among the top
hardware failures that cause server and workload crashes.

Memory scrubbing is a feature where an ECC (Error-Correcting Code) engine
reads data from each memory media location, corrects if necessary and writes
the corrected data back to the same memory media location.

DIMMs can be scrubbed at a configurable rate to detect uncorrected memory
errors and attempt recovery from detected errors, providing the following
benefits:

1. Proactively scrubbing DIMMs reduces the chance of a correctable error
   becoming uncorrectable.

2. When detected, uncorrected errors caught in unallocated memory pages are
   isolated and prevented from being allocated to an application or the OS.

3. This reduces the likelihood of software or hardware products encountering
   memory errors.

4. The additional data on failures in memory may be used to build up
   statistics that are later used to decide whether to use memory repair
   technologies such as Post Package Repair or Sparing.

There are 2 types of memory scrubbing:

1. Background (patrol) scrubbing while the DRAM is otherwise idle.

2. On-demand scrubbing for a specific address range or region of memory.

Several types of interfaces to hardware memory scrubbers have been
identified, such as CXL memory device patrol scrub, CXL DDR5 ECS, ACPI
RAS2 memory scrubbing, and ACPI NVDIMM ARS (Address Range Scrub).

The control mechanisms vary across different memory scrubbers. To enable
standardized userspace tooling, there is a need to present these controls
through a standardized ABI.

A generic memory EDAC scrub control allows users to manage underlying
scrubbers in the system through a standardized sysfs control interface.  It
abstracts the management of various scrubbing functionalities into a unified
set of functions.

공통 scrub control의 사용 사례

Hardware scrubber에는 CXL patrol scrub, CXL DDR5 ECS, ACPI RAS2, ACPI NVDIMM ARS와 software scrubber가 있습니다. 일부는 background 제어를, 일부는 on-demand 제어를 지원하지만 인터페이스가 달라 공통 sysfs scrub control이 필요합니다.

Userspace는 성능에 민감한 작업 중 background activity를 줄이기 위해 patrol scrubbing을 끄거나 scrub rate를 조정할 수 있습니다. Scrubber가 지원한다면 특정 주소 범위를 on-demand로 scrub할 수도 있습니다.

구성 가능한 DIMM scrub rate는 사용자 접근 전에 uncorrectable error를 조기에 발견해 복구를 돕고, correctable error가 uncorrectable 상태로 발전할 가능성을 줄입니다.

부팅 뒤 hotplug된 CXL memory는 시스템 전체 BIOS 설정이 관리하지 못할 수 있습니다. Reliability와 performance 사이의 정책 판단은 userspace가 내려야 하므로 별도 인터페이스를 만들기보다 다른 scrubber와 같은 통합 제어를 사용하는 것이 권장됩니다.

Use cases of common scrub control feature
-----------------------------------------

1. Several types of interfaces for hardware memory scrubbers have been
   identified, including the CXL memory device patrol scrub, CXL DDR5 ECS,
   ACPI RAS2 memory scrubbing features, ACPI NVDIMM ARS (Address Range Scrub),
   and software-based memory scrubbers.

   Of the identified interfaces to hardware memory scrubbers some support
   control over patrol (background) scrubbing (e.g., ACPI RAS2, CXL) and/or
   on-demand scrubbing (e.g., ACPI RAS2, ACPI ARS). However, the scrub control
   interfaces vary between memory scrubbers, highlighting the need for
   a standardized, generic sysfs scrub control interface that is accessible to
   userspace for administration and use by scripts/tools.

2. User-space scrub controls allow users to disable scrubbing if necessary,
   for example, to disable background patrol scrubbing or adjust the scrub
   rate for performance-aware operations where background activities need to
   be minimized or disabled.

3. User-space tools enable on-demand scrubbing for specific address ranges,
   provided that the scrubber supports this functionality.

4. User-space tools can also control memory DIMM scrubbing at a configurable
   scrub rate via sysfs scrub controls. This approach offers several benefits:

   4.1. Detects uncorrectable memory errors early, before user access to affected
        memory, helping facilitate recovery.

   4.2. Reduces the likelihood of correctable errors developing into uncorrectable
        errors.

5. Policy control for hotplugged memory is necessary because there may not
   be a system-wide BIOS or similar control to manage scrub settings for a CXL
   device added after boot. Determining these settings is a policy decision,
   balancing reliability against performance, so userspace should control it.
   Therefore, a unified interface is recommended for handling this function in
   a way that aligns with other similar interfaces, rather than creating a
   separate one.

CXL memory patrol scrub

CXL 3.1 규격 8.2.9.9.11.1은 memory device patrol scrub control을 정의합니다. Patrol scrub은 일정 주기로 메모리를 선제적으로 검사하고 오류를 교정합니다.

Userspace는 CXL patrol scrubber의 구성을 바꿀 수 있으며, patrol cycle을 완료해야 하는 시간 수를 지정합니다. 요청한 rate는 장치가 지원하는 범위 안에 있어야 합니다.

CXL 드라이버는 sysfs에서 요청한 cycle당 초 값을 장치가 사용하는 cycle당 시간 값으로 변환합니다. Performance-sensitive operation과 충돌할 때는 host가 background patrol 기능을 끌 수도 있습니다.

Scrubbing features
------------------

CXL Memory Scrubbing features
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CXL spec r3.1 [1]_ section 8.2.9.9.11.1 describes the memory device patrol
scrub control feature. The device patrol scrub proactively locates and makes
corrections to errors in regular cycle. The patrol scrub control allows the
userspace request to change CXL patrol scrubber's configurations.

The patrol scrub control allows the requester to specify the number of
hours in which the patrol scrub cycles must be completed, provided that
the requested scrub rate must be within the supported range of the
scrub rate that the device is capable of. In the CXL driver, the
number of seconds per scrub cycles, which user requests via sysfs, is
rescaled to hours per scrub cycles.

In addition, they allow the host to disable the feature in case it interferes
with performance-aware operations which require the background operations to
be turned off.

DDR5 Error Check Scrub

CXL 3.1 8.2.9.9.11.2와 JEDEC DDR5 SDRAM 규격 JESD79-5는 Error Check Scrub(ECS)를 정의합니다.

ECS는 DRAM 내부에서 데이터를 읽고 single-bit error를 교정한 뒤 수정된 bit를 DRAM array에 다시 쓰며, error count도 외부에 제공합니다.

DDR5 장치에는 여러 memory-media Field Replaceable Unit(FRU)이 있으므로 DDR5 ECS와 ECS control driver는 FRU별로 ECS parameter를 구성합니다.

Error Check Scrub (ECS)
~~~~~~~~~~~~~~~~~~~~~~~

CXL spec r3.1 [1]_ section 8.2.9.9.11.2 describes Error Check Scrub (ECS)
- a feature defined in the JEDEC DDR5 SDRAM Specification (JESD79-5) and
allowing DRAM to internally read, correct single-bit errors, and write back
corrected data bits to the DRAM array while providing transparency to error
counts.

The DDR5 device contains number of memory media Field Replaceable Units (FRU)
per device. The DDR5 ECS feature and thus the ECS control driver supports
configuring the ECS parameters per FRU.

ACPI RAS2 hardware scrub

ACPI 6.5 5.2.21은 platform RAS 기능의 interface를 제공하는 ACPI RAS2 table을 정의합니다. 같은 종류의 component가 여러 개 있는 시스템에서도 instance마다 독립된 RAS capability와 control을 제공합니다.

Memory-specific RAS2 PCC subspace의 Feature Type은 `0x00`(Memory)입니다. Platform은 이를 통해 hardware memory scrub engine의 control과 capability를 노출합니다.

RAS2 memory scrubbing은 proximity domain으로 식별한 NUMA domain별 독립 제어, 전체 메모리의 background patrol scrub, 특정 영역의 on-demand scrub을 지원합니다.

ACPI RAS2 Hardware-based Memory Scrubbing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ACPI spec 6.5 [2]_ section 5.2.21 ACPI RAS2 describes an ACPI RAS2 table
which provides interfaces for platform RAS features and supports independent
RAS controls and capabilities for a given RAS feature for multiple instances
of the same component in a given system.

Memory RAS features apply to RAS capabilities, controls and operations that
are specific to memory. RAS2 PCC sub-spaces for memory-specific RAS features
have a Feature Type of 0x00 (Memory).

The platform can use the hardware-based memory scrubbing feature to expose
controls and capabilities associated with hardware-based memory scrub
engines. The RAS2 memory scrubbing feature supports as per spec,

1. Independent memory scrubbing controls for each NUMA domain, identified
   using its proximity domain.

2. Provision for background (patrol) scrubbing of the entire memory system,
   as well as on-demand scrubbing for a specific region of memory.

ACPI Address Range Scrubbing

ACPI 6.5 9.19.7.2의 Address Range Scrubbing(ARS)은 platform이 memory error를 system software에 전달하고, software가 uncorrectable error가 있는 주소의 접근을 막을 수 있게 합니다.

ARS는 시스템의 모든 NVDIMM을 관리하며 시스템 전체에서 한 번에 하나의 scrub만 실행할 수 있습니다.

규격은 지정 범위의 ARS capability 조회, volatile·persistent memory의 ARS 시작, 진행률과 error record를 포함한 status 조회, uncorrectable error 제거, SPA 변환, ARS error injection 등을 지원합니다.

Kernel에는 기존 ARS control이 있지만 현재 ARS는 EDAC에서 지원하지 않습니다. 관련 규격은 CXL specification과 ACPI 6.5 문서에서 확인합니다.

ACPI Address Range Scrubbing (ARS)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ACPI spec 6.5 [2]_ section 9.19.7.2 describes Address Range Scrubbing (ARS).
ARS allows the platform to communicate memory errors to system software.
This capability allows system software to prevent accesses to addresses with
uncorrectable errors in memory. ARS functions manage all NVDIMMs present in
the system. Only one scrub can be in progress system wide at any given time.

The following functions are supported as per the specification:

1. Query ARS Capabilities for a given address range, indicates platform
   supports the ACPI NVDIMM Root Device Unconsumed Error Notification.

2. Start ARS triggers an Address Range Scrub for the given memory range.
   Address scrubbing can be done for volatile or persistent memory, or both.

3. Query ARS Status command allows software to get the status of ARS,
   including the progress of ARS and ARS error record.

4. Clear Uncorrectable Error.

5. Translate SPA

6. ARS Error Inject etc.

The kernel supports an existing control for ARS and ARS is currently not
supported in EDAC.

.. [1] https://computeexpresslink.org/cxl-specification/

.. [2] https://uefi.org/specs/ACPI/6.5/

Scrubbing 기능 비교

원문 표는 ACPI RAS2, CXL patrol scrub, CXL ECS, ARS의 on-demand·background 지원, 제어 단위, capability 조회, 주소 범위, rate, status, UC error reporting, EDAC 지원 여부를 비교합니다.

ACPI RAS2만 on-demand와 background를 모두 지원합니다. CXL patrol과 ECS는 background 방식이며, ARS는 on-demand 전용입니다. 주소 범위 설정은 RAS2와 ARS, scrub rate 설정은 RAS2와 CXL patrol에서 지원합니다.

모든 방식이 capability 조회를 지원하지만 EDAC control은 RAS2, CXL patrol, CXL ECS에서만 지원되고 ARS는 현재 제외됩니다.

Comparison of various scrubbing features
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 +--------------+-----------+-----------+-----------+-----------+
 |              |   ACPI    | CXL patrol|  CXL ECS  |  ARS      |
 |  Name        |   RAS2    | scrub     |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | On-demand    | Supported | No        | No        | Supported |
 | Scrubbing    |           |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Background   | Supported | Supported | Supported | No        |
 | scrubbing    |           |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Mode of      | Scrub ctrl| per device| per memory|  Unknown  |
 | scrubbing    | per NUMA  |           | media     |           |
 |              | domain.   |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Query scrub  | Supported | Supported | Supported | Supported |
 | capabilities |           |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Setting      | Supported | No        | No        | Supported |
 | address range|           |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Setting      | Supported | Supported | No        | No        |
 | scrub rate   |           |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Unit for     | Not       | in hours  | No        | No        |
 | scrub rate   | Defined   |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 |              | Supported |           |           |           |
 | Scrub        | on-demand | No        | No        | Supported |
 | status/      | scrubbing |           |           |           |
 | Completion   | only      |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+
 | UC error     |           |CXL general|CXL general| ACPI UCE  |
 | reporting    | Exception |media/DRAM |media/DRAM | notify and|
 |              |           |event/media|event/media| query     |
 |              |           |scan?      |scan?      | ARS status|
 +--------------+-----------+-----------+-----------+-----------+
 |              |           |           |           |           |
 | Support for  | Supported | Supported | Supported | No        |
 | EDAC control |           |           |           |           |
 |              |           |           |           |           |
 +--------------+-----------+-----------+-----------+-----------+

Scrubber sysfs 위치와 ABI

등록된 scrubber instance의 제어 속성은 `/sys/bus/edac/devices/<dev-name>/scrubX/`에서 접근합니다.

일반 scrub sysfs ABI는 `Documentation/ABI/testing/sysfs-edac-scrub`, ECS ABI는 `Documentation/ABI/testing/sysfs-edac-ecs`에 정의됩니다.

The File System
---------------

The control attributes of a registered scrubber instance could be
accessed in:

/sys/bus/edac/devices/<dev-name>/scrubX/

sysfs
-----

Sysfs files are documented in
`Documentation/ABI/testing/sysfs-edac-scrub`

`Documentation/ABI/testing/sysfs-edac-ecs`

CXL patrol과 device-based scrubbing

CXL patrol rate를 높이는 이유에는 특정 장치에서 예상보다 많은 오류가 발생하는 경우, 아직 online되지 않은 메모리에 부팅 기본값을 적용하는 경우, 특정 dataset에 더 높은 reliability가 필요하다고 monitor software가 판단한 경우가 있습니다. 마지막 방식은 Differentiated Reliability라고 부릅니다.

CXL memory는 CXL device를 통해 memory management subsystem과 userspace에 노출됩니다. 특정 장치의 오류율이 높은 첫 번째 사례에는 device-based scrubbing을 사용합니다.

Device interface와 region interface를 함께 제어할 때는 이어지는 region-based 규칙을 따라야 합니다. 실제 속성은 `Documentation/ABI/testing/sysfs-edac-scrub`에 정의됩니다.

Examples
--------

The usage takes the form shown in these examples:

1. CXL memory Patrol Scrub

The following are the use cases identified why we might increase the scrub rate.

- Scrubbing is needed at device granularity because a device is showing
  unexpectedly high errors.

- Scrubbing may apply to memory that isn't online at all yet. Likely this
  is a system wide default setting on boot.

- Scrubbing at a higher rate because the monitor software has determined that
  more reliability is necessary for a particular data set. This is called
  Differentiated Reliability.

1.1. Device based scrubbing

CXL memory is exposed to memory management subsystem and ultimately userspace
via CXL devices. Device-based scrubbing is used for the first use case
described in "Section 1 CXL Memory Patrol Scrub".

When combining control via the device interfaces and region interfaces,
"see Section 1.2 Region based scrubbing".

Sysfs files for scrubbing are documented in
`Documentation/ABI/testing/sysfs-edac-scrub`

CXL region 기반 scrub rate 설정

CXL region은 system physical address에 매핑된 메모리 용량을 나타내며, 여러 CXL memory device의 일부를 포함하고 traffic을 interleave할 수 있습니다. 사용자는 구성 장치를 일일이 찾는 대신 이 추상 region을 통해 scrub rate를 제어할 수 있습니다.

하지만 장치 scrub rate는 장치 전체에 적용됩니다. 같은 장치의 일부를 여러 region이 사용하면 다른 region의 요청 때문에 특정 region이 요구한 값보다 높은 rate가 적용될 수 있습니다.

혼합 요구를 설정할 때 userspace는 먼저 원하는 scrub rate가 낮은 region부터 높은 region 순서로 rate를 설정합니다. 뒤의 region은 구성 장치와 그 장치가 참여하는 전체 region의 효과를 덮어쓸 수 있습니다.

그다음 강화된 scrub이 필요한 각 장치에 더 높은 rate를 설정합니다. 특정 rate가 이미 정해진 경우를 제외하고, 장치가 뒷받침하는 모든 region의 요구 중 최대 rate가 장치에 적용됩니다.

1.2. Region based scrubbing

CXL memory is exposed to memory management subsystem and ultimately userspace
via CXL regions. CXL Regions represent mapped memory capacity in system
physical address space. These can incorporate one or more parts of multiple CXL
memory devices with traffic interleaved across them. The user may want to control
the scrub rate via this more abstract region instead of having to figure out the
constituent devices and program them separately. The scrub rate for each device
covers the whole device. Thus if multiple regions use parts of that device then
requests for scrubbing of other regions may result in a higher scrub rate than
requested for this specific region.

Region-based scrubbing is used for the third use case described in
"Section 1 CXL Memory Patrol Scrub".

Userspace must follow below set of rules on how to set the scrub rates for any
mixture of requirements.

1. Taking each region in turn from lowest desired scrub rate to highest and set
   their scrub rates. Later regions may override the scrub rate on individual
   devices (and hence potentially whole regions).

2. Take each device for which enhanced scrubbing is required (higher rate) and
   set those scrub rates. This will override the scrub rates of individual devices,
   setting them to the maximum rate required for any of the regions they help back,
   unless a specific rate is already defined.

Sysfs files for scrubbing are documented in
`Documentation/ABI/testing/sysfs-edac-scrub`

CXL ECS 제어 예제

ECS는 메모리 장치가 ECC error checking과 correction을 수행하고 single-bit error를 계수하도록 합니다. 연결된 memory controller가 trigger를 보내 ECS mode를 설정합니다.

CXL ECS control을 통해 host와 userspace는 error-count mode, error report에 사용할 segment별 threshold, ECS counter reset 속성을 바꿀 수 있습니다. Threshold는 최소 해당 오류 수를 가진 segment가 얼마나 있는지 보고하는 기준입니다.

예상보다 높은 error rate가 감지되면 ECS를 시작할 책임은 memory controller 또는 platform에 있을 수 있습니다. ABI는 `Documentation/ABI/testing/sysfs-edac-ecs`에 정의됩니다.

2. CXL memory Error Check Scrub (ECS)

The Error Check Scrub (ECS) feature enables a memory device to perform error
checking and correction (ECC) and count single-bit errors. The associated
memory controller sets the ECS mode with a trigger sent to the memory
device. CXL ECS control allows the host, thus the userspace, to change the
attributes for error count mode, threshold number of errors per segment
(indicating how many segments have at least that number of errors) for
reporting errors, and reset the ECS counter. Thus the responsibility for
initiating Error Check Scrub on a memory device may lie with the memory
controller or platform when unexpectedly high error rates are detected.

Sysfs files for scrubbing are documented in
`Documentation/ABI/testing/sysfs-edac-ecs`