← Documents Documentation/arch/powerpc/imc.rst GitHub 원문 ↗

Linux 6.18.37 · Architecture

IMC (In-Memory Collection Counters)

Nest·Core·Thread PMU와 POWER9 trace mode의 register layout, perf 사용법과 PMI 회피 효과를 설명합니다.

Source pathDocumentation/arch/powerpc/imc.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

요약과 해설

imc.rst:1-199

IMC는 performance counter를 memory로 직접 전달합니다. Trace mode는 overflow마다 program counter와 count를 snapshot해 PMI 처리 overhead 없이 instruction sampling을 수행합니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 .. SPDX-License-Identifier: GPL-2.0
2 .. _imc:
3
4 ===================================
5 IMC (In-Memory Collection Counters)
6 ===================================
7
8 Anju T Sudhakar, 10 May 2019
9
10 .. contents::
11 :depth: 3
12
13
14 Basic overview
15 ==============
16
17 IMC (In-Memory collection counters) is a hardware monitoring facility that
18 collects large numbers of hardware performance events at Nest level (these are
19 on-chip but off-core), Core level and Thread level.
20
21 The Nest PMU counters are handled by a Nest IMC microcode which runs in the OCC
22 (On-Chip Controller) complex. The microcode collects the counter data and moves
23 the nest IMC counter data to memory.
24
25 The Core and Thread IMC PMU counters are handled in the core. Core level PMU
26 counters give us the IMC counters' data per core and thread level PMU counters
27 give us the IMC counters' data per CPU thread.
28
29 OPAL obtains the IMC PMU and supported events information from the IMC Catalog
30 and passes on to the kernel via the device tree. The event's information
31 contains:
32
33 - Event name
34 - Event Offset
35 - Event description
36
37 and possibly also:
38
39 - Event scale
40 - Event unit
41
42 Some PMUs may have a common scale and unit values for all their supported
43 events. For those cases, the scale and unit properties for those events must be
44 inherited from the PMU.
45
46 The event offset in the memory is where the counter data gets accumulated.
47
48 IMC catalog is available at:
49 https://github.com/open-power/ima-catalog
50
51 The kernel discovers the IMC counters information in the device tree at the
52 `imc-counters` device node which has a compatible field
53 `ibm,opal-in-memory-counters`. From the device tree, the kernel parses the PMUs
54 and their event's information and register the PMU and its attributes in the
55 kernel.
56
57 IMC example usage
58 =================
59
60 .. code-block:: sh
61
62 # perf list
63 [...]
64 nest_mcs01/PM_MCS01_64B_RD_DISP_PORT01/ [Kernel PMU event]
65 nest_mcs01/PM_MCS01_64B_RD_DISP_PORT23/ [Kernel PMU event]
66 [...]
67 core_imc/CPM_0THRD_NON_IDLE_PCYC/ [Kernel PMU event]
68 core_imc/CPM_1THRD_NON_IDLE_INST/ [Kernel PMU event]
69 [...]
70 thread_imc/CPM_0THRD_NON_IDLE_PCYC/ [Kernel PMU event]
71 thread_imc/CPM_1THRD_NON_IDLE_INST/ [Kernel PMU event]
72
73 To see per chip data for nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/:
74
75 .. code-block:: sh
76
77 # ./perf stat -e "nest_mcs01/PM_MCS01_64B_WR_DISP_PORT01/" -a --per-socket
78
79 To see non-idle instructions for core 0:
80
81 .. code-block:: sh
82
83 # ./perf stat -e "core_imc/CPM_NON_IDLE_INST/" -C 0 -I 1000
84
85 To see non-idle instructions for a "make":
86
87 .. code-block:: sh
88
89 # ./perf stat -e "thread_imc/CPM_NON_IDLE_PCYC/" make
90
91
92 IMC Trace-mode
93 ===============
94
95 POWER9 supports two modes for IMC which are the Accumulation mode and Trace
96 mode. In Accumulation mode, event counts are accumulated in system Memory.
97 Hypervisor then reads the posted counts periodically or when requested. In IMC
98 Trace mode, the 64 bit trace SCOM value is initialized with the event
99 information. The CPMCxSEL and CPMC_LOAD in the trace SCOM, specifies the event
100 to be monitored and the sampling duration. On each overflow in the CPMCxSEL,
101 hardware snapshots the program counter along with event counts and writes into
102 memory pointed by LDBAR.
103
104 LDBAR is a 64 bit special purpose per thread register, it has bits to indicate
105 whether hardware is configured for accumulation or trace mode.
106
107 LDBAR Register Layout
108 ---------------------
109
110 +-------+----------------------+
111 | 0 | Enable/Disable |
112 +-------+----------------------+
113 | 1 | 0: Accumulation Mode |
114 | +----------------------+
115 | | 1: Trace Mode |
116 +-------+----------------------+
117 | 2:3 | Reserved |
118 +-------+----------------------+
119 | 4-6 | PB scope |
120 +-------+----------------------+
121 | 7 | Reserved |
122 +-------+----------------------+
123 | 8:50 | Counter Address |
124 +-------+----------------------+
125 | 51:63 | Reserved |
126 +-------+----------------------+
127
128 TRACE_IMC_SCOM bit representation
129 ---------------------------------
130
131 +-------+------------+
132 | 0:1 | SAMPSEL |
133 +-------+------------+
134 | 2:33 | CPMC_LOAD |
135 +-------+------------+
136 | 34:40 | CPMC1SEL |
137 +-------+------------+
138 | 41:47 | CPMC2SEL |
139 +-------+------------+
140 | 48:50 | BUFFERSIZE |
141 +-------+------------+
142 | 51:63 | RESERVED |
143 +-------+------------+
144
145 CPMC_LOAD contains the sampling duration. SAMPSEL and CPMCxSEL determines the
146 event to count. BUFFERSIZE indicates the memory range. On each overflow,
147 hardware snapshots the program counter along with event counts and updates the
148 memory and reloads the CMPC_LOAD value for the next sampling duration. IMC
149 hardware does not support exceptions, so it quietly wraps around if memory
150 buffer reaches the end.
151
152 *Currently the event monitored for trace-mode is fixed as cycle.*
153
154 Trace IMC example usage
155 =======================
156
157 .. code-block:: sh
158
159 # perf list
160 [....]
161 trace_imc/trace_cycles/ [Kernel PMU event]
162
163 To record an application/process with trace-imc event:
164
165 .. code-block:: sh
166
167 # perf record -e trace_imc/trace_cycles/ yes > /dev/null
168 [ perf record: Woken up 1 times to write data ]
169 [ perf record: Captured and wrote 0.012 MB perf.data (21 samples) ]
170
171 The `perf.data` generated, can be read using perf report.
172
173 Benefits of using IMC trace-mode
174 ================================
175
176 PMI (Performance Monitoring Interrupts) interrupt handling is avoided, since IMC
177 trace mode snapshots the program counter and updates to the memory. And this
178 also provide a way for the operating system to do instruction sampling in real
179 time without PMI processing overhead.
180
181 Performance data using `perf top` with and without trace-imc event.
182
183 PMI interrupts count when `perf top` command is executed without trace-imc event.
184
185 .. code-block:: sh
186
187 # grep PMI /proc/interrupts
188 PMI: 0 0 0 0 Performance monitoring interrupts
189 # ./perf top
190 ...
191 # grep PMI /proc/interrupts
192 PMI: 39735 8710 17338 17801 Performance monitoring interrupts
193 # ./perf top -e trace_imc/trace_cycles/
194 ...
195 # grep PMI /proc/interrupts
196 PMI: 39735 8710 17338 17801 Performance monitoring interrupts
197
198
199 That is, the PMI interrupt counts do not increment when using the `trace_imc` event.
200

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

IMC 문서 정보

1-13

IMC(In-Memory Collection Counters)는 Anju T Sudhakar가 2019년 5월 10일 정리한 PowerPC 성능 계수기 문서입니다.

기본 개요

14-56

IMC는 on-chip이지만 core 밖에 있는 Nest level, Core level, Thread level에서 많은 hardware performance event를 수집하는 monitoring facility입니다.

Nest PMU counter는 OCC(On-Chip Controller) complex에서 실행되는 Nest IMC microcode가 관리합니다. Microcode가 counter data를 모아 memory로 옮깁니다.

Core와 Thread IMC PMU counter는 core가 관리합니다. Core level은 core별 counter data를, Thread level은 CPU thread별 counter data를 제공합니다.

OPAL은 IMC Catalog에서 PMU와 지원 event 정보를 얻어 device tree로 kernel에 전달합니다. Event 정보에는 name, memory offset, description이 들어가고 경우에 따라 scale과 unit도 포함됩니다. PMU 전체가 공통 scale/unit을 쓴다면 event가 PMU property를 상속합니다.

Event offset은 memory에서 counter가 누적되는 위치입니다. IMC Catalog는 `https://github.com/open-power/ima-catalog`에서 제공됩니다.

Kernel은 compatible이 `ibm,opal-in-memory-counters`인 `imc-counters` device tree node를 찾아 PMU와 event property를 parse하고 PMU 및 attribute를 등록합니다.

IMC 기본 사용 예

57-91

`perf list`는 Nest, Core, Thread IMC event를 다음처럼 표시합니다.

# perf list
[...]
nest_mcs01/PM_MCS01_64B_RD_DISP_PORT01/            [Kernel PMU event]
nest_mcs01/PM_MCS01_64B_RD_DISP_PORT23/            [Kernel PMU event]
[...]
core_imc/CPM_0THRD_NON_IDLE_PCYC/                  [Kernel PMU event]
core_imc/CPM_1THRD_NON_IDLE_INST/                  [Kernel PMU event]
[...]
thread_imc/CPM_0THRD_NON_IDLE_PCYC/                [Kernel PMU event]
thread_imc/CPM_1THRD_NON_IDLE_INST/                [Kernel PMU event]

Nest event의 chip별 값을 보려면 system-wide per-socket 측정을 사용합니다.

# ./perf stat -e "nest_mcs01/PM_MCS01_64B_WR_DISP_PORT01/" -a --per-socket

Core 0의 non-idle instruction을 1초 간격으로 보려면 다음과 같이 실행합니다.

# ./perf stat -e "core_imc/CPM_NON_IDLE_INST/" -C 0 -I 1000

`make` process의 thread-level non-idle cycle은 다음과 같이 측정합니다.

# ./perf stat -e "thread_imc/CPM_NON_IDLE_PCYC/" make

IMC trace mode

92-106

POWER9 IMC는 Accumulation mode와 Trace mode를 지원합니다. Accumulation mode는 event count를 system memory에 누적하고 hypervisor가 주기적으로 또는 요청 시 읽습니다.

Trace mode에서는 64-bit trace SCOM 값을 event 정보로 초기화합니다. `CPMCxSEL`과 `CPMC_LOAD`가 관찰 event와 sampling duration을 정합니다. `CPMCxSEL` overflow마다 hardware가 program counter와 event count를 snapshot하여 `LDBAR`가 가리키는 memory에 씁니다.

`LDBAR`는 thread별 64-bit special-purpose register이며 hardware가 accumulation mode인지 trace mode인지 나타내는 bit를 포함합니다.

IMC trace sample 생성
`CPMCxSEL` event countOverflowProgram counter + count snapshot`LDBAR` memory`CPMC_LOAD` reload

Counter overflow가 interrupt 대신 memory snapshot을 직접 만듭니다.

LDBAR register layout

107-127
Bit의미
0Enable/Disable
1`0`: Accumulation mode, `1`: Trace mode
2:3Reserved
4:6PB scope
7Reserved
8:50Counter Address
51:63Reserved
LDBAR 64-bit 배치
Bit rangeField
0Enable
1Mode
2:3Reserved
4:6PB scope
7Reserved
8:50Counter address
51:63Reserved

Thread별 register가 mode와 counter memory address를 지정합니다.

TRACE_IMC_SCOM bit 표현

128-153
BitField
0:1`SAMPSEL`
2:33`CPMC_LOAD`
34:40`CPMC1SEL`
41:47`CPMC2SEL`
48:50`BUFFERSIZE`
51:63`RESERVED`

`CPMC_LOAD`는 sampling duration입니다. `SAMPSEL`과 `CPMCxSEL`은 계수할 event를 정하고 `BUFFERSIZE`는 memory range를 나타냅니다.

Overflow마다 hardware는 program counter와 count를 memory에 기록하고 다음 sampling period를 위해 `CPMC_LOAD`를 다시 적재합니다. IMC hardware는 exception을 지원하지 않으므로 buffer 끝에 도달하면 조용히 처음으로 wrap합니다.

현재 trace mode가 관찰하는 event는 cycle로 고정되어 있습니다.

TRACE_IMC_SCOM 64-bit 배치
Bit rangeField
0:1SAMPSEL
2:33CPMC_LOAD
34:40CPMC1SEL
41:47CPMC2SEL
48:50BUFFERSIZE
51:63RESERVED

Sampling source, duration, event selector와 buffer size를 한 register에 담습니다.

Trace IMC 사용 예

154-172

`perf list`에서 `trace_imc/trace_cycles/` event를 확인할 수 있습니다.

  # perf list
  [....]
  trace_imc/trace_cycles/                            [Kernel PMU event]

To record an application/process with trace-imc event:

.. code-block:: sh

  # perf record -e trace_imc/trace_cycles/ yes > /dev/null
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.012 MB perf.data (21 samples) ]

생성된 `perf.data`는 `perf report`로 읽습니다.

Trace mode의 장점

173-199

IMC trace mode는 program counter를 snapshot하여 memory에 직접 갱신하므로 PMI(Performance Monitoring Interrupt) 처리를 피합니다. Operating system은 PMI 처리 overhead 없이 실시간 instruction sampling을 수행할 수 있습니다.

다음 비교에서는 일반 `perf top` 실행 중 PMI count가 증가하지만 `trace_imc` event를 사용한 실행 중에는 증가하지 않습니다.

# grep PMI /proc/interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
# ./perf top
...
# grep PMI /proc/interrupts
PMI:      39735       8710      17338      17801   Performance monitoring interrupts
# ./perf top -e trace_imc/trace_cycles/
...
# grep PMI /proc/interrupts
PMI:      39735       8710      17338      17801   Performance monitoring interrupts

즉 `trace_imc` event를 사용할 때 PMI interrupt count가 증가하지 않는 것이 확인됩니다.