요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
Scheduling과 workload
amdnpu.rst:119-209Resource Solver가 spatial·temporal partition을 배치하고 overlay·ctrlcode·special buffer를 이용해 workload를 실행합니다.
Boot와 runtime service
amdnpu.rst:210-281Signed firmware boot, compiler·XRT stack, DMA opcode, privileged error capture와 telemetry를 정리합니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0-only
.. include:: <isonum.txt>
=========
AMD NPU
=========
:Copyright: |copy| 2024 Advanced Micro Devices, Inc.
:Author: Sonal Santan <sonal.santan@amd.com>
Overview
========
AMD NPU (Neural Processing Unit) is a multi-user AI inference accelerator
integrated into AMD client APU. NPU enables efficient execution of Machine
Learning applications like CNN, LLM, etc. NPU is based on
`AMD XDNA Architecture`_. NPU is managed by **amdxdna** driver.
Hardware Description
====================
AMD NPU consists of the following hardware components:
AMD XDNA Array
--------------
AMD XDNA Array comprises of 2D array of compute and memory tiles built with
`AMD AI Engine Technology`_. Each column has 4 rows of compute tiles and 1
row of memory tile. Each compute tile contains a VLIW processor with its own
dedicated program and data memory. The memory tile acts as L2 memory. The 2D
array can be partitioned at a column boundary creating a spatially isolated
partition which can be bound to a workload context.
Each column also has dedicated DMA engines to move data between host DDR and
memory tile.
AMD Phoenix and AMD Hawk Point client NPU have a 4x5 topology, i.e., 4 rows of
compute tiles arranged into 5 columns. AMD Strix Point client APU have 4x8
topology, i.e., 4 rows of compute tiles arranged into 8 columns.
Shared L2 Memory
----------------
The single row of memory tiles create a pool of software managed on chip L2
memory. DMA engines are used to move data between host DDR and memory tiles.
AMD Phoenix and AMD Hawk Point NPUs have a total of 2560 KB of L2 memory.
AMD Strix Point NPU has a total of 4096 KB of L2 memory.
Microcontroller
---------------
A microcontroller runs NPU Firmware which is responsible for command processing,
XDNA Array partition setup, XDNA Array configuration, workload context
management and workload orchestration.
NPU Firmware uses a dedicated instance of an isolated non-privileged context
called ERT to service each workload context. ERT is also used to execute user
provided ``ctrlcode`` associated with the workload context.
NPU Firmware uses a single isolated privileged context called MERT to service
management commands from the amdxdna driver.
Mailboxes
---------
The microcontroller and amdxdna driver use a privileged channel for management
tasks like setting up of contexts, telemetry, query, error handling, setting up
user channel, etc. As mentioned before, privileged channel requests are
serviced by MERT. The privileged channel is bound to a single mailbox.
The microcontroller and amdxdna driver use a dedicated user channel per
workload context. The user channel is primarily used for submitting work to
the NPU. As mentioned before, a user channel requests are serviced by an
instance of ERT. Each user channel is bound to its own dedicated mailbox.
PCIe EP
-------
NPU is visible to the x86 host CPU as a PCIe device with multiple BARs and some
MSI-X interrupt vectors. NPU uses a dedicated high bandwidth SoC level fabric
for reading or writing into host memory. Each instance of ERT gets its own
dedicated MSI-X interrupt. MERT gets a single instance of MSI-X interrupt.
The number of PCIe BARs varies depending on the specific device. Based on their
functions, PCIe BARs can generally be categorized into the following types.
* PSP BAR: Expose the AMD PSP (Platform Security Processor) function
* SMU BAR: Expose the AMD SMU (System Management Unit) function
* SRAM BAR: Expose ring buffers for the mailbox
* Mailbox BAR: Expose the mailbox control registers (head, tail and ISR
registers etc.)
* Public Register BAR: Expose public registers
On specific devices, the above-mentioned BAR type might be combined into a
single physical PCIe BAR. Or a module might require two physical PCIe BARs to
be fully functional. For example,
* On AMD Phoenix device, PSP, SMU, Public Register BARs are on PCIe BAR index 0.
* On AMD Strix Point device, Mailbox and Public Register BARs are on PCIe BAR
index 0. The PSP has some registers in PCIe BAR index 0 (Public Register BAR)
and PCIe BAR index 4 (PSP BAR).
Process Isolation Hardware
--------------------------
As explained before, XDNA Array can be dynamically divided into isolated
spatial partitions, each of which may have one or more columns. The spatial
partition is setup by programming the column isolation registers by the
microcontroller. Each spatial partition is associated with a PASID which is
also programmed by the microcontroller. Hence multiple spatial partitions in
the NPU can make concurrent host access protected by PASID.
The NPU FW itself uses microcontroller MMU enforced isolated contexts for
servicing user and privileged channel requests.
Mixed Spatial and Temporal Scheduling
=====================================
AMD XDNA architecture supports mixed spatial and temporal (time sharing)
scheduling of 2D array. This means that spatial partitions may be setup and
torn down dynamically to accommodate various workloads. A *spatial* partition
may be *exclusively* bound to one workload context while another partition may
be *temporarily* bound to more than one workload contexts. The microcontroller
updates the PASID for a temporarily shared partition to match the context that
has been bound to the partition at any moment.
Resource Solver
---------------
The Resource Solver component of the amdxdna driver manages the allocation
of 2D array among various workloads. Every workload describes the number
of columns required to run the NPU binary in its metadata. The Resource Solver
component uses hints passed by the workload and its own heuristics to
decide 2D array (re)partition strategy and mapping of workloads for spatial and
temporal sharing of columns. The FW enforces the context-to-column(s) resource
binding decisions made by the Resource Solver.
AMD Phoenix and AMD Hawk Point client NPU can support 6 concurrent workload
contexts. AMD Strix Point can support 16 concurrent workload contexts.
Application Binaries
====================
A NPU application workload is comprised of two separate binaries which are
generated by the NPU compiler.
1. AMD XDNA Array overlay, which is used to configure a NPU spatial partition.
The overlay contains instructions for setting up the stream switch
configuration and ELF for the compute tiles. The overlay is loaded on the
spatial partition bound to the workload by the associated ERT instance.
Refer to the
`Versal Adaptive SoC AIE-ML Architecture Manual (AM020)`_ for more details.
2. ``ctrlcode``, used for orchestrating the overlay loaded on the spatial
partition. ``ctrlcode`` is executed by the ERT running in protected mode on
the microcontroller in the context of the workload. ``ctrlcode`` is made up
of a sequence of opcodes named ``XAie_TxnOpcode``. Refer to the
`AI Engine Run Time`_ for more details.
Special Host Buffers
====================
Per-context Instruction Buffer
------------------------------
Every workload context uses a host resident 64 MB buffer which is memory
mapped into the ERT instance created to service the workload. The ``ctrlcode``
used by the workload is copied into this special memory. This buffer is
protected by PASID like all other input/output buffers used by that workload.
Instruction buffer is also mapped into the user space of the workload.
Global Privileged Buffer
------------------------
In addition, the driver also allocates a single buffer for maintenance tasks
like recording errors from MERT. This global buffer uses the global IOMMU
domain and is only accessible by MERT.
High-level Use Flow
===================
Here are the steps to run a workload on AMD NPU:
1. Compile the workload into an overlay and a ``ctrlcode`` binary.
2. Userspace opens a context in the driver and provides the overlay.
3. The driver checks with the Resource Solver for provisioning a set of columns
for the workload.
4. The driver then asks MERT to create a context on the device with the desired
columns.
5. MERT then creates an instance of ERT. MERT also maps the Instruction Buffer
into ERT memory.
6. The userspace then copies the ``ctrlcode`` to the Instruction Buffer.
7. Userspace then creates a command buffer with pointers to input, output, and
instruction buffer; it then submits command buffer with the driver and goes
to sleep waiting for completion.
8. The driver sends the command over the Mailbox to ERT.
9. ERT *executes* the ``ctrlcode`` in the instruction buffer.
10. Execution of the ``ctrlcode`` kicks off DMAs to and from the host DDR while
AMD XDNA Array is running.
11. When ERT reaches end of ``ctrlcode``, it raises an MSI-X to send completion
signal to the driver which then wakes up the waiting workload.
Boot Flow
=========
amdxdna driver uses PSP to securely load signed NPU FW and kick off the boot
of the NPU microcontroller. amdxdna driver then waits for the alive signal in
a special location on BAR 0. The NPU is switched off during SoC suspend and
turned on after resume where the NPU FW is reloaded, and the handshake is
performed again.
Userspace components
====================
Compiler
--------
Peano is an LLVM based open-source single core compiler for AMD XDNA Array
compute tile. Peano is available at:
https://github.com/Xilinx/llvm-aie
IRON is an open-source array compiler for AMD XDNA Array based NPU which uses
Peano underneath. IRON is available at:
https://github.com/Xilinx/mlir-aie
Usermode Driver (UMD)
---------------------
The open-source XRT runtime stack interfaces with amdxdna kernel driver. XRT
can be found at:
https://github.com/Xilinx/XRT
The open-source XRT shim for NPU is can be found at:
https://github.com/amd/xdna-driver
DMA Operation
=============
DMA operation instructions are encoded in the ``ctrlcode`` as
``XAIE_IO_BLOCKWRITE`` opcode. When ERT executes ``XAIE_IO_BLOCKWRITE``, DMA
operations between host DDR and L2 memory are effected.
Error Handling
==============
When MERT detects an error in AMD XDNA Array, it pauses execution for that
workload context and sends an asynchronous message to the driver over the
privileged channel. The driver then sends a buffer pointer to MERT to capture
the register states for the partition bound to faulting workload context. The
driver then decodes the error by reading the contents of the buffer pointer.
Telemetry
=========
MERT can report various kinds of telemetry information like the following:
* L1 interrupt counter
* DMA counter
* Deep Sleep counter
* etc.
References
==========
- `AMD XDNA Architecture <https://www.amd.com/en/technologies/xdna.html>`_
- `AMD AI Engine Technology <https://www.xilinx.com/products/technology/ai-engine.html>`_
- `Peano <https://github.com/Xilinx/llvm-aie>`_
- `Versal Adaptive SoC AIE-ML Architecture Manual (AM020) <https://docs.amd.com/r/en-US/am020-versal-aie-ml>`_
- `AI Engine Run Time <https://github.com/Xilinx/aie-rt/tree/release/main_aig>`_
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 표제와 저작권
1-10이 문서는 `GPL-2.0-only`로 배포되는 AMD NPU 설명서이며 `isonum.txt`를 포함합니다. 저작권은 2024 Advanced Micro Devices, Inc.에 있고 저자는 Sonal Santan <sonal.santan@amd.com>입니다.
.. SPDX-License-Identifier: GPL-2.0-only
.. include:: <isonum.txt>
AMD NPU
:Copyright: |copy| 2024 Advanced Micro Devices, Inc.
:Author: Sonal Santan <sonal.santan@amd.com>개요
11-19AMD NPU(Neural Processing Unit)는 AMD client APU에 통합된 multi-user AI inference accelerator입니다. CNN, LLM 등의 Machine Learning application을 효율적으로 실행하며 `AMD XDNA Architecture`를 기반으로 합니다. NPU는 `amdxdna` driver가 관리합니다.
AMD XDNA Array
20-42AMD NPU의 XDNA Array는 `AMD AI Engine Technology`로 만든 compute tile과 memory tile의 2D array입니다. 각 column에는 compute tile 4개 row와 memory tile 1개 row가 있습니다. 각 compute tile에는 전용 program memory와 data memory를 가진 VLIW processor가 있고 memory tile은 L2 memory로 동작합니다.
2D array는 column 경계에서 나누어 spatially isolated partition을 만들 수 있고 partition은 workload context에 bind됩니다. 각 column의 전용 DMA engine은 host DDR와 memory tile 사이에서 data를 이동합니다. Phoenix와 Hawk Point client NPU는 compute tile 4개 row와 5개 column의 `4x5` topology이고 Strix Point client APU는 4개 row와 8개 column의 `4x8` topology입니다.
각 세대의 compute-array column 수와 공통 tile 구성을 비교합니다.
공유 L2 memory
43-50Memory tile 한 row는 software가 관리하는 on-chip L2 memory pool을 만듭니다. DMA engine이 host DDR와 memory tile 사이의 data를 이동합니다. Phoenix와 Hawk Point NPU의 L2 memory 총량은 `2560 KB`, Strix Point NPU의 총량은 `4096 KB`입니다.
Memory-tile row가 제공하는 software-managed L2 pool입니다.
Microcontroller와 firmware context
51-64Microcontroller에서 실행되는 NPU Firmware는 command 처리, XDNA Array partition 설정과 구성, workload context 관리와 workload orchestration을 담당합니다.
NPU Firmware는 workload context마다 isolated non-privileged context인 ERT 전용 instance를 사용합니다. ERT는 workload context와 연결된 사용자의 `ctrlcode`도 실행합니다. Management command는 하나의 isolated privileged context인 MERT가 `amdxdna` driver를 대신해 처리합니다.
User workload와 privileged management를 서로 다른 context로 격리합니다.
Privileged channel과 user channel
65-77Microcontroller와 `amdxdna` driver는 context 설정, telemetry, query, error handling, user channel 설정 등의 management task에 privileged channel을 사용합니다. 이 request는 MERT가 처리하며 privileged channel은 mailbox 하나에 bind됩니다.
각 workload context에는 NPU work submit에 주로 쓰는 전용 user channel이 있습니다. User channel request는 해당 ERT instance가 처리하며 channel마다 고유한 mailbox 하나에 bind됩니다.
Management와 workload traffic이 별도의 execution context와 mailbox로 향합니다.
PCIe endpoint와 BAR
78-104NPU는 x86 host CPU에 여러 BAR와 일부 MSI-X interrupt vector를 가진 PCIe device로 보입니다. Host memory read/write에는 전용 high-bandwidth SoC fabric을 사용합니다. ERT instance마다 전용 MSI-X interrupt가 하나씩 있고 MERT에는 MSI-X interrupt instance 하나가 있습니다.
PCIe BAR 수는 device에 따라 다릅니다. 기능별 분류는 AMD PSP(Platform Security Processor)를 노출하는 PSP BAR, AMD SMU(System Management Unit)를 노출하는 SMU BAR, mailbox ring buffer를 노출하는 SRAM BAR, head·tail·ISR 등의 mailbox control register를 노출하는 Mailbox BAR, public register를 노출하는 Public Register BAR입니다.
Device에 따라 여러 BAR type이 하나의 physical PCIe BAR에 합쳐지거나 module 하나가 완전히 동작하는 데 physical BAR 두 개가 필요할 수 있습니다. Phoenix에서는 PSP, SMU, Public Register BAR가 PCIe BAR index 0에 있습니다. Strix Point에서는 Mailbox와 Public Register BAR가 index 0에 있고 PSP register 일부는 index 0(Public Register BAR), 나머지는 index 4(PSP BAR)에 있습니다.
Physical BAR 배치는 device별로 결합되거나 분할될 수 있습니다.
Process isolation hardware
105-118XDNA Array는 column 하나 이상으로 이루어진 isolated spatial partition들로 동적으로 나눌 수 있습니다. Microcontroller가 column isolation register를 programming해 partition을 설정하고 각 partition에 연관된 PASID도 programming합니다. 따라서 NPU의 여러 spatial partition은 PASID로 보호된 host access를 동시에 수행할 수 있습니다.
NPU Firmware 자체는 user channel과 privileged channel request를 처리할 때 microcontroller MMU가 강제하는 isolated context를 사용합니다.
Spatial·temporal scheduling과 Resource Solver
119-144AMD XDNA architecture는 2D array의 mixed spatial scheduling과 temporal(time sharing) scheduling을 지원합니다. 다양한 workload에 맞춰 spatial partition을 동적으로 만들고 제거할 수 있습니다. Spatial partition 하나를 workload context 하나에 exclusive하게 bind할 수도 있고, 다른 partition을 둘 이상의 workload context가 시간에 따라 잠시 공유할 수도 있습니다. Microcontroller는 일시 공유 partition의 PASID를 그 순간 bind된 context와 일치하도록 갱신합니다.
`amdxdna` driver의 Resource Solver는 workload 사이의 2D array 할당을 관리합니다. 각 workload metadata는 NPU binary 실행에 필요한 column 수를 기술합니다. Solver는 workload hint와 자체 heuristic으로 2D array (재)partition 전략, spatial·temporal column sharing을 위한 workload mapping을 결정합니다. Firmware는 Solver가 정한 context-to-column(s) resource binding을 강제합니다. Phoenix와 Hawk Point는 concurrent workload context 6개, Strix Point는 16개를 지원합니다.
Workload 요구와 hint에서 partition binding까지의 결정 흐름입니다.
Application binaries
145-164NPU compiler가 만드는 application workload는 서로 분리된 binary 두 개로 구성됩니다. 첫째, AMD XDNA Array overlay는 NPU spatial partition을 구성합니다. Overlay에는 stream switch 설정 instruction과 compute tile용 ELF가 들어 있으며 workload에 bind된 spatial partition에 연관 ERT instance가 load합니다. 자세한 내용은 `Versal Adaptive SoC AIE-ML Architecture Manual (AM020)`을 참조합니다.
둘째, `ctrlcode`는 spatial partition에 load된 overlay를 orchestration합니다. Workload context에서 microcontroller의 protected mode로 실행되는 ERT가 `ctrlcode`를 실행합니다. `ctrlcode`는 `XAie_TxnOpcode`라는 opcode sequence로 구성되며 자세한 내용은 `AI Engine Run Time`을 참조합니다.
Configuration binary와 orchestration binary의 역할을 구분합니다.
Special host buffers
165-184각 workload context는 자신을 처리하는 ERT instance에 memory-map된 host-resident `64 MB` instruction buffer를 사용합니다. Workload의 `ctrlcode`를 이 special memory에 복사합니다. 다른 input/output buffer와 마찬가지로 PASID로 보호되며 workload userspace에도 map됩니다.
Driver는 MERT의 error 기록 같은 maintenance task를 위해 global privileged buffer 하나도 할당합니다. 이 global buffer는 global IOMMU domain을 사용하며 MERT만 접근할 수 있습니다.
Per-context user buffer와 global privileged buffer의 보호 범위를 비교합니다.
Workload 실행 흐름
185-209AMD NPU workload는 다음 순서로 실행합니다. (1) workload를 overlay와 `ctrlcode` binary로 compile합니다. (2) Userspace가 driver context를 열고 overlay를 제공합니다. (3) Driver가 Resource Solver에 필요한 column provision을 요청합니다. (4) Driver가 MERT에 원하는 column을 가진 device context 생성을 요청합니다. (5) MERT가 ERT instance를 만들고 Instruction Buffer를 ERT memory에 map합니다.
(6) Userspace가 `ctrlcode`를 Instruction Buffer에 복사합니다. (7) Input, output, instruction buffer pointer를 가진 command buffer를 만들어 driver에 submit한 뒤 completion을 기다리며 sleep합니다. (8) Driver가 Mailbox로 ERT에 command를 보냅니다. (9) ERT가 instruction buffer의 `ctrlcode`를 실행합니다. (10) `ctrlcode` 실행이 AMD XDNA Array 동작 중 host DDR와의 DMA를 시작합니다. (11) ERT가 `ctrlcode` 끝에 도달하면 MSI-X로 driver에 completion signal을 보내고 driver는 기다리던 workload를 깨웁니다.
Compilation부터 ERT completion interrupt까지의 11단계입니다.
Boot와 suspend/resume
210-219`amdxdna` driver는 PSP를 사용해 signed NPU Firmware를 secure하게 load하고 NPU microcontroller boot를 시작합니다. 이후 BAR 0의 special location에서 alive signal을 기다립니다. SoC suspend 동안 NPU를 끄고 resume 후 다시 켜며, 이때 NPU Firmware를 다시 load하고 handshake도 다시 수행합니다.
Initial boot와 resume가 같은 firmware load·alive handshake를 수행합니다.
Compiler와 usermode driver
220-244Peano는 AMD XDNA Array compute tile용 LLVM 기반 open-source single-core compiler이며 `https://github.com/Xilinx/llvm-aie`에서 제공합니다. IRON은 Peano를 내부적으로 사용하는 AMD XDNA Array 기반 NPU용 open-source array compiler이며 `https://github.com/Xilinx/mlir-aie`에서 제공합니다.
Open-source XRT runtime stack은 `amdxdna` kernel driver와 interface하며 `https://github.com/Xilinx/XRT`에서 제공합니다. NPU용 open-source XRT shim은 `https://github.com/amd/xdna-driver`에서 제공합니다.
Compilation과 runtime components 및 upstream locations입니다.
DMA operation
245-252DMA operation instruction은 `ctrlcode` 안에 `XAIE_IO_BLOCKWRITE` opcode로 encode됩니다. ERT가 `XAIE_IO_BLOCKWRITE`를 실행하면 host DDR와 L2 memory 사이의 DMA operation이 수행됩니다.
Error handling
253-262MERT가 AMD XDNA Array error를 감지하면 해당 workload context의 실행을 pause하고 privileged channel을 통해 driver에 asynchronous message를 보냅니다. Driver는 faulting workload context에 bind된 partition의 register state를 capture하도록 MERT에 buffer pointer를 보냅니다. 이후 driver는 그 buffer 내용을 읽어 error를 decode합니다.
Fault detection에서 register-state decoding까지의 privileged path입니다.
Telemetry
263-273MERT는 L1 interrupt counter, DMA counter, Deep Sleep counter 등 여러 종류의 telemetry information을 보고할 수 있습니다.
- L1 interrupt counter
- DMA counter
- Deep Sleep counter
- 기타 telemetry
참고 자료
274-281- AMD XDNA Architecture
https://www.amd.com/en/technologies/xdna.html - AMD AI Engine Technology
https://www.xilinx.com/products/technology/ai-engine.html - Peano
https://github.com/Xilinx/llvm-aie - Versal Adaptive SoC AIE-ML Architecture Manual (AM020)
https://docs.amd.com/r/en-US/am020-versal-aie-ml - AI Engine Run Time
https://github.com/Xilinx/aie-rt/tree/release/main_aig
XDNA hardware
amdnpu.rst:1-118Tile array와 shared L2, microcontroller, ERT·MERT mailbox, PCIe BAR와 PASID 기반 process isolation을 설명합니다.