요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
===================================================
PCI Express I/O Virtualization Resource on Powerenv
===================================================
Wei Yang <weiyang@linux.vnet.ibm.com>
Benjamin Herrenschmidt <benh@au1.ibm.com>
Bjorn Helgaas <bhelgaas@google.com>
26 Aug 2014
This document describes the requirement from hardware for PCI MMIO resource
sizing and assignment on PowerKVM and how generic PCI code handles this
requirement. The first two sections describe the concepts of Partitionable
Endpoints and the implementation on P8 (IODA2). The next two sections talks
about considerations on enabling SRIOV on IODA2.
1. Introduction to Partitionable Endpoints
==========================================
A Partitionable Endpoint (PE) is a way to group the various resources
associated with a device or a set of devices to provide isolation between
partitions (i.e., filtering of DMA, MSIs etc.) and to provide a mechanism
to freeze a device that is causing errors in order to limit the possibility
of propagation of bad data.
There is thus, in HW, a table of PE states that contains a pair of "frozen"
state bits (one for MMIO and one for DMA, they get set together but can be
cleared independently) for each PE.
When a PE is frozen, all stores in any direction are dropped and all loads
return all 1's value. MSIs are also blocked. There's a bit more state that
captures things like the details of the error that caused the freeze etc., but
that's not critical.
The interesting part is how the various PCIe transactions (MMIO, DMA, ...)
are matched to their corresponding PEs.
The following section provides a rough description of what we have on P8
(IODA2). Keep in mind that this is all per PHB (PCI host bridge). Each PHB
is a completely separate HW entity that replicates the entire logic, so has
its own set of PEs, etc.
2. Implementation of Partitionable Endpoints on P8 (IODA2)
==========================================================
P8 supports up to 256 Partitionable Endpoints per PHB.
* Inbound
For DMA, MSIs and inbound PCIe error messages, we have a table (in
memory but accessed in HW by the chip) that provides a direct
correspondence between a PCIe RID (bus/dev/fn) with a PE number.
We call this the RTT.
- For DMA we then provide an entire address space for each PE that can
contain two "windows", depending on the value of PCI address bit 59.
Each window can be configured to be remapped via a "TCE table" (IOMMU
translation table), which has various configurable characteristics
not described here.
- For MSIs, we have two windows in the address space (one at the top of
the 32-bit space and one much higher) which, via a combination of the
address and MSI value, will result in one of the 2048 interrupts per
bridge being triggered. There's a PE# in the interrupt controller
descriptor table as well which is compared with the PE# obtained from
the RTT to "authorize" the device to emit that specific interrupt.
- Error messages just use the RTT.
* Outbound. That's where the tricky part is.
Like other PCI host bridges, the Power8 IODA2 PHB supports "windows"
from the CPU address space to the PCI address space. There is one M32
window and sixteen M64 windows. They have different characteristics.
First what they have in common: they forward a configurable portion of
the CPU address space to the PCIe bus and must be naturally aligned
power of two in size. The rest is different:
- The M32 window:
* Is limited to 4GB in size.
* Drops the top bits of the address (above the size) and replaces
them with a configurable value. This is typically used to generate
32-bit PCIe accesses. We configure that window at boot from FW and
don't touch it from Linux; it's usually set to forward a 2GB
portion of address space from the CPU to PCIe
0x8000_0000..0xffff_ffff. (Note: The top 64KB are actually
reserved for MSIs but this is not a problem at this point; we just
need to ensure Linux doesn't assign anything there, the M32 logic
ignores that however and will forward in that space if we try).
* It is divided into 256 segments of equal size. A table in the chip
maps each segment to a PE#. That allows portions of the MMIO space
to be assigned to PEs on a segment granularity. For a 2GB window,
the segment granularity is 2GB/256 = 8MB.
Now, this is the "main" window we use in Linux today (excluding
SR-IOV). We basically use the trick of forcing the bridge MMIO windows
onto a segment alignment/granularity so that the space behind a bridge
can be assigned to a PE.
Ideally we would like to be able to have individual functions in PEs
but that would mean using a completely different address allocation
scheme where individual function BARs can be "grouped" to fit in one or
more segments.
- The M64 windows:
* Must be at least 256MB in size.
* Do not translate addresses (the address on PCIe is the same as the
address on the PowerBus). There is a way to also set the top 14
bits which are not conveyed by PowerBus but we don't use this.
* Can be configured to be segmented. When not segmented, we can
specify the PE# for the entire window. When segmented, a window
has 256 segments; however, there is no table for mapping a segment
to a PE#. The segment number *is* the PE#.
* Support overlaps. If an address is covered by multiple windows,
there's a defined ordering for which window applies.
We have code (fairly new compared to the M32 stuff) that exploits that
for large BARs in 64-bit space:
We configure an M64 window to cover the entire region of address space
that has been assigned by FW for the PHB (about 64GB, ignore the space
for the M32, it comes out of a different "reserve"). We configure it
as segmented.
Then we do the same thing as with M32, using the bridge alignment
trick, to match to those giant segments.
Since we cannot remap, we have two additional constraints:
- We do the PE# allocation *after* the 64-bit space has been assigned
because the addresses we use directly determine the PE#. We then
update the M32 PE# for the devices that use both 32-bit and 64-bit
spaces or assign the remaining PE# to 32-bit only devices.
- We cannot "group" segments in HW, so if a device ends up using more
than one segment, we end up with more than one PE#. There is a HW
mechanism to make the freeze state cascade to "companion" PEs but
that only works for PCIe error messages (typically used so that if
you freeze a switch, it freezes all its children). So we do it in
SW. We lose a bit of effectiveness of EEH in that case, but that's
the best we found. So when any of the PEs freezes, we freeze the
other ones for that "domain". We thus introduce the concept of
"master PE" which is the one used for DMA, MSIs, etc., and "secondary
PEs" that are used for the remaining M64 segments.
We would like to investigate using additional M64 windows in "single
PE" mode to overlay over specific BARs to work around some of that, for
example for devices with very large BARs, e.g., GPUs. It would make
sense, but we haven't done it yet.
3. Considerations for SR-IOV on PowerKVM
========================================
* SR-IOV Background
The PCIe SR-IOV feature allows a single Physical Function (PF) to
support several Virtual Functions (VFs). Registers in the PF's SR-IOV
Capability control the number of VFs and whether they are enabled.
When VFs are enabled, they appear in Configuration Space like normal
PCI devices, but the BARs in VF config space headers are unusual. For
a non-VF device, software uses BARs in the config space header to
discover the BAR sizes and assign addresses for them. For VF devices,
software uses VF BAR registers in the *PF* SR-IOV Capability to
discover sizes and assign addresses. The BARs in the VF's config space
header are read-only zeros.
When a VF BAR in the PF SR-IOV Capability is programmed, it sets the
base address for all the corresponding VF(n) BARs. For example, if the
PF SR-IOV Capability is programmed to enable eight VFs, and it has a
1MB VF BAR0, the address in that VF BAR sets the base of an 8MB region.
This region is divided into eight contiguous 1MB regions, each of which
is a BAR0 for one of the VFs. Note that even though the VF BAR
describes an 8MB region, the alignment requirement is for a single VF,
i.e., 1MB in this example.
There are several strategies for isolating VFs in PEs:
- M32 window: There's one M32 window, and it is split into 256
equally-sized segments. The finest granularity possible is a 256MB
window with 1MB segments. VF BARs that are 1MB or larger could be
mapped to separate PEs in this window. Each segment can be
individually mapped to a PE via the lookup table, so this is quite
flexible, but it works best when all the VF BARs are the same size. If
they are different sizes, the entire window has to be small enough that
the segment size matches the smallest VF BAR, which means larger VF
BARs span several segments.
- Non-segmented M64 window: A non-segmented M64 window is mapped entirely
to a single PE, so it could only isolate one VF.
- Single segmented M64 windows: A segmented M64 window could be used just
like the M32 window, but the segments can't be individually mapped to
PEs (the segment number is the PE#), so there isn't as much
flexibility. A VF with multiple BARs would have to be in a "domain" of
multiple PEs, which is not as well isolated as a single PE.
- Multiple segmented M64 windows: As usual, each window is split into 256
equally-sized segments, and the segment number is the PE#. But if we
use several M64 windows, they can be set to different base addresses
and different segment sizes. If we have VFs that each have a 1MB BAR
and a 32MB BAR, we could use one M64 window to assign 1MB segments and
another M64 window to assign 32MB segments.
Finally, the plan to use M64 windows for SR-IOV, which will be described
more in the next two sections. For a given VF BAR, we need to
effectively reserve the entire 256 segments (256 * VF BAR size) and
position the VF BAR to start at the beginning of a free range of
segments/PEs inside that M64 window.
The goal is of course to be able to give a separate PE for each VF.
The IODA2 platform has 16 M64 windows, which are used to map MMIO
range to PE#. Each M64 window defines one MMIO range and this range is
divided into 256 segments, with each segment corresponding to one PE.
We decide to leverage this M64 window to map VFs to individual PEs, since
SR-IOV VF BARs are all the same size.
But doing so introduces another problem: total_VFs is usually smaller
than the number of M64 window segments, so if we map one VF BAR directly
to one M64 window, some part of the M64 window will map to another
device's MMIO range.
IODA supports 256 PEs, so segmented windows contain 256 segments, so if
total_VFs is less than 256, we have the situation in Figure 1.0, where
segments [total_VFs, 255] of the M64 window may map to some MMIO range on
other devices::
0 1 total_VFs - 1
+------+------+- -+------+------+
| | | ... | | |
+------+------+- -+------+------+
VF(n) BAR space
0 1 total_VFs - 1 255
+------+------+- -+------+------+- -+------+------+
| | | ... | | | ... | | |
+------+------+- -+------+------+- -+------+------+
M64 window
Figure 1.0 Direct map VF(n) BAR space
Our current solution is to allocate 256 segments even if the VF(n) BAR
space doesn't need that much, as shown in Figure 1.1::
0 1 total_VFs - 1 255
+------+------+- -+------+------+- -+------+------+
| | | ... | | | ... | | |
+------+------+- -+------+------+- -+------+------+
VF(n) BAR space + extra
0 1 total_VFs - 1 255
+------+------+- -+------+------+- -+------+------+
| | | ... | | | ... | | |
+------+------+- -+------+------+- -+------+------+
M64 window
Figure 1.1 Map VF(n) BAR space + extra
Allocating the extra space ensures that the entire M64 window will be
assigned to this one SR-IOV device and none of the space will be
available for other devices. Note that this only expands the space
reserved in software; there are still only total_VFs VFs, and they only
respond to segments [0, total_VFs - 1]. There's nothing in hardware that
responds to segments [total_VFs, 255].
4. Implications for the Generic PCI Code
========================================
The PCIe SR-IOV spec requires that the base of the VF(n) BAR space be
aligned to the size of an individual VF BAR.
In IODA2, the MMIO address determines the PE#. If the address is in an M32
window, we can set the PE# by updating the table that translates segments
to PE#s. Similarly, if the address is in an unsegmented M64 window, we can
set the PE# for the window. But if it's in a segmented M64 window, the
segment number is the PE#.
Therefore, the only way to control the PE# for a VF is to change the base
of the VF(n) BAR space in the VF BAR. If the PCI core allocates the exact
amount of space required for the VF(n) BAR space, the VF BAR value is fixed
and cannot be changed.
On the other hand, if the PCI core allocates additional space, the VF BAR
value can be changed as long as the entire VF(n) BAR space remains inside
the space allocated by the core.
Ideally the segment size will be the same as an individual VF BAR size.
Then each VF will be in its own PE. The VF BARs (and therefore the PE#s)
are contiguous. If VF0 is in PE(x), then VF(n) is in PE(x+n). If we
allocate 256 segments, there are (256 - numVFs) choices for the PE# of VF0.
If the segment size is smaller than the VF BAR size, it will take several
segments to cover a VF BAR, and a VF will be in several PEs. This is
possible, but the isolation isn't as good, and it reduces the number of PE#
choices because instead of consuming only numVFs segments, the VF(n) BAR
space will consume (numVFs * n) segments. That means there aren't as many
available segments for adjusting base of the VF(n) BAR space.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
PowerNV PCI I/O virtualization resource
1-18이 문서는 PowerKVM에서 PCI MMIO resource를 sizing·assignment하기 위한 hardware 요구 사항과 generic PCI code의 처리 방식을 설명합니다. P8 IODA2의 Partitionable Endpoint와 SR-IOV enable 고려 사항이 중심입니다.
문서는 Wei Yang, Benjamin Herrenschmidt, Bjorn Helgaas가 작성했으며 날짜는 2014년 8월 26일입니다.
Partitionable Endpoint
19-44Partitionable Endpoint(PE)는 device 또는 device 집합의 DMA, MSI 등 resource를 묶어 partition 사이를 isolate하고, 오류 device를 freeze하여 bad data propagation을 제한하는 hardware 단위입니다.
Hardware PE state table에는 PE마다 MMIO용과 DMA용 frozen bit가 있습니다. 두 bit는 함께 set되지만 독립적으로 clear할 수 있습니다.
PE가 frozen이면 어느 방향이든 store는 drop되고 load는 all-ones를 반환하며 MSI도 block됩니다. Freeze 원인 같은 추가 state도 저장합니다.
핵심은 MMIO, DMA 등 각 PCIe transaction을 올바른 PE에 match하는 방식입니다. 이 logic은 PHB마다 완전히 독립적으로 복제됩니다.
P8 IODA2 inbound mapping
45-71P8은 PHB마다 최대 256개 PE를 지원합니다.
DMA, MSI, inbound PCIe error message는 PCIe RID(bus/dev/fn)를 PE number와 직접 대응시키는 RTT table을 사용합니다. Table은 memory에 있지만 chip hardware가 접근합니다.
- DMA는 PE마다 별도 address space를 제공하고 PCI address bit 59에 따라 두 window를 가질 수 있습니다. 각 window는 TCE table(IOMMU translation table)로 remap할 수 있습니다.
- MSI는 32-bit space 상단과 더 높은 address의 두 window를 사용합니다. Address와 MSI value 조합으로 bridge당 2048 interrupt 중 하나를 trigger하며 interrupt descriptor의 PE#와 RTT PE#를 비교해 device 권한을 확인합니다.
- Inbound PCIe error message는 RTT를 직접 사용합니다.
RID가 PE를 선택하고 DMA, MSI와 error path가 각 PE isolation에 연결됩니다.
Outbound M32 window
72-109IODA2 PHB는 CPU address space에서 PCI address space로 향하는 M32 window 1개와 M64 window 16개를 지원합니다. 모두 configurable CPU range를 PCIe bus로 전달하고 크기는 naturally aligned power of two여야 합니다.
M32는 최대 4GB이며 size 위쪽 address bit를 버리고 configurable value로 대체해 보통 32-bit PCIe access를 만듭니다. Firmware가 boot 때 보통 CPU `0x8000_0000..0xffff_ffff`의 2GB를 forward하도록 설정하고 Linux는 변경하지 않습니다.
M32 top 64KB는 MSI용으로 reserved입니다. M32 logic 자체는 이 공간도 forward하므로 Linux가 다른 resource를 assign하지 않도록 해야 합니다.
M32는 같은 크기의 256 segment로 나뉘고 chip table이 segment를 PE#에 map합니다. 2GB window라면 granularity는 `2GB/256 = 8MB`입니다.
Linux는 bridge MMIO window를 segment alignment와 granularity에 맞춰 bridge 뒤의 space를 한 PE에 assign합니다. Function별 PE를 만들려면 BAR를 한 개 이상의 segment에 group하는 완전히 다른 allocation scheme이 필요합니다.
Outbound M64 window
110-159M64는 최소 256MB이며 PowerBus와 PCIe에서 같은 address를 사용해 translation하지 않습니다. 전달되지 않는 상위 14bit를 설정할 수 있지만 Linux는 사용하지 않습니다.
Unsegmented mode에서는 window 전체 PE#를 지정합니다. Segmented mode에서는 256 segment가 있고 별도 mapping table이 없으므로 segment number 자체가 PE#입니다. Window overlap도 지원하며 적용 우선순위가 정의되어 있습니다.
| 특성 | M32 | M64 |
|---|---|---|
| 개수 | PHB당 1 | PHB당 16 |
| 최소/최대 | 최대 4GB | 최소 256MB |
| Address | 상위 bit replace 가능 | PowerBus와 PCIe address 동일 |
| Segment | 256개, table로 PE# mapping | 256개, segment number = PE# |
| 주 용도 | 32-bit access와 일반 bridge MMIO | 64-bit large BAR 및 SR-IOV |
Linux는 firmware가 PHB에 배정한 약 64GB region 전체를 segmented M64 window로 덮고 bridge alignment 기법을 적용합니다.
Address가 PE#를 직접 결정하므로 64-bit space를 assign한 뒤 PE#를 배정합니다. 32-bit와 64-bit를 함께 쓰는 device는 M32 PE#도 갱신하고, 32-bit only device에는 남은 PE#를 할당합니다.
한 device가 여러 segment를 쓰면 여러 PE#가 생깁니다. Hardware companion-PE freeze cascade는 PCIe error message에만 적용되므로 software가 domain의 다른 PE도 freeze합니다. DMA/MSI용 `master PE`와 나머지 M64 segment용 `secondary PE`를 구분합니다.
매우 큰 GPU BAR 같은 경우 추가 M64 window를 single-PE mode로 overlay하는 방안이 있지만 문서 작성 시점에는 구현되지 않았습니다.
M32는 mapping table, segmented M64는 address-derived PE number를 사용합니다.
SR-IOV 배경
160-185PCIe SR-IOV는 하나의 Physical Function(PF)이 여러 Virtual Function(VF)을 제공하게 합니다. PF의 SR-IOV Capability register가 VF 수와 enable 상태를 제어합니다.
VF가 enable되면 normal PCI device처럼 Configuration Space에 나타나지만 VF header의 BAR는 read-only zero입니다. Software는 PF SR-IOV Capability의 VF BAR register에서 size를 알아내고 address를 assign합니다.
PF의 VF BAR 하나가 모든 대응 VF(n) BAR의 base를 정합니다. 예를 들어 1MB VF BAR0을 가진 VF 8개를 enable하면 8MB region이 1MB씩 여덟 구간으로 나뉩니다. 전체 region은 8MB지만 alignment 요구 사항은 단일 VF 크기인 1MB입니다.
VF isolation 전략
186-213| 전략 | 장점 | 제약 |
|---|---|---|
| M32 window | 각 segment를 lookup table로 개별 PE에 유연하게 mapping | 최소 1MB granularity이며 VF BAR 크기가 다르면 큰 BAR가 여러 segment 사용 |
| Non-segmented M64 | Window 전체를 한 PE에 단순 mapping | VF 하나만 isolate 가능 |
| Single segmented M64 | 256 segment를 PE#와 직접 대응 | 여러 BAR VF는 여러 PE domain이 되어 isolation 약화 |
| Multiple segmented M64 | Window별 base와 segment size를 달리해 1MB BAR와 32MB BAR 등을 분리 | 여러 M64 window 소비 |
VF BAR 크기 조합에 따라 window 수와 isolation granularity를 선택합니다.
M64 256-segment reservation
214-280목표는 VF마다 별도 PE를 주는 것입니다. IODA2의 M64 window는 MMIO range를 256 segment로 나누고 각 segment가 PE 하나에 대응하므로 동일 크기인 SR-IOV VF BAR mapping에 적합합니다.
하지만 `total_VFs`는 보통 256보다 작습니다. VF(n) BAR space만 직접 map하면 `[total_VFs, 255]` segment가 다른 device의 MMIO range와 겹칠 수 있습니다.
VF가 사용하는 앞쪽 segment 뒤의 M64 tail이 다른 MMIO와 겹칠 위험이 있습니다.
현재 해법은 VF(n) BAR가 실제로 필요로 하는 양보다 많더라도 256 segment 전체를 software에서 reserve하는 것입니다.
사용하지 않는 tail까지 같은 SR-IOV device에 reserve해 window overlap을 막습니다.
| 항목 | 식/범위 | 의미 |
|---|---|---|
| M64 segment 수 | 256 | IODA2 PE 수와 동일 |
| 실제 VF segment | `0 .. total_VFs-1` | VF가 실제 응답 |
| Extra reserve | `total_VFs .. 255` | 다른 device assignment 방지 |
Generic PCI code에 미치는 영향
281-312PCIe SR-IOV specification은 VF(n) BAR space base가 개별 VF BAR size에 align되도록 요구합니다.
M32는 segment→PE table을 바꿀 수 있고 unsegmented M64는 window PE#를 설정할 수 있지만 segmented M64에서는 segment number가 곧 PE#입니다. 따라서 VF PE#를 제어하는 유일한 방법은 PF VF BAR의 VF(n) BAR base를 바꾸는 것입니다.
PCI core가 정확히 필요한 크기만 allocate하면 VF BAR value를 움직일 수 없습니다. Extra space를 allocate하면 전체 VF(n) BAR space가 core allocation 안에 머무는 범위에서 base를 조절할 수 있습니다.
Segment size가 단일 VF BAR size와 같으면 VF마다 한 PE를 사용하고 PE#는 연속입니다. VF0가 `PE(x)`이면 VF(n)은 `PE(x+n)`이며 256 segment reservation에서는 VF0 PE#를 고를 수 있는 위치가 `(256 - numVFs)`개입니다.
Segment가 VF BAR보다 작으면 한 VF가 여러 PE를 사용해 isolation이 약해지고 VF(n) space가 `(numVFs * n)` segment를 소비하므로 base 조정 선택지도 줄어듭니다.
Extra allocation 안에서 BAR base를 이동해 연속 PE range를 고릅니다.
요약과 해설
pci_iov_resource_on_powernv.rst:1-312Segmented M64에서는 address가 곧 PE#를 결정합니다. 따라서 PCI core가 extra space를 reserve해야 VF BAR base를 움직여 VF마다 연속된 PE를 배정하고 남는 tail을 다른 device로부터 격리할 수 있습니다.