PCI NTB function

두 host를 연결하는 NTB endpoint

저자는 Kishon Vijay Abraham I입니다. PCI Non-Transparent Bridge(`NTB`)는 각 host를 상대 host에 device로 노출해 두 host system이 통신하게 합니다.

NTB는 일반적으로 remote machine interrupt 발생, memory range의 BAR 노출, DMA, 양쪽 machine에서 접근 가능한 NTB 내부 scratchpad를 지원합니다.

PCI NTB Function은 한 system의 transaction이 다른 system으로 route되도록 endpoint instance를 구성합니다. SoC의 여러 PCI EP instance에서 한 EP controller의 transaction을 다른 controller로 전달해 HOST1과 HOST2 사이 bridge 역할을 합니다.

.. SPDX-License-Identifier: GPL-2.0

=================
PCI NTB Function
=================

:Author: Kishon Vijay Abraham I <kishon@ti.com>

PCI Non-Transparent Bridges (NTB) allow two host systems to communicate
with each other by exposing each host as a device to the other host.
NTBs typically support the ability to generate interrupts on the remote
machine, expose memory ranges as BARs, and perform DMA.  They also support
scratchpads, which are areas of memory within the NTB that are accessible
from both machines.

PCI NTB Function allows two different systems (or hosts) to communicate
with each other by configuring the endpoint instances in such a way that
transactions from one system are routed to the other system.

In the below diagram, PCI NTB function configures the SoC with multiple
PCI Endpoint (EP) instances in such a way that transactions from one EP
controller are routed to the other EP controller. Once PCI NTB function
configures the SoC with multiple EP instances, HOST1 and HOST2 can
communicate with each other using SoC as a bridge.

.. code-block:: text

    +-------------+                                   +-------------+
    |             |                                   |             |
    |    HOST1    |                                   |    HOST2    |
    |             |                                   |             |
    +------^------+                                   +------^------+
           |                                                 |
           |                                                 |
 +---------|-------------------------------------------------|---------+
 |  +------v------+                                   +------v------+  |
 |  |             |                                   |             |  |
 |  |     EP      |                                   |     EP      |  |
 |  | CONTROLLER1 |                                   | CONTROLLER2 |  |
 |  |             <----------------------------------->             |  |
 |  |             |                                   |             |  |
 |  |             |                                   |             |  |
 |  |             |  SoC With Multiple EP Instances   |             |  |
 |  |             |  (Configured using NTB Function)  |             |  |
 |  +-------------+                                   +-------------+  |
 +---------------------------------------------------------------------+

NTB 구현 요소

Constructs used for Implementing NTB
====================================

        1) Config Region
        2) Self Scratchpad Registers
        3) Peer Scratchpad Registers
        4) Doorbell (DB) Registers
        5) Memory Window (MW)

Config Region layout

Config Region은 NTB Endpoint Function Driver 구현에 고유한 영역이며 host와 endpoint 측 NTB function driver가 정보를 교환합니다.

이 영역의 control/status register로 endpoint controller를 구성합니다. Host는 outbound Address Translation Unit(`ATU`) 설정과 link status 전달을 위해 기록하고, endpoint는 host command status, scratchpad offset, memory window 수를 알립니다.

Config Region의 모든 field는 32bit입니다. 64bit address는 lower와 upper 32bit로 나눕니다.

Config Region:
--------------

Config Region is a construct that is specific to NTB implemented using NTB
Endpoint Function Driver. The host and endpoint side NTB function driver will
exchange information with each other using this region. Config Region has
Control/Status Registers for configuring the Endpoint Controller. Host can
write into this region for configuring the outbound Address Translation Unit
(ATU) and to indicate the link status. Endpoint can indicate the status of
commands issued by host in this region. Endpoint can also indicate the
scratchpad offset and number of memory windows to the host using this region.

The format of Config Region is given below. All the fields here are 32 bits.

.. code-block:: text

        +------------------------+
        |         COMMAND        |
        +------------------------+
        |         ARGUMENT       |
        +------------------------+
        |         STATUS         |
        +------------------------+
        |         TOPOLOGY       |
        +------------------------+
        |    ADDRESS (LOWER 32)  |
        +------------------------+
        |    ADDRESS (UPPER 32)  |
        +------------------------+
        |           SIZE         |
        +------------------------+
        |   NO OF MEMORY WINDOW  |
        +------------------------+
        |  MEMORY WINDOW1 OFFSET |
        +------------------------+
        |       SPAD OFFSET      |
        +------------------------+
        |        SPAD COUNT      |
        +------------------------+
        |      DB ENTRY SIZE     |
        +------------------------+
        |         DB DATA        |
        +------------------------+
        |            :           |
        +------------------------+
        |            :           |
        +------------------------+
        |         DB DATA        |
        +------------------------+

COMMAND register

NTB function은 세 command를 지원합니다.

`CMD_CONFIGURE_DOORBELL`의 `ARGUMENT` lower 16bit에는 구성할 DB 수를 넣고 bit 16으로 MSI 또는 MSI-X를 선택합니다.

`CMD_CONFIGURE_MW` 전에 host가 remote host에서 접근 가능한 buffer를 할당해야 합니다. 64bit address는 `ADDRESS`, 크기는 `SIZE`, memory window index는 `ARGUMENT`에 넣습니다.

  COMMAND:

        NTB function supports three commands:

          CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
        invoking this command, the host should allocate and initialize
        MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
        Endpoint). The endpoint on receiving this command will configure
        the outbound ATU such that transactions to Doorbell BAR will be routed
        to the MSI/MSI-X address programmed by the host. The ARGUMENT
        register should be populated with number of DBs to configure (in the
        lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).

          CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
        host invokes this command after allocating a buffer that can be
        accessed by remote host. The allocated address should be programmed
        in the ADDRESS register (64 bit), the size should be programmed in
        the SIZE register and the memory window index should be programmed
        in the ARGUMENT register. The endpoint on receiving this command
        will configure the outbound ATU such that transactions to MW BAR
        are routed to the address provided by the host.

          CMD_LINK_UP (0x3): Command to indicate an NTB application is
        bound to the EP device on the host side. Once the endpoint
        receives this command from both the hosts, the endpoint will
        raise a LINK_UP event to both the hosts to indicate the host
        NTB applications can start communicating with each other.

Config field 세부 의미

`ARGUMENT` 값은 `COMMAND`에 따라 달라집니다. `TOPOLOGY`는 primary interface에서 `NTB_TOPO_B2B_USD`, secondary interface에서 `NTB_TOPO_B2B_DSD`로 설정합니다.

`ADDRESS/SIZE`는 `CMD_CONFIGURE_MW`가 memory window를 구성할 때 사용합니다.

Memory Window 1과 doorbell register는 같은 BAR에 pack됩니다. 앞부분은 doorbell이고 뒷부분은 MW1이며 `MEMORY WINDOW1 OFFSET`이 MW1 시작 위치를 지정합니다. `NO OF MEMORY WINDOW`는 지원하는 MW 수입니다.

Self scratchpad와 config region도 같은 BAR에 pack됩니다. 앞부분은 config region이고 뒷부분은 self scratchpad이며 `SPAD OFFSET`이 scratchpad 시작 위치를, `SPAD COUNT`가 register 수를 지정합니다.

`DB ENTRY SIZE`는 doorbell을 울리기 위해 DB BAR 안에서 기록할 offset 간격을 정합니다. EPF NTB는 MSI 또는 MSI-X를 사용할 수 있으며 원문 시점에는 MSI-X 지원이 추후 추가될 예정입니다.

MSI는 모든 interrupt에 같은 address를 사용하지만 MSI-X는 서로 다른 address를 제공할 수 있습니다. ARM GIC ITS처럼 모든 MSI-X interrupt에 같은 address를 쓰는 platform도 있습니다.

양쪽 경우를 동일하게 처리하려고 EPF NTB는 interrupt마다 별도 outbound address-space region을 만들고 host가 제공한 MSI/MSI-X address에 mapping합니다. Host address가 같으면 모든 region이 같은 주소로, 다르면 각각 다른 주소로 translate되므로 doorbell 발생 방식은 같게 유지됩니다.

EPF NTB는 interrupt 32개를 지원하므로 `DB DATA` register도 32개입니다. 각 register는 doorbell interrupt 발생 시 MSI address에 쓸 MSI/MSI-X data를 보관하며 `CMD_CONFIGURE_DOORBELL` 처리 때 채웁니다.

  ARGUMENT:

        The value of this register is based on the commands issued in
        command register. See COMMAND section for more information.

  TOPOLOGY:

        Set to NTB_TOPO_B2B_USD for Primary interface
        Set to NTB_TOPO_B2B_DSD for Secondary interface

  ADDRESS/SIZE:

        Address and Size to be used while configuring the memory window.
        See "CMD_CONFIGURE_MW" for more info.

  MEMORY WINDOW1 OFFSET:

        Memory Window 1 and Doorbell registers are packed together in the
        same BAR. The initial portion of the region will have doorbell
        registers and the latter portion of the region is for memory window 1.
        This register will specify the offset of the memory window 1.

  NO OF MEMORY WINDOW:

        Specifies the number of memory windows supported by the NTB device.

  SPAD OFFSET:

        Self scratchpad region and config region are packed together in the
        same BAR. The initial portion of the region will have config region
        and the latter portion of the region is for self scratchpad. This
        register will specify the offset of the self scratchpad registers.

  SPAD COUNT:

        Specifies the number of scratchpad registers supported by the NTB
        device.

  DB ENTRY SIZE:

        Used to determine the offset within the DB BAR that should be written
        in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
        ring doorbell (MSI-X support will be added later). MSI uses same
        address for all the interrupts and MSI-X can provide different
        addresses for different interrupts. The MSI/MSI-X address is provided
        by the host and the address it gives is based on the MSI/MSI-X
        implementation supported by the host. For instance, ARM platform
        using GIC ITS will have the same MSI-X address for all the interrupts.
        In order to support all the combinations and use the same mechanism
        for both MSI and MSI-X, EPF NTB allocates a separate region in the
        Outbound Address Space for each of the interrupts. This region will
        be mapped to the MSI/MSI-X address provided by the host. If a host
        provides the same address for all the interrupts, all the regions
        will be translated to the same address. If a host provides different
        addresses, the regions will be translated to different addresses. This
        will ensure there is no difference while raising the doorbell.

  DB DATA:

        EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
        This holds the MSI/MSI-X data that has to be written to MSI address
        for raising doorbell interrupt. This will be populated by EPF NTB
        while invoking CMD_CONFIGURE_DOORBELL.

Scratchpad, doorbell과 memory window

각 host는 NTB endpoint controller memory에 자체 register space를 가지며 bridge 양쪽에서 읽고 쓸 수 있습니다. NTB application은 이 scratchpad로 control·status 정보를 교환합니다.

Scratchpad Registers:
---------------------

  Each host has its own register space allocated in the memory of NTB endpoint
  controller. They are both readable and writable from both sides of the bridge.
  They are used by applications built over NTB and can be used to pass control
  and status information between both sides of a device.

  Scratchpad registers has 2 parts
        1) Self Scratchpad: Host's own register space
        2) Peer Scratchpad: Remote host's register space.

Doorbell Registers:
-------------------

  Doorbell Registers are used by the hosts to interrupt each other.

Memory Window:
--------------

  Actual transfer of data between the two hosts will happen using the
  memory window.

NTB construct의 BAR packing

NTB에는 config, self scratchpad, peer scratchpad, doorbell, 하나 이상의 memory window 등 최소 다섯 region이 필요합니다. Host가 접근할 수 있도록 모두 BAR에 mapping해야 하며 MW는 최소 하나가 필요하고 여러 개도 허용됩니다.

Region마다 BAR를 하나씩 배정하면 64bit BAR만 지원하는 platform에서는 BAR가 부족합니다. 대부분의 platform을 지원하면서 host가 접근하면 안 되는 region을 차단하려면 region을 pack해 mapping해야 합니다.

이 layout에서는 기본 NTB 기능에 BAR 세 개면 충분합니다.

Modeling Constructs:
====================

There are 5 or more distinct regions (config, self scratchpad, peer
scratchpad, doorbell, one or more memory windows) to be modeled to achieve
NTB functionality. At least one memory window is required while more than
one is permitted. All these regions should be mapped to BARs for hosts to
access these regions.

If one 32-bit BAR is allocated for each of these regions, the scheme would
look like this:

======  ===============
BAR NO  CONSTRUCTS USED
======  ===============
BAR0    Config Region
BAR1    Self Scratchpad
BAR2    Peer Scratchpad
BAR3    Doorbell
BAR4    Memory Window 1
BAR5    Memory Window 2
======  ===============

However if we allocate a separate BAR for each of the regions, there would not
be enough BARs for all the regions in a platform that supports only 64-bit
BARs.

In order to be supported by most of the platforms, the regions should be
packed and mapped to BARs in a way that provides NTB functionality and
also makes sure the host doesn't access any region that it is not supposed
to.

The following scheme is used in EPF NTB Function:

======  ===============================
BAR NO  CONSTRUCTS USED
======  ===============================
BAR0    Config Region + Self Scratchpad
BAR1    Peer Scratchpad
BAR2    Doorbell + Memory Window 1
BAR3    Memory Window 2
BAR4    Memory Window 3
BAR5    Memory Window 4
======  ===============================

With this scheme, for the basic NTB functionality 3 BARs should be sufficient.

Config와 scratchpad mapping

HOST1에 대응하는 config region과 scratchpad region은 local memory에 함께 할당됩니다. HOST1은 EP controller 1의 BAR0으로 config와 self scratchpad에 접근합니다.

HOST2는 EP controller 2의 BAR1을 통해 같은 scratchpad를 peer scratchpad로 접근합니다. 그림은 HOST1의 영역을 예로 들지만 HOST2 영역도 대칭으로 구성합니다.

Modeling Config/Scratchpad Region:
----------------------------------

.. code-block:: text

 +-----------------+------->+------------------+        +-----------------+
 |       BAR0      |        |  CONFIG REGION   |        |       BAR0      |
 +-----------------+----+   +------------------+<-------+-----------------+
 |       BAR1      |    |   |SCRATCHPAD REGION |        |       BAR1      |
 +-----------------+    +-->+------------------+<-------+-----------------+
 |       BAR2      |            Local Memory            |       BAR2      |
 +-----------------+                                    +-----------------+
 |       BAR3      |                                    |       BAR3      |
 +-----------------+                                    +-----------------+
 |       BAR4      |                                    |       BAR4      |
 +-----------------+                                    +-----------------+
 |       BAR5      |                                    |       BAR5      |
 +-----------------+                                    +-----------------+
   EP CONTROLLER 1                                        EP CONTROLLER 2

Above diagram shows Config region + Scratchpad region for HOST1 (connected to
EP controller 1) allocated in local memory. The HOST1 can access the config
region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
The peer host (HOST2 connected to EP controller 2) can also access this
scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
diagram shows the case where Config region and Scratchpad regions are allocated
for HOST1, however the same is applicable for HOST2.

Doorbell과 MW1 outbound mapping

Doorbell과 Memory Window 1을 mapping하면 HOST1이 HOST2에 doorbell interrupt를 발생시키고 HOST2가 공개한 buffer를 MW1으로 접근할 수 있습니다.

Doorbell과 MW1 region은 EP controller 2의 outbound(`OB`) address space에 할당합니다. BAR 할당과 region 구성은 NTB endpoint function driver 초기화 단계에서 수행합니다.

HOST2가 `CMD_CONFIGURE_MW`와 `CMD_CONFIGURE_DOORBELL`을 보내면 EP controller 2 OB space에서 HOST2가 관리하는 PCI address space로 mapping합니다.

Modeling Doorbell/Memory Window 1:
----------------------------------

.. code-block:: text

 +-----------------+    +----->+----------------+-----------+-----------------+
 |       BAR0      |    |      |   Doorbell 1   +-----------> MSI-X ADDRESS 1 |
 +-----------------+    |      +----------------+           +-----------------+
 |       BAR1      |    |      |   Doorbell 2   +---------+ |                 |
 +-----------------+----+      +----------------+         | |                 |
 |       BAR2      |           |   Doorbell 3   +-------+ | +-----------------+
 +-----------------+----+      +----------------+       | +-> MSI-X ADDRESS 2 |
 |       BAR3      |    |      |   Doorbell 4   +-----+ |   +-----------------+
 +-----------------+    |      |----------------+     | |   |                 |
 |       BAR4      |    |      |                |     | |   +-----------------+
 +-----------------+    |      |      MW1       +---+ | +-->+ MSI-X ADDRESS 3||
 |       BAR5      |    |      |                |   | |     +-----------------+
 +-----------------+    +----->-----------------+   | |     |                 |
   EP CONTROLLER 1             |                |   | |     +-----------------+
                               |                |   | +---->+ MSI-X ADDRESS 4 |
                               +----------------+   |       +-----------------+
                                EP CONTROLLER 2     |       |                 |
                                  (OB SPACE)        |       |                 |
                                                    +------->      MW1        |
                                                            |                 |
                                                            |                 |
                                                            +-----------------+
                                                            |                 |
                                                            |                 |
                                                            |                 |
                                                            |                 |
                                                            |                 |
                                                            +-----------------+
                                                             PCI Address Space
                                                             (Managed by HOST2)

Above diagram shows how the doorbell and memory window 1 is mapped so that
HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
memory window 1 regions are allocated in EP controller 2 outbound (OB) address
space. Allocating and configuring BARs for doorbell and memory window1
is done during the initialization phase of NTB endpoint function driver.
Mapping from EP controller 2 OB space to PCI address space is done when HOST2
sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.

선택적 memory window

추가 memory window도 MW1과 같은 방식으로 modeling하지만 각각 별도 BAR에 mapping합니다.

Modeling Optional Memory Windows:
---------------------------------

This is modeled the same was as MW1 but each of the additional memory windows
is mapped to separate BARs.