PCI Express Port Bus 드라이버 안내서 HOWTO

안내서 소개

이 문서는 GPL-2.0으로 배포되며 저자는 Tom L Nguyen, 저작권자는 2004년 Intel Corporation입니다.

이 안내서는 PCI Express Port Bus 드라이버의 기본 원리를 설명하고, service driver가 PCI Express Port Bus Driver에 등록하거나 등록을 해제하도록 구현하는 방법을 제공합니다.

.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

===========================================
The PCI Express Port Bus Driver Guide HOWTO
===========================================

:Author: Tom L Nguyen tom.l.nguyen@intel.com 11/03/2004
:Copyright: |copy| 2004 Intel Corporation

About this guide
================

This guide describes the basics of the PCI Express Port Bus driver
and provides information on how to enable the service drivers to
register/unregister with the PCI Express Port Bus Driver.

PCI Express Port와 service

PCI Express Port는 논리적인 PCI-PCI Bridge 구조입니다. PCI Express Port에는 Root Port와 Switch Port 두 종류가 있습니다.

Root Port는 PCI Express Root Complex에서 PCI Express link를 시작합니다. Switch Port는 PCI Express link를 switch 내부의 논리 PCI bus에 연결합니다.

Secondary bus가 switch 내부 routing logic을 나타내는 Switch Port를 switch의 Upstream Port라고 합니다. Downstream Port는 switch 내부 routing bus에서 PCI Express Switch의 downstream PCI Express link를 나타내는 bus로 bridge합니다.

PCI Express Port는 port type에 따라 서로 구별되는 기능을 최대 네 개까지 제공할 수 있으며 이 문서에서는 이를 service라고 부릅니다.

Service에는 native hotplug(HP), power management event(PME), Advanced Error Reporting(AER), Virtual Channel(VC)이 있습니다. 하나의 복합 driver가 모두 처리할 수도 있고 각각을 대응하는 service driver에 분배할 수도 있습니다.

What is the PCI Express Port Bus Driver
=======================================

A PCI Express Port is a logical PCI-PCI Bridge structure. There
are two types of PCI Express Port: the Root Port and the Switch
Port. The Root Port originates a PCI Express link from a PCI Express
Root Complex and the Switch Port connects PCI Express links to
internal logical PCI buses. The Switch Port, which has its secondary
bus representing the switch's internal routing logic, is called the
switch's Upstream Port. The switch's Downstream Port is bridging from
switch's internal routing bus to a bus representing the downstream
PCI Express link from the PCI Express Switch.

A PCI Express Port can provide up to four distinct functions,
referred to in this document as services, depending on its port type.
PCI Express Port's services include native hotplug support (HP),
power management event support (PME), advanced error reporting
support (AER), and virtual channel support (VC). These services may
be handled by a single complex driver or be individually distributed
and handled by corresponding service drivers.

Port Bus 드라이버가 필요한 이유

기존 Linux Device Driver Model에서는 하나의 물리 device를 driver 하나만 처리할 수 있습니다. 그러나 PCI Express Port는 여러 독립 service를 가진 하나의 PCI-PCI Bridge device입니다.

각 service를 단순하고 명확하게 유지하려면 별도 software service driver가 필요하지만, 그러면 여러 service driver가 같은 PCI-PCI Bridge device를 두고 경쟁하게 됩니다.

예를 들어 Root Port native hotplug service driver가 먼저 load되면 PCI-PCI Bridge Root Port를 claim합니다. Kernel은 그 Root Port에 다른 service driver를 load하지 않으므로 현재 driver model만으로는 여러 service driver를 동시에 실행할 수 없습니다.

PCI Express Port Bus driver는 존재하는 모든 PCI Express Port를 관리하고 각 service 요청을 대응하는 service driver에 분배하여 이 제약을 해결합니다.

이 구조를 사용하면 여러 service driver를 한 Port에서 동시에 실행하고, 각 service를 독립적인 단계로 구현하며, 하나의 service driver를 여러 PCI-PCI Bridge Port에서 실행할 수 있습니다. 또한 Port device의 resource를 요청한 service driver에 관리·분배할 수 있습니다.

Why use the PCI Express Port Bus Driver?
========================================

In existing Linux kernels, the Linux Device Driver Model allows a
physical device to be handled by only a single driver. The PCI
Express Port is a PCI-PCI Bridge device with multiple distinct
services. To maintain a clean and simple solution each service
may have its own software service driver. In this case several
service drivers will compete for a single PCI-PCI Bridge device.
For example, if the PCI Express Root Port native hotplug service
driver is loaded first, it claims a PCI-PCI Bridge Root Port. The
kernel therefore does not load other service drivers for that Root
Port. In other words, it is impossible to have multiple service
drivers load and run on a PCI-PCI Bridge device simultaneously
using the current driver model.

To enable multiple service drivers running simultaneously requires
having a PCI Express Port Bus driver, which manages all populated
PCI Express Ports and distributes all provided service requests
to the corresponding service drivers as required. Some key
advantages of using the PCI Express Port Bus driver are listed below:

  - Allow multiple service drivers to run simultaneously on
    a PCI-PCI Bridge Port device.

  - Allow service drivers implemented in an independent
    staged approach.

  - Allow one service driver to run on multiple PCI-PCI Bridge
    Port devices.

  - Manage and distribute resources of a PCI-PCI Bridge Port
    device to requested service drivers.

Kernel에 Port Bus 지원 포함하기

PCI Express Port Bus 드라이버 포함 여부는 kernel configuration에서 PCI Express 지원을 포함했는지에 따라 결정됩니다.

Kernel에서 PCI Express 지원을 활성화하면 PCI Express Port Bus driver가 kernel driver로 자동 포함됩니다.

Configuring the PCI Express Port Bus Driver vs. Service Drivers
===============================================================

Including the PCI Express Port Bus Driver Support into the Kernel
-----------------------------------------------------------------

Including the PCI Express Port Bus driver depends on whether the PCI
Express support is included in the kernel config. The kernel will
automatically include the PCI Express Port Bus driver as a kernel
driver when the PCI Express support is enabled in the kernel.

Service driver 지원 활성화

모든 service driver는 Linux Device Driver Model을 기반으로 한 PCI device driver입니다. 그러나 Port Bus driver가 물리 Port를 이미 소유하므로 기존 방식 그대로는 service driver를 load할 수 없습니다.

기존 service driver에는 기능에 영향을 주지 않는 최소한의 변경만 적용하여 PCI Express Port Bus Driver Model에 맞출 수 있습니다.

Service driver는 아래의 두 API로 Port Bus driver에 service를 등록하고 해제해야 합니다. API를 호출하기 전에 `/include/linux/pcieport_if.h`에 정의된 `pcie_port_service_driver` data structure를 반드시 초기화해야 합니다.

초기화하지 않으면 identity mismatch가 발생해 PCI Express Port Bus driver가 해당 service driver를 load하지 못합니다.

Enabling Service Driver Support
-------------------------------

PCI device drivers are implemented based on Linux Device Driver Model.
All service drivers are PCI device drivers. As discussed above, it is
impossible to load any service driver once the kernel has loaded the
PCI Express Port Bus Driver. To meet the PCI Express Port Bus Driver
Model requires some minimal changes on existing service drivers that
imposes no impact on the functionality of existing service drivers.

A service driver is required to use the two APIs shown below to
register its service with the PCI Express Port Bus driver (see
section 5.2.1 & 5.2.2). It is important that a service driver
initializes the pcie_port_service_driver data structure, included in
header file /include/linux/pcieport_if.h, before calling these APIs.
Failure to do so will result an identity mismatch, which prevents
the PCI Express Port Bus driver from loading a service driver.

Service 등록·해제 API

`pcie_port_service_register()`는 Linux Driver Model의 `pci_register_driver()`를 대신합니다. Service driver는 module 초기화 때 항상 이 함수를 호출해야 합니다.

int pcie_port_service_register(struct pcie_port_service_driver *new)

Service driver가 load된 뒤에는 `pci_enable_device(dev)`와 `pci_set_master(dev)`를 호출할 필요가 없습니다. PCI Port Bus driver가 이 작업을 이미 수행하기 때문입니다.

`pcie_port_service_unregister()`는 `pci_unregister_driver()`를 대신하며 service driver module이 종료될 때 항상 호출합니다.

void pcie_port_service_unregister(struct pcie_port_service_driver *new)

pcie_port_service_register
~~~~~~~~~~~~~~~~~~~~~~~~~~
::

  int pcie_port_service_register(struct pcie_port_service_driver *new)

This API replaces the Linux Driver Model's pci_register_driver API. A
service driver should always calls pcie_port_service_register at
module init. Note that after service driver being loaded, calls
such as pci_enable_device(dev) and pci_set_master(dev) are no longer
necessary since these calls are executed by the PCI Port Bus driver.

pcie_port_service_unregister
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::

  void pcie_port_service_unregister(struct pcie_port_service_driver *new)

pcie_port_service_unregister replaces the Linux Driver Model's
pci_unregister_driver. It's always called by service driver when a
module exits.

AER service driver 예제

첫 예제는 AER Root Port service driver가 사용할 ID table과 `struct pcie_port_service_driver`를 초기화합니다.

ID table은 vendor와 device를 `PCI_ANY_ID`로 두고 port type을 `PCIE_RC_PORT`, service type을 `PCIE_PORT_SERVICE_AER`로 제한한 뒤 all-zero entry로 끝냅니다.

static struct pcie_port_service_id service_id[] = { {
  .vendor = PCI_ANY_ID,
  .device = PCI_ANY_ID,
  .port_type = PCIE_RC_PORT,
  .service_type = PCIE_PORT_SERVICE_AER,
  }, { /* end: all zeroes */ }
};

`root_aerdrv`는 이름과 ID table, `aerdrv_load`·`aerdrv_unload` probe/remove callback, `aerdrv_suspend`·`aerdrv_resume` power-management callback을 연결합니다.

static struct pcie_port_service_driver root_aerdrv = {
  .name      = (char *)device_name,
  .id_table  = service_id,
  .probe     = aerdrv_load,
  .remove    = aerdrv_unload,
  .suspend   = aerdrv_suspend,
  .resume    = aerdrv_resume,
};

Module init 함수 `aerdrv_service_init()`는 `pcie_port_service_register(&root_aerdrv)`를 호출하고 반환값을 전달합니다. Module exit 함수 `aerdrv_service_exit()`는 같은 구조체를 `pcie_port_service_unregister()`에 넘깁니다.

static int __init aerdrv_service_init(void)
{
  int retval = 0;

  retval = pcie_port_service_register(&root_aerdrv);
  if (!retval) {
    /*
    * FIX ME
    */
  }
  return retval;
}

static void __exit aerdrv_service_exit(void)
{
  pcie_port_service_unregister(&root_aerdrv);
}

module_init(aerdrv_service_init);
module_exit(aerdrv_service_exit);

Sample Code
~~~~~~~~~~~

Below is sample service driver code to initialize the port service
driver data structure.
::

  static struct pcie_port_service_id service_id[] = { {
    .vendor = PCI_ANY_ID,
    .device = PCI_ANY_ID,
    .port_type = PCIE_RC_PORT,
    .service_type = PCIE_PORT_SERVICE_AER,
    }, { /* end: all zeroes */ }
  };

  static struct pcie_port_service_driver root_aerdrv = {
    .name                = (char *)device_name,
    .id_table        = service_id,

    .probe                = aerdrv_load,
    .remove                = aerdrv_unload,

    .suspend        = aerdrv_suspend,
    .resume                = aerdrv_resume,
  };

Below is a sample code for registering/unregistering a service
driver.
::

  static int __init aerdrv_service_init(void)
  {
    int retval = 0;

    retval = pcie_port_service_register(&root_aerdrv);
    if (!retval) {
      /*
      * FIX ME
      */
    }
    return retval;
  }

  static void __exit aerdrv_service_exit(void)
  {
    pcie_port_service_unregister(&root_aerdrv);
  }

  module_init(aerdrv_service_init);
  module_exit(aerdrv_service_exit);

MSI와 MSI-X vector 충돌 방지

한 PCI-PCI Bridge Port의 모든 service driver가 동시에 실행되므로 공유 resource 충돌을 조정해야 합니다.

Device에서 MSI 또는 MSI-X interrupt를 enable하면 다시 disable할 때까지 그 mode가 유지됩니다. 같은 Port의 service driver들은 하나의 물리 device를 공유하므로 개별 service driver가 MSI/MSI-X mode를 바꾸면 예측할 수 없는 동작이 생길 수 있습니다.

따라서 service driver는 device의 interrupt mode를 전환할 수 없습니다. PCI Express Port Bus driver가 mode를 결정하며 이 과정은 service driver에 투명해야 합니다.

Service driver가 알아야 하는 것은 probe 때 전달된 `struct pcie_device`의 `irq` field에 할당된 vector IRQ뿐입니다. `request_irq()`와 `free_irq()`에는 `(struct pcie_device *)dev->irq`를 사용합니다.

선택된 interrupt mode는 `struct pcie_device`의 `interrupt_mode` field에도 저장됩니다.

Possible Resource Conflicts
===========================

Since all service drivers of a PCI-PCI Bridge Port device are
allowed to run simultaneously, below lists a few of possible resource
conflicts with proposed solutions.

MSI and MSI-X Vector Resource
-----------------------------

Once MSI or MSI-X interrupts are enabled on a device, it stays in this
mode until they are disabled again.  Since service drivers of the same
PCI-PCI Bridge port share the same physical device, if an individual
service driver enables or disables MSI/MSI-X mode it may result
unpredictable behavior.

To avoid this situation all service drivers are not permitted to
switch interrupt mode on its device. The PCI Express Port Bus driver
is responsible for determining the interrupt mode and this should be
transparent to service drivers. Service drivers need to know only
the vector IRQ assigned to the field irq of struct pcie_device, which
is passed in when the PCI Express Port Bus driver probes each service
driver. Service drivers should use (struct pcie_device*)dev->irq to
call request_irq/free_irq. In addition, the interrupt mode is stored
in the field interrupt_mode of struct pcie_device.

PCI Memory·I/O mapped region

PCI Express PME, AER, HP, VC service driver는 모두 PCI Express Port의 PCI configuration space에 접근합니다.

각 service가 접근하는 register는 서로 독립적입니다. 이 설계는 모든 service driver가 올바르게 동작하여 다른 service driver의 configuration setting을 덮어쓰지 않는다고 가정합니다.

PCI Memory/IO Mapped Regions
----------------------------

Service drivers for PCI Express Power Management (PME), Advanced
Error Reporting (AER), Hot-Plug (HP) and Virtual Channel (VC) access
PCI configuration space on the PCI Express port. In all cases the
registers accessed are independent of each other. This patch assumes
that all service drivers will be well behaved and not overwrite
other service driver's configuration settings.

공유 PCI Express Capability register

각 service driver는 자기 capability structure에서 PCI config operation을 수행합니다. 예외는 service driver를 포함한 여러 driver가 공유하는 PCI Express capability structure입니다.

Read-modify-write(RMW) capability accessor인 `pcie_capability_clear_and_set_word()`, `pcie_capability_set_word()`, `pcie_capability_clear_word()`는 선택된 PCI Express Capability Register를 동시 갱신으로부터 보호합니다.

보호 대상에는 Link Control Register, Root Control Register, Link Control 2 Register가 포함됩니다.

이 register를 변경할 때는 concurrent update 문제를 피하도록 반드시 RMW accessor를 사용해야 합니다. 최신 보호 register 목록은 `pcie_capability_clear_and_set_word()` 구현을 참조하십시오.

PCI Config Registers
--------------------

Each service driver runs its PCI config operations on its own
capability structure except the PCI Express capability structure,
that is shared between many drivers including the service drivers.
RMW Capability accessors (pcie_capability_clear_and_set_word(),
pcie_capability_set_word(), and pcie_capability_clear_word()) protect
a selected set of PCI Express Capability Registers:

* Link Control Register
* Root Control Register
* Link Control 2 Register

Any change to those registers should be performed using RMW accessors to
avoid problems due to concurrent updates. For the up-to-date list of
protected registers, see pcie_capability_clear_and_set_word().