Boot interrupt

PCIe legacy INTx routing 개요

저자는 Sean V Kelley입니다. PCI Express의 interrupt는 MSI 또는 inbound interrupt message인 `Assert_INTx/Deassert_INTx`로 표현됩니다.

Core IO에 통합된 IO-APIC은 PCI Express의 legacy interrupt message를 MSI interrupt로 변환합니다. IO-APIC table entry의 mask bit로 IO-APIC을 비활성화하면 message가 legacy PCH로 route됩니다.

이 in-band interrupt mechanism은 IO-APIC을 지원하지 않는 system과 boot 과정에 전통적으로 필요했습니다. Intel은 과거 이 mechanism을 `boot interrupt`라고 불렀습니다.

PCI Express protocol은 I/O device가 PCI 방식 level interrupt를 신호하는 in-band legacy wire-interrupt INTx mechanism으로 이를 정의합니다. 이 문서는 Core IO가 INTx message를 PCH로 route할 때의 문제와 BIOS·OS mitigation을 설명합니다.

.. SPDX-License-Identifier: GPL-2.0

===============
Boot Interrupts
===============

:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>

Overview
========

On PCI Express, interrupts are represented with either MSI or inbound
interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a
given Core IO converts the legacy interrupt messages from PCI Express to
MSI interrupts.  If the IO-APIC is disabled (via the mask bits in the
IO-APIC table entries), the messages are routed to the legacy PCH. This
in-band interrupt mechanism was traditionally necessary for systems that
did not support the IO-APIC and for boot. Intel in the past has used the
term "boot interrupts" to describe this mechanism. Further, the PCI Express
protocol describes this in-band legacy wire-interrupt INTx mechanism for
I/O devices to signal PCI-style level interrupts. The subsequent paragraphs
describe problems with the Core IO handling of INTx message routing to the
PCH and mitigation within BIOS and the OS.

Unhandled interrupt와 IRQ 비활성화

In-band legacy INTx message가 PCH로 전달되면 PCH는 OS에 handler가 없을 가능성이 높은 새 interrupt를 발생시킵니다.

처리되지 않은 interrupt가 누적되면 Linux kernel은 이를 Spurious Interrupt로 추적합니다. 특정 횟수에 도달하면 `nobody cared` error와 함께 IRQ를 비활성화합니다.

비활성화된 IRQ line을 정상 interrupt가 공유하고 있었다면 그 정상 사용까지 막힙니다. 원문 예제에서는 IRQ 19에서 call trace와 등록 handler를 출력한 뒤 `Disabling IRQ #19`를 보고합니다.

irq 19: nobody cared (try booting with the "irqpoll" option)
...
handlers:
irq_default_primary_handler threaded usb_hcd_irq
Disabling IRQ #19

Issue
=====

When in-band legacy INTx messages are forwarded to the PCH, they in turn
trigger a new interrupt for which the OS likely lacks a handler. When an
interrupt goes unhandled over time, they are tracked by the Linux kernel as
Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it
reaches a specific count with the error "nobody cared". This disabled IRQ
now prevents valid usage by an existing interrupt which may happen to share
the IRQ line::

  irq 19: nobody cared (try booting with the "irqpoll" option)
  CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
  Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
  Call Trace:

  <IRQ>
   ? dump_stack+0x46/0x5e
   ? __report_bad_irq+0x2e/0xb0
   ? note_interrupt+0x242/0x290
   ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
   ? handle_irq_event_percpu+0x55/0x70
   ? handle_irq_event+0x4f/0x80
   ? handle_fasteoi_irq+0x81/0x180
   ? handle_irq+0x1c/0x30
   ? do_IRQ+0x41/0xd0
   ? common_interrupt+0x84/0x84
  </IRQ>

  handlers:
  irq_default_primary_handler threaded usb_hcd_irq
  Disabling IRQ #19

Threaded interrupt 조건

현재 이 문제를 일으킬 가능성이 가장 높은 조건은 threaded interrupt 사용입니다. IRQ handler가 깨어난 뒤 threaded interrupt가 다시 활성화되지 않을 수 있습니다.

이런 one-shot 조건에서는 threaded handler가 실행될 때까지 interrupt line을 masked 상태로 유지해야 합니다.

특히 data rate가 높은 interrupt에서는 thread가 끝까지 실행돼야 합니다. Issuing device의 interrupt가 계속 active한 상태에서 완료되지 않으면 일부 handler가 stack overflow에 이를 수 있습니다.

Conditions
==========

The use of threaded interrupts is the most likely condition to trigger
this problem today. Threaded interrupts may not be re-enabled after the IRQ
handler wakes. These "one shot" conditions mean that the threaded interrupt
needs to keep the interrupt line masked until the threaded handler has run.
Especially when dealing with high data rate interrupts, the thread needs to
run to completion; otherwise some handlers will end up in stack overflows
since the interrupt of the issuing device is still active.

영향받는 chipset

Legacy interrupt forwarding mechanism은 AMD/ATI, Broadcom, Intel chipset을 포함한 여러 device에 존재합니다. 아래 mitigation 변경은 `drivers/pci/quirks.c`에 적용돼 있습니다.

ICX부터 Core IO device에는 IO-APIC이 없고 PCH에만 존재합니다. Core IO의 PCIe Root Port에 연결된 device는 native MSI/MSI-X mechanism을 사용합니다.

Affected Chipsets
=================

The legacy interrupt forwarding mechanism exists today in a number of
devices including but not limited to chipsets from AMD/ATI, Broadcom, and
Intel. Changes made through the mitigations below have been applied to
drivers/pci/quirks.c

Starting with ICX there are no longer any IO-APICs in the Core IO's
devices.  IO-APIC is only in the PCH.  Devices connected to the Core IO's
PCIe Root Ports will use native MSI/MSI-X mechanisms.

PCH routing과 boot interrupt 비활성화

Mitigation은 PCI quirk 형태입니다. 우선 PCH routing을 끄는 방법을 찾아 사용하며, 가능하면 boot interrupt 생성을 비활성화하는 quirk를 추가합니다[1].

Intel 6300ESB I/O Controller Hub의 Alternate Base Address Register에는 `BIE`(Boot Interrupt Enable)가 있습니다.

Sandy Bridge부터 Sky Lake 기반 Xeon server의 Coherent Interface Protocol Interrupt Control에는 `dis_intx_route2pch`, `dis_intx_route2ich`, `dis_intx_route2dmi2`가 있습니다.

이 bit를 설정하면 Intel Quick Data DMA/PCI Express port에서 받은 local INTx message를 legacy PCH로 route하지 않습니다. 적절한 IO-APIC entry의 mask bit가 clear면 integrated IO-APIC이 MSI로 변환하고, mask bit가 set이면 추가 동작을 하지 않습니다.

Mitigations
===========

The mitigations take the form of PCI quirks. The preference has been to
first identify and make use of a means to disable the routing to the PCH.
In such a case a quirk to disable boot interrupt generation can be
added. [1]_

Intel® 6300ESB I/O Controller Hub
  Alternate Base Address Register:
   BIE: Boot Interrupt Enable

          ==  ===========================
          0   Boot interrupt is enabled.
          1   Boot interrupt is disabled.
          ==  ===========================

Intel® Sandy Bridge through Sky Lake based Xeon servers:
  Coherent Interface Protocol Interrupt Control
   dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
          When this bit is set. Local INTx messages received from the
          Intel® Quick Data DMA/PCI Express ports are not routed to legacy
          PCH - they are either converted into MSI via the integrated IO-APIC
          (if the IO-APIC mask bit is clear in the appropriate entries)
          or cause no further action (when mask bit is set)

INTx handler rerouting

PCH routing을 직접 비활성화할 방법이 없으면 PCI Interrupt pin-to-INTx routing table을 이용해 기본 interrupt handler를 reroute된 interrupt line으로 redirect할 수 있습니다.

INTx routing을 끌 수 없는 chipset에서 Linux kernel은 정상 interrupt를 legacy interrupt로 reroute합니다. 이 handler redirect는 unhandled count가 쌓여 IRQ line이 비활성화되는 spurious interrupt detection을 방지합니다[2].

`X86_REROUTE_FOR_BROKEN_BOOT_IRQS` config option으로 handler를 PCH interrupt line으로 redirect하는 기능을 켜거나 끕니다. Kernel command line의 `pci=ioapicreroute` 또는 `pci=noioapicreroute`로 override할 수 있습니다[3].

In the absence of a way to directly disable the routing, another approach
has been to make use of PCI Interrupt pin to INTx routing tables for
purposes of redirecting the interrupt handler to the rerouted interrupt
line by default.  Therefore, on chipsets where this INTx routing cannot be
disabled, the Linux kernel will reroute the valid interrupt to its legacy
interrupt. This redirection of the handler will prevent the occurrence of
the spurious interrupt detection which would ordinarily disable the IRQ
line due to excessive unhandled counts. [2]_

The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or
disable) the redirection of the interrupt handler to the PCH interrupt
line. The option can be overridden by either pci=ioapicreroute or
pci=noioapicreroute. [3]_

추가 datasheet

6300ESB와 6700PXH 등 여러 datasheet에 legacy interrupt handling 개요가 있습니다. 구현은 대체로 같지만 chipset 세대에 따른 처리 방식의 변화를 보여줍니다.

More Documentation
==================

There is an overview of the legacy interrupt handling in several datasheets
(6300ESB and 6700PXH below). While largely the same, it provides insight
into the evolution of its handling with chipsets.

Example of disabling of the boot interrupt
------------------------------------------

      - Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
        5.7.3 Boot Interrupt
        https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf

      - Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
        Datasheet - Volume 2: Registers (Document # 330784-003)
        6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
        https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf

Example of handler rerouting
----------------------------

      - Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
        2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
        https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf

문의와 참고 링크

문서에서 답하지 않은 legacy PCI interrupt 질문은 Sean V Kelley(`sean.v.kelley@linux.intel.com`)에게 문의할 수 있습니다.

참고 링크 [1]~[3]은 각각 boot interrupt disable quirk, handler rerouting, 관련 option 논의를 가리킵니다.

[1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
[2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
[3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/

If you have any legacy PCI interrupt questions that aren't answered, email me.

Cheers,
    Sean V Kelley
    sean.v.kelley@linux.intel.com

.. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
.. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
.. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/