mlx5 Switchdev

브리지와 VLAN 오프로딩

이 문서는 `GPL-2.0 OR Linux-OpenIB` 이중 라이선스를 따릅니다.

Switchdev

mlx5 드라이버는 switchdev 모드에서 브리지 규칙 오프로딩을 지원합니다. mlx5 switchdev representor를 bridge에 연결하면 Linux bridge FDB가 자동으로 하드웨어에 내려갑니다.

장치를 switchdev 모드로 바꿉니다.

$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

mlx5 representor `enp8s0f0`을 bridge netdev `bridge1`에 연결합니다.

$ ip link set enp8s0f0 master bridge1

VLAN

• 포트마다 여러 VLAN을 둘 수 있는 VLAN filtering을 지원합니다.
• 브리지 ingress에서 PVID를 넣는 VLAN push를 지원합니다.
• 브리지 egress에서 untagged로 내보내는 VLAN pop을 지원합니다.

$ ip link set bridge1 type bridge vlan_filtering 1
$ bridge vlan add dev enp8s0f0 vid 2-3
$ bridge vlan add dev enp8s0f0 vid 3 pvid
$ bridge vlan add dev enp8s0f0 vid 3 untagged

.. SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
.. include:: <isonum.txt>

=========
Switchdev
=========

:Copyright: |copy| 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

.. _mlx5_bridge_offload:

Bridge offload
==============

The mlx5 driver implements support for offloading bridge rules when in switchdev
mode. Linux bridge FDBs are automatically offloaded when mlx5 switchdev
representor is attached to bridge.

- Change device to switchdev mode::

    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

- Attach mlx5 switchdev representor 'enp8s0f0' to bridge netdev 'bridge1'::

    $ ip link set enp8s0f0 master bridge1

VLANs
-----

Following bridge VLAN functions are supported by mlx5:

- VLAN filtering (including multiple VLANs per port)::

    $ ip link set bridge1 type bridge vlan_filtering 1
    $ bridge vlan add dev enp8s0f0 vid 2-3

- VLAN push on bridge ingress::

    $ bridge vlan add dev enp8s0f0 vid 3 pvid

- VLAN pop on bridge egress::

    $ bridge vlan add dev enp8s0f0 vid 3 untagged

Subfunction 생성과 자원 모델

Subfunction

E-Switch 위에 만드는 SF는 devlink 장치로만 생성됩니다. 기본 상태에서는 모든 SF auxiliary device가 비활성화되므로 완전히 probe하기 전에 SF를 구성할 수 있고 초기화 시간을 줄일 수 있습니다.

SF를 만들고 MAC 주소와 active 상태를 설정한 뒤 ETH auxiliary device를 켜고 reload합니다.

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11
$ devlink port function set pci/0000:08:00.0/32768 hw_addr 00:00:00:00:00:11 state active
$ devlink dev param set auxiliary/mlx5_core.sf.1 name enable_eth value true cmode driverinit
$ devlink dev reload auxiliary/mlx5_core.sf.1

mlx5는 ETH, RDMA와 vDPA(vnet) auxiliary device의 devlink parameter를 지원합니다. SF 관리는 `Documentation/networking/devlink/devlink-port.rst`의 devlink port 인터페이스를 사용합니다.

각 SF에는 고유 기능과 전용 `txq`, `rxq`, `cq`, `eq`가 있습니다. 부모 PCI function과 큐를 공유하거나 빼앗지 않습니다. RDMA가 가능하면 QP1, GID table과 RDMA 자원도 전용입니다.

SF마다 다른 SF나 부모 PCI function과 공유하지 않는 PCI BAR window가 있어 netdev, RDMA, vDPA 장치는 배정된 BAR 공간만 접근합니다. eSwitch representor를 통해 TC 오프로딩과 SF 포트 송수신을 구성합니다. 다만 PCI MSI-X IRQ 같은 PCI 계층 자원은 다른 SF 및 부모와 공유합니다.

devlink port 인터페이스로 SF를 생성하고 조회한 뒤 사용이 끝나면 삭제합니다.

$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ devlink port del pci/0000:06:00.0/32768

Subfunction
===========

Subfunction which are spawned over the E-switch are created only with devlink
device, and by default all the SF auxiliary devices are disabled.
This will allow user to configure the SF before the SF have been fully probed,
which will save time.

Usage example:

- Create SF::

    $ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11
    $ devlink port function set pci/0000:08:00.0/32768 hw_addr 00:00:00:00:00:11 state active

- Enable ETH auxiliary device::

    $ devlink dev param set auxiliary/mlx5_core.sf.1 name enable_eth value true cmode driverinit

- Now, in order to fully probe the SF, use devlink reload::

    $ devlink dev reload auxiliary/mlx5_core.sf.1

mlx5 supports ETH,rdma and vdpa (vnet) auxiliary devices devlink params (see :ref:`Documentation/networking/devlink/devlink-params.rst <devlink_params_generic>`).

mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.

A subfunction has its own function capabilities and its own resources. This
means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
queues are neither shared nor stolen from the parent PCI function.

When a subfunction is RDMA capable, it has its own QP1, GID table, and RDMA
resources neither shared nor stolen from the parent PCI function.

A subfunction has a dedicated window in PCI BAR space that is not shared
with the other subfunctions or the parent PCI function. This ensures that all
devices (netdev, rdma, vdpa, etc.) of the subfunction accesses only assigned
PCI BAR space.

A subfunction supports eswitch representation through which it supports tc
offloads. The user configures eswitch to send/receive packets from/to
the subfunction port.

Subfunctions share PCI level resources such as PCI MSI-X IRQs with
other subfunctions and/or with its parent PCI function.

Example mlx5 software, system, and device view::

       _______
      | admin |
      | user  |----------
      |_______|         |
          |             |
      ____|____       __|______            _________________
     |         |     |         |          |                 |
     | devlink |     | tc tool |          |    user         |
     | tool    |     |_________|          | applications    |
     |_________|         |                |_________________|
           |             |                   |          |
           |             |                   |          |         Userspace
 +---------|-------------|-------------------|----------|--------------------+
           |             |           +----------+   +----------+   Kernel
           |             |           |  netdev  |   | rdma dev |
           |             |           +----------+   +----------+
   (devlink port add/del |              ^               ^
    port function set)   |              |               |
           |             |              +---------------|
      _____|___          |              |        _______|_______
     |         |         |              |       | mlx5 class    |
     | devlink |   +------------+       |       |   drivers     |
     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
     |_________|   +------------+       |       |_______________|
           |             |              |               ^
   (devlink ops)         |              |          (probe/remove)
  _________|________     |              |           ____|________
 | subfunction      |    |     +---------------+   | subfunction |
 | management driver|-----     | subfunction   |---|  driver     |
 | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
 |__________________|          +---------------+   |_____________|
           |                                            ^
  (sf add/del, vhca events)                             |
           |                                      (device add/del)
      _____|____                                    ____|________
     |          |                                  | subfunction |
     |  PCI NIC |--- activate/deactivate events--->| host driver |
     |__________|                                  | (mlx5_core) |
                                                   |_____________|

Subfunction is created using devlink port interface.

- Change device to switchdev mode::

    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

- Add a devlink port of subfunction flavour::

    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
      function:
        hw_addr 00:00:00:00:00:00 state inactive opstate detached

- Show a devlink port of the subfunction::

    $ devlink port show pci/0000:06:00.0/32768
    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
      function:
        hw_addr 00:00:00:00:00:00 state inactive opstate detached

- Delete a devlink port of subfunction after use::

    $ devlink port del pci/0000:06:00.0/32768

VF/SF function 속성과 capability

Function 속성

mlx5는 SmartNIC과 일반 NIC 모두에서 PCI VF/SF function 속성을 같은 방식으로 설정합니다. eSwitch가 switchdev 모드일 때만 지원하며 devlink eSwitch port를 사용합니다. 드라이버가 PCI VF/SF를 열거하기 전에 속성을 설정해야 합니다.

Function attributes
===================

The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
a unified way for SmartNIC and non-SmartNIC.

This is supported only when the eswitch mode is set to switchdev. Port function
configuration of the PCI VF/SF is supported through devlink eswitch port.

Port function attributes should be set before PCI VF/SF is enumerated by the
driver.

MAC address setup
-----------------

mlx5 driver support devlink port function attr mechanism to setup MAC
address. (refer to Documentation/networking/devlink/devlink-port.rst)

RoCE capability setup
~~~~~~~~~~~~~~~~~~~~~
Not all mlx5 PCI devices/SFs require RoCE capability.

When RoCE capability is disabled, it saves 1 Mbytes worth of system memory per
PCI devices/SF.

mlx5 driver support devlink port function attr mechanism to setup RoCE
capability. (refer to Documentation/networking/devlink/devlink-port.rst)

migratable capability setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~
User who wants mlx5 PCI VFs to be able to perform live migration need to
explicitly enable the VF migratable capability.

mlx5 driver support devlink port function attr mechanism to setup migratable
capability. (refer to Documentation/networking/devlink/devlink-port.rst)

IPsec crypto capability setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
User who wants mlx5 PCI VFs to be able to perform IPsec crypto offloading need
to explicitly enable the VF ipsec_crypto capability. Enabling IPsec capability
for VFs is supported starting with ConnectX6dx devices and above. When a VF has
IPsec capability enabled, any IPsec offloading is blocked on the PF.

mlx5 driver support devlink port function attr mechanism to setup ipsec_crypto
capability. (refer to Documentation/networking/devlink/devlink-port.rst)

IPsec packet capability setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
User who wants mlx5 PCI VFs to be able to perform IPsec packet offloading need
to explicitly enable the VF ipsec_packet capability. Enabling IPsec capability
for VFs is supported starting with ConnectX6dx devices and above. When a VF has
IPsec capability enabled, any IPsec offloading is blocked on the PF.

mlx5 driver support devlink port function attr mechanism to setup ipsec_packet
capability. (refer to Documentation/networking/devlink/devlink-port.rst)

SF 상태 활성화와 auxiliary device 계층

SF 상태 설정

SF를 사용하려면 SF function state 속성으로 활성화해야 합니다. 먼저 고유 devlink port index의 상태를 확인한 뒤 `state active`를 설정하고 다시 확인합니다.

 function:
   hw_addr 00:00:00:00:88:88 state inactive opstate detached
 function:
   hw_addr 00:00:00:00:88:88 state active opstate detached

function을 활성화하면 PF 드라이버가 장치의 SF 활성화 이벤트를 받습니다. 이 신호를 계기로 장치를 버스에 올리고 probe한 뒤 devlink instance와 class별 auxiliary device를 생성합니다.

SF auxiliary device, 포트와 RDMA 장치를 확인합니다.

$ devlink dev show
devlink dev show auxiliary/mlx5_core.sf.4
$ devlink port show auxiliary/mlx5_core.sf.4/1
auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
$ rdma link show mlx5_0/1
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
$ rdma dev show
8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112

SF 드라이버가 auxiliary device에 attach되면 SF port도 이벤트를 받아 function의 operational state를 바꿉니다. 사용자는 `opstate attached`를 확인해 SF port를 정상 종료 방식으로 안전하게 삭제할 시점을 판단할 수 있습니다.

$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
  function:
    hw_addr 00:00:00:00:88:88 state active opstate attached

SF state setup
--------------

To use the SF, the user must activate the SF using the SF function state
attribute.

- Get the state of the SF identified by its unique devlink port index::

   $ devlink port show ens2f0npf0sf88
   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
     function:
       hw_addr 00:00:00:00:88:88 state inactive opstate detached

- Activate the function and verify its state is active::

   $ devlink port function set ens2f0npf0sf88 state active

   $ devlink port show ens2f0npf0sf88
   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
     function:
       hw_addr 00:00:00:00:88:88 state active opstate detached

Upon function activation, the PF driver instance gets the event from the device
that a particular SF was activated. It's the cue to put the device on bus, probe
it and instantiate the devlink instance and class specific auxiliary devices
for it.

- Show the auxiliary device and port of the subfunction::

    $ devlink dev show
    devlink dev show auxiliary/mlx5_core.sf.4

    $ devlink port show auxiliary/mlx5_core.sf.4/1
    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false

    $ rdma link show mlx5_0/1
    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88

    $ rdma dev show
    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112

- Subfunction auxiliary device and class device hierarchy::

                 mlx5_core.sf.4
          (subfunction auxiliary device)
                       /\
                      /  \
                     /    \
                    /      \
                   /        \
      mlx5_core.eth.4     mlx5_core.rdma.4
     (sf eth aux dev)     (sf rdma aux dev)
         |                      |
         |                      |
      p0sf88                  mlx5_0
     (sf netdev)          (sf rdma device)

Additionally, the SF port also gets the event when the driver attaches to the
auxiliary device of the subfunction. This results in changing the operational
state of the function. This provides visibility to the user to decide when is it
safe to delete the SF port for graceful termination of the subfunction.

- Show the SF port operational state::

    $ devlink port show ens2f0npf0sf88
    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
      function:
        hw_addr 00:00:00:00:88:88 state active opstate attached