Marvell OcteonTx2 RVU Kernel Drivers

RVU 개요와 resource provisioning

이 문서는 `GPL-2.0-only OR BSD-2-Clause` 이중 라이선스를 따릅니다.

Marvell OcteonTx2 RVU Kernel Driver

Copyright (c) 2020 Marvell International Ltd.

목차

• 개요
• Driver
• 기본 packet flow
• Devlink health reporter
• Quality of service
• RVU representor

개요

Marvell OcteonTX2 SoC의 Resource Virtualization Unit(RVU)은 network, crypto와 다른 functional block의 hardware resource를 PCI-compatible Physical Function(PF)과 Virtual Function(VF)에 mapping합니다. 각 functional block에는 PCI device에 provision할 수 있는 여러 Local Function(LF)이 있습니다.

RVU는 여러 PCIe SR-IOV PF와 VF를 지원합니다. PF0는 administrative function 또는 Admin Function(AF)이라 부르며 각 PF/VF에 RVU functional block의 LF를 provision할 권한이 있습니다.

RVU가 관리하는 networking functional block

• Network Pool or buffer Allocator(NPA)
• Network Interface Controller(NIX)
• Network Parser CAM(NPC)
• Schedule/Synchronize/Order unit(SSO)
• Loopback interface(LBK)

RVU가 관리하는 non-networking functional block

• Crypto accelerator(CPT)
• Scheduled Timer unit(TIM)
• networking과 non-networking 모두에 쓰이는 SSO

Resource provisioning 예

• NIX-LF와 NPA-LF resource를 받은 PF/VF는 순수 network device로 동작합니다.
• CPT-LF resource를 받은 PF/VF는 순수 crypto offload device로 동작합니다.

RVU functional block은 software requirement에 맞춰 매우 유연하게 구성할 수 있습니다.

kernel boot 전에 firmware가 설정하는 항목

• physical link 수에 따라 필요한 RVU PF 수를 활성화합니다.
• PF별 VF 수는 static이거나 compile time에 구성할 수 있습니다. firmware는 이 configuration에 따라 각 PF에 VF를 할당합니다.
• 각 PF와 VF에 MSI-X vector를 할당합니다.
• 이 설정은 kernel boot 뒤에는 바뀌지 않습니다.

.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)

====================================
Marvell OcteonTx2 RVU Kernel Drivers
====================================

Copyright (c) 2020 Marvell International Ltd.

Contents
========

- `Overview`_
- `Drivers`_
- `Basic packet flow`_
- `Devlink health reporters`_
- `Quality of service`_
- `RVU representors`_

Overview
========

Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
resources from the network, crypto and other functional blocks into
PCI-compatible physical and virtual functions. Each functional block
again has multiple local functions (LFs) for provisioning to PCI devices.
RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
functions (VFs). PF0 is called the administrative / admin function (AF)
and has privileges to provision RVU functional block's LFs to each of the
PF/VF.

RVU managed networking functional blocks
 - Network pool or buffer allocator (NPA)
 - Network interface controller (NIX)
 - Network parser CAM (NPC)
 - Schedule/Synchronize/Order unit (SSO)
 - Loopback interface (LBK)

RVU managed non-networking functional blocks
 - Crypto accelerator (CPT)
 - Scheduled timers unit (TIM)
 - Schedule/Synchronize/Order unit (SSO)
   Used for both networking and non networking usecases

Resource provisioning examples
 - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
 - A PF/VF with CPT-LF resource works as a pure crypto offload device.

RVU functional blocks are highly configurable as per software requirements.

Firmware setups following stuff before kernel boots
 - Enables required number of RVU PFs based on number of physical links.
 - Number of VFs per PF are either static or configurable at compile time.
   Based on config, firmware assigns VFs to each of the PFs.
 - Also assigns MSIX vectors to each of PF and VFs.
 - These are not changed after kernel boot.

Admin Function driver

Driver

Linux kernel에는 RVU의 서로 다른 PF와 VF에 등록되는 여러 driver가 있습니다. networking 관점에서는 세 종류로 나뉩니다.

Admin Function driver

RVU PF0인 AF driver는 resource provisioning과 functional block configuration을 담당하며 I/O는 직접 처리하지 않습니다. 기본 항목 일부를 설정하지만 대부분의 기능은 PF와 VF가 보내는 configuration request로 수행합니다.

PF/VF는 shared memory region인 mailbox를 통해 AF와 통신합니다. AF는 request를 받으면 resource를 provision하고 다른 hardware configuration을 수행합니다.

AF는 항상 host kernel에 연결됩니다. 반면 PF와 그 VF는 host kernel이 직접 사용하거나 VM 또는 DPDK 같은 user-space application에 연결할 수 있습니다. 따라서 AF는 어느 domain의 device가 보낸 provisioning·configuration request도 처리해야 합니다.

AF driver가 firmware와 상호작용해 수행하는 작업

• CGX LMAC 같은 physical Ethernet link를 관리합니다.
• speed, duplex, auto-negotiation 정보를 가져옵니다.
• PHY EEPROM과 통계를 가져옵니다.
• FEC와 PAM mode를 구성합니다.

순수 networking 관점에서 AF driver가 지원하는 기능

• netdev가 등록될 RVU PF에 physical link를 mapping합니다.
• 일반 networking에 필요한 buffer pool, Receive Queue(RQ), Send Queue(SQ)를 제공하도록 NIX·NPA block LF를 RVU PF/VF에 연결합니다.
• Flow Control(pause frame)을 켜고 끄거나 구성합니다.
• hardware PTP timestamping 관련 항목을 구성합니다.
• packet parsing 방법과 추출 정보를 정하는 NPC parser profile을 구성합니다.
• MCAM entry와 비교할 packet field를 정하는 NPC extract profile을 구성합니다.
• 요청에 따라 packet forwarding rule을 만들고 설치하는 NPC MCAM entry를 관리합니다.
• Receive Side Scaling(RSS) algorithm을 정의합니다.
• TSO 같은 segmentation offload algorithm을 정의합니다.
• VLAN stripping, capture와 insertion을 구성합니다.
• packet scheduling을 제공하는 SSO와 TIM block을 구성합니다.
• 현재 resource provisioning, NPA pool, NIX RQ·SQ·CQ 상태와 통계를 점검할 debugfs를 지원합니다.

Drivers
=======

Linux kernel will have multiple drivers registering to different PF and VFs
of RVU. Wrt networking there will be 3 flavours of drivers.

Admin Function driver
---------------------

As mentioned above RVU PF0 is called the admin function (AF), this driver
supports resource provisioning and configuration of functional blocks.
Doesn't handle any I/O. It sets up few basic stuff but most of the
functionality is achieved via configuration requests from PFs and VFs.

PF/VFs communicates with AF via a shared memory region (mailbox). Upon
receiving requests AF does resource provisioning and other HW configuration.
AF is always attached to host kernel, but PFs and their VFs may be used by host
kernel itself, or attached to VMs or to userspace applications like
DPDK etc. So AF has to handle provisioning/configuration requests sent
by any device from any domain.

AF driver also interacts with underlying firmware to
 - Manage physical ethernet links ie CGX LMACs.
 - Retrieve information like speed, duplex, autoneg etc
 - Retrieve PHY EEPROM and stats.
 - Configure FEC, PAM modes
 - etc

From pure networking side AF driver supports following functionality.
 - Map a physical link to a RVU PF to which a netdev is registered.
 - Attach NIX and NPA block LFs to RVU PF/VF which provide buffer pools, RQs, SQs
   for regular networking functionality.
 - Flow control (pause frames) enable/disable/config.
 - HW PTP timestamping related config.
 - NPC parser profile config, basically how to parse pkt and what info to extract.
 - NPC extract profile config, what to extract from the pkt to match data in MCAM entries.
 - Manage NPC MCAM entries, upon request can frame and install requested packet forwarding rules.
 - Defines receive side scaling (RSS) algorithms.
 - Defines segmentation offload algorithms (eg TSO)
 - VLAN stripping, capture and insertion config.
 - SSO and TIM blocks config which provide packet scheduling support.
 - Debugfs support, to check current resource provising, current status of
   NPA pools, NIX RQ, SQ and CQs, various stats etc which helps in debugging issues.
 - And many more.

Physical Function과 Virtual Function driver

Physical Function driver

RVU PF는 I/O를 처리하고 physical Ethernet link에 mapping되며 driver는 netdev를 등록합니다. SR-IOV를 지원하고 mailbox로 AF와 통신합니다.

PF driver는 physical link 정보를 firmware에서 직접 가져올 수 없습니다. AF에 요청하면 AF가 firmware에서 정보를 가져와 PF에 응답합니다.

ethtool을 이용한 link, RSS, queue 수·크기, Flow Control, ntuple filter, PHY EEPROM dump와 FEC configuration을 지원합니다.

Virtual Function driver

VF에는 parent SR-IOV PF와 physical link를 공유하는 Type 1과 내부 hardware loopback channel(LBK)으로 pair를 이루는 Type 2가 있습니다.

Type 1 VF

• parent PF와 physical link를 공유하며 외부 통신에 사용합니다.
• AF와 직접 통신할 수 없습니다. VF가 PF에 mailbox message를 보내면 PF가 AF로 전달하고, AF의 응답도 PF를 거쳐 VF에 전달됩니다.
• PF와 VF에 같은 type의 hardware resource가 연결되므로 기능상 차이는 없습니다. 다만 PF를 link의 owner/admin으로 취급하므로 일부 항목은 PF에서만 구성할 수 있습니다.

Type 2 VF

• AF인 RVU PF0가 VF를 만들고 loopback block channel에 mapping합니다.
• VF0·VF1, VF2·VF3처럼 두 VF가 pair로 동작합니다. VF0에서 내보낸 packet은 VF1이 받고 그 반대도 같습니다.
• application이나 VM 사이에서 traffic을 외부로 내보내지 않고 통신하는 데 사용할 수 있습니다. hardware에 switch가 없기 때문에 loopback VF를 지원합니다.
• AF(PF0)와 mailbox로 직접 통신합니다.

두 VF type은 packet 송수신에 사용하는 I/O channel 또는 link만 다르고 나머지는 같습니다. AF driver가 I/O channel mapping을 처리하므로 같은 VF driver가 두 type 모두에서 동작합니다.

Physical Function driver
------------------------

This RVU PF handles IO, is mapped to a physical ethernet link and this
driver registers a netdev. This supports SR-IOV. As said above this driver
communicates with AF with a mailbox. To retrieve information from physical
links this driver talks to AF and AF gets that info from firmware and responds
back ie cannot talk to firmware directly.

Supports ethtool for configuring links, RSS, queue count, queue size,
flow control, ntuple filters, dump PHY EEPROM, config FEC etc.

Virtual Function driver
-----------------------

There are two types VFs, VFs that share the physical link with their parent
SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK).

Type1:
 - These VFs and their parent PF share a physical link and used for outside communication.
 - VFs cannot communicate with AF directly, they send mbox message to PF and PF
   forwards that to AF. AF after processing, responds back to PF and PF forwards
   the reply to VF.
 - From functionality point of view there is no difference between PF and VF as same type
   HW resources are attached to both. But user would be able to configure few stuff only
   from PF as PF is treated as owner/admin of the link.

Type2:
 - RVU PF0 ie admin function creates these VFs and maps them to loopback block's channels.
 - A set of two VFs (VF0 & VF1, VF2 & VF3 .. so on) works as a pair ie pkts sent out of
   VF0 will be received by VF1 and vice versa.
 - These VFs can be used by applications or virtual machines to communicate between them
   without sending traffic outside. There is no switch present in HW, hence the support
   for loopback VFs.
 - These communicate directly with AF (PF0) via mbox.

Except for the IO channels or links used for packet reception and transmission there is
no other difference between these VF types. AF driver takes care of IO channel mapping,
hence same VF driver works for both types of devices.

Ingress와 egress packet flow

기본 packet flow

Ingress

• 1. CGX LMAC가 packet을 수신합니다.
• 2. packet을 NIX block으로 전달합니다.
• 3. NPC block이 packet을 parse하고 MCAM lookup으로 destination RVU device를 찾습니다.
• 4. destination RVU device에 연결된 NIX LF가 NPA block LF의 RQ-mapped buffer pool에서 buffer를 allocate합니다.
• 5. RSS 또는 RQ number를 지정한 MCAM rule이 RQ를 선택할 수 있습니다.
• 6. packet을 DMA하고 driver에 알립니다.

Egress

• 1. driver가 send descriptor를 준비해 transmission용 SQ에 submit합니다.
• 2. SQ는 AF가 특정 link 또는 channel로 전송하도록 미리 구성해 둡니다.
• 3. SQ descriptor ring은 NPA block LF의 SQ-mapped pool에서 allocate한 buffer에 유지됩니다.
• 4. NIX block이 지정 channel로 packet을 전송합니다.
• 5. NPC MCAM entry를 설치하면 packet을 다른 channel로 돌릴 수 있습니다.

Basic packet flow
=================

Ingress
-------

1. CGX LMAC receives packet.
2. Forwards the packet to the NIX block.
3. Then submitted to NPC block for parsing and then MCAM lookup to get the destination RVU device.
4. NIX LF attached to the destination RVU device allocates a buffer from RQ mapped buffer pool of NPA block LF.
5. RQ may be selected by RSS or by configuring MCAM rule with a RQ number.
6. Packet is DMA'ed and driver is notified.

Egress
------

1. Driver prepares a send descriptor and submits to SQ for transmission.
2. The SQ is already configured (by AF) to transmit on a specific link/channel.
3. The SQ descriptor ring is maintained in buffers allocated from SQ mapped pool of NPA block LF.
4. NIX block transmits the pkt on the designated channel.
5. NPC MCAM entries can be installed to divert pkt onto a different channel.

NPA devlink health reporter

Devlink health reporter

NPA reporter

NPA reporter는 다음 error group을 보고하고 recovery합니다.

`devlink health` sample에는 `hw_npa_intr`, `hw_npa_gen`, `hw_npa_err`, `hw_npa_ras` reporter와 각 state, error·recover count, 마지막 dump 시각, auto recovery/dump 상태가 표시됩니다.

devlink health

각 reporter dump는 다음 정보를 제공합니다.

• Error Type
• Error Register value
• 문장으로 된 원인

아래 command로 reporter별 dump를 확인할 수 있습니다. 예시에서는 각각 NIX0 free disabled RX, Unmap Slot Error와 AQ Doorbell Error가 보고됩니다.

devlink health dump show pci/0002:01:00.0 reporter hw_npa_gen
devlink health dump show pci/0002:01:00.0 reporter hw_npa_intr
devlink health dump show pci/0002:01:00.0 reporter hw_npa_err

Devlink health reporters
========================

NPA Reporters
-------------
The NPA reporters are responsible for reporting and recovering the following group of errors:

1. GENERAL events

   - Error due to operation of unmapped PF.
   - Error due to disabled alloc/free for other HW blocks (NIX, SSO, TIM, DPI and AURA).

2. ERROR events

   - Fault due to NPA_AQ_INST_S read or NPA_AQ_RES_S write.
   - AQ Doorbell Error.

3. RAS events

   - RAS Error Reporting for NPA_AQ_INST_S/NPA_AQ_RES_S.

4. RVU events

   - Error due to unmapped slot.

Sample Output::

        ~# devlink health
        pci/0002:01:00.0:
          reporter hw_npa_intr
              state healthy error 2872 recover 2872 last_dump_date 2020-12-10 last_dump_time 09:39:09 grace_period 0 auto_recover true auto_dump true
          reporter hw_npa_gen
              state healthy error 2872 recover 2872 last_dump_date 2020-12-11 last_dump_time 04:43:04 grace_period 0 auto_recover true auto_dump true
          reporter hw_npa_err
              state healthy error 2871 recover 2871 last_dump_date 2020-12-10 last_dump_time 09:39:17 grace_period 0 auto_recover true auto_dump true
           reporter hw_npa_ras
              state healthy error 0 recover 0 last_dump_date 2020-12-10 last_dump_time 09:32:40 grace_period 0 auto_recover true auto_dump true

Each reporter dumps the

 - Error Type
 - Error Register value
 - Reason in words

For example::

        ~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_gen
         NPA_AF_GENERAL:
                 NPA General Interrupt Reg : 1
                 NIX0: free disabled RX
        ~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_intr
         NPA_AF_RVU:
                 NPA RVU Interrupt Reg : 1
                 Unmap Slot Error
        ~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_err
         NPA_AF_ERR:
                NPA Error Interrupt Reg : 4096
                AQ Doorbell Error

NIX devlink health reporter

NIX reporter

NIX reporter는 다음 error group을 보고하고 recovery합니다.

`devlink health` sample에는 NPA reporter와 함께 `hw_nix_intr`, `hw_nix_gen`, `hw_nix_err`, `hw_nix_ras` reporter의 상태와 누적 error·recovery 수가 표시됩니다.

각 reporter dump는 다음 정보를 제공합니다.

• Error Type
• Error Register value
• 문장으로 된 원인

아래 command로 NIX reporter dump를 확인합니다. 예시는 Unmap Slot Error, Rx multicast packet drop과 unmapped `PF_FUNC`에서의 Rx를 보여 줍니다.

devlink health dump show pci/0002:01:00.0 reporter hw_nix_intr
devlink health dump show pci/0002:01:00.0 reporter hw_nix_gen
devlink health dump show pci/0002:01:00.0 reporter hw_nix_err

NIX Reporters
-------------
The NIX reporters are responsible for reporting and recovering the following group of errors:

1. GENERAL events

   - Receive mirror/multicast packet drop due to insufficient buffer.
   - SMQ Flush operation.

2. ERROR events

   - Memory Fault due to WQE read/write from multicast/mirror buffer.
   - Receive multicast/mirror replication list error.
   - Receive packet on an unmapped PF.
   - Fault due to NIX_AQ_INST_S read or NIX_AQ_RES_S write.
   - AQ Doorbell Error.

3. RAS events

   - RAS Error Reporting for NIX Receive Multicast/Mirror Entry Structure.
   - RAS Error Reporting for WQE/Packet Data read from Multicast/Mirror Buffer..
   - RAS Error Reporting for NIX_AQ_INST_S/NIX_AQ_RES_S.

4. RVU events

   - Error due to unmapped slot.

Sample Output::

        ~# ./devlink health
        pci/0002:01:00.0:
          reporter hw_npa_intr
            state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
          reporter hw_npa_gen
            state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
          reporter hw_npa_err
            state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
          reporter hw_npa_ras
            state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
          reporter hw_nix_intr
            state healthy error 1121 recover 1121 last_dump_date 2021-01-19 last_dump_time 05:42:26 grace_period 0 auto_recover true auto_dump true
          reporter hw_nix_gen
            state healthy error 949 recover 949 last_dump_date 2021-01-19 last_dump_time 05:42:43 grace_period 0 auto_recover true auto_dump true
          reporter hw_nix_err
            state healthy error 1147 recover 1147 last_dump_date 2021-01-19 last_dump_time 05:42:59 grace_period 0 auto_recover true auto_dump true
          reporter hw_nix_ras
            state healthy error 409 recover 409 last_dump_date 2021-01-19 last_dump_time 05:43:16 grace_period 0 auto_recover true auto_dump true

Each reporter dumps the

 - Error Type
 - Error Register value
 - Reason in words

For example::

        ~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_intr
         NIX_AF_RVU:
                NIX RVU Interrupt Reg : 1
                Unmap Slot Error
        ~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_gen
         NIX_AF_GENERAL:
                NIX General Interrupt Reg : 1
                Rx multicast pkt drop
        ~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_err
         NIX_AF_ERR:
                NIX Error Interrupt Reg : 64
                Rx on unmapped PF_FUNC

QoS scheduler와 HTB offload

Quality of Service

Scheduling에 사용하는 hardware algorithm

OcteonTX2 silicon과 CN10K transmit interface에는 SMQ/MDQ에서 시작해 TL4, TL3, TL2, TL1로 이어지는 다섯 transmit level이 있습니다. 각 packet은 MDQ에서 TL4를 거쳐 TL1까지 통과합니다. 각 level에는 scheduling과 shaping을 위한 queue array가 있습니다.

hardware는 scheduler queue priority에 따라 algorithm을 선택합니다. 사용자가 priority가 다른 tc class를 만들면 driver는 지정 priority와 rate limit configuration을 class에 할당된 scheduler에 구성합니다.

1. Strict Priority

packet이 MDQ에 submit되면 hardware는 priority가 서로 다른 active MDQ를 strict priority 방식으로 선택합니다.

2. Round Robin

priority level이 같은 active MDQ는 round-robin 방식으로 선택합니다.

HTB offload 설정

1. interface에서 hardware TC offload를 켭니다.

ethtool -K <interface> hw-tc-offload on

2. clsact와 HTB root를 만듭니다.

tc qdisc add dev <interface> clsact
tc qdisc replace dev <interface> root handle 1: htb offload

3. priority가 서로 다른 tc class를 만듭니다.

tc class add dev <interface> parent 1: classid 1:1 htb rate 10Gbit prio 1
tc class add dev <interface> parent 1: classid 1:2 htb rate 10Gbit prio 7

4. priority는 같고 quantum이 서로 다른 tc class를 만듭니다.

tc class add dev <interface> parent 1: classid 1:1 htb rate 10Gbit prio 2 quantum 409600
tc class add dev <interface> parent 1: classid 1:2 htb rate 10Gbit prio 2 quantum 188416
tc class add dev <interface> parent 1: classid 1:3 htb rate 10Gbit prio 2 quantum 32768

Quality of service
==================


Hardware algorithms used in scheduling
--------------------------------------

octeontx2 silicon and CN10K transmit interface consists of five transmit levels
starting from SMQ/MDQ, TL4 to TL1. Each packet will traverse MDQ, TL4 to TL1
levels. Each level contains an array of queues to support scheduling and shaping.
The hardware uses the below algorithms depending on the priority of scheduler queues.
once the usercreates tc classes with different priorities, the driver configures
schedulers allocated to the class with specified priority along with rate-limiting
configuration.

1. Strict Priority

      -  Once packets are submitted to MDQ, hardware picks all active MDQs having different priority
         using strict priority.

2. Round Robin

      - Active MDQs having the same priority level are chosen using round robin.


Setup HTB offload
-----------------

1. Enable HW TC offload on the interface::

        # ethtool -K <interface> hw-tc-offload on

2. Crate htb root::

        # tc qdisc add dev <interface> clsact
        # tc qdisc replace dev <interface> root handle 1: htb offload

3. Create tc classes with different priorities::

        # tc class add dev <interface> parent 1: classid 1:1 htb rate 10Gbit prio 1

        # tc class add dev <interface> parent 1: classid 1:2 htb rate 10Gbit prio 7

4. Create tc classes with same priorities and different quantum::

        # tc class add dev <interface> parent 1: classid 1:1 htb rate 10Gbit prio 2 quantum 409600

        # tc class add dev <interface> parent 1: classid 1:2 htb rate 10Gbit prio 2 quantum 188416

        # tc class add dev <interface> parent 1: classid 1:3 htb rate 10Gbit prio 2 quantum 32768

RVU representor와 switchdev mode

RVU Representor

RVU representor driver는 system에서 RVU PF의 VF를 대표하는 representor device를 생성합니다. 사용자가 switchdev mode를 켜면 representor가 만들어지며, SR-IOV `numVFs`를 설정하기 전이나 후 모두 switchdev mode를 켤 수 있습니다.

모든 representor device는 NIX LF 하나를 공유하지만 각각 전용 Rx/Tx queue를 가집니다. RVU PF representor driver는 Rx/Tx queue pair마다 별도 netdev를 등록합니다.

현재 hardware에는 representee와 representor 사이에서 L2 learning과 packet forwarding을 수행하는 built-in switch가 없습니다. 따라서 적절한 NPC MCAM filter를 설정해 두 방향 packet path를 만듭니다.

filter와 일치하는 transmit packet은 MAC interface로 나가지 않고 hardware loopback channel/interface로 되돌아옵니다. 다시 설치된 filter와 일치한 뒤 forwarding되어 `representee => representor`와 `representor => representee` path를 완성합니다.

rule은 representor 생성 때 설치되며 representor와 representee interface state에 따라 활성화되거나 비활성화됩니다.

사용 예

device를 switchdev mode로 바꿉니다.

devlink dev eswitch set pci/0002:1c:00.0 mode switchdev

`ip link show`에는 `Rpf1vf0`, `Rpf1vf1`, `Rpf1vf2`, `Rpf1vf3`, `Rpf2vf0` 같은 representor netdev가 표시됩니다.

ip link show

system에서 representor device를 삭제하려면 device를 legacy mode로 되돌립니다.

devlink dev eswitch set pci/0002:1c:00.0 mode legacy

RVU representor는 devlink port interface로 관리할 수 있습니다. 자세한 내용은 `Documentation/networking/devlink/devlink-port.rst`를 참고하십시오.

representor의 devlink port를 표시합니다.

devlink port

출력에는 physical port와 `flavour pcivf` port별 controller, `pfnum`, `vfnum`과 연결 netdev가 표시됩니다.

RVU Representors
================

RVU representor driver adds support for creation of representor devices for
RVU PFs' VFs in the system. Representor devices are created when user enables
the switchdev mode.
Switchdev mode can be enabled either before or after setting up SRIOV numVFs.
All representor devices share a single NIXLF but each has a dedicated Rx/Tx
queues. RVU PF representor driver registers a separate netdev for each
Rx/Tx queue pair.

Current HW does not support built-in switch which can do L2 learning and
forwarding packets between representee and representor. Hence, packet path
between representee and it's representor is achieved by setting up appropriate
NPC MCAM filters.
Transmit packets matching these filters will be loopbacked through hardware
loopback channel/interface (i.e, instead of sending them out of MAC interface).
Which will again match the installed filters and will be forwarded.
This way representee => representor and representor => representee packet
path is achieved. These rules get installed when representors are created
and gets active/deactivate based on the representor/representee interface state.

Usage example:

 - Change device to switchdev mode::

        # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev

 - List of representor devices on the system::

        # ip link show
        Rpf1vf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000 link/ether f6:43:83:ee:26:21 brd ff:ff:ff:ff:ff:ff
        Rpf1vf1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000 link/ether 12:b2:54:0e:24:54 brd ff:ff:ff:ff:ff:ff
        Rpf1vf2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000 link/ether 4a:12:c4:4c:32:62 brd ff:ff:ff:ff:ff:ff
        Rpf1vf3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000 link/ether ca:cb:68:0e:e2:6e brd ff:ff:ff:ff:ff:ff
        Rpf2vf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000 link/ether 06:cc:ad:b4:f0:93 brd ff:ff:ff:ff:ff:ff


To delete the representors devices from the system. Change the device to legacy mode.

 - Change device to legacy mode::

        # devlink dev eswitch set pci/0002:1c:00.0 mode legacy

RVU representors can be managed using devlink ports
(see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.

 - Show devlink ports of representors::

        # devlink port
        pci/0002:1c:00.0/0: type eth netdev Rpf1vf0 flavour physical port 0 splittable false
        pci/0002:1c:00.0/1: type eth netdev Rpf1vf1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false
        pci/0002:1c:00.0/2: type eth netdev Rpf1vf2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false
        pci/0002:1c:00.0/3: type eth netdev Rpf1vf3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false

Representor function attribute와 MAC address

Function attribute

RVU representor는 representor용 function attribute를 지원합니다. representor의 port function configuration은 devlink eswitch port를 통해 수행합니다.

MAC address 설정

RVU representor driver는 devlink port function attribute mechanism으로 MAC address를 설정할 수 있습니다. `Documentation/networking/devlink/devlink-port.rst`도 참고하십시오.

port 2에 MAC address를 설정하고 결과를 확인합니다.

devlink port function set pci/0002:1c:00.0/2 hw_addr 5c:a1:1b:5e:43:11
devlink port show pci/0002:1c:00.0/2

출력의 `function` 아래에는 설정한 `hw_addr 5c:a1:1b:5e:43:11`이 표시됩니다.

Function attributes
===================

The RVU representor support function attributes for representors.
Port function configuration of the representors are supported through devlink eswitch port.

MAC address setup
-----------------

RVU representor driver support devlink port function attr mechanism to setup MAC
address. (refer to Documentation/networking/devlink/devlink-port.rst)

 - To setup MAC address for port 2::

        # devlink port function set pci/0002:1c:00.0/2 hw_addr 5c:a1:1b:5e:43:11
        # devlink port show pci/0002:1c:00.0/2
        pci/0002:1c:00.0/2: type eth netdev Rpf1vf2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false
        function:
                hw_addr 5c:a1:1b:5e:43:11

Representor TC offload

TC offload

RVU representor driver는 port representor를 이용해 tc rule을 hardware로 offload하는 기능을 구현합니다.

VLAN ID 3인 packet을 drop합니다.

tc filter add dev Rpf1vf0 protocol 802.1Q parent ffff: flower vlan_id 3 vlan_ethtype ipv4 skip_sw action drop

VLAN ID 5인 IPv4 packet에서 VLAN header를 제거한 뒤 `eth1`로 redirect합니다.

tc filter add dev Rpf1vf0 ingress protocol 802.1Q flower vlan_id 5 vlan_ethtype ipv4 skip_sw action vlan pop action mirred ingress redirect dev eth1

TC offload
==========

The rvu representor driver implements support for offloading tc rules using port representors.

 - Drop packets with vlan id 3::

        # tc filter add dev Rpf1vf0 protocol 802.1Q parent ffff: flower vlan_id 3 vlan_ethtype ipv4 skip_sw action drop

 - Redirect packets with vlan id 5 and IPv4 packets to eth1, after stripping vlan header.::

        # tc filter add dev Rpf1vf0 ingress protocol 802.1Q flower vlan_id 5 vlan_ethtype ipv4 skip_sw action vlan pop action mirred ingress redirect dev eth1