← Documents Documentation/networking/device_drivers/ethernet/intel/iavf.rst GitHub 원문 ↗

Linux 6.18.37 · Networking

Linux Base Driver for Intel(R) Ethernet Adaptive Virtual Function

Intel iavf의 PF 의존 조건, AVF base mode, VLAN·QinQ, ADq queue 구성과 VF 운용 문제를 설명합니다.

Source pathDocumentation/networking/device_drivers/ethernet/intel/iavf.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

요약·해설

iavf.rst:1-326

iavf는 Intel 700 Series 계열 PF가 제공하는 기능에 맞춰 동작하는 adaptive VF driver입니다. guest의 MSI-X 지원과 PF·kernel configuration이 선행되어야 하며, 실제 기능 범위는 공통 base mode에 PF가 협상한 기능을 더해 결정됩니다. ADq, VLAN, QinQ와 bonding은 PF가 가진 전역 상태와 queue resource 제약을 함께 확인해야 합니다.

iavf 활성화 조건
Host kernel: CONFIG_PCI_IOVi40e 또는 newer PF driverCONFIG_PCI_MSIVF 생성·할당Guest OS: MSI-X 지원iavf load

host PF부터 guest interrupt support까지 필요한 의존 관계입니다.

AVF base mode
영역기본 제공
QueueTx/Rx QP 4개, control queue 1개
InterruptMSI-X vector 5개, ITR index 1개
InterfaceVF마다 VSI 1개, TC0 1개
RSS64-entry indirection table와 key, PF가 구성
Addressunicast MAC 1개, MAC filter 16개
Offloadnon-tunneled checksum
PF 통신hardware mailbox

PF 기능과 무관하게 제공되는 최소 VF resource입니다.

VLAN tag stripping 결정
VM: rxvlan on/off 요청PF가 요청 처리PF-assigned VLAN 없음tag stripping 변경
VM: rxvlan on/off 요청PF-assigned VLAN 있음요청 무시

VM의 ethtool 요청이 PF 정책에 따라 적용되는 과정입니다.

QinQ 구성과 제한
항목내용
Outer tag802.1ad, 예: VLAN 24
Inner tag802.1Q, 예: VLAN 371
용도L2 tunneling과 VLAN 내부 traffic 분리
Receive checksum offload미지원
Cloud filter미지원
VLAN acceleration미지원

outer 802.1ad와 inner 802.1Q tag stack 운용 조건입니다.

ADq 설정 순서
module·iproute2·NVM 확인mqprio로 TC와 queue map 구성min/max bandwidth 선택hw-tc-offload oningress qdisc 추가TC filter 적용

application 전용 queue와 rate limit을 만드는 핵심 단계입니다.

ADq resource와 호환성
항목조건
TC 수interface당 최대 8개
전체 queue64와 core 수 중 작은 값 이하
Bandwidth 합port speed 이하
필요 modulesch_mqprio, act_mirred, cls_flower
필요 firmwareNVM 6.01 이상
동시 사용 불가cloud filter, DCB, MFP, Sideband Filter
Channel 변경mqprio TC 구성 중 ethtool -L 미지원

활성화 전에 확인할 상한과 배타 기능입니다.

주요 troubleshooting
증상원인·대응
Bonding active slave 실패PF가 설정한 VF MAC 동기화 충돌
VM-client traffic 차단VF trust off + spoof checking on
Port unload 정지active VM이 VF를 release할 때까지 대기
Traffic Class 설정 실패iavf에서는 3개 이하로 제한
Virtual link 누락host PF에서 ethtool -r 반복
VF MAC 변경host에서 static MAC 할당
ARP traffic 불균형arp_filter=1 또는 broadcast domain 분리

VF lifecycle과 network policy에서 자주 마주치는 증상입니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 .. SPDX-License-Identifier: GPL-2.0+
2
3 =================================================================
4 Linux Base Driver for Intel(R) Ethernet Adaptive Virtual Function
5 =================================================================
6
7 Intel Ethernet Adaptive Virtual Function Linux driver.
8 Copyright(c) 2013-2018 Intel Corporation.
9
10 Contents
11 ========
12
13 - Overview
14 - Identifying Your Adapter
15 - Additional Configurations
16 - Known Issues/Troubleshooting
17 - Support
18
19 Overview
20 ========
21
22 This file describes the iavf Linux Base Driver. This driver was formerly
23 called i40evf.
24
25 The iavf driver supports the below mentioned virtual function devices and
26 can only be activated on kernels running the i40e or newer Physical Function
27 (PF) driver compiled with CONFIG_PCI_IOV. The iavf driver requires
28 CONFIG_PCI_MSI to be enabled.
29
30 The guest OS loading the iavf driver must support MSI-X interrupts.
31
32 Identifying Your Adapter
33 ========================
34
35 The driver in this kernel is compatible with devices based on the following:
36 * Intel(R) XL710 X710 Virtual Function
37 * Intel(R) X722 Virtual Function
38 * Intel(R) XXV710 Virtual Function
39 * Intel(R) Ethernet Adaptive Virtual Function
40
41 For the best performance, make sure the latest NVM/FW is installed on your
42 device.
43
44 For information on how to identify your adapter, and for the latest NVM/FW
45 images and Intel network drivers, refer to the Intel Support website:
46 https://www.intel.com/support
47
48
49 Additional Features and Configurations
50 ======================================
51
52 Viewing Link Messages
53 ---------------------
54 Link messages will not be displayed to the console if the distribution is
55 restricting system messages. In order to see network driver link messages on
56 your console, set dmesg to eight by entering the following::
57
58 # dmesg -n 8
59
60 NOTE:
61 This setting is not saved across reboots.
62
63 ethtool
64 -------
65 The driver utilizes the ethtool interface for driver configuration and
66 diagnostics, as well as displaying statistical information. The latest ethtool
67 version is required for this functionality. Download it at:
68 https://www.kernel.org/pub/software/network/ethtool/
69
70 Setting VLAN Tag Stripping
71 --------------------------
72 If you have applications that require Virtual Functions (VFs) to receive
73 packets with VLAN tags, you can disable VLAN tag stripping for the VF. The
74 Physical Function (PF) processes requests issued from the VF to enable or
75 disable VLAN tag stripping. Note that if the PF has assigned a VLAN to a VF,
76 then requests from that VF to set VLAN tag stripping will be ignored.
77
78 To enable/disable VLAN tag stripping for a VF, issue the following command
79 from inside the VM in which you are running the VF::
80
81 # ethtool -K <if_name> rxvlan on/off
82
83 or alternatively::
84
85 # ethtool --offload <if_name> rxvlan on/off
86
87 Adaptive Virtual Function
88 -------------------------
89 Adaptive Virtual Function (AVF) allows the virtual function driver, or VF, to
90 adapt to changing feature sets of the physical function driver (PF) with which
91 it is associated. This allows system administrators to update a PF without
92 having to update all the VFs associated with it. All AVFs have a single common
93 device ID and branding string.
94
95 AVFs have a minimum set of features known as "base mode," but may provide
96 additional features depending on what features are available in the PF with
97 which the AVF is associated. The following are base mode features:
98
99 - 4 Queue Pairs (QP) and associated Configuration Status Registers (CSRs)
100 for Tx/Rx
101 - i40e descriptors and ring format
102 - Descriptor write-back completion
103 - 1 control queue, with i40e descriptors, CSRs and ring format
104 - 5 MSI-X interrupt vectors and corresponding i40e CSRs
105 - 1 Interrupt Throttle Rate (ITR) index
106 - 1 Virtual Station Interface (VSI) per VF
107 - 1 Traffic Class (TC), TC0
108 - Receive Side Scaling (RSS) with 64 entry indirection table and key,
109 configured through the PF
110 - 1 unicast MAC address reserved per VF
111 - 16 MAC address filters for each VF
112 - Stateless offloads - non-tunneled checksums
113 - AVF device ID
114 - HW mailbox is used for VF to PF communications (including on Windows)
115
116 IEEE 802.1ad (QinQ) Support
117 ---------------------------
118 The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
119 IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
120 "tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks
121 allow L2 tunneling and the ability to segregate traffic within a particular
122 VLAN ID, among other uses.
123
124 The following are examples of how to configure 802.1ad (QinQ)::
125
126 # ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
127 # ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
128
129 Where "24" and "371" are example VLAN IDs.
130
131 NOTES:
132 Receive checksum offloads, cloud filters, and VLAN acceleration are not
133 supported for 802.1ad (QinQ) packets.
134
135 Application Device Queues (ADq)
136 -------------------------------
137 Application Device Queues (ADq) allows you to dedicate one or more queues to a
138 specific application. This can reduce latency for the specified application,
139 and allow Tx traffic to be rate limited per application. Follow the steps below
140 to set ADq.
141
142 Requirements:
143
144 - The sch_mqprio, act_mirred and cls_flower modules must be loaded
145 - The latest version of iproute2
146 - If another driver (for example, DPDK) has set cloud filters, you cannot
147 enable ADQ
148 - Depending on the underlying PF device, ADQ cannot be enabled when the
149 following features are enabled:
150
151 + Data Center Bridging (DCB)
152 + Multiple Functions per Port (MFP)
153 + Sideband Filters
154
155 1. Create traffic classes (TCs). Maximum of 8 TCs can be created per interface.
156 The shaper bw_rlimit parameter is optional.
157
158 Example: Sets up two tcs, tc0 and tc1, with 16 queues each and max tx rate set
159 to 1Gbit for tc0 and 3Gbit for tc1.
160
161 ::
162
163 tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
164 queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
165 max_rate 1Gbit 3Gbit
166
167 map: priority mapping for up to 16 priorities to tcs (e.g. map 0 0 0 0 1 1 1 1
168 sets priorities 0-3 to use tc0 and 4-7 to use tc1)
169
170 queues: for each tc, <num queues>@<offset> (e.g. queues 16@0 16@16 assigns
171 16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16. Max total
172 number of queues for all tcs is 64 or number of cores, whichever is lower.)
173
174 hw 1 mode channel: ‘channel’ with ‘hw’ set to 1 is a new new hardware
175 offload mode in mqprio that makes full use of the mqprio options, the
176 TCs, the queue configurations, and the QoS parameters.
177
178 shaper bw_rlimit: for each tc, sets minimum and maximum bandwidth rates.
179 Totals must be equal or less than port speed.
180
181 For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
182 monitoring tools such as ``ifstat`` or ``sar -n DEV [interval] [number of samples]``
183
184 NOTE:
185 Setting up channels via ethtool (ethtool -L) is not supported when the
186 TCs are configured using mqprio.
187
188 2. Enable HW TC offload on interface::
189
190 # ethtool -K <interface> hw-tc-offload on
191
192 3. Apply TCs to ingress (RX) flow of interface::
193
194 # tc qdisc add dev <interface> ingress
195
196 NOTES:
197 - Run all tc commands from the iproute2 <pathtoiproute2>/tc/ directory
198 - ADq is not compatible with cloud filters
199 - Setting up channels via ethtool (ethtool -L) is not supported when the TCs
200 are configured using mqprio
201 - You must have iproute2 latest version
202 - NVM version 6.01 or later is required
203 - ADq cannot be enabled when any the following features are enabled: Data
204 Center Bridging (DCB), Multiple Functions per Port (MFP), or Sideband Filters
205 - If another driver (for example, DPDK) has set cloud filters, you cannot
206 enable ADq
207 - Tunnel filters are not supported in ADq. If encapsulated packets do arrive
208 in non-tunnel mode, filtering will be done on the inner headers. For example,
209 for VXLAN traffic in non-tunnel mode, PCTYPE is identified as a VXLAN
210 encapsulated packet, outer headers are ignored. Therefore, inner headers are
211 matched.
212 - If a TC filter on a PF matches traffic over a VF (on the PF), that traffic
213 will be routed to the appropriate queue of the PF, and will not be passed on
214 the VF. Such traffic will end up getting dropped higher up in the TCP/IP
215 stack as it does not match PF address data.
216 - If traffic matches multiple TC filters that point to different TCs, that
217 traffic will be duplicated and sent to all matching TC queues. The hardware
218 switch mirrors the packet to a VSI list when multiple filters are matched.
219
220
221 Known Issues/Troubleshooting
222 ============================
223
224 Bonding fails with VFs bound to an Intel(R) Ethernet Controller 700 series device
225 ---------------------------------------------------------------------------------
226 If you bind Virtual Functions (VFs) to an Intel(R) Ethernet Controller 700
227 series based device, the VF slaves may fail when they become the active slave.
228 If the MAC address of the VF is set by the PF (Physical Function) of the
229 device, when you add a slave, or change the active-backup slave, Linux bonding
230 tries to sync the backup slave's MAC address to the same MAC address as the
231 active slave. Linux bonding will fail at this point. This issue will not occur
232 if the VF's MAC address is not set by the PF.
233
234 Traffic Is Not Being Passed Between VM and Client
235 -------------------------------------------------
236 You may not be able to pass traffic between a client system and a
237 Virtual Machine (VM) running on a separate host if the Virtual Function
238 (VF, or Virtual NIC) is not in trusted mode and spoof checking is enabled
239 on the VF. Note that this situation can occur in any combination of client,
240 host, and guest operating system. For information on how to set the VF to
241 trusted mode, refer to the section "VLAN Tag Packet Steering" in this
242 readme document. For information on setting spoof checking, refer to the
243 section "MAC and VLAN anti-spoofing feature" in this readme document.
244
245 Do not unload port driver if VF with active VM is bound to it
246 -------------------------------------------------------------
247 Do not unload a port's driver if a Virtual Function (VF) with an active Virtual
248 Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
249 Once the VM shuts down, or otherwise releases the VF, the command will complete.
250
251 Using four traffic classes fails
252 --------------------------------
253 Do not try to reserve more than three traffic classes in the iavf driver. Doing
254 so will fail to set any traffic classes and will cause the driver to write
255 errors to stdout. Use a maximum of three queues to avoid this issue.
256
257 Multiple log error messages on iavf driver removal
258 --------------------------------------------------
259 If you have several VFs and you remove the iavf driver, several instances of
260 the following log errors are written to the log::
261
262 Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err ok
263 Unable to send the message to VF 2 aq_err 12
264 ARQ Overflow Error detected
265
266 Virtual machine does not get link
267 ---------------------------------
268 If the virtual machine has more than one virtual port assigned to it, and those
269 virtual ports are bound to different physical ports, you may not get link on
270 all of the virtual ports. The following command may work around the issue::
271
272 # ethtool -r <PF>
273
274 Where <PF> is the PF interface in the host, for example: p5p1. You may need to
275 run the command more than once to get link on all virtual ports.
276
277 MAC address of Virtual Function changes unexpectedly
278 ----------------------------------------------------
279 If a Virtual Function's MAC address is not assigned in the host, then the VF
280 (virtual function) driver will use a random MAC address. This random MAC
281 address may change each time the VF driver is reloaded. You can assign a static
282 MAC address in the host machine. This static MAC address will survive
283 a VF driver reload.
284
285 Driver Buffer Overflow Fix
286 --------------------------
287 The fix to resolve CVE-2016-8105, referenced in Intel SA-00069
288 https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00069.html
289 is included in this and future versions of the driver.
290
291 Multiple Interfaces on Same Ethernet Broadcast Network
292 ------------------------------------------------------
293 Due to the default ARP behavior on Linux, it is not possible to have one system
294 on two IP networks in the same Ethernet broadcast domain (non-partitioned
295 switch) behave as expected. All Ethernet interfaces will respond to IP traffic
296 for any IP address assigned to the system. This results in unbalanced receive
297 traffic.
298
299 If you have multiple interfaces in a server, either turn on ARP filtering by
300 entering::
301
302 # echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
303
304 NOTE:
305 This setting is not saved across reboots. The configuration change can be
306 made permanent by adding the following line to the file /etc/sysctl.conf::
307
308 net.ipv4.conf.all.arp_filter = 1
309
310 Another alternative is to install the interfaces in separate broadcast domains
311 (either in different switches or in a switch partitioned to VLANs).
312
313 Rx Page Allocation Errors
314 -------------------------
315 'Page allocation failure. order:0' errors may occur under stress.
316 This is caused by the way the Linux kernel reports this stressed condition.
317
318
319 Support
320 =======
321 For general information, go to the Intel support website at:
322 https://support.intel.com
323
324 If an issue is identified with the released source code on the supported kernel
325 with a supported adapter, email the specific information related to the issue
326 to intel-wired-lan@lists.osuosl.org.
327

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

개요와 adapter 식별

1-48

이 문서는 `GPL-2.0+` 라이선스를 따릅니다.

Intel(R) Ethernet Adaptive Virtual Function용 Linux Base Driver

Intel Ethernet Adaptive Virtual Function Linux driver.

Copyright(c) 2013-2018 Intel Corporation.

목차

  • 개요
  • Adapter 식별
  • 추가 configuration
  • 알려진 문제와 troubleshooting
  • 지원

개요

이 문서는 iavf Linux Base Driver를 설명합니다. 이 driver의 이전 이름은 i40evf였습니다.

iavf driver는 아래에 열거한 virtual function device를 지원합니다. `CONFIG_PCI_IOV`를 사용해 compile한 i40e 또는 그보다 새로운 Physical Function(PF) driver가 실행 중인 kernel에서만 활성화할 수 있습니다. iavf driver를 사용하려면 `CONFIG_PCI_MSI`도 활성화해야 합니다.

iavf driver를 load하는 guest OS는 MSI-X interrupt를 지원해야 합니다.

Adapter 식별

이 kernel의 driver는 다음 device 기반 adapter와 호환됩니다.

  • Intel(R) XL710 X710 Virtual Function
  • Intel(R) X722 Virtual Function
  • Intel(R) XXV710 Virtual Function
  • Intel(R) Ethernet Adaptive Virtual Function

최상의 performance를 위해 device에 최신 NVM/FW를 설치하십시오.

adapter 식별 방법, 최신 NVM/FW image와 Intel network driver 정보는 Intel Support website `https://www.intel.com/support`를 참고하십시오.

.. SPDX-License-Identifier: GPL-2.0+

=================================================================
Linux Base Driver for Intel(R) Ethernet Adaptive Virtual Function
=================================================================

Intel Ethernet Adaptive Virtual Function Linux driver.
Copyright(c) 2013-2018 Intel Corporation.

Contents
========

- Overview
- Identifying Your Adapter
- Additional Configurations
- Known Issues/Troubleshooting
- Support

Overview
========

This file describes the iavf Linux Base Driver. This driver was formerly
called i40evf.

The iavf driver supports the below mentioned virtual function devices and
can only be activated on kernels running the i40e or newer Physical Function
(PF) driver compiled with CONFIG_PCI_IOV.  The iavf driver requires
CONFIG_PCI_MSI to be enabled.

The guest OS loading the iavf driver must support MSI-X interrupts.

Identifying Your Adapter
========================

The driver in this kernel is compatible with devices based on the following:
 * Intel(R) XL710 X710 Virtual Function
 * Intel(R) X722 Virtual Function
 * Intel(R) XXV710 Virtual Function
 * Intel(R) Ethernet Adaptive Virtual Function

For the best performance, make sure the latest NVM/FW is installed on your
device.

For information on how to identify your adapter, and for the latest NVM/FW
images and Intel network drivers, refer to the Intel Support website:
https://www.intel.com/support

Link message, ethtool과 VLAN tag stripping

49-86

추가 기능과 configuration

Link message 보기

distribution이 system message를 제한하면 link message가 console에 표시되지 않습니다. console에서 network driver link message를 보려면 dmesg level을 8로 설정합니다.

# dmesg -n 8

참고: 이 설정은 reboot 후 유지되지 않습니다.

ethtool

driver는 configuration과 진단, 통계 표시를 위해 ethtool interface를 사용합니다. 이 기능에는 최신 ethtool version이 필요하며 `https://www.kernel.org/pub/software/network/ethtool/`에서 받을 수 있습니다.

VLAN tag stripping 설정

Virtual Function(VF)이 VLAN tag가 붙은 packet을 받아야 하는 application에서는 VF의 VLAN tag stripping을 비활성화할 수 있습니다. Physical Function(PF)은 VLAN tag stripping을 켜거나 끄기 위한 VF의 요청을 처리합니다.

PF가 VF에 VLAN을 이미 할당했다면 해당 VF가 보낸 VLAN tag stripping 변경 요청은 무시됩니다.

VF를 실행하는 VM 내부에서 다음 command 중 하나로 VLAN tag stripping을 켜거나 끕니다.

# ethtool -K <if_name> rxvlan on/off
# ethtool --offload <if_name> rxvlan on/off
Additional Features and Configurations
======================================

Viewing Link Messages
---------------------
Link messages will not be displayed to the console if the distribution is
restricting system messages. In order to see network driver link messages on
your console, set dmesg to eight by entering the following::

    # dmesg -n 8

NOTE:
  This setting is not saved across reboots.

ethtool
-------
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at:
https://www.kernel.org/pub/software/network/ethtool/

Setting VLAN Tag Stripping
--------------------------
If you have applications that require Virtual Functions (VFs) to receive
packets with VLAN tags, you can disable VLAN tag stripping for the VF. The
Physical Function (PF) processes requests issued from the VF to enable or
disable VLAN tag stripping. Note that if the PF has assigned a VLAN to a VF,
then requests from that VF to set VLAN tag stripping will be ignored.

To enable/disable VLAN tag stripping for a VF, issue the following command
from inside the VM in which you are running the VF::

    # ethtool -K <if_name> rxvlan on/off

or alternatively::

    # ethtool --offload <if_name> rxvlan on/off

Adaptive Virtual Function과 QinQ

87-134

Adaptive Virtual Function

Adaptive Virtual Function(AVF)은 virtual function driver, 즉 VF가 자신과 연결된 Physical Function driver(PF)의 변경되는 feature set에 적응하게 합니다. 따라서 system administrator는 연결된 모든 VF를 함께 update하지 않고도 PF를 update할 수 있습니다. 모든 AVF는 하나의 공통 device ID와 branding string을 사용합니다.

AVF에는 `base mode`라는 최소 feature set이 있으며, 연결된 PF에서 제공하는 기능에 따라 추가 feature를 제공할 수 있습니다. base mode 기능은 다음과 같습니다.

  • Tx/Rx용 4개 Queue Pair(QP)와 관련 Configuration Status Register(CSR)
  • i40e descriptor와 ring format
  • descriptor write-back completion
  • i40e descriptor, CSR, ring format을 사용하는 control queue 1개
  • MSI-X interrupt vector 5개와 대응하는 i40e CSR
  • Interrupt Throttle Rate(ITR) index 1개
  • VF마다 Virtual Station Interface(VSI) 1개
  • Traffic Class(TC) 1개인 TC0
  • PF를 통해 구성하는 64-entry indirection table과 key 기반 Receive Side Scaling(RSS)
  • VF마다 예약된 unicast MAC address 1개
  • VF마다 MAC address filter 16개
  • non-tunneled checksum용 stateless offload
  • AVF device ID
  • Windows를 포함한 VF-PF 통신용 hardware mailbox

IEEE 802.1ad(QinQ) 지원

QinQ로도 불리는 IEEE 802.1ad 표준은 하나의 Ethernet frame 안에 여러 VLAN ID를 넣을 수 있게 합니다. VLAN ID는 흔히 tag라고 하며, 여러 VLAN ID의 집합은 tag stack이라고 합니다. tag stack은 L2 tunneling과 특정 VLAN ID 내부의 traffic 분리 등에 사용합니다.

802.1ad QinQ 구성 예시는 다음과 같습니다.

# ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
# ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371

여기서 `24`와 `371`은 VLAN ID 예시입니다.

참고: 802.1ad(QinQ) packet에는 receive checksum offload, cloud filter와 VLAN acceleration을 지원하지 않습니다.

Adaptive Virtual Function
-------------------------
Adaptive Virtual Function (AVF) allows the virtual function driver, or VF, to
adapt to changing feature sets of the physical function driver (PF) with which
it is associated. This allows system administrators to update a PF without
having to update all the VFs associated with it. All AVFs have a single common
device ID and branding string.

AVFs have a minimum set of features known as "base mode," but may provide
additional features depending on what features are available in the PF with
which the AVF is associated. The following are base mode features:

- 4 Queue Pairs (QP) and associated Configuration Status Registers (CSRs)
  for Tx/Rx
- i40e descriptors and ring format
- Descriptor write-back completion
- 1 control queue, with i40e descriptors, CSRs and ring format
- 5 MSI-X interrupt vectors and corresponding i40e CSRs
- 1 Interrupt Throttle Rate (ITR) index
- 1 Virtual Station Interface (VSI) per VF
- 1 Traffic Class (TC), TC0
- Receive Side Scaling (RSS) with 64 entry indirection table and key,
  configured through the PF
- 1 unicast MAC address reserved per VF
- 16 MAC address filters for each VF
- Stateless offloads - non-tunneled checksums
- AVF device ID
- HW mailbox is used for VF to PF communications (including on Windows)

IEEE 802.1ad (QinQ) Support
---------------------------
The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
"tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks
allow L2 tunneling and the ability to segregate traffic within a particular
VLAN ID, among other uses.

The following are examples of how to configure 802.1ad (QinQ)::

    # ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
    # ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371

Where "24" and "371" are example VLAN IDs.

NOTES:
  Receive checksum offloads, cloud filters, and VLAN acceleration are not
  supported for 802.1ad (QinQ) packets.

Application Device Queues 요구사항과 TC 생성

135-186

Application Device Queues(ADq)

ADq를 사용하면 하나 이상의 queue를 특정 application 전용으로 지정할 수 있습니다. 지정한 application의 latency를 줄이고 application별 Tx traffic rate limit을 적용할 수 있습니다.

요구사항

  • `sch_mqprio`, `act_mirred`, `cls_flower` module을 load해야 합니다.
  • 최신 version의 iproute2가 필요합니다.
  • DPDK 같은 다른 driver가 cloud filter를 설정했다면 ADq를 활성화할 수 없습니다.
  • 기반 PF device에 따라 DCB, MFP 또는 Sideband Filter가 활성화된 상태에서는 ADq를 활성화할 수 없습니다.

1. Traffic Class(TC)를 만듭니다. interface마다 최대 8개 TC를 만들 수 있으며 `shaper bw_rlimit` parameter는 선택 사항입니다.

다음 예시는 TC0와 TC1에 각각 queue 16개를 배치하고 최대 Tx rate를 TC0는 1 Gbit, TC1은 3 Gbit로 설정합니다.

tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
max_rate 1Gbit 3Gbit

`map`은 최대 16개 priority를 TC에 대응시킵니다. 예를 들어 `map 0 0 0 0 1 1 1 1`은 priority 0-3을 TC0에, 4-7을 TC1에 연결합니다.

`queues`는 각 TC에 `<queue 수>@<offset>` 형식으로 queue를 배치합니다. `16@0 16@16`은 offset 0부터 16개를 TC0에, offset 16부터 16개를 TC1에 할당합니다. 모든 TC의 전체 queue 수 상한은 64와 core 수 중 더 작은 값입니다.

`hw 1 mode channel`은 mqprio option, TC, queue configuration과 QoS parameter를 온전히 사용하는 hardware offload mode입니다.

`shaper bw_rlimit`은 TC마다 최소·최대 bandwidth rate를 설정합니다. 전체 합계는 port speed 이하여야 합니다.

`min_rate 1Gbit 3Gbit` 같은 bandwidth limit은 `ifstat` 또는 `sar -n DEV [interval] [number of samples]` 같은 network monitoring tool로 확인합니다.

참고: mqprio로 TC를 구성한 상태에서는 `ethtool -L`로 channel을 설정할 수 없습니다.

Application Device Queues (ADq)
-------------------------------
Application Device Queues (ADq) allows you to dedicate one or more queues to a
specific application. This can reduce latency for the specified application,
and allow Tx traffic to be rate limited per application. Follow the steps below
to set ADq.

Requirements:

- The sch_mqprio, act_mirred and cls_flower modules must be loaded
- The latest version of iproute2
- If another driver (for example, DPDK) has set cloud filters, you cannot
  enable ADQ
- Depending on the underlying PF device, ADQ cannot be enabled when the
  following features are enabled:

  + Data Center Bridging (DCB)
  + Multiple Functions per Port (MFP)
  + Sideband Filters

1. Create traffic classes (TCs). Maximum of 8 TCs can be created per interface.
The shaper bw_rlimit parameter is optional.

Example: Sets up two tcs, tc0 and tc1, with 16 queues each and max tx rate set
to 1Gbit for tc0 and 3Gbit for tc1.

::

    tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
    queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
    max_rate 1Gbit 3Gbit

map: priority mapping for up to 16 priorities to tcs (e.g. map 0 0 0 0 1 1 1 1
sets priorities 0-3 to use tc0 and 4-7 to use tc1)

queues: for each tc, <num queues>@<offset> (e.g. queues 16@0 16@16 assigns
16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16. Max total
number of queues for all tcs is 64 or number of cores, whichever is lower.)

hw 1 mode channel: ‘channel’ with ‘hw’ set to 1 is a new new hardware
offload mode in mqprio that makes full use of the mqprio options, the
TCs, the queue configurations, and the QoS parameters.

shaper bw_rlimit: for each tc, sets minimum and maximum bandwidth rates.
Totals must be equal or less than port speed.

For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
monitoring tools such as ``ifstat`` or ``sar -n DEV [interval] [number of samples]``

NOTE:
  Setting up channels via ethtool (ethtool -L) is not supported when the
  TCs are configured using mqprio.

ADq hardware offload와 제약

187-220

2. interface에서 hardware TC offload를 활성화합니다.

# ethtool -K <interface> hw-tc-offload on

3. interface의 ingress(RX) flow에 TC를 적용합니다.

# tc qdisc add dev <interface> ingress

참고

  • 모든 tc command는 iproute2의 `<pathtoiproute2>/tc/` directory에서 실행합니다.
  • ADq는 cloud filter와 호환되지 않습니다.
  • mqprio로 TC를 구성한 상태에서는 `ethtool -L`로 channel을 설정할 수 없습니다.
  • 최신 iproute2가 필요합니다.
  • NVM version 6.01 이상이 필요합니다.
  • DCB, MFP 또는 Sideband Filter가 활성화되어 있으면 ADq를 활성화할 수 없습니다.
  • DPDK 같은 다른 driver가 cloud filter를 설정했다면 ADq를 활성화할 수 없습니다.
  • ADq는 tunnel filter를 지원하지 않습니다.

encapsulated packet이 non-tunnel mode로 들어오면 inner header를 기준으로 filtering합니다. 예를 들어 non-tunnel mode의 VXLAN traffic은 VXLAN encapsulated packet PCTYPE로 식별되고 outer header는 무시되므로 inner header가 matching 대상입니다.

PF의 TC filter가 PF를 통과하는 VF traffic과 일치하면 해당 traffic은 PF의 적절한 queue로 route되며 VF에 전달되지 않습니다. 이후 PF address data와 맞지 않아 TCP/IP stack의 상위 계층에서 drop됩니다.

traffic이 서로 다른 TC를 가리키는 여러 TC filter와 일치하면 packet은 복제되어 일치하는 모든 TC queue로 전송됩니다. 여러 filter가 일치할 때 hardware switch는 packet을 VSI list에 mirror합니다.


2. Enable HW TC offload on interface::

    # ethtool -K <interface> hw-tc-offload on

3. Apply TCs to ingress (RX) flow of interface::

    # tc qdisc add dev <interface> ingress

NOTES:
 - Run all tc commands from the iproute2 <pathtoiproute2>/tc/ directory
 - ADq is not compatible with cloud filters
 - Setting up channels via ethtool (ethtool -L) is not supported when the TCs
   are configured using mqprio
 - You must have iproute2 latest version
 - NVM version 6.01 or later is required
 - ADq cannot be enabled when any the following features are enabled: Data
   Center Bridging (DCB), Multiple Functions per Port (MFP), or Sideband Filters
 - If another driver (for example, DPDK) has set cloud filters, you cannot
   enable ADq
 - Tunnel filters are not supported in ADq. If encapsulated packets do arrive
   in non-tunnel mode, filtering will be done on the inner headers.  For example,
   for VXLAN traffic in non-tunnel mode, PCTYPE is identified as a VXLAN
   encapsulated packet, outer headers are ignored. Therefore, inner headers are
   matched.
 - If a TC filter on a PF matches traffic over a VF (on the PF), that traffic
   will be routed to the appropriate queue of the PF, and will not be passed on
   the VF. Such traffic will end up getting dropped higher up in the TCP/IP
   stack as it does not match PF address data.
 - If traffic matches multiple TC filters that point to different TCs, that
   traffic will be duplicated and sent to all matching TC queues.  The hardware
   switch mirrors the packet to a VSI list when multiple filters are matched.

Bonding, trusted mode와 traffic class 문제

221-255

알려진 문제와 troubleshooting

Intel(R) Ethernet Controller 700 Series device에 bind된 VF에서 bonding 실패

Intel 700 Series 기반 device에 Virtual Function(VF)을 bind하면 VF slave가 active slave가 될 때 실패할 수 있습니다. PF가 VF의 MAC address를 설정한 경우 slave를 추가하거나 active-backup slave를 바꿀 때 Linux bonding이 backup slave MAC address를 active slave와 같게 동기화하려 하며 이 단계에서 실패합니다. PF가 VF MAC address를 설정하지 않았다면 이 문제는 발생하지 않습니다.

VM과 client 사이에 traffic이 전달되지 않음

Virtual Function(VF 또는 virtual NIC)이 trusted mode가 아니고 VF에서 spoof checking이 활성화되어 있으면 서로 다른 host에 있는 client system과 Virtual Machine(VM) 사이에서 traffic이 전달되지 않을 수 있습니다. client, host, guest OS의 어떤 조합에서도 발생할 수 있습니다.

VF trusted mode 설정은 이 readme의 `VLAN Tag Packet Steering` 절을, spoof checking 설정은 `MAC and VLAN anti-spoofing feature` 절을 참고하십시오.

active VM에 bind된 VF가 있으면 port driver를 unload하지 마십시오.

active VM에 bind된 VF가 있는 port의 driver를 unload하면 port가 정지한 것처럼 보입니다. VM이 종료되거나 다른 방식으로 VF를 release하면 command가 완료됩니다.

Traffic Class 4개 사용 실패

iavf driver에 3개를 초과하는 traffic class를 예약하지 마십시오. 4개 이상을 시도하면 모든 traffic class 설정이 실패하고 driver가 stdout에 error를 기록합니다. 이 문제를 피하려면 queue를 최대 3개 사용하십시오.

Known Issues/Troubleshooting
============================

Bonding fails with VFs bound to an Intel(R) Ethernet Controller 700 series device
---------------------------------------------------------------------------------
If you bind Virtual Functions (VFs) to an Intel(R) Ethernet Controller 700
series based device, the VF slaves may fail when they become the active slave.
If the MAC address of the VF is set by the PF (Physical Function) of the
device, when you add a slave, or change the active-backup slave, Linux bonding
tries to sync the backup slave's MAC address to the same MAC address as the
active slave. Linux bonding will fail at this point. This issue will not occur
if the VF's MAC address is not set by the PF.

Traffic Is Not Being Passed Between VM and Client
-------------------------------------------------
You may not be able to pass traffic between a client system and a
Virtual Machine (VM) running on a separate host if the Virtual Function
(VF, or Virtual NIC) is not in trusted mode and spoof checking is enabled
on the VF. Note that this situation can occur in any combination of client,
host, and guest operating system. For information on how to set the VF to
trusted mode, refer to the section "VLAN Tag Packet Steering" in this
readme document. For information on setting spoof checking, refer to the
section "MAC and VLAN anti-spoofing feature" in this readme document.

Do not unload port driver if VF with active VM is bound to it
-------------------------------------------------------------
Do not unload a port's driver if a Virtual Function (VF) with an active Virtual
Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
Once the VM shuts down, or otherwise releases the VF, the command will complete.

Using four traffic classes fails
--------------------------------
Do not try to reserve more than three traffic classes in the iavf driver. Doing
so will fail to set any traffic classes and will cause the driver to write
errors to stdout. Use a maximum of three queues to avoid this issue.

Driver 제거, virtual link와 MAC address 문제

256-283

iavf driver 제거 시 여러 log error message

VF가 여러 개 있는 상태에서 iavf driver를 제거하면 다음 log error가 여러 번 기록됩니다.

Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err ok
Unable to send the message to VF 2 aq_err 12
ARQ Overflow Error detected

Virtual machine이 link를 얻지 못함

virtual machine에 둘 이상의 virtual port가 할당되고 그 port들이 서로 다른 physical port에 bind되어 있으면 모든 virtual port에서 link를 얻지 못할 수 있습니다. 다음 command로 문제를 우회할 수 있습니다.

# ethtool -r <PF>

`<PF>`는 host의 PF interface이며 예를 들면 `p5p1`입니다. 모든 virtual port에서 link를 얻으려면 command를 여러 번 실행해야 할 수 있습니다.

Virtual Function의 MAC address가 예기치 않게 변경됨

host에서 VF MAC address를 할당하지 않으면 VF driver가 random MAC address를 사용합니다. 이 random address는 VF driver를 reload할 때마다 바뀔 수 있습니다. host machine에 static MAC address를 할당하면 VF driver reload 후에도 유지됩니다.


Multiple log error messages on iavf driver removal
--------------------------------------------------
If you have several VFs and you remove the iavf driver, several instances of
the following log errors are written to the log::

    Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err ok
    Unable to send the message to VF 2 aq_err 12
    ARQ Overflow Error detected

Virtual machine does not get link
---------------------------------
If the virtual machine has more than one virtual port assigned to it, and those
virtual ports are bound to different physical ports, you may not get link on
all of the virtual ports. The following command may work around the issue::

    # ethtool -r <PF>

Where <PF> is the PF interface in the host, for example: p5p1. You may need to
run the command more than once to get link on all virtual ports.

MAC address of Virtual Function changes unexpectedly
----------------------------------------------------
If a Virtual Function's MAC address is not assigned in the host, then the VF
(virtual function) driver will use a random MAC address. This random MAC
address may change each time the VF driver is reloaded. You can assign a static
MAC address in the host machine. This static MAC address will survive
a VF driver reload.

Security fix, ARP filtering과 page allocation

284-318

Driver buffer overflow 수정

Intel SA-00069에서 참조한 CVE-2016-8105 해결 수정은 이 driver version과 이후 version에 포함되어 있습니다. advisory는 `https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00069.html`을 참고하십시오.

같은 Ethernet broadcast network의 여러 interface

Linux의 default ARP 동작 때문에 partition되지 않은 같은 Ethernet broadcast domain에서 한 system이 두 IP network에 연결될 때 기대한 대로 동작하지 않습니다. 모든 Ethernet interface가 system에 할당된 어떤 IP address로 온 traffic에도 응답하므로 receive traffic이 불균형해집니다.

server에 interface가 여러 개라면 ARP filtering을 활성화합니다.

# echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

참고: 이 설정은 reboot 후 유지되지 않습니다. `/etc/sysctl.conf`에 다음 줄을 추가하면 영구적으로 적용할 수 있습니다.

net.ipv4.conf.all.arp_filter = 1

또 다른 방법은 interface를 서로 다른 broadcast domain에 두는 것입니다. 서로 다른 switch를 사용하거나 switch를 VLAN으로 partition할 수 있습니다.

Rx page allocation error

부하가 높은 상황에서 `Page allocation failure. order:0` error가 발생할 수 있습니다. 이는 Linux kernel이 이 stress 상태를 보고하는 방식 때문에 나타납니다.


Driver Buffer Overflow Fix
--------------------------
The fix to resolve CVE-2016-8105, referenced in Intel SA-00069
https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00069.html
is included in this and future versions of the driver.

Multiple Interfaces on Same Ethernet Broadcast Network
------------------------------------------------------
Due to the default ARP behavior on Linux, it is not possible to have one system
on two IP networks in the same Ethernet broadcast domain (non-partitioned
switch) behave as expected. All Ethernet interfaces will respond to IP traffic
for any IP address assigned to the system. This results in unbalanced receive
traffic.

If you have multiple interfaces in a server, either turn on ARP filtering by
entering::

    # echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

NOTE:
  This setting is not saved across reboots. The configuration change can be
  made permanent by adding the following line to the file /etc/sysctl.conf::

    net.ipv4.conf.all.arp_filter = 1

Another alternative is to install the interfaces in separate broadcast domains
(either in different switches or in a switch partitioned to VLANs).

Rx Page Allocation Errors
-------------------------
'Page allocation failure. order:0' errors may occur under stress.
This is caused by the way the Linux kernel reports this stressed condition.

지원

319-326

지원

일반 정보는 Intel support website `https://support.intel.com`을 참고하십시오.

지원되는 adapter와 kernel에서 공개 source code의 문제가 확인되면 문제에 관한 구체적인 정보를 `intel-wired-lan@lists.osuosl.org`로 보내십시오.

Support
=======
For general information, go to the Intel support website at:
https://support.intel.com

If an issue is identified with the released source code on the supported kernel
with a supported adapter, email the specific information related to the issue
to intel-wired-lan@lists.osuosl.org.