요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
Port counter와 hardware counter
sysfs-class-infiniband:85-216Link error, packet·octet, discard·wait counter와 hardware counter update lifespan을 다룹니다.
GID, MAD와 verbs ABI
sysfs-class-infiniband:219-284RoCE GID의 netdev·type, umad·issm port mapping과 uverbs ABI version을 다룹니다.
Mellanox mthca·mlx4와 IOV
sysfs-class-infiniband:287-400HCA identity와 SR-IOV GID·P_Key paravirtualization, VF QP0 SMI 제어를 다룹니다.
Driver별 HCA 확장
sysfs-class-infiniband:403-784cxgb4, qib, mlx5, usnic, ocrdma, hfi1, qedr, PVRDMA, bnxt_re의 board·context·congestion·lane mapping·SDMA attribute를 다룹니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
sysfs interface common for all infiniband devices
-------------------------------------------------
What: /sys/class/infiniband/<device>/node_type
What: /sys/class/infiniband/<device>/node_guid
What: /sys/class/infiniband/<device>/sys_image_guid
Date: Apr, 2005
KernelVersion: v2.6.12
Contact: linux-rdma@vger.kernel.org
Description:
=============== ===========================================
node_type: (RO) Node type (CA, RNIC, usNIC, usNIC UDP,
switch or router)
node_guid: (RO) Node GUID
sys_image_guid: (RO) System image GUID
=============== ===========================================
What: /sys/class/infiniband/<device>/node_desc
Date: Feb, 2006
KernelVersion: v2.6.17
Contact: linux-rdma@vger.kernel.org
Description:
(RW) Update the node description with information such as the
node's hostname, so that IB network management software can tie
its view to the real world.
What: /sys/class/infiniband/<device>/fw_ver
Date: Jun, 2016
KernelVersion: v4.10
Contact: linux-rdma@vger.kernel.org
Description:
(RO) Display firmware version
What: /sys/class/infiniband/<device>/ports/<port-num>/lid
What: /sys/class/infiniband/<device>/ports/<port-num>/rate
What: /sys/class/infiniband/<device>/ports/<port-num>/lid_mask_count
What: /sys/class/infiniband/<device>/ports/<port-num>/sm_sl
What: /sys/class/infiniband/<device>/ports/<port-num>/sm_lid
What: /sys/class/infiniband/<device>/ports/<port-num>/state
What: /sys/class/infiniband/<device>/ports/<port-num>/phys_state
What: /sys/class/infiniband/<device>/ports/<port-num>/cap_mask
Date: Apr, 2005
KernelVersion: v2.6.12
Contact: linux-rdma@vger.kernel.org
Description:
=============== ===============================================
lid: (RO) Port LID
rate: (RO) Port data rate (active width * active
speed)
lid_mask_count: (RO) Port LID mask count
sm_sl: (RO) Subnet manager SL for port's subnet
sm_lid: (RO) Subnet manager LID for port's subnet
state: (RO) Port state (DOWN, INIT, ARMED, ACTIVE or
ACTIVE_DEFER)
phys_state: (RO) Port physical state (Sleep, Polling,
LinkUp, etc)
cap_mask: (RO) Port capability mask. 2 bits here are
settable- IsCommunicationManagementSupported
(set when CM module is loaded) and IsSM (set
via open of issmN file).
=============== ===============================================
What: /sys/class/infiniband/<device>/ports/<port-num>/link_layer
Date: Oct, 2010
KernelVersion: v2.6.37
Contact: linux-rdma@vger.kernel.org
Description:
(RO) Link layer type information (Infiniband or Ethernet type)
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/symbol_error
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_remote_physical_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_switch_relay_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/link_error_recovery
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_constraint_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_contraint_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/local_link_integrity_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/excessive_buffer_overrun_errors
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_data
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_data
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_packets
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_packets
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/unicast_rcv_packets
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/unicast_xmit_packets
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/multicast_rcv_packets
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/multicast_xmit_packets
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/link_downed
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_discards
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/VL15_dropped
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_wait
Date: Apr, 2005
KernelVersion: v2.6.12
Contact: linux-rdma@vger.kernel.org
Description:
**Errors info**:
symbol_error: (RO) Total number of minor link errors detected on
one or more physical lanes.
port_rcv_errors : (RO) Total number of packets containing an
error that were received on the port.
port_rcv_remote_physical_errors : (RO) Total number of packets
marked with the EBP delimiter received on the port.
port_rcv_switch_relay_errors : (RO) Total number of packets
received on the port that were discarded because they could not
be forwarded by the switch relay.
link_error_recovery: (RO) Total number of times the Port
Training state machine has successfully completed the link error
recovery process.
port_xmit_constraint_errors: (RO) Total number of packets not
transmitted from the switch physical port due to outbound raw
filtering or failing outbound partition or IP version check.
port_rcv_constraint_errors: (RO) Total number of packets
received on the switch physical port that are discarded due to
inbound raw filtering or failing inbound partition or IP version
check.
local_link_integrity_errors: (RO) The number of times that the
count of local physical errors exceeded the threshold specified
by LocalPhyErrors
excessive_buffer_overrun_errors: (RO) This counter, indicates an
input buffer overrun. It indicates possible misconfiguration of
a port, either by the Subnet Manager (SM) or by user
intervention. It can also indicate hardware issues or extremely
poor link signal integrity
**Data info**:
port_xmit_data: (RO) Total number of data octets, divided by 4
(lanes), transmitted on all VLs. This is 64 bit counter
port_rcv_data: (RO) Total number of data octets, divided by 4
(lanes), received on all VLs. This is 64 bit counter.
port_xmit_packets: (RO) Total number of packets transmitted on
all VLs from this port. This may include packets with errors.
This is 64 bit counter.
port_rcv_packets: (RO) Total number of packets (this may include
packets containing Errors. This is 64 bit counter.
link_downed: (RO) Total number of times the Port Training state
machine has failed the link error recovery process and downed
the link.
unicast_rcv_packets: (RO) Total number of unicast packets,
including unicast packets containing errors.
unicast_xmit_packets: (RO) Total number of unicast packets
transmitted on all VLs from the port. This may include unicast
packets with errors.
multicast_rcv_packets: (RO) Total number of multicast packets,
including multicast packets containing errors.
multicast_xmit_packets: (RO) Total number of multicast packets
transmitted on all VLs from the port. This may include multicast
packets with errors.
**Misc info**:
port_xmit_discards: (RO) Total number of outbound packets
discarded by the port because the port is down or congested.
VL15_dropped: (RO) Number of incoming VL15 packets dropped due
to resource limitations (e.g., lack of buffers) of the port.
port_xmit_wait: (RO) The number of ticks during which the port
had data to transmit but no data was sent during the entire tick
(either because of insufficient credits or because of lack of
arbitration).
Each of these files contains the corresponding value from the
port's Performance Management PortCounters attribute, as
described in the InfiniBand Architecture Specification.
What: /sys/class/infiniband/<device-name>/hw_counters/lifespan
What: /sys/class/infiniband/<device-name>/ports/<port-num>/hw_counters/lifespan
Date: May, 2016
KernelVersion: 4.6
Contact: linux-rdma@vger.kernel.org
Description:
The optional "hw_counters" subdirectory can be under either the
parent device or the port subdirectories or both. If present,
there are a list of counters provided by the hardware. They may
match some of the counters in the counters directory, but they
often include many other counters. In addition to the various
counters, there will be a file named "lifespan" that configures
how frequently the core should update the counters when they are
being accessed (counters are not updated if they are not being
accessed). The lifespan is in milliseconds and defaults to 10
unless set to something else by the driver. Users may echo a
value between 0-10000 to the lifespan file to set the length
of time between updates in milliseconds.
What: /sys/class/infiniband/<hca>/ports/<port-number>/gid_attrs/ndevs/<gid-index>
Date: November 29, 2015
KernelVersion: 4.4.0
Contact: linux-rdma@vger.kernel.org
Description: The net-device's name associated with the GID resides
at index <gid-index>.
What: /sys/class/infiniband/<hca>/ports/<port-number>/gid_attrs/types/<gid-index>
Date: November 29, 2015
KernelVersion: 4.4.0
Contact: linux-rdma@vger.kernel.org
Description: The RoCE type of the associated GID resides at index <gid-index>.
This could either be "IB/RoCE v1" for IB and RoCE v1 based GIDs
or "RoCE v2" for RoCE v2 based GIDs.
What: /sys/class/infiniband_mad/umad<N>/ibdev
What: /sys/class/infiniband_mad/umad<N>/port
What: /sys/class/infiniband_mad/issm<N>/ibdev
What: /sys/class/infiniband_mad/issm<N>/port
Date: Apr, 2005
KernelVersion: v2.6.12
Contact: linux-rdma@vger.kernel.org
Description:
Each port of each InfiniBand device has a "umad" device and an
"issm" device attached. For example, a two-port HCA will have
two umad devices and two issm devices, while a switch will have
one device of each type (for switch port 0).
======= =====================================
ibdev: (RO) Show Infiniband (IB) device name
port: (RO) Display port number
======= =====================================
What: /sys/class/infiniband_mad/abi_version
Date: Apr, 2005
KernelVersion: v2.6.12
Contact: linux-rdma@vger.kernel.org
Description:
(RO) Value is incremented if any changes are made that break
userspace ABI compatibility of umad & issm devices.
What: /sys/class/infiniband_verbs/uverbs<N>/ibdev
What: /sys/class/infiniband_verbs/uverbs<N>/abi_version
Date: Sept, 2005
KernelVersion: v2.6.14
Contact: linux-rdma@vger.kernel.org
Description:
=============== ===========================================
ibdev: (RO) Display Infiniband (IB) device name
abi_version: (RO) Show ABI version of IB device specific
interfaces.
=============== ===========================================
What: /sys/class/infiniband_verbs/abi_version
Date: Sep, 2005
KernelVersion: v2.6.14
Contact: linux-rdma@vger.kernel.org
Description:
(RO) Value is incremented if any changes are made that break
userspace ABI compatibility of uverbs devices.
sysfs interface for Mellanox IB HCA low-level driver (mthca)
------------------------------------------------------------
What: /sys/class/infiniband/mthcaX/hw_rev
What: /sys/class/infiniband/mthcaX/hca_type
What: /sys/class/infiniband/mthcaX/board_id
Date: Apr, 2005
KernelVersion: v2.6.12
Contact: linux-rdma@vger.kernel.org
Description:
=============== ================================================
hw_rev: (RO) Hardware revision number
hca_type: (RO) Host Channel Adapter type: MT23108, MT25208
(MT23108 compat mode), MT25208 or MT25204
board_id: (RO) Manufacturing board ID
=============== ================================================
sysfs interface for Mellanox ConnectX HCA IB driver (mlx4)
----------------------------------------------------------
What: /sys/class/infiniband/mlx4_X/hw_rev
What: /sys/class/infiniband/mlx4_X/hca_type
What: /sys/class/infiniband/mlx4_X/board_id
Date: Sep, 2007
KernelVersion: v2.6.24
Contact: linux-rdma@vger.kernel.org
Description:
=============== ===============================
hw_rev: (RO) Hardware revision number
hca_type: (RO) Host channel adapter type
board_id: (RO) Manufacturing board ID
=============== ===============================
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/gids/<n>
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/admin_guids/<n>
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/pkeys/<n>
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/mcgs/
What: /sys/class/infiniband/mlx4_X/iov/ports/<pci-slot-num>/ports/<m>/gid_idx/0
What: /sys/class/infiniband/mlx4_X/iov/ports/<pci-slot-num>/ports/<m>/pkey_idx/<n>
Date: Aug, 2012
KernelVersion: v3.6.15
Contact: linux-rdma@vger.kernel.org
Description:
The sysfs iov directory is used to manage and examine the port
P_Key and guid paravirtualization. This directory is added only
for the master -- slaves do not have it.
Under iov/ports, the administrator may examine the gid and P_Key
tables as they are present in the device (and as are seen in the
"network view" presented to the SM).
The "pkeys" and "gids" subdirectories contain one file for each
entry in the port's P_Key or GID table respectively. For
example, ports/1/pkeys/10 contains the value at index 10 in port
1's P_Key table.
======================= ==========================================
gids/<n>: (RO) The physical port gids n = 0..127
admin_guids/<n>: (RW) Allows examining or changing the
administrative state of a given GUID
n = 0..127
pkeys/<n>: (RO) Displays the contents of the physical
key table n = 0..126
mcgs/: (RO) Multicast group table
<m>/gid_idx/0: (RO) Display the GID mapping m = 1..2
<m>/pkey_idx/<n>: (RW) Writable except for RoCE pkeys.
m = 1..2, n = 0..126
Under the iov/<pci slot number>
directories, the admin may map the index
numbers in the physical tables (as under
iov/ports) to the paravirtualized index
numbers that guests see.
For example, if the administrator, for
port 1 on guest 2 maps physical pkey
index 10 to virtual index 1, then that
guest, whenever it uses its pkey index
1, will actually be using the real pkey
index 10.
======================= ==========================================
What: /sys/class/infiniband/mlx4_X/iov/<pci-slot-num>/ports/<m>/smi_enabled
What: /sys/class/infiniband/mlx4_X/iov/<pci-slot-num>/ports/<m>/enable_smi_admin
Date: May, 2014
KernelVersion: v3.15.7
Contact: linux-rdma@vger.kernel.org
Description:
Enabling QP0 on VFs for selected VF/port. By default, no VFs are
enabled for QP0 operation.
================= ==== ===========================================
smi_enabled: (RO) Indicates whether smi is currently enabled
for the indicated VF/port
enable_smi_admin: (RW) Used by the admin to request that smi
capability be enabled or disabled for the
indicated VF/port. 0 = disable, 1 = enable.
================= ==== ===========================================
The requested enablement will occur at the next reset of the VF
(e.g. driver restart on the VM which owns the VF).
sysfs interface for Chelsio T4/T5 RDMA driver (cxgb4)
-----------------------------------------------------
What: /sys/class/infiniband/cxgb4_X/hw_rev
What: /sys/class/infiniband/cxgb4_X/hca_type
What: /sys/class/infiniband/cxgb4_X/board_id
Date: Apr, 2010
KernelVersion: v2.6.35
Contact: linux-rdma@vger.kernel.org
Description:
=============== =============================================
hw_rev: (RO) Hardware revision number
hca_type: (RO) Driver short name. Should normally match
the name in its bus driver structure (e.g.
pci_driver::name)
board_id: (RO) Manufacturing board id. (Vendor + device
information)
=============== =============================================
sysfs interface for Intel IB driver qib
---------------------------------------
What: /sys/class/infiniband/qibX/version
What: /sys/class/infiniband/qibX/hw_rev
What: /sys/class/infiniband/qibX/hca_type
What: /sys/class/infiniband/qibX/board_id
What: /sys/class/infiniband/qibX/boardversion
What: /sys/class/infiniband/qibX/nctxts
What: /sys/class/infiniband/qibX/localbus_info
What: /sys/class/infiniband/qibX/tempsense
What: /sys/class/infiniband/qibX/serial
What: /sys/class/infiniband/qibX/nfreectxts
What: /sys/class/infiniband/qibX/chip_reset
Date: May, 2010
KernelVersion: v2.6.35
Contact: linux-rdma@vger.kernel.org
Description:
=============== ======================================================
version: (RO) Display version information of installed software
and drivers.
hw_rev: (RO) Hardware revision number
hca_type: (RO) Host channel adapter type
board_id: (RO) Manufacturing board id
boardversion: (RO) Current version of the chip architecture
nctxts: (RO) Return the number of user ports (contexts)
available
localbus_info: (RO) Human readable localbus info
tempsense: (RO) Display temp sense registers in decimal
serial: (RO) Serial number of the HCA
nfreectxts: (RO) The number of free user ports (contexts)
available.
chip_reset: (WO) Reset the chip if possible by writing
"reset" to this file. Only allowed if no user
contexts are open that use chip resources.
=============== ======================================================
What: /sys/class/infiniband/qibX/ports/<N>/sl2vl/[0-15]
Date: May, 2010
KernelVersion: v2.6.35
Contact: linux-rdma@vger.kernel.org
Description:
(RO) The directory contains 16 files numbered 0-15 that specify
the Service Level (SL). Listing the SL files returns the Virtual
Lane (VL) as programmed by the SL.
What: /sys/class/infiniband/qibX/ports/<N>/CCMgtA/cc_settings_bin
What: /sys/class/infiniband/qibX/ports/<N>/CCMgtA/cc_table_bin
Date: May, 2010
KernelVersion: v2.6.35
Contact: linux-rdma@vger.kernel.org
Description:
Per-port congestion control. Both are binary attributes.
=============== ================================================
cc_table_bin (RO) Congestion control table size followed by
table entries.
cc_settings_bin (RO) Congestion settings: port control, control
map and an array of 16 entries for the
congestion entries - increase, timer, event log
trigger threshold and the minimum injection rate
delay.
=============== ================================================
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/loopback
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/led_override
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/hrtbt_enable
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/status
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/status_str
Date: May, 2010
KernelVersion: v2.6.35
Contact: linux-rdma@vger.kernel.org
Description:
[to be documented]
=============== ===============================================
loopback: (WO)
led_override: (WO)
hrtbt_enable: (RW)
status: (RO)
status_str: (RO) Displays information about the link state,
possible cable/switch problems, and hardware
errors. Possible states are- "Initted",
"Present", "IB_link_up", "IB_configured" or
"Fatal_Hardware_Error".
=============== ===============================================
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rc_resends
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/seq_naks
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rdma_seq
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rnr_naks
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/other_naks
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rc_timeouts
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/look_pkts
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/pkt_drops
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/dma_wait
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/unaligned
Date: May, 2010
KernelVersion: v2.6.35
Contact: linux-rdma@vger.kernel.org
Description:
[to be documented]
sysfs interface for Mellanox Connect-IB HCA driver mlx5
-------------------------------------------------------
What: /sys/class/infiniband/mlx5_X/hw_rev
What: /sys/class/infiniband/mlx5_X/hca_type
What: /sys/class/infiniband/mlx5_X/reg_pages
What: /sys/class/infiniband/mlx5_X/fw_pages
Date: Jul, 2013
KernelVersion: v3.11
Contact: linux-rdma@vger.kernel.org
Description:
[to be documented]
sysfs interface for Cisco VIC (usNIC) Verbs Driver
--------------------------------------------------
What: /sys/class/infiniband/usnic_X/board_id
What: /sys/class/infiniband/usnic_X/config
What: /sys/class/infiniband/usnic_X/qp_per_vf
What: /sys/class/infiniband/usnic_X/max_vf
What: /sys/class/infiniband/usnic_X/cq_per_vf
What: /sys/class/infiniband/usnic_X/iface
Date: Sep, 2013
KernelVersion: v3.14
Contact: Christian Benvenuti <benve@cisco.com>,
Dave Goodell <dgoodell@cisco.com>,
linux-rdma@vger.kernel.org
Description:
=============== ===============================================
board_id: (RO) Manufacturing board id
config: (RO) Report the configuration for this PF
qp_per_vf: (RO) Queue pairs per virtual function.
max_vf: (RO) Max virtual functions
cq_per_vf: (RO) Completion queue per virtual function
iface: (RO) Shows which network interface this usNIC
entry is associated to (visible with ifconfig).
=============== ===============================================
What: /sys/class/infiniband/usnic_X/qpn/summary
What: /sys/class/infiniband/usnic_X/qpn/context
Date: Sep, 2013
KernelVersion: v3.14
Contact: Christian Benvenuti <benve@cisco.com>,
Dave Goodell <dgoodell@cisco.com>,
linux-rdma@vger.kernel.org
Description:
[to be documented]
sysfs interface for Emulex RoCE HCA Driver
------------------------------------------
What: /sys/class/infiniband/ocrdmaX/hw_rev
Date: Feb, 2014
KernelVersion: v3.14
Description:
hw_rev: (RO) Hardware revision number
What: /sys/class/infiniband/ocrdmaX/hca_type
Date: Jun, 2014
KernelVersion: v3.16
Contact: linux-rdma@vger.kernel.org
Description:
hca_type: (RO) Display FW version
sysfs interface for Intel Omni-Path driver (HFI1)
-------------------------------------------------
What: /sys/class/infiniband/hfi1_X/hw_rev
What: /sys/class/infiniband/hfi1_X/board_id
What: /sys/class/infiniband/hfi1_X/nctxts
What: /sys/class/infiniband/hfi1_X/serial
What: /sys/class/infiniband/hfi1_X/chip_reset
What: /sys/class/infiniband/hfi1_X/boardversion
What: /sys/class/infiniband/hfi1_X/nfreectxts
What: /sys/class/infiniband/hfi1_X/tempsense
Date: May, 2016
KernelVersion: v4.6
Contact: linux-rdma@vger.kernel.org
Description:
=============== =============================================
hw_rev: (RO) Hardware revision number
board_id: (RO) Manufacturing board id
nctxts: (RO) Total contexts available.
serial: (RO) Board serial number
chip_reset: (WO) Write "reset" to this file to reset the
chip if possible. Only allowed if no user
contexts are open that use chip resources.
boardversion: (RO) Human readable board info
nfreectxts: (RO) The number of free user ports (contexts)
available.
tempsense: (RO) Thermal sense information
=============== =============================================
What: /sys/class/infiniband/hfi1_X/ports/<N>/CCMgtA/cc_settings_bin
What: /sys/class/infiniband/hfi1_X/ports/<N>/CCMgtA/cc_table_bin
What: /sys/class/infiniband/hfi1_X/ports/<N>/CCMgtA/cc_prescan
Date: May, 2016
KernelVersion: v4.6
Contact: linux-rdma@vger.kernel.org
Description:
Per-port congestion control.
=============== ================================================
cc_table_bin (RO) CCA tables used by PSM2 Congestion control
table size followed by table entries. Binary
attribute.
cc_settings_bin (RO) Congestion settings: port control, control
map and an array of 16 entries for the
congestion entries - increase, timer, event log
trigger threshold and the minimum injection rate
delay. Binary attribute.
cc_prescan (RW) enable prescanning for faster BECN
response. Write "on" to enable and "off" to
disable.
=============== ================================================
What: /sys/class/infiniband/hfi1_X/ports/<N>/sc2vl/[0-31]
What: /sys/class/infiniband/hfi1_X/ports/<N>/sl2sc/[0-31]
What: /sys/class/infiniband/hfi1_X/ports/<N>/vl2mtu/[0-15]
Date: May, 2016
KernelVersion: v4.6
Contact: linux-rdma@vger.kernel.org
Description:
=============== ===================================================
sc2vl/: (RO) 32 files (0 - 31) used to translate sl->vl
sl2sc/: (RO) 32 files (0 - 31) used to translate sl->sc
vl2mtu/: (RO) 16 files (0 - 15) used to determine MTU for vl
=============== ===================================================
What: /sys/class/infiniband/hfi1_X/sdma_<N>/cpu_list
What: /sys/class/infiniband/hfi1_X/sdma_<N>/vl
Date: Sept, 2016
KernelVersion: v4.8
Contact: linux-rdma@vger.kernel.org
Description:
sdma<N>/ contains one directory per sdma engine (0 - 15)
=============== ==============================================
cpu_list: (RW) List of cpus for user-process to sdma
engine assignment.
vl: (RO) Displays the virtual lane (vl) the sdma
engine maps to.
=============== ==============================================
This interface gives the user control on the affinity settings
for the device. As an example, to set an sdma engine irq
affinity and thread affinity of a user processes to use the
sdma engine, which is "near" in terms of NUMA configuration, or
physical cpu location, the user will do::
echo "3" > /proc/irq/<N>/smp_affinity_list
echo "4-7" > /sys/devices/.../sdma3/cpu_list
cat /sys/devices/.../sdma3/vl
0
echo "8" > /proc/irq/<M>/smp_affinity_list
echo "9-12" > /sys/devices/.../sdma4/cpu_list
cat /sys/devices/.../sdma4/vl
1
to make sure that when a process runs on cpus 4,5,6, or 7, and
uses vl=0, then sdma engine 3 is selected by the driver, and
also the interrupt of the sdma engine 3 is steered to cpu 3.
Similarly, when a process runs on cpus 9,10,11, or 12 and sets
vl=1, then engine 4 will be selected and the irq of the sdma
engine 4 is steered to cpu 8. This assumes that in the above N
is the irq number of "sdma3", and M is irq number of "sdma4" in
the /proc/interrupts file.
sysfs interface for QLogic qedr NIC Driver
------------------------------------------
What: /sys/class/infiniband/qedrX/hw_rev
What: /sys/class/infiniband/qedrX/hca_type
Date: Oct, 2016
KernelVersion: v4.10
Contact: linux-rdma@vger.kernel.org
Description:
=============== ==== ========================
hw_rev: (RO) Hardware revision number
hca_type: (RO) Display HCA type
=============== ==== ========================
sysfs interface for VMware Paravirtual RDMA driver
--------------------------------------------------
What: /sys/class/infiniband/vmw_pvrdmaX/hw_rev
What: /sys/class/infiniband/vmw_pvrdmaX/hca_type
What: /sys/class/infiniband/vmw_pvrdmaX/board_id
Date: Oct, 2016
KernelVersion: v4.10
Contact: linux-rdma@vger.kernel.org
Description:
=============== ==== =====================================
hw_rev: (RO) Hardware revision number
hca_type: (RO) Host channel adapter type
board_id: (RO) Display PVRDMA manufacturing board ID
=============== ==== =====================================
sysfs interface for Broadcom NetXtreme-E RoCE driver
----------------------------------------------------
What: /sys/class/infiniband/bnxt_reX/hw_rev
What: /sys/class/infiniband/bnxt_reX/hca_type
Date: Feb, 2017
KernelVersion: v4.11
Contact: linux-rdma@vger.kernel.org
Description:
=============== ==== =========================
hw_rev: (RO) Hardware revision number
hca_type: (RO) Host channel adapter type
=============== ==== =========================
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
모든 InfiniBand device의 공통 attribute
1-36공통 interface의 contact는 linux-rdma@vger.kernel.org다.
| 경로 | 도입·접근 | 의미 |
|---|---|---|
| /sys/class/infiniband/<device>/node_type | 2005년 4월 · kernel 2.6.12 · read-only | Node type이다. CA, RNIC, usNIC, usNIC UDP, switch 또는 router 가운데 하나다. |
| /sys/class/infiniband/<device>/node_guid | 2005년 4월 · kernel 2.6.12 · read-only | Node GUID다. |
| /sys/class/infiniband/<device>/sys_image_guid | 2005년 4월 · kernel 2.6.12 · read-only | System image GUID다. |
| /sys/class/infiniband/<device>/node_desc | 2006년 2월 · kernel 2.6.17 · read/write | Hostname 같은 정보로 node description을 갱신해 IB network management software의 view를 실제 system과 연결한다. |
| /sys/class/infiniband/<device>/fw_ver | 2016년 6월 · kernel 4.10 · read-only | Firmware version을 표시한다. |
Port identity, state와 capability
39-82Port 기본 attribute는 2005년 4월 kernel 2.6.12에 도입됐고 모두 read-only다.
| attribute | 의미 |
|---|---|
| lid | Port LID |
| rate | active width와 active speed의 곱인 port data rate |
| lid_mask_count | Port LID mask count |
| sm_sl | Port subnet의 subnet manager SL |
| sm_lid | Port subnet의 subnet manager LID |
| state | DOWN, INIT, ARMED, ACTIVE 또는 ACTIVE_DEFER인 port state |
| phys_state | Sleep, Polling, LinkUp 등 port physical state |
| cap_mask | Port capability mask다. IsCommunicationManagementSupported는 CM module이 load되면 설정되고, IsSM은 issmN file을 open하면 설정되는 두 writable bit다. |
/sys/class/infiniband/<device>/ports/<port-num>/link_layer는 2010년 10월 kernel 2.6.37에 추가된 read-only attribute로 link layer가 InfiniBand인지 Ethernet인지 표시한다.
Port error counter
85-146아래 read-only counter는 2005년 4월 kernel 2.6.12에 도입됐다.
| counter | 의미 |
|---|---|
| symbol_error | 하나 이상의 physical lane에서 감지한 minor link error의 전체 수 |
| port_rcv_errors | Port가 수신한 error 포함 packet의 전체 수 |
| port_rcv_remote_physical_errors | Port가 수신한 EBP delimiter 표시 packet의 전체 수 |
| port_rcv_switch_relay_errors | Switch relay가 forward할 수 없어 버린 수신 packet의 전체 수 |
| link_error_recovery | Port Training state machine이 link error recovery를 성공적으로 완료한 횟수 |
| port_xmit_constraint_errors | Outbound raw filtering 또는 partition/IP version check 실패로 switch physical port에서 전송하지 않은 packet의 전체 수 |
| port_rcv_contraint_errors | Inbound raw filtering 또는 partition/IP version check 실패로 switch physical port가 버린 수신 packet의 전체 수. Path에는 원문의 contraint 철자가 사용된다. |
| local_link_integrity_errors | Local physical error 수가 LocalPhyErrors threshold를 넘은 횟수 |
| excessive_buffer_overrun_errors | Input buffer overrun counter다. SM이나 user의 port misconfiguration, hardware 문제 또는 매우 나쁜 link signal integrity를 나타낼 수 있다. |
Port data와 기타 counter
148-196| counter | 의미 |
|---|---|
| port_xmit_data | 모든 VL에서 전송한 data octet 수를 4 lane으로 나눈 64-bit 값 |
| port_rcv_data | 모든 VL에서 수신한 data octet 수를 4 lane으로 나눈 64-bit 값 |
| port_xmit_packets | 이 port가 모든 VL에서 전송한 전체 packet 수. Error packet을 포함할 수 있는 64-bit counter다. |
| port_rcv_packets | 수신한 전체 packet 수. Error 포함 packet을 포함할 수 있는 64-bit counter다. |
| unicast_rcv_packets | Error 포함 unicast packet을 포함한 전체 수신 수 |
| unicast_xmit_packets | 모든 VL에서 전송한 전체 unicast packet 수. Error packet을 포함할 수 있다. |
| multicast_rcv_packets | Error 포함 multicast packet을 포함한 전체 수신 수 |
| multicast_xmit_packets | 모든 VL에서 전송한 전체 multicast packet 수. Error packet을 포함할 수 있다. |
| link_downed | Port Training state machine이 link error recovery에 실패하고 link를 down시킨 전체 횟수 |
| port_xmit_discards | Port down 또는 congestion 때문에 port가 버린 outbound packet 전체 수 |
| VL15_dropped | Buffer 부족 같은 port resource 제한 때문에 버린 incoming VL15 packet 수 |
| port_xmit_wait | 전송할 data가 있었지만 credit 부족 또는 arbitration 부재로 tick 전체 동안 아무 data도 보내지 못한 tick 수 |
각 file은 InfiniBand Architecture Specification의 port Performance Management PortCounters attribute에 있는 대응 값을 담는다.
Hardware counter cache와 RoCE GID
199-2322016년 5월 kernel 4.6에 추가된 optional hw_counters subdirectory는 parent device, port 또는 양쪽 모두에 있을 수 있다. Hardware가 제공하는 counter 목록은 일반 counters와 일부 겹칠 수 있으나 다른 counter도 흔히 포함한다.
device 또는 port의 hw_counters/lifespan은 counter에 접근할 때 core가 값을 갱신하는 주기를 millisecond로 설정한다. 접근하지 않으면 갱신하지 않는다. Driver가 다르게 정하지 않으면 기본값은 10ms이고, 0부터 10000 사이 값을 써서 갱신 간격을 바꿀 수 있다.
| 경로 | 도입 | 의미 |
|---|---|---|
| .../gid_attrs/ndevs/<gid-index> | 2015년 11월 29일 · kernel 4.4.0 | 해당 GID에 연결된 net-device 이름을 gid-index 위치에 제공한다. |
| .../gid_attrs/types/<gid-index> | 2015년 11월 29일 · kernel 4.4.0 | GID의 RoCE type을 제공한다. IB와 RoCE v1 기반 GID는 IB/RoCE v1, RoCE v2 기반 GID는 RoCE v2다. |
MAD, issm와 uverbs ABI
235-284각 InfiniBand device port에는 umad device와 issm device가 하나씩 연결된다. 2-port HCA는 각각 두 개를 가지며 switch는 port 0에 대해 각 type 하나를 갖는다.
| 경로 | 도입 | 의미 |
|---|---|---|
| /sys/class/infiniband_mad/umad<N>/ibdev 및 port | 2005년 4월 · kernel 2.6.12 | ibdev는 IB device 이름, port는 port number를 표시하는 read-only attribute다. |
| /sys/class/infiniband_mad/issm<N>/ibdev 및 port | 2005년 4월 · kernel 2.6.12 | ibdev는 IB device 이름, port는 port number를 표시하는 read-only attribute다. |
| /sys/class/infiniband_mad/abi_version | 2005년 4월 · kernel 2.6.12 | umad 또는 issm device의 user space ABI compatibility를 깨는 변경이 생기면 증가하는 read-only 값이다. |
| /sys/class/infiniband_verbs/uverbs<N>/ibdev | 2005년 9월 · kernel 2.6.14 | IB device 이름을 표시한다. |
| /sys/class/infiniband_verbs/uverbs<N>/abi_version | 2005년 9월 · kernel 2.6.14 | IB device별 interface의 ABI version을 표시한다. |
| /sys/class/infiniband_verbs/abi_version | 2005년 9월 · kernel 2.6.14 | uverbs device의 user space ABI compatibility를 깨는 변경이 생기면 증가하는 read-only 값이다. |
Mellanox mthca와 mlx4 HCA identity
287-323| Driver·도입 | attribute | 의미 |
|---|---|---|
| mthca · 2005년 4월 · kernel 2.6.12 | hw_rev | Hardware revision number |
| mthca | hca_type | MT23108, MT25208(MT23108 compatibility mode), MT25208 또는 MT25204 HCA type |
| mthca | board_id | Manufacturing board ID |
| mlx4 · 2007년 9월 · kernel 2.6.24 | hw_rev | Hardware revision number |
| mlx4 | hca_type | Host channel adapter type |
| mlx4 | board_id | Manufacturing board ID |
모두 /sys/class/infiniband/mthcaX/ 또는 /sys/class/infiniband/mlx4_X/ 아래의 read-only attribute다.
mlx4 SR-IOV GID와 P_Key paravirtualization
326-4002012년 8월 kernel 3.6.15에 도입된 iov directory는 port P_Key와 GUID paravirtualization을 관리·검사한다. Master에만 생기며 slave에는 없다. Administrator는 iov/ports에서 device의 실제 GID·P_Key table과 SM에 보이는 network view를 확인한다.
| 경로 | 접근과 범위 |
|---|---|
| iov/ports/<port-num>/gids/<n> | read-only physical port GID, n=0..127 |
| iov/ports/<port-num>/admin_guids/<n> | GUID administrative state 조회·변경, read/write, n=0..127 |
| iov/ports/<port-num>/pkeys/<n> | read-only physical P_Key table, n=0..126 |
| iov/ports/<port-num>/mcgs/ | read-only multicast group table |
| iov/ports/<pci-slot-num>/ports/<m>/gid_idx/0 | read-only GID mapping, m=1..2 |
| iov/ports/<pci-slot-num>/ports/<m>/pkey_idx/<n> | RoCE P_Key를 제외하고 read/write, m=1..2, n=0..126 |
pkeys와 gids subdirectory는 physical table entry마다 file 하나를 둔다. 예를 들어 ports/1/pkeys/10은 port 1 P_Key table의 index 10 값이다. iov/<pci slot number> 아래에서 physical table index를 guest가 보는 paravirtualized index에 mapping한다. Guest 2의 port 1에서 physical P_Key index 10을 virtual index 1에 mapping하면 guest가 index 1을 쓸 때 실제 index 10을 사용한다.
2014년 5월 kernel 3.15.7에 추가된 smi_enabled는 해당 VF/port의 SMI 활성 상태를 읽고, enable_smi_admin은 administrator가 0으로 비활성화 또는 1로 활성화를 요청한다. 기본적으로 어떤 VF도 QP0 operation이 허용되지 않는다. 요청은 다음 VF reset, 예를 들어 VF를 소유한 VM의 driver restart 때 적용된다.
Chelsio cxgb4와 Intel qib
403-540cxgb4 attribute는 2010년 4월 kernel 2.6.35에 도입된 read-only 값이다. hw_rev는 hardware revision, hca_type은 보통 pci_driver::name 같은 bus driver structure 이름과 일치하는 driver short name, board_id는 vendor와 device 정보를 포함한 manufacturing board ID다.
qib attribute는 2010년 5월 kernel 2.6.35에 도입됐다.
| qib attribute | 접근 | 의미 |
|---|---|---|
| version | read-only | 설치된 software와 driver version |
| hw_rev | read-only | Hardware revision number |
| hca_type | read-only | Host channel adapter type |
| board_id | read-only | Manufacturing board ID |
| boardversion | read-only | 현재 chip architecture version |
| nctxts | read-only | 사용 가능한 user port(context) 전체 수 |
| localbus_info | read-only | 사람이 읽을 수 있는 local bus 정보 |
| tempsense | read-only | Temperature sense register의 10진수 값 |
| serial | read-only | HCA serial number |
| nfreectxts | read-only | 사용 가능한 free user context 수 |
| chip_reset | write-only | reset 문자열을 써서 가능한 경우 chip reset. Chip resource를 쓰는 open user context가 없어야 한다. |
qib ports/<N>/sl2vl/[0-15]는 0부터 15까지 16개 read-only file이며 각 Service Level에 programming된 Virtual Lane을 반환한다.
| qib congestion attribute | 의미 |
|---|---|
| CCMgtA/cc_table_bin | read-only binary. Congestion control table 크기 뒤에 table entry가 이어진다. |
| CCMgtA/cc_settings_bin | read-only binary. Port control, control map과 increase, timer, event log trigger threshold, minimum injection rate delay를 담은 16개 congestion entry 배열이다. |
qib linkstate의 loopback(write-only), led_override(write-only), hrtbt_enable(read/write), status(read-only)는 원문에 추가 설명이 없다. status_str은 link state, cable·switch 문제와 hardware error 정보를 표시하며 가능한 상태는 Initted, Present, IB_link_up, IB_configured, Fatal_Hardware_Error다.
qib diag_counters의 rc_resends, seq_naks, rdma_seq, rnr_naks, other_naks, rc_timeouts, look_pkts, pkt_drops, dma_wait, unaligned는 원문에서 [to be documented]로 남아 있다.
mlx5, usNIC와 ocrdma
543-613mlx5의 hw_rev, hca_type, reg_pages, fw_pages는 2013년 7월 kernel 3.11에 도입됐으나 원문에는 [to be documented]만 적혀 있다.
Cisco usNIC attribute는 2013년 9월 kernel 3.14에 도입됐고 담당자는 Christian Benvenuti, Dave Goodell과 linux-rdma mailing list다.
| usNIC attribute | 의미 |
|---|---|
| board_id | Manufacturing board ID |
| config | 이 PF의 configuration |
| qp_per_vf | Virtual function당 queue pair 수 |
| max_vf | 최대 virtual function 수 |
| cq_per_vf | Virtual function당 completion queue 수 |
| iface | 이 usNIC entry와 연결된 ifconfig에서 보이는 network interface |
usnic_X/qpn/summary와 qpn/context는 원문에서 [to be documented]로 남아 있다.
Emulex ocrdma의 hw_rev는 2014년 2월 kernel 3.14에 도입된 read-only hardware revision number다. hca_type은 2014년 6월 kernel 3.16에 도입됐으며 원문 설명은 read-only FW version 표시라고 되어 있다.
Intel Omni-Path HFI1
616-691HFI1 기본 attribute는 2016년 5월 kernel 4.6에 도입됐다.
| attribute | 접근 | 의미 |
|---|---|---|
| hw_rev | read-only | Hardware revision number |
| board_id | read-only | Manufacturing board ID |
| nctxts | read-only | 사용 가능한 전체 context 수 |
| serial | read-only | Board serial number |
| chip_reset | write-only | reset을 써서 가능한 경우 chip을 reset한다. Chip resource를 쓰는 open user context가 없어야 한다. |
| boardversion | read-only | 사람이 읽을 수 있는 board 정보 |
| nfreectxts | read-only | 사용 가능한 free user context 수 |
| tempsense | read-only | Thermal sense 정보 |
| Per-port congestion attribute | 의미 |
|---|---|
| CCMgtA/cc_table_bin | PSM2 congestion control이 쓰는 CCA table 크기와 entry를 담은 read-only binary attribute |
| CCMgtA/cc_settings_bin | Port control, control map, increase·timer·event log threshold·minimum injection rate delay의 16개 entry를 담은 read-only binary attribute |
| CCMgtA/cc_prescan | 더 빠른 BECN response를 위한 prescan을 on으로 켜고 off로 끄는 read/write attribute |
| Mapping directory | 의미 |
|---|---|
| sc2vl/[0-31] | SL에서 VL로 변환하는 32개 read-only file |
| sl2sc/[0-31] | SL에서 SC로 변환하는 32개 read-only file |
| vl2mtu/[0-15] | VL의 MTU를 정하는 16개 read-only file |
HFI1 SDMA engine affinity
694-7322016년 9월 kernel 4.8에 도입됐다. sdma_<N>/은 0부터 15까지 SDMA engine마다 directory 하나를 둔다. cpu_list는 user process를 SDMA engine에 할당할 CPU 목록을 설정하는 read/write attribute고, vl은 engine이 mapping되는 virtual lane을 표시하는 read-only attribute다.
NUMA 구성이나 physical CPU 위치상 가까운 SDMA engine을 사용하도록 user process thread affinity와 engine IRQ affinity를 함께 제어할 수 있다.
echo "3" > /proc/irq/<N>/smp_affinity_list
echo "4-7" > /sys/devices/.../sdma3/cpu_list
cat /sys/devices/.../sdma3/vl
0
echo "8" > /proc/irq/<M>/smp_affinity_list
echo "9-12" > /sys/devices/.../sdma4/cpu_list
cat /sys/devices/.../sdma4/vl
1
CPU 4–7에서 실행하며 VL 0을 쓰는 process에는 engine 3을 선택하고 engine 3 interrupt는 CPU 3으로 보낸다. CPU 9–12에서 VL 1을 쓰는 process에는 engine 4를 선택하고 interrupt는 CPU 8로 보낸다. N과 M은 /proc/interrupts에 나온 sdma3과 sdma4의 IRQ number라고 가정한다.
qedr, VMware PVRDMA와 Broadcom bnxt_re
734-784| Driver·도입 | attribute | 의미 |
|---|---|---|
| QLogic qedr · 2016년 10월 · kernel 4.10 | hw_rev | read-only hardware revision number |
| QLogic qedr | hca_type | read-only HCA type |
| VMware PVRDMA · 2016년 10월 · kernel 4.10 | hw_rev | read-only hardware revision number |
| VMware PVRDMA | hca_type | read-only host channel adapter type |
| VMware PVRDMA | board_id | read-only PVRDMA manufacturing board ID |
| Broadcom bnxt_re · 2017년 2월 · kernel 4.11 | hw_rev | read-only hardware revision number |
| Broadcom bnxt_re | hca_type | read-only host channel adapter type |
이 driver별 attribute의 contact는 linux-rdma@vger.kernel.org다.
RDMA core device와 port
sysfs-class-infiniband:1-82Node GUID·description·firmware와 port LID·rate·state·capability·link layer를 다룹니다.