bnx2fc로 Broadcom FCoE 운영

Stateful hardware offload와 network 전제

`bnx2fc`는 Broadcom FCoE를 full-stateful hardware offload하면서 Linux FC/FCoE와 SCSI interface에 통합한다. 기능을 활성화하면 SAN에서 발견된 device가 upper storage layer에 자동 등록·해제된다.

Offload 자체는 hardware에서 수행하지만 associated network interface, 예를 들어 `eth0`의 link state에 의존하므로 interface를 up으로 유지하고 boot 때 자동 활성화하는 것이 권장된다.

Solution은 발견된 FCoE VLAN마다 `eth0.1001-fcoe` 같은 VLAN interface를 자동 생성한다. 이를 삭제하거나 비활성화하면 FCoE가 중단된다.

.. SPDX-License-Identifier: GPL-2.0

===========================
Operating FCoE using bnx2fc
===========================
Broadcom FCoE offload through bnx2fc is full stateful hardware offload that
cooperates with all interfaces provided by the Linux ecosystem for FC/FCoE and
SCSI controllers.  As such, FCoE functionality, once enabled is largely
transparent. Devices discovered on the SAN will be registered and unregistered
automatically with the upper storage layers.

Despite the fact that the Broadcom's FCoE offload is fully offloaded, it does
depend on the state of the network interfaces to operate. As such, the network
interface (e.g. eth0) associated with the FCoE offload initiator must be 'up'.
It is recommended that the network interfaces be configured to be brought up
automatically at boot time.

Furthermore, the Broadcom FCoE offload solution creates VLAN interfaces to
support the VLANs that have been discovered for FCoE operation (e.g.

fcoe-utils 설정과 service 시작

먼저 `fcoe-utils`를 설치한다. `/etc/fcoe`에서 `cfg-ethx`를 필요한 interface 이름으로 복사하고 각 `cfg-eth*` 파일에서 `DCB_REQUIRED=no`, `AUTO_VLAN=yes`로 설정하며 나머지는 기본값으로 둔다.

`/etc/fcoe/config`의 `SUPPORTED_DRIVERS`에 `bnx2fc`가 있어야 한다. `service fcoe start`를 실행하면 Broadcom device가 있을 때 driver가 interface를 claim하고 VLAN discovery와 target login을 시작한다.

eth0.1001-fcoe).  Do not delete or disable these interfaces or FCoE operation
will be disrupted.

Driver Usage Model:
===================

1. Ensure that fcoe-utils package is installed.

2. Configure the interfaces on which bnx2fc driver has to operate on.
Here are the steps to configure:

        a. cd /etc/fcoe
        b. copy cfg-ethx to cfg-eth5 if FCoE has to be enabled on eth5.
        c. Repeat this for all the interfaces where FCoE has to be enabled.
        d. Edit all the cfg-eth files to set "no" for DCB_REQUIRED** field, and
           "yes" for AUTO_VLAN.
        e. Other configuration parameters should be left as default

3. Ensure that "bnx2fc" is in SUPPORTED_DRIVERS list in /etc/fcoe/config.

4. Start fcoe service. (service fcoe start). If Broadcom devices are present in
the system, bnx2fc driver would automatically claim the interfaces, starts vlan
discovery and log into the targets.

5. "Symbolic Name" in 'fcoeadm -i' output would display if bnx2fc has claimed
the interface.

Eg::

fcoeadm과 VLAN discovery 확인

`fcoeadm -i` 출력의 `Symbolic Name`에 `bnx2fc v1.0.5 over eth5.4`처럼 표시되면 driver가 interface를 claim한 것이다. 예시는 BCM57712 10GbE, host11, node·port WWN, fabric name, 10Gbit speed, max frame 2048, FC-ID와 Online state를 보여 준다.

`ifconfig`에서 `<INTERFACE>.<VLAN>-fcoe` interface가 자동 생성됐는지 확인한다. Interface 생성·삭제와 LUN·target 표시는 `fcoeadm` manual을 참고한다.

 [root@bh2 ~]# fcoeadm -i
    Description:      NetXtreme II BCM57712 10 Gigabit Ethernet
    Revision:         01
    Manufacturer:     Broadcom Corporation
    Serial Number:    0010186FD558
    Driver:           bnx2x 1.70.00-0
    Number of Ports:  2

        Symbolic Name:     bnx2fc v1.0.5 over eth5.4
        OS Device Name:    host11
        Node Name:         0x10000010186FD559
        Port Name:         0x20000010186FD559
        FabricName:        0x2001000DECB3B681
        Speed:             10 Gbit
        Supported Speed:   10 Gbit
        MaxFrameSize:      2048
        FC-ID (Port ID):   0x0F0377
        State:             Online

6. Verify the vlan discovery is performed by running ifconfig and notice
   <INTERFACE>.<VLAN>-fcoe interfaces are automatically created.

Refer to fcoeadm manpage for more information on fcoeadm operations to
create/destroy interfaces or to display lun/target information.

On-chip DCBX/LLDP client 주의

Broadcom FCoE-capable device는 chip 안에 DCBX/LLDP client를 구현한다. Interface당 LLDP client는 하나만 허용되므로 `lldpad` 같은 host software client를 해당 interface에서 비활성화해야 한다.

명령은 `lldptool set-lldp -i <interface_name> adminStatus=disabled`다. 이 조치는 host LLDP와 on-chip client가 동시에 동작하는 충돌을 피한다.

NOTE
====
** Broadcom FCoE capable devices implement a DCBX/LLDP client on-chip. Only one
LLDP client is allowed per interface. For proper operation all host software
based DCBX/LLDP clients (e.g. lldpad) must be disabled. To disable lldpad on a
given interface, run the following command::

        lldptool set-lldp -i <interface_name> adminStatus=disabled