요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
=====================================
Network Devices, the Kernel, and You!
=====================================
Introduction
============
The following is a random collection of documentation regarding
network devices. It is intended for driver developers.
struct net_device lifetime rules
================================
Network device structures need to persist even after module is unloaded and
must be allocated with alloc_netdev_mqs() and friends.
If device has registered successfully, it will be freed on last use
by free_netdev(). This is required to handle the pathological case cleanly
(example: ``rmmod mydriver </sys/class/net/myeth/mtu``)
alloc_netdev_mqs() / alloc_netdev() reserve extra space for driver
private data which gets freed when the network device is freed. If
separately allocated data is attached to the network device
(netdev_priv()) then it is up to the module exit handler to free that.
There are two groups of APIs for registering struct net_device.
First group can be used in normal contexts where ``rtnl_lock`` is not already
held: register_netdev(), unregister_netdev().
Second group can be used when ``rtnl_lock`` is already held:
register_netdevice(), unregister_netdevice(), free_netdevice().
Simple drivers
--------------
Most drivers (especially device drivers) handle lifetime of struct net_device
in context where ``rtnl_lock`` is not held (e.g. driver probe and remove paths).
In that case the struct net_device registration is done using
the register_netdev(), and unregister_netdev() functions:
.. code-block:: c
int probe()
{
struct my_device_priv *priv;
int err;
dev = alloc_netdev_mqs(...);
if (!dev)
return -ENOMEM;
priv = netdev_priv(dev);
/* ... do all device setup before calling register_netdev() ...
*/
err = register_netdev(dev);
if (err)
goto err_undo;
/* net_device is visible to the user! */
err_undo:
/* ... undo the device setup ... */
free_netdev(dev);
return err;
}
void remove()
{
unregister_netdev(dev);
free_netdev(dev);
}
Note that after calling register_netdev() the device is visible in the system.
Users can open it and start sending / receiving traffic immediately,
or run any other callback, so all initialization must be done prior to
registration.
unregister_netdev() closes the device and waits for all users to be done
with it. The memory of struct net_device itself may still be referenced
by sysfs but all operations on that device will fail.
free_netdev() can be called after unregister_netdev() returns on when
register_netdev() failed.
Device management under RTNL
----------------------------
Registering struct net_device while in context which already holds
the ``rtnl_lock`` requires extra care. In those scenarios most drivers
will want to make use of struct net_device's ``needs_free_netdev``
and ``priv_destructor`` members for freeing of state.
Example flow of netdev handling under ``rtnl_lock``:
.. code-block:: c
static void my_setup(struct net_device *dev)
{
dev->needs_free_netdev = true;
}
static void my_destructor(struct net_device *dev)
{
some_obj_destroy(priv->obj);
some_uninit(priv);
}
int create_link()
{
struct my_device_priv *priv;
int err;
ASSERT_RTNL();
dev = alloc_netdev(sizeof(*priv), "net%d", NET_NAME_UNKNOWN, my_setup);
if (!dev)
return -ENOMEM;
priv = netdev_priv(dev);
/* Implicit constructor */
err = some_init(priv);
if (err)
goto err_free_dev;
priv->obj = some_obj_create();
if (!priv->obj) {
err = -ENOMEM;
goto err_some_uninit;
}
/* End of constructor, set the destructor: */
dev->priv_destructor = my_destructor;
err = register_netdevice(dev);
if (err)
/* register_netdevice() calls destructor on failure */
goto err_free_dev;
/* If anything fails now unregister_netdevice() (or unregister_netdev())
* will take care of calling my_destructor and free_netdev().
*/
return 0;
err_some_uninit:
some_uninit(priv);
err_free_dev:
free_netdev(dev);
return err;
}
If struct net_device.priv_destructor is set it will be called by the core
some time after unregister_netdevice(), it will also be called if
register_netdevice() fails. The callback may be invoked with or without
``rtnl_lock`` held.
There is no explicit constructor callback, driver "constructs" the private
netdev state after allocating it and before registration.
Setting struct net_device.needs_free_netdev makes core call free_netdevice()
automatically after unregister_netdevice() when all references to the device
are gone. It only takes effect after a successful call to register_netdevice()
so if register_netdevice() fails driver is responsible for calling
free_netdev().
free_netdev() is safe to call on error paths right after unregister_netdevice()
or when register_netdevice() fails. Parts of netdev (de)registration process
happen after ``rtnl_lock`` is released, therefore in those cases free_netdev()
will defer some of the processing until ``rtnl_lock`` is released.
Devices spawned from struct rtnl_link_ops should never free the
struct net_device directly.
.ndo_init and .ndo_uninit
~~~~~~~~~~~~~~~~~~~~~~~~~
``.ndo_init`` and ``.ndo_uninit`` callbacks are called during net_device
registration and de-registration, under ``rtnl_lock``. Drivers can use
those e.g. when parts of their init process need to run under ``rtnl_lock``.
``.ndo_init`` runs before device is visible in the system, ``.ndo_uninit``
runs during de-registering after device is closed but other subsystems
may still have outstanding references to the netdevice.
MTU
===
Each network device has a Maximum Transfer Unit. The MTU does not
include any link layer protocol overhead. Upper layer protocols must
not pass a socket buffer (skb) to a device to transmit with more data
than the mtu. The MTU does not include link layer header overhead, so
for example on Ethernet if the standard MTU is 1500 bytes used, the
actual skb will contain up to 1514 bytes because of the Ethernet
header. Devices should allow for the 4 byte VLAN header as well.
Segmentation Offload (GSO, TSO) is an exception to this rule. The
upper layer protocol may pass a large socket buffer to the device
transmit routine, and the device will break that up into separate
packets based on the current MTU.
MTU is symmetrical and applies both to receive and transmit. A device
must be able to receive at least the maximum size packet allowed by
the MTU. A network device may use the MTU as mechanism to size receive
buffers, but the device should allow packets with VLAN header. With
standard Ethernet mtu of 1500 bytes, the device should allow up to
1518 byte packets (1500 + 14 header + 4 tag). The device may either:
drop, truncate, or pass up oversize packets, but dropping oversize
packets is preferred.
struct net_device synchronization rules
=======================================
ndo_open:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
ndo_stop:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
Note: netif_running() is guaranteed false
ndo_do_ioctl:
Synchronization: rtnl_lock() semaphore.
This is only called by network subsystems internally,
not by user space calling ioctl as it was in before
linux-5.14.
ndo_siocbond:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
Used by the bonding driver for the SIOCBOND family of
ioctl commands.
ndo_siocwandev:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
Used by the drivers/net/wan framework to handle
the SIOCWANDEV ioctl with the if_settings structure.
ndo_siocdevprivate:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
This is used to implement SIOCDEVPRIVATE ioctl helpers.
These should not be added to new drivers, so don't use.
ndo_eth_ioctl:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
ndo_get_stats:
Synchronization: RCU (can be called concurrently with the stats
update path).
Context: atomic (can't sleep under RCU)
ndo_start_xmit:
Synchronization: __netif_tx_lock spinlock.
When the driver sets dev->lltx this will be
called without holding netif_tx_lock. In this case the driver
has to lock by itself when needed.
The locking there should also properly protect against
set_rx_mode. WARNING: use of dev->lltx is deprecated.
Don't use it for new drivers.
Context: Process with BHs disabled or BH (timer),
will be called with interrupts disabled by netconsole.
Return codes:
* NETDEV_TX_OK everything ok.
* NETDEV_TX_BUSY Cannot transmit packet, try later
Usually a bug, means queue start/stop flow control is broken in
the driver. Note: the driver must NOT put the skb in its DMA ring.
ndo_tx_timeout:
Synchronization: netif_tx_lock spinlock; all TX queues frozen.
Context: BHs disabled
Notes: netif_queue_stopped() is guaranteed true
ndo_set_rx_mode:
Synchronization: netif_addr_lock spinlock.
Context: BHs disabled
ndo_setup_tc:
``TC_SETUP_BLOCK`` and ``TC_SETUP_FT`` are running under NFT locks
(i.e. no ``rtnl_lock`` and no device instance lock). The rest of
``tc_setup_type`` types run under netdev instance lock if the driver
implements queue management or shaper API.
Most ndo callbacks not specified in the list above are running
under ``rtnl_lock``. In addition, netdev instance lock is taken as well if
the driver implements queue management or shaper API.
struct napi_struct synchronization rules
========================================
napi->poll:
Synchronization:
NAPI_STATE_SCHED bit in napi->state. Device
driver's ndo_stop method will invoke napi_disable() on
all NAPI instances which will do a sleeping poll on the
NAPI_STATE_SCHED napi->state bit, waiting for all pending
NAPI activity to cease.
Context:
softirq
will be called with interrupts disabled by netconsole.
netdev instance lock
====================
Historically, all networking control operations were protected by a single
global lock known as ``rtnl_lock``. There is an ongoing effort to replace this
global lock with separate locks for each network namespace. Additionally,
properties of individual netdev are increasingly protected by per-netdev locks.
For device drivers that implement shaping or queue management APIs, all control
operations will be performed under the netdev instance lock.
Drivers can also explicitly request instance lock to be held during ops
by setting ``request_ops_lock`` to true. Code comments and docs refer
to drivers which have ops called under the instance lock as "ops locked".
See also the documentation of the ``lock`` member of struct net_device.
In the future, there will be an option for individual
drivers to opt out of using ``rtnl_lock`` and instead perform their control
operations directly under the netdev instance lock.
Devices drivers are encouraged to rely on the instance lock where possible.
For the (mostly software) drivers that need to interact with the core stack,
there are two sets of interfaces: ``dev_xxx``/``netdev_xxx`` and ``netif_xxx``
(e.g., ``dev_set_mtu`` and ``netif_set_mtu``). The ``dev_xxx``/``netdev_xxx``
functions handle acquiring the instance lock themselves, while the
``netif_xxx`` functions assume that the driver has already acquired
the instance lock.
struct net_device_ops
---------------------
``ndos`` are called without holding the instance lock for most drivers.
"Ops locked" drivers will have most of the ``ndos`` invoked under
the instance lock.
struct ethtool_ops
------------------
Similarly to ``ndos`` the instance lock is only held for select drivers.
For "ops locked" drivers all ethtool ops without exceptions should
be called under the instance lock.
struct netdev_stat_ops
----------------------
"qstat" ops are invoked under the instance lock for "ops locked" drivers,
and under rtnl_lock for all other drivers.
struct net_shaper_ops
---------------------
All net shaper callbacks are invoked while holding the netdev instance
lock. ``rtnl_lock`` may or may not be held.
Note that supporting net shapers automatically enables "ops locking".
struct netdev_queue_mgmt_ops
----------------------------
All queue management callbacks are invoked while holding the netdev instance
lock. ``rtnl_lock`` may or may not be held.
Note that supporting struct netdev_queue_mgmt_ops automatically enables
"ops locking".
Notifiers and netdev instance lock
----------------------------------
For device drivers that implement shaping or queue management APIs,
some of the notifiers (``enum netdev_cmd``) are running under the netdev
instance lock.
The following netdev notifiers are always run under the instance lock:
* ``NETDEV_XDP_FEAT_CHANGE``
For devices with locked ops, currently only the following notifiers are
running under the lock:
* ``NETDEV_CHANGE``
* ``NETDEV_REGISTER``
* ``NETDEV_UP``
The following notifiers are running without the lock:
* ``NETDEV_UNREGISTER``
There are no clear expectations for the remaining notifiers. Notifiers not on
the list may run with or without the instance lock, potentially even invoking
the same notifier type with and without the lock from different code paths.
The goal is to eventually ensure that all (or most, with a few documented
exceptions) notifiers run under the instance lock. Please extend this
documentation whenever you make explicit assumption about lock being held
from a notifier.
NETDEV_INTERNAL symbol namespace
================================
Symbols exported as NETDEV_INTERNAL can only be used in networking
core and drivers which exclusively flow via the main networking list and trees.
Note that the inverse is not true, most symbols outside of NETDEV_INTERNAL
are not expected to be used by random code outside netdev either.
Symbols may lack the designation because they predate the namespaces,
or simply due to an oversight.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서의 범위
1-12이 문서는 네트워크 장치와 관련된 여러 규칙을 한곳에 모은 드라이버 개발자용 참고 자료입니다. `struct net_device`의 수명, 등록, MTU, NDO와 NAPI의 동기화, 장치별 instance lock 및 내부 심볼 영역을 다룹니다.
.. SPDX-License-Identifier: GPL-2.0
=====================================
Network Devices, the Kernel, and You!
=====================================
Introduction
============
The following is a random collection of documentation regarding
network devices. It is intended for driver developers.
struct net_device 수명 규칙
13-31네트워크 장치 구조체는 모듈이 언로드된 뒤에도 참조될 수 있으므로 `alloc_netdev_mqs()` 계열로 할당해야 합니다. 등록에 성공한 장치는 마지막 사용이 끝날 때 `free_netdev()`로 해제됩니다. 이는 sysfs 파일을 입력으로 둔 채 모듈을 제거하는 병적인 상황도 안전하게 처리하기 위한 규칙입니다.
`alloc_netdev_mqs()`와 `alloc_netdev()`가 예약한 드라이버 private data는 장치와 함께 해제됩니다. `netdev_priv()`와 별개로 할당한 데이터를 장치에 연결했다면 모듈 종료 처리기가 직접 해제해야 합니다.
등록 API는 두 그룹입니다. `rtnl_lock`을 아직 잡지 않은 일반 문맥에서는 `register_netdev()`와 `unregister_netdev()`를 사용합니다. 이미 잠금을 보유한 문맥에서는 `register_netdevice()`, `unregister_netdevice()`, `free_netdevice()`를 사용합니다.
호출 시 RTNL 보유 여부가 API 계열을 결정합니다.
struct net_device lifetime rules
================================
Network device structures need to persist even after module is unloaded and
must be allocated with alloc_netdev_mqs() and friends.
If device has registered successfully, it will be freed on last use
by free_netdev(). This is required to handle the pathological case cleanly
(example: ``rmmod mydriver </sys/class/net/myeth/mtu``)
alloc_netdev_mqs() / alloc_netdev() reserve extra space for driver
private data which gets freed when the network device is freed. If
separately allocated data is attached to the network device
(netdev_priv()) then it is up to the module exit handler to free that.
There are two groups of APIs for registering struct net_device.
First group can be used in normal contexts where ``rtnl_lock`` is not already
held: register_netdev(), unregister_netdev().
Second group can be used when ``rtnl_lock`` is already held:
register_netdevice(), unregister_netdevice(), free_netdevice().
일반 드라이버의 등록과 제거
32-85대부분의 장치 드라이버는 probe와 remove처럼 `rtnl_lock`을 잡지 않은 문맥에서 수명을 관리합니다. probe에서는 `alloc_netdev_mqs()`로 할당하고 `netdev_priv()`를 얻은 뒤 모든 장치 설정을 끝내고 `register_netdev()`를 호출합니다. 등록이 실패하면 설정을 되돌리고 `free_netdev()`를 호출합니다.
`register_netdev()`가 성공하는 즉시 장치는 사용자에게 보입니다. 사용자가 바로 열어 송수신하거나 임의의 콜백을 실행할 수 있으므로 등록 전에 초기화를 모두 끝내야 합니다.
remove에서는 `unregister_netdev()`로 장치를 닫고 모든 사용자가 끝날 때까지 기다린 다음 `free_netdev()`를 호출합니다. sysfs가 `struct net_device` 메모리를 잠시 참조할 수는 있지만 등록 해제 뒤 장치 작업은 모두 실패합니다. `free_netdev()`는 등록 해제가 반환한 뒤 또는 등록 실패 경로에서 호출할 수 있습니다.
probe 성공과 오류, remove 경로를 나눕니다.
Simple drivers
--------------
Most drivers (especially device drivers) handle lifetime of struct net_device
in context where ``rtnl_lock`` is not held (e.g. driver probe and remove paths).
In that case the struct net_device registration is done using
the register_netdev(), and unregister_netdev() functions:
.. code-block:: c
int probe()
{
struct my_device_priv *priv;
int err;
dev = alloc_netdev_mqs(...);
if (!dev)
return -ENOMEM;
priv = netdev_priv(dev);
/* ... do all device setup before calling register_netdev() ...
*/
err = register_netdev(dev);
if (err)
goto err_undo;
/* net_device is visible to the user! */
err_undo:
/* ... undo the device setup ... */
free_netdev(dev);
return err;
}
void remove()
{
unregister_netdev(dev);
free_netdev(dev);
}
Note that after calling register_netdev() the device is visible in the system.
Users can open it and start sending / receiving traffic immediately,
or run any other callback, so all initialization must be done prior to
registration.
unregister_netdev() closes the device and waits for all users to be done
with it. The memory of struct net_device itself may still be referenced
by sysfs but all operations on that device will fail.
free_netdev() can be called after unregister_netdev() returns on when
register_netdev() failed.
RTNL 아래의 장치 관리
86-173이미 `rtnl_lock`을 보유한 문맥에서 등록한다면 일반적으로 `needs_free_netdev`와 `priv_destructor`로 상태 해제를 위임합니다. setup 콜백에서 `needs_free_netdev = true`로 설정하고, private state 해제 함수들을 destructor에 모읍니다.
문서의 `create_link()` 예제는 `ASSERT_RTNL()` 뒤 장치를 할당하고 private state를 단계별로 구성합니다. 암시적 생성 과정이 모두 성공한 뒤에만 `dev->priv_destructor`를 설정합니다. `register_netdevice()`가 실패하면 코어가 destructor를 호출하며, 등록 뒤의 오류는 `unregister_netdevice()` 또는 `unregister_netdev()`가 destructor와 `free_netdev()` 호출을 처리합니다.
`priv_destructor`는 `unregister_netdevice()` 이후 어느 시점 또는 `register_netdevice()` 실패 시 호출되며, 호출 시 `rtnl_lock` 보유 여부는 보장되지 않습니다. 명시적 constructor 콜백은 없으므로 드라이버가 할당 뒤 등록 전에 private 상태를 직접 구성합니다.
`needs_free_netdev`는 등록 성공 뒤 참조가 모두 사라졌을 때 코어가 `free_netdevice()`를 자동 호출하게 합니다. 등록이 실패하면 효력이 없으므로 드라이버가 `free_netdev()`를 호출해야 합니다. 등록·해제의 일부 처리는 RTNL 해제 뒤 진행되므로 오류 경로에서 즉시 `free_netdev()`를 호출해도 필요한 처리는 잠금 해제 때까지 지연됩니다. `struct rtnl_link_ops`가 만든 장치는 구조체를 직접 해제하면 안 됩니다.
Device management under RTNL
----------------------------
Registering struct net_device while in context which already holds
the ``rtnl_lock`` requires extra care. In those scenarios most drivers
will want to make use of struct net_device's ``needs_free_netdev``
and ``priv_destructor`` members for freeing of state.
Example flow of netdev handling under ``rtnl_lock``:
.. code-block:: c
static void my_setup(struct net_device *dev)
{
dev->needs_free_netdev = true;
}
static void my_destructor(struct net_device *dev)
{
some_obj_destroy(priv->obj);
some_uninit(priv);
}
int create_link()
{
struct my_device_priv *priv;
int err;
ASSERT_RTNL();
dev = alloc_netdev(sizeof(*priv), "net%d", NET_NAME_UNKNOWN, my_setup);
if (!dev)
return -ENOMEM;
priv = netdev_priv(dev);
/* Implicit constructor */
err = some_init(priv);
if (err)
goto err_free_dev;
priv->obj = some_obj_create();
if (!priv->obj) {
err = -ENOMEM;
goto err_some_uninit;
}
/* End of constructor, set the destructor: */
dev->priv_destructor = my_destructor;
err = register_netdevice(dev);
if (err)
/* register_netdevice() calls destructor on failure */
goto err_free_dev;
/* If anything fails now unregister_netdevice() (or unregister_netdev())
* will take care of calling my_destructor and free_netdev().
*/
return 0;
err_some_uninit:
some_uninit(priv);
err_free_dev:
free_netdev(dev);
return err;
}
If struct net_device.priv_destructor is set it will be called by the core
some time after unregister_netdevice(), it will also be called if
register_netdevice() fails. The callback may be invoked with or without
``rtnl_lock`` held.
There is no explicit constructor callback, driver "constructs" the private
netdev state after allocating it and before registration.
Setting struct net_device.needs_free_netdev makes core call free_netdevice()
automatically after unregister_netdevice() when all references to the device
are gone. It only takes effect after a successful call to register_netdevice()
so if register_netdevice() fails driver is responsible for calling
free_netdev().
free_netdev() is safe to call on error paths right after unregister_netdevice()
or when register_netdevice() fails. Parts of netdev (de)registration process
happen after ``rtnl_lock`` is released, therefore in those cases free_netdev()
will defer some of the processing until ``rtnl_lock`` is released.
Devices spawned from struct rtnl_link_ops should never free the
struct net_device directly.
ndo_init과 ndo_uninit
174-184`.ndo_init`과 `.ndo_uninit`은 각각 `net_device` 등록과 등록 해제 중 `rtnl_lock`을 잡은 상태로 호출됩니다. RTNL 아래에서 수행해야 하는 초기화 부분에 사용할 수 있습니다. `.ndo_init`은 장치가 시스템에 보이기 전에 실행되고, `.ndo_uninit`은 장치가 닫힌 뒤 실행되지만 이때 다른 하위 시스템이 아직 netdevice 참조를 갖고 있을 수 있습니다.
.ndo_init and .ndo_uninit
~~~~~~~~~~~~~~~~~~~~~~~~~
``.ndo_init`` and ``.ndo_uninit`` callbacks are called during net_device
registration and de-registration, under ``rtnl_lock``. Drivers can use
those e.g. when parts of their init process need to run under ``rtnl_lock``.
``.ndo_init`` runs before device is visible in the system, ``.ndo_uninit``
runs during de-registering after device is closed but other subsystems
may still have outstanding references to the netdevice.
MTU와 실제 프레임 크기
185-209각 장치는 MTU를 가지며 링크 계층 오버헤드는 MTU에 포함되지 않습니다. 상위 계층은 일반적으로 MTU보다 큰 데이터를 가진 skb를 장치 송신 함수에 넘기면 안 됩니다. Ethernet MTU가 1500이면 14바이트 Ethernet 헤더 때문에 실제 skb는 최대 1514바이트이며 장치는 4바이트 VLAN 헤더도 허용해야 합니다.
GSO와 TSO는 예외입니다. 상위 계층이 큰 skb를 넘기면 장치 송신 경로가 현재 MTU에 맞는 별도 패킷으로 분할합니다.
MTU는 송수신에 대칭으로 적용됩니다. 장치는 최소한 MTU가 허용한 최대 패킷을 받아야 하며 수신 버퍼 크기 계산에 MTU를 쓸 수 있지만 VLAN 헤더 여유를 둬야 합니다. 표준 Ethernet MTU 1500에서는 1500 + 14 + 4인 1518바이트까지 허용해야 합니다. 초과 패킷을 버리거나 자르거나 상위로 전달할 수 있지만 버리는 방식을 권장합니다.
표준 MTU에서 링크 헤더와 VLAN 태그를 더한 실제 수신 여유입니다.
MTU
===
Each network device has a Maximum Transfer Unit. The MTU does not
include any link layer protocol overhead. Upper layer protocols must
not pass a socket buffer (skb) to a device to transmit with more data
than the mtu. The MTU does not include link layer header overhead, so
for example on Ethernet if the standard MTU is 1500 bytes used, the
actual skb will contain up to 1514 bytes because of the Ethernet
header. Devices should allow for the 4 byte VLAN header as well.
Segmentation Offload (GSO, TSO) is an exception to this rule. The
upper layer protocol may pass a large socket buffer to the device
transmit routine, and the device will break that up into separate
packets based on the current MTU.
MTU is symmetrical and applies both to receive and transmit. A device
must be able to receive at least the maximum size packet allowed by
the MTU. A network device may use the MTU as mechanism to size receive
buffers, but the device should allow packets with VLAN header. With
standard Ethernet mtu of 1500 bytes, the device should allow up to
1518 byte packets (1500 + 14 header + 4 tag). The device may either:
drop, truncate, or pass up oversize packets, but dropping oversize
packets is preferred.
net_device_ops 동기화 규칙
210-302`ndo_open`과 `ndo_stop`은 process 문맥에서 `rtnl_lock` 아래 호출되며, queue management 또는 shaper API를 구현한 드라이버는 netdev instance lock도 잡힙니다. `ndo_stop`에서는 `netif_running()`이 false임이 보장됩니다. `ndo_do_ioctl`도 RTNL 아래 호출되지만 Linux 5.14 이후 사용자 ioctl이 직접 들어오는 경로가 아니라 네트워크 하위 시스템 내부 호출입니다.
`ndo_siocbond`, `ndo_siocwandev`, `ndo_siocdevprivate`, `ndo_eth_ioctl`도 RTNL과 해당되는 instance lock 아래 process 문맥에서 실행됩니다. 각각 bonding의 `SIOCBOND`, WAN 프레임워크의 `SIOCWANDEV`, private ioctl helper, Ethernet ioctl을 담당합니다. 새 드라이버에는 `SIOCDEVPRIVATE` helper를 추가하지 않아야 합니다.
`ndo_get_stats`는 갱신 경로와 동시에 호출될 수 있도록 RCU로 동기화되며 atomic 문맥이라 sleep할 수 없습니다. `ndo_start_xmit`은 `__netif_tx_lock` spinlock 아래 실행됩니다. `dev->lltx`를 설정하면 잠금 없이 호출되므로 드라이버가 `set_rx_mode`와의 충돌까지 직접 보호해야 하지만 `lltx`는 폐기 예정이므로 새 드라이버에서 쓰면 안 됩니다.
`ndo_start_xmit`은 BH가 비활성화된 process 또는 BH(timer) 문맥에서 호출되고 Netconsole에서는 인터럽트까지 비활성화됩니다. `NETDEV_TX_OK`는 성공입니다. `NETDEV_TX_BUSY`는 나중에 재시도하라는 뜻이지만 보통 드라이버의 queue start/stop 흐름 제어 결함이며, 이때 드라이버는 skb를 DMA ring에 넣으면 안 됩니다.
`ndo_tx_timeout`은 모든 TX queue가 고정되고 `netif_tx_lock`을 잡은 BH 비활성 문맥에서 실행되며 `netif_queue_stopped()`가 true입니다. `ndo_set_rx_mode`는 `netif_addr_lock` spinlock과 BH 비활성 문맥을 사용합니다. `ndo_setup_tc` 가운데 `TC_SETUP_BLOCK`과 `TC_SETUP_FT`는 NFT 잠금 아래 실행되어 RTNL과 instance lock이 없고, 나머지는 queue/shaper 구현 드라이버에서 instance lock 아래 실행됩니다. 목록에 없는 대부분의 NDO도 RTNL 및 해당되는 instance lock 아래 실행됩니다.
콜백별 잠금과 실행 문맥을 압축했습니다.
struct net_device synchronization rules
=======================================
ndo_open:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
ndo_stop:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
Note: netif_running() is guaranteed false
ndo_do_ioctl:
Synchronization: rtnl_lock() semaphore.
This is only called by network subsystems internally,
not by user space calling ioctl as it was in before
linux-5.14.
ndo_siocbond:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
Used by the bonding driver for the SIOCBOND family of
ioctl commands.
ndo_siocwandev:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
Used by the drivers/net/wan framework to handle
the SIOCWANDEV ioctl with the if_settings structure.
ndo_siocdevprivate:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
This is used to implement SIOCDEVPRIVATE ioctl helpers.
These should not be added to new drivers, so don't use.
ndo_eth_ioctl:
Synchronization: rtnl_lock() semaphore. In addition, netdev instance
lock if the driver implements queue management or shaper API.
Context: process
ndo_get_stats:
Synchronization: RCU (can be called concurrently with the stats
update path).
Context: atomic (can't sleep under RCU)
ndo_start_xmit:
Synchronization: __netif_tx_lock spinlock.
When the driver sets dev->lltx this will be
called without holding netif_tx_lock. In this case the driver
has to lock by itself when needed.
The locking there should also properly protect against
set_rx_mode. WARNING: use of dev->lltx is deprecated.
Don't use it for new drivers.
Context: Process with BHs disabled or BH (timer),
will be called with interrupts disabled by netconsole.
Return codes:
* NETDEV_TX_OK everything ok.
* NETDEV_TX_BUSY Cannot transmit packet, try later
Usually a bug, means queue start/stop flow control is broken in
the driver. Note: the driver must NOT put the skb in its DMA ring.
ndo_tx_timeout:
Synchronization: netif_tx_lock spinlock; all TX queues frozen.
Context: BHs disabled
Notes: netif_queue_stopped() is guaranteed true
ndo_set_rx_mode:
Synchronization: netif_addr_lock spinlock.
Context: BHs disabled
ndo_setup_tc:
``TC_SETUP_BLOCK`` and ``TC_SETUP_FT`` are running under NFT locks
(i.e. no ``rtnl_lock`` and no device instance lock). The rest of
``tc_setup_type`` types run under netdev instance lock if the driver
implements queue management or shaper API.
Most ndo callbacks not specified in the list above are running
under ``rtnl_lock``. In addition, netdev instance lock is taken as well if
the driver implements queue management or shaper API.
napi_struct 동기화
303-316`napi->poll`은 `napi->state`의 `NAPI_STATE_SCHED` 비트로 동기화되고 softirq 문맥에서 실행됩니다. 드라이버의 `ndo_stop`은 모든 NAPI 인스턴스에 `napi_disable()`을 호출하며, 이 함수는 해당 비트를 sleep 가능한 방식으로 polling하여 진행 중인 NAPI 작업이 모두 끝날 때까지 기다립니다. Netconsole이 호출하는 경우 poll은 인터럽트가 비활성화된 상태입니다.
struct napi_struct synchronization rules
========================================
napi->poll:
Synchronization:
NAPI_STATE_SCHED bit in napi->state. Device
driver's ndo_stop method will invoke napi_disable() on
all NAPI instances which will do a sleeping poll on the
NAPI_STATE_SCHED napi->state bit, waiting for all pending
NAPI activity to cease.
Context:
softirq
will be called with interrupts disabled by netconsole.
netdev instance lock 모델
317-344과거에는 모든 네트워크 제어 작업을 전역 `rtnl_lock` 하나로 보호했습니다. 현재는 네트워크 namespace별 잠금과 netdev별 잠금으로 나누는 작업이 진행 중입니다. shaping 또는 queue management API를 구현한 드라이버의 제어 작업은 instance lock 아래 수행됩니다.
드라이버는 `request_ops_lock = true`로 ops 호출 때 instance lock 보유를 명시적으로 요청할 수 있습니다. 문서와 코드 주석에서는 이런 드라이버를 'ops locked'라고 부릅니다. 장기적으로 개별 드라이버가 RTNL 사용을 중단하고 instance lock만으로 제어 작업을 수행하도록 선택할 수 있게 할 예정이며, 가능한 드라이버는 이 잠금에 의존하도록 권장됩니다.
코어 스택과 상호작용하는 주로 소프트웨어인 드라이버를 위해 `dev_xxx`/`netdev_xxx`와 `netif_xxx` 두 API 계열이 있습니다. `dev_set_mtu` 같은 전자는 instance lock을 내부에서 획득하고, `netif_set_mtu` 같은 후자는 호출자가 이미 획득했다고 가정합니다.
netdev instance lock
====================
Historically, all networking control operations were protected by a single
global lock known as ``rtnl_lock``. There is an ongoing effort to replace this
global lock with separate locks for each network namespace. Additionally,
properties of individual netdev are increasingly protected by per-netdev locks.
For device drivers that implement shaping or queue management APIs, all control
operations will be performed under the netdev instance lock.
Drivers can also explicitly request instance lock to be held during ops
by setting ``request_ops_lock`` to true. Code comments and docs refer
to drivers which have ops called under the instance lock as "ops locked".
See also the documentation of the ``lock`` member of struct net_device.
In the future, there will be an option for individual
drivers to opt out of using ``rtnl_lock`` and instead perform their control
operations directly under the netdev instance lock.
Devices drivers are encouraged to rely on the instance lock where possible.
For the (mostly software) drivers that need to interact with the core stack,
there are two sets of interfaces: ``dev_xxx``/``netdev_xxx`` and ``netif_xxx``
(e.g., ``dev_set_mtu`` and ``netif_set_mtu``). The ``dev_xxx``/``netdev_xxx``
functions handle acquiring the instance lock themselves, while the
``netif_xxx`` functions assume that the driver has already acquired
the instance lock.
net_device_ops의 instance lock
345-352대부분의 일반 드라이버에서는 `ndos`가 instance lock 없이 호출됩니다. 'ops locked' 드라이버에서는 대부분의 `struct net_device_ops` 콜백이 instance lock 아래 호출됩니다.
struct net_device_ops
---------------------
``ndos`` are called without holding the instance lock for most drivers.
"Ops locked" drivers will have most of the ``ndos`` invoked under
the instance lock.
ethtool_ops의 instance lock
353-359`ethtool_ops`도 일반 NDO와 마찬가지로 선택된 드라이버에서만 instance lock을 잡습니다. 'ops locked' 드라이버에서는 예외 없이 모든 ethtool 작업이 instance lock 아래 호출되어야 합니다.
struct ethtool_ops
------------------
Similarly to ``ndos`` the instance lock is only held for select drivers.
For "ops locked" drivers all ethtool ops without exceptions should
be called under the instance lock.
netdev_stat_ops의 잠금
360-365`netdev_stat_ops`의 qstat 작업은 'ops locked' 드라이버에서는 instance lock 아래 호출되고, 그 밖의 드라이버에서는 `rtnl_lock` 아래 호출됩니다.
struct netdev_stat_ops
----------------------
"qstat" ops are invoked under the instance lock for "ops locked" drivers,
and under rtnl_lock for all other drivers.
net_shaper_ops의 잠금
366-373모든 net shaper 콜백은 netdev instance lock을 보유한 채 호출됩니다. 이때 `rtnl_lock`은 잡혀 있을 수도 있고 아닐 수도 있습니다. net shaper 지원을 선언하면 'ops locking'이 자동으로 활성화됩니다.
struct net_shaper_ops
---------------------
All net shaper callbacks are invoked while holding the netdev instance
lock. ``rtnl_lock`` may or may not be held.
Note that supporting net shapers automatically enables "ops locking".
queue management ops의 잠금
374-382모든 queue management 콜백도 netdev instance lock 아래 호출되며 RTNL 보유 여부는 일정하지 않습니다. `struct netdev_queue_mgmt_ops`를 지원하면 'ops locking'이 자동으로 활성화됩니다.
struct netdev_queue_mgmt_ops
----------------------------
All queue management callbacks are invoked while holding the netdev instance
lock. ``rtnl_lock`` may or may not be held.
Note that supporting struct netdev_queue_mgmt_ops automatically enables
"ops locking".
notifier와 instance lock
383-409shaping 또는 queue management API를 구현한 드라이버에서는 일부 `enum netdev_cmd` notifier가 instance lock 아래 실행됩니다. `NETDEV_XDP_FEAT_CHANGE`는 언제나 이 잠금을 보유합니다.
locked ops 장치에서는 현재 `NETDEV_CHANGE`, `NETDEV_REGISTER`, `NETDEV_UP`만 잠금 아래 실행되고 `NETDEV_UNREGISTER`는 잠금 없이 실행됩니다. 나머지 notifier는 명확한 보장이 없으며 같은 종류도 코드 경로에 따라 잠금이 있거나 없을 수 있습니다.
목표는 문서화된 소수 예외를 제외한 대부분의 notifier가 instance lock 아래 실행되도록 만드는 것입니다. notifier에서 잠금 보유를 명시적으로 가정하는 코드를 추가할 때는 이 문서도 함께 갱신해야 합니다.
현재 명시된 instance lock 보장을 구분합니다.
Notifiers and netdev instance lock
----------------------------------
For device drivers that implement shaping or queue management APIs,
some of the notifiers (``enum netdev_cmd``) are running under the netdev
instance lock.
The following netdev notifiers are always run under the instance lock:
* ``NETDEV_XDP_FEAT_CHANGE``
For devices with locked ops, currently only the following notifiers are
running under the lock:
* ``NETDEV_CHANGE``
* ``NETDEV_REGISTER``
* ``NETDEV_UP``
The following notifiers are running without the lock:
* ``NETDEV_UNREGISTER``
There are no clear expectations for the remaining notifiers. Notifiers not on
the list may run with or without the instance lock, potentially even invoking
the same notifier type with and without the lock from different code paths.
The goal is to eventually ensure that all (or most, with a few documented
exceptions) notifiers run under the instance lock. Please extend this
documentation whenever you make explicit assumption about lock being held
from a notifier.
NETDEV_INTERNAL 심볼 namespace
410-418`NETDEV_INTERNAL`로 export된 심볼은 네트워킹 코어와 주 네트워킹 목록·트리를 통해서만 데이터가 흐르는 드라이버에서만 사용할 수 있습니다. 역은 성립하지 않습니다. 이 namespace 밖의 심볼 대부분도 임의의 외부 코드가 쓰도록 의도된 것은 아닙니다. 오래되어 namespace 표기가 없거나 단순 누락일 수 있으므로 export 표기만으로 공개 API라고 판단하면 안 됩니다.
NETDEV_INTERNAL symbol namespace
================================
Symbols exported as NETDEV_INTERNAL can only be used in networking
core and drivers which exclusively flow via the main networking list and trees.
Note that the inverse is not true, most symbols outside of NETDEV_INTERNAL
are not expected to be used by random code outside netdev either.
Symbols may lack the designation because they predate the namespaces,
or simply due to an oversight.
요약·해설
netdevices.rst:1-418장치 등록 전에 초기화를 끝내고 호출 문맥의 RTNL 상태에 맞는 API를 선택해야 합니다. 콜백별 잠금·문맥 보장과 점진적으로 확대되는 netdev instance lock 규칙도 반드시 따라야 합니다.
수명과 동기화의 두 축입니다.