요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
===========
NFS LOCALIO
===========
Overview
========
The LOCALIO auxiliary RPC protocol allows the Linux NFS client and
server to reliably handshake to determine if they are on the same
host. Select "NFS client and server support for LOCALIO auxiliary
protocol" in menuconfig to enable CONFIG_NFS_LOCALIO in the kernel
config (both CONFIG_NFS_FS and CONFIG_NFSD must also be enabled).
Once an NFS client and server handshake as "local", the client will
bypass the network RPC protocol for read, write and commit operations.
Due to this XDR and RPC bypass, these operations will operate faster.
The LOCALIO auxiliary protocol's implementation, which uses the same
connection as NFS traffic, follows the pattern established by the NFS
ACL protocol extension.
The LOCALIO auxiliary protocol is needed to allow robust discovery of
clients local to their servers. In a private implementation that
preceded use of this LOCALIO protocol, a fragile sockaddr network
address based match against all local network interfaces was attempted.
But unlike the LOCALIO protocol, the sockaddr-based matching didn't
handle use of iptables or containers.
The robust handshake between local client and server is just the
beginning, the ultimate use case this locality makes possible is the
client is able to open files and issue reads, writes and commits
directly to the server without having to go over the network. The
requirement is to perform these loopback NFS operations as efficiently
as possible, this is particularly useful for container use cases
(e.g. kubernetes) where it is possible to run an IO job local to the
server.
The performance advantage realized from LOCALIO's ability to bypass
using XDR and RPC for reads, writes and commits can be extreme, e.g.:
fio for 20 secs with directio, qd of 8, 16 libaio threads:
- With LOCALIO:
4K read: IOPS=979k, BW=3825MiB/s (4011MB/s)(74.7GiB/20002msec)
4K write: IOPS=165k, BW=646MiB/s (678MB/s)(12.6GiB/20002msec)
128K read: IOPS=402k, BW=49.1GiB/s (52.7GB/s)(982GiB/20002msec)
128K write: IOPS=11.5k, BW=1433MiB/s (1503MB/s)(28.0GiB/20004msec)
- Without LOCALIO:
4K read: IOPS=79.2k, BW=309MiB/s (324MB/s)(6188MiB/20003msec)
4K write: IOPS=59.8k, BW=234MiB/s (245MB/s)(4671MiB/20002msec)
128K read: IOPS=33.9k, BW=4234MiB/s (4440MB/s)(82.7GiB/20004msec)
128K write: IOPS=11.5k, BW=1434MiB/s (1504MB/s)(28.0GiB/20011msec)
fio for 20 secs with directio, qd of 8, 1 libaio thread:
- With LOCALIO:
4K read: IOPS=230k, BW=898MiB/s (941MB/s)(17.5GiB/20001msec)
4K write: IOPS=22.6k, BW=88.3MiB/s (92.6MB/s)(1766MiB/20001msec)
128K read: IOPS=38.8k, BW=4855MiB/s (5091MB/s)(94.8GiB/20001msec)
128K write: IOPS=11.4k, BW=1428MiB/s (1497MB/s)(27.9GiB/20001msec)
- Without LOCALIO:
4K read: IOPS=77.1k, BW=301MiB/s (316MB/s)(6022MiB/20001msec)
4K write: IOPS=32.8k, BW=128MiB/s (135MB/s)(2566MiB/20001msec)
128K read: IOPS=24.4k, BW=3050MiB/s (3198MB/s)(59.6GiB/20001msec)
128K write: IOPS=11.4k, BW=1430MiB/s (1500MB/s)(27.9GiB/20001msec)
FAQ
===
1. What are the use cases for LOCALIO?
a. Workloads where the NFS client and server are on the same host
realize improved IO performance. In particular, it is common when
running containerised workloads for jobs to find themselves
running on the same host as the knfsd server being used for
storage.
2. What are the requirements for LOCALIO?
a. Bypass use of the network RPC protocol as much as possible. This
includes bypassing XDR and RPC for open, read, write and commit
operations.
b. Allow client and server to autonomously discover if they are
running local to each other without making any assumptions about
the local network topology.
c. Support the use of containers by being compatible with relevant
namespaces (e.g. network, user, mount).
d. Support all versions of NFS. NFSv3 is of particular importance
because it has wide enterprise usage and pNFS flexfiles makes use
of it for the data path.
3. Why doesn’t LOCALIO just compare IP addresses or hostnames when
deciding if the NFS client and server are co-located on the same
host?
Since one of the main use cases is containerised workloads, we cannot
assume that IP addresses will be shared between the client and
server. This sets up a requirement for a handshake protocol that
needs to go over the same connection as the NFS traffic in order to
identify that the client and the server really are running on the
same host. The handshake uses a secret that is sent over the wire,
and can be verified by both parties by comparing with a value stored
in shared kernel memory if they are truly co-located.
4. Does LOCALIO improve pNFS flexfiles?
Yes, LOCALIO complements pNFS flexfiles by allowing it to take
advantage of NFS client and server locality. Policy that initiates
client IO as closely to the server where the data is stored naturally
benefits from the data path optimization LOCALIO provides.
5. Why not develop a new pNFS layout to enable LOCALIO?
A new pNFS layout could be developed, but doing so would put the
onus on the server to somehow discover that the client is co-located
when deciding to hand out the layout.
There is value in a simpler approach (as provided by LOCALIO) that
allows the NFS client to negotiate and leverage locality without
requiring more elaborate modeling and discovery of such locality in a
more centralized manner.
6. Why is having the client perform a server-side file OPEN, without
using RPC, beneficial? Is the benefit pNFS specific?
Avoiding the use of XDR and RPC for file opens is beneficial to
performance regardless of whether pNFS is used. Especially when
dealing with small files its best to avoid going over the wire
whenever possible, otherwise it could reduce or even negate the
benefits of avoiding the wire for doing the small file I/O itself.
Given LOCALIO's requirements the current approach of having the
client perform a server-side file open, without using RPC, is ideal.
If in the future requirements change then we can adapt accordingly.
7. Why is LOCALIO only supported with UNIX Authentication (AUTH_UNIX)?
Strong authentication is usually tied to the connection itself. It
works by establishing a context that is cached by the server, and
that acts as the key for discovering the authorisation token, which
can then be passed to rpc.mountd to complete the authentication
process. On the other hand, in the case of AUTH_UNIX, the credential
that was passed over the wire is used directly as the key in the
upcall to rpc.mountd. This simplifies the authentication process, and
so makes AUTH_UNIX easier to support.
8. How do export options that translate RPC user IDs behave for LOCALIO
operations (eg. root_squash, all_squash)?
Export options that translate user IDs are managed by nfsd_setuser()
which is called by nfsd_setuser_and_check_port() which is called by
__fh_verify(). So they get handled exactly the same way for LOCALIO
as they do for non-LOCALIO.
9. How does LOCALIO make certain that object lifetimes are managed
properly given NFSD and NFS operate in different contexts?
See the detailed "NFS Client and Server Interlock" section below.
RPC
===
The LOCALIO auxiliary RPC protocol consists of a single "UUID_IS_LOCAL"
RPC method that allows the Linux NFS client to verify the local Linux
NFS server can see the nonce (single-use UUID) the client generated and
made available in nfs_common. This protocol isn't part of an IETF
standard, nor does it need to be considering it is Linux-to-Linux
auxiliary RPC protocol that amounts to an implementation detail.
The UUID_IS_LOCAL method encodes the client generated uuid_t in terms of
the fixed UUID_SIZE (16 bytes). The fixed size opaque encode and decode
XDR methods are used instead of the less efficient variable sized
methods.
The RPC program number for the NFS_LOCALIO_PROGRAM is 400122 (as assigned
by IANA, see https://www.iana.org/assignments/rpc-program-numbers/ ):
Linux Kernel Organization 400122 nfslocalio
The LOCALIO protocol spec in rpcgen syntax is::
/* raw RFC 9562 UUID */
#define UUID_SIZE 16
typedef u8 uuid_t<UUID_SIZE>;
program NFS_LOCALIO_PROGRAM {
version LOCALIO_V1 {
void
NULL(void) = 0;
void
UUID_IS_LOCAL(uuid_t) = 1;
} = 1;
} = 400122;
LOCALIO uses the same transport connection as NFS traffic. As such,
LOCALIO is not registered with rpcbind.
NFS Common and Client/Server Handshake
======================================
fs/nfs_common/nfslocalio.c provides interfaces that enable an NFS client
to generate a nonce (single-use UUID) and associated short-lived
nfs_uuid_t struct, register it with nfs_common for subsequent lookup and
verification by the NFS server and if matched the NFS server populates
members in the nfs_uuid_t struct. The NFS client then uses nfs_common to
transfer the nfs_uuid_t from its nfs_uuids to the nn->nfsd_serv
clients_list from the nfs_common's uuids_list. See:
fs/nfs/localio.c:nfs_local_probe()
nfs_common's nfs_uuids list is the basis for LOCALIO enablement, as such
it has members that point to nfsd memory for direct use by the client
(e.g. 'net' is the server's network namespace, through it the client can
access nn->nfsd_serv with proper rcu read access). It is this client
and server synchronization that enables advanced usage and lifetime of
objects to span from the host kernel's nfsd to per-container knfsd
instances that are connected to nfs client's running on the same local
host.
NFS Client and Server Interlock
===============================
LOCALIO provides the nfs_uuid_t object and associated interfaces to
allow proper network namespace (net-ns) and NFSD object refcounting.
LOCALIO required the introduction and use of NFSD's percpu nfsd_net_ref
to interlock nfsd_shutdown_net() and nfsd_open_local_fh(), to ensure
each net-ns is not destroyed while in use by nfsd_open_local_fh(), and
warrants a more detailed explanation:
nfsd_open_local_fh() uses nfsd_net_try_get() before opening its
nfsd_file handle and then the caller (NFS client) must drop the
reference for the nfsd_file and associated net-ns using
nfsd_file_put_local() once it has completed its IO.
This interlock working relies heavily on nfsd_open_local_fh() being
afforded the ability to safely deal with the possibility that the
NFSD's net-ns (and nfsd_net by association) may have been destroyed
by nfsd_destroy_serv() via nfsd_shutdown_net().
This interlock of the NFS client and server has been verified to fix an
easy to hit crash that would occur if an NFSD instance running in a
container, with a LOCALIO client mounted, is shutdown. Upon restart of
the container and associated NFSD, the client would go on to crash due
to NULL pointer dereference that occurred due to the LOCALIO client's
attempting to nfsd_open_local_fh() without having a proper reference on
NFSD's net-ns.
NFS Client issues IO instead of Server
======================================
Because LOCALIO is focused on protocol bypass to achieve improved IO
performance, alternatives to the traditional NFS wire protocol (SUNRPC
with XDR) must be provided to access the backing filesystem.
See fs/nfs/localio.c:nfs_local_open_fh() and
fs/nfsd/localio.c:nfsd_open_local_fh() for the interface that makes
focused use of select nfs server objects to allow a client local to a
server to open a file pointer without needing to go over the network.
The client's fs/nfs/localio.c:nfs_local_open_fh() will call into the
server's fs/nfsd/localio.c:nfsd_open_local_fh() and carefully access
both the associated nfsd network namespace and nn->nfsd_serv in terms of
RCU. If nfsd_open_local_fh() finds that the client no longer sees valid
nfsd objects (be it struct net or nn->nfsd_serv) it returns -ENXIO
to nfs_local_open_fh() and the client will try to reestablish the
LOCALIO resources needed by calling nfs_local_probe() again. This
recovery is needed if/when an nfsd instance running in a container were
to reboot while a LOCALIO client is connected to it.
Once the client has an open nfsd_file pointer it will issue reads,
writes and commits directly to the underlying local filesystem (normally
done by the nfs server). As such, for these operations, the NFS client
is issuing IO to the underlying local filesystem that it is sharing with
the NFS server. See: fs/nfs/localio.c:nfs_local_doio() and
fs/nfs/localio.c:nfs_local_commit().
With normal NFS that makes use of RPC to issue IO to the server, if an
application uses O_DIRECT the NFS client will bypass the pagecache but
the NFS server will not. The NFS server's use of buffered IO affords
applications to be less precise with their alignment when issuing IO to
the NFS client. But if all applications properly align their IO, LOCALIO
can be configured to use end-to-end O_DIRECT semantics from the NFS
client to the underlying local filesystem, that it is sharing with
the NFS server, by setting the 'localio_O_DIRECT_semantics' nfs module
parameter to Y, e.g.:
echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_semantics
Once enabled, it will cause LOCALIO to use end-to-end O_DIRECT semantics
(but again, this may cause IO to fail if applications do not properly
align their IO).
Security
========
LOCALIO is only supported when UNIX-style authentication (AUTH_UNIX, aka
AUTH_SYS) is used.
Care is taken to ensure the same NFS security mechanisms are used
(authentication, etc) regardless of whether LOCALIO or regular NFS
access is used. The auth_domain established as part of the traditional
NFS client access to the NFS server is also used for LOCALIO.
Relative to containers, LOCALIO gives the client access to the network
namespace the server has. This is required to allow the client to access
the server's per-namespace nfsd_net struct. With traditional NFS, the
client is afforded this same level of access (albeit in terms of the NFS
protocol via SUNRPC). No other namespaces (user, mount, etc) have been
altered or purposely extended from the server to the client.
Module Parameters
=================
/sys/module/nfs/parameters/localio_enabled (bool)
controls if LOCALIO is enabled, defaults to Y. If client and server are
local but 'localio_enabled' is set to N then LOCALIO will not be used.
/sys/module/nfs/parameters/localio_O_DIRECT_semantics (bool)
controls if O_DIRECT extends down to the underlying filesystem, defaults
to N. Application IO must be logical blocksize aligned, otherwise
O_DIRECT will fail.
/sys/module/nfsv3/parameters/nfs3_localio_probe_throttle (uint)
controls if NFSv3 read and write IOs will trigger (re)enabling of
LOCALIO every N (nfs3_localio_probe_throttle) IOs, defaults to 0
(disabled). Must be power-of-2, admin keeps all the pieces if they
misconfigure (too low a value or non-power-of-2).
Testing
=======
The LOCALIO auxiliary protocol and associated NFS LOCALIO read, write
and commit access have proven stable against various test scenarios:
- Client and server both on the same host.
- All permutations of client and server support enablement for both
local and remote client and server.
- Testing against NFS storage products that don't support the LOCALIO
protocol was also performed.
- Client on host, server within a container (for both v3 and v4.2).
The container testing was in terms of podman managed containers and
includes successful container stop/restart scenario.
- Formalizing these test scenarios in terms of existing test
infrastructure is on-going. Initial regular coverage is provided in
terms of ktest running xfstests against a LOCALIO-enabled NFS loopback
mount configuration, and includes lockdep and KASAN coverage, see:
https://evilpiepirate.org/~testdashboard/ci?user=snitzer&branch=snitm-nfs-next
https://github.com/koverstreet/ktest
- Various kdevops testing (in terms of "Chuck's BuildBot") has been
performed to regularly verify the LOCALIO changes haven't caused any
regressions to non-LOCALIO NFS use cases.
- All of Hammerspace's various sanity tests pass with LOCALIO enabled
(this includes numerous pNFS and flexfiles tests).
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
LOCALIO handshake와 network RPC 우회
1-40LOCALIO auxiliary RPC protocol은 Linux NFS client와 server가 같은 host에 있는지 신뢰성 있게 handshake하여 판별하게 합니다. Kernel config에서 `NFS client and server support for LOCALIO auxiliary protocol`을 선택해 `CONFIG_NFS_LOCALIO`를 켜며, `CONFIG_NFS_FS`와 `CONFIG_NFSD`도 함께 활성화해야 합니다.
Client와 server가 local이라고 확인되면 client는 read, write, commit operation에서 network RPC protocol을 우회합니다. XDR encode/decode와 RPC 전달을 생략하므로 같은 host의 backing filesystem에 더 빠르게 접근할 수 있습니다. LOCALIO는 NFS traffic과 동일한 connection을 사용하며 NFS ACL protocol extension이 세운 구현 pattern을 따릅니다.
이 auxiliary protocol은 local client를 견고하게 발견하기 위해 필요합니다. 이전 private 구현은 모든 local network interface의 `sockaddr`와 주소를 비교했지만 iptables와 container 환경을 처리하지 못했습니다. IP address나 interface topology가 달라도 protocol handshake는 실제 kernel locality를 확인할 수 있습니다.
Handshake는 시작일 뿐이며 최종 목적은 client가 network를 거치지 않고 server-side file을 열어 read, write, commit을 직접 수행하게 하는 것입니다. Loopback NFS operation을 최대한 효율화하며, 특히 Kubernetes 같은 container 환경에서 I/O job이 storage server와 같은 host에 배치될 때 유용합니다.
LOCALIO가 read, write, commit에서 XDR과 RPC를 제거하면 workload에 따라 성능 차이가 매우 커질 수 있습니다. 이어지는 fio 결과는 direct I/O, queue depth 8 조건에서 이를 예시하지만 특정 측정 환경의 결과이므로 일반적인 보장값으로 해석해서는 안 됩니다.
동일 connection의 handshake 결과가 I/O 경로를 결정합니다.
===========
NFS LOCALIO
===========
Overview
========
The LOCALIO auxiliary RPC protocol allows the Linux NFS client and
server to reliably handshake to determine if they are on the same
host. Select "NFS client and server support for LOCALIO auxiliary
protocol" in menuconfig to enable CONFIG_NFS_LOCALIO in the kernel
config (both CONFIG_NFS_FS and CONFIG_NFSD must also be enabled).
Once an NFS client and server handshake as "local", the client will
bypass the network RPC protocol for read, write and commit operations.
Due to this XDR and RPC bypass, these operations will operate faster.
The LOCALIO auxiliary protocol's implementation, which uses the same
connection as NFS traffic, follows the pattern established by the NFS
ACL protocol extension.
The LOCALIO auxiliary protocol is needed to allow robust discovery of
clients local to their servers. In a private implementation that
preceded use of this LOCALIO protocol, a fragile sockaddr network
address based match against all local network interfaces was attempted.
But unlike the LOCALIO protocol, the sockaddr-based matching didn't
handle use of iptables or containers.
The robust handshake between local client and server is just the
beginning, the ultimate use case this locality makes possible is the
client is able to open files and issue reads, writes and commits
directly to the server without having to go over the network. The
requirement is to perform these loopback NFS operations as efficiently
as possible, this is particularly useful for container use cases
(e.g. kubernetes) where it is possible to run an IO job local to the
server.
The performance advantage realized from LOCALIO's ability to bypass
using XDR and RPC for reads, writes and commits can be extreme, e.g.:
fio direct I/O 측정 결과
41-66첫 측정은 20초, direct I/O, queue depth 8, libaio thread 16개 조건입니다. LOCALIO 사용 시 4K read는 979k IOPS와 3825 MiB/s, 4K write는 165k IOPS와 646 MiB/s, 128K read는 402k IOPS와 49.1 GiB/s입니다. 128K write는 11.5k IOPS와 1433 MiB/s입니다.
같은 16-thread 조건에서 LOCALIO를 끄면 4K read는 79.2k IOPS, 4K write는 59.8k IOPS, 128K read는 33.9k IOPS로 낮아집니다. 128K write는 11.5k IOPS로 사실상 동일하여 workload와 operation에 따라 우회 효과가 달라짐을 보여 줍니다.
둘째 측정은 같은 시간·direct I/O·queue depth에서 libaio thread를 1개만 사용합니다. LOCALIO 사용 시 4K read 230k, 4K write 22.6k, 128K read 38.8k, 128K write 11.4k IOPS입니다. LOCALIO를 끄면 각각 77.1k, 32.8k, 24.4k, 11.4k IOPS입니다.
Single-thread 4K write는 이 표에서 LOCALIO 쪽이 더 낮고 128K write는 거의 차이가 없습니다. 따라서 LOCALIO의 이점은 단순한 일괄 배수가 아니라 XDR/RPC overhead 비중, concurrency, I/O size와 backing filesystem 동작에 좌우됩니다.
원문의 IOPS를 thread 수와 I/O 형태별로 비교합니다.
fio for 20 secs with directio, qd of 8, 16 libaio threads:
- With LOCALIO:
4K read: IOPS=979k, BW=3825MiB/s (4011MB/s)(74.7GiB/20002msec)
4K write: IOPS=165k, BW=646MiB/s (678MB/s)(12.6GiB/20002msec)
128K read: IOPS=402k, BW=49.1GiB/s (52.7GB/s)(982GiB/20002msec)
128K write: IOPS=11.5k, BW=1433MiB/s (1503MB/s)(28.0GiB/20004msec)
- Without LOCALIO:
4K read: IOPS=79.2k, BW=309MiB/s (324MB/s)(6188MiB/20003msec)
4K write: IOPS=59.8k, BW=234MiB/s (245MB/s)(4671MiB/20002msec)
128K read: IOPS=33.9k, BW=4234MiB/s (4440MB/s)(82.7GiB/20004msec)
128K write: IOPS=11.5k, BW=1434MiB/s (1504MB/s)(28.0GiB/20011msec)
fio for 20 secs with directio, qd of 8, 1 libaio thread:
- With LOCALIO:
4K read: IOPS=230k, BW=898MiB/s (941MB/s)(17.5GiB/20001msec)
4K write: IOPS=22.6k, BW=88.3MiB/s (92.6MB/s)(1766MiB/20001msec)
128K read: IOPS=38.8k, BW=4855MiB/s (5091MB/s)(94.8GiB/20001msec)
128K write: IOPS=11.4k, BW=1428MiB/s (1497MB/s)(27.9GiB/20001msec)
- Without LOCALIO:
4K read: IOPS=77.1k, BW=301MiB/s (316MB/s)(6022MiB/20001msec)
4K write: IOPS=32.8k, BW=128MiB/s (135MB/s)(2566MiB/20001msec)
128K read: IOPS=24.4k, BW=3050MiB/s (3198MB/s)(59.6GiB/20001msec)
128K write: IOPS=11.4k, BW=1430MiB/s (1500MB/s)(27.9GiB/20001msec)
사용 사례, 요구 사항과 locality 판별
67-120LOCALIO의 주요 사용 사례는 NFS client와 server가 같은 host에서 실행되는 workload입니다. Containerized workload에서는 storage로 사용하는 knfsd server와 job이 같은 host에 놓이는 일이 흔하며, 이때 local I/O path가 성능을 개선합니다.
요구 사항은 네 가지입니다. 첫째 open, read, write, commit에서 가능한 한 XDR과 network RPC를 우회해야 합니다. 둘째 local network topology를 가정하지 않고 client와 server가 서로 local인지 자율적으로 발견해야 합니다. 셋째 network, user, mount namespace 같은 container 관련 namespace와 호환되어야 합니다. 넷째 NFSv3를 포함한 모든 NFS version을 지원해야 합니다. NFSv3는 enterprise에서 널리 사용되고 pNFS flexfiles data path에도 쓰이므로 특히 중요합니다.
IP address나 hostname만 비교하지 않는 이유는 container client와 server가 address를 공유한다고 가정할 수 없기 때문입니다. LOCALIO handshake는 NFS traffic과 같은 connection으로 secret을 전달합니다. 양쪽이 실제로 같은 host라면 shared kernel memory에 저장된 값과 비교해 그 secret을 모두 검증할 수 있습니다.
LOCALIO는 pNFS flexfiles가 client/server locality를 활용하게 하여 보완합니다. Data가 있는 server 가까이에서 client I/O를 시작하는 policy는 LOCALIO가 제공하는 data-path 최적화의 이점을 자연스럽게 얻습니다.
새 pNFS layout으로 같은 기능을 만들 수도 있지만 그러면 layout을 배포할 때 server가 client co-location을 발견해야 하는 부담이 생깁니다. LOCALIO는 client가 locality를 협상하고 활용하므로 중앙화된 정교한 locality model과 discovery가 필요하지 않은 더 단순한 접근입니다.
Protocol 우회와 container locality를 함께 만족해야 합니다.
FAQ
===
1. What are the use cases for LOCALIO?
a. Workloads where the NFS client and server are on the same host
realize improved IO performance. In particular, it is common when
running containerised workloads for jobs to find themselves
running on the same host as the knfsd server being used for
storage.
2. What are the requirements for LOCALIO?
a. Bypass use of the network RPC protocol as much as possible. This
includes bypassing XDR and RPC for open, read, write and commit
operations.
b. Allow client and server to autonomously discover if they are
running local to each other without making any assumptions about
the local network topology.
c. Support the use of containers by being compatible with relevant
namespaces (e.g. network, user, mount).
d. Support all versions of NFS. NFSv3 is of particular importance
because it has wide enterprise usage and pNFS flexfiles makes use
of it for the data path.
3. Why doesn’t LOCALIO just compare IP addresses or hostnames when
deciding if the NFS client and server are co-located on the same
host?
Since one of the main use cases is containerised workloads, we cannot
assume that IP addresses will be shared between the client and
server. This sets up a requirement for a handshake protocol that
needs to go over the same connection as the NFS traffic in order to
identify that the client and the server really are running on the
same host. The handshake uses a secret that is sent over the wire,
and can be verified by both parties by comparing with a value stored
in shared kernel memory if they are truly co-located.
4. Does LOCALIO improve pNFS flexfiles?
Yes, LOCALIO complements pNFS flexfiles by allowing it to take
advantage of NFS client and server locality. Policy that initiates
client IO as closely to the server where the data is stored naturally
benefits from the data path optimization LOCALIO provides.
5. Why not develop a new pNFS layout to enable LOCALIO?
A new pNFS layout could be developed, but doing so would put the
onus on the server to somehow discover that the client is co-located
when deciding to hand out the layout.
There is value in a simpler approach (as provided by LOCALIO) that
allows the NFS client to negotiate and leverage locality without
requiring more elaborate modeling and discovery of such locality in a
more centralized manner.Local open, AUTH_UNIX, ID translation과 lifetime
121-157Client가 RPC 없이 server-side file `OPEN`을 수행하면 pNFS 사용 여부와 무관하게 XDR과 RPC overhead를 줄입니다. 특히 small file에서는 실제 I/O가 network를 피하더라도 open이 wire를 왕복하면 이득이 줄거나 사라질 수 있으므로 local open이 중요합니다.
현재 요구 사항에서는 client가 RPC 없이 server-side file open을 수행하는 방식이 적절합니다. 향후 요구가 바뀌면 구현도 조정할 수 있습니다.
LOCALIO가 `AUTH_UNIX`만 지원하는 이유는 strong authentication이 보통 connection에 묶인 context를 server에 cache하고, 그 context를 authorization token discovery의 key로 사용한 뒤 `rpc.mountd`에 전달하는 구조이기 때문입니다. 반면 `AUTH_UNIX`는 wire로 받은 credential 자체를 `rpc.mountd` upcall의 key로 사용하므로 지원 과정이 단순합니다.
`root_squash`, `all_squash`처럼 RPC user ID를 변환하는 export option은 `__fh_verify()`가 호출하는 `nfsd_setuser_and_check_port()`와 그 안의 `nfsd_setuser()`가 처리합니다. 따라서 LOCALIO에서도 일반 NFS와 완전히 같은 방식으로 적용됩니다.
NFSD와 NFS가 서로 다른 context에서 동작할 때 object lifetime을 올바르게 관리하는 방식은 뒤의 `NFS Client and Server Interlock` 절에서 `nfs_uuid_t`, network namespace reference와 함께 설명합니다.
Network RPC는 우회하지만 기존 NFSD authorization 경로를 재사용합니다.
6. Why is having the client perform a server-side file OPEN, without
using RPC, beneficial? Is the benefit pNFS specific?
Avoiding the use of XDR and RPC for file opens is beneficial to
performance regardless of whether pNFS is used. Especially when
dealing with small files its best to avoid going over the wire
whenever possible, otherwise it could reduce or even negate the
benefits of avoiding the wire for doing the small file I/O itself.
Given LOCALIO's requirements the current approach of having the
client perform a server-side file open, without using RPC, is ideal.
If in the future requirements change then we can adapt accordingly.
7. Why is LOCALIO only supported with UNIX Authentication (AUTH_UNIX)?
Strong authentication is usually tied to the connection itself. It
works by establishing a context that is cached by the server, and
that acts as the key for discovering the authorisation token, which
can then be passed to rpc.mountd to complete the authentication
process. On the other hand, in the case of AUTH_UNIX, the credential
that was passed over the wire is used directly as the key in the
upcall to rpc.mountd. This simplifies the authentication process, and
so makes AUTH_UNIX easier to support.
8. How do export options that translate RPC user IDs behave for LOCALIO
operations (eg. root_squash, all_squash)?
Export options that translate user IDs are managed by nfsd_setuser()
which is called by nfsd_setuser_and_check_port() which is called by
__fh_verify(). So they get handled exactly the same way for LOCALIO
as they do for non-LOCALIO.
9. How does LOCALIO make certain that object lifetimes are managed
properly given NFSD and NFS operate in different contexts?
See the detailed "NFS Client and Server Interlock" section below.
UUID_IS_LOCAL auxiliary RPC 명세
158-195LOCALIO auxiliary RPC protocol은 `UUID_IS_LOCAL` method 하나로 구성됩니다. Linux NFS client가 생성해 `nfs_common`에 공개한 single-use UUID nonce를 local Linux NFS server도 볼 수 있는지 검증합니다. Linux-to-Linux 구현 세부 사항인 auxiliary protocol이므로 IETF standard의 일부가 아니며 그럴 필요도 없습니다.
`UUID_IS_LOCAL`은 client가 만든 `uuid_t`를 고정 `UUID_SIZE`, 즉 16 byte로 encode합니다. 가변 크기 방식보다 효율적인 fixed-size opaque XDR encode/decode method를 사용합니다.
IANA가 할당한 `NFS_LOCALIO_PROGRAM` RPC program number는 400122이며 등록 이름은 `nfslocalio`입니다. Version `LOCALIO_V1`에는 procedure 0인 `NULL(void)`와 procedure 1인 `UUID_IS_LOCAL(uuid_t)`가 있습니다.
/* raw RFC 9562 UUID */
#define UUID_SIZE 16
typedef u8 uuid_t<UUID_SIZE>;
program NFS_LOCALIO_PROGRAM {
version LOCALIO_V1 {
void NULL(void) = 0;
void UUID_IS_LOCAL(uuid_t) = 1;
} = 1;
} = 400122;LOCALIO는 NFS traffic과 같은 transport connection을 사용합니다. 별도 service endpoint를 찾을 필요가 없으므로 `rpcbind`에는 등록하지 않습니다.
고정 UUID를 확인하는 최소 auxiliary protocol입니다.
RPC
===
The LOCALIO auxiliary RPC protocol consists of a single "UUID_IS_LOCAL"
RPC method that allows the Linux NFS client to verify the local Linux
NFS server can see the nonce (single-use UUID) the client generated and
made available in nfs_common. This protocol isn't part of an IETF
standard, nor does it need to be considering it is Linux-to-Linux
auxiliary RPC protocol that amounts to an implementation detail.
The UUID_IS_LOCAL method encodes the client generated uuid_t in terms of
the fixed UUID_SIZE (16 bytes). The fixed size opaque encode and decode
XDR methods are used instead of the less efficient variable sized
methods.
The RPC program number for the NFS_LOCALIO_PROGRAM is 400122 (as assigned
by IANA, see https://www.iana.org/assignments/rpc-program-numbers/ ):
Linux Kernel Organization 400122 nfslocalio
The LOCALIO protocol spec in rpcgen syntax is::
/* raw RFC 9562 UUID */
#define UUID_SIZE 16
typedef u8 uuid_t<UUID_SIZE>;
program NFS_LOCALIO_PROGRAM {
version LOCALIO_V1 {
void
NULL(void) = 0;
void
UUID_IS_LOCAL(uuid_t) = 1;
} = 1;
} = 400122;
LOCALIO uses the same transport connection as NFS traffic. As such,
LOCALIO is not registered with rpcbind.
nfs_common handshake와 nfs_uuid_t 이전
196-216`fs/nfs_common/nfslocalio.c`는 NFS client가 single-use UUID nonce와 수명이 짧은 `nfs_uuid_t`를 생성하는 interface를 제공합니다. Client는 이를 `nfs_common`에 등록해 NFS server가 나중에 lookup하고 검증할 수 있게 합니다.
Server가 nonce를 일치시키면 `nfs_uuid_t`의 member를 채웁니다. 그 뒤 client는 `nfs_common`을 사용해 이 object를 `nfs_common`의 `uuids_list`에 있는 자신의 `nfs_uuids`에서 `nn->nfsd_serv`의 `clients_list`로 옮깁니다. Client 측 진입점은 `fs/nfs/localio.c:nfs_local_probe()`입니다.
`nfs_common`의 `nfs_uuids` list는 LOCALIO enablement의 기반입니다. Entry는 client가 직접 사용할 nfsd memory를 가리키며, 예를 들어 server network namespace인 `net`을 통해 적절한 RCU read access 아래 `nn->nfsd_serv`에 접근할 수 있습니다.
이 client/server synchronization 덕분에 host kernel의 nfsd와 같은 local host에서 NFS client에 연결된 container별 knfsd instance 사이에서도 object의 advanced usage와 lifetime을 안전하게 이어 갈 수 있습니다.
Nonce 생성부터 server client list 편입까지의 object 이동입니다.
NFS Common and Client/Server Handshake
======================================
fs/nfs_common/nfslocalio.c provides interfaces that enable an NFS client
to generate a nonce (single-use UUID) and associated short-lived
nfs_uuid_t struct, register it with nfs_common for subsequent lookup and
verification by the NFS server and if matched the NFS server populates
members in the nfs_uuid_t struct. The NFS client then uses nfs_common to
transfer the nfs_uuid_t from its nfs_uuids to the nn->nfsd_serv
clients_list from the nfs_common's uuids_list. See:
fs/nfs/localio.c:nfs_local_probe()
nfs_common's nfs_uuids list is the basis for LOCALIO enablement, as such
it has members that point to nfsd memory for direct use by the client
(e.g. 'net' is the server's network namespace, through it the client can
access nn->nfsd_serv with proper rcu read access). It is this client
and server synchronization that enables advanced usage and lifetime of
objects to span from the host kernel's nfsd to per-container knfsd
instances that are connected to nfs client's running on the same local
host.
NFS client/server network namespace interlock
217-245LOCALIO는 `nfs_uuid_t` object와 관련 interface를 제공해 network namespace, 즉 net-ns와 NFSD object의 reference count를 올바르게 관리합니다.
이를 위해 NFSD의 per-CPU `nfsd_net_ref`가 도입되었습니다. `nfsd_shutdown_net()`과 `nfsd_open_local_fh()`를 interlock하여 `nfsd_open_local_fh()`가 사용하는 동안 각 net-ns가 파괴되지 않도록 합니다.
`nfsd_open_local_fh()`는 `nfsd_file` handle을 열기 전에 `nfsd_net_try_get()`으로 reference를 얻습니다. I/O가 끝나면 caller인 NFS client가 `nfsd_file_put_local()`을 호출해 `nfsd_file`과 연결된 net-ns reference를 내려야 합니다.
Interlock은 `nfsd_destroy_serv()`가 `nfsd_shutdown_net()`을 거쳐 NFSD net-ns와 연관된 `nfsd_net`을 이미 파괴했을 가능성을 `nfsd_open_local_fh()`가 안전하게 처리할 수 있어야 한다는 점에 크게 의존합니다.
이 구조는 container 안 NFSD instance를 LOCALIO client가 mount한 상태에서 종료하고 다시 시작할 때 발생하던 crash를 고쳤습니다. 이전에는 client가 NFSD net-ns reference 없이 `nfsd_open_local_fh()`를 시도해 NULL pointer dereference가 발생할 수 있었습니다.
Open 전 획득과 I/O 후 해제가 shutdown과 맞물립니다.
NFS Client and Server Interlock
===============================
LOCALIO provides the nfs_uuid_t object and associated interfaces to
allow proper network namespace (net-ns) and NFSD object refcounting.
LOCALIO required the introduction and use of NFSD's percpu nfsd_net_ref
to interlock nfsd_shutdown_net() and nfsd_open_local_fh(), to ensure
each net-ns is not destroyed while in use by nfsd_open_local_fh(), and
warrants a more detailed explanation:
nfsd_open_local_fh() uses nfsd_net_try_get() before opening its
nfsd_file handle and then the caller (NFS client) must drop the
reference for the nfsd_file and associated net-ns using
nfsd_file_put_local() once it has completed its IO.
This interlock working relies heavily on nfsd_open_local_fh() being
afforded the ability to safely deal with the possibility that the
NFSD's net-ns (and nfsd_net by association) may have been destroyed
by nfsd_destroy_serv() via nfsd_shutdown_net().
This interlock of the NFS client and server has been verified to fix an
easy to hit crash that would occur if an NFSD instance running in a
container, with a LOCALIO client mounted, is shutdown. Upon restart of
the container and associated NFSD, the client would go on to crash due
to NULL pointer dereference that occurred due to the LOCALIO client's
attempting to nfsd_open_local_fh() without having a proper reference on
NFSD's net-ns.
Client가 backing filesystem에 직접 I/O 수행
246-290LOCALIO는 I/O 성능을 위해 protocol을 우회하므로 전통적인 NFS wire protocol인 XDR 기반 SUNRPC 대신 backing filesystem에 접근할 interface가 필요합니다.
Client의 `fs/nfs/localio.c:nfs_local_open_fh()`와 server의 `fs/nfsd/localio.c:nfsd_open_local_fh()`는 선택된 NFS server object를 사용해 network를 거치지 않고 local client가 file pointer를 열게 합니다. Client 함수가 server 함수를 직접 호출하고, server의 network namespace와 `nn->nfsd_serv`를 RCU 규칙에 맞춰 조심스럽게 접근합니다.
`nfsd_open_local_fh()`가 `struct net` 또는 `nn->nfsd_serv` 같은 유효한 nfsd object를 더 이상 보지 못하면 `-ENXIO`를 반환합니다. Client의 `nfs_local_open_fh()`는 이를 받아 `nfs_local_probe()`를 다시 호출해 LOCALIO resource를 재설정합니다. Container의 nfsd가 연결 중 reboot할 수 있으므로 이 recovery가 필요합니다.
Client가 열린 `nfsd_file` pointer를 얻으면 보통 NFS server가 수행할 read, write, commit을 공유 중인 local backing filesystem에 직접 발행합니다. 구현 진입점은 `fs/nfs/localio.c:nfs_local_doio()`와 `fs/nfs/localio.c:nfs_local_commit()`입니다.
일반 NFS에서 application이 `O_DIRECT`를 사용하면 client는 page cache를 우회하지만 server는 buffered I/O를 사용할 수 있습니다. 이 server buffering은 application의 alignment가 다소 부정확해도 동작하게 합니다.
모든 application이 I/O를 올바르게 align한다면 `localio_O_DIRECT_semantics` nfs module parameter를 `Y`로 설정해 client부터 backing filesystem까지 end-to-end `O_DIRECT`를 사용할 수 있습니다. 활성화 뒤 alignment가 맞지 않으면 I/O가 실패할 수 있습니다.
echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_semantics일반 NFS와 LOCALIO의 cache·alignment 의미를 비교합니다.
NFS Client issues IO instead of Server
======================================
Because LOCALIO is focused on protocol bypass to achieve improved IO
performance, alternatives to the traditional NFS wire protocol (SUNRPC
with XDR) must be provided to access the backing filesystem.
See fs/nfs/localio.c:nfs_local_open_fh() and
fs/nfsd/localio.c:nfsd_open_local_fh() for the interface that makes
focused use of select nfs server objects to allow a client local to a
server to open a file pointer without needing to go over the network.
The client's fs/nfs/localio.c:nfs_local_open_fh() will call into the
server's fs/nfsd/localio.c:nfsd_open_local_fh() and carefully access
both the associated nfsd network namespace and nn->nfsd_serv in terms of
RCU. If nfsd_open_local_fh() finds that the client no longer sees valid
nfsd objects (be it struct net or nn->nfsd_serv) it returns -ENXIO
to nfs_local_open_fh() and the client will try to reestablish the
LOCALIO resources needed by calling nfs_local_probe() again. This
recovery is needed if/when an nfsd instance running in a container were
to reboot while a LOCALIO client is connected to it.
Once the client has an open nfsd_file pointer it will issue reads,
writes and commits directly to the underlying local filesystem (normally
done by the nfs server). As such, for these operations, the NFS client
is issuing IO to the underlying local filesystem that it is sharing with
the NFS server. See: fs/nfs/localio.c:nfs_local_doio() and
fs/nfs/localio.c:nfs_local_commit().
With normal NFS that makes use of RPC to issue IO to the server, if an
application uses O_DIRECT the NFS client will bypass the pagecache but
the NFS server will not. The NFS server's use of buffered IO affords
applications to be less precise with their alignment when issuing IO to
the NFS client. But if all applications properly align their IO, LOCALIO
can be configured to use end-to-end O_DIRECT semantics from the NFS
client to the underlying local filesystem, that it is sharing with
the NFS server, by setting the 'localio_O_DIRECT_semantics' nfs module
parameter to Y, e.g.:
echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_semantics
Once enabled, it will cause LOCALIO to use end-to-end O_DIRECT semantics
(but again, this may cause IO to fail if applications do not properly
align their IO).
보안 경계와 module parameter
291-326LOCALIO는 UNIX-style authentication인 `AUTH_UNIX`, 즉 `AUTH_SYS`를 사용할 때만 지원됩니다. LOCALIO와 일반 NFS 접근에서 authentication을 포함한 동일한 NFS security mechanism을 사용하도록 하며, 전통적인 NFS client 접근에서 만든 `auth_domain`을 LOCALIO에도 재사용합니다.
Container 환경에서 LOCALIO client는 server의 network namespace에 접근할 수 있습니다. 이는 namespace별 `nfsd_net`에 접근하는 데 필요하며 일반 NFS client도 SUNRPC protocol을 통해 사실상 같은 수준의 접근을 받습니다. User, mount 등 다른 namespace를 server에서 client로 일부러 확장하거나 변경하지는 않습니다.
Boolean `/sys/module/nfs/parameters/localio_enabled`는 LOCALIO 사용 여부를 제어하며 기본값은 `Y`입니다. Client와 server가 local이어도 값을 `N`으로 설정하면 LOCALIO를 사용하지 않습니다.
Boolean `/sys/module/nfs/parameters/localio_O_DIRECT_semantics`는 `O_DIRECT`가 underlying filesystem까지 이어지는지 제어하며 기본값은 `N`입니다. 활성화하면 application I/O가 logical block size에 맞게 align되어야 하며 그렇지 않으면 실패합니다.
Unsigned integer `/sys/module/nfsv3/parameters/nfs3_localio_probe_throttle`은 NFSv3 read/write I/O가 매 N회마다 LOCALIO 재활성화를 시도할지 제어합니다. 기본값 0은 비활성화입니다. 값은 power of 2여야 하며 너무 작거나 power of 2가 아닌 값으로 잘못 구성한 결과는 administrator가 감수해야 합니다.
기본값과 활성화 시 의미입니다.
Security
========
LOCALIO is only supported when UNIX-style authentication (AUTH_UNIX, aka
AUTH_SYS) is used.
Care is taken to ensure the same NFS security mechanisms are used
(authentication, etc) regardless of whether LOCALIO or regular NFS
access is used. The auth_domain established as part of the traditional
NFS client access to the NFS server is also used for LOCALIO.
Relative to containers, LOCALIO gives the client access to the network
namespace the server has. This is required to allow the client to access
the server's per-namespace nfsd_net struct. With traditional NFS, the
client is afforded this same level of access (albeit in terms of the NFS
protocol via SUNRPC). No other namespaces (user, mount, etc) have been
altered or purposely extended from the server to the client.
Module Parameters
=================
/sys/module/nfs/parameters/localio_enabled (bool)
controls if LOCALIO is enabled, defaults to Y. If client and server are
local but 'localio_enabled' is set to N then LOCALIO will not be used.
/sys/module/nfs/parameters/localio_O_DIRECT_semantics (bool)
controls if O_DIRECT extends down to the underlying filesystem, defaults
to N. Application IO must be logical blocksize aligned, otherwise
O_DIRECT will fail.
/sys/module/nfsv3/parameters/nfs3_localio_probe_throttle (uint)
controls if NFSv3 read and write IOs will trigger (re)enabling of
LOCALIO every N (nfs3_localio_probe_throttle) IOs, defaults to 0
(disabled). Must be power-of-2, admin keeps all the pieces if they
misconfigure (too low a value or non-power-of-2).
검증 시나리오와 회귀 검사
327-357LOCALIO auxiliary protocol과 NFS LOCALIO read, write, commit 경로는 여러 test scenario에서 안정성을 검증했습니다. Client와 server가 같은 host인 기본 경우뿐 아니라 local·remote 배치와 양쪽 LOCALIO 지원 enablement의 모든 조합을 시험했습니다.
LOCALIO protocol을 지원하지 않는 NFS storage product와의 호환성도 검사했습니다. Host의 client와 container 내부 server 조합은 NFSv3와 NFSv4.2에서 모두 시험했고, Podman 관리 container의 정상 stop/restart도 성공했습니다.
이 scenario를 기존 test infrastructure에 정식 편입하는 작업은 진행 중입니다. 초기 정기 coverage는 LOCALIO-enabled NFS loopback mount에 `xfstests`를 실행하는 `ktest`로 제공되며 lockdep과 KASAN 검사도 포함합니다. 원문은 test dashboard와 ktest GitHub URL을 제시합니다.
Chuck's BuildBot 형태의 여러 `kdevops` test도 LOCALIO 변경이 일반 non-LOCALIO NFS 사용 사례에 regression을 만들지 않았는지 정기적으로 확인했습니다. Hammerspace의 pNFS와 flexfiles test를 포함한 여러 sanity test도 LOCALIO 활성 상태에서 통과했습니다.
기능과 호환성, lifetime, regression을 함께 확인합니다.
Testing
=======
The LOCALIO auxiliary protocol and associated NFS LOCALIO read, write
and commit access have proven stable against various test scenarios:
- Client and server both on the same host.
- All permutations of client and server support enablement for both
local and remote client and server.
- Testing against NFS storage products that don't support the LOCALIO
protocol was also performed.
- Client on host, server within a container (for both v3 and v4.2).
The container testing was in terms of podman managed containers and
includes successful container stop/restart scenario.
- Formalizing these test scenarios in terms of existing test
infrastructure is on-going. Initial regular coverage is provided in
terms of ktest running xfstests against a LOCALIO-enabled NFS loopback
mount configuration, and includes lockdep and KASAN coverage, see:
https://evilpiepirate.org/~testdashboard/ci?user=snitzer&branch=snitm-nfs-next
https://github.com/koverstreet/ktest
- Various kdevops testing (in terms of "Chuck's BuildBot") has been
performed to regularly verify the LOCALIO changes haven't caused any
regressions to non-LOCALIO NFS use cases.
- All of Hammerspace's various sanity tests pass with LOCALIO enabled
(this includes numerous pNFS and flexfiles tests).
요약·해설
localio.rst:1-357NFS LOCALIO는 기존 NFS connection의 auxiliary RPC로 client와 server가 같은 Linux host인지 single-use UUID를 통해 검증하고, local인 경우 open/read/write/commit에서 SUNRPC와 XDR을 우회합니다. IP address 비교와 달리 iptables·container namespace에서도 locality를 견고하게 판별합니다.
직접 I/O 경로는 `nfs_uuid_t`, per-CPU `nfsd_net_ref`, RCU와 명시적 get/put으로 container별 NFSD network namespace의 수명을 보호합니다. Security는 기존 `AUTH_UNIX`/`AUTH_SYS`, `auth_domain`, export ID translation을 재사용하며 end-to-end O_DIRECT는 별도 parameter로 opt-in합니다.
Discovery, resource interlock, direct I/O와 recovery를 연결합니다.