MSG_ZEROCOPY

기회, 제약과 참고 자료

`MSG_ZEROCOPY` flag는 socket send call에서 data copy를 피하도록 요청하며 현재 TCP, UDP와 virtio transport를 사용하는 VSOCK에 구현되어 있습니다. 큰 buffer를 userspace와 kernel 사이에서 복사하는 비용을 줄이고 `sendfile`, `splice`가 쓰는 copy avoidance 개념을 일반적인 socket send API로 확장합니다.

Copy avoidance도 비용이 없습니다. 현재 구현은 byte별 copy 대신 page pinning의 accounting과 completion notification 비용을 부담하므로 대체로 약 10KB보다 큰 write에서만 효과적입니다. Page pinning 동안 process와 network stack이 buffer를 공유하므로 syscall 반환 직후 process가 buffer를 덮어쓰면 전송 중인 자신의 data stream을 손상시킬 수 있습니다. Kernel integrity에는 영향이 없습니다.

Kernel은 data를 다시 수정해도 안전해진 시점에 notification을 돌려줍니다. 따라서 기존 application을 바꿀 때 단순히 flag 하나만 추가해서는 안 됩니다. 더 깊은 설명은 netdev 2.1 paper·slide·video, LWN article과 최초 patchset에서 확인할 수 있습니다.

============
MSG_ZEROCOPY
============

Intro
=====

The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
The feature is currently implemented for TCP, UDP and VSOCK (with
virtio transport) sockets.


Opportunity and Caveats
-----------------------

Copying large buffers between user process and kernel can be
expensive. Linux supports various interfaces that eschew copying,
such as sendfile and splice. The MSG_ZEROCOPY flag extends the
underlying copy avoidance mechanism to common socket send calls.

Copy avoidance is not a free lunch. As implemented, with page pinning,
it replaces per byte copy cost with page accounting and completion
notification overhead. As a result, MSG_ZEROCOPY is generally only
effective at writes over around 10 KB.

Page pinning also changes system call semantics. It temporarily shares
the buffer between process and network stack. Unlike with copying, the
process cannot immediately overwrite the buffer after system call
return without possibly modifying the data in flight. Kernel integrity
is not affected, but a buggy program can possibly corrupt its own data
stream.

The kernel returns a notification when it is safe to modify data.
Converting an existing application to MSG_ZEROCOPY is not always as
trivial as just passing the flag, then.


More Info
---------

Much of this document was derived from a longer paper presented at
netdev 2.1. For more in-depth information see that paper and talk,
the excellent reporting over at LWN.net or read the original code.

  paper, slides, video
    https://netdevconf.org/2.1/session.html?debruijn

  LWN article
    https://lwn.net/Articles/726917/

  patchset
    [PATCH net-next v4 0/9] socket sendmsg MSG_ZEROCOPY
    https://lore.kernel.org/netdev/20170803202945.70750-1-willemdebruijn.kernel@gmail.com

Socket 설정, 송신과 copy 혼용

Send syscall은 정의되지 않은 flag를 기본적으로 무시하므로 legacy process가 우연히 같은 bit를 쓰더라도 동작이 바뀌지 않게 먼저 `setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, ...)`로 zerocopy 의사를 명시해야 합니다. 그 뒤 `send`, `sendto`, `sendmsg`, `sendmmsg`에 `MSG_ZEROCOPY`를 전달합니다.

Socket의 optmem limit을 넘거나 user의 locked-page ulimit을 초과하면 syscall이 -1과 `ENOBUFS`를 반환합니다. 작은 packet에서는 copy avoidance가 오히려 비싸므로 flag가 있는 call과 없는 call을 같은 socket에서 안전하게 섞을 수 있습니다. Application은 buffer 크기에 따라 copy와 zerocopy를 선택하면 됩니다.

Interface
=========

Passing the MSG_ZEROCOPY flag is the most obvious step to enable copy
avoidance, but not the only one.

Socket Setup
------------

The kernel is permissive when applications pass undefined flags to the
send system call. By default it simply ignores these. To avoid enabling
copy avoidance mode for legacy processes that accidentally already pass
this flag, a process must first signal intent by setting a socket option:

::

        if (setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one)))
                error(1, errno, "setsockopt zerocopy");

Transmission
------------

The change to send (or sendto, sendmsg, sendmmsg) itself is trivial.
Pass the new flag.

::

        ret = send(fd, buf, sizeof(buf), MSG_ZEROCOPY);

A zerocopy failure will return -1 with errno ENOBUFS. This happens if
the socket exceeds its optmem limit or the user exceeds their ulimit on
locked pages.


Mixing copy avoidance and copying
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Many workloads have a mixture of large and small buffers. Because copy
avoidance is more expensive than copying for small packets, the
feature is implemented as a flag. It is safe to mix calls with the flag
with those without.

Completion notification 수신

Kernel은 이전에 넘긴 buffer를 안전하게 재사용할 수 있을 때 transmit timestamping과 비슷하게 socket error queue에 completion notification을 넣습니다. 각 socket은 unsigned 32-bit counter를 유지하며 data를 성공적으로 보낸 각 `MSG_ZEROCOPY` syscall마다 1 증가합니다. 실패하거나 길이가 0이면 증가하지 않고 byte가 아니라 syscall 횟수를 세며 `UINT_MAX` 다음에는 wrap됩니다.

가장 단순한 수신 방식은 `poll()`로 `POLLERR`를 기다린 뒤 `recvmsg(..., MSG_ERRQUEUE)`로 error queue를 읽는 것입니다. Error queue read 자체는 항상 non-blocking이고 error는 `events`에 `POLLERR`를 넣지 않아도 무조건 signal됩니다. 실제 application은 매 send마다 기다리기보다 몇 번 송신한 뒤 non-blocking으로 확인하는 편이 효율적입니다.

Notification은 socket의 다른 operation과 순서가 뒤바뀔 수 있습니다. 일반 error가 queue에 있으면 읽을 때까지 다른 operation을 막지만 zerocopy notification은 error code가 0이어서 send와 recv를 차단하지 않습니다.

Notifications
-------------

The kernel has to notify the process when it is safe to reuse a
previously passed buffer. It queues completion notifications on the
socket error queue, akin to the transmit timestamping interface.

The notification itself is a simple scalar value. Each socket
maintains an internal unsigned 32-bit counter. Each send call with
MSG_ZEROCOPY that successfully sends data increments the counter. The
counter is not incremented on failure or if called with length zero.
The counter counts system call invocations, not bytes. It wraps after
UINT_MAX calls.


Notification Reception
~~~~~~~~~~~~~~~~~~~~~~

The below snippet demonstrates the API. In the simplest case, each
send syscall is followed by a poll and recvmsg on the error queue.

Reading from the error queue is always a non-blocking operation. The
poll call is there to block until an error is outstanding. It will set
POLLERR in its output flags. That flag does not have to be set in the
events field. Errors are signaled unconditionally.

::

        pfd.fd = fd;
        pfd.events = 0;
        if (poll(&pfd, 1, -1) != 1 || pfd.revents & POLLERR == 0)
                error(1, errno, "poll");

        ret = recvmsg(fd, &msg, MSG_ERRQUEUE);
        if (ret == -1)
                error(1, errno, "recvmsg");

        read_notification(msg);

The example is for demonstration purpose only. In practice, it is more
efficient to not wait for notifications, but read without blocking
every couple of send calls.

Notifications can be processed out of order with other operations on
the socket. A socket that has an error queued would normally block
other operations until the error is read. Zerocopy notifications have
a zero error code, however, to not block send and recv calls.

Notification batching과 parsing

여러 outstanding packet의 notification은 `recvmmsg()`로 한 번에 읽을 수 있지만 kernel이 연속 notification을 range로 합치므로 꼭 필요하지는 않습니다. 새 value가 error queue tail notification의 range를 연장하면 새 notification packet을 버리고 기존 range의 upper value만 늘립니다. TCP처럼 in-order acknowledgement인 protocol은 보통 하나의 outstanding notification으로 계속 합칠 수 있습니다. 다만 retransmission이나 socket teardown에서는 out-of-order notification이 가능합니다.

Notification은 표준 `sock_extended_err` control message로 encoding됩니다. TCP·UDP는 protocol family에 따라 `IP_RECVERR` 또는 `IPV6_RECVERR`, VSOCK은 `SOL_VSOCK` level과 `VSOCK_RECVERR` type을 사용합니다. `ee_origin`은 `SO_EE_ORIGIN_ZEROCOPY`, `ee_errno`는 다른 socket I/O를 막지 않도록 0입니다.

32-bit inclusive completion range는 `[ee_info, ee_data]`에 들어갑니다. 아래에서 설명하는 `ee_code`를 제외한 나머지 `sock_extended_err` field는 정의되지 않은 값으로 취급해야 합니다.

Notification Batching
~~~~~~~~~~~~~~~~~~~~~

Multiple outstanding packets can be read at once using the recvmmsg
call. This is often not needed. In each message the kernel returns not
a single value, but a range. It coalesces consecutive notifications
while one is outstanding for reception on the error queue.

When a new notification is about to be queued, it checks whether the
new value extends the range of the notification at the tail of the
queue. If so, it drops the new notification packet and instead increases
the range upper value of the outstanding notification.

For protocols that acknowledge data in-order, like TCP, each
notification can be squashed into the previous one, so that no more
than one notification is outstanding at any one point.

Ordered delivery is the common case, but not guaranteed. Notifications
may arrive out of order on retransmission and socket teardown.


Notification Parsing
~~~~~~~~~~~~~~~~~~~~

The below snippet demonstrates how to parse the control message: the
read_notification() call in the previous snippet. A notification
is encoded in the standard error format, sock_extended_err.

The level and type fields in the control data are protocol family
specific, IP_RECVERR or IPV6_RECVERR (for TCP or UDP socket).
For VSOCK socket, cmsg_level will be SOL_VSOCK and cmsg_type will be
VSOCK_RECVERR.

Error origin is the new type SO_EE_ORIGIN_ZEROCOPY. ee_errno is zero,
as explained before, to avoid blocking read and write system calls on
the socket.

The 32-bit notification range is encoded as [ee_info, ee_data]. This
range is inclusive. Other fields in the struct must be treated as
undefined, bar for ee_code, as discussed below.

::

        struct sock_extended_err *serr;
        struct cmsghdr *cm;

        cm = CMSG_FIRSTHDR(msg);
        if (cm->cmsg_level != SOL_IP &&
            cm->cmsg_type != IP_RECVERR)
                error(1, 0, "cmsg");

        serr = (void *) CMSG_DATA(cm);
        if (serr->ee_errno != 0 ||
            serr->ee_origin != SO_EE_ORIGIN_ZEROCOPY)
                error(1, 0, "serr");

        printf("completed: %u..%u\n", serr->ee_info, serr->ee_data);

Deferred copy와 completion 의미

`MSG_ZEROCOPY`는 kernel에 copy avoidance를 요청하는 hint이자 completion notification을 반드시 queue하라는 계약이지 실제 copy가 항상 생략된다는 보장은 아닙니다. Scatter-gather I/O를 지원하지 않는 device는 kernel protocol header와 zerocopy user data로 구성된 packet을 보낼 수 없고, stack 깊은 곳에서 checksum 계산 등을 위해 private copy로 바꿀 수도 있습니다.

이 경우에도 kernel은 shared page hold를 놓을 때 notification을 보내며 copied data의 실제 송신이 끝나기 전에 올 수 있습니다. 따라서 zerocopy completion은 transmit completion이 아닙니다. Cache가 식은 뒤의 deferred copy와 불필요한 notification 처리는 즉시 copy보다 비쌀 수 있습니다.

Kernel은 copy로 완료한 경우 `ee_code`에 `SO_EE_CODE_ZEROCOPY_COPIED`를 설정합니다. Application은 이 signal을 보고 같은 socket의 이후 request에서 `MSG_ZEROCOPY` 사용을 중단할 수 있습니다.

Deferred copies
~~~~~~~~~~~~~~~

Passing flag MSG_ZEROCOPY is a hint to the kernel to apply copy
avoidance, and a contract that the kernel will queue a completion
notification. It is not a guarantee that the copy is elided.

Copy avoidance is not always feasible. Devices that do not support
scatter-gather I/O cannot send packets made up of kernel generated
protocol headers plus zerocopy user data. A packet may need to be
converted to a private copy of data deep in the stack, say to compute
a checksum.

In all these cases, the kernel returns a completion notification when
it releases its hold on the shared pages. That notification may arrive
before the (copied) data is fully transmitted. A zerocopy completion
notification is not a transmit completion notification, therefore.

Deferred copies can be more expensive than a copy immediately in the
system call, if the data is no longer warm in the cache. The process
also incurs notification processing cost for no benefit. For this
reason, the kernel signals if data was completed with a copy, by
setting flag SO_EE_CODE_ZEROCOPY_COPIED in field ee_code on return.
A process may use this signal to stop passing flag MSG_ZEROCOPY on
subsequent requests on the same socket.

Loopback 구현과 test

TCP와 UDP에서 local socket으로 보낸 data는 receiver가 읽지 않으면 무기한 queue될 수 있어 notification latency가 제한되지 않습니다. 이를 피하려고 local socket, packet socket(tcpdump 등), tun device로 loop되는 모든 `MSG_ZEROCOPY` packet은 deferred copy를 수행합니다. VSOCK은 local과 non-local socket의 data path가 같습니다.

현실적인 TCP·UDP 예제는 `tools/testing/selftests/net/msg_zerocopy.c`에 있습니다. 두 host 사이에서 시험해야 하며, namespace를 가로지르는 veth처럼 local process pair에서 실행하면 loopback 제약 때문에 성능 향상이 나타나지 않습니다. Test 목적으로는 `skb_orphan_frags_rx`를 `skb_orphan_frags`와 같게 만들어 제약을 잠시 완화할 수 있습니다. VSOCK 예제는 `tools/testing/vsock/vsock_test_zerocopy.c`입니다.

Implementation
==============

Loopback
--------

For TCP and UDP:
Data sent to local sockets can be queued indefinitely if the receive
process does not read its socket. Unbound notification latency is not
acceptable. For this reason all packets generated with MSG_ZEROCOPY
that are looped to a local socket will incur a deferred copy. This
includes looping onto packet sockets (e.g., tcpdump) and tun devices.

For VSOCK:
Data path sent to local sockets is the same as for non-local sockets.

Testing
=======

More realistic example code can be found in the kernel source under
tools/testing/selftests/net/msg_zerocopy.c.

Be cognizant of the loopback constraint. The test can be run between
a pair of hosts. But if run between a local pair of processes, for
instance when run with msg_zerocopy.sh between a veth pair across
namespaces, the test will not show any improvement. For testing, the
loopback restriction can be temporarily relaxed by making
skb_orphan_frags_rx identical to skb_orphan_frags.

For VSOCK type of socket example can be found in
tools/testing/vsock/vsock_test_zerocopy.c.