relay interface (formerly relayfs)

Relay channel 개요

Relay 인터페이스는 커널 애플리케이션이 사용자가 정의한 relay channel을 통해 많은 데이터를 효율적으로 기록하고 userspace로 전송하게 한다. 과거 이름은 relayfs였다.

Relay channel은 CPU마다 하나씩 있는 커널 channel buffer 집합으로 구현한 kernel-to-user 데이터 전달 장치다. 각 channel buffer는 userspace에서 일반 파일인 relay file 하나로 보인다.

커널 클라이언트가 효율적인 쓰기 함수로 기록하면 현재 CPU의 channel buffer가 자동 선택된다. Userspace는 relay file을 `mmap()`하거나 `read()`해 데이터가 준비되는 대로 가져온다. Relay file은 debugfs 같은 host filesystem에 생성되고 아래 API로 channel buffer와 연결한다.

Channel buffer에 기록하는 데이터 형식은 전적으로 커널 클라이언트가 정한다. Relay는 클라이언트가 buffer 데이터에 구조를 부여할 hook을 제공하지만 filtering은 구현하지 않는다. 단순성을 유지하기 위해 필터링도 커널 클라이언트의 책임이다.

이 문서는 relay API의 개요이며 각 함수 인자의 상세 계약은 relay 인터페이스 소스의 함수 문서를 참고해야 한다.

.. SPDX-License-Identifier: GPL-2.0

==================================
relay interface (formerly relayfs)
==================================

The relay interface provides a means for kernel applications to
efficiently log and transfer large quantities of data from the kernel
to userspace via user-defined 'relay channels'.

A 'relay channel' is a kernel->user data relay mechanism implemented
as a set of per-cpu kernel buffers ('channel buffers'), each
represented as a regular file ('relay file') in user space.  Kernel
clients write into the channel buffers using efficient write
functions; these automatically log into the current cpu's channel
buffer.  User space applications mmap() or read() from the relay files
and retrieve the data as it becomes available.  The relay files
themselves are files created in a host filesystem, e.g. debugfs, and
are associated with the channel buffers using the API described below.

The format of the data logged into the channel buffers is completely
up to the kernel client; the relay interface does however provide
hooks which allow kernel clients to impose some structure on the
buffer data.  The relay interface doesn't implement any form of data
filtering - this also is left to the kernel client.  The purpose is to
keep things as simple as possible.

This document provides an overview of the relay interface API.  The
details of the function parameters are documented along with the
functions in the relay interface code - please see that for details.

Sub-buffer 전환과 소비 의미

각 relay channel은 CPU마다 buffer 하나를 가지며 각 buffer는 하나 이상의 sub-buffer로 나뉜다. 첫 sub-buffer가 새 메시지 전체를 담을 수 없을 만큼 차면 다음 사용 가능한 sub-buffer에 쓴다. 메시지는 sub-buffer 경계를 가로질러 나뉘지 않는다.

전환 시 userspace에 첫 sub-buffer를 비우라고 알릴 수 있고 커널은 다음 sub-buffer에 계속 쓴다. 가득 찬 sub-buffer에는 완전한 메시지가 들어가지 않아 남은 padding byte 수가 알려지므로 userspace는 유효 데이터만 복사할 수 있다. 복사 뒤에는 해당 sub-buffer를 소비했다고 커널에 알릴 수 있다.

채널은 userspace가 아직 수집하지 않은 데이터를 기다리지 않고 덮어쓰는 overwrite 모드로도 동작할 수 있다.

Relay channel 자체는 padding·소비 상태를 kernel과 userspace 사이에 전달하는 단일 통신 규약을 강제하지 않는다. 커널 쪽을 단순하게 유지하기 위해 이 통신은 클라이언트가 정하고, 별도 helper와 예제를 제공한다.

`read()` 인터페이스는 padding을 제거하고 읽은 sub-buffer를 내부적으로 소비 처리한다. 따라서 `read(2)`로 channel buffer를 비우는 기본 동작에는 별도의 kernel-userspace 상태 통신이 필요 없다.

Relay의 주요 목표는 낮은 overhead다. `read()`는 사용하기 쉽지만 `mmap()` 방식만큼 효율적이지 않으며 예제 코드는 둘의 성능 차이를 줄이려 한다.

Semantics
=========

Each relay channel has one buffer per CPU; each buffer has one or more
sub-buffers.  Messages are written to the first sub-buffer until it is
too full to contain a new message, in which case it is written to
the next (if available).  Messages are never split across sub-buffers.
At this point, userspace can be notified so it empties the first
sub-buffer, while the kernel continues writing to the next.

When notified that a sub-buffer is full, the kernel knows how many
bytes of it are padding, i.e., unused space occurring because a complete
message couldn't fit into a sub-buffer.  Userspace can use this
knowledge to copy only valid data.

After copying it, userspace can notify the kernel that a sub-buffer
has been consumed.

A relay channel can operate in a mode where it will overwrite data not
yet collected by userspace, and not wait for it to be consumed.

The relay channel itself does not provide for communication of such
data between userspace and kernel, allowing the kernel side to remain
simple and not impose a single interface on userspace.  It does
provide a set of examples and a separate helper though, described
below.

The read() interface both removes padding and internally consumes the
read sub-buffers; thus in cases where read(2) is being used to drain
the channel buffers, special-purpose communication between kernel and
user isn't necessary for basic operation.

One of the major goals of the relay interface is to provide a low
overhead mechanism for conveying kernel data to userspace.  While the
read() interface is easy to use, it's not as efficient as the mmap()
approach; the example code attempts to make the tradeoff between the
two approaches as small as possible.

klog와 relay-apps 예제

Relay 인터페이스는 바로 사용할 수 있지만, 단순한 utility 함수와 예제 집합도 제공한다. Relay SourceForge 사이트의 relay-apps tarball에는 userspace와 kernel 쪽 boilerplate C 파일 한 쌍으로 된 자체 완결 예제가 들어 있다. 두 파일을 조합하면 반복적인 상태 관리 코드를 직접 쓰지 않고 데이터를 디스크로 스트리밍할 수 있다.

Relay-apps의 `klog.patch`가 제공하는 klog debugging 함수는 대상 channel이 존재하는지, relay가 커널에 컴파일됐는지와 무관하게 formatted text나 raw data를 쓰는 고수준 API다. 커널이나 모듈 어디에나 무조건 trace 문을 둘 수 있고 `klog handler`가 등록된 경우에만 실제로 기록한다. 자세한 사용은 klog와 kleak 예제를 참고한다.

예제나 klog 없이 relay를 직접 사용할 수도 있다. 이 경우 buffer의 full·empty·padding 양을 양쪽에 전달하는 kernel-userspace 통신을 구현해야 한다.

`read()`는 padding 제거와 sub-buffer 소비를 내부 처리하므로 기본 배출에는 전용 통신이 필요 없지만 buffer-full 같은 상태는 여전히 어떤 channel로 알려야 한다.

Klog와 relay-apps 예제는 `http://relayfs.sourceforge.net`의 relay-apps tarball에서 찾을 수 있다.

klog and relay-apps example code
================================

The relay interface itself is ready to use, but to make things easier,
a couple of simple utility functions and a set of examples are provided.

The relay-apps example tarball, available on the relay sourceforge
site, contains a set of self-contained examples, each consisting of a
pair of .c files containing boilerplate code for each of the user and
kernel sides of a relay application.  When combined these two sets of
boilerplate code provide glue to easily stream data to disk, without
having to bother with mundane housekeeping chores.

The 'klog debugging functions' patch (klog.patch in the relay-apps
tarball) provides a couple of high-level logging functions to the
kernel which allow writing formatted text or raw data to a channel,
regardless of whether a channel to write into exists or not, or even
whether the relay interface is compiled into the kernel or not.  These
functions allow you to put unconditional 'trace' statements anywhere
in the kernel or kernel modules; only when there is a 'klog handler'
registered will data actually be logged (see the klog and kleak
examples for details).

It is of course possible to use the relay interface from scratch,
i.e., without using any of the relay-apps example code or klog, but
you'll have to implement communication between userspace and kernel,
allowing both to convey the state of buffers (full, empty, amount of
padding).  The read() interface both removes padding and internally
consumes the read sub-buffers; thus in cases where read(2) is being
used to drain the channel buffers, special-purpose communication
between kernel and user isn't necessary for basic operation.  Things
such as buffer-full conditions would still need to be communicated via
some channel though.

klog and the relay-apps examples can be found in the relay-apps
tarball on http://relayfs.sourceforge.net

Userspace 파일 연산 API

Relay 인터페이스는 userspace가 channel buffer 데이터에 접근할 기본 파일 연산을 구현한다.

기본 no-overwrite 모드에서는 active kernel writer가 있어도 언제든 `read()`할 수 있다. Overwrite 모드에서 writer가 활동 중이면 결과를 예측할 수 없으므로 모든 channel logging이 끝난 뒤 읽어야 한다.

Userspace에서 relay file을 사용하려면 host filesystem을 마운트해야 한다. Debugfs를 쓰는 대표 명령은 다음과 같다.

mount -t debugfs debugfs /sys/kernel/debug

커널 클라이언트가 channel을 생성하거나 사용하는 데 host filesystem 마운트가 필요한 것은 아니다. Userspace가 buffer 데이터에 접근할 때만 마운트돼 있어야 한다.

The relay interface user space API
==================================

The relay interface implements basic file operations for user space
access to relay channel buffer data.  Here are the file operations
that are available and some comments regarding their behavior:

=========== ============================================================
open()            enables user to open an _existing_ channel buffer.

mmap()      results in channel buffer being mapped into the caller's
            memory space. Note that you can't do a partial mmap - you
            must map the entire file, which is NRBUF * SUBBUFSIZE.

read()      read the contents of a channel buffer.  The bytes read are
            'consumed' by the reader, i.e., they won't be available
            again to subsequent reads.  If the channel is being used
            in no-overwrite mode (the default), it can be read at any
            time even if there's an active kernel writer.  If the
            channel is being used in overwrite mode and there are
            active channel writers, results may be unpredictable -
            users should make sure that all logging to the channel has
            ended before using read() with overwrite mode.  Sub-buffer
            padding is automatically removed and will not be seen by
            the reader.

sendfile()  transfer data from a channel buffer to an output file
            descriptor. Sub-buffer padding is automatically removed
            and will not be seen by the reader.

poll()      POLLIN/POLLRDNORM/POLLERR supported.  User applications are
            notified when sub-buffer boundaries are crossed.

close()     decrements the channel buffer's refcount.  When the refcount
            reaches 0, i.e., when no process or kernel client has the
            buffer open, the channel buffer is freed.
=========== ============================================================

In order for a user application to make use of relay files, the
host filesystem must be mounted.  For example::

        mount -t debugfs debugfs /sys/kernel/debug

.. Note::

        The host filesystem doesn't need to be mounted for kernel
        clients to create or use channels - it only needs to be
        mounted when user space applications need access to the buffer
        data.

커널 API 분류

In-kernel 클라이언트가 사용하는 relay API는 channel 관리, userspace 소비 통지, 쓰기, callback, helper의 다섯 묶음으로 나뉜다.

`relay_open()`은 이름·부모 dentry·sub-buffer 크기와 개수·callback 집합·private data를 받아 channel을 만든다. `relay_close()`는 닫고, `relay_flush()`는 마지막 데이터 처리를 위해 전환을 강제하며, `relay_reset()`은 할당과 mapping을 유지한 채 초기 상태로 되돌린다.

`relay_subbufs_consumed()`는 userspace가 CPU별로 소비한 sub-buffer 수를 알린다. 쓰기 API는 현재 CPU buffer를 사용하며 callback은 경계 전환과 파일 수명, mapping 수명을 다룬다.

원문의 `TBD(curr. line MT:/API/)` 표시는 API 상세 문서 연결이 미완성이던 상태를 그대로 나타낸다.

The relay interface kernel API
==============================

Here's a summary of the API the relay interface provides to in-kernel clients:

TBD(curr. line MT:/API/)
  channel management functions::

    relay_open(base_filename, parent, subbuf_size, n_subbufs,
               callbacks, private_data)
    relay_close(chan)
    relay_flush(chan)
    relay_reset(chan)

  channel management typically called on instigation of userspace::

    relay_subbufs_consumed(chan, cpu, subbufs_consumed)

  write functions::

    relay_write(chan, data, length)
    __relay_write(chan, data, length)
    relay_reserve(chan, length)

  callbacks::

    subbuf_start(buf, subbuf, prev_subbuf, prev_padding)
    buf_mapped(buf, filp)
    buf_unmapped(buf, filp)
    create_buf_file(filename, parent, mode, buf, is_global)
    remove_buf_file(dentry)

  helper functions::

    relay_buf_full(buf)
    subbuf_start_reserve(buf, length)

Channel 생성과 파일 callback

`relay_open()`은 channel과 CPU별 channel buffer를 만든다. 각 buffer에는 host filesystem 파일이 연결되며 userspace가 mmap하거나 읽을 수 있다. 온라인 CPU 수가 `N`이면 기본 파일명은 `basename0`부터 `basenameN-1`이다.

`parent`가 `NULL`이면 host filesystem 루트에 파일을 만든다. 별도 디렉터리가 필요하면 `debugfs_create_dir()` 같은 host filesystem 함수를 호출해 부모를 만들고 `relay_open()`에 넘긴다. Channel을 닫을 때 클라이언트가 `debugfs_remove()` 같은 함수로 자신이 만든 디렉터리 구조를 정리해야 한다.

Channel 생성에는 `create_buf_file()`과 `remove_buf_file()` callback이 필수다. 전자는 `relay_open()`이 CPU별 buffer마다 호출해 표시 파일을 만들고 해당 dentry를 반환한다. 후자는 `relay_close()` 중 전자가 만든 파일을 삭제한다.

static struct dentry *create_buf_file_handler(const char *filename,
                                        struct dentry *parent,
                                        umode_t mode,
                                        struct rchan_buf *buf,
                                        int *is_global)
{
        return debugfs_create_file(filename, mode, parent, buf,
                                   &relay_file_operations);
}

static int remove_buf_file_handler(struct dentry *dentry)
{
        debugfs_remove(dentry);
        return 0;
}

static struct rchan_callbacks relay_callbacks = {
        .create_buf_file = create_buf_file_handler,
        .remove_buf_file = remove_buf_file_handler,
};

chan = relay_open("cpu", NULL, SUBBUF_SIZE, N_SUBBUFS, &relay_callbacks, NULL);

`create_buf_file()` callback이 실패하거나 정의되지 않으면 channel 생성과 `relay_open()`이 실패한다.

CPU별 buffer 전체 크기는 `n_subbufs * subbuf_size`다. Sub-buffer는 double buffering을 N개 buffer로 확장한 구조이며, 높은 처리량 애플리케이션이 buffer 경계 단위 random access를 구현하는 데도 유용하다. 적절한 개수와 크기는 workload에 따라 실험으로 정해야 한다.

Sub-buffer 하나만 두는 것은 좋지 않다. Channel 모드에 따라 기존 데이터를 덮어쓰거나 새 event를 잃는 일이 보장되기 때문이다.

`create_buf_file()`은 기본 CPU별 집합 대신 global buffer 하나를 만들 수도 있다. 시스템 전체 event의 상대 순서를 보고 싶고 CPU별 파일을 timestamp로 병합·정렬하고 싶지 않은 애플리케이션에 유용하다. 이 경우 callback은 파일을 만들면서 `is_global` 출력 인자를 0이 아닌 값으로 설정한다.

Global buffer에서는 `create_buf_file()`과 `remove_buf_file()`이 한 번씩만 호출되고 보통의 `relay_write()`도 사용할 수 있다. 모든 CPU 쓰기가 같은 buffer로 들어가므로 호출자가 spinlock으로 감싸거나 `relay.h` 쓰기 함수를 복사해 내부 locking을 추가한 로컬 함수를 만들어야 한다.

`relay_open()`의 `private_data`는 channel과 사용자 정의 데이터를 연결한다. `create_buf_file()` 실행 시점부터 `chan->private_data` 또는 `buf->chan->private_data`로 즉시 접근할 수 있다.

Creating a channel
------------------

relay_open() is used to create a channel, along with its per-cpu
channel buffers.  Each channel buffer will have an associated file
created for it in the host filesystem, which can be and mmapped or
read from in user space.  The files are named basename0...basenameN-1
where N is the number of online cpus, and by default will be created
in the root of the filesystem (if the parent param is NULL).  If you
want a directory structure to contain your relay files, you should
create it using the host filesystem's directory creation function,
e.g. debugfs_create_dir(), and pass the parent directory to
relay_open().  Users are responsible for cleaning up any directory
structure they create, when the channel is closed - again the host
filesystem's directory removal functions should be used for that,
e.g. debugfs_remove().

In order for a channel to be created and the host filesystem's files
associated with its channel buffers, the user must provide definitions
for two callback functions, create_buf_file() and remove_buf_file().
create_buf_file() is called once for each per-cpu buffer from
relay_open() and allows the user to create the file which will be used
to represent the corresponding channel buffer.  The callback should
return the dentry of the file created to represent the channel buffer.
remove_buf_file() must also be defined; it's responsible for deleting
the file(s) created in create_buf_file() and is called during
relay_close().

Here are some typical definitions for these callbacks, in this case
using debugfs::

    /*
    * create_buf_file() callback.  Creates relay file in debugfs.
    */
    static struct dentry *create_buf_file_handler(const char *filename,
                                                struct dentry *parent,
                                                umode_t mode,
                                                struct rchan_buf *buf,
                                                int *is_global)
    {
            return debugfs_create_file(filename, mode, parent, buf,
                                    &relay_file_operations);
    }

    /*
    * remove_buf_file() callback.  Removes relay file from debugfs.
    */
    static int remove_buf_file_handler(struct dentry *dentry)
    {
            debugfs_remove(dentry);

            return 0;
    }

    /*
    * relay interface callbacks
    */
    static struct rchan_callbacks relay_callbacks =
    {
            .create_buf_file = create_buf_file_handler,
            .remove_buf_file = remove_buf_file_handler,
    };

And an example relay_open() invocation using them::

  chan = relay_open("cpu", NULL, SUBBUF_SIZE, N_SUBBUFS, &relay_callbacks, NULL);

If the create_buf_file() callback fails, or isn't defined, channel
creation and thus relay_open() will fail.

The total size of each per-cpu buffer is calculated by multiplying the
number of sub-buffers by the sub-buffer size passed into relay_open().
The idea behind sub-buffers is that they're basically an extension of
double-buffering to N buffers, and they also allow applications to
easily implement random-access-on-buffer-boundary schemes, which can
be important for some high-volume applications.  The number and size
of sub-buffers is completely dependent on the application and even for
the same application, different conditions will warrant different
values for these parameters at different times.  Typically, the right
values to use are best decided after some experimentation; in general,
though, it's safe to assume that having only 1 sub-buffer is a bad
idea - you're guaranteed to either overwrite data or lose events
depending on the channel mode being used.

The create_buf_file() implementation can also be defined in such a way
as to allow the creation of a single 'global' buffer instead of the
default per-cpu set.  This can be useful for applications interested
mainly in seeing the relative ordering of system-wide events without
the need to bother with saving explicit timestamps for the purpose of
merging/sorting per-cpu files in a postprocessing step.

To have relay_open() create a global buffer, the create_buf_file()
implementation should set the value of the is_global outparam to a
non-zero value in addition to creating the file that will be used to
represent the single buffer.  In the case of a global buffer,
create_buf_file() and remove_buf_file() will be called only once.  The
normal channel-writing functions, e.g. relay_write(), can still be
used - writes from any cpu will transparently end up in the global
buffer - but since it is a global buffer, callers should make sure
they use the proper locking for such a buffer, either by wrapping
writes in a spinlock, or by copying a write function from relay.h and
creating a local version that internally does the proper locking.

The private_data passed into relay_open() allows clients to associate
user-defined data with a channel, and is immediately available
(including in create_buf_file()) via chan->private_data or
buf->chan->private_data.

Overwrite·no-overwrite 모드

Relay channel의 overwrite와 no-overwrite 모드는 `subbuf_start()` callback 구현으로 완전히 결정된다. Callback을 정의하지 않으면 기본은 no-overwrite다. 기본 모드와 `read()`를 사용한다면 이 절의 mmap 중심 세부사항을 대부분 신경 쓰지 않아도 된다.

Overwrite, 즉 flight recorder 모드에서는 쓰기가 원형 buffer를 계속 순환해 실패하지 않지만 소비되지 않은 옛 데이터도 무조건 덮어쓴다. No-overwrite 모드에서는 미소비 sub-buffer 수가 전체 개수와 같으면 쓰기가 실패해 새 데이터가 유실된다.

소비자가 없거나 충분히 빠르지 않으면 어느 모드든 데이터가 유실된다. 차이는 buffer의 시작 쪽 옛 데이터를 잃는지, 끝 쪽 새 event를 잃는지다.

현재 메시지가 sub-buffer에 맞지 않으면 새 sub-buffer 전환 직전에 `subbuf_start()`를 호출한다. 클라이언트는 다음 sub-buffer 초기화, 이전 sub-buffer 마무리, 실제 전환 허용 여부의 boolean 반환을 수행한다.

static int subbuf_start(struct rchan_buf *buf,
                        void *subbuf,
                        void *prev_subbuf,
                        unsigned int prev_padding)
{
        if (prev_subbuf)
                *((unsigned *)prev_subbuf) = prev_padding;

        if (relay_buf_full(buf))
                return 0;

        subbuf_start_reserve(buf, sizeof(unsigned int));
        return 1;
}

No-overwrite callback은 이전 sub-buffer가 있으면 그 header에 `prev_padding`을 기록한다. 모든 sub-buffer가 미소비 상태면 `relay_buf_full()`이 참이므로 0을 반환해 전환을 중지한다. 소비자가 `relay_subbufs_consumed()`로 하나 이상 소비했다고 알린 뒤 재시도할 때만 전환이 진행된다.

쓰기가 다시 시도되면 같은 인자로 `subbuf_start()`가 재호출된다. 준비된 sub-buffer가 소비되면 `relay_buf_full()`이 0을 반환하고 callback이 1을 반환할 수 있다.

static int subbuf_start(struct rchan_buf *buf,
                        void *subbuf,
                        void *prev_subbuf,
                        size_t prev_padding)
{
        if (prev_subbuf)
                *((unsigned *)prev_subbuf) = prev_padding;

        subbuf_start_reserve(buf, sizeof(unsigned int));
        return 1;
}

Overwrite callback에서는 full 검사가 의미 없고 항상 1을 반환해 무조건 전환한다. 이 모드는 소비 상태를 참조하지 않으므로 `relay_subbufs_consumed()`를 호출할 의미도 없다.

클라이언트 callback이 전혀 없거나 `subbuf_start()`만 없을 때 쓰는 기본 구현은 가장 단순한 no-overwrite 모드로, 아무 작업 없이 0만 반환한다.

`subbuf_start_reserve()`는 각 sub-buffer 시작에 header 공간을 예약한다. 예제에서는 padding count를 저장할 `unsigned int` 공간을 예약하고, 값이 sub-buffer를 채운 뒤에야 확정되므로 다음 전환 callback에서 `prev_subbuf`에 기록한다.

Channel을 열 때 첫 sub-buffer에도 callback을 호출해 header를 예약할 기회를 준다. 이때 이전 sub-buffer가 없으므로 `prev_subbuf == NULL`이며, 이전 buffer에 쓰기 전 반드시 포인터를 검사해야 한다.

Channel 'modes'
---------------

relay channels can be used in either of two modes - 'overwrite' or
'no-overwrite'.  The mode is entirely determined by the implementation
of the subbuf_start() callback, as described below.  The default if no
subbuf_start() callback is defined is 'no-overwrite' mode.  If the
default mode suits your needs, and you plan to use the read()
interface to retrieve channel data, you can ignore the details of this
section, as it pertains mainly to mmap() implementations.

In 'overwrite' mode, also known as 'flight recorder' mode, writes
continuously cycle around the buffer and will never fail, but will
unconditionally overwrite old data regardless of whether it's actually
been consumed.  In no-overwrite mode, writes will fail, i.e., data will
be lost, if the number of unconsumed sub-buffers equals the total
number of sub-buffers in the channel.  It should be clear that if
there is no consumer or if the consumer can't consume sub-buffers fast
enough, data will be lost in either case; the only difference is
whether data is lost from the beginning or the end of a buffer.

As explained above, a relay channel is made of up one or more
per-cpu channel buffers, each implemented as a circular buffer
subdivided into one or more sub-buffers.  Messages are written into
the current sub-buffer of the channel's current per-cpu buffer via the
write functions described below.  Whenever a message can't fit into
the current sub-buffer, because there's no room left for it, the
client is notified via the subbuf_start() callback that a switch to a
new sub-buffer is about to occur.  The client uses this callback to 1)
initialize the next sub-buffer if appropriate 2) finalize the previous
sub-buffer if appropriate and 3) return a boolean value indicating
whether or not to actually move on to the next sub-buffer.

To implement 'no-overwrite' mode, the userspace client provides
an implementation of the subbuf_start() callback something like the
following::

    static int subbuf_start(struct rchan_buf *buf,
                            void *subbuf,
                            void *prev_subbuf,
                            unsigned int prev_padding)
    {
            if (prev_subbuf)
                    *((unsigned *)prev_subbuf) = prev_padding;

            if (relay_buf_full(buf))
                    return 0;

            subbuf_start_reserve(buf, sizeof(unsigned int));

            return 1;
    }

If the current buffer is full, i.e., all sub-buffers remain unconsumed,
the callback returns 0 to indicate that the buffer switch should not
occur yet, i.e., until the consumer has had a chance to read the
current set of ready sub-buffers.  For the relay_buf_full() function
to make sense, the consumer is responsible for notifying the relay
interface when sub-buffers have been consumed via
relay_subbufs_consumed().  Any subsequent attempts to write into the
buffer will again invoke the subbuf_start() callback with the same
parameters; only when the consumer has consumed one or more of the
ready sub-buffers will relay_buf_full() return 0, in which case the
buffer switch can continue.

The implementation of the subbuf_start() callback for 'overwrite' mode
would be very similar::

    static int subbuf_start(struct rchan_buf *buf,
                            void *subbuf,
                            void *prev_subbuf,
                            size_t prev_padding)
    {
            if (prev_subbuf)
                    *((unsigned *)prev_subbuf) = prev_padding;

            subbuf_start_reserve(buf, sizeof(unsigned int));

            return 1;
    }

In this case, the relay_buf_full() check is meaningless and the
callback always returns 1, causing the buffer switch to occur
unconditionally.  It's also meaningless for the client to use the
relay_subbufs_consumed() function in this mode, as it's never
consulted.

The default subbuf_start() implementation, used if the client doesn't
define any callbacks, or doesn't define the subbuf_start() callback,
implements the simplest possible 'no-overwrite' mode, i.e., it does
nothing but return 0.

Header information can be reserved at the beginning of each sub-buffer
by calling the subbuf_start_reserve() helper function from within the
subbuf_start() callback.  This reserved area can be used to store
whatever information the client wants.  In the example above, room is
reserved in each sub-buffer to store the padding count for that
sub-buffer.  This is filled in for the previous sub-buffer in the
subbuf_start() implementation; the padding value for the previous
sub-buffer is passed into the subbuf_start() callback along with a
pointer to the previous sub-buffer, since the padding value isn't
known until a sub-buffer is filled.  The subbuf_start() callback is
also called for the first sub-buffer when the channel is opened, to
give the client a chance to reserve space in it.  In this case the
previous sub-buffer pointer passed into the callback will be NULL, so
the client should check the value of the prev_subbuf pointer before
writing into the previous sub-buffer.

쓰기·예약·종료 API

커널 클라이언트는 `relay_write()` 또는 `__relay_write()`로 현재 CPU의 channel buffer에 쓴다. `relay_write()`는 `local_irqsave()`로 buffer를 보호하므로 interrupt context에서 기록할 가능성이 있으면 사용한다. Interrupt context에서 절대 쓰지 않는다면 preemption만 비활성화하는 `__relay_write()`를 사용할 수 있다.

두 함수는 반환값이 없다. 빠른 logging 경로에서 실패값을 검사하지 않는다는 설계이며, no-overwrite 모드에서 buffer가 가득 찬 경우를 제외하면 항상 성공한다고 가정한다. 이 실패는 `subbuf_start()`에서 `relay_buf_full()`을 호출해 감지할 수 있다.

`relay_reserve()`는 나중에 직접 쓸 channel buffer slot을 예약한다. 임시 buffer에 데이터를 staging하지 않고 바로 기록하려는 애플리케이션에 적합하다. 예약 직후 실제 쓰기가 일어나지 않을 수 있으므로, sub-buffer 예약 영역이나 별도 배열에 실제로 쓴 byte 수를 기록할 수 있다.

예약과 쓰기가 분리되고 실제 쓰기가 클라이언트 통제 아래 있으므로 `relay_reserve()`는 buffer를 전혀 보호하지 않는다. 적절한 동기화는 클라이언트 책임이다. Relay-apps tarball의 `reserve` 예제가 구현 방식을 보여 준다.

사용을 마치면 `relay_close()`를 호출한다. 어느 channel buffer에도 참조가 남지 않을 때 channel과 관련 buffer가 파괴된다. 닫기 전 `relay_flush()`로 모든 channel buffer의 sub-buffer 전환을 강제하면 마지막 sub-buffer를 마무리하고 처리할 수 있다.

Writing to a channel
--------------------

Kernel clients write data into the current cpu's channel buffer using
relay_write() or __relay_write().  relay_write() is the main logging
function - it uses local_irqsave() to protect the buffer and should be
used if you might be logging from interrupt context.  If you know
you'll never be logging from interrupt context, you can use
__relay_write(), which only disables preemption.  These functions
don't return a value, so you can't determine whether or not they
failed - the assumption is that you wouldn't want to check a return
value in the fast logging path anyway, and that they'll always succeed
unless the buffer is full and no-overwrite mode is being used, in
which case you can detect a failed write in the subbuf_start()
callback by calling the relay_buf_full() helper function.

relay_reserve() is used to reserve a slot in a channel buffer which
can be written to later.  This would typically be used in applications
that need to write directly into a channel buffer without having to
stage data in a temporary buffer beforehand.  Because the actual write
may not happen immediately after the slot is reserved, applications
using relay_reserve() can keep a count of the number of bytes actually
written, either in space reserved in the sub-buffers themselves or as
a separate array.  See the 'reserve' example in the relay-apps tarball
at http://relayfs.sourceforge.net for an example of how this can be
done.  Because the write is under control of the client and is
separated from the reserve, relay_reserve() doesn't protect the buffer
at all - it's up to the client to provide the appropriate
synchronization when using relay_reserve().

Closing a channel
-----------------

The client calls relay_close() when it's finished using the channel.
The channel and its associated buffers are destroyed when there are no
longer any references to any of the channel buffers.  relay_flush()
forces a sub-buffer switch on all the channel buffers, and can be used
to finalize and process the last sub-buffers before the channel is
closed.

Reset·mapping callback·자료와 기여자

애플리케이션이 사용할 때마다 channel을 열고 닫지 않고 재사용하려면 `relay_reset()`을 쓸 수 있다. Channel buffer 메모리를 다시 할당하거나 기존 mapping을 파괴하지 않고 초기 상태로 되돌린다. 현재 channel에 쓰는 중이 아닐 때만 호출해야 안전하다.

`buf_mapped()`는 userspace가 channel buffer를 mmap할 때, `buf_unmapped()`는 mapping을 해제할 때 호출된다. 커널 클라이언트는 이 알림을 channel logging 활성화·비활성화 같은 동작의 trigger로 사용할 수 있다.

뉴스·예제 코드·메일링 리스트 등 자료는 relay 인터페이스 홈페이지 `http://relayfs.sourceforge.net`에서 제공한다.

Relay의 아이디어와 명세는 tracing 논의에서 나왔다. 원문은 Michel Dagenais, Richard Moore, Bob Wisniewski, Karim Yaghmour, Tom Zanussi를 기여자로 기록하며, 유용한 제안과 bug report를 제공한 Hubertus Franke에게도 감사를 전한다.

Misc
----

Some applications may want to keep a channel around and re-use it
rather than open and close a new channel for each use.  relay_reset()
can be used for this purpose - it resets a channel to its initial
state without reallocating channel buffer memory or destroying
existing mappings.  It should however only be called when it's safe to
do so, i.e., when the channel isn't currently being written to.

Finally, there are a couple of utility callbacks that can be used for
different purposes.  buf_mapped() is called whenever a channel buffer
is mmapped from user space and buf_unmapped() is called when it's
unmapped.  The client can use this notification to trigger actions
within the kernel application, such as enabling/disabling logging to
the channel.


Resources
=========

For news, example code, mailing list, etc. see the relay interface homepage:

    http://relayfs.sourceforge.net


Credits
=======

The ideas and specs for the relay interface came about as a result of
discussions on tracing involving the following:

Michel Dagenais                <michel.dagenais@polymtl.ca>
Richard Moore                <richardj_moore@uk.ibm.com>
Bob Wisniewski                <bob@watson.ibm.com>
Karim Yaghmour                <karim@opersys.com>
Tom Zanussi                <zanussi@us.ibm.com>

Also thanks to Hubertus Franke for a lot of useful suggestions and bug
reports.