Management Component Transport Protocol (MCTP)

Protocol과 interface·network topology

`net/mctp/`는 DMTF 표준 DSP0236의 MCTP protocol을 구현하고, specification에서 binding이라 부르는 물리 interface driver는 `drivers/net/mctp/`에 둡니다. Core는 `AF_MCTP`, `SOCK_DGRAM` socket을 통해 MCTP message를 보내고 받는 interface를 제공합니다.

Kernel은 local MCTP topology를 interface와 network로 모델링합니다. Interface 또는 link는 특정 hardware device에 연결되는 MCTP physical transport binding의 instance이며 `struct netdevice`로 나타냅니다. Network는 EID(endpoint ID)로 endpoint를 식별하는 고유 address space이고 userspace에서 참조할 identifier를 가지며 route도 network별로 정의됩니다.

각 interface는 하나의 network에 속하지만 한 network에는 하나 이상의 interface가 연결될 수 있습니다. 여러 network가 존재하면 서로 다른 network가 같은 EID를 포함해도 충돌하지 않습니다.

.. SPDX-License-Identifier: GPL-2.0

==============================================
Management Component Transport Protocol (MCTP)
==============================================

net/mctp/ contains protocol support for MCTP, as defined by DMTF standard
DSP0236. Physical interface drivers ("bindings" in the specification) are
provided in drivers/net/mctp/.

The core code provides a socket-based interface to send and receive MCTP
messages, through an AF_MCTP, SOCK_DGRAM socket.

Structure: interfaces & networks
================================

The kernel models the local MCTP topology through two items: interfaces and
networks.

An interface (or "link") is an instance of an MCTP physical transport binding
(as defined by DSP0236, section 3.2.47), likely connected to a specific hardware
device. This is represented as a ``struct netdevice``.

A network defines a unique address space for MCTP endpoints by endpoint-ID
(described by DSP0236, section 3.2.31). A network has a user-visible identifier
to allow references from userspace. Route definitions are specific to one
network.

Interfaces are associated with one network. A network may be associated with one
or more interfaces.

If multiple networks are present, each may contain endpoint IDs (EIDs) that are
also present on other networks.

Socket family와 address 구조

MCTP는 address family와 protocol family로 각각 `AF_MCTP`와 `PF_MCTP`를 사용합니다. Message 기반 protocol이므로 `SOCK_DGRAM`만 지원하며 현재 `socket(AF_MCTP, SOCK_DGRAM, protocol)`의 `protocol`에는 0만 사용할 수 있습니다.

Endpoint address는 한 byte인 `mctp_eid_t`이고 `struct mctp_addr.s_addr`에 저장됩니다. `struct sockaddr_mctp`는 family, network identifier, endpoint address, message type, tag를 담습니다. `MCTP_NET_ANY`는 모든 network를 뜻하는 0이고 `MCTP_ADDR_ANY`는 모든 local EID를 뜻하는 `0xff`입니다.

Sockets API
===========

Protocol definitions
--------------------

MCTP uses ``AF_MCTP`` / ``PF_MCTP`` for the address- and protocol- families.
Since MCTP is message-based, only ``SOCK_DGRAM`` sockets are supported.

.. code-block:: C

    int sd = socket(AF_MCTP, SOCK_DGRAM, 0);

The only (current) value for the ``protocol`` argument is 0.

As with all socket address families, source and destination addresses are
specified with a ``sockaddr`` type, with a single-byte endpoint address:

.. code-block:: C

    typedef __u8                mctp_eid_t;

    struct mctp_addr {
            mctp_eid_t                s_addr;
    };

    struct sockaddr_mctp {
            __kernel_sa_family_t smctp_family;
            unsigned int         smctp_network;
            struct mctp_addr     smctp_addr;
            __u8                 smctp_type;
            __u8                 smctp_tag;
    };

    #define MCTP_NET_ANY        0x0
    #define MCTP_ADDR_ANY        0xff


Syscall behaviour
-----------------

The following sections describe the MCTP-specific behaviours of the standard
socket system calls. These behaviours have been chosen to map closely to the
existing sockets APIs.

bind(): local 수신 address

들어오는 request packet을 받을 socket은 `bind()`로 local address를 설정합니다. Kernel은 network, address, message type이 일치하는 incoming MCTP message를 이 socket에 전달합니다. 여기서 incoming은 TO(tag owner) bit가 설정된 request를 뜻하며, bound socket이 response까지 무조건 받는다는 뜻은 아닙니다.

`smctp_tag`에는 remote가 소유한 tag를 받도록 `MCTP_TAG_OWNER`만 지정할 수 있습니다. 이 flag를 쓸 때 최하위 3개 tag value bit는 사용하지 않으므로 caller가 0으로 설정해야 합니다. `smctp_network`가 `MCTP_NET_ANY`이면 locally connected인 모든 network, 특정 값이면 그 network에서 온 message만 받습니다.

`smctp_addr`의 `MCTP_ADDR_ANY`는 모든 local destination EID를 허용합니다. `smctp_type` 비교에는 하위 7 bit만 사용하고 최상위 IC(integrity check) bit는 제외하므로 integrity check footer가 있는 packet과 없는 packet을 모두 받을 수 있습니다.

``bind()`` : set local socket address
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sockets that receive incoming request packets will bind to a local address,
using the ``bind()`` syscall.

.. code-block:: C

    struct sockaddr_mctp addr;

    addr.smctp_family = AF_MCTP;
    addr.smctp_network = MCTP_NET_ANY;
    addr.smctp_addr.s_addr = MCTP_ADDR_ANY;
    addr.smctp_type = MCTP_TYPE_PLDM;
    addr.smctp_tag = MCTP_TAG_OWNER;

    int rc = bind(sd, (struct sockaddr *)&addr, sizeof(addr));

This establishes the local address of the socket. Incoming MCTP messages that
match the network, address, and message type will be received by this socket.
The reference to 'incoming' is important here; a bound socket will only receive
messages with the TO bit set, to indicate an incoming request message, rather
than a response.

The ``smctp_tag`` value will configure the tags accepted from the remote side of
this socket. Given the above, the only valid value is ``MCTP_TAG_OWNER``, which
will result in remotely "owned" tags being routed to this socket. Since
``MCTP_TAG_OWNER`` is set, the 3 least-significant bits of ``smctp_tag`` are not
used; callers must set them to zero.

A ``smctp_network`` value of ``MCTP_NET_ANY`` will configure the socket to
receive incoming packets from any locally-connected network. A specific network
value will cause the socket to only receive incoming messages from that network.

The ``smctp_addr`` field specifies a local address to bind to. A value of
``MCTP_ADDR_ANY`` configures the socket to receive messages addressed to any
local destination EID.

The ``smctp_type`` field specifies which message types to receive. Only the
lower 7 bits of the type is matched on incoming messages (ie., the
most-significant IC bit is not part of the match). This results in the socket
receiving packets with and without a message integrity check footer.

sendto·sendmsg·send(): message 송신

`sendto()`, `sendmsg()`, `send()`로 MCTP message를 전송합니다. Destination address의 network와 EID가 remote 목적지를 정합니다. `smctp_tag`에 `MCTP_TAG_OWNER`가 있으면 kernel은 `MCTP_TAG_VALUE` bit를 무시하고 destination EID에 알맞은 tag를 생성합니다. Owner bit가 없으면 지정한 tag value를 그대로 사용하며 tag를 할당할 수 없으면 `EAGAIN`을 반환합니다.

Application은 송신 buffer의 첫 byte에 message type을 직접 넣어야 합니다. Integrity check가 필요하면 footer도 buffer에 포함하고 type byte의 최상위 bit를 1로 설정합니다. `sendmsg()`는 scatter-gather list를 사용할 수 있지만 현재 `msg_control`에 넣을 ancillary message type은 정의되어 있지 않습니다.

연결되지 않은 socket에서 `MCTP_TAG_OWNER`로 보낼 때 destination용 유효 tag가 없으면 새 tag를 할당합니다. `(destination-eid, tag)` tuple은 outgoing request의 response를 받기 위한 암시적 local socket address가 됩니다. 다른 remote EID에 대한 이전 할당은 사라집니다. Socket은 자신이 TO=1로 보낸 request의 response만 받고, 자신이 받은 request에는 TO=0으로만 응답할 수 있습니다.

``sendto()``, ``sendmsg()``, ``send()`` : transmit an MCTP message
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

An MCTP message is transmitted using one of the ``sendto()``, ``sendmsg()`` or
``send()`` syscalls. Using ``sendto()`` as the primary example:

.. code-block:: C

    struct sockaddr_mctp addr;
    char buf[14];
    ssize_t len;

    /* set message destination */
    addr.smctp_family = AF_MCTP;
    addr.smctp_network = 0;
    addr.smctp_addr.s_addr = 8;
    addr.smctp_tag = MCTP_TAG_OWNER;
    addr.smctp_type = MCTP_TYPE_ECHO;

    /* arbitrary message to send, with message-type header */
    buf[0] = MCTP_TYPE_ECHO;
    memcpy(buf + 1, "hello, world!", sizeof(buf) - 1);

    len = sendto(sd, buf, sizeof(buf), 0,
                    (struct sockaddr_mctp *)&addr, sizeof(addr));

The network and address fields of ``addr`` define the remote address to send to.
If ``smctp_tag`` has the ``MCTP_TAG_OWNER``, the kernel will ignore any bits set
in ``MCTP_TAG_VALUE``, and generate a tag value suitable for the destination
EID. If ``MCTP_TAG_OWNER`` is not set, the message will be sent with the tag
value as specified. If a tag value cannot be allocated, the system call will
report an errno of ``EAGAIN``.

The application must provide the message type byte as the first byte of the
message buffer passed to ``sendto()``. If a message integrity check is to be
included in the transmitted message, it must also be provided in the message
buffer, and the most-significant bit of the message type byte must be 1.

The ``sendmsg()`` system call allows a more compact argument interface, and the
message buffer to be specified as a scatter-gather list. At present no ancillary
message types (used for the ``msg_control`` data passed to ``sendmsg()``) are
defined.

Transmitting a message on an unconnected socket with ``MCTP_TAG_OWNER``
specified will cause an allocation of a tag, if no valid tag is already
allocated for that destination. The (destination-eid,tag) tuple acts as an
implicit local socket address, to allow the socket to receive responses to this
outgoing message. If any previous allocation has been performed (to for a
different remote EID), that allocation is lost.

Sockets will only receive responses to requests they have sent (with TO=1) and
may only respond (with TO=0) to requests they have received.

recvfrom·recvmsg·recv(): message 수신

Application은 `recvfrom()`, `recvmsg()`, `recv()`로 MCTP message를 받습니다. `recvfrom()`과 `recvmsg()`의 address argument에는 incoming message의 remote EID와 reply에 필요한 tag value를 포함한 remote address가 채워집니다.

수신 buffer의 첫 byte는 message type이며 integrity check가 뒤따르면 그것도 buffer에 포함됩니다. `recv()`는 비슷하게 동작하지만 remote address를 돌려주지 않으므로 상대 address를 이미 알거나 reply가 필요 없는 경우에만 유용합니다. 송신과 마찬가지로 socket은 자신이 TO=1로 보낸 request의 response만 받을 수 있고, 수신한 request에는 TO=0으로 응답합니다.

``recvfrom()``, ``recvmsg()``, ``recv()`` : receive an MCTP message
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

An MCTP message can be received by an application using one of the
``recvfrom()``, ``recvmsg()``, or ``recv()`` system calls. Using ``recvfrom()``
as the primary example:

.. code-block:: C

    struct sockaddr_mctp addr;
    socklen_t addrlen;
    char buf[14];
    ssize_t len;

    addrlen = sizeof(addr);

    len = recvfrom(sd, buf, sizeof(buf), 0,
                    (struct sockaddr_mctp *)&addr, &addrlen);

    /* We can expect addr to describe an MCTP address */
    assert(addrlen >= sizeof(buf));
    assert(addr.smctp_family == AF_MCTP);

    printf("received %zd bytes from remote EID %d\n", rc, addr.smctp_addr);

The address argument to ``recvfrom`` and ``recvmsg`` is populated with the
remote address of the incoming message, including tag value (this will be needed
in order to reply to the message).

The first byte of the message buffer will contain the message type byte. If an
integrity check follows the message, it will be included in the received buffer.

The ``recv()`` system call behaves in a similar way, but does not provide a
remote address to the application. Therefore, these are only useful if the
remote address is already known, or the message does not require a reply.

Like the send calls, sockets will only receive responses to requests they have
sent (TO=1) and may only respond (TO=0) to requests they have received.

Tag 명시 할당과 해제 ioctl

`SIOCMCTPALLOCTAG`와 `SIOCMCTPDROPTAG`는 `sendmsg()`마다 kernel이 자동 할당하는 방식 대신 application이 MCTP tag를 명시적으로 할당하고 해제하게 합니다. 여러 request에 같은 tag를 유지하거나 하나의 request가 여러 response를 만드는 등 일반적인 request/response model에 맞지 않을 때 사용합니다.

두 ioctl은 `peer_addr`, `tag`, `flags`를 가진 `struct mctp_ioc_tag_ctl` pointer를 받습니다. `SIOCMCTPALLOCTAG` 호출 전에는 remote EID인 `peer_addr`만 채우고 나머지는 0으로 둡니다. 성공하면 `tag`에 `MCTP_TAG_OWNER`, preallocated임을 나타내는 `MCTP_TAG_PREALLOC`, 그리고 하위 3 bit의 실제 `MCTP_TAG_MASK` 값이 함께 반환됩니다. 실제 값 0도 유효하며 반환된 값을 `sockaddr_mctp.smctp_tag`에 그대로 사용해야 합니다.

`SIOCMCTPDROPTAG`는 앞서 할당한 tag를 해제합니다. `peer_addr`는 할당 때와 같아야 하고 `tag`도 OWNER와 PREALLOC bit를 포함한 반환값 전체가 정확히 일치해야 하며 `flags`는 0이어야 합니다.

``ioctl(SIOCMCTPALLOCTAG)`` and ``ioctl(SIOCMCTPDROPTAG)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These tags give applications more control over MCTP message tags, by allocating
(and dropping) tag values explicitly, rather than the kernel automatically
allocating a per-message tag at ``sendmsg()`` time.

In general, you will only need to use these ioctls if your MCTP protocol does
not fit the usual request/response model. For example, if you need to persist
tags across multiple requests, or a request may generate more than one response.
In these cases, the ioctls allow you to decouple the tag allocation (and
release) from individual message send and receive operations.

Both ioctls are passed a pointer to a ``struct mctp_ioc_tag_ctl``:

.. code-block:: C

    struct mctp_ioc_tag_ctl {
        mctp_eid_t      peer_addr;
        __u8                tag;
        __u16           flags;
    };

``SIOCMCTPALLOCTAG`` allocates a tag for a specific peer, which an application
can use in future ``sendmsg()`` calls. The application populates the
``peer_addr`` member with the remote EID. Other fields must be zero.

On return, the ``tag`` member will be populated with the allocated tag value.
The allocated tag will have the following tag bits set:

 - ``MCTP_TAG_OWNER``: it only makes sense to allocate tags if you're the tag
   owner

 - ``MCTP_TAG_PREALLOC``: to indicate to ``sendmsg()`` that this is a
   preallocated tag.

 - ... and the actual tag value, within the least-significant three bits
   (``MCTP_TAG_MASK``). Note that zero is a valid tag value.

The tag value should be used as-is for the ``smctp_tag`` member of ``struct
sockaddr_mctp``.

``SIOCMCTPDROPTAG`` releases a tag that has been previously allocated by a
``SIOCMCTPALLOCTAG`` ioctl. The ``peer_addr`` must be the same as used for the
allocation, and the ``tag`` value must match exactly the tag returned from the
allocation (including the ``MCTP_TAG_OWNER`` and ``MCTP_TAG_PREALLOC`` bits).
The ``flags`` field must be zero.

Kernel packet 흐름과 key reference

Local 송신 message가 MTU 이하면 `mctp_local_output()`의 route lookup 뒤 `mctp_route_output`과 `dev_queue_xmit()`로 바로 나갑니다. MTU를 넘으면 `mctp_do_fragment_route()`가 packet 크기의 skb를 만들고 각 fragment를 같은 route output 경로로 전송합니다.

Remote에서 local endpoint로 들어온 단일 packet message는 `mctp_pkttype_receive()`에서 route와 `sk_key`를 찾은 뒤 `sock_queue_rcv_skb()`로 socket에 넣습니다. 여러 packet message는 첫 fragment를 `sk_key->reasm_head`에 저장하고 이후 fragment를 기존 reassembly에 이어 붙인 뒤 완성된 skb를 socket에 전달합니다.

Key reference는 route output 중 `skb->cb`에 저장된 skb와 net namespace 및 socket list가 보유합니다. Key가 device와 연결되면 `key->dev`를 통해 device reference를 잡고 `dev->key_count`로 계산하며 여러 key가 같은 device를 참조할 수 있습니다.

Kernel internals
================

There are a few possible packet flows in the MCTP stack:

1. local TX to remote endpoint, message <= MTU::

        sendmsg()
         -> mctp_local_output()
            : route lookup
            -> rt->output() (== mctp_route_output)
               -> dev_queue_xmit()

2. local TX to remote endpoint, message > MTU::

        sendmsg()
        -> mctp_local_output()
            -> mctp_do_fragment_route()
               : creates packet-sized skbs. For each new skb:
               -> rt->output() (== mctp_route_output)
                  -> dev_queue_xmit()

3. remote TX to local endpoint, single-packet message::

        mctp_pkttype_receive()
        : route lookup
        -> rt->output() (== mctp_route_input)
           : sk_key lookup
           -> sock_queue_rcv_skb()

4. remote TX to local endpoint, multiple-packet message::

        mctp_pkttype_receive()
        : route lookup
        -> rt->output() (== mctp_route_input)
           : sk_key lookup
           : stores skb in struct sk_key->reasm_head

        mctp_pkttype_receive()
        : route lookup
        -> rt->output() (== mctp_route_input)
           : sk_key lookup
           : finds existing reassembly in sk_key->reasm_head
           : appends new fragment
           -> sock_queue_rcv_skb()

Key refcounts
-------------

 * keys are refed by:

   - a skb: during route output, stored in ``skb->cb``.

   - netns and sock lists.

 * keys can be associated with a device, in which case they hold a
   reference to the dev (set through ``key->dev``, counted through
   ``dev->key_count``). Multiple keys can reference the device.