TCP Authentication Option Linux 구현 (RFC 5925)

TCP-AO 목적과 MAC

이 문서는 GPL-2.0 라이선스를 따르며 Linux의 TCP Authentication Option(TCP-AO, RFC 5925) 구현을 설명합니다.

TCP-AO는 신뢰하는 peer 사이에서 TCP segment를 검증하는 TCP extension입니다. 양쪽 peer가 아는 password와 hashing function으로 TCP segment 내용의 Message Authentication Code(MAC)를 만들고, 이를 새 TCP header option에 넣습니다.

TCP-AO는 TCP-MD5를 대체하는 것이 목적이며 더 나은 보안, key rotation, 여러 hashing algorithm 지원을 제공합니다.

.. SPDX-License-Identifier: GPL-2.0

========================================================
TCP Authentication Option Linux implementation (RFC5925)
========================================================

TCP Authentication Option (TCP-AO) provides a TCP extension aimed at verifying
segments between trusted peers. It adds a new TCP header option with
a Message Authentication Code (MAC). MACs are produced from the content
of a TCP segment using a hashing function with a password known to both peers.
The intent of TCP-AO is to deprecate TCP-MD5 providing better security,
key rotation and support for a variety of hashing algorithms.

TCP-MD5와 TCP-AO 비교

TCP-MD5와 TCP-AO의 제한적인 간단 비교는 다음과 같습니다.

1. Introduction
===============

.. table:: Short and Limited Comparison of TCP-AO and TCP-MD5

 +----------------------+------------------------+-----------------------+
 |                      |       TCP-MD5          |         TCP-AO        |
 +======================+========================+=======================+
 |Supported hashing     |MD5                     |Must support HMAC-SHA1 |
 |algorithms            |(cryptographically weak)|(chosen-prefix attacks)|
 |                      |                        |and CMAC-AES-128 (only |
 |                      |                        |side-channel attacks). |
 |                      |                        |May support any hashing|
 |                      |                        |algorithm.             |
 +----------------------+------------------------+-----------------------+
 |Length of MACs (bytes)|16                      |Typically 12-16.       |
 |                      |                        |Other variants that fit|
 |                      |                        |TCP header permitted.  |
 +----------------------+------------------------+-----------------------+
 |Number of keys per    |1                       |Many                   |
 |TCP connection        |                        |                       |
 +----------------------+------------------------+-----------------------+
 |Possibility to change |Non-practical (both     |Supported by protocol  |
 |an active key         |peers have to change    |                       |
 |                      |them during MSL)        |                       |
 +----------------------+------------------------+-----------------------+
 |Protection against    |No                      |Yes: ignoring them     |
 |ICMP 'hard errors'    |                        |by default on          |
 |                      |                        |established connections|
 +----------------------+------------------------+-----------------------+
 |Protection against    |No                      |Yes: pseudo-header     |
 |traffic-crossing      |                        |includes TCP ports.    |
 |attack                |                        |                       |
 +----------------------+------------------------+-----------------------+
 |Protection against    |No                      |Sequence Number        |
 |replayed TCP segments |                        |Extension (SNE) and    |
 |                      |                        |Initial Sequence       |
 |                      |                        |Numbers (ISNs)         |
 +----------------------+------------------------+-----------------------+
 |Supports              |Yes                     |No. ISNs+SNE are needed|
 |Connectionless Resets |                        |to correctly sign RST. |
 +----------------------+------------------------+-----------------------+
 |Standards             |RFC 2385                |RFC 5925, RFC 5926     |
 +----------------------+------------------------+-----------------------+

FAQ: ID 고유성, MKT 수명과 강제 삭제

질문: 같은 4-tuple `(srcaddr, srcport, dstaddr, dstport)`에서 SendID 또는 RecvID가 중복될 수 있습니까? 답: 안 됩니다. RFC 5925 3.1에 따르면 TCP connection identifier가 겹치는 MKT의 ID는 서로 겹치면 안 됩니다.

질문: 활성 connection의 Master Key Tuple(MKT)을 제거할 수 있습니까? 답: MKT가 TCB에 복사된 경우가 아니라면 안 됩니다. 특정 connection에 영향을 주는 MKT는 connection이 활성인 동안 파괴할 수 없다고 가정합니다. 동등한 방식으로 parameter를 connection local 영역에 instance화하면 이후 변경은 새 connection에만 영향을 줍니다.

질문: 활성 connection이 나중에도 old MKT를 사용할 수 있을 때 어떻게 삭제해야 합니까? 답: RFC 5925는 이를 지정하지 않으며, 제거 전에 아무도 그 MKT를 사용하지 않도록 보장하는 것은 key management의 책임입니다.

질문: old MKT가 영원히 남아 다른 peer가 사용할 수 있습니까? 답: 가능합니다. RFC 5925 6.1에 따르면 key를 언제 사용하기 시작할지는 성능 문제이고 MKT를 언제 제거할지는 보안 문제입니다. 유효하지 않은 MKT는 제거해야 하지만 TCP-AO는 제거를 조정하는 mechanism을 제공하지 않습니다. 이전 MKT 재사용을 피하는 유일한 방법은 더 이상 허용되지 않을 때 제거하는 것입니다.

Linux TCP-AO는 사용 중인 key 제거를 key management 실패로 보고 가능한 한 막습니다. 그러나 오래된 key 자체가 보안 문제가 될 수 있고 peer가 계속 `RNextKeyID`로 요청해 삭제를 방해할 수도 있으므로 강제 삭제 mechanism을 제공합니다.

강제 삭제 때 userspace는 제거할 key 대신 사용할 KeyID를 제공하고 kernel은 peer가 old key를 계속 요청하더라도 old key 삭제와 새 key 지정을 atomic하게 수행합니다. Peer가 아직 새 key를 가지지 않았을 수 있으므로 성공적인 통신은 보장되지 않고 TCP connection이 끊길 수 있습니다. 대안은 socket을 종료하는 것입니다.

1.1 Frequently Asked Questions (FAQ) with references to RFC 5925
----------------------------------------------------------------

Q: Can either SendID or RecvID be non-unique for the same 4-tuple
(srcaddr, srcport, dstaddr, dstport)?

A: No [3.1]::

   >> The IDs of MKTs MUST NOT overlap where their TCP connection
   identifiers overlap.

Q: Can Master Key Tuple (MKT) for an active connection be removed?

A: No, unless it's copied to Transport Control Block (TCB) [3.1]::

   It is presumed that an MKT affecting a particular connection cannot
   be destroyed during an active connection -- or, equivalently, that
   its parameters are copied to an area local to the connection (i.e.,
   instantiated) and so changes would affect only new connections.

Q: If an old MKT needs to be deleted, how should it be done in order
to not remove it for an active connection? (As it can be still in use
at any moment later)

A: Not specified by RFC 5925, seems to be a problem for key management
to ensure that no one uses such MKT before trying to remove it.

Q: Can an old MKT exist forever and be used by another peer?

A: It can, it's a key management task to decide when to remove an old key [6.1]::

   Deciding when to start using a key is a performance issue. Deciding
   when to remove an MKT is a security issue. Invalid MKTs are expected
   to be removed. TCP-AO provides no mechanism to coordinate their removal,
   as we consider this a key management operation.

also [6.1]::

   The only way to avoid reuse of previously used MKTs is to remove the MKT
   when it is no longer considered permitted.

Linux TCP-AO will try its best to prevent you from removing a key that's
being used, considering it a key management failure. But since keeping
an outdated key may become a security issue and as a peer may
unintentionally prevent the removal of an old key by always setting
it as RNextKeyID - a forced key removal mechanism is provided, where
userspace has to supply KeyID to use instead of the one that's being removed
and the kernel will atomically delete the old key, even if the peer is
still requesting it. There are no guarantees for force-delete as the peer
may yet not have the new key - the TCP connection may just break.
Alternatively, one may choose to shut down the socket.

FAQ: 알 수 없는 key와 RNextKeyID

질문: 새 connection에서 알려진 MKT의 RecvID와 일치하지 않는 packet을 받으면 어떻게 합니까? RFC 5925의 기본값은 warning을 기록하고 받아들이는 것이지만 사용자가 동작을 설정할 수 있습니다.

SYN이면 새 connection의 첫 segment입니다. Segment의 socket pair와 TCP-AO KeyID를 MKT의 TCP connection identifier와 RecvID에 대조해 MKT를 찾습니다. 일치하는 MKT가 없으면 segment에서 TCP-AO를 제거하고 일반 TCP 처리를 계속합니다. 이는 MKT와 일치하지 않는 connection을 조용히 허용한다는 전제입니다.

RFC 5925 7.3은 TCP-AO가 있지만 MKT와 일치하지 않는 segment의 처리 방식을 반드시 설정할 수 있어야 하고, 초기 기본값은 그러한 connection을 조용히 허용하는 것이 바람직하다고 규정합니다. 원치 않으면 해당 connection과 일치하는 MKT를 추가하거나 TCP-AO 필수를 표시하거나, 일치하지 않는 AO option segment를 폐기하도록 설정할 수 있습니다.

10.2.b에 따르면 어떤 MKT와도 일치하지 않는 connection은 TCP-AO가 필요하지 않으며, incoming segment가 TCP-AO option을 포함했다는 이유만으로 폐기하지 않습니다. 단, 어떤 MKT와도 일치하지 않아야 합니다.

Linux 구현은 이 부분에서 RFC와 다릅니다. 현재 알 수 없는 key signature가 있는 TCP-AO segment는 warning을 기록하고 폐기합니다.

질문: RFC가 모든 connection의 key를 동시에 rotate하는 중앙집중식 kernel key management를 요구합니까? 답: 지정하지 않습니다. MKT는 userspace에서 관리할 수 있고, 모든 TCP segment는 TCP connection identifier와 일치하는 MKT 집합을 대상으로 검사해야 합니다.

질문: peer가 요청한 `RNextKeyID`를 모르면 connection을 reset해야 합니까? 답: 아닙니다. 해당 socket pair와 `RNextKeyID`에 맞는 MKT가 준비되지 않았다면 아무 동작도 하지 않습니다. 수신 segment의 `RNextKeyID`는 MKT의 SendID와 일치해야 합니다.

Q: What happens when a packet is received on a new connection with no known
MKT's RecvID?

A: RFC 5925 specifies that by default it is accepted with a warning logged, but
the behaviour can be configured by the user [7.5.1.a]::

   If the segment is a SYN, then this is the first segment of a new
   connection. Find the matching MKT for this segment, using the segment's
   socket pair and its TCP-AO KeyID, matched against the MKT's TCP connection
   identifier and the MKT's RecvID.

      i. If there is no matching MKT, remove TCP-AO from the segment.
         Proceed with further TCP handling of the segment.
         NOTE: this presumes that connections that do not match any MKT
         should be silently accepted, as noted in Section 7.3.

[7.3]::

   >> A TCP-AO implementation MUST allow for configuration of the behavior
   of segments with TCP-AO but that do not match an MKT. The initial default
   of this configuration SHOULD be to silently accept such connections.
   If this is not the desired case, an MKT can be included to match such
   connections, or the connection can indicate that TCP-AO is required.
   Alternately, the configuration can be changed to discard segments with
   the AO option not matching an MKT.

[10.2.b]::

   Connections not matching any MKT do not require TCP-AO. Further, incoming
   segments with TCP-AO are not discarded solely because they include
   the option, provided they do not match any MKT.

Note that Linux TCP-AO implementation differs in this aspect. Currently, TCP-AO
segments with unknown key signatures are discarded with warnings logged.

Q: Does the RFC imply centralized kernel key management in any way?
(i.e. that a key on all connections MUST be rotated at the same time?)

A: Not specified. MKTs can be managed in userspace, the only relevant part to
key changes is [7.3]::

   >> All TCP segments MUST be checked against the set of MKTs for matching
   TCP connection identifiers.

Q: What happens when RNextKeyID requested by a peer is unknown? Should
the connection be reset?

A: It should not, no action needs to be performed [7.5.2.e]::

   ii. If they differ, determine whether the RNextKeyID MKT is ready.

       1. If the MKT corresponding to the segment’s socket pair and RNextKeyID
       is not available, no action is required (RNextKeyID of a received
       segment needs to match the MKT’s SendID).

FAQ: current_key 선택과 보안 방식 전환

질문: `current_key`는 어떻게 설정되고 언제 바뀝니까? `rnext_key`는 수동 userspace 개입이나 MKT management protocol로만 바뀌며 TCP-AO가 조작하지 않습니다. 반면 `current_key`는 수신 TCP segment를 처리할 때 TCP-AO가 `RNextKeyID`에 따라 갱신합니다.

Algorithm은 `current_key`가 새 MKT로 바뀌었다가 이전 MKT로 되돌아가는 ‘backing up’도 허용합니다. MKT 변경 중 segment가 순서 없이 도착할 때 발생할 수 있으며, reordering 때문에 packet을 drop하지 않게 하는 TCP-AO의 기능입니다. 일치하는 `RNextKeyID` MKT가 있으면 `current_key`를 그 MKT로 설정합니다.

질문: 양쪽 peer에 같은 socket pair와 일치하지만 KeyID가 다른 MKT가 여러 개라면 sender/receiver는 어느 KeyID를 고릅니까? 답: 어떤 mechanism이 ‘desired’ MKT 하나를 골라야 합니다. Outgoing segment는 여러 MKT와 일치할 수 있지만 정확히 하나만 desired로 지정되어야 하고 ID가 충돌해서는 안 됩니다. 선택에 다른 segment 정보는 사용할 수 있지만 TCP option field 값은 사용하면 안 됩니다.

질문: TCP-MD5 connection을 TCP-AO로, 또는 그 반대로 migrate할 수 있습니까? 답: 없습니다. TCP-MD5는 established connection의 security algorithm 변경을 지원하지 않습니다.

질문: connection의 모든 MKT를 제거하면 서명하지 않는 non-TCP-AO connection으로 바뀔 수 있습니까? 답: RFC의 established-segment 처리는 SYN 처리처럼 서명 없는 segment를 허용하는 선택지가 없으며, 그렇게 바꾸는 것은 안전하지 않습니다. TCP-AO는 보호되는 모든 TCP segment가 정확히 하나의 MKT와 일치하고 connection에 하나 이상의 MKT와 항상 하나의 `current_key`가 있기를 요구합니다.

질문: non-TCP-AO connection을 TCP-AO-enabled connection으로 바꿀 수 있습니까? 답: established connection에서는 traffic key 생성에 ISN이 필요하므로 불가능합니다. TCP-AO를 시작하려면 TCP connection을 다시 수립해야 합니다.

Q: How is current_key set, and when does it change? Is it a user-triggered
change, or is it triggered by a request from the remote peer? Is it set by the
user explicitly, or by a matching rule?

A: current_key is set by RNextKeyID [6.1]::

   Rnext_key is changed only by manual user intervention or MKT management
   protocol operation. It is not manipulated by TCP-AO. Current_key is updated
   by TCP-AO when processing received TCP segments as discussed in the segment
   processing description in Section 7.5. Note that the algorithm allows
   the current_key to change to a new MKT, then change back to a previously
   used MKT (known as "backing up"). This can occur during an MKT change when
   segments are received out of order, and is considered a feature of TCP-AO,
   because reordering does not result in drops.

[7.5.2.e.ii]::

   2. If the matching MKT corresponding to the segment’s socket pair and
   RNextKeyID is available:

      a. Set current_key to the RNextKeyID MKT.

Q: If both peers have multiple MKTs matching the connection's socket pair
(with different KeyIDs), how should the sender/receiver pick KeyID to use?

A: Some mechanism should pick the "desired" MKT [3.3]::

   Multiple MKTs may match a single outgoing segment, e.g., when MKTs
   are being changed. Those MKTs cannot have conflicting IDs (as noted
   elsewhere), and some mechanism must determine which MKT to use for each
   given outgoing segment.

   >> An outgoing TCP segment MUST match at most one desired MKT, indicated
   by the segment’s socket pair. The segment MAY match multiple MKTs, provided
   that exactly one MKT is indicated as desired. Other information in
   the segment MAY be used to determine the desired MKT when multiple MKTs
   match; such information MUST NOT include values in any TCP option fields.

Q: Can TCP-MD5 connection migrate to TCP-AO (and vice-versa):

A: No [1]::

   TCP MD5-protected connections cannot be migrated to TCP-AO because TCP MD5
   does not support any changes to a connection’s security algorithm
   once established.

Q: If all MKTs are removed on a connection, can it become a non-TCP-AO signed
connection?

A: [7.5.2] doesn't have the same choice as SYN packet handling in [7.5.1.i]
that would allow accepting segments without a sign (which would be insecure).
While switching to non-TCP-AO connection is not prohibited directly, it seems
what the RFC means. Also, there's a requirement for TCP-AO connections to
always have one current_key [3.3]::

   TCP-AO requires that every protected TCP segment match exactly one MKT.

[3.3]::

   >> An incoming TCP segment including TCP-AO MUST match exactly one MKT,
   indicated solely by the segment’s socket pair and its TCP-AO KeyID.

[4.4]::

   One or more MKTs. These are the MKTs that match this connection’s
   socket pair.

Q: Can a non-TCP-AO connection become a TCP-AO-enabled one?

A: No: for an already established non-TCP-AO connection it would be impossible
to switch to using TCP-AO, as the traffic key generation requires the initial
sequence numbers. Paraphrasing, starting using TCP-AO would require
re-establishing the TCP connection.

Kernel MKT database와 userspace 관리 비교

Linux TCP-AO는 TCP-MD5와 비슷하게 `setsockopt()` 방식으로 구현됩니다. Application은 TCP socket에서 MKT 추가·제거·rotation이 필요할 때 `setsockopt()`를 호출합니다.

이 방식은 peer가 `RNextKeyID`를 따르지 않을 때의 정책 같은 corner case 결정과 key management 책임을 userspace로 옮깁니다. Kernel code가 줄고 kernel 전체 database보다 locking이 적어 유연하고 확장성이 좋습니다. 주요 사용자가 임의 application이 아니라 BGP process이며 현대 BGP daemon은 이미 TCP-MD5용 `setsockopt()`를 사용하므로 IPsec tunnel 같은 투명성도 필요하지 않습니다.

2. In-kernel MKTs database vs database in userspace
===================================================

Linux TCP-AO support is implemented using ``setsockopt()s``, in a similar way
to TCP-MD5. It means that a userspace application that wants to use TCP-AO
should perform ``setsockopt()`` on a TCP socket when it wants to add,
remove or rotate MKTs. This approach moves the key management responsibility
to userspace as well as decisions on corner cases, i.e. what to do if
the peer doesn't respect RNextKeyID; moving more code to userspace, especially
responsible for the policy decisions. Besides, it's flexible and scales well
(with less locking needed than in the case of an in-kernel database). One also
should keep in mind that mainly intended users are BGP processes, not any
random applications, which means that compared to IPsec tunnels,
no transparency is really needed and modern BGP daemons already have
``setsockopt()s`` for TCP-MD5 support.

.. table:: Considered pros and cons of the approaches

 +----------------------+------------------------+-----------------------+
 |                      |    ``setsockopt()``    |      in-kernel DB     |
 +======================+========================+=======================+
 | Extendability        | ``setsockopt()``       | Netlink messages are  |
 |                      | commands should be     | simple and extendable |
 |                      | extendable syscalls    |                       |
 +----------------------+------------------------+-----------------------+
 | Required userspace   | BGP or any application | could be transparent  |
 | changes              | that wants TCP-AO needs| as tunnels, providing |
 |                      | to perform             | something like        |
 |                      | ``setsockopt()s``      | ``ip tcpao add key``  |
 |                      | and do key management  | (delete/show/rotate)  |
 +----------------------+------------------------+-----------------------+
 |MKTs removal or adding| harder for userspace   | harder for kernel     |
 +----------------------+------------------------+-----------------------+
 | Dump-ability         | ``getsockopt()``       | Netlink .dump()       |
 |                      |                        | callback              |
 +----------------------+------------------------+-----------------------+
 | Limits on kernel     |                      equal                     |
 | resources/memory     |                                                |
 +----------------------+------------------------+-----------------------+
 | Scalability          | contention on          | contention on         |
 |                      | ``TCP_LISTEN`` sockets | the whole database    |
 +----------------------+------------------------+-----------------------+
 | Monitoring & warnings| ``TCP_DIAG``           | same Netlink socket   |
 +----------------------+------------------------+-----------------------+
 | Matching of MKTs     | half-problem: only     | hard                  |
 |                      | listen sockets         |                       |
 +----------------------+------------------------+-----------------------+

Socket option uAPI와 key operation

Linux는 socket별 TCP-AO를 관리하는 `setsockopt()`와 `getsockopt()` 집합을 제공합니다. MKT 추가와 삭제에는 TCP socket option `TCP_AO_ADD_KEY`와 `TCP_AO_DEL_KEY`를 사용합니다. Established non-TCP-AO connection에 key를 추가하거나 TCP-AO connection의 마지막 key를 제거할 수 없습니다.

`setsockopt(TCP_AO_DEL_KEY)`는 `tcp_ao_del::current_key`와 `tcp_ao_del::set_current`, 또는 `tcp_ao_del::rnext`와 `tcp_ao_del::set_rnext`를 지정해 사용 중인 key를 삭제하면서 대체 key를 atomic하게 설정하는 강제 삭제를 수행할 수 있습니다. 이는 정상 용도가 아니라 peer가 `RNextKeyID`를 무시하며 old key를 계속 요청·사용할 때만 써야 합니다. 신뢰할 수 없는 key를 제거할 수 있지만 connection이 깨질 수 있습니다.

일반적인 key rotation은 `setsockopt(TCP_AO_INFO)`로 수행합니다. 이 option은 ICMP 무시 같은 socket별 TCP-AO 설정 변경과 packet counter 초기화도 제공합니다. 대응하는 `getsockopt(TCP_AO_INFO)`로 현재 설정을 읽습니다.

`getsockopt(TCP_AO_GET_KEYS)`는 TCP socket의 모든 MKT를 나열하거나 peer, sndid/rcvid, VRF L3 interface, `current_key` 또는 `rnext_key`로 filter해 조회합니다.

TCP-AO connection repair에는 먼저 `getsockopt(TCP_AO_REPAIR)`로 socket을 checkpoint/dump한 뒤 `setsockopt(TCP_AO_REPAIR)`를 사용합니다.

3. uAPI
=======

Linux provides a set of ``setsockopt()s`` and ``getsockopt()s`` that let
userspace manage TCP-AO on a per-socket basis. In order to add/delete MKTs
``TCP_AO_ADD_KEY`` and ``TCP_AO_DEL_KEY`` TCP socket options must be used.
It is not allowed to add a key on an established non-TCP-AO connection
as well as to remove the last key from TCP-AO connection.

``setsockopt(TCP_AO_DEL_KEY)`` command may specify ``tcp_ao_del::current_key``
+ ``tcp_ao_del::set_current`` and/or ``tcp_ao_del::rnext``
+ ``tcp_ao_del::set_rnext`` which makes such delete "forced": it
provides userspace a way to delete a key that's being used and atomically set
another one instead. This is not intended for normal use and should be used
only when the peer ignores RNextKeyID and keeps requesting/using an old key.
It provides a way to force-delete a key that's not trusted but may break
the TCP-AO connection.

The usual/normal key-rotation can be performed with ``setsockopt(TCP_AO_INFO)``.
It also provides a uAPI to change per-socket TCP-AO settings, such as
ignoring ICMPs, as well as clear per-socket TCP-AO packet counters.
The corresponding ``getsockopt(TCP_AO_INFO)`` can be used to get those
per-socket TCP-AO settings.

Another useful command is ``getsockopt(TCP_AO_GET_KEYS)``. One can use it
to list all MKTs on a TCP socket or use a filter to get keys for a specific
peer and/or sndid/rcvid, VRF L3 interface or get current_key/rnext_key.

To repair TCP-AO connections ``setsockopt(TCP_AO_REPAIR)`` is available,
provided that the user previously has checkpointed/dumped the socket with
``getsockopt(TCP_AO_REPAIR)``.

대규모 listen, counter, trace와 port matching

수천 개 TCP-AO key를 가질 수 있는 대규모 `TCP_LISTEN` socket에서는 `getsockopt(TCP_AO_GET_KEYS)` filter와 `setsockopt(TCP_AO_DEL_KEY)`의 asynchronous delete를 사용하는 것이 좋습니다.

문제 분석을 위해 각 MKT에는 검증을 통과하거나 실패한 packet 수를 나타내는 good/bad counter가 있습니다. 각 TCP-AO socket은 올바르게 서명된 segment, 검증 실패 segment, 알 수 없는 key의 segment, AO signature가 필요했지만 없던 segment, 무시한 ICMP 수를 셉니다.

Socket별 counter는 SNMP에 공개되는 netns별 counter `TCPAOGood`, `TCPAOBad`, `TCPAOKeyNotFound`, `TCPAORequired`, `TCPAODroppedIcmps`에도 복제됩니다.

Monitoring trace event는 `tcp_hash_bad_header`, `tcp_hash_ao_required`, `tcp_ao_handshake_failure`, `tcp_ao_wrong_maclen`, `tcp_ao_key_not_found`, `tcp_ao_rnext_request`, `tcp_ao_synack_no_key`, `tcp_ao_snd_sne_update`, `tcp_ao_rcv_sne_update`입니다. 각각 따로 활성화할 수 있고 net namespace, 4-tuple, family, L3 index, TCP header flag로 filter할 수 있습니다. TCP-AO header가 있으면 keyid, rnext, maclen도 filter에 넣을 수 있고 SNE update에는 rollover된 숫자가 포함됩니다.

RFC 5925는 MKT의 TCP connection identifier를 local/remote IP와 local/remote TCP port로 정의하고, range·mask·wildcard 등으로 부분 지정하는 것을 폭넓게 허용합니다. 현재 Linux TCP-AO는 TCP port matching을 제공하지 않습니다. Port range가 uAPI에 가장 유연할 가능성이 있지만 아직 구현되지 않았습니다.

A tip here for scaled TCP_LISTEN sockets, that may have some thousands TCP-AO
keys, is: use filters in ``getsockopt(TCP_AO_GET_KEYS)`` and asynchronous
delete with ``setsockopt(TCP_AO_DEL_KEY)``.

Linux TCP-AO also provides a bunch of segment counters that can be helpful
with troubleshooting/debugging issues. Every MKT has good/bad counters
that reflect how many packets passed/failed verification.
Each TCP-AO socket has the following counters:
- for good segments (properly signed)
- for bad segments (failed TCP-AO verification)
- for segments with unknown keys
- for segments where an AO signature was expected, but wasn't found
- for the number of ignored ICMPs

TCP-AO per-socket counters are also duplicated with per-netns counters,
exposed with SNMP. Those are ``TCPAOGood``, ``TCPAOBad``, ``TCPAOKeyNotFound``,
``TCPAORequired`` and ``TCPAODroppedIcmps``.

For monitoring purposes, there are following TCP-AO trace events:
``tcp_hash_bad_header``, ``tcp_hash_ao_required``, ``tcp_ao_handshake_failure``,
``tcp_ao_wrong_maclen``, ``tcp_ao_wrong_maclen``, ``tcp_ao_key_not_found``,
``tcp_ao_rnext_request``, ``tcp_ao_synack_no_key``, ``tcp_ao_snd_sne_update``,
``tcp_ao_rcv_sne_update``. It's possible to separately enable any of them and
one can filter them by net-namespace, 4-tuple, family, L3 index, and TCP header
flags. If a segment has a TCP-AO header, the filters may also include
keyid, rnext, and maclen. SNE updates include the rolled-over numbers.

RFC 5925 very permissively specifies how TCP port matching can be done for
MKTs::

   TCP connection identifier. A TCP socket pair, i.e., a local IP
   address, a remote IP address, a TCP local port, and a TCP remote port.
   Values can be partially specified using ranges (e.g., 2-30), masks
   (e.g., 0xF0), wildcards (e.g., "*"), or any other suitable indication.

Currently Linux TCP-AO implementation doesn't provide any TCP port matching.
Probably, port ranges are the most flexible for uAPI, but so far
not implemented.

setsockopt와 accept의 key race

TCP-MD5 established connection은 key가 하나지만 TCP-AO connection은 여러 key를 가질 수 있으므로 listen socket에서 accept되는 connection도 많은 key를 가질 수 있습니다. 올바르게 서명된 첫 SYN에서 모든 key를 request socket으로 복사하면 request socket이 커지므로 현재 구현은 복사하지 않고 parent listener socket에서 key를 조회합니다.

그 결과 userspace가 TCP-AO key를 제거하면 아직 established되지 않은 request socket connection이 깨질 수 있습니다. 반대로 이미 established됐지만 아직 `accept()`되지 않아 accept queue에 있는 socket에서는 key가 제거되지 않습니다.

반대 방향도 같습니다. Listener socket에 peer용 새 key를 추가해도 accept queue의 established socket에는 새 key가 들어가지 않습니다.

현재 `setsockopt(TCP_AO_ADD_KEY)` 또는 `setsockopt(TCP_AO_DEL_KEY)`와 `accept()` 사이의 race 해결은 userspace 책임입니다. `accept()`가 반환한 socket의 MKT를 검사해 listener에서 일어난 key rotation이 새 established connection에 반영됐는지 확인해야 합니다.

이는 kernel이 아무것도 하지 않는 TCP-MD5와 비슷한 방식이며, 나중에 `tcp_ao_add`와 `tcp_ao_del`의 새 flag로 바뀔 수 있습니다. 이 race는 새 connection의 3-way handshake 동안 key rotation이 일어나야 하므로 드뭅니다.

4. ``setsockopt()`` vs ``accept()`` race
========================================

In contrast with an established TCP-MD5 connection which has just one key,
TCP-AO connections may have many keys, which means that accepted connections
on a listen socket may have any amount of keys as well. As copying all those
keys on a first properly signed SYN would make the request socket bigger, that
would be undesirable. Currently, the implementation doesn't copy keys
to request sockets, but rather look them up on the "parent" listener socket.

The result is that when userspace removes TCP-AO keys, that may break
not-yet-established connections on request sockets as well as not removing
keys from sockets that were already established, but not yet ``accept()``'ed,
hanging in the accept queue.

The reverse is valid as well: if userspace adds a new key for a peer on
a listener socket, the established sockets in the accept queue won't
have the new keys.

At this moment, the resolution for the two races:
``setsockopt(TCP_AO_ADD_KEY)`` vs ``accept()``
and ``setsockopt(TCP_AO_DEL_KEY)`` vs ``accept()`` is delegated to userspace.
This means that it's expected that userspace would check the MKTs on the socket
that was returned by ``accept()`` to verify that any key rotation that
happened on the listen socket is reflected on the newly established connection.

This is a similar "do-nothing" approach to TCP-MD5 from the kernel side and
may be changed later by introducing new flags to ``tcp_ao_add``
and ``tcp_ao_del``.

Note that this race is rare for it needs TCP-AO key rotation to happen
during the 3-way handshake for the new TCP connection.

TCP-MD5와의 상호작용

TCP connection은 TCP-AO와 TCP-MD5 option 사이를 migrate할 수 없습니다. AO 또는 MD5 key가 있는 established socket에는 다른 option의 key를 추가할 수 없습니다.

Listening socket은 다릅니다. BGP server가 TCP-AO client와 deprecated TCP-MD5 client를 모두 받을 수 있으므로 `TCP_CLOSED` 또는 `TCP_LISTEN` socket에는 두 종류 key를 모두 추가할 수 있습니다. 다만 같은 peer에 서로 다른 type의 key를 추가할 수는 없습니다.

5. Interaction with TCP-MD5
===========================

A TCP connection can not migrate between TCP-AO and TCP-MD5 options. The
established sockets that have either AO or MD5 keys are restricted for
adding keys of the other option.

For listening sockets the picture is different: BGP server may want to receive
both TCP-AO and (deprecated) TCP-MD5 clients. As a result, both types of keys
may be added to TCP_CLOSED or TCP_LISTEN sockets. It's not allowed to add
different types of keys for the same peer.

Linux SNE rollover 계산

RFC 5925 6.2는 TCP sequence number를 Sequence Number Extension(SNE)으로 확장하는 algorithm을 설명합니다. 이전 sequence number를 추적하다 현재 SEQ가 rollover하면 `sne_flag`를 설정합니다. 현재와 이전 SEQ가 모두 `0x7fff`, 즉 32KB 경계를 넘으면 flag를 지웁니다.

`sne_flag`가 설정된 동안 각 packet의 SEQ를 `0x7fff`와 비교하고 32KB보다 높으면 SNE 증가 전 값으로 검증해야 하는 packet으로 봅니다. 따라서 `[0, 32KB]` window에서는 `(SNE - 1)` packet도 받아들일 수 있습니다.

Linux는 network stack이 ACK를 기다리는 첫 SEQ byte `snd_una`와 다음에 원하는 SEQ byte `rcv_nxt`를 이미 추적한다는 점을 이용해 단순화합니다. 이 정보로 sender와 receiver가 4GB SEQ 공간의 어디에 있는지 대략 알 수 있고 zero로 rollover할 때 해당 SNE를 증가시킵니다.

각 TCP-AO segment에서 `tcp_ao_compute_sne()`가 segment SEQ를 `snd_una` 또는 `rcv_nxt`와 비교해 주변 2GB window에 맞추면서 rollover를 감지합니다. 이 방식은 code를 크게 단순화하며 각 TCP-AO socket에 SNE number만 저장하면 됩니다.

2GB window는 RFC 5926보다 훨씬 관대해 보이지만 rollover 전후의 올바른 SNE 선택에만 사용됩니다. 더 많은 replay segment가 이 단계를 통과할 수 있어도 검증된 segment에는 `tcp_sequence()`의 일반 TCP 검사가 모두 적용됩니다. Algorithm 단순성과 큰 TCP window에서의 더 나은 동작을 위해 replay/retransmit 허용 범위를 조금 넓힌 절충입니다.

6. SNE Linux implementation
===========================

RFC 5925 [6.2] describes the algorithm of how to extend TCP sequence numbers
with SNE.  In short: TCP has to track the previous sequence numbers and set
sne_flag when the current SEQ number rolls over. The flag is cleared when
both current and previous SEQ numbers cross 0x7fff, which is 32Kb.

In times when sne_flag is set, the algorithm compares SEQ for each packet with
0x7fff and if it's higher than 32Kb, it assumes that the packet should be
verified with SNE before the increment. As a result, there's
this [0; 32Kb] window, when packets with (SNE - 1) can be accepted.

Linux implementation simplifies this a bit: as the network stack already tracks
the first SEQ byte that ACK is wanted for (snd_una) and the next SEQ byte that
is wanted (rcv_nxt) - that's enough information for a rough estimation
on where in the 4GB SEQ number space both sender and receiver are.
When they roll over to zero, the corresponding SNE gets incremented.

tcp_ao_compute_sne() is called for each TCP-AO segment. It compares SEQ numbers
from the segment with snd_una or rcv_nxt and fits the result into a 2GB window around them,
detecting SEQ numbers rolling over. That simplifies the code a lot and only
requires SNE numbers to be stored on every TCP-AO socket.

The 2GB window at first glance seems much more permissive compared to
RFC 5926. But that is only used to pick the correct SNE before/after
a rollover. It allows more TCP segment replays, but yet all regular
TCP checks in tcp_sequence() are applied on the verified segment.
So, it trades a bit more permissive acceptance of replayed/retransmitted
segments for the simplicity of the algorithm and what seems better behaviour
for large TCP windows.

표준과 추가 자료

참고 문서는 TCP Authentication Option을 정의한 RFC 5925, TCP-AO cryptographic algorithm을 정의한 RFC 5926, TCP-AO용 SHA-2 algorithm draft, BGP session용 TCP MD5 Signature Option을 정의한 RFC 2385입니다. 원문 URL을 그대로 보존합니다.

저자는 Dmitry Safonov `<dima@arista.com>`입니다.

7. Links
========

RFC 5925 The TCP Authentication Option
   https://www.rfc-editor.org/rfc/pdfrfc/rfc5925.txt.pdf

RFC 5926 Cryptographic Algorithms for the TCP Authentication Option (TCP-AO)
   https://www.rfc-editor.org/rfc/pdfrfc/rfc5926.txt.pdf

Draft "SHA-2 Algorithm for the TCP Authentication Option (TCP-AO)"
   https://datatracker.ietf.org/doc/html/draft-nayak-tcp-sha2-03

RFC 2385 Protection of BGP Sessions via the TCP MD5 Signature Option
   https://www.rfc-editor.org/rfc/pdfrfc/rfc2385.txt.pdf

:Author: Dmitry Safonov <dima@arista.com>