IBM Virtual Management Channel

문서와 저자

GPL-2.0-or-later로 배포되는 IBM Virtual Management Channel kernel driver 문서입니다. 저자는 Dave Engebretsen, Adam Reznechek, Steven Royer, Bryant G. Ly입니다.

.. SPDX-License-Identifier: GPL-2.0+

======================================================
IBM Virtual Management Channel Kernel Driver (IBMVMC)
======================================================

:Authors:
        Dave Engebretsen <engebret@us.ibm.com>,
        Adam Reznechek <adreznec@linux.vnet.ibm.com>,
        Steven Royer <seroyer@linux.vnet.ibm.com>,
        Bryant G. Ly <bryantly@linux.vnet.ibm.com>,

소개

이 문서를 이해하려면 virtualization 기술 지식이 필요합니다. 참고 자료로 원문에 링크된 Linux on Power Architecture Platform Reference를 제시합니다.

VMC(Virtual Management Channel)는 hypervisor와 management partition 사이에 message-passing interface를 제공하는 logical device입니다. Hardware Management Console(HMC)에 기반한 system management의 대안을 management partition에서 제공하려는 목적입니다.

IBM의 주된 hardware management solution인 HMC는 외부 tower 또는 rack-mounted PC 형태의 appliance server입니다. Power Systems 환경에서는 HMC 하나가 여러 POWER processor 기반 system을 관리할 수 있습니다.

Introduction
============

Note: Knowledge of virtualization technology is required to understand
this document.

A good reference document would be:

https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf

The Virtual Management Channel (VMC) is a logical device which provides an
interface between the hypervisor and a management partition. This interface
is like a message passing interface. This management partition is intended
to provide an alternative to systems that use a Hardware Management
Console (HMC) - based system management.

The primary hardware management solution that is developed by IBM relies
on an appliance server named the Hardware Management Console (HMC),
packaged as an external tower or rack-mounted personal computer. In a
Power Systems environment, a single HMC can manage multiple POWER
processor-based systems.

Management application

Management partition에는 system administrator가 CLI 또는 REST API로 system partitioning 특성을 구성할 수 있게 하는 management application이 있습니다.

이 application은 PowerVM으로 virtualize된 POWER8 이상 server의 Linux logical partition에서 실행됩니다. 기존에 HMC가 맡던 configuration, maintenance, control을 HMC-to-hypervisor interface와 기존 operating-system method의 조합으로 구현합니다.

HMC 기능의 일부와 기본 partition configuration을 제공하며, 지원하는 HMC-to-hypervisor message는 아래 정의한 VMC interface를 통해 hypervisor로 전달합니다.

• Logical partitioning configuration
• 개별 partition start·stop
• Partition status 표시
• Virtual Ethernet 관리
• Virtual storage 관리
• 기본 system management

Management Application
----------------------

In the management partition, a management application exists which enables
a system administrator to configure the system’s partitioning
characteristics via a command line interface (CLI) or Representational
State Transfer Application (REST API's).

The management application runs on a Linux logical partition on a
POWER8 or newer processor-based server that is virtualized by PowerVM.
System configuration, maintenance, and control functions which
traditionally require an HMC can be implemented in the management
application using a combination of HMC to hypervisor interfaces and
existing operating system methods. This tool provides a subset of the
functions implemented by the HMC and enables basic partition configuration.
The set of HMC to hypervisor messages supported by the management
application component are passed to the hypervisor over a VMC interface,
which is defined below.

The VMC enables the management partition to provide basic partitioning
functions:

- Logical Partitioning Configuration
- Start, and stop actions for individual partitions
- Display of partition status
- Management of virtual Ethernet
- Management of virtual Storage
- Basic system management

Virtual Management Channel

VMC는 management application과 hypervisor 사이의 통신용 logical device이며 virtualization management software를 위한 pipe를 만듭니다. 지정된 management partition에 virtual device로 표시됩니다.

Communication device는 CRQ(Command/Response Queue)와 RDMA(Remote Direct Memory Access)를 사용합니다. Protocol message를 송수신하기 전에 hypervisor와 management partition 양쪽 channel이 실행 중임을 확인하는 three-way handshake를 거칩니다.

Driver는 Transport Event CRQ도 사용합니다. Peer partition이 비정상 종료되거나 한쪽이 `H_FREE_CRQ`로 CRQ를 닫으면 hypervisor가 CRQ message를 보냅니다.

VMC device에는 두 CRQ message class가 추가됩니다. VMC Administrative message는 각 partition이 partner에 capability를 알릴 때 사용하고, HMC Interface message는 실제 HMC message 흐름에 사용합니다.

대부분의 HMC message는 CRQ buffer보다 훨씬 크므로 각 HMC Interface CRQ message 전에 HMC message data를 virtual DMA(RDMA)합니다. RDMA operation은 management partition만 구동하며 hypervisor가 직접 message data 이동을 일으키지는 않습니다.

Virtual Management Channel (VMC)
--------------------------------

A logical device, called the Virtual Management Channel (VMC), is defined
for communicating between the management application and the hypervisor. It
basically creates the pipes that enable virtualization management
software. This device is presented to a designated management partition as
a virtual device.

This communication device uses Command/Response Queue (CRQ) and the
Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is
defined that must take place to establish that both the hypervisor and
management partition sides of the channel are running prior to
sending/receiving any of the protocol messages.

This driver also utilizes Transport Event CRQs. CRQ messages are sent
when the hypervisor detects one of the peer partitions has abnormally
terminated, or one side has called H_FREE_CRQ to close their CRQ.
Two new classes of CRQ messages are introduced for the VMC device. VMC
Administrative messages are used for each partition using the VMC to
communicate capabilities to their partner. HMC Interface messages are used
for the actual flow of HMC messages between the management partition and
the hypervisor. As most HMC messages far exceed the size of a CRQ buffer,
a virtual DMA (RMDA) of the HMC message data is done prior to each HMC
Interface CRQ message. Only the management partition drives RDMA
operations; hypervisors never directly cause the movement of message data.

용어

`RDMA`: server와 client 또는 server와 partner partition 사이의 DMA transfer입니다. 여기서 DMA는 memory에 대한 physical I/O뿐 아니라 memory-to-memory 이동도 포함합니다.

`CRQ`: partner partition 사이의 통신에 사용하는 Command/Response Queue입니다. Hypervisor가 partition에 알리는 transport event도 이 queue로 보고됩니다.

Terminology
-----------
RDMA
        Remote Direct Memory Access is DMA transfer from the server to its
        client or from the server to its partner partition. DMA refers
        to both physical I/O to and from memory operations and to memory
        to memory move operations.
CRQ
        Command/Response Queue a facility which is used to communicate
        between partner partitions. Transport events which are signaled
        from the hypervisor to partition are also reported in this queue.

Management partition driver interface

Management application 구현 예에서는 device driver가 VMC device에 연결됩니다. `/dev/ibmvmc` 같은 새 device가 VMC에 대한 `open`, `close`, `read`, `write`, `ioctl` interface를 제공합니다.

Example Management Partition VMC Driver Interface
=================================================

This section provides an example for the management application
implementation where a device driver is used to interface to the VMC
device. This driver consists of a new device, for example /dev/ibmvmc,
which provides interfaces to open, close, read, write, and perform
ioctl’s against the VMC device.

VMC interface 초기화

Driver가 load되면 VMC를 초기화해야 합니다. 먼저 CRQ를 만들고 초기화한 뒤 management partition과 hypervisor가 code version과 사용 가능한 resource 수를 알리는 capability를 교환합니다.

마지막으로 hypervisor가 가능한 HMC connection마다 하나씩 initial VMC buffer pool을 만들도록 management partition에 요청합니다. 이 buffer는 management application session 초기화에 사용합니다.

초기화가 끝나기 전 `open()`에는 `EBUSY`를 반환하고, 그 밖의 모든 `open()` 실패에는 `EIO`를 반환합니다.

VMC Interface Initialization
----------------------------

The device driver is responsible for initializing the VMC when the driver
is loaded. It first creates and initializes the CRQ. Next, an exchange of
VMC capabilities is performed to indicate the code version and number of
resources available in both the management partition and the hypervisor.
Finally, the hypervisor requests that the management partition create an
initial pool of VMC buffers, one buffer for each possible HMC connection,
which will be used for management application  session initialization.
Prior to completion of this initialization sequence, the device returns
EBUSY to open() calls. EIO is returned for all open() failures.

::

        Management Partition                Hypervisor
                        CRQ INIT
        ---------------------------------------->
                   CRQ INIT COMPLETE
        <----------------------------------------
                      CAPABILITIES
        ---------------------------------------->
                 CAPABILITIES RESPONSE
        <----------------------------------------
              ADD BUFFER (HMC IDX=0,1,..)         _
        <----------------------------------------  |
                  ADD BUFFER RESPONSE              | - Perform # HMCs Iterations
        ----------------------------------------> -

VMC interface open

기본 VMC channel 초기화 뒤 HMC session-level connection을 설정할 수 있습니다. Application layer가 VMC device를 `open()`하고 32-byte HMC ID를 지정하는 `ioctl()`을 실행합니다. VMC device state가 유효하지 않으면 `ioctl()`은 `EIO`를 반환합니다.

Driver는 HMC ID에 대해 1~255 범위의 새 HMC session 값과 0~254 범위의 HMC index 값을 만듭니다. HMC ID를 hypervisor로 RDMA한 뒤 Interface Open message를 보냅니다.

Hypervisor는 새 HMC connection의 initial buffer pool을 채우는 Add Buffer message를 보내고, 정상 runtime messaging 준비가 끝났음을 Interface Open Response로 알립니다.

VMC Interface Open
------------------

After the basic VMC channel has been initialized, an HMC session level
connection can be established. The application layer performs an open() to
the VMC device and executes an ioctl() against it, indicating the HMC ID
(32 bytes of data) for this session. If the VMC device is in an invalid
state, EIO will be returned for the ioctl(). The device driver creates a
new HMC session value (ranging from 1 to 255) and HMC index value (starting
at index 0 and ranging to 254) for this HMC ID. The driver then does an
RDMA of the HMC ID to the hypervisor, and then sends an Interface Open
message to the hypervisor to establish the session over the VMC. After the
hypervisor receives this information, it sends Add Buffer messages to the
management partition to seed an initial pool of buffers for the new HMC
connection. Finally, the hypervisor sends an Interface Open Response
message, to indicate that it is ready for normal runtime messaging. The
following illustrates this VMC flow:

::

        Management Partition             Hypervisor
                      RDMA HMC ID
        ---------------------------------------->
                    Interface Open
        ---------------------------------------->
                      Add Buffer                  _
        <----------------------------------------  |
                  Add Buffer Response              | - Perform N Iterations
        ----------------------------------------> -
                Interface Open Response
        <----------------------------------------

VMC interface runtime

정상 runtime에는 management application과 hypervisor가 Signal VMC message와 RDMA operation으로 HMC message를 교환합니다.

Hypervisor로 보낼 때 application이 VMC device에 `write()`합니다. Driver가 data를 hypervisor로 RDMA한 뒤 Signal Message를 보냅니다. Hypervisor가 buffer를 제공하기 전이거나 현재 사용 가능한 buffer가 없으면 `write()`는 `EBUSY`, invalid device state 같은 다른 오류에는 `EIO`를 반환합니다.

Hypervisor가 management 쪽으로 보낼 때는 data를 VMC buffer에 넣고 management partition의 VMC driver에 Signal Message를 보냅니다. Driver가 buffer를 partition으로 RDMA한 뒤 VMC device의 `read()`를 통해 해당 management application에 data를 전달합니다.

읽을 buffer가 없으면 `read()`는 block됩니다. Management application은 `select()`로 VMC device가 읽을 data를 준비할 때까지 기다릴 수 있습니다.

VMC Interface Runtime
---------------------

During normal runtime, the management application and the hypervisor
exchange HMC messages via the Signal VMC message and RDMA operations. When
sending data to the hypervisor, the management application performs a
write() to the VMC device, and the driver RDMA’s the data to the hypervisor
and then sends a Signal Message. If a write() is attempted before VMC
device buffers have been made available by the hypervisor, or no buffers
are currently available, EBUSY is returned in response to the write(). A
write() will return EIO for all other errors, such as an invalid device
state. When the hypervisor sends a message to the management, the data is
put into a VMC buffer and an Signal Message is sent to the VMC driver in
the management partition. The driver RDMA’s the buffer into the partition
and passes the data up to the appropriate management application via a
read() to the VMC device. The read() request blocks if there is no buffer
available to read. The management application may use select() to wait for
the VMC device to become ready with data to read.

::

        Management Partition             Hypervisor
                        MSG RDMA
        ---------------------------------------->
                        SIGNAL MSG
        ---------------------------------------->
                        SIGNAL MSG
        <----------------------------------------
                        MSG RDMA
        <----------------------------------------

VMC interface close

Application layer가 device에 `close()`를 수행하면 management partition이 HMC session-level connection을 닫습니다. Hypervisor로 Interface Close message가 전달되어 session이 종료되고, driver는 이 HMC connection의 buffer에 할당한 storage를 해제해야 합니다.

VMC Interface Close
-------------------

HMC session level connections are closed by the management partition when
the application layer performs a close() against the device. This action
results in an Interface Close message flowing to the hypervisor, which
causes the session to be terminated. The device driver must free any
storage allocated for buffers for this HMC connection.

::

        Management Partition             Hypervisor
                     INTERFACE CLOSE
        ---------------------------------------->
                INTERFACE CLOSE RESPONSE
        <----------------------------------------

추가 정보

CRQ Message, VMC Message, HMC interface buffer, signal message의 자세한 문서는 Linux on Power Architecture Platform Reference Section F를 참조합니다.

Additional Information
======================

For more information on the documentation for CRQ Messages, VMC Messages,
HMC interface Buffers, and signal messages please refer to the Linux on
Power Architecture Platform Reference. Section F.