Remote Processor Framework

AMP remote processor 제어

현대 SoC에는 서로 다른 Linux 또는 RTOS instance를 실행하는 heterogeneous remote processor가 AMP(asymmetric multiprocessing) 구성으로 들어가는 경우가 많다. 예를 들어 OMAP4는 dual Cortex-A9, dual Cortex-M3, C64x+ DSP를 가지며 A9 두 개는 SMP Linux, M3 두 개와 DSP는 각각 별도 RTOS를 실행할 수 있다.

Remoteproc framework는 hardware 차이를 추상화하면서 platform·architecture가 remote processor의 power on, firmware load, power off를 제어하게 해 전체 driver 중복을 피한다.

해당 통신을 지원하는 remote processor에는 rpmsg virtio device도 추가한다. Platform별 remoteproc driver가 소수 low-level handler만 제공하면 rpmsg driver가 그대로 동작한다. 자세한 virtio 기반 rpmsg bus와 driver는 `Documentation/staging/rpmsg.rst`를 참조한다.

다른 종류의 virtio device도 등록할 수 있다. Firmware가 지원 virtio device 종류를 공개하면 remoteproc가 device를 추가하므로 기존 virtio driver를 remote processor backend와 적은 개발 비용으로 재사용할 수 있다.

==========================
Remote Processor Framework
==========================

Introduction
============

Modern SoCs typically have heterogeneous remote processor devices in asymmetric
multiprocessing (AMP) configurations, which may be running different instances
of operating system, whether it's Linux or any other flavor of real-time OS.

OMAP4, for example, has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP.
In a typical configuration, the dual cortex-A9 is running Linux in a SMP
configuration, and each of the other three cores (two M3 cores and a DSP)
is running its own instance of RTOS in an AMP configuration.

The remoteproc framework allows different platforms/architectures to
control (power on, load firmware, power off) those remote processors while
abstracting the hardware differences, so the entire driver doesn't need to be
duplicated. In addition, this framework also adds rpmsg virtio devices
for remote processors that supports this kind of communication. This way,
platform-specific remoteproc drivers only need to provide a few low-level
handlers, and then all rpmsg drivers will then just work
(for more information about the virtio-based rpmsg bus and its drivers,
please read Documentation/staging/rpmsg.rst).
Registration of other types of virtio devices is now also possible. Firmwares
just need to publish what kind of virtio devices do they support, and then
remoteproc will add those devices. This makes it possible to reuse the
existing virtio drivers with remote processor backends at a minimal development
cost.

사용자 API와 reference count

`rproc_boot(struct rproc *rproc)`는 firmware를 load하고 power를 켜 remote processor를 boot한다. 이미 켜져 있으면 즉시 성공한다. 성공 시 0, 실패 시 적절한 error를 반환한다. 호출 전 devres, platform data, `remoteproc_rpmsg.c` 방식 등으로 유효한 `rproc` handle을 확보해야 하며 필요성이 커지면 `dev_archdata` 사용도 고려할 수 있다.

`rproc_shutdown(struct rproc *rproc)`는 `rproc_boot()`로 켠 processor를 끈다. 추가 사용자가 있으면 power refcount만 줄이고 실제 power off 없이 끝난다. 모든 `rproc_boot()` 호출은 결국 `rproc_shutdown()`과 짝을 이뤄야 하며 중복 shutdown은 bug다.

Shutdown은 rproc object refcount가 아니라 power refcount만 줄인다. 따라서 반환 뒤에도 handle은 유효하고 필요하면 다시 `rproc_boot()`에 쓸 수 있다.

`rproc_get_by_phandle(phandle)`은 Device Tree phandle로 handle을 찾는다. 성공하면 handle, 실패하면 `NULL`을 반환하며 remote processor refcount를 증가시키므로 필요가 끝나면 반드시 `rproc_put()`으로 감소시킨다.

User API
========

::

  int rproc_boot(struct rproc *rproc)

Boot a remote processor (i.e. load its firmware, power it on, ...).

If the remote processor is already powered on, this function immediately
returns (successfully).

Returns 0 on success, and an appropriate error value otherwise.
Note: to use this function you should already have a valid rproc
handle. There are several ways to achieve that cleanly (devres, pdata,
the way remoteproc_rpmsg.c does this, or, if this becomes prevalent, we
might also consider using dev_archdata for this).

::

  int rproc_shutdown(struct rproc *rproc)

Power off a remote processor (previously booted with rproc_boot()).
In case @rproc is still being used by an additional user(s), then
this function will just decrement the power refcount and exit,
without really powering off the device.

Returns 0 on success, and an appropriate error value otherwise.
Every call to rproc_boot() must (eventually) be accompanied by a call
to rproc_shutdown(). Calling rproc_shutdown() redundantly is a bug.

.. note::

  we're not decrementing the rproc's refcount, only the power refcount.
  which means that the @rproc handle stays valid even after
  rproc_shutdown() returns, and users can still use it with a subsequent
  rproc_boot(), if needed.

::

  struct rproc *rproc_get_by_phandle(phandle phandle)

Find an rproc handle using a device tree phandle. Returns the rproc
handle on success, and NULL on failure. This function increments
the remote processor's refcount, so always use rproc_put() to
decrement it back once rproc isn't needed anymore.

일반적인 boot·shutdown 사용

일반 사용 code는 `<linux/remoteproc.h>`를 include하고 유효한 `rproc` handle을 받는다. `rproc_boot()` 반환값을 검사해 오류를 처리한 뒤 remote processor에 작업을 주고, 끝나면 `rproc_shutdown()`을 호출한다. 원문 `dummy_rproc_example()`의 code와 comment는 아래 source block에 그대로 보존된다.

Typical usage
=============

::

  #include <linux/remoteproc.h>

  /* in case we were given a valid 'rproc' handle */
  int dummy_rproc_example(struct rproc *my_rproc)
  {
	int ret;

	/* let's power on and boot our remote processor */
	ret = rproc_boot(my_rproc);
	if (ret) {
		/*
		 * something went wrong. handle it and leave.
		 */
	}

	/*
	 * our remote processor is now powered on... give it some work
	 */

	/* let's shut it down now */
	rproc_shutdown(my_rproc);
  }

구현자 API와 crash 보고

`rproc_alloc(dev, name, ops, firmware, len)`은 remote processor handle을 할당하지만 아직 등록하지 않는다. Underlying device, 이름, platform별 ops handler, boot firmware 이름, driver-private data 길이를 byte 단위로 받는다. Remote processor 구현이 초기화 중 호출하고 준비가 끝나면 `rproc_add()`로 등록을 완료한다. 성공 시 새 rproc, 실패 시 `NULL`을 반환한다.

등록 전이라도 `rproc`를 직접 해제하면 안 된다. `rproc_alloc()`을 되돌릴 때는 반드시 `rproc_free()`를 쓴다. `rproc_free()`는 rproc refcount를 줄이며 다른 reference가 없어 0이 될 때만 실제로 free한다.

`rproc_add()`는 할당된 rproc를 framework에 등록하며 platform별 구현이 새 device를 probe할 때 호출한다. 성공 시 0, 아니면 error code를 반환한다. 비동기 firmware loading context를 시작해 firmware가 지원하는 virtio device를 찾고, 발견하면 생성·추가하므로 추가 virtio driver가 probe될 수 있다.

`rproc_del()`은 성공한 `rproc_add()`만 되돌린다. Platform 구현이 device를 제거할 때 호출하며 반환 뒤에도 rproc는 유효하다. 마지막 refcount는 `rproc_free()`로 줄여야 한다. 성공 시 0, rproc가 invalid면 `-EINVAL`을 반환한다.

`rproc_report_crash(rproc, type)`는 remoteproc crash를 보고한다. Platform별 rproc 구현이 crash를 감지할 때마다 호출해야 하며 remoteproc가 아닌 driver가 호출하면 안 된다. Atomic 또는 interrupt context에서 호출할 수 있다.

API for implementers
====================

::

  struct rproc *rproc_alloc(struct device *dev, const char *name,
				const struct rproc_ops *ops,
				const char *firmware, int len)

Allocate a new remote processor handle, but don't register
it yet. Required parameters are the underlying device, the
name of this remote processor, platform-specific ops handlers,
the name of the firmware to boot this rproc with, and the
length of private data needed by the allocating rproc driver (in bytes).

This function should be used by rproc implementations during
initialization of the remote processor.

After creating an rproc handle using this function, and when ready,
implementations should then call rproc_add() to complete
the registration of the remote processor.

On success, the new rproc is returned, and on failure, NULL.

.. note::

  **never** directly deallocate @rproc, even if it was not registered
  yet. Instead, when you need to unroll rproc_alloc(), use rproc_free().

::

  void rproc_free(struct rproc *rproc)

Free an rproc handle that was allocated by rproc_alloc.

This function essentially unrolls rproc_alloc(), by decrementing the
rproc's refcount. It doesn't directly free rproc; that would happen
only if there are no other references to rproc and its refcount now
dropped to zero.

::

  int rproc_add(struct rproc *rproc)

Register @rproc with the remoteproc framework, after it has been
allocated with rproc_alloc().

This is called by the platform-specific rproc implementation, whenever
a new remote processor device is probed.

Returns 0 on success and an appropriate error code otherwise.
Note: this function initiates an asynchronous firmware loading
context, which will look for virtio devices supported by the rproc's
firmware.

If found, those virtio devices will be created and added, so as a result
of registering this remote processor, additional virtio drivers might get
probed.

::

  int rproc_del(struct rproc *rproc)

Unroll rproc_add().

This function should be called when the platform specific rproc
implementation decides to remove the rproc device. it should
_only_ be called if a previous invocation of rproc_add()
has completed successfully.

After rproc_del() returns, @rproc is still valid, and its
last refcount should be decremented by calling rproc_free().

Returns 0 on success and -EINVAL if @rproc isn't valid.

::

  void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type)

Report a crash in a remoteproc

This function must be called every time a crash is detected by the
platform specific rproc implementation. This should not be called from a
non-remoteproc driver. This function can be called from atomic/interrupt
context.

Platform별 rproc_ops callback

Platform별 remoteproc driver는 `struct rproc_ops` callback을 제공한다. 모든 구현은 최소 `start`와 `stop`을 제공해야 하고 rpmsg/virtio 기능이 필요하면 `kick`도 제공한다.

`start(rproc)`는 `rproc->priv`의 platform private data로 device power를 켜고 boot한다. 필요하면 remoteproc core가 ELF entry point를 넣은 `rproc->bootaddr`에서 boot address를 얻는다. 성공 시 0, 실패 시 적절한 error code를 반환한다.

`stop(rproc)`은 device power를 끄고 성공 시 0, 실패 시 error code를 반환한다. `kick(rproc, vqid)`은 새 message가 들어간 virtqueue index를 받아 remote processor에 interrupt를 보내 pending message를 알린다. 정확한 index 전달은 선택 사항이며 existing virtqueue와 used ring을 순회해 새 buffer를 찾는 비용도 크지 않다.

Implementation callbacks
========================

These callbacks should be provided by platform-specific remoteproc
drivers::

  /**
   * struct rproc_ops - platform-specific device handlers
   * @start:	power on the device and boot it
   * @stop:	power off the device
   * @kick:	kick a virtqueue (virtqueue id given as a parameter)
   */
  struct rproc_ops {
	int (*start)(struct rproc *rproc);
	int (*stop)(struct rproc *rproc);
	void (*kick)(struct rproc *rproc, int vqid);
  };

Every remoteproc implementation should at least provide the ->start and ->stop
handlers. If rpmsg/virtio functionality is also desired, then the ->kick handler
should be provided as well.

The ->start() handler takes an rproc handle and should then power on the
device and boot it (use rproc->priv to access platform-specific private data).
The boot address, in case needed, can be found in rproc->bootaddr (remoteproc
core puts there the ELF entry point).
On success, 0 should be returned, and on failure, an appropriate error code.

The ->stop() handler takes an rproc handle and powers the device down.
On success, 0 is returned, and on failure, an appropriate error code.

The ->kick() handler takes an rproc handle, and an index of a virtqueue
where new message was placed in. Implementations should interrupt the remote
processor and let it know it has pending messages. Notifying remote processors
the exact virtqueue index to look in is optional: it is easy (and not
too expensive) to go through the existing virtqueues and look for new buffers
in the used rings.

ELF firmware와 resource table

현재 remoteproc는 ELF32와 ELF64 firmware binary를 지원한다. 다른 platform이 다른 binary format을 요구하면 공통 code를 복제하지 않고 여러 format을 지원하도록 binary format을 framework core에서 분리해야 한다.

Firmware parsing 시 각 segment는 지정 device address에 따라 memory로 load된다. Remote processor가 memory에 직접 접근하면 device address가 physical address일 수 있다.

표준 ELF segment 외에 대다수 remote processor firmware에는 `resource table`이라는 특별 section이 있다. Power on 전에 필요한 physically contiguous memory allocation, on-chip peripheral의 IOMMU mapping 같은 system resource를 담는다. Remoteproc core는 모든 요구가 충족된 뒤에만 device를 켠다.

Resource table은 trace buffer와 지원 virtio device 및 configuration처럼 remote processor가 지원하는 feature나 구성을 공개하는 entry도 담을 수 있다.

`struct resource_table` header는 version `ver`, resource entry 수 `num`, 반드시 0인 `reserved[2]`, 각 entry를 가리키는 offset array를 갖는다. Header 바로 뒤에 entry가 있고 각 entry는 type과 data가 있는 packed `struct fw_rsc_hdr`로 시작한다. Entry body는 type에 따라 parsing한다.

일부 resource entry는 host에 remoteproc configuration을 알리기만 하고, 다른 entry는 system resource 할당 같은 host 작업을 요구한다. Firmware가 resource를 요청하고 host가 할당 뒤 memory address 같은 세부 정보를 돌려주는 negotiation도 있다.

Binary Firmware Structure
=========================

At this point remoteproc supports ELF32 and ELF64 firmware binaries. However,
it is quite expected that other platforms/devices which we'd want to
support with this framework will be based on different binary formats.

When those use cases show up, we will have to decouple the binary format
from the framework core, so we can support several binary formats without
duplicating common code.

When the firmware is parsed, its various segments are loaded to memory
according to the specified device address (might be a physical address
if the remote processor is accessing memory directly).

In addition to the standard ELF segments, most remote processors would
also include a special section which we call "the resource table".

The resource table contains system resources that the remote processor
requires before it should be powered on, such as allocation of physically
contiguous memory, or iommu mapping of certain on-chip peripherals.
Remotecore will only power up the device after all the resource table's
requirement are met.

In addition to system resources, the resource table may also contain
resource entries that publish the existence of supported features
or configurations by the remote processor, such as trace buffers and
supported virtio devices (and their configurations).

The resource table begins with this header::

  /**
   * struct resource_table - firmware resource table header
   * @ver: version number
   * @num: number of resource entries
   * @reserved: reserved (must be zero)
   * @offset: array of offsets pointing at the various resource entries
   *
   * The header of the resource table, as expressed by this structure,
   * contains a version number (should we need to change this format in the
   * future), the number of available resource entries, and their offsets
   * in the table.
   */
  struct resource_table {
	u32 ver;
	u32 num;
	u32 reserved[2];
	u32 offset[0];
  } __packed;

Immediately following this header are the resource entries themselves,
each of which begins with the following resource entry header::

  /**
   * struct fw_rsc_hdr - firmware resource entry header
   * @type: resource type
   * @data: resource data
   *
   * Every resource entry begins with a 'struct fw_rsc_hdr' header providing
   * its @type. The content of the entry itself will immediately follow
   * this header, and it should be parsed according to the resource type.
   */
  struct fw_rsc_hdr {
	u32 type;
	u8 data[0];
  } __packed;

Some resources entries are mere announcements, where the host is informed
of specific remoteproc configuration. Other entries require the host to
do something (e.g. allocate a system resource). Sometimes a negotiation
is expected, where the firmware requests a resource, and once allocated,
the host should provide back its details (e.g. address of an allocated
memory region).

Firmware resource type

현재 `enum fw_resource_type`은 physically contiguous memory region 할당 요청 `RSC_CARVEOUT=0`, memory-mapped peripheral의 IOMMU mapping 요청 `RSC_DEVMEM=1`, remote processor log용 trace buffer 공개 `RSC_TRACE=2`, virtio device 지원과 header를 선언하는 `RSC_VDEV=3`을 지원한다.

`RSC_LAST=4`는 lookup 전 index 유효성 검사와 `rproc_handle_rsc` table index 관리에 쓰이므로 새 일반 type을 추가하면 끝 값을 갱신해야 한다. Vendor별 resource type 범위는 `RSC_VENDOR_START=128`부터 `RSC_VENDOR_END=512`까지다.

각 resource type의 구체 구조는 `include/linux/remoteproc.h`를 참조한다. 향후 platform별 entry가 필요해지면 `RSC_PLATFORM` type을 추가하고 platform별 rproc driver에 처리를 넘길 수 있다.

Here are the various resource types that are currently supported::

  /**
   * enum fw_resource_type - types of resource entries
   *
   * @RSC_CARVEOUT:   request for allocation of a physically contiguous
   *		    memory region.
   * @RSC_DEVMEM:     request to iommu_map a memory-based peripheral.
   * @RSC_TRACE:	    announces the availability of a trace buffer into which
   *		    the remote processor will be writing logs.
   * @RSC_VDEV:       declare support for a virtio device, and serve as its
   *		    virtio header.
   * @RSC_LAST:       just keep this one at the end
   * @RSC_VENDOR_START:	start of the vendor specific resource types range
   * @RSC_VENDOR_END:	end of the vendor specific resource types range
   *
   * Please note that these values are used as indices to the rproc_handle_rsc
   * lookup table, so please keep them sane. Moreover, @RSC_LAST is used to
   * check the validity of an index before the lookup table is accessed, so
   * please update it as needed.
   */
  enum fw_resource_type {
	RSC_CARVEOUT		= 0,
	RSC_DEVMEM		= 1,
	RSC_TRACE		= 2,
	RSC_VDEV		= 3,
	RSC_LAST		= 4,
	RSC_VENDOR_START	= 128,
	RSC_VENDOR_END		= 512,
  };

For more details regarding a specific resource type, please see its
dedicated structure in include/linux/remoteproc.h.

We also expect that platform-specific resource entries will show up
at some point. When that happens, we could easily add a new RSC_PLATFORM
type, and hand those resources to the platform-specific rproc driver to handle.

Virtio와 remoteproc

Firmware는 지원하는 virtio device와 configuration을 remoteproc에 제공해야 한다. `RSC_VDEV` entry는 `virtio_ids.h`의 device id, virtio feature, config space, vring 정보 등을 지정한다.

새 remote processor가 등록되면 framework는 resource table을 찾아 지원 virtio device를 등록한다. Firmware는 종류와 수에 제한 없이 virtio device를 지원할 수 있고, 하나의 remote processor가 여러 rpmsg virtio device를 제공할 수도 있다.

`RSC_VDEV`는 virtio device의 static allocation에만 충분하다. Dynamic allocation은 rpmsg channel을 동적으로 할당하는 방식과 비슷하게 rpmsg bus를 통해 지원할 예정이다. 원문은 자세한 설명으로 `rpmsg.txt`를 가리키며 현재 source tree의 관련 문서는 `Documentation/staging/rpmsg.rst`다.

Virtio and remoteproc
=====================

The firmware should provide remoteproc information about virtio devices
that it supports, and their configurations: a RSC_VDEV resource entry
should specify the virtio device id (as in virtio_ids.h), virtio features,
virtio config space, vrings information, etc.

When a new remote processor is registered, the remoteproc framework
will look for its resource table and will register the virtio devices
it supports. A firmware may support any number of virtio devices, and
of any type (a single remote processor can also easily support several
rpmsg virtio devices this way, if desired).

Of course, RSC_VDEV resource entries are only good enough for static
allocation of virtio devices. Dynamic allocations will also be made possible
using the rpmsg bus (similar to how we already do dynamic allocations of
rpmsg channels; read more about it in rpmsg.txt).