v9fs: Plan 9 Resource Sharing for Linux

v9fs와 9P 자료

`v9fs`는 Plan 9의 9P remote filesystem protocol을 Unix에서 구현한 것입니다. Ron Minnich와 Maya Gokhale가 처음 개발했고 Greg Watson, Eric Van Hensbergen, Latchesar Ionkov, Russ Cox 등이 뒤이어 개발했습니다.

Linux 9P client의 구현과 응용은 USENIX 논문에서 자세히 설명합니다. 관련 응용으로 XCPU clustering, KVM 제어 filesystem인 KVMFS, Cell BE용 CellFS, 9P 기반 application partition I/O인 PROSE I/O, virtualization-aware passthrough filesystem인 VirtFS가 있습니다.

.. SPDX-License-Identifier: GPL-2.0

=======================================
v9fs: Plan 9 Resource Sharing for Linux
=======================================

About
=====

v9fs is a Unix implementation of the Plan 9 9p remote filesystem protocol.

This software was originally developed by Ron Minnich <rminnich@sandia.gov>
and Maya Gokhale.  Additional development by Greg Watson
<gwatson@lanl.gov> and most recently Eric Van Hensbergen
<ericvh@gmail.com>, Latchesar Ionkov <lucho@ionkov.net> and Russ Cox
<rsc@swtch.com>.

The best detailed explanation of the Linux implementation and applications of
the 9p client is available in the form of a USENIX paper:

   https://www.usenix.org/events/usenix05/tech/freenix/hensbergen.html

Other applications are described in the following papers:

        * XCPU & Clustering
          http://xcpu.org/papers/xcpu-talk.pdf
        * KVMFS: control file system for KVM
          http://xcpu.org/papers/kvmfs.pdf
        * CellFS: A New Programming Model for the Cell BE
          http://xcpu.org/papers/cellfs-talk.pdf
        * PROSE I/O: Using 9p to enable Application Partitions
          http://plan9.escet.urjc.es/iwp9/cready/PROSE_iwp9_2006.pdf
        * VirtFS: A Virtualization Aware File System pass-through
          https://kernel.org/doc/ols/2010/ols2010-pages-109-120.pdf

remote·unix·virtio mount

remote file server는 `mount -t 9p 10.10.1.2 /mnt/9`로 mount합니다. Plan 9 From User Space 응용은 `mount -t 9p \`namespace\`/acme /mnt/9 -o trans=unix,uname=$USER`처럼 Unix transport와 사용자 이름을 지정합니다.

QEMU host의 server를 virtio transport로 연결할 때는 `mount -t 9p -o trans=virtio <mount_tag> /mnt/9`를 사용합니다. `<mount_tag>`는 server가 export mount point마다 만든 tag입니다.

client에서는 각 9P export가 `mount_tag` property를 지닌 virtio device로 보입니다. 사용 가능한 tag는 `/sys/bus/virtio/drivers/9pnet_virtio/virtio<n>/mount_tag`에서 확인합니다.

Usage
=====

For remote file server::

        mount -t 9p 10.10.1.2 /mnt/9

For Plan 9 From User Space applications (https://9fans.github.io/plan9port/)::

        mount -t 9p `namespace`/acme /mnt/9 -o trans=unix,uname=$USER

For server running on QEMU host with virtio transport::

        mount -t 9p -o trans=virtio <mount_tag> /mnt/9

where mount_tag is the tag generated by the server to each of the exported
mount points. Each 9P export is seen by the client as a virtio device with an
associated "mount_tag" property. Available mount tags can be
seen by reading /sys/bus/virtio/drivers/9pnet_virtio/virtio<n>/mount_tag files.

USB gadget transport 사용

runtime에 USB host가 export한 filesystem을 gadget을 통해 mount하려면 `mount -t 9p -o trans=usbg,aname=/path/to/fs <device> /mnt/9`를 사용합니다.

root filesystem으로 사용할 때는 kernel command line에 `root=<device> rootfstype=9p rootflags=trans=usbg,cache=loose,uname=root,access=0,dfltuid=0,dfltgid=0,aname=/path/to/rootfs`를 지정합니다.

여기서 `<device>`는 USB gadget transport가 연결한 tag이며 configfs instance name으로 정의됩니다.

USBG Usage
==========

To mount a 9p FS on a USB Host accessible via the gadget at runtime::

        mount -t 9p -o trans=usbg,aname=/path/to/fs <device> /mnt/9

To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::

        root=<device> rootfstype=9p rootflags=trans=usbg,cache=loose,uname=root,access=0,dfltuid=0,dfltgid=0,aname=/path/to/rootfs

where <device> is the tag associated by the usb gadget transport.
It is defined by the configfs instance name.

USBG host·gadget forwarding 절차

USB host가 filesystem을 export하고 USB device 쪽 gadget이 이를 mount 가능하게 만듭니다. 실제 root filesystem이 있는 개발 host에서 Diod 같은 9pfs server와 forwarder를 실행하고, embedded board는 boot 중 또는 이후 gadget을 초기화합니다.

요청은 device에서 시작해 host가 처리합니다. 일반 PC에는 USB device port가 없으므로 반대 방향 연결은 보통 성립하지 않습니다. 현재 usbg transport 요청을 직접 처리하는 native USB host service가 없기 때문에 `tools/usb/p9_fwd.py`를 사용합니다.

먼저 `$ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD`처럼 9pfs server를 시작합니다. gadget이 여러 개면 `$ python $kernel_dir/tools/usb/p9_fwd.py list`로 bus, address, ID, path를 찾습니다.

그 다음 `$ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999`로 transport를 연결하면 gadget driver를 앞 절의 방식으로 사용할 수 있습니다. embedded Linux 개발 중 NFS root boot의 대안이 한 사용 사례입니다.

USBG Example
============

The USB host exports a filesystem, while the gadget on the USB device
side makes it mountable.

Diod (9pfs server) and the forwarder are on the development host, where
the root filesystem is actually stored. The gadget is initialized during
boot (or later) on the embedded board. Then the forwarder will find it
on the USB bus and start forwarding requests.

In this case the 9p requests come from the device and are handled by the
host. The reason is that USB device ports are normally not available on
PCs, so a connection in the other direction would not work.

When using the usbg transport, for now there is no native usb host
service capable to handle the requests from the gadget driver. For
this we have to use the extra python tool p9_fwd.py from tools/usb.

Just start the 9pfs capable network server like diod/nfs-ganesha e.g.::

        $ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD

Optionally scan your bus if there are more then one usbg gadgets to find their path::

        $ python $kernel_dir/tools/usb/p9_fwd.py list

        Bus | Addr | Manufacturer     | Product          | ID        | Path
        --- | ---- | ---------------- | ---------------- | --------- | ----
          2 |   67 | unknown          | unknown          | 1d6b:0109 | 2-1.1.2
          2 |   68 | unknown          | unknown          | 1d6b:0109 | 2-1.1.3

Then start the python transport::

        $ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999

After that the gadget driver can be used as described above.

One use-case is to use it as an alternative to NFS root booting during
the development of embedded Linux devices.

9P mount option 전체

`trans=name`은 transport를 선택합니다. `unix`는 named pipe mount point, `tcp`는 TCP/IP, `fd`는 전달받은 file descriptor, `virtio`는 다음 available virtio channel, `rdma`는 지정 RDMA channel, `usbg`는 지정 USB gadget channel을 사용합니다.

`uname=name`은 remote server에 mount를 시도할 사용자 이름이며 server가 override하거나 무시할 수 있고 인증이 필요할 수 있습니다. `aname=name`은 server가 여러 filesystem을 export할 때 접근할 file tree를 선택합니다.

`cache=mode`는 caching policy입니다. 기본값은 cache 미사용입니다. bitmask에서 `0b00000001`은 file cache, `0b00000010`은 metadata cache, `0b00000100`은 writeback, `0b00001000`은 server와 명시적 일관성이 없는 loose cache, `0b10000000`은 persistent FS-Cache를 켭니다. `0b00000000`은 cache와 mmap을 모두 끕니다.

시험된 shortcut은 `none=0b00000000`, `readahead=0b00000001`, `mmap=0b00000101`, `loose=0b00001111`, `fscache=0b10001111`입니다. 임의의 다른 bit 조합은 동작이 검증되지 않았습니다.

중요하게도 `loose`와 현재 그에 기반한 `fscache`는 server의 cached value를 반드시 재검증하지 않습니다. 따라서 server 변경이 client에 반영된다는 보장이 없습니다. exclusive mount이고 server가 밑의 filesystem을 수정하지 않을 때만 사용해야 합니다.

`debug=n`은 bitmask입니다. `0x01` verbose error, `0x02` developer debug, `0x04` 9P trace, `0x08` VFS trace, `0x10` marshalling, `0x20` RPC, `0x40` transport, `0x80` allocation, `0x100` protocol message, `0x200` Fid, `0x400` packet, `0x800` FS-Cache tracing debug를 표시합니다.

`rfdno=n`과 `wfdno=n`은 `trans=fd`의 read/write file descriptor입니다. `msize=n`은 9P packet payload byte 수, `port=n`은 remote server port입니다. `noextend`는 9P2000.u와 9P2000.L semantics를 사용하지 않는 legacy mode를 강제합니다.

`version=name`은 `9p2000` legacy, `9p2000.u`, `9p2000.L` 중 protocol version을 선택합니다. `dfltuid`와 `dfltgid`는 기본 UID/GID, `afid`는 Plan 9 인증 protocol의 security channel입니다.

`nodevmap`은 special file을 mapping하지 않고 일반 file로 표현하여 device, named pipe, socket을 host 사이에 공유할 수 있게 합니다. `directio`는 모든 read/write에서 page cache를 우회하고, `ignoreqv`는 `qid.version==0`을 cache 무시 표식으로 해석하지 않으며, `noxattr`은 mount에서 xattr 기능을 제공하지 않습니다.

`access` mode는 네 가지입니다. `user`는 사용자가 처음 접근할 때마다 그 사용자의 `Tattach`를 보내는 기본 mode입니다. `<uid>`는 지정 UID만 허용하고, `any`는 한 번 attach하여 모든 작업을 한 사용자로 수행합니다. 원문에 `clien`으로 적힌 mode는 문맥상 client-side ACL 검사로 접근을 검증하는 `client` mode를 뜻합니다.

`cachetag`는 지정 persistent cache의 tag입니다. 기존 cache session tag는 `/sys/fs/9p/caches`에서 열거하며 `cache=fscache`에만 적용됩니다.

Options
=======

  ============= ===============================================================
  trans=name        select an alternative transport.  Valid options are
                  currently:

                        ========  ============================================
                        unix           specifying a named pipe mount point
                        tcp          specifying a normal TCP/IP connection
                        fd             used passed file descriptors for connection
                                  (see rfdno and wfdno)
                        virtio          connect to the next virtio channel available
                                  (from QEMU with trans_virtio module)
                        rdma          connect to a specified RDMA channel
                        usbg          connect to a specified usb gadget channel
                        ========  ============================================

  uname=name        user name to attempt mount as on the remote server.  The
                  server may override or ignore this value.  Certain user
                names may require authentication.

  aname=name        aname specifies the file tree to access when the server is
                  offering several exported file systems.

  cache=mode        specifies a caching policy.  By default, no caches are used.
                The mode can be specified as a bitmask or by using one of the
                preexisting common 'shortcuts'.
                The bitmask is described below: (unspecified bits are reserved)

                        ==========  ====================================================
                        0b00000000  all caches disabled, mmap disabled
                        0b00000001  file caches enabled
                        0b00000010  meta-data caches enabled
                        0b00000100  writeback behavior (as opposed to writethrough)
                        0b00001000  loose caches (no explicit consistency with server)
                        0b10000000  fscache enabled for persistent caching
                        ==========  ====================================================

                The current shortcuts and their associated bitmask are:

                        =========   ====================================================
                        none        0b00000000 (no caching)
                        readahead   0b00000001 (only read-ahead file caching)
                        mmap        0b00000101 (read-ahead + writeback file cache)
                        loose       0b00001111 (non-coherent file and meta-data caches)
                        fscache     0b10001111 (persistent loose cache)
                        =========   ====================================================

                NOTE: only these shortcuts are tested modes of operation at the
                moment, so using other combinations of bit-patterns is not
                known to work.  Work on better cache support is in progress.

                IMPORTANT: loose caches (and by extension at the moment fscache)
                do not necessarily validate cached values on the server.  In other
                words changes on the server are not guaranteed to be reflected
                on the client system.  Only use this mode of operation if you
                have an exclusive mount and the server will not modify the
                filesystem underneath you.

  debug=n        specifies debug level.  The debug level is a bitmask.

                        =====   ================================
                        0x01    display verbose error messages
                        0x02    developer debug (DEBUG_CURRENT)
                        0x04    display 9p trace
                        0x08    display VFS trace
                        0x10    display Marshalling debug
                        0x20    display RPC debug
                        0x40    display transport debug
                        0x80    display allocation debug
                        0x100   display protocol message debug
                        0x200   display Fid debug
                        0x400   display packet debug
                        0x800   display fscache tracing debug
                        =====   ================================

  rfdno=n        the file descriptor for reading with trans=fd

  wfdno=n        the file descriptor for writing with trans=fd

  msize=n        the number of bytes to use for 9p packet payload

  port=n        port to connect to on the remote server

  noextend        force legacy mode (no 9p2000.u or 9p2000.L semantics)

  version=name        Select 9P protocol version. Valid options are:

                        ========        ==============================
                        9p2000          Legacy mode (same as noextend)
                        9p2000.u        Use 9P2000.u protocol
                        9p2000.L        Use 9P2000.L protocol
                        ========        ==============================

  dfltuid        attempt to mount as a particular uid

  dfltgid        attempt to mount with a particular gid

  afid                security channel - used by Plan 9 authentication protocols

  nodevmap        do not map special files - represent them as normal files.
                  This can be used to share devices/named pipes/sockets between
                hosts.  This functionality will be expanded in later versions.

  directio        bypass page cache on all read/write operations

  ignoreqv        ignore qid.version==0 as a marker to ignore cache

  noxattr        do not offer xattr functions on this mount.

  access        there are four access modes.
                        user
                                if a user tries to access a file on v9fs
                                filesystem for the first time, v9fs sends an
                                attach command (Tattach) for that user.
                                This is the default mode.
                        <uid>
                                allows only user with uid=<uid> to access
                                the files on the mounted filesystem
                        any
                                v9fs does single attach and performs all
                                operations as one user
                        clien
                                 ACL based access check on the 9p client
                                 side for access validation

  cachetag        cache tag to use the specified persistent cache.
                cache tags for existing cache sessions can be listed at
                /sys/fs/9p/caches. (applies only to cache=fscache)
  ============= ===============================================================

`O_NONBLOCK` read 동작

이 절은 local filesystem과 다른 9P 특성을 설명합니다. file에 `O_NONBLOCK`을 설정하면 client read는 요청한 byte 수만큼 buffer를 채우거나 EOF에 도달할 때까지 기다리지 않습니다. server가 일부 data를 반환하는 즉시 가능한 한 일찍 반환합니다.

Behavior
========

This section aims at describing 9p 'quirks' that can be different
from a local filesystem behaviors.

 - Setting O_NONBLOCK on a file will make client reads return as early
   as the server returns some data instead of trying to fill the read
   buffer with the requested amount of bytes or end of file is reached.

9P specification·구현·문의 자료

protocol specification은 `http://ericvh.github.com/9p-rfc/`에서 유지합니다. client와 server 구현 목록은 `http://9p.cat-v.org/implementations`에 있습니다.

LLNL이 개발한 9P2000.L server Diod는 `http://code.google.com/p/diod/`에 있습니다. v9fs SourceForge project는 사용자·개발자 mailing list를 제공하며 bug report는 mailing list로 보내는 것이 가장 좋습니다.

Plan 9 운영체제 자료는 `http://plan9.bell-labs.com/plan9`, Linux/BSD/OSX 등으로 port한 Plan 9 application과 library는 `https://9fans.github.io/plan9port/`에서 확인합니다.

Resources
=========

Protocol specifications are maintained on github:
http://ericvh.github.com/9p-rfc/

9p client and server implementations are listed on
http://9p.cat-v.org/implementations

A 9p2000.L server is being developed by LLNL and can be found
at http://code.google.com/p/diod/

There are user and developer mailing lists available through the v9fs project
on sourceforge (http://sourceforge.net/projects/v9fs).

News and other information is maintained on a Wiki.
(http://sf.net/apps/mediawiki/v9fs/index.php).

Bug reports are best issued via the mailing list.

For more information on the Plan 9 Operating System check out
http://plan9.bell-labs.com/plan9

For information on Plan 9 from User Space (Plan 9 applications and libraries
ported to Linux/BSD/OSX/etc) check out https://9fans.github.io/plan9port/