ZoneFS - Zone filesystem for Zoned block devices

Zone을 file로 노출하는 단순 파일 시스템

zonefs는 zoned block device의 각 zone을 하나의 file로 노출하는 매우 단순한 파일 시스템이다. f2fs처럼 zoned block device를 native로 지원하면서 일반 POSIX 파일 시스템처럼 보이게 하는 구현과 달리, sequential write 제약을 사용자에게 숨기지 않는다.

Sequential write zone을 나타내는 file은 현재 file 끝부터 순서대로 써야 한다. 즉 write는 append-only이며, file offset을 임의로 골라 기존 데이터를 덮어쓰는 일반 random write를 허용하지 않는다.

따라서 zonefs는 완전한 POSIX 파일 시스템보다 raw block device access interface에 가깝다. 목표는 개발자에게 익숙하지 않을 수 있는 block device file ioctl과 sector range 관리를, 더 풍부한 file API와 `one file is one zone`이라는 상위 추상화로 바꾸는 것이다.

예를 들어 RocksDB나 LevelDB가 사용하는 LSM(log-structured merge) tree의 SSTable을 디스크 전체의 sector range로 직접 관리하는 대신 zone file에 일반 file처럼 저장할 수 있다. 이 대응은 application 수정량을 줄이고 서로 다른 programming language에서도 zoned storage 지원을 구현하기 쉽게 한다.

.. SPDX-License-Identifier: GPL-2.0

================================================
ZoneFS - Zone filesystem for Zoned block devices
================================================

Introduction
============

zonefs is a very simple file system exposing each zone of a zoned block device
as a file. Unlike a regular POSIX-compliant file system with native zoned block
device support (e.g. f2fs), zonefs does not hide the sequential write
constraint of zoned block devices to the user. Files representing sequential
write zones of the device must be written sequentially starting from the end
of the file (append only writes).

As such, zonefs is in essence closer to a raw block device access interface
than to a full-featured POSIX file system. The goal of zonefs is to simplify
the implementation of zoned block device support in applications by replacing
raw block device file accesses with a richer file API, avoiding relying on
direct block device file ioctls which may be more obscure to developers. One
example of this approach is the implementation of LSM (log-structured merge)
tree structures (such as used in RocksDB and LevelDB) on zoned block devices
by allowing SSTables to be stored in a zone file similarly to a regular file
system rather than as a range of sectors of the entire disk. The introduction
of the higher level construct "one file is one zone" can help reducing the
amount of changes needed in the application as well as introducing support for
different application programming languages.

Conventional zone과 sequential zone

Zoned storage device의 address space는 여러 zone으로 나뉜다. Zone은 연속한 LBA(Logical Block Address)의 집합이며, 모든 zone도 서로 맞닿아 있어 중간에 LBA gap이 없다. 각 zone은 conventional 또는 sequential 같은 type을 가진다.

Conventional zone의 LBA에는 접근 제약이 없다. 일반 block device처럼 zone 안의 임의 위치를 읽거나 쓸 수 있다. 반면 sequential zone은 random read를 허용하지만 write는 반드시 순서대로 수행해야 한다.

각 sequential zone은 device가 유지하는 write pointer를 가진다. 이 pointer가 다음 write의 의무적인 시작 LBA를 가리키므로 이미 기록한 LBA는 덮어쓸 수 없다. 다시 기록하려면 먼저 zone reset command로 zone을 지워 write pointer를 시작 위치로 되돌려야 한다.

현재 흔한 zoned storage는 SMR(Shingled Magnetic Recording) HDD에서 SCSI ZBC(Zoned Block Commands) 또는 ATA ZAC(Zoned ATA Commands) interface를 사용한다. SSD도 garbage collection에 따른 내부 write amplification을 줄이기 위해 zoned interface를 구현할 수 있으며, NVMe protocol에는 ZNS(Zoned Namespace)가 정의되어 있다.

Zoned block devices
-------------------

Zoned storage devices belong to a class of storage devices with an address
space that is divided into zones. A zone is a group of consecutive LBAs and all
zones are contiguous (there are no LBA gaps). Zones may have different types.

* Conventional zones: there are no access constraints to LBAs belonging to
  conventional zones. Any read or write access can be executed, similarly to a
  regular block device.
* Sequential zones: these zones accept random reads but must be written
  sequentially. Each sequential zone has a write pointer maintained by the
  device that keeps track of the mandatory start LBA position of the next write
  to the device. As a result of this write constraint, LBAs in a sequential zone
  cannot be overwritten. Sequential zones must first be erased using a special
  command (zone reset) before rewriting.

Zoned storage devices can be implemented using various recording and media
technologies. The most common form of zoned storage today uses the SCSI Zoned
Block Commands (ZBC) and Zoned ATA Commands (ZAC) interfaces on Shingled
Magnetic Recording (SMR) HDDs.

Solid State Disks (SSD) storage devices can also implement a zoned interface
to, for instance, reduce internal write amplification due to garbage collection.
The NVMe Zoned NameSpace (ZNS) is a technical proposal of the NVMe standard
committee aiming at adding a zoned storage interface to the NVMe protocol.

정적 file tree와 on-disk metadata

zonefs는 device의 zone을 file로, zone type을 sub-directory로 표현한다. File tree는 mount 시 device가 보고한 zone 정보만으로 전부 구성되므로 복잡한 on-disk metadata 구조가 필요하지 않다.

Disk에 영구 저장하는 metadata는 immutable superblock뿐이다. Superblock에는 magic number와 선택적인 feature flag·value가 들어간다. Mount할 때 `blkdev_report_zones()`로 device zone configuration을 얻어 정적 tree를 채우며, file size는 zone type과 device가 관리하는 write pointer 위치에서 결정한다.

Superblock은 항상 sector 0에 기록된다. 이를 저장하는 device의 첫 zone은 zone file로 노출하지 않는다. 첫 zone이 sequential zone이면 `mkzonefs` format tool은 그 zone을 `FULL` 상태로 finish하여 read-only로 만들고, superblock 뒤에 data가 추가로 기록되는 것을 막는다.

Zonefs Overview
===============

Zonefs exposes the zones of a zoned block device as files. The files
representing zones are grouped by zone type, which are themselves represented
by sub-directories. This file structure is built entirely using zone information
provided by the device and so does not require any complex on-disk metadata
structure.

On-disk metadata
----------------

zonefs on-disk metadata is reduced to an immutable super block which
persistently stores a magic number and optional feature flags and values. On
mount, zonefs uses blkdev_report_zones() to obtain the device zone configuration
and populates the mount point with a static file tree solely based on this
information. File sizes come from the device zone type and write pointer
position managed by the device itself.

The super block is always written on disk at sector 0. The first zone of the
device storing the super block is never exposed as a zone file by zonefs. If
the zone containing the super block is a sequential zone, the mkzonefs format
tool always "finishes" the zone, that is, it transitions the zone to a full
state to make it read-only, preventing any data write.

Zone type directory와 file naming

같은 type의 zone file은 mount 시 자동으로 만들어지는 하나의 sub-directory 아래에 모인다. Conventional zone은 `cnv`, sequential write zone은 `seq` directory를 사용한다.

`cnv`는 usable conventional zone이 있을 때만 생긴다. Device에 conventional zone이 sector 0의 첫 zone 하나뿐이면 그 zone은 superblock 저장에 사용되어 file로 노출되지 않으므로 `cnv` directory도 만들지 않는다.

Zonefs에 존재하는 directory는 `cnv`와 `seq`뿐이다. 사용자는 다른 directory를 만들 수 없고 두 directory를 rename하거나 delete할 수도 없다. `stat()` 또는 `fstat()`이 반환한 `struct stat`의 directory `st_size`는 그 아래에 존재하는 file 수를 나타낸다.

각 directory 안의 zone file 이름은 같은 type 집합에서의 zone 번호인 `0`, `1`, `2`, ... 형식이다. 번호가 커질수록 device의 zone start sector도 증가한다. File 최대 size, 즉 zone capacity를 넘는 read·write는 `-EFBIG`로 실패한다.

File과 sub-directory의 create·delete·rename 및 attribute 수정은 허용되지 않는다. `stat()`·`fstat()`이 보고하는 file block 수는 현재 기록량이 아니라 zone file의 capacity, 곧 최대 file size를 뜻한다.

Zone type sub-directories
-------------------------

Files representing zones of the same type are grouped together under the same
sub-directory automatically created on mount.

For conventional zones, the sub-directory "cnv" is used. This directory is
however created if and only if the device has usable conventional zones. If
the device only has a single conventional zone at sector 0, the zone will not
be exposed as a file as it will be used to store the zonefs super block. For
such devices, the "cnv" sub-directory will not be created.

For sequential write zones, the sub-directory "seq" is used.

These two directories are the only directories that exist in zonefs. Users
cannot create other directories and cannot rename nor delete the "cnv" and
"seq" sub-directories.

The size of the directories indicated by the st_size field of struct stat,
obtained with the stat() or fstat() system calls, indicates the number of files
existing under the directory.

Zone files
----------

Zone files are named using the number of the zone they represent within the set
of zones of a particular type. That is, both the "cnv" and "seq" directories
contain files named "0", "1", "2", ... The file numbers also represent
increasing zone start sector on the device.

All read and write operations to zone files are not allowed beyond the file
maximum size, that is, beyond the zone capacity. Any access exceeding the zone
capacity is failed with the -EFBIG error.

Creating, deleting, renaming or modifying any attribute of files and
sub-directories is not allowed.

The number of blocks of a file as reported by stat() and fstat() indicates the
capacity of the zone file, or in other words, the maximum file size.

Conventional·sequential zone file I/O

Conventional zone file의 size는 해당 zone size로 고정되며 truncate할 수 없다. File size 경계 안에서는 buffered I/O, direct I/O, `mmap` 등 모든 방식으로 random read·write할 수 있다.

`seq` directory의 sequential zone file size는 zone start sector를 기준으로 한 write pointer의 상대 위치다. 따라서 file size는 성공적으로 순차 기록된 data 끝을 나타내며 write가 진행될수록 증가한다.

Sequential file write는 file 끝, 또는 asynchronous I/O에서 이미 발행되어 아직 in-flight인 마지막 write의 끝에서 시작해야 한다. Zonefs는 random write를 순차 write로 바꾸지 않으며 start offset이 이 위치와 다르면 request를 실패시킨다.

Page cache의 dirty page writeback은 순차 순서를 보장하지 않으므로 sequential file에는 buffered write와 writable shared mapping을 금지하고 direct I/O write만 허용한다. Block layer에서는 `ELEVATOR_F_ZBD_SEQ_WRITE` feature를 구현한 elevator가 request를 device에 순서대로 전달해야 하며, zoned device 초기화 시 일반적으로 `mq-deadline`이 기본으로 설정된다.

Sequential file read에는 I/O 방식 제한이 없어 buffered I/O, direct I/O, shared read mapping을 모두 사용할 수 있다. Truncate는 두 값만 허용한다. 0으로 줄이면 zone reset으로 write pointer를 zone 시작으로 되돌리고, zone capacity로 늘리면 finish zone operation으로 zone을 `FULL` 상태로 전환한다.

Conventional zone files
-----------------------

The size of conventional zone files is fixed to the size of the zone they
represent. Conventional zone files cannot be truncated.

These files can be randomly read and written using any type of I/O operation:
buffered I/Os, direct I/Os, memory mapped I/Os (mmap), etc. There are no I/O
constraint for these files beyond the file size limit mentioned above.

Sequential zone files
---------------------

The size of sequential zone files grouped in the "seq" sub-directory represents
the file's zone write pointer position relative to the zone start sector.

Sequential zone files can only be written sequentially, starting from the file
end, that is, write operations can only be append writes. Zonefs makes no
attempt at accepting random writes and will fail any write request that has a
start offset not corresponding to the end of the file, or to the end of the last
write issued and still in-flight (for asynchronous I/O operations).

Since dirty page writeback by the page cache does not guarantee a sequential
write pattern, zonefs prevents buffered writes and writeable shared mappings
on sequential files. Only direct I/O writes are accepted for these files.
zonefs relies on the sequential delivery of write I/O requests to the device
implemented by the block layer elevator. An elevator implementing the sequential
write feature for zoned block device (ELEVATOR_F_ZBD_SEQ_WRITE elevator feature)
must be used. This type of elevator (e.g. mq-deadline) is set by default
for zoned block devices on device initialization.

There are no restrictions on the type of I/O used for read operations in
sequential zone files. Buffered I/Os, direct I/Os and shared read mappings are
all accepted.

Truncating sequential zone files is allowed only down to 0, in which case, the
zone is reset to rewind the file zone write pointer position to the start of
the zone, or up to the zone capacity, in which case the file's zone is
transitioned to the FULL state (finish zone operation).

Format 시 선택 기능

여러 선택 기능은 mount할 때가 아니라 zonefs를 format할 때 활성화한다. Format 결과는 immutable superblock과 mount 시 구성되는 file 속성에 반영되므로 변경하려면 device를 다시 format해야 할 수 있다.

Conventional zone aggregation을 사용하면 서로 연속한 conventional zone 범위를 기본적인 zone당 file 하나 대신 더 큰 file 하나로 합친다. Sequential zone의 one-file-per-zone 규칙을 바꾸는 option은 아니다.

Zone file owner는 기본적으로 UID 0, GID 0인 root지만 유효한 UID·GID로 바꿀 수 있다. File access permission의 기본값은 `0640`이며 format option으로 다른 permission을 지정할 수 있다.

Format options
--------------

Several optional features of zonefs can be enabled at format time.

* Conventional zone aggregation: ranges of contiguous conventional zones can be
  aggregated into a single larger file instead of the default one file per zone.
* File ownership: The owner UID and GID of zone files is by default 0 (root)
  but can be changed to any valid UID/GID.
* File access permissions: the default 640 access permissions can be changed.

Zoned device 고유 I/O 오류

Zoned block device도 bad sector처럼 일반 block device와 같은 원인으로 I/O request가 실패할 수 있다. 여기에 zoned standard가 정의하는 read-only·offline 상태 전이, write pointer 불일치, delayed cache error가 추가된다.

Zone이 `BLK_ZONE_COND_READONLY`로 바뀌면 이미 기록한 data는 읽을 수 있지만 더 쓸 수 없다. Zone management command나 read·write로 정상 read/write 상태로 되돌릴 수 없다. Standard는 전이 원인을 규정하지 않지만 HDD write head 결함으로 그 head 아래 zone 전체가 read-only가 되는 경우가 예다.

`BLK_ZONE_COND_OFFLINE` zone은 읽기와 쓰기가 모두 불가능하며 사용자 action으로 정상 상태에 복귀시킬 수 없다. HDD의 read-write head 결함으로 platter의 해당 영역에 접근하지 못하는 경우가 전형적인 예다.

Unaligned write error는 request 실행 시 start sector가 zone write pointer와 다를 때 발생한다. Zonefs가 append를 강제해도 큰 direct I/O가 여러 BIO·request로 나뉜 뒤 일부가 실패하거나 asynchronous write 묶음 중 하나가 실패하면, 뒤에 queue된 request가 모두 write pointer와 어긋나 연쇄 실패할 수 있다.

Device write cache가 켜져 있으면 이미 완료로 보고한 write 범위의 오류가 `fsync()` 같은 cache flush 시점에 늦게 드러날 수 있다. 이런 delayed write error도 문제 sector 뒤의 cached sequential stream을 무효화하여 data가 연속해서 버려질 수 있다.

IO error handling
-----------------

Zoned block devices may fail I/O requests for reasons similar to regular block
devices, e.g. due to bad sectors. However, in addition to such known I/O
failure pattern, the standards governing zoned block devices behavior define
additional conditions that result in I/O errors.

* A zone may transition to the read-only condition (BLK_ZONE_COND_READONLY):
  While the data already written in the zone is still readable, the zone can
  no longer be written. No user action on the zone (zone management command or
  read/write access) can change the zone condition back to a normal read/write
  state. While the reasons for the device to transition a zone to read-only
  state are not defined by the standards, a typical cause for such transition
  would be a defective write head on an HDD (all zones under this head are
  changed to read-only).

* A zone may transition to the offline condition (BLK_ZONE_COND_OFFLINE):
  An offline zone cannot be read nor written. No user action can transition an
  offline zone back to an operational good state. Similarly to zone read-only
  transitions, the reasons for a drive to transition a zone to the offline
  condition are undefined. A typical cause would be a defective read-write head
  on an HDD causing all zones on the platter under the broken head to be
  inaccessible.

* Unaligned write errors: These errors result from the host issuing write
  requests with a start sector that does not correspond to a zone write pointer
  position when the write request is executed by the device. Even though zonefs
  enforces sequential file write for sequential zones, unaligned write errors
  may still happen in the case of a partial failure of a very large direct I/O
  operation split into multiple BIOs/requests or asynchronous I/O operations.
  If one of the write request within the set of sequential write requests
  issued to the device fails, all write requests queued after it will
  become unaligned and fail.

* Delayed write errors: similarly to regular block devices, if the device side
  write cache is enabled, write errors may occur in ranges of previously
  completed writes when the device write cache is flushed, e.g. on fsync().
  Similarly to the previous immediate unaligned write error case, delayed write
  errors can propagate through a stream of cached sequential data for a zone
  causing all data to be dropped after the sector that caused the error.

All I/O errors detected by zonefs are notified to the user with an error code
return for the system call that triggered or detected the error. The recovery
actions taken by zonefs in response to I/O errors depend on the I/O type (read
vs write) and on the reason for the error (bad sector, unaligned writes or zone

오류 통지와 최소 복구

Zonefs가 감지한 모든 I/O 오류는 오류를 유발했거나 감지한 system call의 error code로 사용자에게 통지한다. 복구 action은 read·write I/O type과 bad sector, unaligned write, zone condition change 같은 원인에 따라 달라진다.

Read I/O 오류 자체에는 특별한 복구를 하지 않는다. 단, file zone이 여전히 good condition이고 inode size와 zone write pointer가 일치할 때에만 그렇다. 상태 또는 size 불일치가 발견되면 아래의 I/O error recovery를 실행한다. Write I/O 오류와 read-only·offline condition 전이는 항상 복구를 실행한다.

Sequential write의 immediate·delayed 오류는 inode size와 실제 성공 기록량을 다르게 만들 수 있다. Multi-BIO write가 일부만 성공했어도 전체 operation은 실패로 보고될 수 있으므로 write pointer가 전진한 만큼 inode size를 늘려야 이후 file 끝에서 write를 재시작할 수 있다.

반대로 `fsync()`에서 delayed error를 발견하면 실제로 영구 저장된 data가 inode size보다 적을 수 있다. Zonefs는 오류 후 file size를 zone에 persist된 양과 맞춘다. 즉 복구의 기준은 userspace에 과거 보고한 길이가 아니라 device가 보존한 data와 write pointer다.

Zone이 read-only가 되면 file permission도 read-only로 바꿔 attribute와 data 수정을 막는다. Offline zone은 read·write permission을 모두 제거한다. 이 최소 복구 위에 추가할 조치는 `errors=xxx` mount option이 결정한다.

condition change).

* For read I/O errors, zonefs does not execute any particular recovery action,
  but only if the file zone is still in a good condition and there is no
  inconsistency between the file inode size and its zone write pointer position.
  If a problem is detected, I/O error recovery is executed (see below table).

* For write I/O errors, zonefs I/O error recovery is always executed.

* A zone condition change to read-only or offline also always triggers zonefs
  I/O error recovery.

Zonefs minimal I/O error recovery may change a file size and file access
permissions.

* File size changes:
  Immediate or delayed write errors in a sequential zone file may cause the file
  inode size to be inconsistent with the amount of data successfully written in
  the file zone. For instance, the partial failure of a multi-BIO large write
  operation will cause the zone write pointer to advance partially, even though
  the entire write operation will be reported as failed to the user. In such
  case, the file inode size must be advanced to reflect the zone write pointer
  change and eventually allow the user to restart writing at the end of the
  file.
  A file size may also be reduced to reflect a delayed write error detected on
  fsync(): in this case, the amount of data effectively written in the zone may
  be less than originally indicated by the file inode size. After such I/O
  error, zonefs always fixes the file inode size to reflect the amount of data
  persistently stored in the file zone.

* Access permission changes:
  A zone condition change to read-only is indicated with a change in the file
  access permissions to render the file read-only. This disables changes to the

errors behavior별 post-error 상태

`errors=xxx`는 최소 복구 뒤 zonefs가 취할 추가 action을 선택한다. 원문의 ASCII 표는 아래 구조화 표와 같다. `fixed`는 device write pointer와 persist된 data에 맞춰 file size를 수정한다는 뜻이며, device zone의 read·write 가능 여부와 zone file permission을 구분해야 한다.

Option을 지정하지 않으면 `errors=remount-ro`가 기본이다. 이 behavior는 오류 file 하나가 아니라 모든 file permission을 read-only로 바꾸고 파일 시스템 전체를 read-only로 remount한다.

Device가 zone을 offline으로 전환하여 생긴 size·permission 변화는 영구적이다. `mkfs.zonefs`(`mkzonefs`)로 다시 format하거나 remount해도 offline zone file을 good state로 되돌릴 수 없다. Device가 read-only로 바꾼 zone의 write permission 제거도 마찬가지로 영구적이다.

반면 good condition zone에 `remount-ro`, `zone-ro`, `zone-offline` behavior가 추가한 permission 제한은 임시다. Unmount 후 remount하면 format 시 정한 기본 access right로 복구된다. `repair`는 good zone의 file size 수정이라는 최소 action만 수행하지만, device가 read-only·offline으로 보고한 zone의 permission 제한은 여전히 적용한다.

  file attributes and data modification. For offline zones, all permissions
  (read and write) to the file are disabled.

Further action taken by zonefs I/O error recovery can be controlled by the user
with the "errors=xxx" mount option. The table below summarizes the result of
zonefs I/O error processing depending on the mount option and on the zone
conditions::

    +--------------+-----------+-----------------------------------------+
    |              |           |            Post error state             |
    | "errors=xxx" |  device   |                 access permissions      |
    |    mount     |   zone    | file         file          device zone  |
    |    option    | condition | size     read    write    read    write |
    +--------------+-----------+-----------------------------------------+
    |              | good      | fixed    yes     no       yes     yes   |
    | remount-ro   | read-only | as is    yes     no       yes     no    |
    | (default)    | offline   |   0      no      no       no      no    |
    +--------------+-----------+-----------------------------------------+
    |              | good      | fixed    yes     no       yes     yes   |
    | zone-ro      | read-only | as is    yes     no       yes     no    |
    |              | offline   |   0      no      no       no      no    |
    +--------------+-----------+-----------------------------------------+
    |              | good      |   0      no      no       yes     yes   |
    | zone-offline | read-only |   0      no      no       yes     no    |
    |              | offline   |   0      no      no       no      no    |
    +--------------+-----------+-----------------------------------------+
    |              | good      | fixed    yes     yes      yes     yes   |
    | repair       | read-only | as is    yes     no       yes     no    |
    |              | offline   |   0      no      no       no      no    |
    +--------------+-----------+-----------------------------------------+

Further notes:

* The "errors=remount-ro" mount option is the default behavior of zonefs I/O
  error processing if no errors mount option is specified.
* With the "errors=remount-ro" mount option, the change of the file access
  permissions to read-only applies to all files. The file system is remounted
  read-only.
* Access permission and file size changes due to the device transitioning zones
  to the offline condition are permanent. Remounting or reformatting the device
  with mkfs.zonefs (mkzonefs) will not change back offline zone files to a good
  state.
* File access permission changes to read-only due to the device transitioning
  zones to the read-only condition are permanent. Remounting or reformatting
  the device will not re-enable file write access.
* File access permission changes implied by the remount-ro, zone-ro and
  zone-offline mount options are temporary for zones in a good condition.
  Unmounting and remounting the file system will restore the previous default
  (format time values) access rights to the files affected.
* The repair mount option triggers only the minimal set of I/O error recovery
  actions, that is, file size fixes for zones in a good condition. Zones
  indicated as being read-only or offline by the device still imply changes to
  the zone file access permissions as noted in the table above.

errors mount option과 mount-time 차이

Zonefs mount option은 `errors=<behavior>`와 `explicit-open`이다. `errors=<behavior>`는 I/O error, inode size inconsistency, zone condition change에 대응하는 policy를 `remount-ro`, `zone-ro`, `zone-offline`, `repair` 중에서 고르게 하며 기본값은 `remount-ro`다.

앞 절의 표는 run-time I/O error action을 설명한다. Mount 중 발생한 I/O 오류는 behavior와 관계없이 mount operation 자체를 실패시킨다. Read-only zone 처리도 발견 시점에 따라 다르다.

Mount time에 read-only zone을 발견하면 offline zone과 동일하게 모든 access를 끄고 zone file size를 0으로 설정한다. ZBC와 ZAC standard에서 read-only zone의 write pointer는 invalid로 정의되어 이미 기록된 data 양을 알아낼 수 없기 때문이다.

Run-time에 read-only로 전환된 zone은 마지막으로 갱신된 zone file size를 그대로 둔다. 원문에는 write pointer를 `invalib`라고 쓴 오탈자가 있지만 의미는 `invalid`다. 번역에서는 원문의 철자를 source block에 보존하고 기술적 의미를 이 문장처럼 명확히 한다.

Mount options
-------------

zonefs defines several mount options:
* errors=<behavior>
* explicit-open

"errors=<behavior>" option
~~~~~~~~~~~~~~~~~~~~~~~~~~

The "errors=<behavior>" option mount option allows the user to specify zonefs
behavior in response to I/O errors, inode size inconsistencies or zone
condition changes. The defined behaviors are as follow:

* remount-ro (default)
* zone-ro
* zone-offline
* repair

The run-time I/O error actions defined for each behavior are detailed in the
previous section. Mount time I/O errors will cause the mount operation to fail.
The handling of read-only zones also differs between mount-time and run-time.
If a read-only zone is found at mount time, the zone is always treated in the
same manner as offline zones, that is, all accesses are disabled and the zone
file size set to 0. This is necessary as the write pointer of read-only zones
is defined as invalib by the ZBC and ZAC standards, making it impossible to
discover the amount of data that has been written to the zone. In the case of a
read-only zone discovered at run-time, as indicated in the previous section.
The size of the zone file is left unchanged from its last updated value.

Active zone 제한과 explicit-open

NVMe ZNS 같은 zoned block device는 active zone 수를 제한할 수 있다. Active에는 implicit open, explicit open, closed condition의 zone이 포함된다. 아직 active가 아닌 file zone에 write를 시작할 때 이 limit를 넘으면 application이 write I/O error를 볼 수 있다.

`explicit-open` mount option은 file을 처음 write-open할 때 open zone command를 보내 zone을 미리 active로 만든다. Open command가 성공하면 application은 해당 file의 write request를 device가 처리할 수 있음을 보장받는다.

반대로 zone file의 마지막 `close()`가 실행될 때 zone이 full도 empty도 아니면 zone close command를 device에 보낸다. 이 방식은 file descriptor 수명과 device의 explicit-open resource를 연결하여 open zone limit을 예측 가능하게 관리한다.

"explicit-open" option
~~~~~~~~~~~~~~~~~~~~~~

A zoned block device (e.g. an NVMe Zoned Namespace device) may have limits on
the number of zones that can be active, that is, zones that are in the
implicit open, explicit open or closed conditions.  This potential limitation
translates into a risk for applications to see write IO errors due to this
limit being exceeded if the zone of a file is not already active when a write
request is issued by the user.

To avoid these potential errors, the "explicit-open" mount option forces zones
to be made active using an open zone command when a file is opened for writing
for the first time. If the zone open command succeeds, the application is then
guaranteed that write requests can be processed. Conversely, the
"explicit-open" mount option will result in a zone close command being issued
to the device on the last close() of a zone file if the zone is not full nor
empty.

Runtime sysfs counter

Mount된 device마다 user-readable zonefs attribute가 `/sys/fs/zonefs/<dev>/` 아래에 생긴다. `<dev>`는 mount한 zoned block device 이름이며, counter는 open-for-write 제한과 active-zone 제한을 구분해 보여 준다.

`max_wro_seq_files`는 device가 지원하는 explicit·implicit open zone의 최대 수다. 값이 0이면 제한이 없다. `explicit-open` 사용 중 현재 write-open 수가 limit에 도달하면 zonefs가 새 sequential file의 write `open()`을 실패시킨다.

`nr_wro_seq_files`는 현재 write-open file 수다. `explicit-open`을 쓰면 절대 maximum을 넘지 않는다. 쓰지 않으면 보고값이 maximum보다 클 수 있으므로 application이 동시에 실제 write하는 file 수를 limit 이하로 제한해야 하며, 지키지 않으면 write error가 발생할 수 있다.

`max_active_seq_files`는 partially written, 즉 empty도 full도 아닌 zone file과 explicit-open zone을 합친 active file 최대 수이며 device의 maximum active zones와 같다. `nr_active_seq_files`는 현재 active file 수다.

원문 마지막 문장은 `nr_active_seq_files`가 자기 자신을 넘을 수 없다고 적었지만 문맥상 maximum인 `max_active_seq_files`를 뜻한다. 이 명백한 식별자 오탈자는 영어 source에는 그대로 보존하고, 해설과 구조화 표에서는 `max_active_seq_files != 0`일 때 `nr_active_seq_files <= max_active_seq_files`라는 실제 관계로 제시한다.

Runtime sysfs attributes
------------------------

zonefs defines several sysfs attributes for mounted devices.  All attributes
are user readable and can be found in the directory /sys/fs/zonefs/<dev>/,
where <dev> is the name of the mounted zoned block device.

The attributes defined are as follows.

* **max_wro_seq_files**:  This attribute reports the maximum number of
  sequential zone files that can be open for writing.  This number corresponds
  to the maximum number of explicitly or implicitly open zones that the device
  supports.  A value of 0 means that the device has no limit and that any zone
  (any file) can be open for writing and written at any time, regardless of the
  state of other zones.  When the *explicit-open* mount option is used, zonefs
  will fail any open() system call requesting to open a sequential zone file for
  writing when the number of sequential zone files already open for writing has
  reached the *max_wro_seq_files* limit.
* **nr_wro_seq_files**:  This attribute reports the current number of sequential
  zone files open for writing.  When the "explicit-open" mount option is used,
  this number can never exceed *max_wro_seq_files*.  If the *explicit-open*
  mount option is not used, the reported number can be greater than
  *max_wro_seq_files*.  In such case, it is the responsibility of the
  application to not write simultaneously more than *max_wro_seq_files*
  sequential zone files.  Failure to do so can result in write errors.
* **max_active_seq_files**:  This attribute reports the maximum number of
  sequential zone files that are in an active state, that is, sequential zone
  files that are partially written (not empty nor full) or that have a zone that
  is explicitly open (which happens only if the *explicit-open* mount option is
  used).  This number is always equal to the maximum number of active zones that
  the device supports.  A value of 0 means that the mounted device has no limit
  on the number of sequential zone files that can be active.
* **nr_active_seq_files**:  This attributes reports the current number of
  sequential zone files that are active. If *max_active_seq_files* is not 0,
  then the value of *nr_active_seq_files* can never exceed the value of
  *nr_active_seq_files*, regardless of the use of the *explicit-open* mount
  option.

User space format·test 도구

`mkzonefs`는 zoned block device를 zonefs용으로 format하는 user space tool이다. Source와 배포 정보는 `https://github.com/damien-lemoal/zonefs-tools`에서 제공한다.

같은 `zonefs-tools` project에는 test suite도 포함된다. 실제 zoned block device뿐 아니라 zoned mode로 만든 `null_blk` block device에도 실행할 수 있어 hardware 없이 format·mount·I/O 동작과 error path를 검증할 수 있다.

운영 device에 test suite를 적용할 때는 format과 destructive zone operation 가능성을 먼저 확인해야 한다. 이 문서는 도구 위치와 적용 대상을 설명하며, 실제 command option과 안전 절차는 해당 tool documentation을 따른다.

Zonefs User Space Tools
=======================

The mkzonefs tool is used to format zoned block devices for use with zonefs.
This tool is available on Github at:

https://github.com/damien-lemoal/zonefs-tools

zonefs-tools also includes a test suite which can be run against any zoned
block device, including null_blk block device created with zoned mode.

15TB SMR HDD format과 conventional aggregation

예시는 zone 크기가 256MB인 15TB host-managed SMR HDD를 conventional zone aggregation 기능과 함께 format한다. `mkzonefs -o aggr_cnv /dev/sdX`로 format하고 zonefs로 mount한 뒤 root directory를 확인한다.

출력에서 `cnv` directory의 `st_size`는 1, `seq`의 `st_size`는 55356이다. Directory size가 byte 용량이 아니라 해당 type의 zone file 개수라는 앞의 계약을 실제 `ls -l` 결과로 보여 준다.

모든 conventional zone은 aggregation되어 `/mnt/cnv/0` file 하나가 되고 size는 140391743488 bytes다. 이 aggregated conventional file은 random I/O가 가능한 일반 file처럼 다룰 수 있어 그 안에 ext4를 만들고 loop mount할 수도 있다.

# mkzonefs -o aggr_cnv /dev/sdX
# mount -t zonefs /dev/sdX /mnt
# mkfs.ext4 /mnt/cnv/0
# mount -o loop /mnt/cnv/0 /data

Examples
--------

The following formats a 15TB host-managed SMR HDD with 256 MB zones
with the conventional zones aggregation feature enabled::

    # mkzonefs -o aggr_cnv /dev/sdX
    # mount -t zonefs /dev/sdX /mnt
    # ls -l /mnt/
    total 0
    dr-xr-xr-x 2 root root     1 Nov 25 13:23 cnv
    dr-xr-xr-x 2 root root 55356 Nov 25 13:23 seq

The size of the zone files sub-directories indicate the number of files
existing for each type of zones. In this example, there is only one
conventional zone file (all conventional zones are aggregated under a single
file)::

    # ls -l /mnt/cnv
    total 137101312
    -rw-r----- 1 root root 140391743488 Nov 25 13:23 0

This aggregated conventional zone file can be used as a regular file::

    # mkfs.ext4 /mnt/cnv/0
    # mount -o loop /mnt/cnv/0 /data

The "seq" sub-directory grouping files for sequential write zones has in this
example 55356 zones::

    # ls -lv /mnt/seq

Sequential append와 truncate

`seq` directory에는 예제 device의 sequential write zone 55356개가 file `0`부터 `55355`까지 나타난다. 아직 기록하지 않은 zone file size는 0이지만 `ls`의 total과 `stat` block 수는 각 zone capacity를 반영한다.

예제는 `dd`에 `oflag=direct`를 지정하여 `/mnt/seq/0` 끝에 4096 bytes를 직접 쓴다. 성공 후 file size가 4096으로 증가하며 이는 device write pointer가 zone 시작에서 4096 bytes만큼 전진했음을 뜻한다.

File을 268435456 bytes, 즉 256MB zone capacity로 truncate하면 zone finish operation이 실행되어 추가 write를 막는다. 반대로 size 0으로 truncate하면 zone reset으로 storage space를 비우고 append write를 처음부터 다시 시작할 수 있다.

    total 14511243264
    -rw-r----- 1 root root 0 Nov 25 13:23 0
    -rw-r----- 1 root root 0 Nov 25 13:23 1
    -rw-r----- 1 root root 0 Nov 25 13:23 2
    ...
    -rw-r----- 1 root root 0 Nov 25 13:23 55354
    -rw-r----- 1 root root 0 Nov 25 13:23 55355

For sequential write zone files, the file size changes as data is appended at
the end of the file, similarly to any regular file system::

    # dd if=/dev/zero of=/mnt/seq/0 bs=4096 count=1 conv=notrunc oflag=direct
    1+0 records in
    1+0 records out
    4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00044121 s, 9.3 MB/s

    # ls -l /mnt/seq/0
    -rw-r----- 1 root root 4096 Nov 25 13:23 /mnt/seq/0

The written file can be truncated to the zone size, preventing any further
write operation::

    # truncate -s 268435456 /mnt/seq/0
    # ls -l /mnt/seq/0
    -rw-r----- 1 root root 268435456 Nov 25 13:49 /mnt/seq/0

Truncation to 0 size allows freeing the file zone storage space and restart
append-writes to the file::

stat block 수와 최대 file size

Zonefs file은 disk zone에 정적으로 대응되므로 `stat()`·`fstat()`이 보고하는 `Blocks`는 현재 file size가 아니라 zone capacity를 나타낸다. 예제 file은 size 0이어도 `Blocks: 524288`을 보고한다.

POSIX `stat`의 block count 단위는 512 bytes이므로 최대 file size는 `524288 * 512 B = 256 MB`다. 이는 예제 device의 zone capacity와 정확히 일치한다. Application은 current data length인 `st_size`와 최대 기록 가능량인 `st_blocks * 512`를 구분해야 한다.

출력의 `IO Block: 4096`은 capacity가 아니라 write에 필요한 minimum I/O size를 뜻하며 device physical sector size와 대응한다. Sequential append request는 이 alignment까지 충족해야 device가 요구하는 write granularity를 지킬 수 있다.

    # truncate -s 0 /mnt/seq/0
    # ls -l /mnt/seq/0
    -rw-r----- 1 root root 0 Nov 25 13:49 /mnt/seq/0

Since files are statically mapped to zones on the disk, the number of blocks
of a file as reported by stat() and fstat() indicates the capacity of the file
zone::

    # stat /mnt/seq/0
    File: /mnt/seq/0
    Size: 0                 Blocks: 524288     IO Block: 4096   regular empty file
    Device: 870h/2160d        Inode: 50431       Links: 1
    Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-11-25 13:23:57.048971997 +0900
    Modify: 2019-11-25 13:52:25.553805765 +0900
    Change: 2019-11-25 13:52:25.553805765 +0900
    Birth: -

The number of blocks of the file ("Blocks") in units of 512B blocks gives the
maximum file size of 524288 * 512 B = 256 MB, corresponding to the device zone
capacity in this example. Of note is that the "IO block" field always
indicates the minimum I/O size for writes and corresponds to the device
physical sector size.