NILFS2

Continuous snapshot LFS와 userland tool

NILFS2는 continuous snapshot을 지원하는 log-structured filesystem(LFS)입니다. Filesystem 전체를 versioning할 뿐 아니라 몇 초 전에 잘못 덮어쓰거나 삭제한 file도 복원할 수 있습니다. 전통적 LFS처럼 consistency를 유지하므로 system crash 뒤 빠르게 복구합니다.

변경이 있는 동안 NILFS2는 수 초마다 또는 synchronous write마다 checkpoint를 만듭니다. 사용자는 연속 checkpoint 중 중요한 version을 snapshot으로 바꿀 수 있으며 다시 checkpoint로 되돌리기 전까지 보존됩니다.

Volume이 가득 차기 전까지 snapshot 수에 별도 제한은 없습니다. 각 snapshot은 writable mount와 동시에 read-only filesystem으로 mount할 수 있어 online backup에 편리합니다.

Userland tool은 `nilfs-utils` package에 들어 있습니다. 최소한 `mkfs.nilfs2`, `mount.nilfs2`, `umount.nilfs2`, cleaner 또는 garbage collector인 `nilfs_cleanerd`가 필요하며 자세한 내용은 package의 man page를 참조합니다.

원문은 NILFS project web page, download page와 linux-nilfs mailing-list 정보를 제공합니다.

.. SPDX-License-Identifier: GPL-2.0

======
NILFS2
======

NILFS2 is a log-structured file system (LFS) supporting continuous
snapshotting.  In addition to versioning capability of the entire file
system, users can even restore files mistakenly overwritten or
destroyed just a few seconds ago.  Since NILFS2 can keep consistency
like conventional LFS, it achieves quick recovery after system
crashes.

NILFS2 creates a number of checkpoints every few seconds or per
synchronous write basis (unless there is no change).  Users can select
significant versions among continuously created checkpoints, and can
change them into snapshots which will be preserved until they are
changed back to checkpoints.

There is no limit on the number of snapshots until the volume gets
full.  Each snapshot is mountable as a read-only file system
concurrently with its writable mount, and this feature is convenient
for online backup.

The userland tools are included in nilfs-utils package, which is
available from the following download page.  At least "mkfs.nilfs2",
"mount.nilfs2", "umount.nilfs2", and "nilfs_cleanerd" (so called
cleaner or garbage collector) are required.  Details on the tools are
described in the man pages included in the package.

:Project web page:    https://nilfs.sourceforge.io/
:Download page:       https://nilfs.sourceforge.io/en/download.html
:List info:           http://vger.kernel.org/vger-lists.html#linux-nilfs

현재 지원하지 않는 기능

현재 NILFS2가 아직 지원하지 않는 기능은 access time 갱신인 `atime`, extended attribute, POSIX ACL, quota, `fsck`, defragmentation입니다.

따라서 deployment 전에 application이 xattr·ACL·quota에 의존하는지 확인해야 하며, conventional offline fsck나 defragmentation tool을 기대해서는 안 됩니다.

Caveats
=======

Features which NILFS2 does not support yet:

        - atime
        - extended attributes
        - POSIX ACLs
        - quotas
        - fsck
        - defragmentation

NILFS2 mount option

표에서 `(*)`는 default를 뜻합니다. 기본 `barrier`는 write barrier를 사용하고 `nobarrier`는 끕니다. I/O stack이 barrier를 지원해야 하며 barrier write에서 error가 발생하면 NILFS2는 warning을 내고 barrier를 다시 비활성화합니다.

Filesystem error 대응은 `errors=continue`, 기본 `errors=remount-ro`, `errors=panic` 가운데 선택합니다. 각각 계속 진행, read-only remount, machine panic과 halt를 의미합니다.

`cp=n`은 mount할 snapshot의 checkpoint number를 지정합니다. `lscp`로 checkpoint와 snapshot을 확인하며 snapshot 표시가 된 checkpoint만 이 option으로 mount할 수 있습니다. Snapshot은 read-only이므로 read-only mount option도 함께 지정해야 합니다.

기본 `order=relaxed`는 metadata update가 없을 때 checkpoint를 만들지 않고 modified data block을 disk에 쓸 수 있습니다. Ext3 ordered data mode와 비슷하지만 data-block update atomicity는 보존하며 overwrite synchronous-write 성능을 높입니다.

`order=strict`는 data overwrite를 포함한 모든 file operation 순서를 보존해 crash recovery 뒤 event overtaking이 없음을 보장합니다. `norecovery`는 mount 시 recovery를 끄고 read-only mount나 snapshot의 device write를 모두 막지만, unclean volume의 read/write mount에서는 실패합니다.

`discard`는 block을 해제할 때 underlying block device에 discard/TRIM을 보내 SSD와 sparse/thin-provisioned LUN에 유용합니다. 기본 `nodiscard`는 이를 보내지 않습니다.

Mount options
=============

NILFS2 supports the following mount options:
(*) == default

======================= =======================================================
barrier(*)                This enables/disables the use of write barriers.  This
nobarrier                requires an IO stack which can support barriers, and
                        if nilfs gets an error on a barrier write, it will
                        disable again with a warning.
errors=continue                Keep going on a filesystem error.
errors=remount-ro(*)        Remount the filesystem read-only on an error.
errors=panic                Panic and halt the machine if an error occurs.
cp=n                        Specify the checkpoint-number of the snapshot to be
                        mounted.  Checkpoints and snapshots are listed by lscp
                        user command.  Only the checkpoints marked as snapshot
                        are mountable with this option.  Snapshot is read-only,
                        so a read-only mount option must be specified together.
order=relaxed(*)        Apply relaxed order semantics that allows modified data
                        blocks to be written to disk without making a
                        checkpoint if no metadata update is going.  This mode
                        is equivalent to the ordered data mode of the ext3
                        filesystem except for the updates on data blocks still
                        conserve atomicity.  This will improve synchronous
                        write performance for overwriting.
order=strict                Apply strict in-order semantics that preserves sequence
                        of all file operations including overwriting of data
                        blocks.  That means, it is guaranteed that no
                        overtaking of events occurs in the recovered file
                        system after a crash.
norecovery                Disable recovery of the filesystem on mount.
                        This disables every write access on the device for
                        read-only mounts or snapshots.  This option will fail
                        for r/w mounts on an unclean volume.
discard                        This enables/disables the use of discard/TRIM commands.
nodiscard(*)                The discard/TRIM commands are sent to the underlying
                        block device when blocks are freed.  This is useful
                        for SSD devices and sparse/thinly-provisioned LUNs.
======================= =======================================================

NILFS2 전용 ioctl

Application은 system-call interface를 통해 NILFS2 고유 기능을 사용할 수 있습니다. Checkpoint mode·삭제·조회, segment usage, virtual/block address, garbage collection, sync, resize와 allocation range를 전용 ioctl로 제어합니다.

`NILFS_IOCTL_CHANGE_CPMODE`는 checkpoint와 snapshot mode를 전환하며 `chcp`, `mkcp`가 사용합니다. `NILFS_IOCTL_DELETE_CHECKPOINT`는 checkpoint를 제거하며 `rmcp`가 사용합니다.

`NILFS_IOCTL_GET_CPINFO`와 `NILFS_IOCTL_GET_CPSTAT`은 checkpoint 정보와 통계를 반환합니다. `lscp`, `rmcp`, `nilfs_cleanerd`가 목적에 따라 사용합니다.

Segment 관련 `GET_SUINFO`, `SET_SUINFO`, `GET_SUSTAT`은 segment usage 정보·통계를 조회하거나 수정합니다. `SET_SUINFO`는 cleaner가 불필요한 segment cleaning을 건너뛰어 in-use block의 중복 이동으로 인한 성능 저하와 flash wear를 줄이게 합니다.

`GET_VINFO`는 virtual block address 정보, `GET_BDESCS`는 disk block number descriptor 정보를 반환합니다. `CLEAN_SEGMENTS`는 user-space parameter로 garbage collection을 수행합니다.

`NILFS_IOCTL_SYNC`는 checkpoint를 만들고, `NILFS_IOCTL_RESIZE`는 volume을 resize합니다. `NILFS_IOCTL_SET_ALLOC_RANGE`는 segment lower·upper limit를 byte 단위로 지정합니다.

Ioctls
======

There is some NILFS2 specific functionality which can be accessed by applications
through the system call interfaces. The list of all NILFS2 specific ioctls are
shown in the table below.

Table of NILFS2 specific ioctls:

 ============================== ===============================================
 Ioctl                                Description
 ============================== ===============================================
 NILFS_IOCTL_CHANGE_CPMODE      Change mode of given checkpoint between
                                checkpoint and snapshot state. This ioctl is
                                used in chcp and mkcp utilities.

 NILFS_IOCTL_DELETE_CHECKPOINT  Remove checkpoint from NILFS2 file system.
                                This ioctl is used in rmcp utility.

 NILFS_IOCTL_GET_CPINFO         Return info about requested checkpoints. This
                                ioctl is used in lscp utility and by
                                nilfs_cleanerd daemon.

 NILFS_IOCTL_GET_CPSTAT         Return checkpoints statistics. This ioctl is
                                used by lscp, rmcp utilities and by
                                nilfs_cleanerd daemon.

 NILFS_IOCTL_GET_SUINFO         Return segment usage info about requested
                                segments. This ioctl is used in lssu,
                                nilfs_resize utilities and by nilfs_cleanerd
                                daemon.

 NILFS_IOCTL_SET_SUINFO         Modify segment usage info of requested
                                segments. This ioctl is used by
                                nilfs_cleanerd daemon to skip unnecessary
                                cleaning operation of segments and reduce
                                performance penalty or wear of flash device
                                due to redundant move of in-use blocks.

 NILFS_IOCTL_GET_SUSTAT         Return segment usage statistics. This ioctl
                                is used in lssu, nilfs_resize utilities and
                                by nilfs_cleanerd daemon.

 NILFS_IOCTL_GET_VINFO          Return information on virtual block addresses.
                                This ioctl is used by nilfs_cleanerd daemon.

 NILFS_IOCTL_GET_BDESCS         Return information about descriptors of disk
                                block numbers. This ioctl is used by
                                nilfs_cleanerd daemon.

 NILFS_IOCTL_CLEAN_SEGMENTS     Do garbage collection operation in the
                                environment of requested parameters from
                                userspace. This ioctl is used by
                                nilfs_cleanerd daemon.

 NILFS_IOCTL_SYNC               Make a checkpoint. This ioctl is used in
                                mkcp utility.

 NILFS_IOCTL_RESIZE             Resize NILFS2 volume. This ioctl is used
                                by nilfs_resize utility.

 NILFS_IOCTL_SET_ALLOC_RANGE    Define lower limit of segments in bytes and
                                upper limit of segments in bytes. This ioctl
                                is used by nilfs_resize utility.
 ============================== ===============================================

생성, mount와 snapshot 관리

Local filesystem으로 사용하려면 block device에 NILFS2를 만들고 directory에 mount합니다. `mount.nilfs2` helper가 cleaner daemon도 함께 시작합니다.

# mkfs -t nilfs2 /dev/block_device
# mount -t nilfs2 /dev/block_device /dir

`lscp`는 checkpoint·snapshot 목록, `mkcp`는 checkpoint·snapshot 생성, `chcp`는 checkpoint와 snapshot의 상호 전환, `rmcp`는 지정 checkpoint invalidation에 사용합니다. 각 man page는 `nilfs-utils`에 포함됩니다.

Snapshot은 checkpoint number를 `cp=<cno>`로 지정하고 read-only `-r` option과 함께 mount합니다.

# mount -t nilfs2 -r -o cp=<cno> /dev/block_device /snap_dir

Writable mount나 snapshot을 `umount`하면 `umount.nilfs2` helper가 cleaner daemon을 자동으로 종료합니다.

# umount /dir

NILFS2 usage
============

To use nilfs2 as a local file system, simply::

 # mkfs -t nilfs2 /dev/block_device
 # mount -t nilfs2 /dev/block_device /dir

This will also invoke the cleaner through the mount helper program
(mount.nilfs2).

Checkpoints and snapshots are managed by the following commands.
Their manpages are included in the nilfs-utils package above.

  ====     ===========================================================
  lscp     list checkpoints or snapshots.
  mkcp     make a checkpoint or a snapshot.
  chcp     change an existing checkpoint to a snapshot or vice versa.
  rmcp     invalidate specified checkpoint(s).
  ====     ===========================================================

To mount a snapshot::

 # mount -t nilfs2 -r -o cp=<cno> /dev/block_device /snap_dir

where <cno> is the checkpoint number of the snapshot.

To unmount the NILFS2 mount point or snapshot, simply::

 # umount /dir

Then, the cleaner daemon is automatically shut down by the umount
helper program (umount.nilfs2).

Segment, log와 file payload 배치

NILFS2 volume은 superblock(SB)과 segment 0 주변의 예외 영역을 제외하고 같은 크기의 segment로 나뉩니다. 원문의 4 KiB block 예에서는 SB가 1 KiB, segment 0이 4 KiB에서 시작하고 segment 경계는 8 MiB 간격으로 이어집니다.

Segment는 log container이며 각 log는 summary information block, payload block, 선택적인 super root block(SR)으로 구성됩니다.

Payload block은 file별로 묶이며 각 file은 data block과 B-tree node block으로 구성됩니다. Log에는 modified block만 기록하므로 어떤 file group에는 data block 또는 B-tree node block이 없을 수도 있습니다.

Disk format
===========

A nilfs2 volume is equally divided into a number of segments except
for the super block (SB) and segment #0.  A segment is the container
of logs.  Each log is composed of summary information blocks, payload
blocks, and an optional super root block (SR)::

   ______________________________________________________
  | |SB| | Segment | Segment | Segment | ... | Segment | |
  |_|__|_|____0____|____1____|____2____|_____|____N____|_|
  0 +1K +4K       +8M       +16M      +24M  +(8MB x N)
       .             .            (Typical offsets for 4KB-block)
    .                  .
  .______________________.
  | log | log |... | log |
  |__1__|__2__|____|__m__|
        .       .
      .               .
    .                       .
  .______________________________.
  | Summary | Payload blocks  |SR|
  |_blocks__|_________________|__|

The payload blocks are organized per file, and each file consists of
data blocks and B-tree node blocks::

    |<---       File-A        --->|<---       File-B        --->|
   _______________________________________________________________
    | Data blocks | B-tree blocks | Data blocks | B-tree blocks | ...
   _|_____________|_______________|_____________|_______________|_

Summary block, metadata file과 atomic log

Block organization은 summary information block에 기록됩니다. Summary는 header `nilfs_segment_summary`, file별 `nilfs_finfo`, block별 `nilfs_binfo` structure를 포함합니다.

Log에는 regular file, directory, symbolic link와 filesystem metadata를 유지하는 여러 metadata file이 포함됩니다. 현재 metadata file은 on-disk inode를 저장하는 inode file `ifile`, checkpoint를 저장하는 `cpfile`, segment allocation state를 저장하는 `sufile`, virtual block number를 일반 block number에 mapping해 on-disk block relocation을 가능하게 하는 `DAT`입니다.

File sequence가 segment boundary를 넘으면 여러 log로 분할될 수 있습니다. 논리적으로 하나로 취급해야 하는 log sequence는 segment summary의 flag로 경계를 표시하며 NILFS2 recovery code가 이를 읽어 update atomicity를 보장합니다.

Since only the modified blocks are written in the log, it may have
files without data blocks or B-tree node blocks.

The organization of the blocks is recorded in the summary information
blocks, which contains a header structure (nilfs_segment_summary), per
file structures (nilfs_finfo), and per block structures (nilfs_binfo)::

  _________________________________________________________________________
 | Summary | finfo | binfo | ... | binfo | finfo | binfo | ... | binfo |...
 |_blocks__|___A___|_(A,1)_|_____|(A,Na)_|___B___|_(B,1)_|_____|(B,Nb)_|___


The logs include regular files, directory files, symbolic link files
and several meta data files.  The meta data files are the files used
to maintain file system meta data.  The current version of NILFS2 uses
the following meta data files::

 1) Inode file (ifile)             -- Stores on-disk inodes
 2) Checkpoint file (cpfile)       -- Stores checkpoints
 3) Segment usage file (sufile)    -- Stores allocation state of segments
 4) Data address translation file  -- Maps virtual block numbers to usual
    (DAT)                             block numbers.  This file serves to
                                      make on-disk blocks relocatable.

The following figure shows a typical organization of the logs::

  _________________________________________________________________________
 | Summary | regular file | file  | ... | ifile | cpfile | sufile | DAT |SR|
 |_blocks__|_or_directory_|_______|_____|_______|________|________|_____|__|


To stride over segment boundaries, this sequence of files may be split
into multiple logs.  The sequence of logs that should be treated as
logically one log, is delimited with flags marked in the segment
summary.  The recovery code of nilfs2 looks this boundary information
to ensure atomicity of updates.

Super root와 checkpoint별 inode 계층

Super root block은 모든 checkpoint마다 삽입되며 `DAT`, `cpfile`, `sufile`의 special inode 세 개를 포함합니다.

Regular file, directory, symlink와 다른 special file의 inode는 `ifile`에 들어갑니다. `ifile` 자신의 inode는 해당 checkpoint의 `cpfile` entry에 포함됩니다.

각 on-disk file format의 상세 정의는 `include/uapi/linux/nilfs2_ondisk.h`를 참조합니다.

NILFS2 설계에는 보호하려는 patent나 다른 intellectual property가 없습니다. 다른 operating system도 이 format의 data를 mount·read·write할 수 있기를 바라며 설계를 복제하는 것을 허용합니다.

The super root block is inserted for every checkpoints.  It includes
three special inodes, inodes for the DAT, cpfile, and sufile.  Inodes
of regular files, directories, symlinks and other special files, are
included in the ifile.  The inode of ifile itself is included in the
corresponding checkpoint entry in the cpfile.  Thus, the hierarchy
among NILFS2 files can be depicted as follows::

  Super block (SB)
       |
       v
  Super root block (the latest cno=xx)
       |-- DAT
       |-- sufile
       `-- cpfile
              |-- ifile (cno=c1)
              |-- ifile (cno=c2) ---- file (ino=i1)
              :        :          |-- file (ino=i2)
              `-- ifile (cno=xx)  |-- file (ino=i3)
                                  :        :
                                  `-- file (ino=yy)
                                    ( regular file, directory, or symlink )

For detail on the format of each file, please see nilfs2_ondisk.h
located at include/uapi/linux directory.

There are no patents or other intellectual property that we protect
with regard to the design of NILFS2.  It is allowed to replicate the
design in hopes that other operating systems could share (mount, read,
write, etc.) data stored in this format.