Read/Write HPFS 2.09

HPFS 2.09 credits와 mount option

이 문서는 Mikulas Patocka가 1998~2004년에 작성한 Linux read/write HPFS 2.09 driver 설명입니다. 연락처는 `mikulas@artax.karlin.mff.cuni.cz`, homepage은 `https://artax.karlin.mff.cuni.cz/~mikulas/vyplody/hpfs/index-e.cgi`입니다.

Chris Smith의 1993년 original read-only HPFS에서 일부 code와 HPFS structure file을 가져왔습니다. Jacques Gelinas의 MSDOS mmap은 Jon Tombs가 1993년 8월 15일 작성한 `fs/nfs/mmap.c`에서 영감을 받았고, Werner Almesberger의 1992~1993년 MSDOS option parser와 CR/LF conversion도 기여했습니다.

`uid=xxx,gid=xxx,umask=xxx`는 extended attribute에 값이 없는 file의 owner, group, mode를 정합니다. default는 `uid=gid=0`과 system default umask입니다.

mode는 umask를 반전한 값입니다. 예를 들어 `umask 027`은 owner에게 모든 permission, group에게 read permission을 주고 그 밖의 user에게는 접근을 허용하지 않습니다. 일반 file mode는 추가로 `0666`과 AND되므로 executable 권한이 필요하면 extended attribute를 사용해야 합니다.

`case=lower,asis`는 `readdir`에서 filename을 lowercase로 바꿀지 정하며 default는 `asis`입니다.

`conv=binary,text,auto`는 CR/LF를 LF로 변환하는 방식이며 default는 `binary`입니다. `auto`는 source에 있는 text extension 목록으로 결정합니다.

원문 저자는 binary file 손상을 피하기 위해 text file을 변환하지 않는 편이 낫다고 설명하고, extension 목록 변경은 source에서 하라고 안내합니다. original read-only driver의 heuristic은 DJGPP binary처럼 앞부분에 짧은 text가 있는 file을 오판해 손상시킬 수 있어 제거했습니다.

`check=none,normal,strict`는 검사 수준이며 default는 `normal`입니다. `none`은 속도 향상이 작고 위험이 크며, `normal`은 손상된 filesystem에서도 crash하지 않도록 설계됐습니다. `strict`는 bitmap allocation 같은 추가 검사를 수행하는 debugging mode입니다.

`errors=continue,remount-ro,panic`은 filesystem error 발견 시 동작을 정하며 default는 `remount-ro`입니다.

`chkdsk=no,errors,always`는 OS/2가 검사하도록 filesystem을 dirty로 표시할 시점을 정하며 default는 `errors`입니다.

`eas=no,ro,rw`는 extended attribute 처리 방식이며 default는 `rw`입니다. `no`는 EA를 무시하고 항상 mount option의 UID/GID/mode를 사용합니다. `ro`는 EA를 읽되 만들지 않고, `rw`는 `chmod`, `chown`, `chgrp`, `mknod`, `ln -s` 사용 시 EA를 만듭니다.

`timeshift=(-)nnn`은 시간을 `nnn`초 이동하며 default는 0입니다. Linux에서 OS/2보다 시간이 한 시간 앞서 보이면 `timeshift=-3600`을 사용할 수 있습니다.

.. SPDX-License-Identifier: GPL-2.0

====================
Read/Write HPFS 2.09
====================

1998-2004, Mikulas Patocka

:email: mikulas@artax.karlin.mff.cuni.cz
:homepage: https://artax.karlin.mff.cuni.cz/~mikulas/vyplody/hpfs/index-e.cgi

Credits
=======
Chris Smith, 1993, original read-only HPFS, some code and hpfs structures file
        is taken from it

Jacques Gelinas, MSDos mmap, Inspired by fs/nfs/mmap.c (Jon Tombs 15 Aug 1993)

Werner Almesberger, 1992, 1993, MSDos option parser & CR/LF conversion

Mount options

uid=xxx,gid=xxx,umask=xxx (default uid=gid=0 umask=default_system_umask)
        Set owner/group/mode for files that do not have it specified in extended
        attributes. Mode is inverted umask - for example umask 027 gives owner
        all permission, group read permission and anybody else no access. Note
        that for files mode is anded with 0666. If you want files to have 'x'
        rights, you must use extended attributes.
case=lower,asis (default asis)
        File name lowercasing in readdir.
conv=binary,text,auto (default binary)
        CR/LF -> LF conversion, if auto, decision is made according to extension
        - there is a list of text extensions (I thing it's better to not convert
        text file than to damage binary file). If you want to change that list,
        change it in the source. Original readonly HPFS contained some strange
        heuristic algorithm that I removed. I thing it's danger to let the
        computer decide whether file is text or binary. For example, DJGPP
        binaries contain small text message at the beginning and they could be
        misidentified and damaged under some circumstances.
check=none,normal,strict (default normal)
        Check level. Selecting none will cause only little speedup and big
        danger. I tried to write it so that it won't crash if check=normal on
        corrupted filesystems. check=strict means many superfluous checks -
        used for debugging (for example it checks if file is allocated in
        bitmaps when accessing it).
errors=continue,remount-ro,panic (default remount-ro)
        Behaviour when filesystem errors found.
chkdsk=no,errors,always (default errors)
        When to mark filesystem dirty so that OS/2 checks it.
eas=no,ro,rw (default rw)
        What to do with extended attributes. 'no' - ignore them and use always
        values specified in uid/gid/mode options. 'ro' - read extended
        attributes but do not create them. 'rw' - create extended attributes
        when you use chmod/chown/chgrp/mknod/ln -s on the filesystem.
timeshift=(-)nnn (default 0)
        Shifts the time by nnn seconds. For example, if you see under linux
        one hour more, than under os/2, use timeshift=-3600.

Filename, extended attribute와 symlink

OS/2와 마찬가지로 HPFS filename 비교는 case-insensitive입니다. 따라서 `FOO`를 만든 뒤 `cat FOO`, `cat Foo`, `cat foo`로 접근할 수 있습니다.

하지만 shell glob은 case-sensitive하게 pattern을 확장하므로 `cat F*`는 동작해도 `cat f*`는 `FOO`를 찾지 못합니다. `bootsect.S`와 `bootsect.s`처럼 case만 다른 file을 함께 만드는 Linux kernel build도 HPFS에서는 수행할 수 없습니다.

filename에 128 이상의 character가 있으면 아래에서 설명하는 codepage가 검색에 사용됩니다.

OS/2는 filename 끝의 dot과 space를 무시하며 이 driver도 같은 규칙을 따릅니다. 예를 들어 `'a. ...'`를 만들면 실제 file은 `'a'`이고 `'a.'`, `'a..'`, `'a . . . '` 같은 이름으로도 접근할 수 있습니다.

HPFS extended attribute는 ASCII key와 가변 길이 byte string value의 쌍입니다. OS/2는 window·icon 위치와 file type을 저장하며, Linux driver는 Unix owner와 access right도 여기에 기록합니다.

HPFS에서 `chown`, `chgrp`, `chmod`를 수행하면 2-byte value의 `UID`, `GID`, `MODE` EA가 생성됩니다. mount option default와 값이 다를 때만 처음 생성되지만 한 번 생긴 EA는 삭제되지 않고 값만 바뀝니다.

예를 들어 default가 `uid=0`일 때 `chown luser file; chown root file`을 실행하면 `UID=0` EA가 남습니다. 이후 `uid=luser_uid`로 다시 mount해도 해당 file은 EA 때문에 계속 root 소유입니다.

`chmod file 444`는 `MODE` EA를 만들지 않고 HPFS read-only flag를 설정하는 special case입니다.

block 또는 character device를 `mknod`하면 `MODE` 외에 device number를 담은 4-byte `DEV` EA가 생성됩니다.

현재 driver는 EA 크기를 조정할 수 없습니다. 다른 program이 `UID`, `GID`, `MODE`, `DEV`를 예상과 다른 크기로 만들었다면 driver가 다시 쓸 수 없어 값 변경이 동작하지 않습니다.

Linux symlink는 link target을 값으로 갖는 `SYMLINK` EA로 구현합니다. ext2와 마찬가지로 symlink 자체를 `chown`·`chgrp`할 수 있지만, symlink에 `chmod`하면 target file의 mode가 바뀝니다.

이 symlink는 Linux 전용이고 OS/2와 호환되지 않습니다. OS/2 PmShell symlink는 일부가 directory EA, 일부가 `OS2SYS.INI`에 저장되고 file 이동을 추적하는 복잡한 구조라 지원하지 않습니다.

File names
==========

As in OS/2, filenames are case insensitive. However, shell thinks that names
are case sensitive, so for example when you create a file FOO, you can use
'cat FOO', 'cat Foo', 'cat foo' or 'cat F*' but not 'cat f*'. Note, that you
also won't be able to compile linux kernel (and maybe other things) on HPFS
because kernel creates different files with names like bootsect.S and
bootsect.s. When searching for file whose name has characters >= 128, codepages
are used - see below.
OS/2 ignores dots and spaces at the end of file name, so this driver does as
well. If you create 'a. ...', the file 'a' will be created, but you can still
access it under names 'a.', 'a..', 'a .  . . ' etc.


Extended attributes
===================

On HPFS partitions, OS/2 can associate to each file a special information called
extended attributes. Extended attributes are pairs of (key,value) where key is
an ascii string identifying that attribute and value is any string of bytes of
variable length. OS/2 stores window and icon positions and file types there. So
why not use it for unix-specific info like file owner or access rights? This
driver can do it. If you chown/chgrp/chmod on a hpfs partition, extended
attributes with keys "UID", "GID" or "MODE" and 2-byte values are created. Only
that extended attributes those value differs from defaults specified in mount
options are created. Once created, the extended attributes are never deleted,
they're just changed. It means that when your default uid=0 and you type
something like 'chown luser file; chown root file' the file will contain
extended attribute UID=0. And when you umount the fs and mount it again with
uid=luser_uid, the file will be still owned by root! If you chmod file to 444,
extended attribute "MODE" will not be set, this special case is done by setting
read-only flag. When you mknod a block or char device, besides "MODE", the
special 4-byte extended attribute "DEV" will be created containing the device
number. Currently this driver cannot resize extended attributes - it means
that if somebody (I don't know who?) has set "UID", "GID", "MODE" or "DEV"
attributes with different sizes, they won't be rewritten and changing these
values doesn't work.


Symlinks
========

You can do symlinks on HPFS partition, symlinks are achieved by setting extended
attribute named "SYMLINK" with symlink value. Like on ext2, you can chown and
chgrp symlinks but I don't know what is it good for. chmoding symlink results
in chmoding file where symlink points. These symlinks are just for Linux use and
incompatible with OS/2. OS/2 PmShell symlinks are not supported because they are
stored in very crazy way. They tried to do it so that link changes when file is
moved ... sometimes it works. But the link is partly stored in directory
extended attributes and partly in OS2SYS.INI. I don't want (and don't know how)
to analyze or change OS2SYS.INI.

Codepage 처리와 혼합 언어 OS/2의 위험

HPFS volume에는 여러 codepage용 uppercasing table이 있을 수 있고 각 file은 filename이 속한 codepage를 가리킵니다. 하지만 원문 저자는 OS/2의 multiple codepage 구현이 매우 buggy하다고 설명합니다.

예시는 codepage 852를 쓰는 Czech OS/2 partition에 cp850을 쓰는 English OS/2로 boot하여 file을 만든 경우입니다. file은 codepage 850으로 올바르게 표시됐지만 Czech OS/2로 돌아오자 어떤 이름으로도 접근할 수 없었습니다.

추정 원인은 OS/2가 search pattern은 system codepage 852로 uppercase하고 비교 대상 filename은 file codepage 850으로 uppercase하여 두 값이 match하지 않는 것입니다.

같은 directory에 Czech OS/2로 새 file을 만들자 그 file도 접근할 수 없었습니다. HPFS directory는 filename 순서로 정렬돼야 하는데, file 배치용 uppercase와 검색용 uppercase 방법이 다른 것으로 보입니다.

PmShell에서 directory를 열면 crash했고 reboot 뒤에도 같은 directory를 다시 열려 했습니다. `chkdsk`는 오류를 무시했으며 low-level disk modification만 문제를 해결했습니다.

따라서 원문은 한 system에서 서로 다른 언어 version의 OS/2를 절대 섞지 말라고 경고합니다.

Linux driver는 복잡한 multiple codepage 지원이 OS/2 bug 때문에 이득보다 문제를 더 만들 것으로 보고, file의 codepage index와 무관하게 처음 찾은 codepage 하나로 uppercase와 lowercase를 수행합니다.

일반적으로 앞의 혼합 언어 시나리오를 만들지 않는 한 모든 filename은 그 첫 codepage에 있으므로 이 단순화가 동작합니다.

Codepages
=========

HPFS can contain several uppercasing tables for several codepages and each
file has a pointer to codepage its name is in. However OS/2 was created in
America where people don't care much about codepages and so multiple codepages
support is quite buggy. I have Czech OS/2 working in codepage 852 on my disk.
Once I booted English OS/2 working in cp 850 and I created a file on my 852
partition. It marked file name codepage as 850 - good. But when I again booted
Czech OS/2, the file was completely inaccessible under any name. It seems that
OS/2 uppercases the search pattern with its system code page (852) and file
name it's comparing to with its code page (850). These could never match. Is it
really what IBM developers wanted? But problems continued. When I created in
Czech OS/2 another file in that directory, that file was inaccessible too. OS/2
probably uses different uppercasing method when searching where to place a file
(note, that files in HPFS directory must be sorted) and when searching for
a file. Finally when I opened this directory in PmShell, PmShell crashed (the
funny thing was that, when rebooted, PmShell tried to reopen this directory
again :-). chkdsk happily ignores these errors and only low-level disk
modification saved me.  Never mix different language versions of OS/2 on one
system although HPFS was designed to allow that.
OK, I could implement complex codepage support to this driver but I think it
would cause more problems than benefit with such buggy implementation in OS/2.
So this driver simply uses first codepage it finds for uppercasing and
lowercasing no matter what's file codepage index. Usually all file names are in
this codepage - if you don't try to do what I described above :-)

HPFS386와 알려진 driver 제약

OS/2 server의 HPFS386은 지원하지 않습니다. 일반 OS/2 client에 설치된 HPFS386은 동작해야 하지만 OS/2 server volume은 read-only mode로만 사용해야 합니다.

driver는 HPFS386 access control list나 extended permission list 구조를 처리하는 법, file 삭제 시 이를 제거하는 법, EA로 덮어쓰지 않는 법을 알지 못합니다. HPFS386 구조를 발견하면 read-only로 remount하여 파괴하지 않도록 시도합니다.

extended attribute 공간이 부족하면 EA가 truncate되지만 error가 반환되지 않습니다.

OS/2는 path가 약 256 character보다 길면 file에 접근하지 못하지만 이 driver는 긴 path 생성을 허용합니다. `chkdsk`도 이런 오류를 무시합니다.

매우 가득 찬 filesystem에서는 file 삭제가 `ENOSPC`로 실패할 수 있습니다. HPFS directory tree의 non-leaf dnode에 있는 file을 삭제할 때 다른 node로 교체해야 하는데, 새 filename이 더 길면 dnode에 맞지 않아 tree split과 추가 disk space가 필요하기 때문입니다.

workaround는 leaf file을 먼저 삭제하거나 file을 truncate해 공간을 만드는 것입니다. non-leaf일 확률은 약 1/50입니다.

이 문제는 directory가 매우 많아 preallocated directory band가 가득 찬 경우에만 발생하며 조건은 `number_of_directories / size_of_filesystem_in_mb > 4`입니다.

열린 directory는 삭제할 수 없고 directory 위로 rename할 수도 없습니다.

case만 바꾸는 rename은 driver가 지원하지만 VFS가 지원하지 않아 `mv file FILE`이 동작하지 않습니다.

성능상 모든 atime과 directory mtime을 갱신하지 않습니다. 갱신하도록 구현할 수 있지만 매우 느려집니다.

system이 memory와 swap을 모두 소진하면 lost file이나 unbalanced directory 같은 가벼운 filesystem corruption이 생길 수 있습니다.

compile 시 `function declaration isn't a prototype` warning이 발생한다는 역사적 메모도 원문에 포함돼 있습니다.

Known bugs
==========

HPFS386 on OS/2 server is not supported. HPFS386 installed on normal OS/2 client
should work. If you have OS/2 server, use only read-only mode. I don't know how
to handle some HPFS386 structures like access control list or extended perm
list, I don't know how to delete them when file is deleted and how to not
overwrite them with extended attributes. Send me some info on these structures
and I'll make it. However, this driver should detect presence of HPFS386
structures, remount read-only and not destroy them (I hope).

When there's not enough space for extended attributes, they will be truncated
and no error is returned.

OS/2 can't access files if the path is longer than about 256 chars but this
driver allows you to do it. chkdsk ignores such errors.

Sometimes you won't be able to delete some files on a very full filesystem
(returning error ENOSPC). That's because file in non-leaf node in directory tree
(one directory, if it's large, has dirents in tree on HPFS) must be replaced
with another node when deleted. And that new file might have larger name than
the old one so the new name doesn't fit in directory node (dnode). And that
would result in directory tree splitting, that takes disk space. Workaround is
to delete other files that are leaf (probability that the file is non-leaf is
about 1/50) or to truncate file first to make some space.
You encounter this problem only if you have many directories so that
preallocated directory band is full i.e.::

        number_of_directories / size_of_filesystem_in_mb > 4.

You can't delete open directories.

You can't rename over directories (what is it good for?).

Renaming files so that only case changes doesn't work. This driver supports it
but vfs doesn't. Something like 'mv file FILE' won't work.

All atimes and directory mtimes are not updated. That's because of performance
reasons. If you extremely wish to update them, let me know, I'll write it (but
it will be slow).

When the system is out of memory and swap, it may slightly corrupt filesystem
(lost files, unbalanced directories). (I guess all filesystem may do it).

When compiled, you get warning: function declaration isn't a prototype. Does
anybody know what does it mean?

Unbalanced tree warning과 OS/2 bug

오래된 Linux HPFS driver는 때때로 unbalanced dnode tree를 만들었습니다. OS/2 `chkdsk`는 unbalanced tree를 오류로 보고하지 않고 스스로 만들기도 하지만, HPFS와 HPFS386에는 tree가 unbalanced일 때 드물게 crash하는 bug가 있습니다.

현재 driver는 unbalanced tree를 올바르게 처리하면서 발견 시 warning을 출력합니다. 보통 오래된 driver로 만든 directory가 원인입니다.

workaround는 해당 directory의 모든 file을 다른 곳으로 옮겼다가 다시 되돌리는 것입니다. 이 작업은 OS/2가 아니라 Linux에서 수행해야 합니다.

현재 driver가 전부 새로 만든 directory에서 warning이 나타나면 driver bug이므로 보고해야 합니다.

OS/2 자체의 bug도 기록돼 있습니다. 서로를 가리키는 lost directory가 둘 이상 있으면 `chkdsk`가 filesystem repair 중 lock up합니다.

OS/2에서 한 character 이름의 file을 만들면 때때로 무작위로 `long` flag가 설정되고, `chkdsk`는 `Minor fs error corrected`를 출력하며 flag를 제거합니다.

`a .b` 같은 filename도 OS/2가 `long`으로 표시하지만 `chkdsk`가 short로 바꿉니다. 이 bug은 HPFS386에는 없습니다.

앞서 설명한 codepage bug도 OS/2 bug에 포함되며 fixpack을 설치하지 않으면 더 많은 문제가 존재합니다.

What does "unbalanced tree" message mean?
=========================================

Old versions of this driver created sometimes unbalanced dnode trees. OS/2
chkdsk doesn't scream if the tree is unbalanced (and sometimes creates
unbalanced trees too :-) but both HPFS and HPFS386 contain bug that it rarely
crashes when the tree is not balanced. This driver handles unbalanced trees
correctly and writes warning if it finds them. If you see this message, this is
probably because of directories created with old version of this driver.
Workaround is to move all files from that directory to another and then back
again. Do it in Linux, not OS/2! If you see this message in directory that is
whole created by this driver, it is BUG - let me know about it.


Bugs in OS/2
============

When you have two (or more) lost directories pointing each to other, chkdsk
locks up when repairing filesystem.

Sometimes (I think it's random) when you create a file with one-char name under
OS/2, OS/2 marks it as 'long'. chkdsk then removes this flag saying "Minor fs
error corrected".

File names like "a .b" are marked as 'long' by OS/2 but chkdsk "corrects" it and
marks them as short (and writes "minor fs error corrected"). This bug is not in
HPFS386.

Codepage bugs described above
=============================

If you don't install fixpacks, there are many, many more...

HPFS driver 0.90~2.09 history

이 절은 첫 public release인 0.90부터 extremely fragmented file 오류를 고친 2.09까지 Linux HPFS driver의 변경 history를 보존합니다.

History
=======

====== =========================================================================
0.90   First public release
0.91   Fixed bug that caused shooting to memory when write_inode was called on
       open inode (rarely happened)
0.92   Fixed a little memory leak in freeing directory inodes
0.93   Fixed bug that locked up the machine when there were too many filenames
       with first 15 characters same
       Fixed write_file to zero file when writing behind file end
0.94   Fixed a little memory leak when trying to delete busy file or directory
0.95   Fixed a bug that i_hpfs_parent_dir was not updated when moving files
1.90   First version for 2.1.1xx kernels
1.91   Fixed a bug that chk_sectors failed when sectors were at the end of disk
       Fixed a race-condition when write_inode is called while deleting file
       Fixed a bug that could possibly happen (with very low probability) when
       using 0xff in filenames.

       Rewritten locking to avoid race-conditions

       Mount option 'eas' now works

       Fsync no longer returns error

       Files beginning with '.' are marked hidden

       Remount support added

       Alloc is not so slow when filesystem becomes full

       Atimes are no more updated because it slows down operation

       Code cleanup (removed all commented debug prints)
1.92   Corrected a bug when sync was called just before closing file
1.93   Modified, so that it works with kernels >= 2.1.131, I don't know if it
       works with previous versions

       Fixed a possible problem with disks > 64G (but I don't have one, so I can't
       test it)

       Fixed a file overflow at 2G

       Added new option 'timeshift'

       Changed behaviour on HPFS386: It is now possible to operate on HPFS386 in
       read-only mode

       Fixed a bug that slowed down alloc and prevented allocating 100% space
       (this bug was not destructive)
1.94   Added workaround for one bug in Linux

       Fixed one buffer leak

       Fixed some incompatibilities with large extended attributes (but it's still
       not 100% ok, I have no info on it and OS/2 doesn't want to create them)

       Rewritten allocation

       Fixed a bug with i_blocks (du sometimes didn't display correct values)

       Directories have no longer archive attribute set (some programs don't like
       it)

       Fixed a bug that it set badly one flag in large anode tree (it was not
       destructive)
1.95   Fixed one buffer leak, that could happen on corrupted filesystem

       Fixed one bug in allocation in 1.94
1.96   Added workaround for one bug in OS/2 (HPFS locked up, HPFS386 reported
       error sometimes when opening directories in PMSHELL)

       Fixed a possible bitmap race

       Fixed possible problem on large disks

       You can now delete open files

       Fixed a nondestructive race in rename
1.97   Support for HPFS v3 (on large partitions)

       ZFixed a bug that it didn't allow creation of files > 128M
       (it should be 2G)
1.97.1 Changed names of global symbols

       Fixed a bug when chmoding or chowning root directory
1.98   Fixed a deadlock when using old_readdir
       Better directory handling; workaround for "unbalanced tree" bug in OS/2
1.99   Corrected a possible problem when there's not enough space while deleting
       file

       Now it tries to truncate the file if there's not enough space when
       deleting

       Removed a lot of redundant code
2.00   Fixed a bug in rename (it was there since 1.96)
       Better anti-fragmentation strategy
2.01   Fixed problem with directory listing over NFS

       Directory lseek now checks for proper parameters

       Fixed race-condition in buffer code - it is in all filesystems in Linux;
       when reading device (cat /dev/hda) while creating files on it, files
       could be damaged
2.02   Workaround for bug in breada in Linux. breada could cause accesses beyond
       end of partition
2.03   Char, block devices and pipes are correctly created

       Fixed non-crashing race in unlink (Alexander Viro)

       Now it works with Japanese version of OS/2
2.04   Fixed error when ftruncate used to extend file
2.05   Fixed crash when got mount parameters without =

       Fixed crash when allocation of anode failed due to full disk

       Fixed some crashes when block io or inode allocation failed
2.06   Fixed some crash on corrupted disk structures

       Better allocation strategy

       Reschedule points added so that it doesn't lock CPU long time

       It should work in read-only mode on Warp Server
2.07   More fixes for Warp Server. Now it really works
2.08   Creating new files is not so slow on large disks

       An attempt to sync deleted file does not generate filesystem error
2.09   Fixed error on extremely fragmented files
====== =========================================================================