요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
.. SPDX-License-Identifier: GPL-2.0
2
.. _xfs_online_fsck_design:
4
..
5
Mapping of heading styles within this document:
6
Heading 1 uses "====" above and below
7
Heading 2 uses "===="
8
Heading 3 uses "----"
9
Heading 4 uses "````"
10
Heading 5 uses "^^^^"
11
Heading 6 uses "~~~~"
12
Heading 7 uses "...."
14
Sections are manually numbered because apparently that's what everyone
15
does in the kernel.
17
======================
18
XFS Online Fsck Design
19
======================
21
This document captures the design of the online filesystem check feature for
22
XFS.
23
The purpose of this document is threefold:
25
- To help kernel distributors understand exactly what the XFS online fsck
26
feature is, and issues about which they should be aware.
28
- To help people reading the code to familiarize themselves with the relevant
29
concepts and design points before they start digging into the code.
31
- To help developers maintaining the system by capturing the reasons
32
supporting higher level decision making.
34
As the online fsck code is merged, the links in this document to topic branches
35
will be replaced with links to code.
37
This document is licensed under the terms of the GNU Public License, v2.
38
The primary author is Darrick J. Wong.
40
This design document is split into seven parts.
41
Part 1 defines what fsck tools are and the motivations for writing a new one.
42
Parts 2 and 3 present a high level overview of how online fsck process works
43
and how it is tested to ensure correct functionality.
44
Part 4 discusses the user interface and the intended usage modes of the new
45
program.
46
Parts 5 and 6 show off the high level components and how they fit together, and
47
then present case studies of how each repair function actually works.
48
Part 7 sums up what has been discussed so far and speculates about what else
49
might be built atop online fsck.
51
.. contents:: Table of Contents
52
:local:
54
1. What is a Filesystem Check?
55
==============================
57
A Unix filesystem has four main responsibilities:
59
- Provide a hierarchy of names through which application programs can associate
60
arbitrary blobs of data for any length of time,
62
- Virtualize physical storage media across those names, and
64
- Retrieve the named data blobs at any time.
66
- Examine resource usage.
68
Metadata directly supporting these functions (e.g. files, directories, space
69
mappings) are sometimes called primary metadata.
70
Secondary metadata (e.g. reverse mapping and directory parent pointers) support
71
operations internal to the filesystem, such as internal consistency checking
72
and reorganization.
73
Summary metadata, as the name implies, condense information contained in
74
primary metadata for performance reasons.
76
The filesystem check (fsck) tool examines all the metadata in a filesystem
77
to look for errors.
78
In addition to looking for obvious metadata corruptions, fsck also
79
cross-references different types of metadata records with each other to look
80
for inconsistencies.
81
People do not like losing data, so most fsck tools also contains some ability
82
to correct any problems found.
83
As a word of caution -- the primary goal of most Linux fsck tools is to restore
84
the filesystem metadata to a consistent state, not to maximize the data
85
recovered.
86
That precedent will not be challenged here.
88
Filesystems of the 20th century generally lacked any redundancy in the ondisk
89
format, which means that fsck can only respond to errors by erasing files until
90
errors are no longer detected.
91
More recent filesystem designs contain enough redundancy in their metadata that
92
it is now possible to regenerate data structures when non-catastrophic errors
93
occur; this capability aids both strategies.
95
+--------------------------------------------------------------------------+
96
| **Note**: |
97
+--------------------------------------------------------------------------+
98
| System administrators avoid data loss by increasing the number of |
99
| separate storage systems through the creation of backups; and they avoid |
100
| downtime by increasing the redundancy of each storage system through the |
101
| creation of RAID arrays. |
102
| fsck tools address only the first problem. |
103
+--------------------------------------------------------------------------+
105
TLDR; Show Me the Code!
106
-----------------------
108
Code is posted to the kernel.org git trees as follows:
109
`kernel changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-symlink>`_,
110
`userspace changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_, and
111
`QA test changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs>`_.
112
Each kernel patchset adding an online repair function will use the same branch
113
name across the kernel, xfsprogs, and fstests git repos.
115
Existing Tools
116
--------------
118
The online fsck tool described here will be the third tool in the history of
119
XFS (on Linux) to check and repair filesystems.
120
Two programs precede it:
122
The first program, ``xfs_check``, was created as part of the XFS debugger
123
(``xfs_db``) and can only be used with unmounted filesystems.
124
It walks all metadata in the filesystem looking for inconsistencies in the
125
metadata, though it lacks any ability to repair what it finds.
126
Due to its high memory requirements and inability to repair things, this
127
program is now deprecated and will not be discussed further.
129
The second program, ``xfs_repair``, was created to be faster and more robust
130
than the first program.
131
Like its predecessor, it can only be used with unmounted filesystems.
132
It uses extent-based in-memory data structures to reduce memory consumption,
133
and tries to schedule readahead IO appropriately to reduce I/O waiting time
134
while it scans the metadata of the entire filesystem.
135
The most important feature of this tool is its ability to respond to
136
inconsistencies in file metadata and directory tree by erasing things as needed
137
to eliminate problems.
138
Space usage metadata are rebuilt from the observed file metadata.
140
Problem Statement
141
-----------------
143
The current XFS tools leave several problems unsolved:
145
1. **User programs** suddenly **lose access** to the filesystem when unexpected
146
shutdowns occur as a result of silent corruptions in the metadata.
147
These occur **unpredictably** and often without warning.
149
2. **Users** experience a **total loss of service** during the recovery period
150
after an **unexpected shutdown** occurs.
152
3. **Users** experience a **total loss of service** if the filesystem is taken
153
offline to **look for problems** proactively.
155
4. **Data owners** cannot **check the integrity** of their stored data without
156
reading all of it.
157
This may expose them to substantial billing costs when a linear media scan
158
performed by the storage system administrator might suffice.
160
5. **System administrators** cannot **schedule** a maintenance window to deal
161
with corruptions if they **lack the means** to assess filesystem health
162
while the filesystem is online.
164
6. **Fleet monitoring tools** cannot **automate periodic checks** of filesystem
165
health when doing so requires **manual intervention** and downtime.
167
7. **Users** can be tricked into **doing things they do not desire** when
168
malicious actors **exploit quirks of Unicode** to place misleading names
169
in directories.
171
Given this definition of the problems to be solved and the actors who would
172
benefit, the proposed solution is a third fsck tool that acts on a running
173
filesystem.
175
This new third program has three components: an in-kernel facility to check
176
metadata, an in-kernel facility to repair metadata, and a userspace driver
177
program to drive fsck activity on a live filesystem.
178
``xfs_scrub`` is the name of the driver program.
179
The rest of this document presents the goals and use cases of the new fsck
180
tool, describes its major design points in connection to those goals, and
181
discusses the similarities and differences with existing tools.
183
+--------------------------------------------------------------------------+
184
| **Note**: |
185
+--------------------------------------------------------------------------+
186
| Throughout this document, the existing offline fsck tool can also be |
187
| referred to by its current name "``xfs_repair``". |
188
| The userspace driver program for the new online fsck tool can be |
189
| referred to as "``xfs_scrub``". |
190
| The kernel portion of online fsck that validates metadata is called |
191
| "online scrub", and portion of the kernel that fixes metadata is called |
192
| "online repair". |
193
+--------------------------------------------------------------------------+
195
The naming hierarchy is broken up into objects known as directories and files
196
and the physical space is split into pieces known as allocation groups.
197
Sharding enables better performance on highly parallel systems and helps to
198
contain the damage when corruptions occur.
199
The division of the filesystem into principal objects (allocation groups and
200
inodes) means that there are ample opportunities to perform targeted checks and
201
repairs on a subset of the filesystem.
203
While this is going on, other parts continue processing IO requests.
204
Even if a piece of filesystem metadata can only be regenerated by scanning the
205
entire system, the scan can still be done in the background while other file
206
operations continue.
208
In summary, online fsck takes advantage of resource sharding and redundant
209
metadata to enable targeted checking and repair operations while the system
210
is running.
211
This capability will be coupled to automatic system management so that
212
autonomous self-healing of XFS maximizes service availability.
214
2. Theory of Operation
215
======================
217
Because it is necessary for online fsck to lock and scan live metadata objects,
218
online fsck consists of three separate code components.
219
The first is the userspace driver program ``xfs_scrub``, which is responsible
220
for identifying individual metadata items, scheduling work items for them,
221
reacting to the outcomes appropriately, and reporting results to the system
222
administrator.
223
The second and third are in the kernel, which implements functions to check
224
and repair each type of online fsck work item.
226
+------------------------------------------------------------------+
227
| **Note**: |
228
+------------------------------------------------------------------+
229
| For brevity, this document shortens the phrase "online fsck work |
230
| item" to "scrub item". |
231
+------------------------------------------------------------------+
233
Scrub item types are delineated in a manner consistent with the Unix design
234
philosophy, which is to say that each item should handle one aspect of a
235
metadata structure, and handle it well.
237
Scope
238
-----
240
In principle, online fsck should be able to check and to repair everything that
241
the offline fsck program can handle.
242
However, online fsck cannot be running 100% of the time, which means that
243
latent errors may creep in after a scrub completes.
244
If these errors cause the next mount to fail, offline fsck is the only
245
solution.
246
This limitation means that maintenance of the offline fsck tool will continue.
247
A second limitation of online fsck is that it must follow the same resource
248
sharing and lock acquisition rules as the regular filesystem.
249
This means that scrub cannot take *any* shortcuts to save time, because doing
250
so could lead to concurrency problems.
251
In other words, online fsck is not a complete replacement for offline fsck, and
252
a complete run of online fsck may take longer than online fsck.
253
However, both of these limitations are acceptable tradeoffs to satisfy the
254
different motivations of online fsck, which are to **minimize system downtime**
255
and to **increase predictability of operation**.
257
.. _scrubphases:
259
Phases of Work
260
--------------
262
The userspace driver program ``xfs_scrub`` splits the work of checking and
263
repairing an entire filesystem into seven phases.
264
Each phase concentrates on checking specific types of scrub items and depends
265
on the success of all previous phases.
266
The seven phases are as follows:
268
1. Collect geometry information about the mounted filesystem and computer,
269
discover the online fsck capabilities of the kernel, and open the
270
underlying storage devices.
272
2. Check allocation group metadata, all realtime volume metadata, and all quota
273
files.
274
Each metadata structure is scheduled as a separate scrub item.
275
If corruption is found in the inode header or inode btree and ``xfs_scrub``
276
is permitted to perform repairs, then those scrub items are repaired to
277
prepare for phase 3.
278
Repairs are implemented by using the information in the scrub item to
279
resubmit the kernel scrub call with the repair flag enabled; this is
280
discussed in the next section.
281
Optimizations and all other repairs are deferred to phase 4.
283
3. Check all metadata of every file in the filesystem.
284
Each metadata structure is also scheduled as a separate scrub item.
285
If repairs are needed and ``xfs_scrub`` is permitted to perform repairs,
286
and there were no problems detected during phase 2, then those scrub items
287
are repaired immediately.
288
Optimizations, deferred repairs, and unsuccessful repairs are deferred to
289
phase 4.
291
4. All remaining repairs and scheduled optimizations are performed during this
292
phase, if the caller permits them.
293
Before starting repairs, the summary counters are checked and any necessary
294
repairs are performed so that subsequent repairs will not fail the resource
295
reservation step due to wildly incorrect summary counters.
296
Unsuccessful repairs are requeued as long as forward progress on repairs is
297
made somewhere in the filesystem.
298
Free space in the filesystem is trimmed at the end of phase 4 if the
299
filesystem is clean.
301
5. By the start of this phase, all primary and secondary filesystem metadata
302
must be correct.
303
Summary counters such as the free space counts and quota resource counts
304
are checked and corrected.
305
Directory entry names and extended attribute names are checked for
306
suspicious entries such as control characters or confusing Unicode sequences
307
appearing in names.
309
6. If the caller asks for a media scan, read all allocated and written data
310
file extents in the filesystem.
311
The ability to use hardware-assisted data file integrity checking is new
312
to online fsck; neither of the previous tools have this capability.
313
If media errors occur, they will be mapped to the owning files and reported.
315
7. Re-check the summary counters and presents the caller with a summary of
316
space usage and file counts.
318
This allocation of responsibilities will be :ref:`revisited <scrubcheck>`
319
later in this document.
321
Steps for Each Scrub Item
322
-------------------------
324
The kernel scrub code uses a three-step strategy for checking and repairing
325
the one aspect of a metadata object represented by a scrub item:
327
1. The scrub item of interest is checked for corruptions; opportunities for
328
optimization; and for values that are directly controlled by the system
329
administrator but look suspicious.
330
If the item is not corrupt or does not need optimization, resource are
331
released and the positive scan results are returned to userspace.
332
If the item is corrupt or could be optimized but the caller does not permit
333
this, resources are released and the negative scan results are returned to
334
userspace.
335
Otherwise, the kernel moves on to the second step.
337
2. The repair function is called to rebuild the data structure.
338
Repair functions generally choose rebuild a structure from other metadata
339
rather than try to salvage the existing structure.
340
If the repair fails, the scan results from the first step are returned to
341
userspace.
342
Otherwise, the kernel moves on to the third step.
344
3. In the third step, the kernel runs the same checks over the new metadata
345
item to assess the efficacy of the repairs.
346
The results of the reassessment are returned to userspace.
348
Classification of Metadata
349
--------------------------
351
Each type of metadata object (and therefore each type of scrub item) is
352
classified as follows:
354
Primary Metadata
355
````````````````
357
Metadata structures in this category should be most familiar to filesystem
358
users either because they are directly created by the user or they index
359
objects created by the user
360
Most filesystem objects fall into this class:
362
- Free space and reference count information
364
- Inode records and indexes
366
- Storage mapping information for file data
368
- Directories
370
- Extended attributes
372
- Symbolic links
374
- Quota limits
376
Scrub obeys the same rules as regular filesystem accesses for resource and lock
377
acquisition.
379
Primary metadata objects are the simplest for scrub to process.
380
The principal filesystem object (either an allocation group or an inode) that
381
owns the item being scrubbed is locked to guard against concurrent updates.
382
The check function examines every record associated with the type for obvious
383
errors and cross-references healthy records against other metadata to look for
384
inconsistencies.
385
Repairs for this class of scrub item are simple, since the repair function
386
starts by holding all the resources acquired in the previous step.
387
The repair function scans available metadata as needed to record all the
388
observations needed to complete the structure.
389
Next, it stages the observations in a new ondisk structure and commits it
390
atomically to complete the repair.
391
Finally, the storage from the old data structure are carefully reaped.
393
Because ``xfs_scrub`` locks a primary object for the duration of the repair,
394
this is effectively an offline repair operation performed on a subset of the
395
filesystem.
396
This minimizes the complexity of the repair code because it is not necessary to
397
handle concurrent updates from other threads, nor is it necessary to access
398
any other part of the filesystem.
399
As a result, indexed structures can be rebuilt very quickly, and programs
400
trying to access the damaged structure will be blocked until repairs complete.
401
The only infrastructure needed by the repair code are the staging area for
402
observations and a means to write new structures to disk.
403
Despite these limitations, the advantage that online repair holds is clear:
404
targeted work on individual shards of the filesystem avoids total loss of
405
service.
407
This mechanism is described in section 2.1 ("Off-Line Algorithm") of
408
V. Srinivasan and M. J. Carey, `"Performance of On-Line Index Construction
409
Algorithms" <https://minds.wisconsin.edu/bitstream/handle/1793/59524/TR1047.pdf>`_,
410
*Extending Database Technology*, pp. 293-309, 1992.
412
Most primary metadata repair functions stage their intermediate results in an
413
in-memory array prior to formatting the new ondisk structure, which is very
414
similar to the list-based algorithm discussed in section 2.3 ("List-Based
415
Algorithms") of Srinivasan.
416
However, any data structure builder that maintains a resource lock for the
417
duration of the repair is *always* an offline algorithm.
419
.. _secondary_metadata:
421
Secondary Metadata
422
``````````````````
424
Metadata structures in this category reflect records found in primary metadata,
425
but are only needed for online fsck or for reorganization of the filesystem.
427
Secondary metadata include:
429
- Reverse mapping information
431
- Directory parent pointers
433
This class of metadata is difficult for scrub to process because scrub attaches
434
to the secondary object but needs to check primary metadata, which runs counter
435
to the usual order of resource acquisition.
436
Frequently, this means that full filesystems scans are necessary to rebuild the
437
metadata.
438
Check functions can be limited in scope to reduce runtime.
439
Repairs, however, require a full scan of primary metadata, which can take a
440
long time to complete.
441
Under these conditions, ``xfs_scrub`` cannot lock resources for the entire
442
duration of the repair.
444
Instead, repair functions set up an in-memory staging structure to store
445
observations.
446
Depending on the requirements of the specific repair function, the staging
447
index will either have the same format as the ondisk structure or a design
448
specific to that repair function.
449
The next step is to release all locks and start the filesystem scan.
450
When the repair scanner needs to record an observation, the staging data are
451
locked long enough to apply the update.
452
While the filesystem scan is in progress, the repair function hooks the
453
filesystem so that it can apply pending filesystem updates to the staging
454
information.
455
Once the scan is done, the owning object is re-locked, the live data is used to
456
write a new ondisk structure, and the repairs are committed atomically.
457
The hooks are disabled and the staging area is freed.
458
Finally, the storage from the old data structure are carefully reaped.
460
Introducing concurrency helps online repair avoid various locking problems, but
461
comes at a high cost to code complexity.
462
Live filesystem code has to be hooked so that the repair function can observe
463
updates in progress.
464
The staging area has to become a fully functional parallel structure so that
465
updates can be merged from the hooks.
466
Finally, the hook, the filesystem scan, and the inode locking model must be
467
sufficiently well integrated that a hook event can decide if a given update
468
should be applied to the staging structure.
470
In theory, the scrub implementation could apply these same techniques for
471
primary metadata, but doing so would make it massively more complex and less
472
performant.
473
Programs attempting to access the damaged structures are not blocked from
474
operation, which may cause application failure or an unplanned filesystem
475
shutdown.
477
Inspiration for the secondary metadata repair strategy was drawn from section
478
2.4 of Srinivasan above, and sections 2 ("NSF: Index Build Without Side-File")
479
and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms for
480
Creating Indexes for Very Large Tables Without Quiescing Updates"
481
<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.
483
The sidecar index mentioned above bears some resemblance to the side file
484
method mentioned in Srinivasan and Mohan.
485
Their method consists of an index builder that extracts relevant record data to
486
build the new structure as quickly as possible; and an auxiliary structure that
487
captures all updates that would be committed to the index by other threads were
488
the new index already online.
489
After the index building scan finishes, the updates recorded in the side file
490
are applied to the new index.
491
To avoid conflicts between the index builder and other writer threads, the
492
builder maintains a publicly visible cursor that tracks the progress of the
493
scan through the record space.
494
To avoid duplication of work between the side file and the index builder, side
495
file updates are elided when the record ID for the update is greater than the
496
cursor position within the record ID space.
498
To minimize changes to the rest of the codebase, XFS online repair keeps the
499
replacement index hidden until it's completely ready to go.
500
In other words, there is no attempt to expose the keyspace of the new index
501
while repair is running.
502
The complexity of such an approach would be very high and perhaps more
503
appropriate to building *new* indices.
505
**Future Work Question**: Can the full scan and live update code used to
506
facilitate a repair also be used to implement a comprehensive check?
508
*Answer*: In theory, yes. Check would be much stronger if each scrub function
509
employed these live scans to build a shadow copy of the metadata and then
510
compared the shadow records to the ondisk records.
511
However, doing that is a fair amount more work than what the checking functions
512
do now.
513
The live scans and hooks were developed much later.
514
That in turn increases the runtime of those scrub functions.
516
Summary Information
517
```````````````````
519
Metadata structures in this last category summarize the contents of primary
520
metadata records.
521
These are often used to speed up resource usage queries, and are many times
522
smaller than the primary metadata which they represent.
524
Examples of summary information include:
526
- Summary counts of free space and inodes
528
- File link counts from directories
530
- Quota resource usage counts
532
Check and repair require full filesystem scans, but resource and lock
533
acquisition follow the same paths as regular filesystem accesses.
535
The superblock summary counters have special requirements due to the underlying
536
implementation of the incore counters, and will be treated separately.
537
Check and repair of the other types of summary counters (quota resource counts
538
and file link counts) employ the same filesystem scanning and hooking
539
techniques as outlined above, but because the underlying data are sets of
540
integer counters, the staging data need not be a fully functional mirror of the
541
ondisk structure.
543
Inspiration for quota and file link count repair strategies were drawn from
544
sections 2.12 ("Online Index Operations") through 2.14 ("Incremental View
545
Maintenance") of G. Graefe, `"Concurrent Queries and Updates in Summary Views
546
and Their Indexes"
547
<http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf>`_, 2011.
549
Since quotas are non-negative integer counts of resource usage, online
550
quotacheck can use the incremental view deltas described in section 2.14 to
551
track pending changes to the block and inode usage counts in each transaction,
552
and commit those changes to a dquot side file when the transaction commits.
553
Delta tracking is necessary for dquots because the index builder scans inodes,
554
whereas the data structure being rebuilt is an index of dquots.
555
Link count checking combines the view deltas and commit step into one because
556
it sets attributes of the objects being scanned instead of writing them to a
557
separate data structure.
558
Each online fsck function will be discussed as case studies later in this
559
document.
561
Risk Management
562
---------------
564
During the development of online fsck, several risk factors were identified
565
that may make the feature unsuitable for certain distributors and users.
566
Steps can be taken to mitigate or eliminate those risks, though at a cost to
567
functionality.
569
- **Decreased performance**: Adding metadata indices to the filesystem
570
increases the time cost of persisting changes to disk, and the reverse space
571
mapping and directory parent pointers are no exception.
572
System administrators who require the maximum performance can disable the
573
reverse mapping features at format time, though this choice dramatically
574
reduces the ability of online fsck to find inconsistencies and repair them.
576
- **Incorrect repairs**: As with all software, there might be defects in the
577
software that result in incorrect repairs being written to the filesystem.
578
Systematic fuzz testing (detailed in the next section) is employed by the
579
authors to find bugs early, but it might not catch everything.
580
The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
581
and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose not to
582
accept this risk.
583
The xfsprogs build system has a configure option (``--enable-scrub=no``) that
584
disables building of the ``xfs_scrub`` binary, though this is not a risk
585
mitigation if the kernel functionality remains enabled.
587
- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
588
repairable.
589
If the keyspaces of several metadata indices overlap in some manner but a
590
coherent narrative cannot be formed from records collected, then the repair
591
fails.
592
To reduce the chance that a repair will fail with a dirty transaction and
593
render the filesystem unusable, the online repair functions have been
594
designed to stage and validate all new records before committing the new
595
structure.
597
- **Misbehavior**: Online fsck requires many privileges -- raw IO to block
598
devices, opening files by handle, ignoring Unix discretionary access control,
599
and the ability to perform administrative changes.
600
Running this automatically in the background scares people, so the systemd
601
background service is configured to run with only the privileges required.
602
Obviously, this cannot address certain problems like the kernel crashing or
603
deadlocking, but it should be sufficient to prevent the scrub process from
604
escaping and reconfiguring the system.
605
The cron job does not have this protection.
607
- **Fuzz Kiddiez**: There are many people now who seem to think that running
608
automated fuzz testing of ondisk artifacts to find mischievous behavior and
609
spraying exploit code onto the public mailing list for instant zero-day
610
disclosure is somehow of some social benefit.
611
In the view of this author, the benefit is realized only when the fuzz
612
operators help to **fix** the flaws, but this opinion apparently is not
613
widely shared among security "researchers".
614
The XFS maintainers' continuing ability to manage these events presents an
615
ongoing risk to the stability of the development process.
616
Automated testing should front-load some of the risk while the feature is
617
considered EXPERIMENTAL.
619
Many of these risks are inherent to software programming.
620
Despite this, it is hoped that this new functionality will prove useful in
621
reducing unexpected downtime.
623
3. Testing Plan
624
===============
626
As stated before, fsck tools have three main goals:
628
1. Detect inconsistencies in the metadata;
630
2. Eliminate those inconsistencies; and
632
3. Minimize further loss of data.
634
Demonstrations of correct operation are necessary to build users' confidence
635
that the software behaves within expectations.
636
Unfortunately, it was not really feasible to perform regular exhaustive testing
637
of every aspect of a fsck tool until the introduction of low-cost virtual
638
machines with high-IOPS storage.
639
With ample hardware availability in mind, the testing strategy for the online
640
fsck project involves differential analysis against the existing fsck tools and
641
systematic testing of every attribute of every type of metadata object.
642
Testing can be split into four major categories, as discussed below.
644
Integrated Testing with fstests
645
-------------------------------
647
The primary goal of any free software QA effort is to make testing as
648
inexpensive and widespread as possible to maximize the scaling advantages of
649
community.
650
In other words, testing should maximize the breadth of filesystem configuration
651
scenarios and hardware setups.
652
This improves code quality by enabling the authors of online fsck to find and
653
fix bugs early, and helps developers of new features to find integration
654
issues earlier in their development effort.
656
The Linux filesystem community shares a common QA testing suite,
657
`fstests <https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/>`_, for
658
functional and regression testing.
659
Even before development work began on online fsck, fstests (when run on XFS)
660
would run both the ``xfs_check`` and ``xfs_repair -n`` commands on the test and
661
scratch filesystems between each test.
662
This provides a level of assurance that the kernel and the fsck tools stay in
663
alignment about what constitutes consistent metadata.
664
During development of the online checking code, fstests was modified to run
665
``xfs_scrub -n`` between each test to ensure that the new checking code
666
produces the same results as the two existing fsck tools.
668
To start development of online repair, fstests was modified to run
669
``xfs_repair`` to rebuild the filesystem's metadata indices between tests.
670
This ensures that offline repair does not crash, leave a corrupt filesystem
671
after it exists, or trigger complaints from the online check.
672
This also established a baseline for what can and cannot be repaired offline.
673
To complete the first phase of development of online repair, fstests was
674
modified to be able to run ``xfs_scrub`` in a "force rebuild" mode.
675
This enables a comparison of the effectiveness of online repair as compared to
676
the existing offline repair tools.
678
General Fuzz Testing of Metadata Blocks
679
---------------------------------------
681
XFS benefits greatly from having a very robust debugging tool, ``xfs_db``.
683
Before development of online fsck even began, a set of fstests were created
684
to test the rather common fault that entire metadata blocks get corrupted.
685
This required the creation of fstests library code that can create a filesystem
686
containing every possible type of metadata object.
687
Next, individual test cases were created to create a test filesystem, identify
688
a single block of a specific type of metadata object, trash it with the
689
existing ``blocktrash`` command in ``xfs_db``, and test the reaction of a
690
particular metadata validation strategy.
692
This earlier test suite enabled XFS developers to test the ability of the
693
in-kernel validation functions and the ability of the offline fsck tool to
694
detect and eliminate the inconsistent metadata.
695
This part of the test suite was extended to cover online fsck in exactly the
696
same manner.
698
In other words, for a given fstests filesystem configuration:
700
* For each metadata object existing on the filesystem:
702
* Write garbage to it
704
* Test the reactions of:
706
1. The kernel verifiers to stop obviously bad metadata
707
2. Offline repair (``xfs_repair``) to detect and fix
708
3. Online repair (``xfs_scrub``) to detect and fix
710
Targeted Fuzz Testing of Metadata Records
711
-----------------------------------------
713
The testing plan for online fsck includes extending the existing fs testing
714
infrastructure to provide a much more powerful facility: targeted fuzz testing
715
of every metadata field of every metadata object in the filesystem.
716
``xfs_db`` can modify every field of every metadata structure in every
717
block in the filesystem to simulate the effects of memory corruption and
718
software bugs.
719
Given that fstests already contains the ability to create a filesystem
720
containing every metadata format known to the filesystem, ``xfs_db`` can be
721
used to perform exhaustive fuzz testing!
723
For a given fstests filesystem configuration:
725
* For each metadata object existing on the filesystem...
727
* For each record inside that metadata object...
729
* For each field inside that record...
731
* For each conceivable type of transformation that can be applied to a bit field...
733
1. Clear all bits
734
2. Set all bits
735
3. Toggle the most significant bit
736
4. Toggle the middle bit
737
5. Toggle the least significant bit
738
6. Add a small quantity
739
7. Subtract a small quantity
740
8. Randomize the contents
742
* ...test the reactions of:
744
1. The kernel verifiers to stop obviously bad metadata
745
2. Offline checking (``xfs_repair -n``)
746
3. Offline repair (``xfs_repair``)
747
4. Online checking (``xfs_scrub -n``)
748
5. Online repair (``xfs_scrub``)
749
6. Both repair tools (``xfs_scrub`` and then ``xfs_repair`` if online repair doesn't succeed)
751
This is quite the combinatoric explosion!
753
Fortunately, having this much test coverage makes it easy for XFS developers to
754
check the responses of XFS' fsck tools.
755
Since the introduction of the fuzz testing framework, these tests have been
756
used to discover incorrect repair code and missing functionality for entire
757
classes of metadata objects in ``xfs_repair``.
758
The enhanced testing was used to finalize the deprecation of ``xfs_check`` by
759
confirming that ``xfs_repair`` could detect at least as many corruptions as
760
the older tool.
762
These tests have been very valuable for ``xfs_scrub`` in the same ways -- they
763
allow the online fsck developers to compare online fsck against offline fsck,
764
and they enable XFS developers to find deficiencies in the code base.
766
Proposed patchsets include
767
`general fuzzer improvements
768
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements>`_,
769
`fuzzing baselines
770
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_,
771
and `improvements in fuzz testing comprehensiveness
772
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing>`_.
774
Stress Testing
775
--------------
777
A unique requirement to online fsck is the ability to operate on a filesystem
778
concurrently with regular workloads.
779
Although it is of course impossible to run ``xfs_scrub`` with *zero* observable
780
impact on the running system, the online repair code should never introduce
781
inconsistencies into the filesystem metadata, and regular workloads should
782
never notice resource starvation.
783
To verify that these conditions are being met, fstests has been enhanced in
784
the following ways:
786
* For each scrub item type, create a test to exercise checking that item type
787
while running ``fsstress``.
788
* For each scrub item type, create a test to exercise repairing that item type
789
while running ``fsstress``.
790
* Race ``fsstress`` and ``xfs_scrub -n`` to ensure that checking the whole
791
filesystem doesn't cause problems.
792
* Race ``fsstress`` and ``xfs_scrub`` in force-rebuild mode to ensure that
793
force-repairing the whole filesystem doesn't cause problems.
794
* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
795
freezing and thawing the filesystem.
796
* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
797
remounting the filesystem read-only and read-write.
798
* The same, but running ``fsx`` instead of ``fsstress``. (Not done yet?)
800
Success is defined by the ability to run all of these tests without observing
801
any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
802
check warnings, or any other sort of mischief.
804
Proposed patchsets include `general stress testing
805
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_
806
and the `evolution of existing per-function stress testing
807
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_.
809
4. User Interface
810
=================
812
The primary user of online fsck is the system administrator, just like offline
813
repair.
814
Online fsck presents two modes of operation to administrators:
815
A foreground CLI process for online fsck on demand, and a background service
816
that performs autonomous checking and repair.
818
Checking on Demand
819
------------------
821
For administrators who want the absolute freshest information about the
822
metadata in a filesystem, ``xfs_scrub`` can be run as a foreground process on
823
a command line.
824
The program checks every piece of metadata in the filesystem while the
825
administrator waits for the results to be reported, just like the existing
826
``xfs_repair`` tool.
827
Both tools share a ``-n`` option to perform a read-only scan, and a ``-v``
828
option to increase the verbosity of the information reported.
830
A new feature of ``xfs_scrub`` is the ``-x`` option, which employs the error
831
correction capabilities of the hardware to check data file contents.
832
The media scan is not enabled by default because it may dramatically increase
833
program runtime and consume a lot of bandwidth on older storage hardware.
835
The output of a foreground invocation is captured in the system log.
837
The ``xfs_scrub_all`` program walks the list of mounted filesystems and
838
initiates ``xfs_scrub`` for each of them in parallel.
839
It serializes scans for any filesystems that resolve to the same top level
840
kernel block device to prevent resource overconsumption.
842
Background Service
843
------------------
845
To reduce the workload of system administrators, the ``xfs_scrub`` package
846
provides a suite of `systemd <https://systemd.io/>`_ timers and services that
847
run online fsck automatically on weekends by default.
848
The background service configures scrub to run with as little privilege as
849
possible, the lowest CPU and IO priority, and in a CPU-constrained single
850
threaded mode.
851
This can be tuned by the systemd administrator at any time to suit the latency
852
and throughput requirements of customer workloads.
854
The output of the background service is also captured in the system log.
855
If desired, reports of failures (either due to inconsistencies or mere runtime
856
errors) can be emailed automatically by setting the ``EMAIL_ADDR`` environment
857
variable in the following service files:
859
* ``xfs_scrub_fail@.service``
860
* ``xfs_scrub_media_fail@.service``
861
* ``xfs_scrub_all_fail.service``
863
The decision to enable the background scan is left to the system administrator.
864
This can be done by enabling either of the following services:
866
* ``xfs_scrub_all.timer`` on systemd systems
867
* ``xfs_scrub_all.cron`` on non-systemd systems
869
This automatic weekly scan is configured out of the box to perform an
870
additional media scan of all file data once per month.
871
This is less foolproof than, say, storing file data block checksums, but much
872
more performant if application software provides its own integrity checking,
873
redundancy can be provided elsewhere above the filesystem, or the storage
874
device's integrity guarantees are deemed sufficient.
876
The systemd unit file definitions have been subjected to a security audit
877
(as of systemd 249) to ensure that the xfs_scrub processes have as little
878
access to the rest of the system as possible.
879
This was performed via ``systemd-analyze security``, after which privileges
880
were restricted to the minimum required, sandboxing was set up to the maximal
881
extent possible with sandboxing and system call filtering; and access to the
882
filesystem tree was restricted to the minimum needed to start the program and
883
access the filesystem being scanned.
884
The service definition files restrict CPU usage to 80% of one CPU core, and
885
apply as nice of a priority to IO and CPU scheduling as possible.
886
This measure was taken to minimize delays in the rest of the filesystem.
887
No such hardening has been performed for the cron job.
889
Proposed patchset:
890
`Enabling the xfs_scrub background service
891
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_.
893
Health Reporting
894
----------------
896
XFS caches a summary of each filesystem's health status in memory.
897
The information is updated whenever ``xfs_scrub`` is run, or whenever
898
inconsistencies are detected in the filesystem metadata during regular
899
operations.
900
System administrators should use the ``health`` command of ``xfs_spaceman`` to
901
download this information into a human-readable format.
902
If problems have been observed, the administrator can schedule a reduced
903
service window to run the online repair tool to correct the problem.
904
Failing that, the administrator can decide to schedule a maintenance window to
905
run the traditional offline repair tool to correct the problem.
907
**Future Work Question**: Should the health reporting integrate with the new
908
inotify fs error notification system?
909
Would it be helpful for sysadmins to have a daemon to listen for corruption
910
notifications and initiate a repair?
912
*Answer*: These questions remain unanswered, but should be a part of the
913
conversation with early adopters and potential downstream users of XFS.
915
Proposed patchsets include
916
`wiring up health reports to correction returns
917
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports>`_
918
and
919
`preservation of sickness info during memory reclaim
920
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_.
922
5. Kernel Algorithms and Data Structures
923
========================================
925
This section discusses the key algorithms and data structures of the kernel
926
code that provide the ability to check and repair metadata while the system
927
is running.
928
The first chapters in this section reveal the pieces that provide the
929
foundation for checking metadata.
930
The remainder of this section presents the mechanisms through which XFS
931
regenerates itself.
933
Self Describing Metadata
934
------------------------
936
Starting with XFS version 5 in 2012, XFS updated the format of nearly every
937
ondisk block header to record a magic number, a checksum, a universally
938
"unique" identifier (UUID), an owner code, the ondisk address of the block,
939
and a log sequence number.
940
When loading a block buffer from disk, the magic number, UUID, owner, and
941
ondisk address confirm that the retrieved block matches the specific owner of
942
the current filesystem, and that the information contained in the block is
943
supposed to be found at the ondisk address.
944
The first three components enable checking tools to disregard alleged metadata
945
that doesn't belong to the filesystem, and the fourth component enables the
946
filesystem to detect lost writes.
948
Whenever a file system operation modifies a block, the change is submitted
949
to the log as part of a transaction.
950
The log then processes these transactions marking them done once they are
951
safely persisted to storage.
952
The logging code maintains the checksum and the log sequence number of the last
953
transactional update.
954
Checksums are useful for detecting torn writes and other discrepancies that can
955
be introduced between the computer and its storage devices.
956
Sequence number tracking enables log recovery to avoid applying out of date
957
log updates to the filesystem.
959
These two features improve overall runtime resiliency by providing a means for
960
the filesystem to detect obvious corruption when reading metadata blocks from
961
disk, but these buffer verifiers cannot provide any consistency checking
962
between metadata structures.
964
For more information, please see the documentation for
965
Documentation/filesystems/xfs/xfs-self-describing-metadata.rst
967
Reverse Mapping
968
---------------
970
The original design of XFS (circa 1993) is an improvement upon 1980s Unix
971
filesystem design.
972
In those days, storage density was expensive, CPU time was scarce, and
973
excessive seek time could kill performance.
974
For performance reasons, filesystem authors were reluctant to add redundancy to
975
the filesystem, even at the cost of data integrity.
976
Filesystems designers in the early 21st century choose different strategies to
977
increase internal redundancy -- either storing nearly identical copies of
978
metadata, or more space-efficient encoding techniques.
980
For XFS, a different redundancy strategy was chosen to modernize the design:
981
a secondary space usage index that maps allocated disk extents back to their
982
owners.
983
By adding a new index, the filesystem retains most of its ability to scale
984
well to heavily threaded workloads involving large datasets, since the primary
985
file metadata (the directory tree, the file block map, and the allocation
986
groups) remain unchanged.
987
Like any system that improves redundancy, the reverse-mapping feature increases
988
overhead costs for space mapping activities.
989
However, it has two critical advantages: first, the reverse index is key to
990
enabling online fsck and other requested functionality such as free space
991
defragmentation, better media failure reporting, and filesystem shrinking.
992
Second, the different ondisk storage format of the reverse mapping btree
993
defeats device-level deduplication because the filesystem requires real
994
redundancy.
996
+--------------------------------------------------------------------------+
997
| **Sidebar**: |
998
+--------------------------------------------------------------------------+
999
| A criticism of adding the secondary index is that it does nothing to |
1000
| improve the robustness of user data storage itself. |
1001
| This is a valid point, but adding a new index for file data block |
1002
| checksums increases write amplification by turning data overwrites into |
1003
| copy-writes, which age the filesystem prematurely. |
1004
| In keeping with thirty years of precedent, users who want file data |
1005
| integrity can supply as powerful a solution as they require. |
1006
| As for metadata, the complexity of adding a new secondary index of space |
1007
| usage is much less than adding volume management and storage device |
1008
| mirroring to XFS itself. |
1009
| Perfection of RAID and volume management are best left to existing |
1010
| layers in the kernel. |
1011
+--------------------------------------------------------------------------+
1013
The information captured in a reverse space mapping record is as follows:
1015
.. code-block:: c
1017
struct xfs_rmap_irec {
1018
xfs_agblock_t rm_startblock; /* extent start block */
1019
xfs_extlen_t rm_blockcount; /* extent length */
1020
uint64_t rm_owner; /* extent owner */
1021
uint64_t rm_offset; /* offset within the owner */
1022
unsigned int rm_flags; /* state flags */
1023
};
1025
The first two fields capture the location and size of the physical space,
1026
in units of filesystem blocks.
1027
The owner field tells scrub which metadata structure or file inode have been
1028
assigned this space.
1029
For space allocated to files, the offset field tells scrub where the space was
1030
mapped within the file fork.
1031
Finally, the flags field provides extra information about the space usage --
1032
is this an attribute fork extent? A file mapping btree extent? Or an
1033
unwritten data extent?
1035
Online filesystem checking judges the consistency of each primary metadata
1036
record by comparing its information against all other space indices.
1037
The reverse mapping index plays a key role in the consistency checking process
1038
because it contains a centralized alternate copy of all space allocation
1039
information.
1040
Program runtime and ease of resource acquisition are the only real limits to
1041
what online checking can consult.
1042
For example, a file data extent mapping can be checked against:
1044
* The absence of an entry in the free space information.
1045
* The absence of an entry in the inode index.
1046
* The absence of an entry in the reference count data if the file is not
1047
marked as having shared extents.
1048
* The correspondence of an entry in the reverse mapping information.
1050
There are several observations to make about reverse mapping indices:
1052
1. Reverse mappings can provide a positive affirmation of correctness if any of
1053
the above primary metadata are in doubt.
1054
The checking code for most primary metadata follows a path similar to the
1055
one outlined above.
1057
2. Proving the consistency of secondary metadata with the primary metadata is
1058
difficult because that requires a full scan of all primary space metadata,
1059
which is very time intensive.
1060
For example, checking a reverse mapping record for a file extent mapping
1061
btree block requires locking the file and searching the entire btree to
1062
confirm the block.
1063
Instead, scrub relies on rigorous cross-referencing during the primary space
1064
mapping structure checks.
1066
3. Consistency scans must use non-blocking lock acquisition primitives if the
1067
required locking order is not the same order used by regular filesystem
1068
operations.
1069
For example, if the filesystem normally takes a file ILOCK before taking
1070
the AGF buffer lock but scrub wants to take a file ILOCK while holding
1071
an AGF buffer lock, scrub cannot block on that second acquisition.
1072
This means that forward progress during this part of a scan of the reverse
1073
mapping data cannot be guaranteed if system load is heavy.
1075
In summary, reverse mappings play a key role in reconstruction of primary
1076
metadata.
1077
The details of how these records are staged, written to disk, and committed
1078
into the filesystem are covered in subsequent sections.
1080
Checking and Cross-Referencing
1081
------------------------------
1083
The first step of checking a metadata structure is to examine every record
1084
contained within the structure and its relationship with the rest of the
1085
system.
1086
XFS contains multiple layers of checking to try to prevent inconsistent
1087
metadata from wreaking havoc on the system.
1088
Each of these layers contributes information that helps the kernel to make
1089
three decisions about the health of a metadata structure:
1091
- Is a part of this structure obviously corrupt (``XFS_SCRUB_OFLAG_CORRUPT``) ?
1092
- Is this structure inconsistent with the rest of the system
1093
(``XFS_SCRUB_OFLAG_XCORRUPT``) ?
1094
- Is there so much damage around the filesystem that cross-referencing is not
1095
possible (``XFS_SCRUB_OFLAG_XFAIL``) ?
1096
- Can the structure be optimized to improve performance or reduce the size of
1097
metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
1098
- Does the structure contain data that is not inconsistent but deserves review
1099
by the system administrator (``XFS_SCRUB_OFLAG_WARNING``) ?
1101
The following sections describe how the metadata scrubbing process works.
1103
Metadata Buffer Verification
1104
````````````````````````````
1106
The lowest layer of metadata protection in XFS are the metadata verifiers built
1107
into the buffer cache.
1108
These functions perform inexpensive internal consistency checking of the block
1109
itself, and answer these questions:
1111
- Does the block belong to this filesystem?
1113
- Does the block belong to the structure that asked for the read?
1114
This assumes that metadata blocks only have one owner, which is always true
1115
in XFS.
1117
- Is the type of data stored in the block within a reasonable range of what
1118
scrub is expecting?
1120
- Does the physical location of the block match the location it was read from?
1122
- Does the block checksum match the data?
1124
The scope of the protections here are very limited -- verifiers can only
1125
establish that the filesystem code is reasonably free of gross corruption bugs
1126
and that the storage system is reasonably competent at retrieval.
1127
Corruption problems observed at runtime cause the generation of health reports,
1128
failed system calls, and in the extreme case, filesystem shutdowns if the
1129
corrupt metadata force the cancellation of a dirty transaction.
1131
Every online fsck scrubbing function is expected to read every ondisk metadata
1132
block of a structure in the course of checking the structure.
1133
Corruption problems observed during a check are immediately reported to
1134
userspace as corruption; during a cross-reference, they are reported as a
1135
failure to cross-reference once the full examination is complete.
1136
Reads satisfied by a buffer already in cache (and hence already verified)
1137
bypass these checks.
1139
Internal Consistency Checks
1140
```````````````````````````
1142
After the buffer cache, the next level of metadata protection is the internal
1143
record verification code built into the filesystem.
1144
These checks are split between the buffer verifiers, the in-filesystem users of
1145
the buffer cache, and the scrub code itself, depending on the amount of higher
1146
level context required.
1147
The scope of checking is still internal to the block.
1148
These higher level checking functions answer these questions:
1150
- Does the type of data stored in the block match what scrub is expecting?
1152
- Does the block belong to the owning structure that asked for the read?
1154
- If the block contains records, do the records fit within the block?
1156
- If the block tracks internal free space information, is it consistent with
1157
the record areas?
1159
- Are the records contained inside the block free of obvious corruptions?
1161
Record checks in this category are more rigorous and more time-intensive.
1162
For example, block pointers and inumbers are checked to ensure that they point
1163
within the dynamically allocated parts of an allocation group and within
1164
the filesystem.
1165
Names are checked for invalid characters, and flags are checked for invalid
1166
combinations.
1167
Other record attributes are checked for sensible values.
1168
Btree records spanning an interval of the btree keyspace are checked for
1169
correct order and lack of mergeability (except for file fork mappings).
1170
For performance reasons, regular code may skip some of these checks unless
1171
debugging is enabled or a write is about to occur.
1172
Scrub functions, of course, must check all possible problems.
1174
Validation of Userspace-Controlled Record Attributes
1175
````````````````````````````````````````````````````
1177
Various pieces of filesystem metadata are directly controlled by userspace.
1178
Because of this nature, validation work cannot be more precise than checking
1179
that a value is within the possible range.
1180
These fields include:
1182
- Superblock fields controlled by mount options
1183
- Filesystem labels
1184
- File timestamps
1185
- File permissions
1186
- File size
1187
- File flags
1188
- Names present in directory entries, extended attribute keys, and filesystem
1189
labels
1190
- Extended attribute key namespaces
1191
- Extended attribute values
1192
- File data block contents
1193
- Quota limits
1194
- Quota timer expiration (if resource usage exceeds the soft limit)
1196
Cross-Referencing Space Metadata
1197
````````````````````````````````
1199
After internal block checks, the next higher level of checking is
1200
cross-referencing records between metadata structures.
1201
For regular runtime code, the cost of these checks is considered to be
1202
prohibitively expensive, but as scrub is dedicated to rooting out
1203
inconsistencies, it must pursue all avenues of inquiry.
1204
The exact set of cross-referencing is highly dependent on the context of the
1205
data structure being checked.
1207
The XFS btree code has keyspace scanning functions that online fsck uses to
1208
cross reference one structure with another.
1209
Specifically, scrub can scan the key space of an index to determine if that
1210
keyspace is fully, sparsely, or not at all mapped to records.
1211
For the reverse mapping btree, it is possible to mask parts of the key for the
1212
purposes of performing a keyspace scan so that scrub can decide if the rmap
1213
btree contains records mapping a certain extent of physical space without the
1214
sparsenses of the rest of the rmap keyspace getting in the way.
1216
Btree blocks undergo the following checks before cross-referencing:
1218
- Does the type of data stored in the block match what scrub is expecting?
1220
- Does the block belong to the owning structure that asked for the read?
1222
- Do the records fit within the block?
1224
- Are the records contained inside the block free of obvious corruptions?
1226
- Are the name hashes in the correct order?
1228
- Do node pointers within the btree point to valid block addresses for the type
1229
of btree?
1231
- Do child pointers point towards the leaves?
1233
- Do sibling pointers point across the same level?
1235
- For each node block record, does the record key accurate reflect the contents
1236
of the child block?
1238
Space allocation records are cross-referenced as follows:
1240
1. Any space mentioned by any metadata structure are cross-referenced as
1241
follows:
1243
- Does the reverse mapping index list only the appropriate owner as the
1244
owner of each block?
1246
- Are none of the blocks claimed as free space?
1248
- If these aren't file data blocks, are none of the blocks claimed as space
1249
shared by different owners?
1251
2. Btree blocks are cross-referenced as follows:
1253
- Everything in class 1 above.
1255
- If there's a parent node block, do the keys listed for this block match the
1256
keyspace of this block?
1258
- Do the sibling pointers point to valid blocks? Of the same level?
1260
- Do the child pointers point to valid blocks? Of the next level down?
1262
3. Free space btree records are cross-referenced as follows:
1264
- Everything in class 1 and 2 above.
1266
- Does the reverse mapping index list no owners of this space?
1268
- Is this space not claimed by the inode index for inodes?
1270
- Is it not mentioned by the reference count index?
1272
- Is there a matching record in the other free space btree?
1274
4. Inode btree records are cross-referenced as follows:
1276
- Everything in class 1 and 2 above.
1278
- Is there a matching record in free inode btree?
1280
- Do cleared bits in the holemask correspond with inode clusters?
1282
- Do set bits in the freemask correspond with inode records with zero link
1283
count?
1285
5. Inode records are cross-referenced as follows:
1287
- Everything in class 1.
1289
- Do all the fields that summarize information about the file forks actually
1290
match those forks?
1292
- Does each inode with zero link count correspond to a record in the free
1293
inode btree?
1295
6. File fork space mapping records are cross-referenced as follows:
1297
- Everything in class 1 and 2 above.
1299
- Is this space not mentioned by the inode btrees?
1301
- If this is a CoW fork mapping, does it correspond to a CoW entry in the
1302
reference count btree?
1304
7. Reference count records are cross-referenced as follows:
1306
- Everything in class 1 and 2 above.
1308
- Within the space subkeyspace of the rmap btree (that is to say, all
1309
records mapped to a particular space extent and ignoring the owner info),
1310
are there the same number of reverse mapping records for each block as the
1311
reference count record claims?
1313
Proposed patchsets are the series to find gaps in
1314
`refcount btree
1315
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps>`_,
1316
`inode btree
1317
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps>`_, and
1318
`rmap btree
1319
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps>`_ records;
1320
to find
1321
`mergeable records
1322
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records>`_;
1323
and to
1324
`improve cross referencing with rmap
1325
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking>`_
1326
before starting a repair.
1328
Checking Extended Attributes
1329
````````````````````````````
1331
Extended attributes implement a key-value store that enable fragments of data
1332
to be attached to any file.
1333
Both the kernel and userspace can access the keys and values, subject to
1334
namespace and privilege restrictions.
1335
Most typically these fragments are metadata about the file -- origins, security
1336
contexts, user-supplied labels, indexing information, etc.
1338
Names can be as long as 255 bytes and can exist in several different
1339
namespaces.
1340
Values can be as large as 64KB.
1341
A file's extended attributes are stored in blocks mapped by the attr fork.
1342
The mappings point to leaf blocks, remote value blocks, or dabtree blocks.
1343
Block 0 in the attribute fork is always the top of the structure, but otherwise
1344
each of the three types of blocks can be found at any offset in the attr fork.
1345
Leaf blocks contain attribute key records that point to the name and the value.
1346
Names are always stored elsewhere in the same leaf block.
1347
Values that are less than 3/4 the size of a filesystem block are also stored
1348
elsewhere in the same leaf block.
1349
Remote value blocks contain values that are too large to fit inside a leaf.
1350
If the leaf information exceeds a single filesystem block, a dabtree (also
1351
rooted at block 0) is created to map hashes of the attribute names to leaf
1352
blocks in the attr fork.
1354
Checking an extended attribute structure is not so straightforward due to the
1355
lack of separation between attr blocks and index blocks.
1356
Scrub must read each block mapped by the attr fork and ignore the non-leaf
1357
blocks:
1359
1. Walk the dabtree in the attr fork (if present) to ensure that there are no
1360
irregularities in the blocks or dabtree mappings that do not point to
1361
attr leaf blocks.
1363
2. Walk the blocks of the attr fork looking for leaf blocks.
1364
For each entry inside a leaf:
1366
a. Validate that the name does not contain invalid characters.
1368
b. Read the attr value.
1369
This performs a named lookup of the attr name to ensure the correctness
1370
of the dabtree.
1371
If the value is stored in a remote block, this also validates the
1372
integrity of the remote value block.
1374
Checking and Cross-Referencing Directories
1375
``````````````````````````````````````````
1377
The filesystem directory tree is a directed acylic graph structure, with files
1378
constituting the nodes, and directory entries (dirents) constituting the edges.
1379
Directories are a special type of file containing a set of mappings from a
1380
255-byte sequence (name) to an inumber.
1381
These are called directory entries, or dirents for short.
1382
Each directory file must have exactly one directory pointing to the file.
1383
A root directory points to itself.
1384
Directory entries point to files of any type.
1385
Each non-directory file may have multiple directories point to it.
1387
In XFS, directories are implemented as a file containing up to three 32GB
1388
partitions.
1389
The first partition contains directory entry data blocks.
1390
Each data block contains variable-sized records associating a user-provided
1391
name with an inumber and, optionally, a file type.
1392
If the directory entry data grows beyond one block, the second partition (which
1393
exists as post-EOF extents) is populated with a block containing free space
1394
information and an index that maps hashes of the dirent names to directory data
1395
blocks in the first partition.
1396
This makes directory name lookups very fast.
1397
If this second partition grows beyond one block, the third partition is
1398
populated with a linear array of free space information for faster
1399
expansions.
1400
If the free space has been separated and the second partition grows again
1401
beyond one block, then a dabtree is used to map hashes of dirent names to
1402
directory data blocks.
1404
Checking a directory is pretty straightforward:
1406
1. Walk the dabtree in the second partition (if present) to ensure that there
1407
are no irregularities in the blocks or dabtree mappings that do not point to
1408
dirent blocks.
1410
2. Walk the blocks of the first partition looking for directory entries.
1411
Each dirent is checked as follows:
1413
a. Does the name contain no invalid characters?
1415
b. Does the inumber correspond to an actual, allocated inode?
1417
c. Does the child inode have a nonzero link count?
1419
d. If a file type is included in the dirent, does it match the type of the
1420
inode?
1422
e. If the child is a subdirectory, does the child's dotdot pointer point
1423
back to the parent?
1425
f. If the directory has a second partition, perform a named lookup of the
1426
dirent name to ensure the correctness of the dabtree.
1428
3. Walk the free space list in the third partition (if present) to ensure that
1429
the free spaces it describes are really unused.
1431
Checking operations involving :ref:`parents <dirparent>` and
1432
:ref:`file link counts <nlinks>` are discussed in more detail in later
1433
sections.
1435
Checking Directory/Attribute Btrees
1436
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1438
As stated in previous sections, the directory/attribute btree (dabtree) index
1439
maps user-provided names to improve lookup times by avoiding linear scans.
1440
Internally, it maps a 32-bit hash of the name to a block offset within the
1441
appropriate file fork.
1443
The internal structure of a dabtree closely resembles the btrees that record
1444
fixed-size metadata records -- each dabtree block contains a magic number, a
1445
checksum, sibling pointers, a UUID, a tree level, and a log sequence number.
1446
The format of leaf and node records are the same -- each entry points to the
1447
next level down in the hierarchy, with dabtree node records pointing to dabtree
1448
leaf blocks, and dabtree leaf records pointing to non-dabtree blocks elsewhere
1449
in the fork.
1451
Checking and cross-referencing the dabtree is very similar to what is done for
1452
space btrees:
1454
- Does the type of data stored in the block match what scrub is expecting?
1456
- Does the block belong to the owning structure that asked for the read?
1458
- Do the records fit within the block?
1460
- Are the records contained inside the block free of obvious corruptions?
1462
- Are the name hashes in the correct order?
1464
- Do node pointers within the dabtree point to valid fork offsets for dabtree
1465
blocks?
1467
- Do leaf pointers within the dabtree point to valid fork offsets for directory
1468
or attr leaf blocks?
1470
- Do child pointers point towards the leaves?
1472
- Do sibling pointers point across the same level?
1474
- For each dabtree node record, does the record key accurate reflect the
1475
contents of the child dabtree block?
1477
- For each dabtree leaf record, does the record key accurate reflect the
1478
contents of the directory or attr block?
1480
Cross-Referencing Summary Counters
1481
``````````````````````````````````
1483
XFS maintains three classes of summary counters: available resources, quota
1484
resource usage, and file link counts.
1486
In theory, the amount of available resources (data blocks, inodes, realtime
1487
extents) can be found by walking the entire filesystem.
1488
This would make for very slow reporting, so a transactional filesystem can
1489
maintain summaries of this information in the superblock.
1490
Cross-referencing these values against the filesystem metadata should be a
1491
simple matter of walking the free space and inode metadata in each AG and the
1492
realtime bitmap, but there are complications that will be discussed in
1493
:ref:`more detail <fscounters>` later.
1495
:ref:`Quota usage <quotacheck>` and :ref:`file link count <nlinks>`
1496
checking are sufficiently complicated to warrant separate sections.
1498
Post-Repair Reverification
1499
``````````````````````````
1501
After performing a repair, the checking code is run a second time to validate
1502
the new structure, and the results of the health assessment are recorded
1503
internally and returned to the calling process.
1504
This step is critical for enabling system administrator to monitor the status
1505
of the filesystem and the progress of any repairs.
1506
For developers, it is a useful means to judge the efficacy of error detection
1507
and correction in the online and offline checking tools.
1509
Eventual Consistency vs. Online Fsck
1510
------------------------------------
1512
Complex operations can make modifications to multiple per-AG data structures
1513
with a chain of transactions.
1514
These chains, once committed to the log, are restarted during log recovery if
1515
the system crashes while processing the chain.
1516
Because the AG header buffers are unlocked between transactions within a chain,
1517
online checking must coordinate with chained operations that are in progress to
1518
avoid incorrectly detecting inconsistencies due to pending chains.
1519
Furthermore, online repair must not run when operations are pending because
1520
the metadata are temporarily inconsistent with each other, and rebuilding is
1521
not possible.
1523
Only online fsck has this requirement of total consistency of AG metadata, and
1524
should be relatively rare as compared to filesystem change operations.
1525
Online fsck coordinates with transaction chains as follows:
1527
* For each AG, maintain a count of intent items targeting that AG.
1528
The count should be bumped whenever a new item is added to the chain.
1529
The count should be dropped when the filesystem has locked the AG header
1530
buffers and finished the work.
1532
* When online fsck wants to examine an AG, it should lock the AG header
1533
buffers to quiesce all transaction chains that want to modify that AG.
1534
If the count is zero, proceed with the checking operation.
1535
If it is nonzero, cycle the buffer locks to allow the chain to make forward
1536
progress.
1538
This may lead to online fsck taking a long time to complete, but regular
1539
filesystem updates take precedence over background checking activity.
1540
Details about the discovery of this situation are presented in the
1541
:ref:`next section <chain_coordination>`, and details about the solution
1542
are presented :ref:`after that<intent_drains>`.
1544
.. _chain_coordination:
1546
Discovery of the Problem
1547
````````````````````````
1549
Midway through the development of online scrubbing, the fsstress tests
1550
uncovered a misinteraction between online fsck and compound transaction chains
1551
created by other writer threads that resulted in false reports of metadata
1552
inconsistency.
1553
The root cause of these reports is the eventual consistency model introduced by
1554
the expansion of deferred work items and compound transaction chains when
1555
reverse mapping and reflink were introduced.
1557
Originally, transaction chains were added to XFS to avoid deadlocks when
1558
unmapping space from files.
1559
Deadlock avoidance rules require that AGs only be locked in increasing order,
1560
which makes it impossible (say) to use a single transaction to free a space
1561
extent in AG 7 and then try to free a now superfluous block mapping btree block
1562
in AG 3.
1563
To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
1564
items to commit to freeing some space in one transaction while deferring the
1565
actual metadata updates to a fresh transaction.
1566
The transaction sequence looks like this:
1568
1. The first transaction contains a physical update to the file's block mapping
1569
structures to remove the mapping from the btree blocks.
1570
It then attaches to the in-memory transaction an action item to schedule
1571
deferred freeing of space.
1572
Concretely, each transaction maintains a list of ``struct
1573
xfs_defer_pending`` objects, each of which maintains a list of ``struct
1574
xfs_extent_free_item`` objects.
1575
Returning to the example above, the action item tracks the freeing of both
1576
the unmapped space from AG 7 and the block mapping btree (BMBT) block from
1577
AG 3.
1578
Deferred frees recorded in this manner are committed in the log by creating
1579
an EFI log item from the ``struct xfs_extent_free_item`` object and
1580
attaching the log item to the transaction.
1581
When the log is persisted to disk, the EFI item is written into the ondisk
1582
transaction record.
1583
EFIs can list up to 16 extents to free, all sorted in AG order.
1585
2. The second transaction contains a physical update to the free space btrees
1586
of AG 3 to release the former BMBT block and a second physical update to the
1587
free space btrees of AG 7 to release the unmapped file space.
1588
Observe that the physical updates are resequenced in the correct order
1589
when possible.
1590
Attached to the transaction is a an extent free done (EFD) log item.
1591
The EFD contains a pointer to the EFI logged in transaction #1 so that log
1592
recovery can tell if the EFI needs to be replayed.
1594
If the system goes down after transaction #1 is written back to the filesystem
1595
but before #2 is committed, a scan of the filesystem metadata would show
1596
inconsistent filesystem metadata because there would not appear to be any owner
1597
of the unmapped space.
1598
Happily, log recovery corrects this inconsistency for us -- when recovery finds
1599
an intent log item but does not find a corresponding intent done item, it will
1600
reconstruct the incore state of the intent item and finish it.
1601
In the example above, the log must replay both frees described in the recovered
1602
EFI to complete the recovery phase.
1604
There are subtleties to XFS' transaction chaining strategy to consider:
1606
* Log items must be added to a transaction in the correct order to prevent
1607
conflicts with principal objects that are not held by the transaction.
1608
In other words, all per-AG metadata updates for an unmapped block must be
1609
completed before the last update to free the extent, and extents should not
1610
be reallocated until that last update commits to the log.
1612
* AG header buffers are released between each transaction in a chain.
1613
This means that other threads can observe an AG in an intermediate state,
1614
but as long as the first subtlety is handled, this should not affect the
1615
correctness of filesystem operations.
1617
* Unmounting the filesystem flushes all pending work to disk, which means that
1618
offline fsck never sees the temporary inconsistencies caused by deferred
1619
work item processing.
1621
In this manner, XFS employs a form of eventual consistency to avoid deadlocks
1622
and increase parallelism.
1624
During the design phase of the reverse mapping and reflink features, it was
1625
decided that it was impractical to cram all the reverse mapping updates for a
1626
single filesystem change into a single transaction because a single file
1627
mapping operation can explode into many small updates:
1629
* The block mapping update itself
1630
* A reverse mapping update for the block mapping update
1631
* Fixing the freelist
1632
* A reverse mapping update for the freelist fix
1634
* A shape change to the block mapping btree
1635
* A reverse mapping update for the btree update
1636
* Fixing the freelist (again)
1637
* A reverse mapping update for the freelist fix
1639
* An update to the reference counting information
1640
* A reverse mapping update for the refcount update
1641
* Fixing the freelist (a third time)
1642
* A reverse mapping update for the freelist fix
1644
* Freeing any space that was unmapped and not owned by any other file
1645
* Fixing the freelist (a fourth time)
1646
* A reverse mapping update for the freelist fix
1648
* Freeing the space used by the block mapping btree
1649
* Fixing the freelist (a fifth time)
1650
* A reverse mapping update for the freelist fix
1652
Free list fixups are not usually needed more than once per AG per transaction
1653
chain, but it is theoretically possible if space is very tight.
1654
For copy-on-write updates this is even worse, because this must be done once to
1655
remove the space from a staging area and again to map it into the file!
1657
To deal with this explosion in a calm manner, XFS expands its use of deferred
1658
work items to cover most reverse mapping updates and all refcount updates.
1659
This reduces the worst case size of transaction reservations by breaking the
1660
work into a long chain of small updates, which increases the degree of eventual
1661
consistency in the system.
1662
Again, this generally isn't a problem because XFS orders its deferred work
1663
items carefully to avoid resource reuse conflicts between unsuspecting threads.
1665
However, online fsck changes the rules -- remember that although physical
1666
updates to per-AG structures are coordinated by locking the buffers for AG
1667
headers, buffer locks are dropped between transactions.
1668
Once scrub acquires resources and takes locks for a data structure, it must do
1669
all the validation work without releasing the lock.
1670
If the main lock for a space btree is an AG header buffer lock, scrub may have
1671
interrupted another thread that is midway through finishing a chain.
1672
For example, if a thread performing a copy-on-write has completed a reverse
1673
mapping update but not the corresponding refcount update, the two AG btrees
1674
will appear inconsistent to scrub and an observation of corruption will be
1675
recorded. This observation will not be correct.
1676
If a repair is attempted in this state, the results will be catastrophic!
1678
Several other solutions to this problem were evaluated upon discovery of this
1679
flaw and rejected:
1681
1. Add a higher level lock to allocation groups and require writer threads to
1682
acquire the higher level lock in AG order before making any changes.
1683
This would be very difficult to implement in practice because it is
1684
difficult to determine which locks need to be obtained, and in what order,
1685
without simulating the entire operation.
1686
Performing a dry run of a file operation to discover necessary locks would
1687
make the filesystem very slow.
1689
2. Make the deferred work coordinator code aware of consecutive intent items
1690
targeting the same AG and have it hold the AG header buffers locked across
1691
the transaction roll between updates.
1692
This would introduce a lot of complexity into the coordinator since it is
1693
only loosely coupled with the actual deferred work items.
1694
It would also fail to solve the problem because deferred work items can
1695
generate new deferred subtasks, but all subtasks must be complete before
1696
work can start on a new sibling task.
1698
3. Teach online fsck to walk all transactions waiting for whichever lock(s)
1699
protect the data structure being scrubbed to look for pending operations.
1700
The checking and repair operations must factor these pending operations into
1701
the evaluations being performed.
1702
This solution is a nonstarter because it is *extremely* invasive to the main
1703
filesystem.
1705
.. _intent_drains:
1707
Intent Drains
1708
`````````````
1710
Online fsck uses an atomic intent item counter and lock cycling to coordinate
1711
with transaction chains.
1712
There are two key properties to the drain mechanism.
1713
First, the counter is incremented when a deferred work item is *queued* to a
1714
transaction, and it is decremented after the associated intent done log item is
1715
*committed* to another transaction.
1716
The second property is that deferred work can be added to a transaction without
1717
holding an AG header lock, but per-AG work items cannot be marked done without
1718
locking that AG header buffer to log the physical updates and the intent done
1719
log item.
1720
The first property enables scrub to yield to running transaction chains, which
1721
is an explicit deprioritization of online fsck to benefit file operations.
1722
The second property of the drain is key to the correct coordination of scrub,
1723
since scrub will always be able to decide if a conflict is possible.
1725
For regular filesystem code, the drain works as follows:
1727
1. Call the appropriate subsystem function to add a deferred work item to a
1728
transaction.
1730
2. The function calls ``xfs_defer_drain_bump`` to increase the counter.
1732
3. When the deferred item manager wants to finish the deferred work item, it
1733
calls ``->finish_item`` to complete it.
1735
4. The ``->finish_item`` implementation logs some changes and calls
1736
``xfs_defer_drain_drop`` to decrease the sloppy counter and wake up any threads
1737
waiting on the drain.
1739
5. The subtransaction commits, which unlocks the resource associated with the
1740
intent item.
1742
For scrub, the drain works as follows:
1744
1. Lock the resource(s) associated with the metadata being scrubbed.
1745
For example, a scan of the refcount btree would lock the AGI and AGF header
1746
buffers.
1748
2. If the counter is zero (``xfs_defer_drain_busy`` returns false), there are no
1749
chains in progress and the operation may proceed.
1751
3. Otherwise, release the resources grabbed in step 1.
1753
4. Wait for the intent counter to reach zero (``xfs_defer_drain_intents``), then go
1754
back to step 1 unless a signal has been caught.
1756
To avoid polling in step 4, the drain provides a waitqueue for scrub threads to
1757
be woken up whenever the intent count drops to zero.
1759
The proposed patchset is the
1760
`scrub intent drain series
1761
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_.
1763
.. _jump_labels:
1765
Static Keys (aka Jump Label Patching)
1766
`````````````````````````````````````
1768
Online fsck for XFS separates the regular filesystem from the checking and
1769
repair code as much as possible.
1770
However, there are a few parts of online fsck (such as the intent drains, and
1771
later, live update hooks) where it is useful for the online fsck code to know
1772
what's going on in the rest of the filesystem.
1773
Since it is not expected that online fsck will be constantly running in the
1774
background, it is very important to minimize the runtime overhead imposed by
1775
these hooks when online fsck is compiled into the kernel but not actively
1776
running on behalf of userspace.
1777
Taking locks in the hot path of a writer thread to access a data structure only
1778
to find that no further action is necessary is expensive -- on the author's
1779
computer, this have an overhead of 40-50ns per access.
1780
Fortunately, the kernel supports dynamic code patching, which enables XFS to
1781
replace a static branch to hook code with ``nop`` sleds when online fsck isn't
1782
running.
1783
This sled has an overhead of however long it takes the instruction decoder to
1784
skip past the sled, which seems to be on the order of less than 1ns and
1785
does not access memory outside of instruction fetching.
1787
When online fsck enables the static key, the sled is replaced with an
1788
unconditional branch to call the hook code.
1789
The switchover is quite expensive (~22000ns) but is paid entirely by the
1790
program that invoked online fsck, and can be amortized if multiple threads
1791
enter online fsck at the same time, or if multiple filesystems are being
1792
checked at the same time.
1793
Changing the branch direction requires taking the CPU hotplug lock, and since
1794
CPU initialization requires memory allocation, online fsck must be careful not
1795
to change a static key while holding any locks or resources that could be
1796
accessed in the memory reclaim paths.
1797
To minimize contention on the CPU hotplug lock, care should be taken not to
1798
enable or disable static keys unnecessarily.
1800
Because static keys are intended to minimize hook overhead for regular
1801
filesystem operations when xfs_scrub is not running, the intended usage
1802
patterns are as follows:
1804
- The hooked part of XFS should declare a static-scoped static key that
1805
defaults to false.
1806
The ``DEFINE_STATIC_KEY_FALSE`` macro takes care of this.
1807
The static key itself should be declared as a ``static`` variable.
1809
- When deciding to invoke code that's only used by scrub, the regular
1810
filesystem should call the ``static_branch_unlikely`` predicate to avoid the
1811
scrub-only hook code if the static key is not enabled.
1813
- The regular filesystem should export helper functions that call
1814
``static_branch_inc`` to enable and ``static_branch_dec`` to disable the
1815
static key.
1816
Wrapper functions make it easy to compile out the relevant code if the kernel
1817
distributor turns off online fsck at build time.
1819
- Scrub functions wanting to turn on scrub-only XFS functionality should call
1820
the ``xchk_fsgates_enable`` from the setup function to enable a specific
1821
hook.
1822
This must be done before obtaining any resources that are used by memory
1823
reclaim.
1824
Callers had better be sure they really need the functionality gated by the
1825
static key; the ``TRY_HARDER`` flag is useful here.
1827
Online scrub has resource acquisition helpers (e.g. ``xchk_perag_lock``) to
1828
handle locking AGI and AGF buffers for all scrubber functions.
1829
If it detects a conflict between scrub and the running transactions, it will
1830
try to wait for intents to complete.
1831
If the caller of the helper has not enabled the static key, the helper will
1832
return -EDEADLOCK, which should result in the scrub being restarted with the
1833
``TRY_HARDER`` flag set.
1834
The scrub setup function should detect that flag, enable the static key, and
1835
try the scrub again.
1836
Scrub teardown disables all static keys obtained by ``xchk_fsgates_enable``.
1838
For more information, please see the kernel documentation of
1839
Documentation/staging/static-keys.rst.
1841
.. _xfile:
1843
Pageable Kernel Memory
1844
----------------------
1846
Some online checking functions work by scanning the filesystem to build a
1847
shadow copy of an ondisk metadata structure in memory and comparing the two
1848
copies.
1849
For online repair to rebuild a metadata structure, it must compute the record
1850
set that will be stored in the new structure before it can persist that new
1851
structure to disk.
1852
Ideally, repairs complete with a single atomic commit that introduces
1853
a new data structure.
1854
To meet these goals, the kernel needs to collect a large amount of information
1855
in a place that doesn't require the correct operation of the filesystem.
1857
Kernel memory isn't suitable because:
1859
* Allocating a contiguous region of memory to create a C array is very
1860
difficult, especially on 32-bit systems.
1862
* Linked lists of records introduce double pointer overhead which is very high
1863
and eliminate the possibility of indexed lookups.
1865
* Kernel memory is pinned, which can drive the system into OOM conditions.
1867
* The system might not have sufficient memory to stage all the information.
1869
At any given time, online fsck does not need to keep the entire record set in
1870
memory, which means that individual records can be paged out if necessary.
1871
Continued development of online fsck demonstrated that the ability to perform
1872
indexed data storage would also be very useful.
1873
Fortunately, the Linux kernel already has a facility for byte-addressable and
1874
pageable storage: tmpfs.
1875
In-kernel graphics drivers (most notably i915) take advantage of tmpfs files
1876
to store intermediate data that doesn't need to be in memory at all times, so
1877
that usage precedent is already established.
1878
Hence, the ``xfile`` was born!
1880
+--------------------------------------------------------------------------+
1881
| **Historical Sidebar**: |
1882
+--------------------------------------------------------------------------+
1883
| The first edition of online repair inserted records into a new btree as |
1884
| it found them, which failed because filesystem could shut down with a |
1885
| built data structure, which would be live after recovery finished. |
1886
| |
1887
| The second edition solved the half-rebuilt structure problem by storing |
1888
| everything in memory, but frequently ran the system out of memory. |
1889
| |
1890
| The third edition solved the OOM problem by using linked lists, but the |
1891
| memory overhead of the list pointers was extreme. |
1892
+--------------------------------------------------------------------------+
1894
xfile Access Models
1895
```````````````````
1897
A survey of the intended uses of xfiles suggested these use cases:
1899
1. Arrays of fixed-sized records (space management btrees, directory and
1900
extended attribute entries)
1902
2. Sparse arrays of fixed-sized records (quotas and link counts)
1904
3. Large binary objects (BLOBs) of variable sizes (directory and extended
1905
attribute names and values)
1907
4. Staging btrees in memory (reverse mapping btrees)
1909
5. Arbitrary contents (realtime space management)
1911
To support the first four use cases, high level data structures wrap the xfile
1912
to share functionality between online fsck functions.
1913
The rest of this section discusses the interfaces that the xfile presents to
1914
four of those five higher level data structures.
1915
The fifth use case is discussed in the :ref:`realtime summary <rtsummary>` case
1916
study.
1918
XFS is very record-based, which suggests that the ability to load and store
1919
complete records is important.
1920
To support these cases, a pair of ``xfile_load`` and ``xfile_store``
1921
functions are provided to read and persist objects into an xfile that treat any
1922
error as an out of memory error. For online repair, squashing error conditions
1923
in this manner is an acceptable behavior because the only reaction is to abort
1924
the operation back to userspace.
1926
However, no discussion of file access idioms is complete without answering the
1927
question, "But what about mmap?"
1928
It is convenient to access storage directly with pointers, just like userspace
1929
code does with regular memory.
1930
Online fsck must not drive the system into OOM conditions, which means that
1931
xfiles must be responsive to memory reclamation.
1932
tmpfs can only push a pagecache folio to the swap cache if the folio is neither
1933
pinned nor locked, which means the xfile must not pin too many folios.
1935
Short term direct access to xfile contents is done by locking the pagecache
1936
folio and mapping it into kernel address space. Object load and store uses this
1937
mechanism. Folio locks are not supposed to be held for long periods of time, so
1938
long term direct access to xfile contents is done by bumping the folio refcount,
1939
mapping it into kernel address space, and dropping the folio lock.
1940
These long term users *must* be responsive to memory reclaim by hooking into
1941
the shrinker infrastructure to know when to release folios.
1943
The ``xfile_get_folio`` and ``xfile_put_folio`` functions are provided to
1944
retrieve the (locked) folio that backs part of an xfile and to release it.
1945
The only code to use these folio lease functions are the xfarray
1946
:ref:`sorting<xfarray_sort>` algorithms and the :ref:`in-memory
1947
btrees<xfbtree>`.
1949
xfile Access Coordination
1950
`````````````````````````
1952
For security reasons, xfiles must be owned privately by the kernel.
1953
They are marked ``S_PRIVATE`` to prevent interference from the security system,
1954
must never be mapped into process file descriptor tables, and their pages must
1955
never be mapped into userspace processes.
1957
To avoid locking recursion issues with the VFS, all accesses to the shmfs file
1958
are performed by manipulating the page cache directly.
1959
xfile writers call the ``->write_begin`` and ``->write_end`` functions of the
1960
xfile's address space to grab writable pages, copy the caller's buffer into the
1961
page, and release the pages.
1962
xfile readers call ``shmem_read_mapping_page_gfp`` to grab pages directly
1963
before copying the contents into the caller's buffer.
1964
In other words, xfiles ignore the VFS read and write code paths to avoid
1965
having to create a dummy ``struct kiocb`` and to avoid taking inode and
1966
freeze locks.
1967
tmpfs cannot be frozen, and xfiles must not be exposed to userspace.
1969
If an xfile is shared between threads to stage repairs, the caller must provide
1970
its own locks to coordinate access.
1971
For example, if a scrub function stores scan results in an xfile and needs
1972
other threads to provide updates to the scanned data, the scrub function must
1973
provide a lock for all threads to share.
1975
.. _xfarray:
1977
Arrays of Fixed-Sized Records
1978
`````````````````````````````
1980
In XFS, each type of indexed space metadata (free space, inodes, reference
1981
counts, file fork space, and reverse mappings) consists of a set of fixed-size
1982
records indexed with a classic B+ tree.
1983
Directories have a set of fixed-size dirent records that point to the names,
1984
and extended attributes have a set of fixed-size attribute keys that point to
1985
names and values.
1986
Quota counters and file link counters index records with numbers.
1987
During a repair, scrub needs to stage new records during the gathering step and
1988
retrieve them during the btree building step.
1990
Although this requirement can be satisfied by calling the read and write
1991
methods of the xfile directly, it is simpler for callers for there to be a
1992
higher level abstraction to take care of computing array offsets, to provide
1993
iterator functions, and to deal with sparse records and sorting.
1994
The ``xfarray`` abstraction presents a linear array for fixed-size records atop
1995
the byte-accessible xfile.
1997
.. _xfarray_access_patterns:
1999
Array Access Patterns
2000
^^^^^^^^^^^^^^^^^^^^^
2002
Array access patterns in online fsck tend to fall into three categories.
2003
Iteration of records is assumed to be necessary for all cases and will be
2004
covered in the next section.
2006
The first type of caller handles records that are indexed by position.
2007
Gaps may exist between records, and a record may be updated multiple times
2008
during the collection step.
2009
In other words, these callers want a sparse linearly addressed table file.
2010
The typical use case are quota records or file link count records.
2011
Access to array elements is performed programmatically via ``xfarray_load`` and
2012
``xfarray_store`` functions, which wrap the similarly-named xfile functions to
2013
provide loading and storing of array elements at arbitrary array indices.
2014
Gaps are defined to be null records, and null records are defined to be a
2015
sequence of all zero bytes.
2016
Null records are detected by calling ``xfarray_element_is_null``.
2017
They are created either by calling ``xfarray_unset`` to null out an existing
2018
record or by never storing anything to an array index.
2020
The second type of caller handles records that are not indexed by position
2021
and do not require multiple updates to a record.
2022
The typical use case here is rebuilding space btrees and key/value btrees.
2023
These callers can add records to the array without caring about array indices
2024
via the ``xfarray_append`` function, which stores a record at the end of the
2025
array.
2026
For callers that require records to be presentable in a specific order (e.g.
2027
rebuilding btree data), the ``xfarray_sort`` function can arrange the sorted
2028
records; this function will be covered later.
2030
The third type of caller is a bag, which is useful for counting records.
2031
The typical use case here is constructing space extent reference counts from
2032
reverse mapping information.
2033
Records can be put in the bag in any order, they can be removed from the bag
2034
at any time, and uniqueness of records is left to callers.
2035
The ``xfarray_store_anywhere`` function is used to insert a record in any
2036
null record slot in the bag; and the ``xfarray_unset`` function removes a
2037
record from the bag.
2039
The proposed patchset is the
2040
`big in-memory array
2041
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array>`_.
2043
Iterating Array Elements
2044
^^^^^^^^^^^^^^^^^^^^^^^^
2046
Most users of the xfarray require the ability to iterate the records stored in
2047
the array.
2048
Callers can probe every possible array index with the following:
2050
.. code-block:: c
2052
xfarray_idx_t i;
2053
foreach_xfarray_idx(array, i) {
2054
xfarray_load(array, i, &rec);
2056
/* do something with rec */
2057
}
2059
All users of this idiom must be prepared to handle null records or must already
2060
know that there aren't any.
2062
For xfarray users that want to iterate a sparse array, the ``xfarray_iter``
2063
function ignores indices in the xfarray that have never been written to by
2064
calling ``xfile_seek_data`` (which internally uses ``SEEK_DATA``) to skip areas
2065
of the array that are not populated with memory pages.
2066
Once it finds a page, it will skip the zeroed areas of the page.
2068
.. code-block:: c
2070
xfarray_idx_t i = XFARRAY_CURSOR_INIT;
2071
while ((ret = xfarray_iter(array, &i, &rec)) == 1) {
2072
/* do something with rec */
2073
}
2075
.. _xfarray_sort:
2077
Sorting Array Elements
2078
^^^^^^^^^^^^^^^^^^^^^^
2080
During the fourth demonstration of online repair, a community reviewer remarked
2081
that for performance reasons, online repair ought to load batches of records
2082
into btree record blocks instead of inserting records into a new btree one at a
2083
time.
2084
The btree insertion code in XFS is responsible for maintaining correct ordering
2085
of the records, so naturally the xfarray must also support sorting the record
2086
set prior to bulk loading.
2088
Case Study: Sorting xfarrays
2089
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2091
The sorting algorithm used in the xfarray is actually a combination of adaptive
2092
quicksort and a heapsort subalgorithm in the spirit of
2093
`Sedgewick <https://algs4.cs.princeton.edu/23quicksort/>`_ and
2094
`pdqsort <https://github.com/orlp/pdqsort>`_, with customizations for the Linux
2095
kernel.
2096
To sort records in a reasonably short amount of time, ``xfarray`` takes
2097
advantage of the binary subpartitioning offered by quicksort, but it also uses
2098
heapsort to hedge against performance collapse if the chosen quicksort pivots
2099
are poor.
2100
Both algorithms are (in general) O(n * lg(n)), but there is a wide performance
2101
gulf between the two implementations.
2103
The Linux kernel already contains a reasonably fast implementation of heapsort.
2104
It only operates on regular C arrays, which limits the scope of its usefulness.
2105
There are two key places where the xfarray uses it:
2107
* Sorting any record subset backed by a single xfile page.
2109
* Loading a small number of xfarray records from potentially disparate parts
2110
of the xfarray into a memory buffer, and sorting the buffer.
2112
In other words, ``xfarray`` uses heapsort to constrain the nested recursion of
2113
quicksort, thereby mitigating quicksort's worst runtime behavior.
2115
Choosing a quicksort pivot is a tricky business.
2116
A good pivot splits the set to sort in half, leading to the divide and conquer
2117
behavior that is crucial to O(n * lg(n)) performance.
2118
A poor pivot barely splits the subset at all, leading to O(n\ :sup:`2`)
2119
runtime.
2120
The xfarray sort routine tries to avoid picking a bad pivot by sampling nine
2121
records into a memory buffer and using the kernel heapsort to identify the
2122
median of the nine.
2124
Most modern quicksort implementations employ Tukey's "ninther" to select a
2125
pivot from a classic C array.
2126
Typical ninther implementations pick three unique triads of records, sort each
2127
of the triads, and then sort the middle value of each triad to determine the
2128
ninther value.
2129
As stated previously, however, xfile accesses are not entirely cheap.
2130
It turned out to be much more performant to read the nine elements into a
2131
memory buffer, run the kernel's in-memory heapsort on the buffer, and choose
2132
the 4th element of that buffer as the pivot.
2133
Tukey's ninthers are described in J. W. Tukey, `The ninther, a technique for
2134
low-effort robust (resistant) location in large samples`, in *Contributions to
2135
Survey Sampling and Applied Statistics*, edited by H. David, (Academic Press,
2136
1978), pp. 251–257.
2138
The partitioning of quicksort is fairly textbook -- rearrange the record
2139
subset around the pivot, then set up the current and next stack frames to
2140
sort with the larger and the smaller halves of the pivot, respectively.
2141
This keeps the stack space requirements to log2(record count).
2143
As a final performance optimization, the hi and lo scanning phase of quicksort
2144
keeps examined xfile pages mapped in the kernel for as long as possible to
2145
reduce map/unmap cycles.
2146
Surprisingly, this reduces overall sort runtime by nearly half again after
2147
accounting for the application of heapsort directly onto xfile pages.
2149
.. _xfblob:
2151
Blob Storage
2152
````````````
2154
Extended attributes and directories add an additional requirement for staging
2155
records: arbitrary byte sequences of finite length.
2156
Each directory entry record needs to store entry name,
2157
and each extended attribute needs to store both the attribute name and value.
2158
The names, keys, and values can consume a large amount of memory, so the
2159
``xfblob`` abstraction was created to simplify management of these blobs
2160
atop an xfile.
2162
Blob arrays provide ``xfblob_load`` and ``xfblob_store`` functions to retrieve
2163
and persist objects.
2164
The store function returns a magic cookie for every object that it persists.
2165
Later, callers provide this cookie to the ``xblob_load`` to recall the object.
2166
The ``xfblob_free`` function frees a specific blob, and the ``xfblob_truncate``
2167
function frees them all because compaction is not needed.
2169
The details of repairing directories and extended attributes will be discussed
2170
in a subsequent section about atomic file content exchanges.
2171
However, it should be noted that these repair functions only use blob storage
2172
to cache a small number of entries before adding them to a temporary ondisk
2173
file, which is why compaction is not required.
2175
The proposed patchset is at the start of the
2176
`extended attribute repair
2177
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series.
2179
.. _xfbtree:
2181
In-Memory B+Trees
2182
`````````````````
2184
The chapter about :ref:`secondary metadata<secondary_metadata>` mentioned that
2185
checking and repairing of secondary metadata commonly requires coordination
2186
between a live metadata scan of the filesystem and writer threads that are
2187
updating that metadata.
2188
Keeping the scan data up to date requires the ability to propagate
2189
metadata updates from the filesystem into the data being collected by the scan.
2190
This *can* be done by appending concurrent updates into a separate log file and
2191
applying them before writing the new metadata to disk, but this leads to
2192
unbounded memory consumption if the rest of the system is very busy.
2193
Another option is to skip the side-log and commit live updates from the
2194
filesystem directly into the scan data, which trades more overhead for a lower
2195
maximum memory requirement.
2196
In both cases, the data structure holding the scan results must support indexed
2197
access to perform well.
2199
Given that indexed lookups of scan data is required for both strategies, online
2200
fsck employs the second strategy of committing live updates directly into
2201
scan data.
2202
Because xfarrays are not indexed and do not enforce record ordering, they
2203
are not suitable for this task.
2204
Conveniently, however, XFS has a library to create and maintain ordered reverse
2205
mapping records: the existing rmap btree code!
2206
If only there was a means to create one in memory.
2208
Recall that the :ref:`xfile <xfile>` abstraction represents memory pages as a
2209
regular file, which means that the kernel can create byte or block addressable
2210
virtual address spaces at will.
2211
The XFS buffer cache specializes in abstracting IO to block-oriented address
2212
spaces, which means that adaptation of the buffer cache to interface with
2213
xfiles enables reuse of the entire btree library.
2214
Btrees built atop an xfile are collectively known as ``xfbtrees``.
2215
The next few sections describe how they actually work.
2217
The proposed patchset is the
2218
`in-memory btree
2219
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=in-memory-btrees>`_
2220
series.
2222
Using xfiles as a Buffer Cache Target
2223
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2225
Two modifications are necessary to support xfiles as a buffer cache target.
2226
The first is to make it possible for the ``struct xfs_buftarg`` structure to
2227
host the ``struct xfs_buf`` rhashtable, because normally those are held by a
2228
per-AG structure.
2229
The second change is to modify the buffer ``ioapply`` function to "read" cached
2230
pages from the xfile and "write" cached pages back to the xfile.
2231
Multiple access to individual buffers is controlled by the ``xfs_buf`` lock,
2232
since the xfile does not provide any locking on its own.
2233
With this adaptation in place, users of the xfile-backed buffer cache use
2234
exactly the same APIs as users of the disk-backed buffer cache.
2235
The separation between xfile and buffer cache implies higher memory usage since
2236
they do not share pages, but this property could some day enable transactional
2237
updates to an in-memory btree.
2238
Today, however, it simply eliminates the need for new code.
2240
Space Management with an xfbtree
2241
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2243
Space management for an xfile is very simple -- each btree block is one memory
2244
page in size.
2245
These blocks use the same header format as an on-disk btree, but the in-memory
2246
block verifiers ignore the checksums, assuming that xfile memory is no more
2247
corruption-prone than regular DRAM.
2248
Reusing existing code here is more important than absolute memory efficiency.
2250
The very first block of an xfile backing an xfbtree contains a header block.
2251
The header describes the owner, height, and the block number of the root
2252
xfbtree block.
2254
To allocate a btree block, use ``xfile_seek_data`` to find a gap in the file.
2255
If there are no gaps, create one by extending the length of the xfile.
2256
Preallocate space for the block with ``xfile_prealloc``, and hand back the
2257
location.
2258
To free an xfbtree block, use ``xfile_discard`` (which internally uses
2259
``FALLOC_FL_PUNCH_HOLE``) to remove the memory page from the xfile.
2261
Populating an xfbtree
2262
^^^^^^^^^^^^^^^^^^^^^
2264
An online fsck function that wants to create an xfbtree should proceed as
2265
follows:
2267
1. Call ``xfile_create`` to create an xfile.
2269
2. Call ``xfs_alloc_memory_buftarg`` to create a buffer cache target structure
2270
pointing to the xfile.
2272
3. Pass the buffer cache target, buffer ops, and other information to
2273
``xfbtree_init`` to initialize the passed in ``struct xfbtree`` and write an
2274
initial root block to the xfile.
2275
Each btree type should define a wrapper that passes necessary arguments to
2276
the creation function.
2277
For example, rmap btrees define ``xfs_rmapbt_mem_create`` to take care of
2278
all the necessary details for callers.
2280
4. Pass the xfbtree object to the btree cursor creation function for the
2281
btree type.
2282
Following the example above, ``xfs_rmapbt_mem_cursor`` takes care of this
2283
for callers.
2285
5. Pass the btree cursor to the regular btree functions to make queries against
2286
and to update the in-memory btree.
2287
For example, a btree cursor for an rmap xfbtree can be passed to the
2288
``xfs_rmap_*`` functions just like any other btree cursor.
2289
See the :ref:`next section<xfbtree_commit>` for information on dealing with
2290
xfbtree updates that are logged to a transaction.
2292
6. When finished, delete the btree cursor, destroy the xfbtree object, free the
2293
buffer target, and the destroy the xfile to release all resources.
2295
.. _xfbtree_commit:
2297
Committing Logged xfbtree Buffers
2298
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2300
Although it is a clever hack to reuse the rmap btree code to handle the staging
2301
structure, the ephemeral nature of the in-memory btree block storage presents
2302
some challenges of its own.
2303
The XFS transaction manager must not commit buffer log items for buffers backed
2304
by an xfile because the log format does not understand updates for devices
2305
other than the data device.
2306
An ephemeral xfbtree probably will not exist by the time the AIL checkpoints
2307
log transactions back into the filesystem, and certainly won't exist during
2308
log recovery.
2309
For these reasons, any code updating an xfbtree in transaction context must
2310
remove the buffer log items from the transaction and write the updates into the
2311
backing xfile before committing or cancelling the transaction.
2313
The ``xfbtree_trans_commit`` and ``xfbtree_trans_cancel`` functions implement
2314
this functionality as follows:
2316
1. Find each buffer log item whose buffer targets the xfile.
2318
2. Record the dirty/ordered status of the log item.
2320
3. Detach the log item from the buffer.
2322
4. Queue the buffer to a special delwri list.
2324
5. Clear the transaction dirty flag if the only dirty log items were the ones
2325
that were detached in step 3.
2327
6. Submit the delwri list to commit the changes to the xfile, if the updates
2328
are being committed.
2330
After removing xfile logged buffers from the transaction in this manner, the
2331
transaction can be committed or cancelled.
2333
Bulk Loading of Ondisk B+Trees
2334
------------------------------
2336
As mentioned previously, early iterations of online repair built new btree
2337
structures by creating a new btree and adding observations individually.
2338
Loading a btree one record at a time had a slight advantage of not requiring
2339
the incore records to be sorted prior to commit, but was very slow and leaked
2340
blocks if the system went down during a repair.
2341
Loading records one at a time also meant that repair could not control the
2342
loading factor of the blocks in the new btree.
2344
Fortunately, the venerable ``xfs_repair`` tool had a more efficient means for
2345
rebuilding a btree index from a collection of records -- bulk btree loading.
2346
This was implemented rather inefficiently code-wise, since ``xfs_repair``
2347
had separate copy-pasted implementations for each btree type.
2349
To prepare for online fsck, each of the four bulk loaders were studied, notes
2350
were taken, and the four were refactored into a single generic btree bulk
2351
loading mechanism.
2352
Those notes in turn have been refreshed and are presented below.
2354
Geometry Computation
2355
````````````````````
2357
The zeroth step of bulk loading is to assemble the entire record set that will
2358
be stored in the new btree, and sort the records.
2359
Next, call ``xfs_btree_bload_compute_geometry`` to compute the shape of the
2360
btree from the record set, the type of btree, and any load factor preferences.
2361
This information is required for resource reservation.
2363
First, the geometry computation computes the minimum and maximum records that
2364
will fit in a leaf block from the size of a btree block and the size of the
2365
block header.
2366
Roughly speaking, the maximum number of records is::
2368
maxrecs = (block_size - header_size) / record_size
2370
The XFS design specifies that btree blocks should be merged when possible,
2371
which means the minimum number of records is half of maxrecs::
2373
minrecs = maxrecs / 2
2375
The next variable to determine is the desired loading factor.
2376
This must be at least minrecs and no more than maxrecs.
2377
Choosing minrecs is undesirable because it wastes half the block.
2378
Choosing maxrecs is also undesirable because adding a single record to each
2379
newly rebuilt leaf block will cause a tree split, which causes a noticeable
2380
drop in performance immediately afterwards.
2381
The default loading factor was chosen to be 75% of maxrecs, which provides a
2382
reasonably compact structure without any immediate split penalties::
2384
default_load_factor = (maxrecs + minrecs) / 2
2386
If space is tight, the loading factor will be set to maxrecs to try to avoid
2387
running out of space::
2389
leaf_load_factor = enough space ? default_load_factor : maxrecs
2391
Load factor is computed for btree node blocks using the combined size of the
2392
btree key and pointer as the record size::
2394
maxrecs = (block_size - header_size) / (key_size + ptr_size)
2395
minrecs = maxrecs / 2
2396
node_load_factor = enough space ? default_load_factor : maxrecs
2398
Once that's done, the number of leaf blocks required to store the record set
2399
can be computed as::
2401
leaf_blocks = ceil(record_count / leaf_load_factor)
2403
The number of node blocks needed to point to the next level down in the tree
2404
is computed as::
2406
n_blocks = (n == 0 ? leaf_blocks : node_blocks[n])
2407
node_blocks[n + 1] = ceil(n_blocks / node_load_factor)
2409
The entire computation is performed recursively until the current level only
2410
needs one block.
2411
The resulting geometry is as follows:
2413
- For AG-rooted btrees, this level is the root level, so the height of the new
2414
tree is ``level + 1`` and the space needed is the summation of the number of
2415
blocks on each level.
2417
- For inode-rooted btrees where the records in the top level do not fit in the
2418
inode fork area, the height is ``level + 2``, the space needed is the
2419
summation of the number of blocks on each level, and the inode fork points to
2420
the root block.
2422
- For inode-rooted btrees where the records in the top level can be stored in
2423
the inode fork area, then the root block can be stored in the inode, the
2424
height is ``level + 1``, and the space needed is one less than the summation
2425
of the number of blocks on each level.
2426
This only becomes relevant when non-bmap btrees gain the ability to root in
2427
an inode, which is a future patchset and only included here for completeness.
2429
.. _newbt:
2431
Reserving New B+Tree Blocks
2432
```````````````````````````
2434
Once repair knows the number of blocks needed for the new btree, it allocates
2435
those blocks using the free space information.
2436
Each reserved extent is tracked separately by the btree builder state data.
2437
To improve crash resilience, the reservation code also logs an Extent Freeing
2438
Intent (EFI) item in the same transaction as each space allocation and attaches
2439
its in-memory ``struct xfs_extent_free_item`` object to the space reservation.
2440
If the system goes down, log recovery will use the unfinished EFIs to free the
2441
unused space, the free space, leaving the filesystem unchanged.
2443
Each time the btree builder claims a block for the btree from a reserved
2444
extent, it updates the in-memory reservation to reflect the claimed space.
2445
Block reservation tries to allocate as much contiguous space as possible to
2446
reduce the number of EFIs in play.
2448
While repair is writing these new btree blocks, the EFIs created for the space
2449
reservations pin the tail of the ondisk log.
2450
It's possible that other parts of the system will remain busy and push the head
2451
of the log towards the pinned tail.
2452
To avoid livelocking the filesystem, the EFIs must not pin the tail of the log
2453
for too long.
2454
To alleviate this problem, the dynamic relogging capability of the deferred ops
2455
mechanism is reused here to commit a transaction at the log head containing an
2456
EFD for the old EFI and new EFI at the head.
2457
This enables the log to release the old EFI to keep the log moving forwards.
2459
EFIs have a role to play during the commit and reaping phases; please see the
2460
next section and the section about :ref:`reaping<reaping>` for more details.
2462
Proposed patchsets are the
2463
`bitmap rework
2464
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework>`_
2465
and the
2466
`preparation for bulk loading btrees
2467
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_.
2470
Writing the New Tree
2471
````````````````````
2473
This part is pretty simple -- the btree builder (``xfs_btree_bulkload``) claims
2474
a block from the reserved list, writes the new btree block header, fills the
2475
rest of the block with records, and adds the new leaf block to a list of
2476
written blocks::
2478
┌────┐
2479
│leaf│
2480
│RRR │
2481
└────┘
2483
Sibling pointers are set every time a new block is added to the level::
2485
┌────┐ ┌────┐ ┌────┐ ┌────┐
2486
│leaf│→│leaf│→│leaf│→│leaf│
2487
│RRR │←│RRR │←│RRR │←│RRR │
2488
└────┘ └────┘ └────┘ └────┘
2490
When it finishes writing the record leaf blocks, it moves on to the node
2491
blocks
2492
To fill a node block, it walks each block in the next level down in the tree
2493
to compute the relevant keys and write them into the parent node::
2495
┌────┐ ┌────┐
2496
│node│──────→│node│
2497
│PP │←──────│PP │
2498
└────┘ └────┘
2499
↙ ↘ ↙ ↘
2500
┌────┐ ┌────┐ ┌────┐ ┌────┐
2501
│leaf│→│leaf│→│leaf│→│leaf│
2502
│RRR │←│RRR │←│RRR │←│RRR │
2503
└────┘ └────┘ └────┘ └────┘
2505
When it reaches the root level, it is ready to commit the new btree!::
2507
┌─────────┐
2508
│ root │
2509
│ PP │
2510
└─────────┘
2511
↙ ↘
2512
┌────┐ ┌────┐
2513
│node│──────→│node│
2514
│PP │←──────│PP │
2515
└────┘ └────┘
2516
↙ ↘ ↙ ↘
2517
┌────┐ ┌────┐ ┌────┐ ┌────┐
2518
│leaf│→│leaf│→│leaf│→│leaf│
2519
│RRR │←│RRR │←│RRR │←│RRR │
2520
└────┘ └────┘ └────┘ └────┘
2522
The first step to commit the new btree is to persist the btree blocks to disk
2523
synchronously.
2524
This is a little complicated because a new btree block could have been freed
2525
in the recent past, so the builder must use ``xfs_buf_delwri_queue_here`` to
2526
remove the (stale) buffer from the AIL list before it can write the new blocks
2527
to disk.
2528
Blocks are queued for IO using a delwri list and written in one large batch
2529
with ``xfs_buf_delwri_submit``.
2531
Once the new blocks have been persisted to disk, control returns to the
2532
individual repair function that called the bulk loader.
2533
The repair function must log the location of the new root in a transaction,
2534
clean up the space reservations that were made for the new btree, and reap the
2535
old metadata blocks:
2537
1. Commit the location of the new btree root.
2539
2. For each incore reservation:
2541
a. Log Extent Freeing Done (EFD) items for all the space that was consumed
2542
by the btree builder. The new EFDs must point to the EFIs attached to
2543
the reservation to prevent log recovery from freeing the new blocks.
2545
b. For unclaimed portions of incore reservations, create a regular deferred
2546
extent free work item to be free the unused space later in the
2547
transaction chain.
2549
c. The EFDs and EFIs logged in steps 2a and 2b must not overrun the
2550
reservation of the committing transaction.
2551
If the btree loading code suspects this might be about to happen, it must
2552
call ``xrep_defer_finish`` to clear out the deferred work and obtain a
2553
fresh transaction.
2555
3. Clear out the deferred work a second time to finish the commit and clean
2556
the repair transaction.
2558
The transaction rolling in steps 2c and 3 represent a weakness in the repair
2559
algorithm, because a log flush and a crash before the end of the reap step can
2560
result in space leaking.
2561
Online repair functions minimize the chances of this occurring by using very
2562
large transactions, which each can accommodate many thousands of block freeing
2563
instructions.
2564
Repair moves on to reaping the old blocks, which will be presented in a
2565
subsequent :ref:`section<reaping>` after a few case studies of bulk loading.
2567
Case Study: Rebuilding the Inode Index
2568
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2570
The high level process to rebuild the inode index btree is:
2572
1. Walk the reverse mapping records to generate ``struct xfs_inobt_rec``
2573
records from the inode chunk information and a bitmap of the old inode btree
2574
blocks.
2576
2. Append the records to an xfarray in inode order.
2578
3. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
2579
of blocks needed for the inode btree.
2580
If the free space inode btree is enabled, call it again to estimate the
2581
geometry of the finobt.
2583
4. Allocate the number of blocks computed in the previous step.
2585
5. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
2586
generate the internal node blocks.
2587
If the free space inode btree is enabled, call it again to load the finobt.
2589
6. Commit the location of the new btree root block(s) to the AGI.
2591
7. Reap the old btree blocks using the bitmap created in step 1.
2593
Details are as follows.
2595
The inode btree maps inumbers to the ondisk location of the associated
2596
inode records, which means that the inode btrees can be rebuilt from the
2597
reverse mapping information.
2598
Reverse mapping records with an owner of ``XFS_RMAP_OWN_INOBT`` marks the
2599
location of the old inode btree blocks.
2600
Each reverse mapping record with an owner of ``XFS_RMAP_OWN_INODES`` marks the
2601
location of at least one inode cluster buffer.
2602
A cluster is the smallest number of ondisk inodes that can be allocated or
2603
freed in a single transaction; it is never smaller than 1 fs block or 4 inodes.
2605
For the space represented by each inode cluster, ensure that there are no
2606
records in the free space btrees nor any records in the reference count btree.
2607
If there are, the space metadata inconsistencies are reason enough to abort the
2608
operation.
2609
Otherwise, read each cluster buffer to check that its contents appear to be
2610
ondisk inodes and to decide if the file is allocated
2611
(``xfs_dinode.i_mode != 0``) or free (``xfs_dinode.i_mode == 0``).
2612
Accumulate the results of successive inode cluster buffer reads until there is
2613
enough information to fill a single inode chunk record, which is 64 consecutive
2614
numbers in the inumber keyspace.
2615
If the chunk is sparse, the chunk record may include holes.
2617
Once the repair function accumulates one chunk's worth of data, it calls
2618
``xfarray_append`` to add the inode btree record to the xfarray.
2619
This xfarray is walked twice during the btree creation step -- once to populate
2620
the inode btree with all inode chunk records, and a second time to populate the
2621
free inode btree with records for chunks that have free non-sparse inodes.
2622
The number of records for the inode btree is the number of xfarray records,
2623
but the record count for the free inode btree has to be computed as inode chunk
2624
records are stored in the xfarray.
2626
The proposed patchset is the
2627
`AG btree repair
2628
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
2629
series.
2631
Case Study: Rebuilding the Space Reference Counts
2632
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2634
Reverse mapping records are used to rebuild the reference count information.
2635
Reference counts are required for correct operation of copy on write for shared
2636
file data.
2637
Imagine the reverse mapping entries as rectangles representing extents of
2638
physical blocks, and that the rectangles can be laid down to allow them to
2639
overlap each other.
2640
From the diagram below, it is apparent that a reference count record must start
2641
or end wherever the height of the stack changes.
2642
In other words, the record emission stimulus is level-triggered::
2644
█ ███
2645
██ █████ ████ ███ ██████
2646
██ ████ ███████████ ████ █████████
2647
████████████████████████████████ ███████████
2648
^ ^ ^^ ^^ ^ ^^ ^^^ ^^^^ ^ ^^ ^ ^ ^
2649
2 1 23 21 3 43 234 2123 1 01 2 3 0
2651
The ondisk reference count btree does not store the refcount == 0 cases because
2652
the free space btree already records which blocks are free.
2653
Extents being used to stage copy-on-write operations should be the only records
2654
with refcount == 1.
2655
Single-owner file blocks aren't recorded in either the free space or the
2656
reference count btrees.
2658
The high level process to rebuild the reference count btree is:
2660
1. Walk the reverse mapping records to generate ``struct xfs_refcount_irec``
2661
records for any space having more than one reverse mapping and add them to
2662
the xfarray.
2663
Any records owned by ``XFS_RMAP_OWN_COW`` are also added to the xfarray
2664
because these are extents allocated to stage a copy on write operation and
2665
are tracked in the refcount btree.
2667
Use any records owned by ``XFS_RMAP_OWN_REFC`` to create a bitmap of old
2668
refcount btree blocks.
2670
2. Sort the records in physical extent order, putting the CoW staging extents
2671
at the end of the xfarray.
2672
This matches the sorting order of records in the refcount btree.
2674
3. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
2675
of blocks needed for the new tree.
2677
4. Allocate the number of blocks computed in the previous step.
2679
5. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
2680
generate the internal node blocks.
2682
6. Commit the location of new btree root block to the AGF.
2684
7. Reap the old btree blocks using the bitmap created in step 1.
2686
Details are as follows; the same algorithm is used by ``xfs_repair`` to
2687
generate refcount information from reverse mapping records.
2689
- Until the reverse mapping btree runs out of records:
2691
- Retrieve the next record from the btree and put it in a bag.
2693
- Collect all records with the same starting block from the btree and put
2694
them in the bag.
2696
- While the bag isn't empty:
2698
- Among the mappings in the bag, compute the lowest block number where the
2699
reference count changes.
2700
This position will be either the starting block number of the next
2701
unprocessed reverse mapping or the next block after the shortest mapping
2702
in the bag.
2704
- Remove all mappings from the bag that end at this position.
2706
- Collect all reverse mappings that start at this position from the btree
2707
and put them in the bag.
2709
- If the size of the bag changed and is greater than one, create a new
2710
refcount record associating the block number range that we just walked to
2711
the size of the bag.
2713
The bag-like structure in this case is a type 2 xfarray as discussed in the
2714
:ref:`xfarray access patterns<xfarray_access_patterns>` section.
2715
Reverse mappings are added to the bag using ``xfarray_store_anywhere`` and
2716
removed via ``xfarray_unset``.
2717
Bag members are examined through ``xfarray_iter`` loops.
2719
The proposed patchset is the
2720
`AG btree repair
2721
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
2722
series.
2724
Case Study: Rebuilding File Fork Mapping Indices
2725
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2727
The high level process to rebuild a data/attr fork mapping btree is:
2729
1. Walk the reverse mapping records to generate ``struct xfs_bmbt_rec``
2730
records from the reverse mapping records for that inode and fork.
2731
Append these records to an xfarray.
2732
Compute the bitmap of the old bmap btree blocks from the ``BMBT_BLOCK``
2733
records.
2735
2. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
2736
of blocks needed for the new tree.
2738
3. Sort the records in file offset order.
2740
4. If the extent records would fit in the inode fork immediate area, commit the
2741
records to that immediate area and skip to step 8.
2743
5. Allocate the number of blocks computed in the previous step.
2745
6. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
2746
generate the internal node blocks.
2748
7. Commit the new btree root block to the inode fork immediate area.
2750
8. Reap the old btree blocks using the bitmap created in step 1.
2752
There are some complications here:
2753
First, it's possible to move the fork offset to adjust the sizes of the
2754
immediate areas if the data and attr forks are not both in BMBT format.
2755
Second, if there are sufficiently few fork mappings, it may be possible to use
2756
EXTENTS format instead of BMBT, which may require a conversion.
2757
Third, the incore extent map must be reloaded carefully to avoid disturbing
2758
any delayed allocation extents.
2760
The proposed patchset is the
2761
`file mapping repair
2762
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings>`_
2763
series.
2765
.. _reaping:
2767
Reaping Old Metadata Blocks
2768
---------------------------
2770
Whenever online fsck builds a new data structure to replace one that is
2771
suspect, there is a question of how to find and dispose of the blocks that
2772
belonged to the old structure.
2773
The laziest method of course is not to deal with them at all, but this slowly
2774
leads to service degradations as space leaks out of the filesystem.
2775
Hopefully, someone will schedule a rebuild of the free space information to
2776
plug all those leaks.
2777
Offline repair rebuilds all space metadata after recording the usage of
2778
the files and directories that it decides not to clear, hence it can build new
2779
structures in the discovered free space and avoid the question of reaping.
2781
As part of a repair, online fsck relies heavily on the reverse mapping records
2782
to find space that is owned by the corresponding rmap owner yet truly free.
2783
Cross referencing rmap records with other rmap records is necessary because
2784
there may be other data structures that also think they own some of those
2785
blocks (e.g. crosslinked trees).
2786
Permitting the block allocator to hand them out again will not push the system
2787
towards consistency.
2789
For space metadata, the process of finding extents to dispose of generally
2790
follows this format:
2792
1. Create a bitmap of space used by data structures that must be preserved.
2793
The space reservations used to create the new metadata can be used here if
2794
the same rmap owner code is used to denote all of the objects being rebuilt.
2796
2. Survey the reverse mapping data to create a bitmap of space owned by the
2797
same ``XFS_RMAP_OWN_*`` number for the metadata that is being preserved.
2799
3. Use the bitmap disunion operator to subtract (1) from (2).
2800
The remaining set bits represent candidate extents that could be freed.
2801
The process moves on to step 4 below.
2803
Repairs for file-based metadata such as extended attributes, directories,
2804
symbolic links, quota files and realtime bitmaps are performed by building a
2805
new structure attached to a temporary file and exchanging all mappings in the
2806
file forks.
2807
Afterward, the mappings in the old file fork are the candidate blocks for
2808
disposal.
2810
The process for disposing of old extents is as follows:
2812
4. For each candidate extent, count the number of reverse mapping records for
2813
the first block in that extent that do not have the same rmap owner for the
2814
data structure being repaired.
2816
- If zero, the block has a single owner and can be freed.
2818
- If not, the block is part of a crosslinked structure and must not be
2819
freed.
2821
5. Starting with the next block in the extent, figure out how many more blocks
2822
have the same zero/nonzero other owner status as that first block.
2824
6. If the region is crosslinked, delete the reverse mapping entry for the
2825
structure being repaired and move on to the next region.
2827
7. If the region is to be freed, mark any corresponding buffers in the buffer
2828
cache as stale to prevent log writeback.
2830
8. Free the region and move on.
2832
However, there is one complication to this procedure.
2833
Transactions are of finite size, so the reaping process must be careful to roll
2834
the transactions to avoid overruns.
2835
Overruns come from two sources:
2837
a. EFIs logged on behalf of space that is no longer occupied
2839
b. Log items for buffer invalidations
2841
This is also a window in which a crash during the reaping process can leak
2842
blocks.
2843
As stated earlier, online repair functions use very large transactions to
2844
minimize the chances of this occurring.
2846
The proposed patchset is the
2847
`preparation for bulk loading btrees
2848
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_
2849
series.
2851
Case Study: Reaping After a Regular Btree Repair
2852
````````````````````````````````````````````````
2854
Old reference count and inode btrees are the easiest to reap because they have
2855
rmap records with special owner codes: ``XFS_RMAP_OWN_REFC`` for the refcount
2856
btree, and ``XFS_RMAP_OWN_INOBT`` for the inode and free inode btrees.
2857
Creating a list of extents to reap the old btree blocks is quite simple,
2858
conceptually:
2860
1. Lock the relevant AGI/AGF header buffers to prevent allocation and frees.
2862
2. For each reverse mapping record with an rmap owner corresponding to the
2863
metadata structure being rebuilt, set the corresponding range in a bitmap.
2865
3. Walk the current data structures that have the same rmap owner.
2866
For each block visited, clear that range in the above bitmap.
2868
4. Each set bit in the bitmap represents a block that could be a block from the
2869
old data structures and hence is a candidate for reaping.
2870
In other words, ``(rmap_records_owned_by & ~blocks_reachable_by_walk)``
2871
are the blocks that might be freeable.
2873
If it is possible to maintain the AGF lock throughout the repair (which is the
2874
common case), then step 2 can be performed at the same time as the reverse
2875
mapping record walk that creates the records for the new btree.
2877
Case Study: Rebuilding the Free Space Indices
2878
`````````````````````````````````````````````
2880
The high level process to rebuild the free space indices is:
2882
1. Walk the reverse mapping records to generate ``struct xfs_alloc_rec_incore``
2883
records from the gaps in the reverse mapping btree.
2885
2. Append the records to an xfarray.
2887
3. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
2888
of blocks needed for each new tree.
2890
4. Allocate the number of blocks computed in the previous step from the free
2891
space information collected.
2893
5. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
2894
generate the internal node blocks for the free space by length index.
2895
Call it again for the free space by block number index.
2897
6. Commit the locations of the new btree root blocks to the AGF.
2899
7. Reap the old btree blocks by looking for space that is not recorded by the
2900
reverse mapping btree, the new free space btrees, or the AGFL.
2902
Repairing the free space btrees has three key complications over a regular
2903
btree repair:
2905
First, free space is not explicitly tracked in the reverse mapping records.
2906
Hence, the new free space records must be inferred from gaps in the physical
2907
space component of the keyspace of the reverse mapping btree.
2909
Second, free space repairs cannot use the common btree reservation code because
2910
new blocks are reserved out of the free space btrees.
2911
This is impossible when repairing the free space btrees themselves.
2912
However, repair holds the AGF buffer lock for the duration of the free space
2913
index reconstruction, so it can use the collected free space information to
2914
supply the blocks for the new free space btrees.
2915
It is not necessary to back each reserved extent with an EFI because the new
2916
free space btrees are constructed in what the ondisk filesystem thinks is
2917
unowned space.
2918
However, if reserving blocks for the new btrees from the collected free space
2919
information changes the number of free space records, repair must re-estimate
2920
the new free space btree geometry with the new record count until the
2921
reservation is sufficient.
2922
As part of committing the new btrees, repair must ensure that reverse mappings
2923
are created for the reserved blocks and that unused reserved blocks are
2924
inserted into the free space btrees.
2925
Deferrred rmap and freeing operations are used to ensure that this transition
2926
is atomic, similar to the other btree repair functions.
2928
Third, finding the blocks to reap after the repair is not overly
2929
straightforward.
2930
Blocks for the free space btrees and the reverse mapping btrees are supplied by
2931
the AGFL.
2932
Blocks put onto the AGFL have reverse mapping records with the owner
2933
``XFS_RMAP_OWN_AG``.
2934
This ownership is retained when blocks move from the AGFL into the free space
2935
btrees or the reverse mapping btrees.
2936
When repair walks reverse mapping records to synthesize free space records, it
2937
creates a bitmap (``ag_owner_bitmap``) of all the space claimed by
2938
``XFS_RMAP_OWN_AG`` records.
2939
The repair context maintains a second bitmap corresponding to the rmap btree
2940
blocks and the AGFL blocks (``rmap_agfl_bitmap``).
2941
When the walk is complete, the bitmap disunion operation ``(ag_owner_bitmap &
2942
~rmap_agfl_bitmap)`` computes the extents that are used by the old free space
2943
btrees.
2944
These blocks can then be reaped using the methods outlined above.
2946
The proposed patchset is the
2947
`AG btree repair
2948
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
2949
series.
2951
.. _rmap_reap:
2953
Case Study: Reaping After Repairing Reverse Mapping Btrees
2954
``````````````````````````````````````````````````````````
2956
Old reverse mapping btrees are less difficult to reap after a repair.
2957
As mentioned in the previous section, blocks on the AGFL, the two free space
2958
btree blocks, and the reverse mapping btree blocks all have reverse mapping
2959
records with ``XFS_RMAP_OWN_AG`` as the owner.
2960
The full process of gathering reverse mapping records and building a new btree
2961
are described in the case study of
2962
:ref:`live rebuilds of rmap data <rmap_repair>`, but a crucial point from that
2963
discussion is that the new rmap btree will not contain any records for the old
2964
rmap btree, nor will the old btree blocks be tracked in the free space btrees.
2965
The list of candidate reaping blocks is computed by setting the bits
2966
corresponding to the gaps in the new rmap btree records, and then clearing the
2967
bits corresponding to extents in the free space btrees and the current AGFL
2968
blocks.
2969
The result ``(new_rmapbt_gaps & ~(agfl | bnobt_records))`` are reaped using the
2970
methods outlined above.
2972
The rest of the process of rebuildng the reverse mapping btree is discussed
2973
in a separate :ref:`case study<rmap_repair>`.
2975
The proposed patchset is the
2976
`AG btree repair
2977
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
2978
series.
2980
Case Study: Rebuilding the AGFL
2981
```````````````````````````````
2983
The allocation group free block list (AGFL) is repaired as follows:
2985
1. Create a bitmap for all the space that the reverse mapping data claims is
2986
owned by ``XFS_RMAP_OWN_AG``.
2988
2. Subtract the space used by the two free space btrees and the rmap btree.
2990
3. Subtract any space that the reverse mapping data claims is owned by any
2991
other owner, to avoid re-adding crosslinked blocks to the AGFL.
2993
4. Once the AGFL is full, reap any blocks leftover.
2995
5. The next operation to fix the freelist will right-size the list.
2997
See `fs/xfs/scrub/agheader_repair.c <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/xfs/scrub/agheader_repair.c>`_ for more details.
2999
Inode Record Repairs
3000
--------------------
3002
Inode records must be handled carefully, because they have both ondisk records
3003
("dinodes") and an in-memory ("cached") representation.
3004
There is a very high potential for cache coherency issues if online fsck is not
3005
careful to access the ondisk metadata *only* when the ondisk metadata is so
3006
badly damaged that the filesystem cannot load the in-memory representation.
3007
When online fsck wants to open a damaged file for scrubbing, it must use
3008
specialized resource acquisition functions that return either the in-memory
3009
representation *or* a lock on whichever object is necessary to prevent any
3010
update to the ondisk location.
3012
The only repairs that should be made to the ondisk inode buffers are whatever
3013
is necessary to get the in-core structure loaded.
3014
This means fixing whatever is caught by the inode cluster buffer and inode fork
3015
verifiers, and retrying the ``iget`` operation.
3016
If the second ``iget`` fails, the repair has failed.
3018
Once the in-memory representation is loaded, repair can lock the inode and can
3019
subject it to comprehensive checks, repairs, and optimizations.
3020
Most inode attributes are easy to check and constrain, or are user-controlled
3021
arbitrary bit patterns; these are both easy to fix.
3022
Dealing with the data and attr fork extent counts and the file block counts is
3023
more complicated, because computing the correct value requires traversing the
3024
forks, or if that fails, leaving the fields invalid and waiting for the fork
3025
fsck functions to run.
3027
The proposed patchset is the
3028
`inode
3029
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes>`_
3030
repair series.
3032
Quota Record Repairs
3033
--------------------
3035
Similar to inodes, quota records ("dquots") also have both ondisk records and
3036
an in-memory representation, and hence are subject to the same cache coherency
3037
issues.
3038
Somewhat confusingly, both are known as dquots in the XFS codebase.
3040
The only repairs that should be made to the ondisk quota record buffers are
3041
whatever is necessary to get the in-core structure loaded.
3042
Once the in-memory representation is loaded, the only attributes needing
3043
checking are obviously bad limits and timer values.
3045
Quota usage counters are checked, repaired, and discussed separately in the
3046
section about :ref:`live quotacheck <quotacheck>`.
3048
The proposed patchset is the
3049
`quota
3050
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota>`_
3051
repair series.
3053
.. _fscounters:
3055
Freezing to Fix Summary Counters
3056
--------------------------------
3058
Filesystem summary counters track availability of filesystem resources such
3059
as free blocks, free inodes, and allocated inodes.
3060
This information could be compiled by walking the free space and inode indexes,
3061
but this is a slow process, so XFS maintains a copy in the ondisk superblock
3062
that should reflect the ondisk metadata, at least when the filesystem has been
3063
unmounted cleanly.
3064
For performance reasons, XFS also maintains incore copies of those counters,
3065
which are key to enabling resource reservations for active transactions.
3066
Writer threads reserve the worst-case quantities of resources from the
3067
incore counter and give back whatever they don't use at commit time.
3068
It is therefore only necessary to serialize on the superblock when the
3069
superblock is being committed to disk.
3071
The lazy superblock counter feature introduced in XFS v5 took this even further
3072
by training log recovery to recompute the summary counters from the AG headers,
3073
which eliminated the need for most transactions even to touch the superblock.
3074
The only time XFS commits the summary counters is at filesystem unmount.
3075
To reduce contention even further, the incore counter is implemented as a
3076
percpu counter, which means that each CPU is allocated a batch of blocks from a
3077
global incore counter and can satisfy small allocations from the local batch.
3079
The high-performance nature of the summary counters makes it difficult for
3080
online fsck to check them, since there is no way to quiesce a percpu counter
3081
while the system is running.
3082
Although online fsck can read the filesystem metadata to compute the correct
3083
values of the summary counters, there's no way to hold the value of a percpu
3084
counter stable, so it's quite possible that the counter will be out of date by
3085
the time the walk is complete.
3086
Earlier versions of online scrub would return to userspace with an incomplete
3087
scan flag, but this is not a satisfying outcome for a system administrator.
3088
For repairs, the in-memory counters must be stabilized while walking the
3089
filesystem metadata to get an accurate reading and install it in the percpu
3090
counter.
3092
To satisfy this requirement, online fsck must prevent other programs in the
3093
system from initiating new writes to the filesystem, it must disable background
3094
garbage collection threads, and it must wait for existing writer programs to
3095
exit the kernel.
3096
Once that has been established, scrub can walk the AG free space indexes, the
3097
inode btrees, and the realtime bitmap to compute the correct value of all
3098
four summary counters.
3099
This is very similar to a filesystem freeze, though not all of the pieces are
3100
necessary:
3102
- The final freeze state is set one higher than ``SB_FREEZE_COMPLETE`` to
3103
prevent other threads from thawing the filesystem, or other scrub threads
3104
from initiating another fscounters freeze.
3106
- It does not quiesce the log.
3108
With this code in place, it is now possible to pause the filesystem for just
3109
long enough to check and correct the summary counters.
3111
+--------------------------------------------------------------------------+
3112
| **Historical Sidebar**: |
3113
+--------------------------------------------------------------------------+
3114
| The initial implementation used the actual VFS filesystem freeze |
3115
| mechanism to quiesce filesystem activity. |
3116
| With the filesystem frozen, it is possible to resolve the counter values |
3117
| with exact precision, but there are many problems with calling the VFS |
3118
| methods directly: |
3119
| |
3120
| - Other programs can unfreeze the filesystem without our knowledge. |
3121
| This leads to incorrect scan results and incorrect repairs. |
3122
| |
3123
| - Adding an extra lock to prevent others from thawing the filesystem |
3124
| required the addition of a ``->freeze_super`` function to wrap |
3125
| ``freeze_fs()``. |
3126
| This in turn caused other subtle problems because it turns out that |
3127
| the VFS ``freeze_super`` and ``thaw_super`` functions can drop the |
3128
| last reference to the VFS superblock, and any subsequent access |
3129
| becomes a UAF bug! |
3130
| This can happen if the filesystem is unmounted while the underlying |
3131
| block device has frozen the filesystem. |
3132
| This problem could be solved by grabbing extra references to the |
3133
| superblock, but it felt suboptimal given the other inadequacies of |
3134
| this approach. |
3135
| |
3136
| - The log need not be quiesced to check the summary counters, but a VFS |
3137
| freeze initiates one anyway. |
3138
| This adds unnecessary runtime to live fscounter fsck operations. |
3139
| |
3140
| - Quiescing the log means that XFS flushes the (possibly incorrect) |
3141
| counters to disk as part of cleaning the log. |
3142
| |
3143
| - A bug in the VFS meant that freeze could complete even when |
3144
| sync_filesystem fails to flush the filesystem and returns an error. |
3145
| This bug was fixed in Linux 5.17. |
3146
+--------------------------------------------------------------------------+
3148
The proposed patchset is the
3149
`summary counter cleanup
3150
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters>`_
3151
series.
3153
Full Filesystem Scans
3154
---------------------
3156
Certain types of metadata can only be checked by walking every file in the
3157
entire filesystem to record observations and comparing the observations against
3158
what's recorded on disk.
3159
Like every other type of online repair, repairs are made by writing those
3160
observations to disk in a replacement structure and committing it atomically.
3161
However, it is not practical to shut down the entire filesystem to examine
3162
hundreds of billions of files because the downtime would be excessive.
3163
Therefore, online fsck must build the infrastructure to manage a live scan of
3164
all the files in the filesystem.
3165
There are two questions that need to be solved to perform a live walk:
3167
- How does scrub manage the scan while it is collecting data?
3169
- How does the scan keep abreast of changes being made to the system by other
3170
threads?
3172
.. _iscan:
3174
Coordinated Inode Scans
3175
```````````````````````
3177
In the original Unix filesystems of the 1970s, each directory entry contained
3178
an index number (*inumber*) which was used as an index into on ondisk array
3179
(*itable*) of fixed-size records (*inodes*) describing a file's attributes and
3180
its data block mapping.
3181
This system is described by J. Lions, `"inode (5659)"
3182
<http://www.lemis.com/grog/Documentation/Lions/>`_ in *Lions' Commentary on
3183
UNIX, 6th Edition*, (Dept. of Computer Science, the University of New South
3184
Wales, November 1977), pp. 18-2; and later by D. Ritchie and K. Thompson,
3185
`"Implementation of the File System"
3186
<https://archive.org/details/bstj57-6-1905/page/n8/mode/1up>`_, from *The UNIX
3187
Time-Sharing System*, (The Bell System Technical Journal, July 1978), pp.
3188
1913-4.
3190
XFS retains most of this design, except now inumbers are search keys over all
3191
the space in the data section filesystem.
3192
They form a continuous keyspace that can be expressed as a 64-bit integer,
3193
though the inodes themselves are sparsely distributed within the keyspace.
3194
Scans proceed in a linear fashion across the inumber keyspace, starting from
3195
``0x0`` and ending at ``0xFFFFFFFFFFFFFFFF``.
3196
Naturally, a scan through a keyspace requires a scan cursor object to track the
3197
scan progress.
3198
Because this keyspace is sparse, this cursor contains two parts.
3199
The first part of this scan cursor object tracks the inode that will be
3200
examined next; call this the examination cursor.
3201
Somewhat less obviously, the scan cursor object must also track which parts of
3202
the keyspace have already been visited, which is critical for deciding if a
3203
concurrent filesystem update needs to be incorporated into the scan data.
3204
Call this the visited inode cursor.
3206
Advancing the scan cursor is a multi-step process encapsulated in
3207
``xchk_iscan_iter``:
3209
1. Lock the AGI buffer of the AG containing the inode pointed to by the visited
3210
inode cursor.
3211
This guarantee that inodes in this AG cannot be allocated or freed while
3212
advancing the cursor.
3214
2. Use the per-AG inode btree to look up the next inumber after the one that
3215
was just visited, since it may not be keyspace adjacent.
3217
3. If there are no more inodes left in this AG:
3219
a. Move the examination cursor to the point of the inumber keyspace that
3220
corresponds to the start of the next AG.
3222
b. Adjust the visited inode cursor to indicate that it has "visited" the
3223
last possible inode in the current AG's inode keyspace.
3224
XFS inumbers are segmented, so the cursor needs to be marked as having
3225
visited the entire keyspace up to just before the start of the next AG's
3226
inode keyspace.
3228
c. Unlock the AGI and return to step 1 if there are unexamined AGs in the
3229
filesystem.
3231
d. If there are no more AGs to examine, set both cursors to the end of the
3232
inumber keyspace.
3233
The scan is now complete.
3235
4. Otherwise, there is at least one more inode to scan in this AG:
3237
a. Move the examination cursor ahead to the next inode marked as allocated
3238
by the inode btree.
3240
b. Adjust the visited inode cursor to point to the inode just prior to where
3241
the examination cursor is now.
3242
Because the scanner holds the AGI buffer lock, no inodes could have been
3243
created in the part of the inode keyspace that the visited inode cursor
3244
just advanced.
3246
5. Get the incore inode for the inumber of the examination cursor.
3247
By maintaining the AGI buffer lock until this point, the scanner knows that
3248
it was safe to advance the examination cursor across the entire keyspace,
3249
and that it has stabilized this next inode so that it cannot disappear from
3250
the filesystem until the scan releases the incore inode.
3252
6. Drop the AGI lock and return the incore inode to the caller.
3254
Online fsck functions scan all files in the filesystem as follows:
3256
1. Start a scan by calling ``xchk_iscan_start``.
3258
2. Advance the scan cursor (``xchk_iscan_iter``) to get the next inode.
3259
If one is provided:
3261
a. Lock the inode to prevent updates during the scan.
3263
b. Scan the inode.
3265
c. While still holding the inode lock, adjust the visited inode cursor
3266
(``xchk_iscan_mark_visited``) to point to this inode.
3268
d. Unlock and release the inode.
3270
8. Call ``xchk_iscan_teardown`` to complete the scan.
3272
There are subtleties with the inode cache that complicate grabbing the incore
3273
inode for the caller.
3274
Obviously, it is an absolute requirement that the inode metadata be consistent
3275
enough to load it into the inode cache.
3276
Second, if the incore inode is stuck in some intermediate state, the scan
3277
coordinator must release the AGI and push the main filesystem to get the inode
3278
back into a loadable state.
3280
The proposed patches are the
3281
`inode scanner
3282
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_
3283
series.
3284
The first user of the new functionality is the
3285
`online quotacheck
3286
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_
3287
series.
3289
Inode Management
3290
````````````````
3292
In regular filesystem code, references to allocated XFS incore inodes are
3293
always obtained (``xfs_iget``) outside of transaction context because the
3294
creation of the incore context for an existing file does not require metadata
3295
updates.
3296
However, it is important to note that references to incore inodes obtained as
3297
part of file creation must be performed in transaction context because the
3298
filesystem must ensure the atomicity of the ondisk inode btree index updates
3299
and the initialization of the actual ondisk inode.
3301
References to incore inodes are always released (``xfs_irele``) outside of
3302
transaction context because there are a handful of activities that might
3303
require ondisk updates:
3305
- The VFS may decide to kick off writeback as part of a ``DONTCACHE`` inode
3306
release.
3308
- Speculative preallocations need to be unreserved.
3310
- An unlinked file may have lost its last reference, in which case the entire
3311
file must be inactivated, which involves releasing all of its resources in
3312
the ondisk metadata and freeing the inode.
3314
These activities are collectively called inode inactivation.
3315
Inactivation has two parts -- the VFS part, which initiates writeback on all
3316
dirty file pages, and the XFS part, which cleans up XFS-specific information
3317
and frees the inode if it was unlinked.
3318
If the inode is unlinked (or unconnected after a file handle operation), the
3319
kernel drops the inode into the inactivation machinery immediately.
3321
During normal operation, resource acquisition for an update follows this order
3322
to avoid deadlocks:
3324
1. Inode reference (``iget``).
3326
2. Filesystem freeze protection, if repairing (``mnt_want_write_file``).
3328
3. Inode ``IOLOCK`` (VFS ``i_rwsem``) lock to control file IO.
3330
4. Inode ``MMAPLOCK`` (page cache ``invalidate_lock``) lock for operations that
3331
can update page cache mappings.
3333
5. Log feature enablement.
3335
6. Transaction log space grant.
3337
7. Space on the data and realtime devices for the transaction.
3339
8. Incore dquot references, if a file is being repaired.
3340
Note that they are not locked, merely acquired.
3342
9. Inode ``ILOCK`` for file metadata updates.
3344
10. AG header buffer locks / Realtime metadata inode ILOCK.
3346
11. Realtime metadata buffer locks, if applicable.
3348
12. Extent mapping btree blocks, if applicable.
3350
Resources are often released in the reverse order, though this is not required.
3351
However, online fsck differs from regular XFS operations because it may examine
3352
an object that normally is acquired in a later stage of the locking order, and
3353
then decide to cross-reference the object with an object that is acquired
3354
earlier in the order.
3355
The next few sections detail the specific ways in which online fsck takes care
3356
to avoid deadlocks.
3358
iget and irele During a Scrub
3359
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3361
An inode scan performed on behalf of a scrub operation runs in transaction
3362
context, and possibly with resources already locked and bound to it.
3363
This isn't much of a problem for ``iget`` since it can operate in the context
3364
of an existing transaction, as long as all of the bound resources are acquired
3365
before the inode reference in the regular filesystem.
3367
When the VFS ``iput`` function is given a linked inode with no other
3368
references, it normally puts the inode on an LRU list in the hope that it can
3369
save time if another process re-opens the file before the system runs out
3370
of memory and frees it.
3371
Filesystem callers can short-circuit the LRU process by setting a ``DONTCACHE``
3372
flag on the inode to cause the kernel to try to drop the inode into the
3373
inactivation machinery immediately.
3375
In the past, inactivation was always done from the process that dropped the
3376
inode, which was a problem for scrub because scrub may already hold a
3377
transaction, and XFS does not support nesting transactions.
3378
On the other hand, if there is no scrub transaction, it is desirable to drop
3379
otherwise unused inodes immediately to avoid polluting caches.
3380
To capture these nuances, the online fsck code has a separate ``xchk_irele``
3381
function to set or clear the ``DONTCACHE`` flag to get the required release
3382
behavior.
3384
Proposed patchsets include fixing
3385
`scrub iget usage
3386
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iget-fixes>`_ and
3387
`dir iget usage
3388
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-dir-iget-fixes>`_.
3390
.. _ilocking:
3392
Locking Inodes
3393
^^^^^^^^^^^^^^
3395
In regular filesystem code, the VFS and XFS will acquire multiple IOLOCK locks
3396
in a well-known order: parent → child when updating the directory tree, and
3397
in numerical order of the addresses of their ``struct inode`` object otherwise.
3398
For regular files, the MMAPLOCK can be acquired after the IOLOCK to stop page
3399
faults.
3400
If two MMAPLOCKs must be acquired, they are acquired in numerical order of
3401
the addresses of their ``struct address_space`` objects.
3402
Due to the structure of existing filesystem code, IOLOCKs and MMAPLOCKs must be
3403
acquired before transactions are allocated.
3404
If two ILOCKs must be acquired, they are acquired in inumber order.
3406
Inode lock acquisition must be done carefully during a coordinated inode scan.
3407
Online fsck cannot abide these conventions, because for a directory tree
3408
scanner, the scrub process holds the IOLOCK of the file being scanned and it
3409
needs to take the IOLOCK of the file at the other end of the directory link.
3410
If the directory tree is corrupt because it contains a cycle, ``xfs_scrub``
3411
cannot use the regular inode locking functions and avoid becoming trapped in an
3412
ABBA deadlock.
3414
Solving both of these problems is straightforward -- any time online fsck
3415
needs to take a second lock of the same class, it uses trylock to avoid an ABBA
3416
deadlock.
3417
If the trylock fails, scrub drops all inode locks and use trylock loops to
3418
(re)acquire all necessary resources.
3419
Trylock loops enable scrub to check for pending fatal signals, which is how
3420
scrub avoids deadlocking the filesystem or becoming an unresponsive process.
3421
However, trylock loops means that online fsck must be prepared to measure the
3422
resource being scrubbed before and after the lock cycle to detect changes and
3423
react accordingly.
3425
.. _dirparent:
3427
Case Study: Finding a Directory Parent
3428
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3430
Consider the directory parent pointer repair code as an example.
3431
Online fsck must verify that the dotdot dirent of a directory points up to a
3432
parent directory, and that the parent directory contains exactly one dirent
3433
pointing down to the child directory.
3434
Fully validating this relationship (and repairing it if possible) requires a
3435
walk of every directory on the filesystem while holding the child locked, and
3436
while updates to the directory tree are being made.
3437
The coordinated inode scan provides a way to walk the filesystem without the
3438
possibility of missing an inode.
3439
The child directory is kept locked to prevent updates to the dotdot dirent, but
3440
if the scanner fails to lock a parent, it can drop and relock both the child
3441
and the prospective parent.
3442
If the dotdot entry changes while the directory is unlocked, then a move or
3443
rename operation must have changed the child's parentage, and the scan can
3444
exit early.
3446
The proposed patchset is the
3447
`directory repair
3448
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_
3449
series.
3451
.. _fshooks:
3453
Filesystem Hooks
3454
`````````````````
3456
The second piece of support that online fsck functions need during a full
3457
filesystem scan is the ability to stay informed about updates being made by
3458
other threads in the filesystem, since comparisons against the past are useless
3459
in a dynamic environment.
3460
Two pieces of Linux kernel infrastructure enable online fsck to monitor regular
3461
filesystem operations: filesystem hooks and :ref:`static keys<jump_labels>`.
3463
Filesystem hooks convey information about an ongoing filesystem operation to
3464
a downstream consumer.
3465
In this case, the downstream consumer is always an online fsck function.
3466
Because multiple fsck functions can run in parallel, online fsck uses the Linux
3467
notifier call chain facility to dispatch updates to any number of interested
3468
fsck processes.
3469
Call chains are a dynamic list, which means that they can be configured at
3470
run time.
3471
Because these hooks are private to the XFS module, the information passed along
3472
contains exactly what the checking function needs to update its observations.
3474
The current implementation of XFS hooks uses SRCU notifier chains to reduce the
3475
impact to highly threaded workloads.
3476
Regular blocking notifier chains use a rwsem and seem to have a much lower
3477
overhead for single-threaded applications.
3478
However, it may turn out that the combination of blocking chains and static
3479
keys are a more performant combination; more study is needed here.
3481
The following pieces are necessary to hook a certain point in the filesystem:
3483
- A ``struct xfs_hooks`` object must be embedded in a convenient place such as
3484
a well-known incore filesystem object.
3486
- Each hook must define an action code and a structure containing more context
3487
about the action.
3489
- Hook providers should provide appropriate wrapper functions and structs
3490
around the ``xfs_hooks`` and ``xfs_hook`` objects to take advantage of type
3491
checking to ensure correct usage.
3493
- A callsite in the regular filesystem code must be chosen to call
3494
``xfs_hooks_call`` with the action code and data structure.
3495
This place should be adjacent to (and not earlier than) the place where
3496
the filesystem update is committed to the transaction.
3497
In general, when the filesystem calls a hook chain, it should be able to
3498
handle sleeping and should not be vulnerable to memory reclaim or locking
3499
recursion.
3500
However, the exact requirements are very dependent on the context of the hook
3501
caller and the callee.
3503
- The online fsck function should define a structure to hold scan data, a lock
3504
to coordinate access to the scan data, and a ``struct xfs_hook`` object.
3505
The scanner function and the regular filesystem code must acquire resources
3506
in the same order; see the next section for details.
3508
- The online fsck code must contain a C function to catch the hook action code
3509
and data structure.
3510
If the object being updated has already been visited by the scan, then the
3511
hook information must be applied to the scan data.
3513
- Prior to unlocking inodes to start the scan, online fsck must call
3514
``xfs_hooks_setup`` to initialize the ``struct xfs_hook``, and
3515
``xfs_hooks_add`` to enable the hook.
3517
- Online fsck must call ``xfs_hooks_del`` to disable the hook once the scan is
3518
complete.
3520
The number of hooks should be kept to a minimum to reduce complexity.
3521
Static keys are used to reduce the overhead of filesystem hooks to nearly
3522
zero when online fsck is not running.
3524
.. _liveupdate:
3526
Live Updates During a Scan
3527
``````````````````````````
3529
The code paths of the online fsck scanning code and the :ref:`hooked<fshooks>`
3530
filesystem code look like this::
3532
other program
3533
↓
3534
inode lock ←────────────────────┐
3535
↓ │
3536
AG header lock │
3537
↓ │
3538
filesystem function │
3539
↓ │
3540
notifier call chain │ same
3541
↓ ├─── inode
3542
scrub hook function │ lock
3543
↓ │
3544
scan data mutex ←──┐ same │
3545
↓ ├─── scan │
3546
update scan data │ lock │
3547
↑ │ │
3548
scan data mutex ←──┘ │
3549
↑ │
3550
inode lock ←────────────────────┘
3551
↑
3552
scrub function
3553
↑
3554
inode scanner
3555
↑
3556
xfs_scrub
3558
These rules must be followed to ensure correct interactions between the
3559
checking code and the code making an update to the filesystem:
3561
- Prior to invoking the notifier call chain, the filesystem function being
3562
hooked must acquire the same lock that the scrub scanning function acquires
3563
to scan the inode.
3565
- The scanning function and the scrub hook function must coordinate access to
3566
the scan data by acquiring a lock on the scan data.
3568
- Scrub hook function must not add the live update information to the scan
3569
observations unless the inode being updated has already been scanned.
3570
The scan coordinator has a helper predicate (``xchk_iscan_want_live_update``)
3571
for this.
3573
- Scrub hook functions must not change the caller's state, including the
3574
transaction that it is running.
3575
They must not acquire any resources that might conflict with the filesystem
3576
function being hooked.
3578
- The hook function can abort the inode scan to avoid breaking the other rules.
3580
The inode scan APIs are pretty simple:
3582
- ``xchk_iscan_start`` starts a scan
3584
- ``xchk_iscan_iter`` grabs a reference to the next inode in the scan or
3585
returns zero if there is nothing left to scan
3587
- ``xchk_iscan_want_live_update`` to decide if an inode has already been
3588
visited in the scan.
3589
This is critical for hook functions to decide if they need to update the
3590
in-memory scan information.
3592
- ``xchk_iscan_mark_visited`` to mark an inode as having been visited in the
3593
scan
3595
- ``xchk_iscan_teardown`` to finish the scan
3597
This functionality is also a part of the
3598
`inode scanner
3599
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_
3600
series.
3602
.. _quotacheck:
3604
Case Study: Quota Counter Checking
3605
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3607
It is useful to compare the mount time quotacheck code to the online repair
3608
quotacheck code.
3609
Mount time quotacheck does not have to contend with concurrent operations, so
3610
it does the following:
3612
1. Make sure the ondisk dquots are in good enough shape that all the incore
3613
dquots will actually load, and zero the resource usage counters in the
3614
ondisk buffer.
3616
2. Walk every inode in the filesystem.
3617
Add each file's resource usage to the incore dquot.
3619
3. Walk each incore dquot.
3620
If the incore dquot is not being flushed, add the ondisk buffer backing the
3621
incore dquot to a delayed write (delwri) list.
3623
4. Write the buffer list to disk.
3625
Like most online fsck functions, online quotacheck can't write to regular
3626
filesystem objects until the newly collected metadata reflect all filesystem
3627
state.
3628
Therefore, online quotacheck records file resource usage to a shadow dquot
3629
index implemented with a sparse ``xfarray``, and only writes to the real dquots
3630
once the scan is complete.
3631
Handling transactional updates is tricky because quota resource usage updates
3632
are handled in phases to minimize contention on dquots:
3634
1. The inodes involved are joined and locked to a transaction.
3636
2. For each dquot attached to the file:
3638
a. The dquot is locked.
3640
b. A quota reservation is added to the dquot's resource usage.
3641
The reservation is recorded in the transaction.
3643
c. The dquot is unlocked.
3645
3. Changes in actual quota usage are tracked in the transaction.
3647
4. At transaction commit time, each dquot is examined again:
3649
a. The dquot is locked again.
3651
b. Quota usage changes are logged and unused reservation is given back to
3652
the dquot.
3654
c. The dquot is unlocked.
3656
For online quotacheck, hooks are placed in steps 2 and 4.
3657
The step 2 hook creates a shadow version of the transaction dquot context
3658
(``dqtrx``) that operates in a similar manner to the regular code.
3659
The step 4 hook commits the shadow ``dqtrx`` changes to the shadow dquots.
3660
Notice that both hooks are called with the inode locked, which is how the
3661
live update coordinates with the inode scanner.
3663
The quotacheck scan looks like this:
3665
1. Set up a coordinated inode scan.
3667
2. For each inode returned by the inode scan iterator:
3669
a. Grab and lock the inode.
3671
b. Determine that inode's resource usage (data blocks, inode counts,
3672
realtime blocks) and add that to the shadow dquots for the user, group,
3673
and project ids associated with the inode.
3675
c. Unlock and release the inode.
3677
3. For each dquot in the system:
3679
a. Grab and lock the dquot.
3681
b. Check the dquot against the shadow dquots created by the scan and updated
3682
by the live hooks.
3684
Live updates are key to being able to walk every quota record without
3685
needing to hold any locks for a long duration.
3686
If repairs are desired, the real and shadow dquots are locked and their
3687
resource counts are set to the values in the shadow dquot.
3689
The proposed patchset is the
3690
`online quotacheck
3691
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_
3692
series.
3694
.. _nlinks:
3696
Case Study: File Link Count Checking
3697
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3699
File link count checking also uses live update hooks.
3700
The coordinated inode scanner is used to visit all directories on the
3701
filesystem, and per-file link count records are stored in a sparse ``xfarray``
3702
indexed by inumber.
3703
During the scanning phase, each entry in a directory generates observation
3704
data as follows:
3706
1. If the entry is a dotdot (``'..'``) entry of the root directory, the
3707
directory's parent link count is bumped because the root directory's dotdot
3708
entry is self referential.
3710
2. If the entry is a dotdot entry of a subdirectory, the parent's backref
3711
count is bumped.
3713
3. If the entry is neither a dot nor a dotdot entry, the target file's parent
3714
count is bumped.
3716
4. If the target is a subdirectory, the parent's child link count is bumped.
3718
A crucial point to understand about how the link count inode scanner interacts
3719
with the live update hooks is that the scan cursor tracks which *parent*
3720
directories have been scanned.
3721
In other words, the live updates ignore any update about ``A → B`` when A has
3722
not been scanned, even if B has been scanned.
3723
Furthermore, a subdirectory A with a dotdot entry pointing back to B is
3724
accounted as a backref counter in the shadow data for A, since child dotdot
3725
entries affect the parent's link count.
3726
Live update hooks are carefully placed in all parts of the filesystem that
3727
create, change, or remove directory entries, since those operations involve
3728
bumplink and droplink.
3730
For any file, the correct link count is the number of parents plus the number
3731
of child subdirectories.
3732
Non-directories never have children of any kind.
3733
The backref information is used to detect inconsistencies in the number of
3734
links pointing to child subdirectories and the number of dotdot entries
3735
pointing back.
3737
After the scan completes, the link count of each file can be checked by locking
3738
both the inode and the shadow data, and comparing the link counts.
3739
A second coordinated inode scan cursor is used for comparisons.
3740
Live updates are key to being able to walk every inode without needing to hold
3741
any locks between inodes.
3742
If repairs are desired, the inode's link count is set to the value in the
3743
shadow information.
3744
If no parents are found, the file must be :ref:`reparented <orphanage>` to the
3745
orphanage to prevent the file from being lost forever.
3747
The proposed patchset is the
3748
`file link count repair
3749
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks>`_
3750
series.
3752
.. _rmap_repair:
3754
Case Study: Rebuilding Reverse Mapping Records
3755
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3757
Most repair functions follow the same pattern: lock filesystem resources,
3758
walk the surviving ondisk metadata looking for replacement metadata records,
3759
and use an :ref:`in-memory array <xfarray>` to store the gathered observations.
3760
The primary advantage of this approach is the simplicity and modularity of the
3761
repair code -- code and data are entirely contained within the scrub module,
3762
do not require hooks in the main filesystem, and are usually the most efficient
3763
in memory use.
3764
A secondary advantage of this repair approach is atomicity -- once the kernel
3765
decides a structure is corrupt, no other threads can access the metadata until
3766
the kernel finishes repairing and revalidating the metadata.
3768
For repairs going on within a shard of the filesystem, these advantages
3769
outweigh the delays inherent in locking the shard while repairing parts of the
3770
shard.
3771
Unfortunately, repairs to the reverse mapping btree cannot use the "standard"
3772
btree repair strategy because it must scan every space mapping of every fork of
3773
every file in the filesystem, and the filesystem cannot stop.
3774
Therefore, rmap repair foregoes atomicity between scrub and repair.
3775
It combines a :ref:`coordinated inode scanner <iscan>`, :ref:`live update hooks
3776
<liveupdate>`, and an :ref:`in-memory rmap btree <xfbtree>` to complete the
3777
scan for reverse mapping records.
3779
1. Set up an xfbtree to stage rmap records.
3781
2. While holding the locks on the AGI and AGF buffers acquired during the
3782
scrub, generate reverse mappings for all AG metadata: inodes, btrees, CoW
3783
staging extents, and the internal log.
3785
3. Set up an inode scanner.
3787
4. Hook into rmap updates for the AG being repaired so that the live scan data
3788
can receive updates to the rmap btree from the rest of the filesystem during
3789
the file scan.
3791
5. For each space mapping found in either fork of each file scanned,
3792
decide if the mapping matches the AG of interest.
3793
If so:
3795
a. Create a btree cursor for the in-memory btree.
3797
b. Use the rmap code to add the record to the in-memory btree.
3799
c. Use the :ref:`special commit function <xfbtree_commit>` to write the
3800
xfbtree changes to the xfile.
3802
6. For each live update received via the hook, decide if the owner has already
3803
been scanned.
3804
If so, apply the live update into the scan data:
3806
a. Create a btree cursor for the in-memory btree.
3808
b. Replay the operation into the in-memory btree.
3810
c. Use the :ref:`special commit function <xfbtree_commit>` to write the
3811
xfbtree changes to the xfile.
3812
This is performed with an empty transaction to avoid changing the
3813
caller's state.
3815
7. When the inode scan finishes, create a new scrub transaction and relock the
3816
two AG headers.
3818
8. Compute the new btree geometry using the number of rmap records in the
3819
shadow btree, like all other btree rebuilding functions.
3821
9. Allocate the number of blocks computed in the previous step.
3823
10. Perform the usual btree bulk loading and commit to install the new rmap
3824
btree.
3826
11. Reap the old rmap btree blocks as discussed in the case study about how
3827
to :ref:`reap after rmap btree repair <rmap_reap>`.
3829
12. Free the xfbtree now that it not needed.
3831
The proposed patchset is the
3832
`rmap repair
3833
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree>`_
3834
series.
3836
Staging Repairs with Temporary Files on Disk
3837
--------------------------------------------
3839
XFS stores a substantial amount of metadata in file forks: directories,
3840
extended attributes, symbolic link targets, free space bitmaps and summary
3841
information for the realtime volume, and quota records.
3842
File forks map 64-bit logical file fork space extents to physical storage space
3843
extents, similar to how a memory management unit maps 64-bit virtual addresses
3844
to physical memory addresses.
3845
Therefore, file-based tree structures (such as directories and extended
3846
attributes) use blocks mapped in the file fork offset address space that point
3847
to other blocks mapped within that same address space, and file-based linear
3848
structures (such as bitmaps and quota records) compute array element offsets in
3849
the file fork offset address space.
3851
Because file forks can consume as much space as the entire filesystem, repairs
3852
cannot be staged in memory, even when a paging scheme is available.
3853
Therefore, online repair of file-based metadata createas a temporary file in
3854
the XFS filesystem, writes a new structure at the correct offsets into the
3855
temporary file, and atomically exchanges all file fork mappings (and hence the
3856
fork contents) to commit the repair.
3857
Once the repair is complete, the old fork can be reaped as necessary; if the
3858
system goes down during the reap, the iunlink code will delete the blocks
3859
during log recovery.
3861
**Note**: All space usage and inode indices in the filesystem *must* be
3862
consistent to use a temporary file safely!
3863
This dependency is the reason why online repair can only use pageable kernel
3864
memory to stage ondisk space usage information.
3866
Exchanging metadata file mappings with a temporary file requires the owner
3867
field of the block headers to match the file being repaired and not the
3868
temporary file.
3869
The directory, extended attribute, and symbolic link functions were all
3870
modified to allow callers to specify owner numbers explicitly.
3872
There is a downside to the reaping process -- if the system crashes during the
3873
reap phase and the fork extents are crosslinked, the iunlink processing will
3874
fail because freeing space will find the extra reverse mappings and abort.
3876
Temporary files created for repair are similar to ``O_TMPFILE`` files created
3877
by userspace.
3878
They are not linked into a directory and the entire file will be reaped when
3879
the last reference to the file is lost.
3880
The key differences are that these files must have no access permission outside
3881
the kernel at all, they must be specially marked to prevent them from being
3882
opened by handle, and they must never be linked into the directory tree.
3884
+--------------------------------------------------------------------------+
3885
| **Historical Sidebar**: |
3886
+--------------------------------------------------------------------------+
3887
| In the initial iteration of file metadata repair, the damaged metadata |
3888
| blocks would be scanned for salvageable data; the extents in the file |
3889
| fork would be reaped; and then a new structure would be built in its |
3890
| place. |
3891
| This strategy did not survive the introduction of the atomic repair |
3892
| requirement expressed earlier in this document. |
3893
| |
3894
| The second iteration explored building a second structure at a high |
3895
| offset in the fork from the salvage data, reaping the old extents, and |
3896
| using a ``COLLAPSE_RANGE`` operation to slide the new extents into |
3897
| place. |
3898
| |
3899
| This had many drawbacks: |
3900
| |
3901
| - Array structures are linearly addressed, and the regular filesystem |
3902
| codebase does not have the concept of a linear offset that could be |
3903
| applied to the record offset computation to build an alternate copy. |
3904
| |
3905
| - Extended attributes are allowed to use the entire attr fork offset |
3906
| address space. |
3907
| |
3908
| - Even if repair could build an alternate copy of a data structure in a |
3909
| different part of the fork address space, the atomic repair commit |
3910
| requirement means that online repair would have to be able to perform |
3911
| a log assisted ``COLLAPSE_RANGE`` operation to ensure that the old |
3912
| structure was completely replaced. |
3913
| |
3914
| - A crash after construction of the secondary tree but before the range |
3915
| collapse would leave unreachable blocks in the file fork. |
3916
| This would likely confuse things further. |
3917
| |
3918
| - Reaping blocks after a repair is not a simple operation, and |
3919
| initiating a reap operation from a restarted range collapse operation |
3920
| during log recovery is daunting. |
3921
| |
3922
| - Directory entry blocks and quota records record the file fork offset |
3923
| in the header area of each block. |
3924
| An atomic range collapse operation would have to rewrite this part of |
3925
| each block header. |
3926
| Rewriting a single field in block headers is not a huge problem, but |
3927
| it's something to be aware of. |
3928
| |
3929
| - Each block in a directory or extended attributes btree index contains |
3930
| sibling and child block pointers. |
3931
| Were the atomic commit to use a range collapse operation, each block |
3932
| would have to be rewritten very carefully to preserve the graph |
3933
| structure. |
3934
| Doing this as part of a range collapse means rewriting a large number |
3935
| of blocks repeatedly, which is not conducive to quick repairs. |
3936
| |
3937
| This lead to the introduction of temporary file staging. |
3938
+--------------------------------------------------------------------------+
3940
Using a Temporary File
3941
``````````````````````
3943
Online repair code should use the ``xrep_tempfile_create`` function to create a
3944
temporary file inside the filesystem.
3945
This allocates an inode, marks the in-core inode private, and attaches it to
3946
the scrub context.
3947
These files are hidden from userspace, may not be added to the directory tree,
3948
and must be kept private.
3950
Temporary files only use two inode locks: the IOLOCK and the ILOCK.
3951
The MMAPLOCK is not needed here, because there must not be page faults from
3952
userspace for data fork blocks.
3953
The usage patterns of these two locks are the same as for any other XFS file --
3954
access to file data are controlled via the IOLOCK, and access to file metadata
3955
are controlled via the ILOCK.
3956
Locking helpers are provided so that the temporary file and its lock state can
3957
be cleaned up by the scrub context.
3958
To comply with the nested locking strategy laid out in the :ref:`inode
3959
locking<ilocking>` section, it is recommended that scrub functions use the
3960
xrep_tempfile_ilock*_nowait lock helpers.
3962
Data can be written to a temporary file by two means:
3964
1. ``xrep_tempfile_copyin`` can be used to set the contents of a regular
3965
temporary file from an xfile.
3967
2. The regular directory, symbolic link, and extended attribute functions can
3968
be used to write to the temporary file.
3970
Once a good copy of a data file has been constructed in a temporary file, it
3971
must be conveyed to the file being repaired, which is the topic of the next
3972
section.
3974
The proposed patches are in the
3975
`repair temporary files
3976
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-tempfiles>`_
3977
series.
3979
Logged File Content Exchanges
3980
-----------------------------
3982
Once repair builds a temporary file with a new data structure written into
3983
it, it must commit the new changes into the existing file.
3984
It is not possible to swap the inumbers of two files, so instead the new
3985
metadata must replace the old.
3986
This suggests the need for the ability to swap extents, but the existing extent
3987
swapping code used by the file defragmenting tool ``xfs_fsr`` is not sufficient
3988
for online repair because:
3990
a. When the reverse-mapping btree is enabled, the swap code must keep the
3991
reverse mapping information up to date with every exchange of mappings.
3992
Therefore, it can only exchange one mapping per transaction, and each
3993
transaction is independent.
3995
b. Reverse-mapping is critical for the operation of online fsck, so the old
3996
defragmentation code (which swapped entire extent forks in a single
3997
operation) is not useful here.
3999
c. Defragmentation is assumed to occur between two files with identical
4000
contents.
4001
For this use case, an incomplete exchange will not result in a user-visible
4002
change in file contents, even if the operation is interrupted.
4004
d. Online repair needs to swap the contents of two files that are by definition
4005
*not* identical.
4006
For directory and xattr repairs, the user-visible contents might be the
4007
same, but the contents of individual blocks may be very different.
4009
e. Old blocks in the file may be cross-linked with another structure and must
4010
not reappear if the system goes down mid-repair.
4012
These problems are overcome by creating a new deferred operation and a new type
4013
of log intent item to track the progress of an operation to exchange two file
4014
ranges.
4015
The new exchange operation type chains together the same transactions used by
4016
the reverse-mapping extent swap code, but records intermedia progress in the
4017
log so that operations can be restarted after a crash.
4018
This new functionality is called the file contents exchange (xfs_exchrange)
4019
code.
4020
The underlying implementation exchanges file fork mappings (xfs_exchmaps).
4021
The new log item records the progress of the exchange to ensure that once an
4022
exchange begins, it will always run to completion, even there are
4023
interruptions.
4024
The new ``XFS_SB_FEAT_INCOMPAT_EXCHRANGE`` incompatible feature flag
4025
in the superblock protects these new log item records from being replayed on
4026
old kernels.
4028
The proposed patchset is the
4029
`file contents exchange
4030
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates>`_
4031
series.
4033
+--------------------------------------------------------------------------+
4034
| **Sidebar: Using Log-Incompatible Feature Flags** |
4035
+--------------------------------------------------------------------------+
4036
| Starting with XFS v5, the superblock contains a |
4037
| ``sb_features_log_incompat`` field to indicate that the log contains |
4038
| records that might not readable by all kernels that could mount this |
4039
| filesystem. |
4040
| In short, log incompat features protect the log contents against kernels |
4041
| that will not understand the contents. |
4042
| Unlike the other superblock feature bits, log incompat bits are |
4043
| ephemeral because an empty (clean) log does not need protection. |
4044
| The log cleans itself after its contents have been committed into the |
4045
| filesystem, either as part of an unmount or because the system is |
4046
| otherwise idle. |
4047
| Because upper level code can be working on a transaction at the same |
4048
| time that the log cleans itself, it is necessary for upper level code to |
4049
| communicate to the log when it is going to use a log incompatible |
4050
| feature. |
4051
| |
4052
| The log coordinates access to incompatible features through the use of |
4053
| one ``struct rw_semaphore`` for each feature. |
4054
| The log cleaning code tries to take this rwsem in exclusive mode to |
4055
| clear the bit; if the lock attempt fails, the feature bit remains set. |
4056
| The code supporting a log incompat feature should create wrapper |
4057
| functions to obtain the log feature and call |
4058
| ``xfs_add_incompat_log_feature`` to set the feature bits in the primary |
4059
| superblock. |
4060
| The superblock update is performed transactionally, so the wrapper to |
4061
| obtain log assistance must be called just prior to the creation of the |
4062
| transaction that uses the functionality. |
4063
| For a file operation, this step must happen after taking the IOLOCK |
4064
| and the MMAPLOCK, but before allocating the transaction. |
4065
| When the transaction is complete, the ``xlog_drop_incompat_feat`` |
4066
| function is called to release the feature. |
4067
| The feature bit will not be cleared from the superblock until the log |
4068
| becomes clean. |
4069
| |
4070
| Log-assisted extended attribute updates and file content exchanges bothe |
4071
| use log incompat features and provide convenience wrappers around the |
4072
| functionality. |
4073
+--------------------------------------------------------------------------+
4075
Mechanics of a Logged File Content Exchange
4076
```````````````````````````````````````````
4078
Exchanging contents between file forks is a complex task.
4079
The goal is to exchange all file fork mappings between two file fork offset
4080
ranges.
4081
There are likely to be many extent mappings in each fork, and the edges of
4082
the mappings aren't necessarily aligned.
4083
Furthermore, there may be other updates that need to happen after the exchange,
4084
such as exchanging file sizes, inode flags, or conversion of fork data to local
4085
format.
4086
This is roughly the format of the new deferred exchange-mapping work item:
4088
.. code-block:: c
4090
struct xfs_exchmaps_intent {
4091
/* Inodes participating in the operation. */
4092
struct xfs_inode *xmi_ip1;
4093
struct xfs_inode *xmi_ip2;
4095
/* File offset range information. */
4096
xfs_fileoff_t xmi_startoff1;
4097
xfs_fileoff_t xmi_startoff2;
4098
xfs_filblks_t xmi_blockcount;
4100
/* Set these file sizes after the operation, unless negative. */
4101
xfs_fsize_t xmi_isize1;
4102
xfs_fsize_t xmi_isize2;
4104
/* XFS_EXCHMAPS_* log operation flags */
4105
uint64_t xmi_flags;
4106
};
4108
The new log intent item contains enough information to track two logical fork
4109
offset ranges: ``(inode1, startoff1, blockcount)`` and ``(inode2, startoff2,
4110
blockcount)``.
4111
Each step of an exchange operation exchanges the largest file range mapping
4112
possible from one file to the other.
4113
After each step in the exchange operation, the two startoff fields are
4114
incremented and the blockcount field is decremented to reflect the progress
4115
made.
4116
The flags field captures behavioral parameters such as exchanging attr fork
4117
mappings instead of the data fork and other work to be done after the exchange.
4118
The two isize fields are used to exchange the file sizes at the end of the
4119
operation if the file data fork is the target of the operation.
4121
When the exchange is initiated, the sequence of operations is as follows:
4123
1. Create a deferred work item for the file mapping exchange.
4124
At the start, it should contain the entirety of the file block ranges to be
4125
exchanged.
4127
2. Call ``xfs_defer_finish`` to process the exchange.
4128
This is encapsulated in ``xrep_tempexch_contents`` for scrub operations.
4129
This will log an extent swap intent item to the transaction for the deferred
4130
mapping exchange work item.
4132
3. Until ``xmi_blockcount`` of the deferred mapping exchange work item is zero,
4134
a. Read the block maps of both file ranges starting at ``xmi_startoff1`` and
4135
``xmi_startoff2``, respectively, and compute the longest extent that can
4136
be exchanged in a single step.
4137
This is the minimum of the two ``br_blockcount`` s in the mappings.
4138
Keep advancing through the file forks until at least one of the mappings
4139
contains written blocks.
4140
Mutual holes, unwritten extents, and extent mappings to the same physical
4141
space are not exchanged.
4143
For the next few steps, this document will refer to the mapping that came
4144
from file 1 as "map1", and the mapping that came from file 2 as "map2".
4146
b. Create a deferred block mapping update to unmap map1 from file 1.
4148
c. Create a deferred block mapping update to unmap map2 from file 2.
4150
d. Create a deferred block mapping update to map map1 into file 2.
4152
e. Create a deferred block mapping update to map map2 into file 1.
4154
f. Log the block, quota, and extent count updates for both files.
4156
g. Extend the ondisk size of either file if necessary.
4158
h. Log a mapping exchange done log item for th mapping exchange intent log
4159
item that was read at the start of step 3.
4161
i. Compute the amount of file range that has just been covered.
4162
This quantity is ``(map1.br_startoff + map1.br_blockcount -
4163
xmi_startoff1)``, because step 3a could have skipped holes.
4165
j. Increase the starting offsets of ``xmi_startoff1`` and ``xmi_startoff2``
4166
by the number of blocks computed in the previous step, and decrease
4167
``xmi_blockcount`` by the same quantity.
4168
This advances the cursor.
4170
k. Log a new mapping exchange intent log item reflecting the advanced state
4171
of the work item.
4173
l. Return the proper error code (EAGAIN) to the deferred operation manager
4174
to inform it that there is more work to be done.
4175
The operation manager completes the deferred work in steps 3b-3e before
4176
moving back to the start of step 3.
4178
4. Perform any post-processing.
4179
This will be discussed in more detail in subsequent sections.
4181
If the filesystem goes down in the middle of an operation, log recovery will
4182
find the most recent unfinished mapping exchange log intent item and restart
4183
from there.
4184
This is how atomic file mapping exchanges guarantees that an outside observer
4185
will either see the old broken structure or the new one, and never a mismash of
4186
both.
4188
Preparation for File Content Exchanges
4189
``````````````````````````````````````
4191
There are a few things that need to be taken care of before initiating an
4192
atomic file mapping exchange operation.
4193
First, regular files require the page cache to be flushed to disk before the
4194
operation begins, and directio writes to be quiesced.
4195
Like any filesystem operation, file mapping exchanges must determine the
4196
maximum amount of disk space and quota that can be consumed on behalf of both
4197
files in the operation, and reserve that quantity of resources to avoid an
4198
unrecoverable out of space failure once it starts dirtying metadata.
4199
The preparation step scans the ranges of both files to estimate:
4201
- Data device blocks needed to handle the repeated updates to the fork
4202
mappings.
4203
- Change in data and realtime block counts for both files.
4204
- Increase in quota usage for both files, if the two files do not share the
4205
same set of quota ids.
4206
- The number of extent mappings that will be added to each file.
4207
- Whether or not there are partially written realtime extents.
4208
User programs must never be able to access a realtime file extent that maps
4209
to different extents on the realtime volume, which could happen if the
4210
operation fails to run to completion.
4212
The need for precise estimation increases the run time of the exchange
4213
operation, but it is very important to maintain correct accounting.
4214
The filesystem must not run completely out of free space, nor can the mapping
4215
exchange ever add more extent mappings to a fork than it can support.
4216
Regular users are required to abide the quota limits, though metadata repairs
4217
may exceed quota to resolve inconsistent metadata elsewhere.
4219
Special Features for Exchanging Metadata File Contents
4220
``````````````````````````````````````````````````````
4222
Extended attributes, symbolic links, and directories can set the fork format to
4223
"local" and treat the fork as a literal area for data storage.
4224
Metadata repairs must take extra steps to support these cases:
4226
- If both forks are in local format and the fork areas are large enough, the
4227
exchange is performed by copying the incore fork contents, logging both
4228
forks, and committing.
4229
The atomic file mapping exchange mechanism is not necessary, since this can
4230
be done with a single transaction.
4232
- If both forks map blocks, then the regular atomic file mapping exchange is
4233
used.
4235
- Otherwise, only one fork is in local format.
4236
The contents of the local format fork are converted to a block to perform the
4237
exchange.
4238
The conversion to block format must be done in the same transaction that
4239
logs the initial mapping exchange intent log item.
4240
The regular atomic mapping exchange is used to exchange the metadata file
4241
mappings.
4242
Special flags are set on the exchange operation so that the transaction can
4243
be rolled one more time to convert the second file's fork back to local
4244
format so that the second file will be ready to go as soon as the ILOCK is
4245
dropped.
4247
Extended attributes and directories stamp the owning inode into every block,
4248
but the buffer verifiers do not actually check the inode number!
4249
Although there is no verification, it is still important to maintain
4250
referential integrity, so prior to performing the mapping exchange, online
4251
repair builds every block in the new data structure with the owner field of the
4252
file being repaired.
4254
After a successful exchange operation, the repair operation must reap the old
4255
fork blocks by processing each fork mapping through the standard :ref:`file
4256
extent reaping <reaping>` mechanism that is done post-repair.
4257
If the filesystem should go down during the reap part of the repair, the
4258
iunlink processing at the end of recovery will free both the temporary file and
4259
whatever blocks were not reaped.
4260
However, this iunlink processing omits the cross-link detection of online
4261
repair, and is not completely foolproof.
4263
Exchanging Temporary File Contents
4264
``````````````````````````````````
4266
To repair a metadata file, online repair proceeds as follows:
4268
1. Create a temporary repair file.
4270
2. Use the staging data to write out new contents into the temporary repair
4271
file.
4272
The same fork must be written to as is being repaired.
4274
3. Commit the scrub transaction, since the exchange resource estimation step
4275
must be completed before transaction reservations are made.
4277
4. Call ``xrep_tempexch_trans_alloc`` to allocate a new scrub transaction with
4278
the appropriate resource reservations, locks, and fill out a ``struct
4279
xfs_exchmaps_req`` with the details of the exchange operation.
4281
5. Call ``xrep_tempexch_contents`` to exchange the contents.
4283
6. Commit the transaction to complete the repair.
4285
.. _rtsummary:
4287
Case Study: Repairing the Realtime Summary File
4288
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4290
In the "realtime" section of an XFS filesystem, free space is tracked via a
4291
bitmap, similar to Unix FFS.
4292
Each bit in the bitmap represents one realtime extent, which is a multiple of
4293
the filesystem block size between 4KiB and 1GiB in size.
4294
The realtime summary file indexes the number of free extents of a given size to
4295
the offset of the block within the realtime free space bitmap where those free
4296
extents begin.
4297
In other words, the summary file helps the allocator find free extents by
4298
length, similar to what the free space by count (cntbt) btree does for the data
4299
section.
4301
The summary file itself is a flat file (with no block headers or checksums!)
4302
partitioned into ``log2(total rt extents)`` sections containing enough 32-bit
4303
counters to match the number of blocks in the rt bitmap.
4304
Each counter records the number of free extents that start in that bitmap block
4305
and can satisfy a power-of-two allocation request.
4307
To check the summary file against the bitmap:
4309
1. Take the ILOCK of both the realtime bitmap and summary files.
4311
2. For each free space extent recorded in the bitmap:
4313
a. Compute the position in the summary file that contains a counter that
4314
represents this free extent.
4316
b. Read the counter from the xfile.
4318
c. Increment it, and write it back to the xfile.
4320
3. Compare the contents of the xfile against the ondisk file.
4322
To repair the summary file, write the xfile contents into the temporary file
4323
and use atomic mapping exchange to commit the new contents.
4324
The temporary file is then reaped.
4326
The proposed patchset is the
4327
`realtime summary repair
4328
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary>`_
4329
series.
4331
Case Study: Salvaging Extended Attributes
4332
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4334
In XFS, extended attributes are implemented as a namespaced name-value store.
4335
Values are limited in size to 64KiB, but there is no limit in the number of
4336
names.
4337
The attribute fork is unpartitioned, which means that the root of the attribute
4338
structure is always in logical block zero, but attribute leaf blocks, dabtree
4339
index blocks, and remote value blocks are intermixed.
4340
Attribute leaf blocks contain variable-sized records that associate
4341
user-provided names with the user-provided values.
4342
Values larger than a block are allocated separate extents and written there.
4343
If the leaf information expands beyond a single block, a directory/attribute
4344
btree (``dabtree``) is created to map hashes of attribute names to entries
4345
for fast lookup.
4347
Salvaging extended attributes is done as follows:
4349
1. Walk the attr fork mappings of the file being repaired to find the attribute
4350
leaf blocks.
4351
When one is found,
4353
a. Walk the attr leaf block to find candidate keys.
4354
When one is found,
4356
1. Check the name for problems, and ignore the name if there are.
4358
2. Retrieve the value.
4359
If that succeeds, add the name and value to the staging xfarray and
4360
xfblob.
4362
2. If the memory usage of the xfarray and xfblob exceed a certain amount of
4363
memory or there are no more attr fork blocks to examine, unlock the file and
4364
add the staged extended attributes to the temporary file.
4366
3. Use atomic file mapping exchange to exchange the new and old extended
4367
attribute structures.
4368
The old attribute blocks are now attached to the temporary file.
4370
4. Reap the temporary file.
4372
The proposed patchset is the
4373
`extended attribute repair
4374
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_
4375
series.
4377
Fixing Directories
4378
------------------
4380
Fixing directories is difficult with currently available filesystem features,
4381
since directory entries are not redundant.
4382
The offline repair tool scans all inodes to find files with nonzero link count,
4383
and then it scans all directories to establish parentage of those linked files.
4384
Damaged files and directories are zapped, and files with no parent are
4385
moved to the ``/lost+found`` directory.
4386
It does not try to salvage anything.
4388
The best that online repair can do at this time is to read directory data
4389
blocks and salvage any dirents that look plausible, correct link counts, and
4390
move orphans back into the directory tree.
4391
The salvage process is discussed in the case study at the end of this section.
4392
The :ref:`file link count fsck <nlinks>` code takes care of fixing link counts
4393
and moving orphans to the ``/lost+found`` directory.
4395
Case Study: Salvaging Directories
4396
`````````````````````````````````
4398
Unlike extended attributes, directory blocks are all the same size, so
4399
salvaging directories is straightforward:
4401
1. Find the parent of the directory.
4402
If the dotdot entry is not unreadable, try to confirm that the alleged
4403
parent has a child entry pointing back to the directory being repaired.
4404
Otherwise, walk the filesystem to find it.
4406
2. Walk the first partition of data fork of the directory to find the directory
4407
entry data blocks.
4408
When one is found,
4410
a. Walk the directory data block to find candidate entries.
4411
When an entry is found:
4413
i. Check the name for problems, and ignore the name if there are.
4415
ii. Retrieve the inumber and grab the inode.
4416
If that succeeds, add the name, inode number, and file type to the
4417
staging xfarray and xblob.
4419
3. If the memory usage of the xfarray and xfblob exceed a certain amount of
4420
memory or there are no more directory data blocks to examine, unlock the
4421
directory and add the staged dirents into the temporary directory.
4422
Truncate the staging files.
4424
4. Use atomic file mapping exchange to exchange the new and old directory
4425
structures.
4426
The old directory blocks are now attached to the temporary file.
4428
5. Reap the temporary file.
4430
**Future Work Question**: Should repair revalidate the dentry cache when
4431
rebuilding a directory?
4433
*Answer*: Yes, it should.
4435
In theory it is necessary to scan all dentry cache entries for a directory to
4436
ensure that one of the following apply:
4438
1. The cached dentry reflects an ondisk dirent in the new directory.
4440
2. The cached dentry no longer has a corresponding ondisk dirent in the new
4441
directory and the dentry can be purged from the cache.
4443
3. The cached dentry no longer has an ondisk dirent but the dentry cannot be
4444
purged.
4445
This is the problem case.
4447
Unfortunately, the current dentry cache design doesn't provide a means to walk
4448
every child dentry of a specific directory, which makes this a hard problem.
4449
There is no known solution.
4451
The proposed patchset is the
4452
`directory repair
4453
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_
4454
series.
4456
Parent Pointers
4457
```````````````
4459
A parent pointer is a piece of file metadata that enables a user to locate the
4460
file's parent directory without having to traverse the directory tree from the
4461
root.
4462
Without them, reconstruction of directory trees is hindered in much the same
4463
way that the historic lack of reverse space mapping information once hindered
4464
reconstruction of filesystem space metadata.
4465
The parent pointer feature, however, makes total directory reconstruction
4466
possible.
4468
XFS parent pointers contain the information needed to identify the
4469
corresponding directory entry in the parent directory.
4470
In other words, child files use extended attributes to store pointers to
4471
parents in the form ``(dirent_name) → (parent_inum, parent_gen)``.
4472
The directory checking process can be strengthened to ensure that the target of
4473
each dirent also contains a parent pointer pointing back to the dirent.
4474
Likewise, each parent pointer can be checked by ensuring that the target of
4475
each parent pointer is a directory and that it contains a dirent matching
4476
the parent pointer.
4477
Both online and offline repair can use this strategy.
4479
+--------------------------------------------------------------------------+
4480
| **Historical Sidebar**: |
4481
+--------------------------------------------------------------------------+
4482
| Directory parent pointers were first proposed as an XFS feature more |
4483
| than a decade ago by SGI. |
4484
| Each link from a parent directory to a child file is mirrored with an |
4485
| extended attribute in the child that could be used to identify the |
4486
| parent directory. |
4487
| Unfortunately, this early implementation had major shortcomings and was |
4488
| never merged into Linux XFS: |
4489
| |
4490
| 1. The XFS codebase of the late 2000s did not have the infrastructure to |
4491
| enforce strong referential integrity in the directory tree. |
4492
| It did not guarantee that a change in a forward link would always be |
4493
| followed up with the corresponding change to the reverse links. |
4494
| |
4495
| 2. Referential integrity was not integrated into offline repair. |
4496
| Checking and repairs were performed on mounted filesystems without |
4497
| taking any kernel or inode locks to coordinate access. |
4498
| It is not clear how this actually worked properly. |
4499
| |
4500
| 3. The extended attribute did not record the name of the directory entry |
4501
| in the parent, so the SGI parent pointer implementation cannot be |
4502
| used to reconnect the directory tree. |
4503
| |
4504
| 4. Extended attribute forks only support 65,536 extents, which means |
4505
| that parent pointer attribute creation is likely to fail at some |
4506
| point before the maximum file link count is achieved. |
4507
| |
4508
| The original parent pointer design was too unstable for something like |
4509
| a file system repair to depend on. |
4510
| Allison Henderson, Chandan Babu, and Catherine Hoang are working on a |
4511
| second implementation that solves all shortcomings of the first. |
4512
| During 2022, Allison introduced log intent items to track physical |
4513
| manipulations of the extended attribute structures. |
4514
| This solves the referential integrity problem by making it possible to |
4515
| commit a dirent update and a parent pointer update in the same |
4516
| transaction. |
4517
| Chandan increased the maximum extent counts of both data and attribute |
4518
| forks, thereby ensuring that the extended attribute structure can grow |
4519
| to handle the maximum hardlink count of any file. |
4520
| |
4521
| For this second effort, the ondisk parent pointer format as originally |
4522
| proposed was ``(parent_inum, parent_gen, dirent_pos) → (dirent_name)``. |
4523
| The format was changed during development to eliminate the requirement |
4524
| of repair tools needing to ensure that the ``dirent_pos`` field always |
4525
| matched when reconstructing a directory. |
4526
| |
4527
| There were a few other ways to have solved that problem: |
4528
| |
4529
| 1. The field could be designated advisory, since the other three values |
4530
| are sufficient to find the entry in the parent. |
4531
| However, this makes indexed key lookup impossible while repairs are |
4532
| ongoing. |
4533
| |
4534
| 2. We could allow creating directory entries at specified offsets, which |
4535
| solves the referential integrity problem but runs the risk that |
4536
| dirent creation will fail due to conflicts with the free space in the |
4537
| directory. |
4538
| |
4539
| These conflicts could be resolved by appending the directory entry |
4540
| and amending the xattr code to support updating an xattr key and |
4541
| reindexing the dabtree, though this would have to be performed with |
4542
| the parent directory still locked. |
4543
| |
4544
| 3. Same as above, but remove the old parent pointer entry and add a new |
4545
| one atomically. |
4546
| |
4547
| 4. Change the ondisk xattr format to |
4548
| ``(parent_inum, name) → (parent_gen)``, which would provide the attr |
4549
| name uniqueness that we require, without forcing repair code to |
4550
| update the dirent position. |
4551
| Unfortunately, this requires changes to the xattr code to support |
4552
| attr names as long as 263 bytes. |
4553
| |
4554
| 5. Change the ondisk xattr format to ``(parent_inum, hash(name)) → |
4555
| (name, parent_gen)``. |
4556
| If the hash is sufficiently resistant to collisions (e.g. sha256) |
4557
| then this should provide the attr name uniqueness that we require. |
4558
| Names shorter than 247 bytes could be stored directly. |
4559
| |
4560
| 6. Change the ondisk xattr format to ``(dirent_name) → (parent_ino, |
4561
| parent_gen)``. This format doesn't require any of the complicated |
4562
| nested name hashing of the previous suggestions. However, it was |
4563
| discovered that multiple hardlinks to the same inode with the same |
4564
| filename caused performance problems with hashed xattr lookups, so |
4565
| the parent inumber is now xor'd into the hash index. |
4566
| |
4567
| In the end, it was decided that solution #6 was the most compact and the |
4568
| most performant. A new hash function was designed for parent pointers. |
4569
+--------------------------------------------------------------------------+
4572
Case Study: Repairing Directories with Parent Pointers
4573
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4575
Directory rebuilding uses a :ref:`coordinated inode scan <iscan>` and
4576
a :ref:`directory entry live update hook <liveupdate>` as follows:
4578
1. Set up a temporary directory for generating the new directory structure,
4579
an xfblob for storing entry names, and an xfarray for stashing the fixed
4580
size fields involved in a directory update: ``(child inumber, add vs.
4581
remove, name cookie, ftype)``.
4583
2. Set up an inode scanner and hook into the directory entry code to receive
4584
updates on directory operations.
4586
3. For each parent pointer found in each file scanned, decide if the parent
4587
pointer references the directory of interest.
4588
If so:
4590
a. Stash the parent pointer name and an addname entry for this dirent in the
4591
xfblob and xfarray, respectively.
4593
b. When finished scanning that file or the kernel memory consumption exceeds
4594
a threshold, flush the stashed updates to the temporary directory.
4596
4. For each live directory update received via the hook, decide if the child
4597
has already been scanned.
4598
If so:
4600
a. Stash the parent pointer name an addname or removename entry for this
4601
dirent update in the xfblob and xfarray for later.
4602
We cannot write directly to the temporary directory because hook
4603
functions are not allowed to modify filesystem metadata.
4604
Instead, we stash updates in the xfarray and rely on the scanner thread
4605
to apply the stashed updates to the temporary directory.
4607
5. When the scan is complete, replay any stashed entries in the xfarray.
4609
6. When the scan is complete, atomically exchange the contents of the temporary
4610
directory and the directory being repaired.
4611
The temporary directory now contains the damaged directory structure.
4613
7. Reap the temporary directory.
4615
The proposed patchset is the
4616
`parent pointers directory repair
4617
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_
4618
series.
4620
Case Study: Repairing Parent Pointers
4621
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4623
Online reconstruction of a file's parent pointer information works similarly to
4624
directory reconstruction:
4626
1. Set up a temporary file for generating a new extended attribute structure,
4627
an xfblob for storing parent pointer names, and an xfarray for stashing the
4628
fixed size fields involved in a parent pointer update: ``(parent inumber,
4629
parent generation, add vs. remove, name cookie)``.
4631
2. Set up an inode scanner and hook into the directory entry code to receive
4632
updates on directory operations.
4634
3. For each directory entry found in each directory scanned, decide if the
4635
dirent references the file of interest.
4636
If so:
4638
a. Stash the dirent name and an addpptr entry for this parent pointer in the
4639
xfblob and xfarray, respectively.
4641
b. When finished scanning the directory or the kernel memory consumption
4642
exceeds a threshold, flush the stashed updates to the temporary file.
4644
4. For each live directory update received via the hook, decide if the parent
4645
has already been scanned.
4646
If so:
4648
a. Stash the dirent name and an addpptr or removepptr entry for this dirent
4649
update in the xfblob and xfarray for later.
4650
We cannot write parent pointers directly to the temporary file because
4651
hook functions are not allowed to modify filesystem metadata.
4652
Instead, we stash updates in the xfarray and rely on the scanner thread
4653
to apply the stashed parent pointer updates to the temporary file.
4655
5. When the scan is complete, replay any stashed entries in the xfarray.
4657
6. Copy all non-parent pointer extended attributes to the temporary file.
4659
7. When the scan is complete, atomically exchange the mappings of the attribute
4660
forks of the temporary file and the file being repaired.
4661
The temporary file now contains the damaged extended attribute structure.
4663
8. Reap the temporary file.
4665
The proposed patchset is the
4666
`parent pointers repair
4667
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_
4668
series.
4670
Digression: Offline Checking of Parent Pointers
4671
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4673
Examining parent pointers in offline repair works differently because corrupt
4674
files are erased long before directory tree connectivity checks are performed.
4675
Parent pointer checks are therefore a second pass to be added to the existing
4676
connectivity checks:
4678
1. After the set of surviving files has been established (phase 6),
4679
walk the surviving directories of each AG in the filesystem.
4680
This is already performed as part of the connectivity checks.
4682
2. For each directory entry found,
4684
a. If the name has already been stored in the xfblob, then use that cookie
4685
and skip the next step.
4687
b. Otherwise, record the name in an xfblob, and remember the xfblob cookie.
4688
Unique mappings are critical for
4690
1. Deduplicating names to reduce memory usage, and
4692
2. Creating a stable sort key for the parent pointer indexes so that the
4693
parent pointer validation described below will work.
4695
c. Store ``(child_ag_inum, parent_inum, parent_gen, name_hash, name_len,
4696
name_cookie)`` tuples in a per-AG in-memory slab. The ``name_hash``
4697
referenced in this section is the regular directory entry name hash, not
4698
the specialized one used for parent pointer xattrs.
4700
3. For each AG in the filesystem,
4702
a. Sort the per-AG tuple set in order of ``child_ag_inum``, ``parent_inum``,
4703
``name_hash``, and ``name_cookie``.
4704
Having a single ``name_cookie`` for each ``name`` is critical for
4705
handling the uncommon case of a directory containing multiple hardlinks
4706
to the same file where all the names hash to the same value.
4708
b. For each inode in the AG,
4710
1. Scan the inode for parent pointers.
4711
For each parent pointer found,
4713
a. Validate the ondisk parent pointer.
4714
If validation fails, move on to the next parent pointer in the
4715
file.
4717
b. If the name has already been stored in the xfblob, then use that
4718
cookie and skip the next step.
4720
c. Record the name in a per-file xfblob, and remember the xfblob
4721
cookie.
4723
d. Store ``(parent_inum, parent_gen, name_hash, name_len,
4724
name_cookie)`` tuples in a per-file slab.
4726
2. Sort the per-file tuples in order of ``parent_inum``, ``name_hash``,
4727
and ``name_cookie``.
4729
3. Position one slab cursor at the start of the inode's records in the
4730
per-AG tuple slab.
4731
This should be trivial since the per-AG tuples are in child inumber
4732
order.
4734
4. Position a second slab cursor at the start of the per-file tuple slab.
4736
5. Iterate the two cursors in lockstep, comparing the ``parent_ino``,
4737
``name_hash``, and ``name_cookie`` fields of the records under each
4738
cursor:
4740
a. If the per-AG cursor is at a lower point in the keyspace than the
4741
per-file cursor, then the per-AG cursor points to a missing parent
4742
pointer.
4743
Add the parent pointer to the inode and advance the per-AG
4744
cursor.
4746
b. If the per-file cursor is at a lower point in the keyspace than
4747
the per-AG cursor, then the per-file cursor points to a dangling
4748
parent pointer.
4749
Remove the parent pointer from the inode and advance the per-file
4750
cursor.
4752
c. Otherwise, both cursors point at the same parent pointer.
4753
Update the parent_gen component if necessary.
4754
Advance both cursors.
4756
4. Move on to examining link counts, as we do today.
4758
The proposed patchset is the
4759
`offline parent pointers repair
4760
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fsck>`_
4761
series.
4763
Rebuilding directories from parent pointers in offline repair would be very
4764
challenging because xfs_repair currently uses two single-pass scans of the
4765
filesystem during phases 3 and 4 to decide which files are corrupt enough to be
4766
zapped.
4767
This scan would have to be converted into a multi-pass scan:
4769
1. The first pass of the scan zaps corrupt inodes, forks, and attributes
4770
much as it does now.
4771
Corrupt directories are noted but not zapped.
4773
2. The next pass records parent pointers pointing to the directories noted
4774
as being corrupt in the first pass.
4775
This second pass may have to happen after the phase 4 scan for duplicate
4776
blocks, if phase 4 is also capable of zapping directories.
4778
3. The third pass resets corrupt directories to an empty shortform directory.
4779
Free space metadata has not been ensured yet, so repair cannot yet use the
4780
directory building code in libxfs.
4782
4. At the start of phase 6, space metadata have been rebuilt.
4783
Use the parent pointer information recorded during step 2 to reconstruct
4784
the dirents and add them to the now-empty directories.
4786
This code has not yet been constructed.
4788
.. _dirtree:
4790
Case Study: Directory Tree Structure
4791
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4793
As mentioned earlier, the filesystem directory tree is supposed to be a
4794
directed acylic graph structure.
4795
However, each node in this graph is a separate ``xfs_inode`` object with its
4796
own locks, which makes validating the tree qualities difficult.
4797
Fortunately, non-directories are allowed to have multiple parents and cannot
4798
have children, so only directories need to be scanned.
4799
Directories typically constitute 5-10% of the files in a filesystem, which
4800
reduces the amount of work dramatically.
4802
If the directory tree could be frozen, it would be easy to discover cycles and
4803
disconnected regions by running a depth (or breadth) first search downwards
4804
from the root directory and marking a bitmap for each directory found.
4805
At any point in the walk, trying to set an already set bit means there is a
4806
cycle.
4807
After the scan completes, XORing the marked inode bitmap with the inode
4808
allocation bitmap reveals disconnected inodes.
4809
However, one of online repair's design goals is to avoid locking the entire
4810
filesystem unless it's absolutely necessary.
4811
Directory tree updates can move subtrees across the scanner wavefront on a live
4812
filesystem, so the bitmap algorithm cannot be applied.
4814
Directory parent pointers enable an incremental approach to validation of the
4815
tree structure.
4816
Instead of using one thread to scan the entire filesystem, multiple threads can
4817
walk from individual subdirectories upwards towards the root.
4818
For this to work, all directory entries and parent pointers must be internally
4819
consistent, each directory entry must have a parent pointer, and the link
4820
counts of all directories must be correct.
4821
Each scanner thread must be able to take the IOLOCK of an alleged parent
4822
directory while holding the IOLOCK of the child directory to prevent either
4823
directory from being moved within the tree.
4824
This is not possible since the VFS does not take the IOLOCK of a child
4825
subdirectory when moving that subdirectory, so instead the scanner stabilizes
4826
the parent -> child relationship by taking the ILOCKs and installing a dirent
4827
update hook to detect changes.
4829
The scanning process uses a dirent hook to detect changes to the directories
4830
mentioned in the scan data.
4831
The scan works as follows:
4833
1. For each subdirectory in the filesystem,
4835
a. For each parent pointer of that subdirectory,
4837
1. Create a path object for that parent pointer, and mark the
4838
subdirectory inode number in the path object's bitmap.
4840
2. Record the parent pointer name and inode number in a path structure.
4842
3. If the alleged parent is the subdirectory being scrubbed, the path is
4843
a cycle.
4844
Mark the path for deletion and repeat step 1a with the next
4845
subdirectory parent pointer.
4847
4. Try to mark the alleged parent inode number in a bitmap in the path
4848
object.
4849
If the bit is already set, then there is a cycle in the directory
4850
tree.
4851
Mark the path as a cycle and repeat step 1a with the next subdirectory
4852
parent pointer.
4854
5. Load the alleged parent.
4855
If the alleged parent is not a linked directory, abort the scan
4856
because the parent pointer information is inconsistent.
4858
6. For each parent pointer of this alleged ancestor directory,
4860
a. Record the parent pointer name and inode number in the path object
4861
if no parent has been set for that level.
4863
b. If an ancestor has more than one parent, mark the path as corrupt.
4864
Repeat step 1a with the next subdirectory parent pointer.
4866
c. Repeat steps 1a3-1a6 for the ancestor identified in step 1a6a.
4867
This repeats until the directory tree root is reached or no parents
4868
are found.
4870
7. If the walk terminates at the root directory, mark the path as ok.
4872
8. If the walk terminates without reaching the root, mark the path as
4873
disconnected.
4875
2. If the directory entry update hook triggers, check all paths already found
4876
by the scan.
4877
If the entry matches part of a path, mark that path and the scan stale.
4878
When the scanner thread sees that the scan has been marked stale, it deletes
4879
all scan data and starts over.
4881
Repairing the directory tree works as follows:
4883
1. Walk each path of the target subdirectory.
4885
a. Corrupt paths and cycle paths are counted as suspect.
4887
b. Paths already marked for deletion are counted as bad.
4889
c. Paths that reached the root are counted as good.
4891
2. If the subdirectory is either the root directory or has zero link count,
4892
delete all incoming directory entries in the immediate parents.
4893
Repairs are complete.
4895
3. If the subdirectory has exactly one path, set the dotdot entry to the
4896
parent and exit.
4898
4. If the subdirectory has at least one good path, delete all the other
4899
incoming directory entries in the immediate parents.
4901
5. If the subdirectory has no good paths and more than one suspect path, delete
4902
all the other incoming directory entries in the immediate parents.
4904
6. If the subdirectory has zero paths, attach it to the lost and found.
4906
The proposed patches are in the
4907
`directory tree repair
4908
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-directory-tree>`_
4909
series.
4912
.. _orphanage:
4914
The Orphanage
4915
-------------
4917
Filesystems present files as a directed, and hopefully acyclic, graph.
4918
In other words, a tree.
4919
The root of the filesystem is a directory, and each entry in a directory points
4920
downwards either to more subdirectories or to non-directory files.
4921
Unfortunately, a disruption in the directory graph pointers result in a
4922
disconnected graph, which makes files impossible to access via regular path
4923
resolution.
4925
Without parent pointers, the directory parent pointer online scrub code can
4926
detect a dotdot entry pointing to a parent directory that doesn't have a link
4927
back to the child directory and the file link count checker can detect a file
4928
that isn't pointed to by any directory in the filesystem.
4929
If such a file has a positive link count, the file is an orphan.
4931
With parent pointers, directories can be rebuilt by scanning parent pointers
4932
and parent pointers can be rebuilt by scanning directories.
4933
This should reduce the incidence of files ending up in ``/lost+found``.
4935
When orphans are found, they should be reconnected to the directory tree.
4936
Offline fsck solves the problem by creating a directory ``/lost+found`` to
4937
serve as an orphanage, and linking orphan files into the orphanage by using the
4938
inumber as the name.
4939
Reparenting a file to the orphanage does not reset any of its permissions or
4940
ACLs.
4942
This process is more involved in the kernel than it is in userspace.
4943
The directory and file link count repair setup functions must use the regular
4944
VFS mechanisms to create the orphanage directory with all the necessary
4945
security attributes and dentry cache entries, just like a regular directory
4946
tree modification.
4948
Orphaned files are adopted by the orphanage as follows:
4950
1. Call ``xrep_orphanage_try_create`` at the start of the scrub setup function
4951
to try to ensure that the lost and found directory actually exists.
4952
This also attaches the orphanage directory to the scrub context.
4954
2. If the decision is made to reconnect a file, take the IOLOCK of both the
4955
orphanage and the file being reattached.
4956
The ``xrep_orphanage_iolock_two`` function follows the inode locking
4957
strategy discussed earlier.
4959
3. Use ``xrep_adoption_trans_alloc`` to reserve resources to the repair
4960
transaction.
4962
4. Call ``xrep_orphanage_compute_name`` to compute the new name in the
4963
orphanage.
4965
5. If the adoption is going to happen, call ``xrep_adoption_reparent`` to
4966
reparent the orphaned file into the lost and found and invalidate the dentry
4967
cache.
4969
6. Call ``xrep_adoption_finish`` to commit any filesystem updates, release the
4970
orphanage ILOCK, and clean the scrub transaction. Call
4971
``xrep_adoption_commit`` to commit the updates and the scrub transaction.
4973
7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all
4974
resources.
4976
The proposed patches are in the
4977
`orphanage adoption
4978
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-orphanage>`_
4979
series.
4981
6. Userspace Algorithms and Data Structures
4982
===========================================
4984
This section discusses the key algorithms and data structures of the userspace
4985
program, ``xfs_scrub``, that provide the ability to drive metadata checks and
4986
repairs in the kernel, verify file data, and look for other potential problems.
4988
.. _scrubcheck:
4990
Checking Metadata
4991
-----------------
4993
Recall the :ref:`phases of fsck work<scrubphases>` outlined earlier.
4994
That structure follows naturally from the data dependencies designed into the
4995
filesystem from its beginnings in 1993.
4996
In XFS, there are several groups of metadata dependencies:
4998
a. Filesystem summary counts depend on consistency within the inode indices,
4999
the allocation group space btrees, and the realtime volume space
5000
information.
5002
b. Quota resource counts depend on consistency within the quota file data
5003
forks, inode indices, inode records, and the forks of every file on the
5004
system.
5006
c. The naming hierarchy depends on consistency within the directory and
5007
extended attribute structures.
5008
This includes file link counts.
5010
d. Directories, extended attributes, and file data depend on consistency within
5011
the file forks that map directory and extended attribute data to physical
5012
storage media.
5014
e. The file forks depends on consistency within inode records and the space
5015
metadata indices of the allocation groups and the realtime volume.
5016
This includes quota and realtime metadata files.
5018
f. Inode records depends on consistency within the inode metadata indices.
5020
g. Realtime space metadata depend on the inode records and data forks of the
5021
realtime metadata inodes.
5023
h. The allocation group metadata indices (free space, inodes, reference count,
5024
and reverse mapping btrees) depend on consistency within the AG headers and
5025
between all the AG metadata btrees.
5027
i. ``xfs_scrub`` depends on the filesystem being mounted and kernel support
5028
for online fsck functionality.
5030
Therefore, a metadata dependency graph is a convenient way to schedule checking
5031
operations in the ``xfs_scrub`` program:
5033
- Phase 1 checks that the provided path maps to an XFS filesystem and detect
5034
the kernel's scrubbing abilities, which validates group (i).
5036
- Phase 2 scrubs groups (g) and (h) in parallel using a threaded workqueue.
5038
- Phase 3 scans inodes in parallel.
5039
For each inode, groups (f), (e), and (d) are checked, in that order.
5041
- Phase 4 repairs everything in groups (i) through (d) so that phases 5 and 6
5042
may run reliably.
5044
- Phase 5 starts by checking groups (b) and (c) in parallel before moving on
5045
to checking names.
5047
- Phase 6 depends on groups (i) through (b) to find file data blocks to verify,
5048
to read them, and to report which blocks of which files are affected.
5050
- Phase 7 checks group (a), having validated everything else.
5052
Notice that the data dependencies between groups are enforced by the structure
5053
of the program flow.
5055
Parallel Inode Scans
5056
--------------------
5058
An XFS filesystem can easily contain hundreds of millions of inodes.
5059
Given that XFS targets installations with large high-performance storage,
5060
it is desirable to scrub inodes in parallel to minimize runtime, particularly
5061
if the program has been invoked manually from a command line.
5062
This requires careful scheduling to keep the threads as evenly loaded as
5063
possible.
5065
Early iterations of the ``xfs_scrub`` inode scanner naïvely created a single
5066
workqueue and scheduled a single workqueue item per AG.
5067
Each workqueue item walked the inode btree (with ``XFS_IOC_INUMBERS``) to find
5068
inode chunks and then called bulkstat (``XFS_IOC_BULKSTAT``) to gather enough
5069
information to construct file handles.
5070
The file handle was then passed to a function to generate scrub items for each
5071
metadata object of each inode.
5072
This simple algorithm leads to thread balancing problems in phase 3 if the
5073
filesystem contains one AG with a few large sparse files and the rest of the
5074
AGs contain many smaller files.
5075
The inode scan dispatch function was not sufficiently granular; it should have
5076
been dispatching at the level of individual inodes, or, to constrain memory
5077
consumption, inode btree records.
5079
Thanks to Dave Chinner, bounded workqueues in userspace enable ``xfs_scrub`` to
5080
avoid this problem with ease by adding a second workqueue.
5081
Just like before, the first workqueue is seeded with one workqueue item per AG,
5082
and it uses INUMBERS to find inode btree chunks.
5083
The second workqueue, however, is configured with an upper bound on the number
5084
of items that can be waiting to be run.
5085
Each inode btree chunk found by the first workqueue's workers are queued to the
5086
second workqueue, and it is this second workqueue that queries BULKSTAT,
5087
creates a file handle, and passes it to a function to generate scrub items for
5088
each metadata object of each inode.
5089
If the second workqueue is too full, the workqueue add function blocks the
5090
first workqueue's workers until the backlog eases.
5091
This doesn't completely solve the balancing problem, but reduces it enough to
5092
move on to more pressing issues.
5094
The proposed patchsets are the scrub
5095
`performance tweaks
5096
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-performance-tweaks>`_
5097
and the
5098
`inode scan rebalance
5099
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance>`_
5100
series.
5102
.. _scrubrepair:
5104
Scheduling Repairs
5105
------------------
5107
During phase 2, corruptions and inconsistencies reported in any AGI header or
5108
inode btree are repaired immediately, because phase 3 relies on proper
5109
functioning of the inode indices to find inodes to scan.
5110
Failed repairs are rescheduled to phase 4.
5111
Problems reported in any other space metadata are deferred to phase 4.
5112
Optimization opportunities are always deferred to phase 4, no matter their
5113
origin.
5115
During phase 3, corruptions and inconsistencies reported in any part of a
5116
file's metadata are repaired immediately if all space metadata were validated
5117
during phase 2.
5118
Repairs that fail or cannot be repaired immediately are scheduled for phase 4.
5120
In the original design of ``xfs_scrub``, it was thought that repairs would be
5121
so infrequent that the ``struct xfs_scrub_metadata`` objects used to
5122
communicate with the kernel could also be used as the primary object to
5123
schedule repairs.
5124
With recent increases in the number of optimizations possible for a given
5125
filesystem object, it became much more memory-efficient to track all eligible
5126
repairs for a given filesystem object with a single repair item.
5127
Each repair item represents a single lockable object -- AGs, metadata files,
5128
individual inodes, or a class of summary information.
5130
Phase 4 is responsible for scheduling a lot of repair work in as quick a
5131
manner as is practical.
5132
The :ref:`data dependencies <scrubcheck>` outlined earlier still apply, which
5133
means that ``xfs_scrub`` must try to complete the repair work scheduled by
5134
phase 2 before trying repair work scheduled by phase 3.
5135
The repair process is as follows:
5137
1. Start a round of repair with a workqueue and enough workers to keep the CPUs
5138
as busy as the user desires.
5140
a. For each repair item queued by phase 2,
5142
i. Ask the kernel to repair everything listed in the repair item for a
5143
given filesystem object.
5145
ii. Make a note if the kernel made any progress in reducing the number
5146
of repairs needed for this object.
5148
iii. If the object no longer requires repairs, revalidate all metadata
5149
associated with this object.
5150
If the revalidation succeeds, drop the repair item.
5151
If not, requeue the item for more repairs.
5153
b. If any repairs were made, jump back to 1a to retry all the phase 2 items.
5155
c. For each repair item queued by phase 3,
5157
i. Ask the kernel to repair everything listed in the repair item for a
5158
given filesystem object.
5160
ii. Make a note if the kernel made any progress in reducing the number
5161
of repairs needed for this object.
5163
iii. If the object no longer requires repairs, revalidate all metadata
5164
associated with this object.
5165
If the revalidation succeeds, drop the repair item.
5166
If not, requeue the item for more repairs.
5168
d. If any repairs were made, jump back to 1c to retry all the phase 3 items.
5170
2. If step 1 made any repair progress of any kind, jump back to step 1 to start
5171
another round of repair.
5173
3. If there are items left to repair, run them all serially one more time.
5174
Complain if the repairs were not successful, since this is the last chance
5175
to repair anything.
5177
Corruptions and inconsistencies encountered during phases 5 and 7 are repaired
5178
immediately.
5179
Corrupt file data blocks reported by phase 6 cannot be recovered by the
5180
filesystem.
5182
The proposed patchsets are the
5183
`repair warning improvements
5184
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings>`_,
5185
refactoring of the
5186
`repair data dependency
5187
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps>`_
5188
and
5189
`object tracking
5190
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking>`_,
5191
and the
5192
`repair scheduling
5193
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling>`_
5194
improvement series.
5196
Checking Names for Confusable Unicode Sequences
5197
-----------------------------------------------
5199
If ``xfs_scrub`` succeeds in validating the filesystem metadata by the end of
5200
phase 4, it moves on to phase 5, which checks for suspicious looking names in
5201
the filesystem.
5202
These names consist of the filesystem label, names in directory entries, and
5203
the names of extended attributes.
5204
Like most Unix filesystems, XFS imposes the sparest of constraints on the
5205
contents of a name:
5207
- Slashes and null bytes are not allowed in directory entries.
5209
- Null bytes are not allowed in userspace-visible extended attributes.
5211
- Null bytes are not allowed in the filesystem label.
5213
Directory entries and attribute keys store the length of the name explicitly
5214
ondisk, which means that nulls are not name terminators.
5215
For this section, the term "naming domain" refers to any place where names are
5216
presented together -- all the names in a directory, or all the attributes of a
5217
file.
5219
Although the Unix naming constraints are very permissive, the reality of most
5220
modern-day Linux systems is that programs work with Unicode character code
5221
points to support international languages.
5222
These programs typically encode those code points in UTF-8 when interfacing
5223
with the C library because the kernel expects null-terminated names.
5224
In the common case, therefore, names found in an XFS filesystem are actually
5225
UTF-8 encoded Unicode data.
5227
To maximize its expressiveness, the Unicode standard defines separate control
5228
points for various characters that render similarly or identically in writing
5229
systems around the world.
5230
For example, the character "Cyrillic Small Letter A" U+0430 "а" often renders
5231
identically to "Latin Small Letter A" U+0061 "a".
5233
The standard also permits characters to be constructed in multiple ways --
5234
either by using a defined code point, or by combining one code point with
5235
various combining marks.
5236
For example, the character "Angstrom Sign U+212B "Å" can also be expressed
5237
as "Latin Capital Letter A" U+0041 "A" followed by "Combining Ring Above"
5238
U+030A "◌̊".
5239
Both sequences render identically.
5241
Like the standards that preceded it, Unicode also defines various control
5242
characters to alter the presentation of text.
5243
For example, the character "Right-to-Left Override" U+202E can trick some
5244
programs into rendering "moo\\xe2\\x80\\xaegnp.txt" as "mootxt.png".
5245
A second category of rendering problems involves whitespace characters.
5246
If the character "Zero Width Space" U+200B is encountered in a file name, the
5247
name will render identically to a name that does not have the zero width
5248
space.
5250
If two names within a naming domain have different byte sequences but render
5251
identically, a user may be confused by it.
5252
The kernel, in its indifference to upper level encoding schemes, permits this.
5253
Most filesystem drivers persist the byte sequence names that are given to them
5254
by the VFS.
5256
Techniques for detecting confusable names are explained in great detail in
5257
sections 4 and 5 of the
5258
`Unicode Security Mechanisms <https://unicode.org/reports/tr39/>`_
5259
document.
5260
When ``xfs_scrub`` detects UTF-8 encoding in use on a system, it uses the
5261
Unicode normalization form NFD in conjunction with the confusable name
5262
detection component of
5263
`libicu <https://github.com/unicode-org/icu>`_
5264
to identify names with a directory or within a file's extended attributes that
5265
could be confused for each other.
5266
Names are also checked for control characters, non-rendering characters, and
5267
mixing of bidirectional characters.
5268
All of these potential issues are reported to the system administrator during
5269
phase 5.
5271
Media Verification of File Data Extents
5272
---------------------------------------
5274
The system administrator can elect to initiate a media scan of all file data
5275
blocks.
5276
This scan after validation of all filesystem metadata (except for the summary
5277
counters) as phase 6.
5278
The scan starts by calling ``FS_IOC_GETFSMAP`` to scan the filesystem space map
5279
to find areas that are allocated to file data fork extents.
5280
Gaps between data fork extents that are smaller than 64k are treated as if
5281
they were data fork extents to reduce the command setup overhead.
5282
When the space map scan accumulates a region larger than 32MB, a media
5283
verification request is sent to the disk as a directio read of the raw block
5284
device.
5286
If the verification read fails, ``xfs_scrub`` retries with single-block reads
5287
to narrow down the failure to the specific region of the media and recorded.
5288
When it has finished issuing verification requests, it again uses the space
5289
mapping ioctl to map the recorded media errors back to metadata structures
5290
and report what has been lost.
5291
For media errors in blocks owned by files, parent pointers can be used to
5292
construct file paths from inode numbers for user-friendly reporting.
5294
7. Conclusion and Future Work
5295
=============================
5297
It is hoped that the reader of this document has followed the designs laid out
5298
in this document and now has some familiarity with how XFS performs online
5299
rebuilding of its metadata indices, and how filesystem users can interact with
5300
that functionality.
5301
Although the scope of this work is daunting, it is hoped that this guide will
5302
make it easier for code readers to understand what has been built, for whom it
5303
has been built, and why.
5304
Please feel free to contact the XFS mailing list with questions.
5306
XFS_IOC_EXCHANGE_RANGE
5307
----------------------
5309
As discussed earlier, a second frontend to the atomic file mapping exchange
5310
mechanism is a new ioctl call that userspace programs can use to commit updates
5311
to files atomically.
5312
This frontend has been out for review for several years now, though the
5313
necessary refinements to online repair and lack of customer demand mean that
5314
the proposal has not been pushed very hard.
5316
File Content Exchanges with Regular User Files
5317
``````````````````````````````````````````````
5319
As mentioned earlier, XFS has long had the ability to swap extents between
5320
files, which is used almost exclusively by ``xfs_fsr`` to defragment files.
5321
The earliest form of this was the fork swap mechanism, where the entire
5322
contents of data forks could be exchanged between two files by exchanging the
5323
raw bytes in each inode fork's immediate area.
5324
When XFS v5 came along with self-describing metadata, this old mechanism grew
5325
some log support to continue rewriting the owner fields of BMBT blocks during
5326
log recovery.
5327
When the reverse mapping btree was later added to XFS, the only way to maintain
5328
the consistency of the fork mappings with the reverse mapping index was to
5329
develop an iterative mechanism that used deferred bmap and rmap operations to
5330
swap mappings one at a time.
5331
This mechanism is identical to steps 2-3 from the procedure above except for
5332
the new tracking items, because the atomic file mapping exchange mechanism is
5333
an iteration of an existing mechanism and not something totally novel.
5334
For the narrow case of file defragmentation, the file contents must be
5335
identical, so the recovery guarantees are not much of a gain.
5337
Atomic file content exchanges are much more flexible than the existing swapext
5338
implementations because it can guarantee that the caller never sees a mix of
5339
old and new contents even after a crash, and it can operate on two arbitrary
5340
file fork ranges.
5341
The extra flexibility enables several new use cases:
5343
- **Atomic commit of file writes**: A userspace process opens a file that it
5344
wants to update.
5345
Next, it opens a temporary file and calls the file clone operation to reflink
5346
the first file's contents into the temporary file.
5347
Writes to the original file should instead be written to the temporary file.
5348
Finally, the process calls the atomic file mapping exchange system call
5349
(``XFS_IOC_EXCHANGE_RANGE``) to exchange the file contents, thereby
5350
committing all of the updates to the original file, or none of them.
5352
.. _exchrange_if_unchanged:
5354
- **Transactional file updates**: The same mechanism as above, but the caller
5355
only wants the commit to occur if the original file's contents have not
5356
changed.
5357
To make this happen, the calling process snapshots the file modification and
5358
change timestamps of the original file before reflinking its data to the
5359
temporary file.
5360
When the program is ready to commit the changes, it passes the timestamps
5361
into the kernel as arguments to the atomic file mapping exchange system call.
5362
The kernel only commits the changes if the provided timestamps match the
5363
original file.
5364
A new ioctl (``XFS_IOC_COMMIT_RANGE``) is provided to perform this.
5366
- **Emulation of atomic block device writes**: Export a block device with a
5367
logical sector size matching the filesystem block size to force all writes
5368
to be aligned to the filesystem block size.
5369
Stage all writes to a temporary file, and when that is complete, call the
5370
atomic file mapping exchange system call with a flag to indicate that holes
5371
in the temporary file should be ignored.
5372
This emulates an atomic device write in software, and can support arbitrary
5373
scattered writes.
5375
Vectorized Scrub
5376
----------------
5378
As it turns out, the :ref:`refactoring <scrubrepair>` of repair items mentioned
5379
earlier was a catalyst for enabling a vectorized scrub system call.
5380
Since 2018, the cost of making a kernel call has increased considerably on some
5381
systems to mitigate the effects of speculative execution attacks.
5382
This incentivizes program authors to make as few system calls as possible to
5383
reduce the number of times an execution path crosses a security boundary.
5385
With vectorized scrub, userspace pushes to the kernel the identity of a
5386
filesystem object, a list of scrub types to run against that object, and a
5387
simple representation of the data dependencies between the selected scrub
5388
types.
5389
The kernel executes as much of the caller's plan as it can until it hits a
5390
dependency that cannot be satisfied due to a corruption, and tells userspace
5391
how much was accomplished.
5392
It is hoped that ``io_uring`` will pick up enough of this functionality that
5393
online fsck can use that instead of adding a separate vectored scrub system
5394
call to XFS.
5396
The relevant patchsets are the
5397
`kernel vectorized scrub
5398
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=vectorized-scrub>`_
5399
and
5400
`userspace vectorized scrub
5401
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=vectorized-scrub>`_
5402
series.
5404
Quality of Service Targets for Scrub
5405
------------------------------------
5407
One serious shortcoming of the online fsck code is that the amount of time that
5408
it can spend in the kernel holding resource locks is basically unbounded.
5409
Userspace is allowed to send a fatal signal to the process which will cause
5410
``xfs_scrub`` to exit when it reaches a good stopping point, but there's no way
5411
for userspace to provide a time budget to the kernel.
5412
Given that the scrub codebase has helpers to detect fatal signals, it shouldn't
5413
be too much work to allow userspace to specify a timeout for a scrub/repair
5414
operation and abort the operation if it exceeds budget.
5415
However, most repair functions have the property that once they begin to touch
5416
ondisk metadata, the operation cannot be cancelled cleanly, after which a QoS
5417
timeout is no longer useful.
5419
Defragmenting Free Space
5420
------------------------
5422
Over the years, many XFS users have requested the creation of a program to
5423
clear a portion of the physical storage underlying a filesystem so that it
5424
becomes a contiguous chunk of free space.
5425
Call this free space defragmenter ``clearspace`` for short.
5427
The first piece the ``clearspace`` program needs is the ability to read the
5428
reverse mapping index from userspace.
5429
This already exists in the form of the ``FS_IOC_GETFSMAP`` ioctl.
5430
The second piece it needs is a new fallocate mode
5431
(``FALLOC_FL_MAP_FREE_SPACE``) that allocates the free space in a region and
5432
maps it to a file.
5433
Call this file the "space collector" file.
5434
The third piece is the ability to force an online repair.
5436
To clear all the metadata out of a portion of physical storage, clearspace
5437
uses the new fallocate map-freespace call to map any free space in that region
5438
to the space collector file.
5439
Next, clearspace finds all metadata blocks in that region by way of
5440
``GETFSMAP`` and issues forced repair requests on the data structure.
5441
This often results in the metadata being rebuilt somewhere that is not being
5442
cleared.
5443
After each relocation, clearspace calls the "map free space" function again to
5444
collect any newly freed space in the region being cleared.
5446
To clear all the file data out of a portion of the physical storage, clearspace
5447
uses the FSMAP information to find relevant file data blocks.
5448
Having identified a good target, it uses the ``FICLONERANGE`` call on that part
5449
of the file to try to share the physical space with a dummy file.
5450
Cloning the extent means that the original owners cannot overwrite the
5451
contents; any changes will be written somewhere else via copy-on-write.
5452
Clearspace makes its own copy of the frozen extent in an area that is not being
5453
cleared, and uses ``FIEDEUPRANGE`` (or the :ref:`atomic file content exchanges
5454
<exchrange_if_unchanged>` feature) to change the target file's data extent
5455
mapping away from the area being cleared.
5456
When all other mappings have been moved, clearspace reflinks the space into the
5457
space collector file so that it becomes unavailable.
5459
There are further optimizations that could apply to the above algorithm.
5460
To clear a piece of physical storage that has a high sharing factor, it is
5461
strongly desirable to retain this sharing factor.
5462
In fact, these extents should be moved first to maximize sharing factor after
5463
the operation completes.
5464
To make this work smoothly, clearspace needs a new ioctl
5465
(``FS_IOC_GETREFCOUNTS``) to report reference count information to userspace.
5466
With the refcount information exposed, clearspace can quickly find the longest,
5467
most shared data extents in the filesystem, and target them first.
5469
**Future Work Question**: How might the filesystem move inode chunks?
5471
*Answer*: To move inode chunks, Dave Chinner constructed a prototype program
5472
that creates a new file with the old contents and then locklessly runs around
5473
the filesystem updating directory entries.
5474
The operation cannot complete if the filesystem goes down.
5475
That problem isn't totally insurmountable: create an inode remapping table
5476
hidden behind a jump label, and a log item that tracks the kernel walking the
5477
filesystem to update directory entries.
5478
The trouble is, the kernel can't do anything about open files, since it cannot
5479
revoke them.
5481
**Future Work Question**: Can static keys be used to minimize the cost of
5482
supporting ``revoke()`` on XFS files?
5484
*Answer*: Yes.
5485
Until the first revocation, the bailout code need not be in the call path at
5486
all.
5488
The relevant patchsets are the
5489
`kernel freespace defrag
5490
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace>`_
5491
and
5492
`userspace freespace defrag
5493
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace>`_
5494
series.
5496
Shrinking Filesystems
5497
---------------------
5499
Removing the end of the filesystem ought to be a simple matter of evacuating
5500
the data and metadata at the end of the filesystem, and handing the freed space
5501
to the shrink code.
5502
That requires an evacuation of the space at end of the filesystem, which is a
5503
use of free space defragmentation!
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 목적과 일곱 부분의 구성
1-53이 문서는 `GPL-2.0`으로 배포되는 XFS online filesystem check 설계서이며 anchor는 `xfs_online_fsck_design`, 주 저자는 Darrick J. Wong입니다. Heading style mapping과 절 번호 수동 부여 규칙은 RST comment로 남아 있습니다.
목적은 kernel distributor에게 기능과 주의점을 알리고, code reader에게 구현을 읽기 전 개념과 design point를 제공하며, maintainer가 상위 의사결정의 근거를 보존하게 하는 것입니다. Online fsck code가 merge되면 topic-branch link는 실제 code link로 교체됩니다.
전체는 일곱 부분입니다. 1부는 fsck 정의와 동기, 2·3부는 동작 이론과 testing, 4부는 user interface, 5·6부는 kernel/userspace component와 repair case study, 7부는 결론과 향후 확장을 다룹니다.
1부: filesystem check의 정의와 문제→2부: online fsck의 작업 이론→3부: differential·fuzz·stress testing→4부: foreground·background·health interface→5·6부: kernel/userspace algorithm과 case study→7부: 결론과 future work
이번 초안은 4부까지 번역하며 나머지는 후속 묶음에서 이어집니다.
.. SPDX-License-Identifier: GPL-2.0
.. _xfs_online_fsck_design:
..
Mapping of heading styles within this document:
Heading 1 uses "====" above and below
Heading 2 uses "===="
Heading 3 uses "----"
Heading 4 uses "````"
Heading 5 uses "^^^^"
Heading 6 uses "~~~~"
Heading 7 uses "...."
Sections are manually numbered because apparently that's what everyone
does in the kernel.
======================
XFS Online Fsck Design
======================
This document captures the design of the online filesystem check feature for
XFS.
The purpose of this document is threefold:
- To help kernel distributors understand exactly what the XFS online fsck
feature is, and issues about which they should be aware.
- To help people reading the code to familiarize themselves with the relevant
concepts and design points before they start digging into the code.
- To help developers maintaining the system by capturing the reasons
supporting higher level decision making.
As the online fsck code is merged, the links in this document to topic branches
will be replaced with links to code.
This document is licensed under the terms of the GNU Public License, v2.
The primary author is Darrick J. Wong.
This design document is split into seven parts.
Part 1 defines what fsck tools are and the motivations for writing a new one.
Parts 2 and 3 present a high level overview of how online fsck process works
and how it is tested to ensure correct functionality.
Part 4 discusses the user interface and the intended usage modes of the new
program.
Parts 5 and 6 show off the high level components and how they fit together, and
then present case studies of how each repair function actually works.
Part 7 sums up what has been discussed so far and speculates about what else
might be built atop online fsck.
.. contents:: Table of Contents
:local:
Filesystem check의 책임과 복구 목표
54-104Unix filesystem의 주요 책임은 application이 임의 data blob에 장기간 이름을 붙일 hierarchy를 제공하고, 그 이름 전체에 physical storage를 virtualize하며, 언제든 이름으로 data를 되찾고 resource usage를 조사하는 것입니다.
File, directory, space mapping처럼 이 기능을 직접 지원하는 구조는 primary metadata입니다. Reverse mapping과 directory parent pointer 같은 secondary metadata는 consistency check와 reorganization 등 filesystem 내부 operation을 지원합니다. Summary metadata는 성능을 위해 primary metadata 정보를 압축합니다.
Fsck는 모든 metadata를 검사해 명백한 corruption뿐 아니라 서로 다른 record 유형의 cross-reference inconsistency도 찾습니다. 대부분의 도구는 문제를 수정할 수 있지만 Linux fsck의 우선 목표는 회수 data 최대화가 아니라 metadata를 consistent state로 복원하는 것입니다.
20세기 filesystem은 on-disk redundancy가 거의 없어 오류가 사라질 때까지 file을 지우는 방식이 일반적이었습니다. 현대 format은 non-catastrophic error에서 structure를 재생성할 만큼 metadata redundancy를 제공할 수 있습니다.
원문 주의 상자는 backup으로 별도 storage system 수를 늘려 data loss를 줄이고 RAID로 각 system의 redundancy를 높여 downtime을 줄인다고 구분합니다. Fsck tool은 이 가운데 metadata inconsistency 문제만 다룹니다.
분류예주요 역할
PrimaryFile·directory·space mapping사용자 object와 storage 직접 표현
SecondaryReverse mapping·parent pointer검증과 reorganization 지원
SummaryFree-space·inode countPrimary 정보를 압축해 query 가속
각 종류가 제공하는 기능과 repair 정보원이 다릅니다.
1. What is a Filesystem Check?
==============================
A Unix filesystem has four main responsibilities:
- Provide a hierarchy of names through which application programs can associate
arbitrary blobs of data for any length of time,
- Virtualize physical storage media across those names, and
- Retrieve the named data blobs at any time.
- Examine resource usage.
Metadata directly supporting these functions (e.g. files, directories, space
mappings) are sometimes called primary metadata.
Secondary metadata (e.g. reverse mapping and directory parent pointers) support
operations internal to the filesystem, such as internal consistency checking
and reorganization.
Summary metadata, as the name implies, condense information contained in
primary metadata for performance reasons.
The filesystem check (fsck) tool examines all the metadata in a filesystem
to look for errors.
In addition to looking for obvious metadata corruptions, fsck also
cross-references different types of metadata records with each other to look
for inconsistencies.
People do not like losing data, so most fsck tools also contains some ability
to correct any problems found.
As a word of caution -- the primary goal of most Linux fsck tools is to restore
the filesystem metadata to a consistent state, not to maximize the data
recovered.
That precedent will not be challenged here.
Filesystems of the 20th century generally lacked any redundancy in the ondisk
format, which means that fsck can only respond to errors by erasing files until
errors are no longer detected.
More recent filesystem designs contain enough redundancy in their metadata that
it is now possible to regenerate data structures when non-catastrophic errors
occur; this capability aids both strategies.
+--------------------------------------------------------------------------+
| **Note**: |
+--------------------------------------------------------------------------+
| System administrators avoid data loss by increasing the number of |
| separate storage systems through the creation of backups; and they avoid |
| downtime by increasing the redundancy of each storage system through the |
| creation of RAID arrays. |
| fsck tools address only the first problem. |
+--------------------------------------------------------------------------+
Kernel·userspace·QA topic branch
105-114Online repair code는 kernel, xfsprogs, fstests의 kernel.org git tree에 각각 게시됩니다. Kernel patchset이 repair function 하나를 추가할 때 세 repository에서 같은 branch 이름을 사용해 대응 변경을 찾기 쉽게 합니다.
영역Branch URL
Kernelhttps://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-symlink
Userspacehttps://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service
QAhttps://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs
원문 URL과 branch를 그대로 보존합니다.
TLDR; Show Me the Code!
-----------------------
Code is posted to the kernel.org git trees as follows:
`kernel changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-symlink>`_,
`userspace changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_, and
`QA test changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs>`_.
Each kernel patchset adding an online repair function will use the same branch
name across the kernel, xfsprogs, and fstests git repos.
xfs_check와 xfs_repair의 한계
115-139이 online fsck는 Linux XFS 역사상 세 번째 check·repair tool입니다. 첫 도구 `xfs_check`는 `xfs_db`의 일부로 만들어졌고 unmounted filesystem에서만 실행됩니다. 모든 metadata inconsistency를 찾지만 repair 기능이 없고 memory 요구량이 커 현재 deprecated 상태입니다.
두 번째 `xfs_repair`도 unmounted filesystem 전용이지만 더 빠르고 견고하도록 만들어졌습니다. Extent 기반 in-memory structure로 memory consumption을 줄이고, 전체 metadata scan 중 I/O 대기를 줄이도록 readahead를 schedule합니다.
`xfs_repair`의 핵심 기능은 file metadata와 directory tree inconsistency를 없애기 위해 필요하면 object를 지우는 것입니다. Space-usage metadata는 관찰한 file metadata에서 다시 구축합니다.
도구Mounted 상태Repair상태
xfs_checkUnmounted only없음Deprecated
xfs_repairUnmounted only삭제와 index rebuild현재 offline fsck
xfs_scrubMounted onlineKernel online repair 호출새 online fsck driver
가용성·repair·운영 상태를 비교합니다.
Existing Tools
--------------
The online fsck tool described here will be the third tool in the history of
XFS (on Linux) to check and repair filesystems.
Two programs precede it:
The first program, ``xfs_check``, was created as part of the XFS debugger
(``xfs_db``) and can only be used with unmounted filesystems.
It walks all metadata in the filesystem looking for inconsistencies in the
metadata, though it lacks any ability to repair what it finds.
Due to its high memory requirements and inability to repair things, this
program is now deprecated and will not be discussed further.
The second program, ``xfs_repair``, was created to be faster and more robust
than the first program.
Like its predecessor, it can only be used with unmounted filesystems.
It uses extent-based in-memory data structures to reduce memory consumption,
and tries to schedule readahead IO appropriately to reduce I/O waiting time
while it scans the metadata of the entire filesystem.
The most important feature of this tool is its ability to respond to
inconsistencies in file metadata and directory tree by erasing things as needed
to eliminate problems.
Space usage metadata are rebuilt from the observed file metadata.
Online fsck가 해결하려는 일곱 문제
140-213기존 도구는 silent metadata corruption으로 갑작스러운 shutdown이 발생하면 user program이 예측 불가능하게 filesystem access를 잃는 문제와, crash recovery 또는 사전 점검을 위해 filesystem을 offline으로 만들 때 전체 service가 중단되는 문제를 남깁니다.
Data owner는 모든 data를 읽지 않고 integrity를 확인할 수 없어 storage administrator의 linear media scan으로 충분할 상황에도 큰 비용을 부담할 수 있습니다. Administrator는 online health를 평가할 수 없으면 maintenance window를 계획하기 어렵고 fleet monitoring은 manual intervention과 downtime 때문에 periodic health check를 자동화할 수 없습니다.
또한 악의적 사용자가 Unicode의 특성을 이용해 directory에 오해를 부르는 이름을 만들면 user가 원하지 않는 operation을 수행하게 할 수 있습니다.
해결책은 running filesystem에서 동작하는 세 번째 fsck입니다. In-kernel metadata check, in-kernel metadata repair, live filesystem의 작업을 구동하는 userspace `xfs_scrub` 세 component로 구성됩니다. 문서에서는 validate kernel 부분을 `online scrub`, 수정 부분을 `online repair`, 기존 offline tool을 `xfs_repair`라 부릅니다.
XFS는 naming hierarchy를 file과 directory로, physical space를 allocation group으로 shard합니다. 이 구조와 redundant metadata 덕분에 일부 shard만 targeted check·repair하는 동안 다른 영역은 I/O를 계속 처리할 수 있습니다. 전체 scan이 필요한 경우도 background에서 수행할 수 있어 automatic management와 결합한 self-healing으로 availability를 높입니다.
주체현재 문제Online 목표
User programSilent corruption 뒤 갑작스러운 access loss조기 발견과 targeted repair
UserRecovery·proactive check의 total downtimeMounted 상태에서 검사
Data owner전체 read 없이는 integrity 확인 불가선택적 media verification
AdministratorOnline health 부재로 window 계획 불가Health reporting
Fleet toolingManual check로 automation 불가Scheduled background service
UserConfusable Unicode name에 기만 가능Name validation
원문 일곱 문제를 운영 목표에 연결합니다.
Problem Statement
-----------------
The current XFS tools leave several problems unsolved:
1. **User programs** suddenly **lose access** to the filesystem when unexpected
shutdowns occur as a result of silent corruptions in the metadata.
These occur **unpredictably** and often without warning.
2. **Users** experience a **total loss of service** during the recovery period
after an **unexpected shutdown** occurs.
3. **Users** experience a **total loss of service** if the filesystem is taken
offline to **look for problems** proactively.
4. **Data owners** cannot **check the integrity** of their stored data without
reading all of it.
This may expose them to substantial billing costs when a linear media scan
performed by the storage system administrator might suffice.
5. **System administrators** cannot **schedule** a maintenance window to deal
with corruptions if they **lack the means** to assess filesystem health
while the filesystem is online.
6. **Fleet monitoring tools** cannot **automate periodic checks** of filesystem
health when doing so requires **manual intervention** and downtime.
7. **Users** can be tricked into **doing things they do not desire** when
malicious actors **exploit quirks of Unicode** to place misleading names
in directories.
Given this definition of the problems to be solved and the actors who would
benefit, the proposed solution is a third fsck tool that acts on a running
filesystem.
This new third program has three components: an in-kernel facility to check
metadata, an in-kernel facility to repair metadata, and a userspace driver
program to drive fsck activity on a live filesystem.
``xfs_scrub`` is the name of the driver program.
The rest of this document presents the goals and use cases of the new fsck
tool, describes its major design points in connection to those goals, and
discusses the similarities and differences with existing tools.
+--------------------------------------------------------------------------+
| **Note**: |
+--------------------------------------------------------------------------+
| Throughout this document, the existing offline fsck tool can also be |
| referred to by its current name "``xfs_repair``". |
| The userspace driver program for the new online fsck tool can be |
| referred to as "``xfs_scrub``". |
| The kernel portion of online fsck that validates metadata is called |
| "online scrub", and portion of the kernel that fixes metadata is called |
| "online repair". |
+--------------------------------------------------------------------------+
The naming hierarchy is broken up into objects known as directories and files
and the physical space is split into pieces known as allocation groups.
Sharding enables better performance on highly parallel systems and helps to
contain the damage when corruptions occur.
The division of the filesystem into principal objects (allocation groups and
inodes) means that there are ample opportunities to perform targeted checks and
repairs on a subset of the filesystem.
While this is going on, other parts continue processing IO requests.
Even if a piece of filesystem metadata can only be regenerated by scanning the
entire system, the scan can still be done in the background while other file
operations continue.
In summary, online fsck takes advantage of resource sharding and redundant
metadata to enable targeted checking and repair operations while the system
is running.
This capability will be coupled to automatic system management so that
autonomous self-healing of XFS maximizes service availability.
Userspace scheduler와 kernel check·repair
214-236Online fsck는 live metadata object를 lock하고 scan해야 하므로 세 code component로 나뉩니다. Userspace `xfs_scrub`는 개별 metadata item 식별, work scheduling, 결과 대응, administrator report를 담당하고 kernel은 scrub-item 유형별 check와 repair function을 구현합니다.
문서는 `online fsck work item`을 줄여 `scrub item`이라 부릅니다. Unix philosophy에 맞게 각 item type은 metadata structure의 한 측면만 맡아 잘 처리하도록 나눕니다.
xfs_scrub가 metadata item과 capability 발견→Userspace가 scrub item을 schedule→Kernel check function이 live metadata 검사→허용되고 필요하면 kernel repair function 실행→Userspace가 결과를 해석해 administrator에게 보고
Control과 metadata lock operation을 분리합니다.
2. Theory of Operation
======================
Because it is necessary for online fsck to lock and scan live metadata objects,
online fsck consists of three separate code components.
The first is the userspace driver program ``xfs_scrub``, which is responsible
for identifying individual metadata items, scheduling work items for them,
reacting to the outcomes appropriately, and reporting results to the system
administrator.
The second and third are in the kernel, which implements functions to check
and repair each type of online fsck work item.
+------------------------------------------------------------------+
| **Note**: |
+------------------------------------------------------------------+
| For brevity, this document shortens the phrase "online fsck work |
| item" to "scrub item". |
+------------------------------------------------------------------+
Scrub item types are delineated in a manner consistent with the Unix design
philosophy, which is to say that each item should handle one aspect of a
metadata structure, and handle it well.
Offline fsck를 대체하지 않는 이유
237-258원칙적으로 online fsck는 offline fsck가 처리할 모든 대상을 check·repair해야 합니다. 그러나 계속 실행되는 것은 아니므로 scrub 이후 latent error가 생겨 다음 mount가 실패하면 offline fsck만이 해결책입니다. 따라서 `xfs_repair` 유지보수는 계속됩니다.
Online fsck는 일반 filesystem과 같은 resource sharing·lock acquisition rule을 따라야 하며 시간을 줄이려고 shortcut을 쓸 수 없습니다. 그러므로 전체 online run이 offline fsck보다 오래 걸릴 수도 있습니다.
이 한계는 목표가 system downtime 최소화와 operation predictability 향상이기 때문에 수용합니다. Online fsck는 offline fsck의 완전한 replacement가 아니라 서로 다른 운영 상황을 담당합니다.
항목Online fsckOffline fsck
Filesystem 상태Mounted·activeUnmounted
Lock ruleNormal filesystem rule 준수동시 workload 없음
Latent mount failure실행 시점 밖 오류는 놓칠 수 있음Mount 전 전체 repair
주요 목표Downtime·예측 불가능성 감소완전한 offline recovery
속도보다 서비스 지속성과 lock 안전성이 online 경로의 우선순위입니다.
Scope
-----
In principle, online fsck should be able to check and to repair everything that
the offline fsck program can handle.
However, online fsck cannot be running 100% of the time, which means that
latent errors may creep in after a scrub completes.
If these errors cause the next mount to fail, offline fsck is the only
solution.
This limitation means that maintenance of the offline fsck tool will continue.
A second limitation of online fsck is that it must follow the same resource
sharing and lock acquisition rules as the regular filesystem.
This means that scrub cannot take *any* shortcuts to save time, because doing
so could lead to concurrency problems.
In other words, online fsck is not a complete replacement for offline fsck, and
a complete run of online fsck may take longer than online fsck.
However, both of these limitations are acceptable tradeoffs to satisfy the
different motivations of online fsck, which are to **minimize system downtime**
and to **increase predictability of operation**.
.. _scrubphases:
xfs_scrub의 일곱 작업 단계
259-320Userspace `xfs_scrub`는 전체 filesystem check·repair를 일곱 phase로 나누며 각 phase는 이전 phase 성공에 의존합니다.
1단계는 mounted filesystem과 computer geometry, kernel online-fsck capability, underlying storage device를 수집합니다. 2단계는 allocation-group metadata, realtime-volume metadata, quota file을 item별로 검사합니다. Inode header나 inode btree corruption은 3단계 준비를 위해 허용 시 즉시 repair하며 나머지는 4단계로 미룹니다.
3단계는 모든 file의 각 metadata structure를 검사하고 2단계가 깨끗하며 repair가 허용되면 즉시 수정합니다. Optimization, deferred repair, 실패한 repair는 4단계로 보냅니다.
4단계는 허용된 모든 잔여 repair와 optimization을 수행합니다. 먼저 summary counter를 check·repair해 잘못된 값 때문에 resource reservation이 실패하지 않게 합니다. 어디선가 forward progress가 있는 동안 실패 item을 requeue하고, filesystem이 깨끗하면 끝에 free space를 trim합니다.
5단계는 primary·secondary metadata가 모두 올바른 상태에서 free-space, quota usage 같은 summary counter를 고치고 directory entry·xattr name의 control character와 confusable Unicode를 검사합니다. 6단계는 요청 시 allocated·written data extent를 모두 media scan하고 hardware error-correction을 활용해 error를 owning file에 mapping합니다. 7단계는 summary counter를 다시 검사하고 space·file count summary를 보고합니다.
1. Geometry·capability·device discovery→2. AG·realtime·quota metadata check, inode index 선행 repair→3. 모든 inode/file metadata check와 가능한 즉시 repair→4. Deferred repair·optimization·counter 선행 수정·trim→5. Summary counter와 suspicious name 검사→6. 선택적 data extent media scan→7. Counter 재검증과 최종 usage summary
앞 단계의 metadata 신뢰성을 다음 단계의 전제로 사용합니다.
Phases of Work
--------------
The userspace driver program ``xfs_scrub`` splits the work of checking and
repairing an entire filesystem into seven phases.
Each phase concentrates on checking specific types of scrub items and depends
on the success of all previous phases.
The seven phases are as follows:
1. Collect geometry information about the mounted filesystem and computer,
discover the online fsck capabilities of the kernel, and open the
underlying storage devices.
2. Check allocation group metadata, all realtime volume metadata, and all quota
files.
Each metadata structure is scheduled as a separate scrub item.
If corruption is found in the inode header or inode btree and ``xfs_scrub``
is permitted to perform repairs, then those scrub items are repaired to
prepare for phase 3.
Repairs are implemented by using the information in the scrub item to
resubmit the kernel scrub call with the repair flag enabled; this is
discussed in the next section.
Optimizations and all other repairs are deferred to phase 4.
3. Check all metadata of every file in the filesystem.
Each metadata structure is also scheduled as a separate scrub item.
If repairs are needed and ``xfs_scrub`` is permitted to perform repairs,
and there were no problems detected during phase 2, then those scrub items
are repaired immediately.
Optimizations, deferred repairs, and unsuccessful repairs are deferred to
phase 4.
4. All remaining repairs and scheduled optimizations are performed during this
phase, if the caller permits them.
Before starting repairs, the summary counters are checked and any necessary
repairs are performed so that subsequent repairs will not fail the resource
reservation step due to wildly incorrect summary counters.
Unsuccessful repairs are requeued as long as forward progress on repairs is
made somewhere in the filesystem.
Free space in the filesystem is trimmed at the end of phase 4 if the
filesystem is clean.
5. By the start of this phase, all primary and secondary filesystem metadata
must be correct.
Summary counters such as the free space counts and quota resource counts
are checked and corrected.
Directory entry names and extended attribute names are checked for
suspicious entries such as control characters or confusing Unicode sequences
appearing in names.
6. If the caller asks for a media scan, read all allocated and written data
file extents in the filesystem.
The ability to use hardware-assisted data file integrity checking is new
to online fsck; neither of the previous tools have this capability.
If media errors occur, they will be mapped to the owning files and reported.
7. Re-check the summary counters and presents the caller with a summary of
space usage and file counts.
This allocation of responsibilities will be :ref:`revisited <scrubcheck>`
later in this document.
각 scrub item의 check·repair·재검증
321-347Kernel은 scrub item이 나타내는 metadata 측면 하나를 세 단계로 처리합니다. 먼저 corruption, optimization 가능성, administrator-controlled value의 suspicious 상태를 검사합니다.
정상이고 optimization이 필요 없으면 resource를 놓고 positive result를 반환합니다. 문제가 있지만 caller가 repair를 허용하지 않으면 resource를 놓고 negative result를 반환합니다.
Repair가 허용되면 기존 structure를 부분 salvage하기보다 다른 metadata에서 새 structure를 rebuild하는 것이 일반적입니다. 실패하면 첫 scan 결과를 userspace에 돌려주고, 성공하면 같은 check를 새 item에 다시 실행해 repair 효과를 평가한 결과를 반환합니다.
1. Corruption·optimization·suspicious attribute check→2. 허용된 경우 다른 metadata에서 structure rebuild→3. 새 metadata에 같은 check를 다시 실행→Reassessment 결과를 userspace에 반환
Repair 뒤 동일 검사를 반복하는 것이 완료 조건입니다.
Steps for Each Scrub Item
-------------------------
The kernel scrub code uses a three-step strategy for checking and repairing
the one aspect of a metadata object represented by a scrub item:
1. The scrub item of interest is checked for corruptions; opportunities for
optimization; and for values that are directly controlled by the system
administrator but look suspicious.
If the item is not corrupt or does not need optimization, resource are
released and the positive scan results are returned to userspace.
If the item is corrupt or could be optimized but the caller does not permit
this, resources are released and the negative scan results are returned to
userspace.
Otherwise, the kernel moves on to the second step.
2. The repair function is called to rebuild the data structure.
Repair functions generally choose rebuild a structure from other metadata
rather than try to salvage the existing structure.
If the repair fails, the scan results from the first step are returned to
userspace.
Otherwise, the kernel moves on to the third step.
3. In the third step, the kernel runs the same checks over the new metadata
item to assess the efficacy of the repairs.
The results of the reassessment are returned to userspace.
Primary metadata의 잠금 기반 rebuild
348-420Primary metadata에는 free-space와 reference-count 정보, inode record·index, file data mapping, directory, xattr, symlink, quota limit가 포함됩니다. Scrub는 일반 access와 같은 resource·lock acquisition rule을 따릅니다.
Scrub는 해당 item을 소유한 principal object, 즉 allocation group이나 inode를 lock해 concurrent update를 막습니다. Check function은 해당 type의 모든 record를 명백한 오류에 대해 검사하고 건강한 record를 다른 metadata와 cross-reference합니다.
Repair는 이미 획득한 resource를 유지한 채 필요한 metadata를 scan해 observation을 수집하고 새 on-disk structure에 stage한 뒤 atomic commit합니다. 이후 old structure의 storage를 주의 깊게 reap합니다.
Repair 내내 primary object를 lock하므로 filesystem 일부에 대한 사실상 offline repair입니다. 동시 update와 다른 filesystem 영역을 처리할 필요가 없어 indexed structure를 빠르게 rebuild할 수 있지만, 손상 structure를 access하려는 program은 repair 완료까지 block됩니다. Target shard만 막아 전체 service loss를 피하는 것이 장점입니다.
원문은 이를 Srinivasan과 Carey의 offline index construction 및 list-based algorithm과 비교합니다. Resource lock을 repair 전 기간 유지하는 builder는 항상 offline algorithm입니다.
Allocation group 또는 inode lock→Record check·cross-reference·observation 수집→In-memory staging에서 새 structure format→새 on-disk structure atomic commit→Old metadata block reap 후 unlock
소유 object를 잠근 상태에서 replacement를 완성합니다.
Classification of Metadata
--------------------------
Each type of metadata object (and therefore each type of scrub item) is
classified as follows:
Primary Metadata
````````````````
Metadata structures in this category should be most familiar to filesystem
users either because they are directly created by the user or they index
objects created by the user
Most filesystem objects fall into this class:
- Free space and reference count information
- Inode records and indexes
- Storage mapping information for file data
- Directories
- Extended attributes
- Symbolic links
- Quota limits
Scrub obeys the same rules as regular filesystem accesses for resource and lock
acquisition.
Primary metadata objects are the simplest for scrub to process.
The principal filesystem object (either an allocation group or an inode) that
owns the item being scrubbed is locked to guard against concurrent updates.
The check function examines every record associated with the type for obvious
errors and cross-references healthy records against other metadata to look for
inconsistencies.
Repairs for this class of scrub item are simple, since the repair function
starts by holding all the resources acquired in the previous step.
The repair function scans available metadata as needed to record all the
observations needed to complete the structure.
Next, it stages the observations in a new ondisk structure and commits it
atomically to complete the repair.
Finally, the storage from the old data structure are carefully reaped.
Because ``xfs_scrub`` locks a primary object for the duration of the repair,
this is effectively an offline repair operation performed on a subset of the
filesystem.
This minimizes the complexity of the repair code because it is not necessary to
handle concurrent updates from other threads, nor is it necessary to access
any other part of the filesystem.
As a result, indexed structures can be rebuilt very quickly, and programs
trying to access the damaged structure will be blocked until repairs complete.
The only infrastructure needed by the repair code are the staging area for
observations and a means to write new structures to disk.
Despite these limitations, the advantage that online repair holds is clear:
targeted work on individual shards of the filesystem avoids total loss of
service.
This mechanism is described in section 2.1 ("Off-Line Algorithm") of
V. Srinivasan and M. J. Carey, `"Performance of On-Line Index Construction
Algorithms" <https://minds.wisconsin.edu/bitstream/handle/1793/59524/TR1047.pdf>`_,
*Extending Database Technology*, pp. 293-309, 1992.
Most primary metadata repair functions stage their intermediate results in an
in-memory array prior to formatting the new ondisk structure, which is very
similar to the list-based algorithm discussed in section 2.3 ("List-Based
Algorithms") of Srinivasan.
However, any data structure builder that maintains a resource lock for the
duration of the repair is *always* an offline algorithm.
.. _secondary_metadata:
Secondary metadata의 live scan과 hook
421-515Secondary metadata는 primary record를 반영하지만 online fsck나 filesystem reorganization에만 필요한 reverse mapping과 directory parent pointer입니다. Secondary object에 attach한 scrub가 primary metadata를 확인해야 해 일반 resource-acquisition 순서와 반대가 되고, rebuild에는 흔히 full filesystem scan이 필요합니다.
Repair는 전체 scan 동안 resource를 lock할 수 없으므로 in-memory staging structure를 준비한 뒤 모든 lock을 놓습니다. Scanner가 observation을 기록할 때 staging data만 잠깐 lock합니다. 동시에 live filesystem hook이 scan 중 발생하는 update를 staging information에 반영합니다.
Scan이 끝나면 owning object를 다시 lock하고 live staging data로 새 on-disk structure를 작성해 atomic commit합니다. Hook을 끄고 staging area를 해제한 뒤 old storage를 reap합니다.
Concurrency는 lock 문제를 피하지만 code complexity가 큽니다. Live path hook, parallel staging structure, scan progress와 inode-locking model이 통합돼 hook event가 update를 적용할지 결정해야 합니다. 손상 structure를 쓰는 application을 막지 않기 때문에 failure나 unplanned shutdown 가능성도 남습니다.
원문의 참고 algorithm은 index builder가 새 index를 scan해 만들고 side file이 동시 update를 기록하는 방식입니다. 공개 cursor보다 뒤의 record는 builder가 아직 처리할 것이므로 side-file update를 생략해 중복을 피합니다. XFS는 replacement index가 완성될 때까지 keyspace를 숨깁니다.
향후 live scan·hook으로 check도 shadow metadata를 만들고 on-disk record와 비교할 수 있지만 현재 check보다 작업량과 runtime이 커집니다.
Repair-specific staging index 준비→Filesystem lock을 놓고 primary metadata full scan→Observation을 staging에 병합→Live hook으로 scan 중 update를 동시 반영→Owner 재잠금 후 replacement atomic commit→Hook 해제·old storage reap
Lock을 놓은 full scan과 live update merge가 핵심입니다.
Secondary Metadata
``````````````````
Metadata structures in this category reflect records found in primary metadata,
but are only needed for online fsck or for reorganization of the filesystem.
Secondary metadata include:
- Reverse mapping information
- Directory parent pointers
This class of metadata is difficult for scrub to process because scrub attaches
to the secondary object but needs to check primary metadata, which runs counter
to the usual order of resource acquisition.
Frequently, this means that full filesystems scans are necessary to rebuild the
metadata.
Check functions can be limited in scope to reduce runtime.
Repairs, however, require a full scan of primary metadata, which can take a
long time to complete.
Under these conditions, ``xfs_scrub`` cannot lock resources for the entire
duration of the repair.
Instead, repair functions set up an in-memory staging structure to store
observations.
Depending on the requirements of the specific repair function, the staging
index will either have the same format as the ondisk structure or a design
specific to that repair function.
The next step is to release all locks and start the filesystem scan.
When the repair scanner needs to record an observation, the staging data are
locked long enough to apply the update.
While the filesystem scan is in progress, the repair function hooks the
filesystem so that it can apply pending filesystem updates to the staging
information.
Once the scan is done, the owning object is re-locked, the live data is used to
write a new ondisk structure, and the repairs are committed atomically.
The hooks are disabled and the staging area is freed.
Finally, the storage from the old data structure are carefully reaped.
Introducing concurrency helps online repair avoid various locking problems, but
comes at a high cost to code complexity.
Live filesystem code has to be hooked so that the repair function can observe
updates in progress.
The staging area has to become a fully functional parallel structure so that
updates can be merged from the hooks.
Finally, the hook, the filesystem scan, and the inode locking model must be
sufficiently well integrated that a hook event can decide if a given update
should be applied to the staging structure.
In theory, the scrub implementation could apply these same techniques for
primary metadata, but doing so would make it massively more complex and less
performant.
Programs attempting to access the damaged structures are not blocked from
operation, which may cause application failure or an unplanned filesystem
shutdown.
Inspiration for the secondary metadata repair strategy was drawn from section
2.4 of Srinivasan above, and sections 2 ("NSF: Index Build Without Side-File")
and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms for
Creating Indexes for Very Large Tables Without Quiescing Updates"
<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.
The sidecar index mentioned above bears some resemblance to the side file
method mentioned in Srinivasan and Mohan.
Their method consists of an index builder that extracts relevant record data to
build the new structure as quickly as possible; and an auxiliary structure that
captures all updates that would be committed to the index by other threads were
the new index already online.
After the index building scan finishes, the updates recorded in the side file
are applied to the new index.
To avoid conflicts between the index builder and other writer threads, the
builder maintains a publicly visible cursor that tracks the progress of the
scan through the record space.
To avoid duplication of work between the side file and the index builder, side
file updates are elided when the record ID for the update is greater than the
cursor position within the record ID space.
To minimize changes to the rest of the codebase, XFS online repair keeps the
replacement index hidden until it's completely ready to go.
In other words, there is no attempt to expose the keyspace of the new index
while repair is running.
The complexity of such an approach would be very high and perhaps more
appropriate to building *new* indices.
**Future Work Question**: Can the full scan and live update code used to
facilitate a repair also be used to implement a comprehensive check?
*Answer*: In theory, yes. Check would be much stronger if each scrub function
employed these live scans to build a shadow copy of the metadata and then
compared the shadow records to the ondisk records.
However, doing that is a fair amount more work than what the checking functions
do now.
The live scans and hooks were developed much later.
That in turn increases the runtime of those scrub functions.
Summary counter의 scan과 delta 유지
516-560Summary information은 primary record 내용을 압축해 resource-usage query를 빠르게 하며 원본 metadata보다 훨씬 작습니다. Free-space·inode count, directory에서 계산한 file link count, quota resource usage가 예입니다.
Check와 repair에는 full filesystem scan이 필요하지만 resource·lock 획득은 일반 access 경로를 따릅니다. Superblock incore counter는 별도 요구가 있고, quota와 link count는 secondary repair와 같은 scan·hook 기법을 사용하되 staging이 integer counter 집합이므로 on-disk structure 전체 mirror일 필요는 없습니다.
Online quotacheck는 transaction마다 block·inode usage delta를 추적해 commit 때 dquot side file에 반영합니다. Builder가 inode를 scan하지만 rebuild 대상은 dquot index이므로 delta tracking이 필요합니다. Link-count check는 별도 structure 대신 scan object의 attribute를 설정하므로 view delta와 commit을 하나로 합칩니다.
대상Scan source동시 update 처리
Quota usage모든 inode resource usageTransaction delta를 dquot side file에 commit
File link countDirectory referencesScanned inode attribute에 delta 직접 결합
Superblock countersFilesystem-wide state특수 incore counter 절차
Primary scan 결과와 live delta를 작은 counter 집합으로 통합합니다.
Summary Information
```````````````````
Metadata structures in this last category summarize the contents of primary
metadata records.
These are often used to speed up resource usage queries, and are many times
smaller than the primary metadata which they represent.
Examples of summary information include:
- Summary counts of free space and inodes
- File link counts from directories
- Quota resource usage counts
Check and repair require full filesystem scans, but resource and lock
acquisition follow the same paths as regular filesystem accesses.
The superblock summary counters have special requirements due to the underlying
implementation of the incore counters, and will be treated separately.
Check and repair of the other types of summary counters (quota resource counts
and file link counts) employ the same filesystem scanning and hooking
techniques as outlined above, but because the underlying data are sets of
integer counters, the staging data need not be a fully functional mirror of the
ondisk structure.
Inspiration for quota and file link count repair strategies were drawn from
sections 2.12 ("Online Index Operations") through 2.14 ("Incremental View
Maintenance") of G. Graefe, `"Concurrent Queries and Updates in Summary Views
and Their Indexes"
<http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf>`_, 2011.
Since quotas are non-negative integer counts of resource usage, online
quotacheck can use the incremental view deltas described in section 2.14 to
track pending changes to the block and inode usage counts in each transaction,
and commit those changes to a dquot side file when the transaction commits.
Delta tracking is necessary for dquots because the index builder scans inodes,
whereas the data structure being rebuilt is an index of dquots.
Link count checking combines the view deltas and commit step into one because
it sets attributes of the objects being scanned instead of writing them to a
separate data structure.
Each online fsck function will be discussed as case studies later in this
document.
기능·안전·운영 risk와 완화
561-622Online fsck 개발에서 distributor와 user에게 부적합할 수 있는 risk가 확인됐으며, 기능을 줄이는 대가로 완화할 수 있습니다.
Reverse mapping과 parent pointer 같은 metadata index는 변경을 disk에 지속하는 비용을 늘립니다. Format 시 reverse mapping을 끄면 최대 performance를 얻지만 inconsistency 발견과 repair 능력이 크게 줄어듭니다.
Software defect가 잘못된 repair를 기록할 수 있습니다. Systematic fuzz testing으로 조기에 찾되 모든 bug를 보장할 수 없으므로 kernel은 `CONFIG_XFS_ONLINE_SCRUB`, `CONFIG_XFS_ONLINE_REPAIR` Kconfig로 distributor가 기능을 끌 수 있게 합니다. Xfsprogs의 `--enable-scrub=no`는 binary만 만들지 않으므로 kernel 기능이 켜져 있으면 완전한 완화가 아닙니다.
Filesystem 손상이 너무 심하거나 겹치는 index keyspace에서 일관된 record narrative를 만들 수 없으면 repair가 실패합니다. Dirty transaction 상태의 실패를 줄이기 위해 모든 새 record를 stage·validate한 뒤 replacement를 commit합니다.
Online fsck는 raw block I/O, handle open, DAC 우회, administrative change 등 많은 privilege가 필요합니다. Systemd service는 필요한 privilege만 주지만 kernel crash·deadlock까지 막을 수는 없습니다. Cron job에는 이 hardening이 없습니다.
원문은 공개 zero-day 방식의 automated fuzz disclosure가 development process 안정성에 주는 위험도 강한 어조로 지적합니다. EXPERIMENTAL 기간의 automated testing으로 일부 위험을 앞당겨 발견하려 합니다.
Risk완화대가·한계
PerformanceReverse mapping format feature 비활성화Online repair 능력 급감
Incorrect repairKconfig·fuzz testing모든 defect 제거 보장 없음
Unrepairable damage새 record 사전 stage·validateCoherent narrative 없으면 실패
Privilege misuseSystemd sandbox·최소 capabilityKernel crash·cron은 보호 못함
Disclosure pressureEXPERIMENTAL 단계 자동 testMaintainer 대응 부담 지속
완화 수단이 기능 또는 coverage에 미치는 대가도 함께 봅니다.
Risk Management
---------------
During the development of online fsck, several risk factors were identified
that may make the feature unsuitable for certain distributors and users.
Steps can be taken to mitigate or eliminate those risks, though at a cost to
functionality.
- **Decreased performance**: Adding metadata indices to the filesystem
increases the time cost of persisting changes to disk, and the reverse space
mapping and directory parent pointers are no exception.
System administrators who require the maximum performance can disable the
reverse mapping features at format time, though this choice dramatically
reduces the ability of online fsck to find inconsistencies and repair them.
- **Incorrect repairs**: As with all software, there might be defects in the
software that result in incorrect repairs being written to the filesystem.
Systematic fuzz testing (detailed in the next section) is employed by the
authors to find bugs early, but it might not catch everything.
The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose not to
accept this risk.
The xfsprogs build system has a configure option (``--enable-scrub=no``) that
disables building of the ``xfs_scrub`` binary, though this is not a risk
mitigation if the kernel functionality remains enabled.
- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
repairable.
If the keyspaces of several metadata indices overlap in some manner but a
coherent narrative cannot be formed from records collected, then the repair
fails.
To reduce the chance that a repair will fail with a dirty transaction and
render the filesystem unusable, the online repair functions have been
designed to stage and validate all new records before committing the new
structure.
- **Misbehavior**: Online fsck requires many privileges -- raw IO to block
devices, opening files by handle, ignoring Unix discretionary access control,
and the ability to perform administrative changes.
Running this automatically in the background scares people, so the systemd
background service is configured to run with only the privileges required.
Obviously, this cannot address certain problems like the kernel crashing or
deadlocking, but it should be sufficient to prevent the scrub process from
escaping and reconfiguring the system.
The cron job does not have this protection.
- **Fuzz Kiddiez**: There are many people now who seem to think that running
automated fuzz testing of ondisk artifacts to find mischievous behavior and
spraying exploit code onto the public mailing list for instant zero-day
disclosure is somehow of some social benefit.
In the view of this author, the benefit is realized only when the fuzz
operators help to **fix** the flaws, but this opinion apparently is not
widely shared among security "researchers".
The XFS maintainers' continuing ability to manage these events presents an
ongoing risk to the stability of the development process.
Automated testing should front-load some of the risk while the feature is
considered EXPERIMENTAL.
Many of these risks are inherent to software programming.
Despite this, it is hoped that this new functionality will prove useful in
reducing unexpected downtime.
Testing plan의 세 목표와 네 범주
623-643Fsck의 세 목표는 metadata inconsistency 발견, inconsistency 제거, 추가 data loss 최소화입니다. User 신뢰를 얻으려면 software가 예상 범위 안에서 동작함을 보여야 합니다.
과거에는 정기적 exhaustive testing이 현실적이지 않았지만 저비용 VM과 high-IOPS storage가 가능하게 했습니다. Online fsck는 기존 tool과의 differential analysis, 모든 metadata object의 모든 attribute를 systematic test하는 전략을 사용하며 네 범주로 나눕니다.
범주핵심 질문
Fstests integration기존 check·repair와 결과가 일치하는가
Block fuzzing전체 metadata block 손상을 처리하는가
Record-field fuzzing모든 field 변형을 탐지·수정하는가
Stress testingLive workload와 경합해도 corruption·starvation이 없는가
기능 일치와 corruption coverage, concurrency를 함께 검증합니다.
3. Testing Plan
===============
As stated before, fsck tools have three main goals:
1. Detect inconsistencies in the metadata;
2. Eliminate those inconsistencies; and
3. Minimize further loss of data.
Demonstrations of correct operation are necessary to build users' confidence
that the software behaves within expectations.
Unfortunately, it was not really feasible to perform regular exhaustive testing
of every aspect of a fsck tool until the introduction of low-cost virtual
machines with high-IOPS storage.
With ample hardware availability in mind, the testing strategy for the online
fsck project involves differential analysis against the existing fsck tools and
systematic testing of every attribute of every type of metadata object.
Testing can be split into four major categories, as discussed below.
fstests differential integration
644-677Free-software QA는 community scale을 활용하도록 testing 비용을 낮추고 널리 실행해 filesystem configuration과 hardware setup의 breadth를 최대화해야 합니다. 그래야 online-fsck bug와 새 feature integration issue를 조기에 찾습니다.
공용 suite `fstests`는 각 test 사이에 test·scratch filesystem에 `xfs_check`와 `xfs_repair -n`을 실행해 kernel과 fsck tool이 consistent metadata 정의에 동의하는지 확인해 왔습니다. Online check 개발 중 `xfs_scrub -n`을 추가해 세 checker의 결과를 비교했습니다.
Online repair 개발을 시작할 때 test 사이마다 `xfs_repair`로 metadata index를 rebuild해 crash, 남은 corruption, online-check complaint가 없는지 확인하고 offline capability baseline을 만들었습니다. 이어 `xfs_scrub` force-rebuild mode로 online과 offline repair effectiveness를 비교합니다.
`xfs_check`와 `xfs_repair -n` 결과 일치 확인→각 test 뒤 `xfs_scrub -n` 추가→Offline `xfs_repair` rebuild baseline 수립→`xfs_scrub` force-rebuild와 offline 결과 비교
새 도구를 기존 두 도구의 결과와 단계적으로 대조합니다.
Integrated Testing with fstests
-------------------------------
The primary goal of any free software QA effort is to make testing as
inexpensive and widespread as possible to maximize the scaling advantages of
community.
In other words, testing should maximize the breadth of filesystem configuration
scenarios and hardware setups.
This improves code quality by enabling the authors of online fsck to find and
fix bugs early, and helps developers of new features to find integration
issues earlier in their development effort.
The Linux filesystem community shares a common QA testing suite,
`fstests <https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/>`_, for
functional and regression testing.
Even before development work began on online fsck, fstests (when run on XFS)
would run both the ``xfs_check`` and ``xfs_repair -n`` commands on the test and
scratch filesystems between each test.
This provides a level of assurance that the kernel and the fsck tools stay in
alignment about what constitutes consistent metadata.
During development of the online checking code, fstests was modified to run
``xfs_scrub -n`` between each test to ensure that the new checking code
produces the same results as the two existing fsck tools.
To start development of online repair, fstests was modified to run
``xfs_repair`` to rebuild the filesystem's metadata indices between tests.
This ensures that offline repair does not crash, leave a corrupt filesystem
after it exists, or trigger complaints from the online check.
This also established a baseline for what can and cannot be repaired offline.
To complete the first phase of development of online repair, fstests was
modified to be able to run ``xfs_scrub`` in a "force rebuild" mode.
This enables a comparison of the effectiveness of online repair as compared to
the existing offline repair tools.
Metadata block 단위 general fuzz
678-709XFS의 `xfs_db`와 `blocktrash`를 이용해 metadata block 전체가 손상되는 흔한 fault를 test합니다. 모든 metadata object type을 담은 filesystem을 만들고, type별 block 하나를 찾아 garbage로 덮은 뒤 validation strategy의 반응을 확인합니다.
기존 suite는 in-kernel verifier와 offline fsck의 detect·repair를 검증했고 같은 방식으로 online fsck를 확장했습니다.
Metadata object type 하나 선택→`xfs_db blocktrash`로 block에 garbage 기록→Kernel verifier가 명백히 나쁜 metadata를 차단하는지 확인→`xfs_repair`의 detect·fix 확인→`xfs_scrub`의 detect·fix 확인
Filesystem configuration마다 모든 metadata object를 반복합니다.
General Fuzz Testing of Metadata Blocks
---------------------------------------
XFS benefits greatly from having a very robust debugging tool, ``xfs_db``.
Before development of online fsck even began, a set of fstests were created
to test the rather common fault that entire metadata blocks get corrupted.
This required the creation of fstests library code that can create a filesystem
containing every possible type of metadata object.
Next, individual test cases were created to create a test filesystem, identify
a single block of a specific type of metadata object, trash it with the
existing ``blocktrash`` command in ``xfs_db``, and test the reaction of a
particular metadata validation strategy.
This earlier test suite enabled XFS developers to test the ability of the
in-kernel validation functions and the ability of the offline fsck tool to
detect and eliminate the inconsistent metadata.
This part of the test suite was extended to cover online fsck in exactly the
same manner.
In other words, for a given fstests filesystem configuration:
* For each metadata object existing on the filesystem:
* Write garbage to it
* Test the reactions of:
1. The kernel verifiers to stop obviously bad metadata
2. Offline repair (``xfs_repair``) to detect and fix
3. Online repair (``xfs_scrub``) to detect and fix
Record field별 targeted fuzz
710-773더 강한 facility는 filesystem의 모든 metadata object, record, field에 targeted fuzz를 적용합니다. `xfs_db`가 모든 metadata field를 수정할 수 있고 fstests가 모든 format을 생성하므로 memory corruption과 software bug를 systematic하게 모사할 수 있습니다.
각 bit field에 all-clear, all-set, MSB·middle·LSB toggle, 작은 값 add·subtract, randomize를 적용합니다. 그때마다 kernel verifier, `xfs_repair -n`, `xfs_repair`, `xfs_scrub -n`, `xfs_scrub`, 필요하면 online 뒤 offline repair의 반응을 test합니다.
조합 수는 폭발하지만 이 coverage는 `xfs_repair`의 잘못된 repair와 metadata class 전체의 누락을 찾았고, 구 tool 이상을 발견함을 확인해 `xfs_check` deprecation을 마무리했습니다. `xfs_scrub`도 offline과 비교하고 code deficiency를 찾는 데 같은 이점을 얻습니다.
원문은 `fuzzer-improvements`, `fuzz-baseline`, `more-fuzz-testing` 세 proposed fstests branch를 연결합니다.
변형의도
Clear/Set all최솟값·최댓값과 reserved bit 검사
Toggle MSB/middle/LSB위치별 bit corruption
Add/Subtract small quantity경계 인접 값 검사
Randomize비정형 corruption 탐색
각 변형 뒤 여섯 검사 경로를 반복합니다.
Targeted Fuzz Testing of Metadata Records
-----------------------------------------
The testing plan for online fsck includes extending the existing fs testing
infrastructure to provide a much more powerful facility: targeted fuzz testing
of every metadata field of every metadata object in the filesystem.
``xfs_db`` can modify every field of every metadata structure in every
block in the filesystem to simulate the effects of memory corruption and
software bugs.
Given that fstests already contains the ability to create a filesystem
containing every metadata format known to the filesystem, ``xfs_db`` can be
used to perform exhaustive fuzz testing!
For a given fstests filesystem configuration:
* For each metadata object existing on the filesystem...
* For each record inside that metadata object...
* For each field inside that record...
* For each conceivable type of transformation that can be applied to a bit field...
1. Clear all bits
2. Set all bits
3. Toggle the most significant bit
4. Toggle the middle bit
5. Toggle the least significant bit
6. Add a small quantity
7. Subtract a small quantity
8. Randomize the contents
* ...test the reactions of:
1. The kernel verifiers to stop obviously bad metadata
2. Offline checking (``xfs_repair -n``)
3. Offline repair (``xfs_repair``)
4. Online checking (``xfs_scrub -n``)
5. Online repair (``xfs_scrub``)
6. Both repair tools (``xfs_scrub`` and then ``xfs_repair`` if online repair doesn't succeed)
This is quite the combinatoric explosion!
Fortunately, having this much test coverage makes it easy for XFS developers to
check the responses of XFS' fsck tools.
Since the introduction of the fuzz testing framework, these tests have been
used to discover incorrect repair code and missing functionality for entire
classes of metadata objects in ``xfs_repair``.
The enhanced testing was used to finalize the deprecation of ``xfs_check`` by
confirming that ``xfs_repair`` could detect at least as many corruptions as
the older tool.
These tests have been very valuable for ``xfs_scrub`` in the same ways -- they
allow the online fsck developers to compare online fsck against offline fsck,
and they enable XFS developers to find deficiencies in the code base.
Proposed patchsets include
`general fuzzer improvements
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements>`_,
`fuzzing baselines
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_,
and `improvements in fuzz testing comprehensiveness
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing>`_.
Live workload와 scrub 경합
774-808Online fsck만의 요구는 regular workload와 동시에 동작하는 것입니다. 영향이 완전히 0일 수는 없지만 repair가 metadata inconsistency를 만들면 안 되고 workload가 resource starvation을 느껴서도 안 됩니다.
Scrub-item type마다 `fsstress` 중 check와 repair를 실행합니다. 전체 `xfs_scrub -n` 및 force-rebuild를 fsstress와 race하고, filesystem freeze/thaw 및 read-only/read-write remount 중에도 두 mode를 race합니다. 같은 test를 `fsx`로도 수행하는 항목은 원문에 아직 미완료 가능성으로 표시됩니다.
성공 기준은 corrupted metadata에 따른 unexpected shutdown, kernel hang-check warning, 기타 예상 밖 문제 없이 모든 test를 실행하는 것입니다. 원문은 mount-state race와 per-function stress refactor branch를 제안합니다.
Per-item check/repair × fsstress→Whole-filesystem check × fsstress→Force rebuild × fsstress→Check/repair × freeze·thaw→Check/repair × read-only/read-write remount→동일 matrix의 fsx variant
Scrub mode와 workload·mount state를 교차합니다.
Stress Testing
--------------
A unique requirement to online fsck is the ability to operate on a filesystem
concurrently with regular workloads.
Although it is of course impossible to run ``xfs_scrub`` with *zero* observable
impact on the running system, the online repair code should never introduce
inconsistencies into the filesystem metadata, and regular workloads should
never notice resource starvation.
To verify that these conditions are being met, fstests has been enhanced in
the following ways:
* For each scrub item type, create a test to exercise checking that item type
while running ``fsstress``.
* For each scrub item type, create a test to exercise repairing that item type
while running ``fsstress``.
* Race ``fsstress`` and ``xfs_scrub -n`` to ensure that checking the whole
filesystem doesn't cause problems.
* Race ``fsstress`` and ``xfs_scrub`` in force-rebuild mode to ensure that
force-repairing the whole filesystem doesn't cause problems.
* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
freezing and thawing the filesystem.
* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
remounting the filesystem read-only and read-write.
* The same, but running ``fsx`` instead of ``fsstress``. (Not done yet?)
Success is defined by the ability to run all of these tests without observing
any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
check warnings, or any other sort of mischief.
Proposed patchsets include `general stress testing
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_
and the `evolution of existing per-function stress testing
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_.
Foreground CLI와 media scan
809-841주 사용자는 offline repair와 마찬가지로 system administrator입니다. Interface는 요청 시 foreground CLI와 autonomous background service 두 mode입니다.
가장 최신 metadata 상태가 필요하면 `xfs_scrub`를 foreground에서 실행해 모든 metadata를 검사하고 결과를 기다립니다. `xfs_repair`와 같이 `-n`은 read-only scan, `-v`는 report verbosity 증가입니다.
새 `-x` option은 hardware error-correction capability로 data file content를 검사합니다. Runtime과 구형 storage bandwidth를 크게 쓸 수 있어 기본 enable은 아닙니다. Foreground output은 system log에 저장됩니다.
`xfs_scrub_all`은 mounted filesystem 목록을 돌며 병렬 실행하지만 같은 top-level kernel block device로 resolve되는 filesystem은 resource overconsumption을 막기 위해 직렬화합니다.
Command·option동작
xfs_scrubMounted filesystem 전체 metadata check
-nRead-only scan
-vReport verbosity 증가
-xHardware-assisted data media scan
xfs_scrub_allMount별 병렬 실행, shared top-level device는 직렬화
선택한 option이 검사 범위와 비용을 바꿉니다.
4. User Interface
=================
The primary user of online fsck is the system administrator, just like offline
repair.
Online fsck presents two modes of operation to administrators:
A foreground CLI process for online fsck on demand, and a background service
that performs autonomous checking and repair.
Checking on Demand
------------------
For administrators who want the absolute freshest information about the
metadata in a filesystem, ``xfs_scrub`` can be run as a foreground process on
a command line.
The program checks every piece of metadata in the filesystem while the
administrator waits for the results to be reported, just like the existing
``xfs_repair`` tool.
Both tools share a ``-n`` option to perform a read-only scan, and a ``-v``
option to increase the verbosity of the information reported.
A new feature of ``xfs_scrub`` is the ``-x`` option, which employs the error
correction capabilities of the hardware to check data file contents.
The media scan is not enabled by default because it may dramatically increase
program runtime and consume a lot of bandwidth on older storage hardware.
The output of a foreground invocation is captured in the system log.
The ``xfs_scrub_all`` program walks the list of mounted filesystems and
initiates ``xfs_scrub`` for each of them in parallel.
It serializes scans for any filesystems that resolve to the same top level
kernel block device to prevent resource overconsumption.
Systemd·cron 자동 service와 hardening
842-892Xfs_scrub package는 기본적으로 주말마다 online fsck를 자동 실행하는 systemd timer·service를 제공합니다. Background mode는 최소 privilege, 최저 CPU·I/O priority, CPU-constrained single-threaded mode로 동작하며 administrator가 workload latency·throughput 요구에 맞게 조정할 수 있습니다.
Output은 system log에 저장되고 `EMAIL_ADDR`를 설정하면 `xfs_scrub_fail@.service`, `xfs_scrub_media_fail@.service`, `xfs_scrub_all_fail.service`가 inconsistency 또는 runtime failure를 email로 보낼 수 있습니다.
Administrator는 systemd의 `xfs_scrub_all.timer` 또는 non-systemd의 `xfs_scrub_all.cron`을 enable합니다. 기본 weekly scan에 매달 한 번 모든 file data media scan도 추가합니다. File checksum보다는 확실하지 않지만 application integrity, 상위 redundancy, device 보장을 신뢰할 수 있다면 더 빠릅니다.
Systemd 249 기준 unit은 `systemd-analyze security` 감사를 거쳐 최소 privilege, 가능한 최대 sandbox와 syscall filtering, 필요한 filesystem-tree access만 허용합니다. CPU 한 core의 80%로 제한하고 CPU·I/O scheduling priority를 가능한 낮춰 다른 operation delay를 줄입니다. Cron job에는 같은 hardening이 없습니다.
단위역할
xfs_scrub_all.timerSystemd weekly activation
xfs_scrub_all.cronNon-systemd weekly activation
xfs_scrub_fail@.serviceMetadata scrub failure report
xfs_scrub_media_fail@.serviceMedia scan failure report
xfs_scrub_all_fail.serviceAll-filesystem run failure report
자동 실행과 failure notification의 구성 요소입니다.
Background Service
------------------
To reduce the workload of system administrators, the ``xfs_scrub`` package
provides a suite of `systemd <https://systemd.io/>`_ timers and services that
run online fsck automatically on weekends by default.
The background service configures scrub to run with as little privilege as
possible, the lowest CPU and IO priority, and in a CPU-constrained single
threaded mode.
This can be tuned by the systemd administrator at any time to suit the latency
and throughput requirements of customer workloads.
The output of the background service is also captured in the system log.
If desired, reports of failures (either due to inconsistencies or mere runtime
errors) can be emailed automatically by setting the ``EMAIL_ADDR`` environment
variable in the following service files:
* ``xfs_scrub_fail@.service``
* ``xfs_scrub_media_fail@.service``
* ``xfs_scrub_all_fail.service``
The decision to enable the background scan is left to the system administrator.
This can be done by enabling either of the following services:
* ``xfs_scrub_all.timer`` on systemd systems
* ``xfs_scrub_all.cron`` on non-systemd systems
This automatic weekly scan is configured out of the box to perform an
additional media scan of all file data once per month.
This is less foolproof than, say, storing file data block checksums, but much
more performant if application software provides its own integrity checking,
redundancy can be provided elsewhere above the filesystem, or the storage
device's integrity guarantees are deemed sufficient.
The systemd unit file definitions have been subjected to a security audit
(as of systemd 249) to ensure that the xfs_scrub processes have as little
access to the rest of the system as possible.
This was performed via ``systemd-analyze security``, after which privileges
were restricted to the minimum required, sandboxing was set up to the maximal
extent possible with sandboxing and system call filtering; and access to the
filesystem tree was restricted to the minimum needed to start the program and
access the filesystem being scanned.
The service definition files restrict CPU usage to 80% of one CPU core, and
apply as nice of a priority to IO and CPU scheduling as possible.
This measure was taken to minimize delays in the rest of the filesystem.
No such hardening has been performed for the cron job.
Proposed patchset:
`Enabling the xfs_scrub background service
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_.
In-memory health summary와 maintenance 결정
893-921XFS는 filesystem별 health summary를 memory에 cache합니다. `xfs_scrub` 실행 또는 일반 operation 중 metadata inconsistency 발견 때 갱신됩니다.
Administrator는 `xfs_spaceman health` command로 human-readable report를 얻습니다. 문제가 있으면 짧은 service window에 online repair를 실행하고, 실패하면 maintenance window를 정해 전통적 offline `xfs_repair`를 실행할 수 있습니다.
Future-work 질문은 health report를 inotify filesystem-error notification과 연결하고 corruption notification daemon이 repair를 시작할지입니다. 답은 아직 없으며 early adopter와 downstream user와 논의해야 합니다.
원문은 correction return을 health report에 연결하는 `corruption-health-reports`와 memory reclaim 중 sickness 정보를 유지하는 `indirect-health-reporting` branch를 제시합니다.
Runtime 또는 xfs_scrub가 sickness 상태 갱신→`xfs_spaceman health`로 report 조회→Reduced service window에서 online repair→해결되지 않으면 maintenance window와 offline xfs_repair
상태 관찰과 repair 강도를 단계적으로 높입니다.
Health Reporting
----------------
XFS caches a summary of each filesystem's health status in memory.
The information is updated whenever ``xfs_scrub`` is run, or whenever
inconsistencies are detected in the filesystem metadata during regular
operations.
System administrators should use the ``health`` command of ``xfs_spaceman`` to
download this information into a human-readable format.
If problems have been observed, the administrator can schedule a reduced
service window to run the online repair tool to correct the problem.
Failing that, the administrator can decide to schedule a maintenance window to
run the traditional offline repair tool to correct the problem.
**Future Work Question**: Should the health reporting integrate with the new
inotify fs error notification system?
Would it be helpful for sysadmins to have a daemon to listen for corruption
notifications and initiate a repair?
*Answer*: These questions remain unanswered, but should be a part of the
conversation with early adopters and potential downstream users of XFS.
Proposed patchsets include
`wiring up health reports to correction returns
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports>`_
and
`preservation of sickness info during memory reclaim
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_.
Kernel algorithm과 재생성 mechanism
922-932이 절에서는 시스템이 실행 중인 동안 메타데이터를 검사하고 복구할 수 있게 하는 커널 코드의 핵심 알고리즘과 자료 구조를 설명합니다. 이 절의 앞쪽 장들은 메타데이터 검사의 토대를 이루는 구성 요소를 밝힙니다. 나머지 부분은 XFS가 스스로를 재생성하는 메커니즘을 제시합니다.
Self-describing metadata와 buffer verifier→Reverse mapping과 cross-reference→Consistency·concurrency coordination→Metadata staging·rebuild·atomic commit
검증 기반 위에 replacement 구조 생성이 놓입니다.
5. Kernel Algorithms and Data Structures
========================================
This section discusses the key algorithms and data structures of the kernel
code that provide the ability to check and repair metadata while the system
is running.
The first chapters in this section reveal the pieces that provide the
foundation for checking metadata.
The remainder of this section presents the mechanisms through which XFS
regenerates itself.
Self-describing block header와 verifier 범위
933-9662012년에 도입된 XFS 버전 5부터 XFS는 거의 모든 온디스크 블록 헤더의 형식을 갱신하여 매직 넘버, 체크섬, 범용 "고유" 식별자(UUID), 소유자 코드, 블록의 온디스크 주소, 로그 시퀀스 번호를 기록합니다. 디스크에서 블록 버퍼를 읽을 때 매직 넘버, UUID, 소유자, 온디스크 주소를 확인하면 가져온 블록이 현재 파일시스템의 특정 소유자와 일치하는지, 블록에 든 정보가 해당 온디스크 주소에 있어야 하는 정보인지 판별할 수 있습니다. 앞의 세 구성 요소 덕분에 검사 도구는 그 파일시스템에 속하지 않는 메타데이터로 추정되는 내용을 무시할 수 있고, 네 번째 구성 요소 덕분에 파일시스템은 쓰기 유실을 탐지할 수 있습니다.
파일시스템 작업이 블록을 수정할 때마다 그 변경은 트랜잭션의 일부로 로그에 제출됩니다. 로그는 이 트랜잭션들을 처리하고, 저장 장치에 안전하게 영속화되면 완료로 표시합니다. 로깅 코드는 체크섬과 마지막 트랜잭션 갱신의 로그 시퀀스 번호를 유지합니다. 체크섬은 컴퓨터와 저장 장치 사이에서 생길 수 있는 분할 쓰기와 그 밖의 불일치를 탐지하는 데 유용합니다. 시퀀스 번호 추적을 통해 로그 복구는 오래되어 유효하지 않은 로그 갱신을 파일시스템에 적용하지 않을 수 있습니다.
이 두 기능은 파일시스템이 디스크에서 메타데이터 블록을 읽을 때 명백한 손상을 탐지할 수단을 제공하여 전체적인 런타임 복원력을 높입니다. 그러나 이러한 버퍼 검증기는 메타데이터 구조 사이의 일관성을 검사할 수 없습니다.
자세한 내용은 `Documentation/filesystems/xfs/xfs-self-describing-metadata.rst` 문서를 참조하십시오.
Field확인 대상
Magic numberMetadata block type
UUID현재 filesystem 소속
Owner code요청한 structure의 소유
On-disk address읽은 위치와 기록 위치, lost write
ChecksumTorn write·전송/저장 discrepancy
LSN마지막 update 순서와 stale replay 방지
각 field가 담당하는 identity와 durability 검사를 구분합니다.
Self Describing Metadata
------------------------
Starting with XFS version 5 in 2012, XFS updated the format of nearly every
ondisk block header to record a magic number, a checksum, a universally
"unique" identifier (UUID), an owner code, the ondisk address of the block,
and a log sequence number.
When loading a block buffer from disk, the magic number, UUID, owner, and
ondisk address confirm that the retrieved block matches the specific owner of
the current filesystem, and that the information contained in the block is
supposed to be found at the ondisk address.
The first three components enable checking tools to disregard alleged metadata
that doesn't belong to the filesystem, and the fourth component enables the
filesystem to detect lost writes.
Whenever a file system operation modifies a block, the change is submitted
to the log as part of a transaction.
The log then processes these transactions marking them done once they are
safely persisted to storage.
The logging code maintains the checksum and the log sequence number of the last
transactional update.
Checksums are useful for detecting torn writes and other discrepancies that can
be introduced between the computer and its storage devices.
Sequence number tracking enables log recovery to avoid applying out of date
log updates to the filesystem.
These two features improve overall runtime resiliency by providing a means for
the filesystem to detect obvious corruption when reading metadata blocks from
disk, but these buffer verifiers cannot provide any consistency checking
between metadata structures.
For more information, please see the documentation for
Documentation/filesystems/xfs/xfs-self-describing-metadata.rst
공간을 owner로 되돌리는 secondary index
967-1079XFS의 원래 설계(1993년 무렵)는 1980년대 Unix 파일시스템 설계를 개선한 것입니다. 당시에는 저장 밀도를 높이는 비용이 컸고 CPU 시간은 부족했으며, 탐색 시간이 지나치게 길면 성능이 무너질 수 있었습니다. 성능상의 이유로 파일시스템 개발자들은 데이터 무결성을 희생하는 한이 있더라도 파일시스템에 중복 정보를 추가하기를 꺼렸습니다. 21세기 초의 파일시스템 설계자들은 내부 중복성을 높이기 위해 거의 동일한 메타데이터 사본을 저장하거나 공간 효율이 더 높은 인코딩 기법을 쓰는 등 다른 전략을 선택합니다.
XFS는 설계를 현대화하기 위해 이와 다른 중복성 전략을 선택했습니다. 할당된 디스크 익스텐트를 그 소유자에게 역으로 대응시키는 보조 공간 사용량 인덱스를 추가한 것입니다. 새 인덱스를 추가해도 디렉터리 트리, 파일 블록 맵, 할당 그룹과 같은 기본 파일 메타데이터는 바뀌지 않으므로, 대규모 데이터셋을 다루는 스레드가 많은 워크로드에서도 파일시스템의 확장성을 대부분 유지할 수 있습니다. 중복성을 높이는 모든 시스템과 마찬가지로 reverse mapping 기능은 공간 매핑 작업의 오버헤드를 증가시킵니다. 하지만 여기에는 두 가지 중요한 장점이 있습니다. 첫째, reverse index는 online fsck를 가능하게 하는 핵심 요소이며, 여유 공간 조각 모음, 더 정확한 매체 장애 보고, 파일시스템 축소처럼 요청되어 온 다른 기능의 기반이기도 합니다. 둘째, reverse mapping btree는 온디스크 저장 형식이 다르므로 파일시스템에 필요한 실질적인 중복성이 장치 수준 중복 제거에 의해 사라지지 않습니다.
논점설명
비판보조 인덱스를 추가해도 사용자 데이터 저장 자체의 견고성은 향상되지 않습니다.
데이터 체크섬의 비용파일 데이터 블록 체크섬용 인덱스를 새로 만들면 데이터 덮어쓰기가 copy-write로 바뀌어 쓰기 증폭이 커지고 파일시스템이 일찍 노화합니다.
사용자 데이터 정책30년 동안 이어진 관례에 따라 파일 데이터 무결성이 필요한 사용자는 요구 수준에 맞는 강력한 해결책을 제공할 수 있습니다.
메타데이터와 저장 계층공간 사용량 보조 인덱스를 추가하는 복잡성은 XFS 자체에 볼륨 관리와 저장 장치 미러링을 추가하는 것보다 훨씬 작습니다. RAID와 볼륨 관리의 고도화는 커널의 기존 계층에 맡기는 편이 좋습니다.
원문의 ASCII 상자를 내용이 같은 구조화 표로 옮겼습니다.
Reverse space mapping 레코드가 담는 정보는 다음과 같습니다.
struct xfs_rmap_irec {
xfs_agblock_t rm_startblock; /* extent start block */
xfs_extlen_t rm_blockcount; /* extent length */
uint64_t rm_owner; /* extent owner */
uint64_t rm_offset; /* offset within the owner */
unsigned int rm_flags; /* state flags */
};앞의 두 필드는 파일시스템 블록 단위로 물리 공간의 위치와 크기를 기록합니다. 소유자 필드는 이 공간이 어느 메타데이터 구조 또는 파일 inode에 할당되었는지를 scrub에 알려 줍니다. 파일에 할당된 공간의 경우 offset 필드는 그 공간이 파일 fork 안의 어디에 매핑되었는지를 알려 줍니다. 마지막으로 flags 필드는 공간 사용 방식에 관한 추가 정보를 제공합니다. 즉, 이 익스텐트가 attribute fork 익스텐트인지, 파일 mapping btree 익스텐트인지, 아니면 unwritten data 익스텐트인지를 나타냅니다.
필드원문 주석과 의미
rm_startblock`extent start block`: 물리 익스텐트의 시작 블록
rm_blockcount`extent length`: 익스텐트 길이
rm_owner`extent owner`: 메타데이터 구조 또는 파일 inode
rm_offset`offset within the owner`: 소유자 file fork 안의 매핑 위치
rm_flags`state flags`: attribute fork·mapping btree·unwritten data 상태
물리 익스텐트를 소유자의 논리 위치와 상태에 연결합니다.
Online filesystem checking은 각 기본 메타데이터 레코드의 정보를 다른 모든 공간 인덱스와 비교하여 그 일관성을 판단합니다. Reverse mapping index에는 모든 공간 할당 정보의 중앙화된 대체 사본이 들어 있으므로 일관성 검사 과정에서 핵심 역할을 합니다. Online checking이 무엇을 참조할 수 있는지를 실질적으로 제한하는 것은 프로그램 실행 시간과 자원 획득의 용이성뿐입니다. 예를 들어 파일 데이터 익스텐트 매핑은 다음 항목과 대조할 수 있습니다.
여유 공간 정보에 해당 entry가 없어야 합니다.→Inode index에 해당 entry가 없어야 합니다.→파일이 shared extent를 가진 것으로 표시되지 않았다면 reference count 데이터에 해당 entry가 없어야 합니다.→Reverse mapping 정보에는 대응하는 entry가 있어야 합니다.
한 mapping을 네 space view와 비교합니다.
Reverse mapping index에 관해서는 몇 가지 사항을 짚을 수 있습니다.
1. 위의 기본 메타데이터 중 어느 하나라도 의심스러울 때 reverse mapping은 그 내용이 올바르다는 적극적인 확인 근거를 제공할 수 있습니다. 대부분의 기본 메타데이터 검사 코드는 위에서 설명한 것과 비슷한 경로를 따릅니다.
2. 보조 메타데이터와 기본 메타데이터의 일관성을 입증하기는 어렵습니다. 이를 위해서는 모든 기본 공간 메타데이터를 전부 스캔해야 하며, 시간이 매우 오래 걸리기 때문입니다. 예를 들어 파일 익스텐트 mapping btree 블록에 대한 reverse mapping 레코드를 검사하려면 파일을 잠그고 btree 전체를 검색해 해당 블록을 확인해야 합니다. 따라서 scrub은 기본 공간 매핑 구조를 검사할 때 엄격하게 cross-reference하는 방식에 의존합니다.
3. 필요한 잠금 순서가 일반 파일시스템 작업에서 사용하는 순서와 다르다면 일관성 스캔은 non-blocking lock acquisition primitive를 사용해야 합니다. 예를 들어 파일시스템이 평소에는 file ILOCK을 획득한 다음 AGF buffer lock을 획득하는데, scrub이 AGF buffer lock을 보유한 상태에서 file ILOCK을 획득하려 한다면 scrub은 두 번째 잠금 획득을 기다리며 block할 수 없습니다. 이는 시스템 부하가 높을 때 reverse mapping 데이터를 스캔하는 이 구간의 forward progress를 보장할 수 없다는 뜻입니다.
번호핵심
1의심스러운 기본 메타데이터를 적극적으로 확인하는 근거가 됩니다.
2보조 레코드 하나를 기본 메타데이터로 입증하려면 전체 스캔·파일 잠금·btree 검색이 필요할 수 있으므로, 기본 구조 검사 중 엄격한 cross-reference를 수행합니다.
3잠금 순서가 일반 경로와 다르면 non-blocking 획득만 사용해야 하므로 높은 부하에서는 진행을 보장할 수 없습니다.
기본 메타데이터와 보조 인덱스를 검증하는 비용과 잠금 제약을 구분합니다.
요약하면 reverse mapping은 기본 메타데이터를 재구성할 때 핵심적인 역할을 합니다. 이러한 레코드를 staging하고 디스크에 기록하며 파일시스템에 commit하는 방법의 세부 사항은 뒤 절에서 다룹니다.
Reverse Mapping
---------------
The original design of XFS (circa 1993) is an improvement upon 1980s Unix
filesystem design.
In those days, storage density was expensive, CPU time was scarce, and
excessive seek time could kill performance.
For performance reasons, filesystem authors were reluctant to add redundancy to
the filesystem, even at the cost of data integrity.
Filesystems designers in the early 21st century choose different strategies to
increase internal redundancy -- either storing nearly identical copies of
metadata, or more space-efficient encoding techniques.
For XFS, a different redundancy strategy was chosen to modernize the design:
a secondary space usage index that maps allocated disk extents back to their
owners.
By adding a new index, the filesystem retains most of its ability to scale
well to heavily threaded workloads involving large datasets, since the primary
file metadata (the directory tree, the file block map, and the allocation
groups) remain unchanged.
Like any system that improves redundancy, the reverse-mapping feature increases
overhead costs for space mapping activities.
However, it has two critical advantages: first, the reverse index is key to
enabling online fsck and other requested functionality such as free space
defragmentation, better media failure reporting, and filesystem shrinking.
Second, the different ondisk storage format of the reverse mapping btree
defeats device-level deduplication because the filesystem requires real
redundancy.
+--------------------------------------------------------------------------+
| **Sidebar**: |
+--------------------------------------------------------------------------+
| A criticism of adding the secondary index is that it does nothing to |
| improve the robustness of user data storage itself. |
| This is a valid point, but adding a new index for file data block |
| checksums increases write amplification by turning data overwrites into |
| copy-writes, which age the filesystem prematurely. |
| In keeping with thirty years of precedent, users who want file data |
| integrity can supply as powerful a solution as they require. |
| As for metadata, the complexity of adding a new secondary index of space |
| usage is much less than adding volume management and storage device |
| mirroring to XFS itself. |
| Perfection of RAID and volume management are best left to existing |
| layers in the kernel. |
+--------------------------------------------------------------------------+
The information captured in a reverse space mapping record is as follows:
.. code-block:: c
struct xfs_rmap_irec {
xfs_agblock_t rm_startblock; /* extent start block */
xfs_extlen_t rm_blockcount; /* extent length */
uint64_t rm_owner; /* extent owner */
uint64_t rm_offset; /* offset within the owner */
unsigned int rm_flags; /* state flags */
};
The first two fields capture the location and size of the physical space,
in units of filesystem blocks.
The owner field tells scrub which metadata structure or file inode have been
assigned this space.
For space allocated to files, the offset field tells scrub where the space was
mapped within the file fork.
Finally, the flags field provides extra information about the space usage --
is this an attribute fork extent? A file mapping btree extent? Or an
unwritten data extent?
Online filesystem checking judges the consistency of each primary metadata
record by comparing its information against all other space indices.
The reverse mapping index plays a key role in the consistency checking process
because it contains a centralized alternate copy of all space allocation
information.
Program runtime and ease of resource acquisition are the only real limits to
what online checking can consult.
For example, a file data extent mapping can be checked against:
* The absence of an entry in the free space information.
* The absence of an entry in the inode index.
* The absence of an entry in the reference count data if the file is not
marked as having shared extents.
* The correspondence of an entry in the reverse mapping information.
There are several observations to make about reverse mapping indices:
1. Reverse mappings can provide a positive affirmation of correctness if any of
the above primary metadata are in doubt.
The checking code for most primary metadata follows a path similar to the
one outlined above.
2. Proving the consistency of secondary metadata with the primary metadata is
difficult because that requires a full scan of all primary space metadata,
which is very time intensive.
For example, checking a reverse mapping record for a file extent mapping
btree block requires locking the file and searching the entire btree to
confirm the block.
Instead, scrub relies on rigorous cross-referencing during the primary space
mapping structure checks.
3. Consistency scans must use non-blocking lock acquisition primitives if the
required locking order is not the same order used by regular filesystem
operations.
For example, if the filesystem normally takes a file ILOCK before taking
the AGF buffer lock but scrub wants to take a file ILOCK while holding
an AGF buffer lock, scrub cannot block on that second acquisition.
This means that forward progress during this part of a scan of the reverse
mapping data cannot be guaranteed if system load is heavy.
In summary, reverse mappings play a key role in reconstruction of primary
metadata.
The details of how these records are staged, written to disk, and committed
into the filesystem are covered in subsequent sections.
Checking outcome의 다섯 flag
1080-1102메타데이터 구조를 검사하는 첫 단계는 구조 안에 든 모든 레코드와 각 레코드가 시스템의 나머지 부분과 맺는 관계를 조사하는 것입니다. XFS에는 일관되지 않은 메타데이터가 시스템에 큰 피해를 주지 못하도록 여러 검사 계층이 있습니다. 각 계층은 커널이 메타데이터 구조의 상태에 관해 판단할 때 도움이 되는 정보를 제공합니다.
커널이 판단할 질문Flag
이 구조의 일부가 명백히 손상되었는가?XFS_SCRUB_OFLAG_CORRUPT
이 구조가 시스템의 나머지 부분과 일관되지 않은가?XFS_SCRUB_OFLAG_XCORRUPT
파일시스템 주변의 손상이 너무 심해 cross-reference할 수 없는가?XFS_SCRUB_OFLAG_XFAIL
성능을 높이거나 메타데이터 크기를 줄이도록 구조를 최적화할 수 있는가?XFS_SCRUB_OFLAG_PREEN
일관성 위반은 아니지만 시스템 관리자가 검토해야 할 데이터가 구조에 들어 있는가?XFS_SCRUB_OFLAG_WARNING
원문의 질문과 결과 flag를 일대일로 보존합니다.
다음 절에서는 메타데이터 scrubbing 과정이 어떻게 동작하는지 설명합니다.
Checking and Cross-Referencing
------------------------------
The first step of checking a metadata structure is to examine every record
contained within the structure and its relationship with the rest of the
system.
XFS contains multiple layers of checking to try to prevent inconsistent
metadata from wreaking havoc on the system.
Each of these layers contributes information that helps the kernel to make
three decisions about the health of a metadata structure:
- Is a part of this structure obviously corrupt (``XFS_SCRUB_OFLAG_CORRUPT``) ?
- Is this structure inconsistent with the rest of the system
(``XFS_SCRUB_OFLAG_XCORRUPT``) ?
- Is there so much damage around the filesystem that cross-referencing is not
possible (``XFS_SCRUB_OFLAG_XFAIL``) ?
- Can the structure be optimized to improve performance or reduce the size of
metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
- Does the structure contain data that is not inconsistent but deserves review
by the system administrator (``XFS_SCRUB_OFLAG_WARNING``) ?
The following sections describe how the metadata scrubbing process works.
Buffer cache의 저비용 metadata verifier
1103-1138XFS에서 가장 낮은 메타데이터 보호 계층은 buffer cache에 내장된 metadata verifier입니다. 이 함수들은 블록 자체의 내부 일관성을 저렴한 비용으로 검사하고 다음 질문에 답합니다.
번호질문
1이 블록은 이 파일시스템에 속하는가?
2이 블록은 읽기를 요청한 구조에 속하는가? 이는 메타데이터 블록의 소유자가 하나뿐이라고 가정하며, XFS에서는 항상 참입니다.
3블록에 저장된 데이터 형식이 scrub이 예상하는 합리적인 범위 안에 있는가?
4블록의 물리 위치가 실제로 읽어 온 위치와 일치하는가?
5블록 체크섬이 데이터와 일치하는가?
블록 하나에서 확인하는 소속·형식·위치·무결성 질문을 원문 순서대로 보존합니다.
이 계층의 보호 범위는 매우 제한적입니다. Verifier는 파일시스템 코드에 심각한 손상 버그가 상당한 정도로 없고 저장 시스템이 데이터를 가져오는 작업을 상당한 정도로 제대로 수행한다는 사실만 확인할 수 있습니다. 런타임에 발견된 손상 문제는 상태 보고와 시스템 호출 실패를 일으키며, 손상된 메타데이터 때문에 dirty transaction을 취소해야 하는 극단적인 경우에는 파일시스템을 shutdown할 수 있습니다.
모든 online fsck scrubbing 함수는 구조를 검사하는 동안 그 구조의 온디스크 메타데이터 블록을 모두 읽어야 합니다. 검사 중 발견된 손상은 즉시 손상으로 userspace에 보고합니다. Cross-reference 중 발견된 손상은 전체 조사가 끝난 뒤 cross-reference 실패로 보고합니다. 이미 cache에 있어 앞서 검증된 버퍼로 충족되는 읽기는 이러한 검사를 건너뜁니다.
Metadata Buffer Verification
````````````````````````````
The lowest layer of metadata protection in XFS are the metadata verifiers built
into the buffer cache.
These functions perform inexpensive internal consistency checking of the block
itself, and answer these questions:
- Does the block belong to this filesystem?
- Does the block belong to the structure that asked for the read?
This assumes that metadata blocks only have one owner, which is always true
in XFS.
- Is the type of data stored in the block within a reasonable range of what
scrub is expecting?
- Does the physical location of the block match the location it was read from?
- Does the block checksum match the data?
The scope of the protections here are very limited -- verifiers can only
establish that the filesystem code is reasonably free of gross corruption bugs
and that the storage system is reasonably competent at retrieval.
Corruption problems observed at runtime cause the generation of health reports,
failed system calls, and in the extreme case, filesystem shutdowns if the
corrupt metadata force the cancellation of a dirty transaction.
Every online fsck scrubbing function is expected to read every ondisk metadata
block of a structure in the course of checking the structure.
Corruption problems observed during a check are immediately reported to
userspace as corruption; during a cross-reference, they are reported as a
failure to cross-reference once the full examination is complete.
Reads satisfied by a buffer already in cache (and hence already verified)
bypass these checks.
Block 내부 record의 엄격한 검증
1139-1173Buffer cache 다음의 메타데이터 보호 계층은 파일시스템에 내장된 내부 레코드 검증 코드입니다. 이 검사는 필요한 상위 수준 문맥의 양에 따라 buffer verifier, 파일시스템 내부의 buffer-cache 사용자, scrub 코드 자체에 나뉘어 있습니다. 검사 범위는 여전히 블록 내부에 한정됩니다. 이 상위 수준 검사 함수는 다음 질문에 답합니다.
번호질문
1블록에 저장된 데이터 형식이 scrub이 예상한 형식과 일치하는가?
2블록이 읽기를 요청한 소유 구조에 속하는가?
3블록에 레코드가 있다면 그 레코드들이 블록 안에 들어맞는가?
4블록이 내부 여유 공간 정보를 추적한다면 그 정보가 레코드 영역과 일관되는가?
5블록 안에 든 레코드에 명백한 손상이 없는가?
블록 내부 layout과 record 자체의 타당성을 원문 순서대로 확인합니다.
이 범주의 레코드 검사는 더 엄격하며 시간도 더 많이 듭니다. 예를 들어 block pointer와 inumber가 allocation group 및 파일시스템에서 동적으로 할당되는 범위 안을 가리키는지 확인합니다. 이름에는 잘못된 문자가 없는지, flag에는 유효하지 않은 조합이 없는지 검사합니다. 그 밖의 레코드 attribute도 값이 합리적인지 확인합니다. Btree keyspace의 한 구간에 걸친 btree 레코드는 순서가 올바르고 서로 병합할 수 없는지 검사하며, file fork mapping은 이 병합 가능성 검사에서 제외합니다. 성능상의 이유로 일반 코드는 debugging이 활성화되어 있거나 곧 write가 일어날 상황이 아니면 이러한 검사 일부를 생략할 수 있습니다. 물론 scrub 함수는 발생 가능한 문제를 모두 검사해야 합니다.
Internal Consistency Checks
```````````````````````````
After the buffer cache, the next level of metadata protection is the internal
record verification code built into the filesystem.
These checks are split between the buffer verifiers, the in-filesystem users of
the buffer cache, and the scrub code itself, depending on the amount of higher
level context required.
The scope of checking is still internal to the block.
These higher level checking functions answer these questions:
- Does the type of data stored in the block match what scrub is expecting?
- Does the block belong to the owning structure that asked for the read?
- If the block contains records, do the records fit within the block?
- If the block tracks internal free space information, is it consistent with
the record areas?
- Are the records contained inside the block free of obvious corruptions?
Record checks in this category are more rigorous and more time-intensive.
For example, block pointers and inumbers are checked to ensure that they point
within the dynamically allocated parts of an allocation group and within
the filesystem.
Names are checked for invalid characters, and flags are checked for invalid
combinations.
Other record attributes are checked for sensible values.
Btree records spanning an interval of the btree keyspace are checked for
correct order and lack of mergeability (except for file fork mappings).
For performance reasons, regular code may skip some of these checks unless
debugging is enabled or a write is about to occur.
Scrub functions, of course, must check all possible problems.
Userspace-controlled 값의 검증 한계
1174-1195파일시스템 메타데이터의 여러 부분은 userspace가 직접 제어합니다. 이러한 특성 때문에 검증 작업은 값이 가능한 범위 안에 있는지 확인하는 것보다 더 정밀할 수 없습니다. 해당 필드는 다음과 같습니다.
번호필드
1Mount option이 제어하는 superblock field
2Filesystem label
3File timestamp
4File permission
5File size
6File flag
7Directory entry, extended attribute key, filesystem label에 나타나는 이름
8Extended attribute key namespace
9Extended attribute value
10File data block content
11Quota limit
12Resource usage가 soft limit를 넘었을 때의 quota timer expiration
원문 항목을 합치지 않고 각각 보존합니다.
Validation of Userspace-Controlled Record Attributes
````````````````````````````````````````````````````
Various pieces of filesystem metadata are directly controlled by userspace.
Because of this nature, validation work cannot be more precise than checking
that a value is within the possible range.
These fields include:
- Superblock fields controlled by mount options
- Filesystem labels
- File timestamps
- File permissions
- File size
- File flags
- Names present in directory entries, extended attribute keys, and filesystem
labels
- Extended attribute key namespaces
- Extended attribute values
- File data block contents
- Quota limits
- Quota timer expiration (if resource usage exceeds the soft limit)
Space metadata의 계층적 cross-reference
1196-1327블록 내부 검사가 끝나면 그보다 한 단계 높은 검사로서 메타데이터 구조 사이의 레코드를 cross-reference합니다. 일반 런타임 코드에서 이 검사의 비용은 감당하기 어려울 정도로 비싸다고 여겨집니다. 그러나 scrub은 불일치를 뿌리 뽑는 데 전념하므로 조사할 수 있는 모든 경로를 따라가야 합니다. 정확히 어떤 항목을 cross-reference할지는 검사 중인 자료 구조의 문맥에 크게 좌우됩니다.
XFS btree 코드에는 online fsck가 한 구조를 다른 구조와 cross-reference할 때 사용하는 keyspace scanning 함수가 있습니다. 구체적으로 scrub은 인덱스의 keyspace를 스캔하여 그 keyspace가 레코드에 완전히 매핑되어 있는지, 드문드문 매핑되어 있는지, 전혀 매핑되지 않았는지를 판단할 수 있습니다. Reverse mapping btree에서는 keyspace scan을 수행할 때 key의 일부를 mask할 수 있습니다. 그러면 나머지 rmap keyspace가 희소하다는 특성에 방해받지 않고, rmap btree에 특정 물리 공간 익스텐트를 매핑하는 레코드가 있는지 판단할 수 있습니다.
Btree 블록은 cross-reference에 들어가기 전에 다음 검사를 거칩니다.
번호검사 질문
1블록에 저장된 데이터 형식이 scrub이 예상한 형식과 일치하는가?
2블록이 읽기를 요청한 소유 구조에 속하는가?
3레코드가 블록 안에 들어맞는가?
4블록 안의 레코드에 명백한 손상이 없는가?
5Name hash가 올바른 순서로 정렬되어 있는가?
6Btree 안의 node pointer가 해당 btree 형식에 유효한 블록 주소를 가리키는가?
7Child pointer가 leaf 방향을 가리키는가?
8Sibling pointer가 같은 level을 가로질러 가리키는가?
9각 node block record에서 record key가 child block의 내용을 정확히 반영하는가?
원문의 아홉 질문을 순서대로 보존합니다.
공간 할당 레코드는 다음과 같이 cross-reference합니다.
Class상속 검사추가 검사
1. 모든 메타데이터 구조가 언급하는 공간없음Reverse mapping index가 각 블록의 소유자로 올바른 소유자만 기록하는가? 어느 블록도 여유 공간으로 주장되지 않는가? File data block이 아니라면 어느 블록도 서로 다른 소유자가 공유하는 공간으로 주장되지 않는가?
2. Btree blockClass 1의 모든 검사Parent node block이 있으면 이 블록에 열거된 key가 이 블록의 keyspace와 일치하는가? Sibling pointer가 같은 level의 유효한 블록을 가리키는가? Child pointer가 바로 아래 level의 유효한 블록을 가리키는가?
3. Free space btree recordClass 1과 2의 모든 검사Reverse mapping index가 이 공간의 소유자를 하나도 기록하지 않는가? Inode index가 이 공간을 inode용으로 주장하지 않는가? Reference count index가 이 공간을 언급하지 않는가? 다른 free space btree에 대응 레코드가 있는가?
4. Inode btree recordClass 1과 2의 모든 검사Free inode btree에 대응 레코드가 있는가? Holemask에서 clear된 bit가 inode cluster와 대응하는가? Freemask에서 set된 bit가 link count 0인 inode record와 대응하는가?
5. Inode recordClass 1의 모든 검사File fork 정보를 요약하는 모든 field가 실제 fork와 일치하는가? Link count 0인 각 inode가 free inode btree의 레코드와 대응하는가?
6. File fork space mapping recordClass 1과 2의 모든 검사Inode btree가 이 공간을 언급하지 않는가? CoW fork mapping이라면 reference count btree의 CoW entry와 대응하는가?
7. Reference count recordClass 1과 2의 모든 검사Rmap btree의 space subkeyspace, 즉 소유자 정보를 무시하고 특정 공간 익스텐트에 매핑된 모든 레코드 안에서 각 블록의 reverse mapping record 수가 reference count record가 주장하는 수와 같은가?
각 class가 상속하는 검사와 추가 검사를 분리해 원문의 중첩 관계를 유지합니다.
제안된 patchset은 복구를 시작하기 전에 다음 문제를 탐지하거나 cross-reference를 강화하는 일련의 변경입니다.
목적Link
Refcount btree record의 gap 탐지https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps
Inode btree record의 gap 탐지https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps
Rmap btree record의 gap 탐지https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps
병합할 수 있는 record 탐지https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records
Rmap을 이용한 cross-reference 강화https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking
원문의 link와 branch 이름을 그대로 보존합니다.
Cross-Referencing Space Metadata
````````````````````````````````
After internal block checks, the next higher level of checking is
cross-referencing records between metadata structures.
For regular runtime code, the cost of these checks is considered to be
prohibitively expensive, but as scrub is dedicated to rooting out
inconsistencies, it must pursue all avenues of inquiry.
The exact set of cross-referencing is highly dependent on the context of the
data structure being checked.
The XFS btree code has keyspace scanning functions that online fsck uses to
cross reference one structure with another.
Specifically, scrub can scan the key space of an index to determine if that
keyspace is fully, sparsely, or not at all mapped to records.
For the reverse mapping btree, it is possible to mask parts of the key for the
purposes of performing a keyspace scan so that scrub can decide if the rmap
btree contains records mapping a certain extent of physical space without the
sparsenses of the rest of the rmap keyspace getting in the way.
Btree blocks undergo the following checks before cross-referencing:
- Does the type of data stored in the block match what scrub is expecting?
- Does the block belong to the owning structure that asked for the read?
- Do the records fit within the block?
- Are the records contained inside the block free of obvious corruptions?
- Are the name hashes in the correct order?
- Do node pointers within the btree point to valid block addresses for the type
of btree?
- Do child pointers point towards the leaves?
- Do sibling pointers point across the same level?
- For each node block record, does the record key accurate reflect the contents
of the child block?
Space allocation records are cross-referenced as follows:
1. Any space mentioned by any metadata structure are cross-referenced as
follows:
- Does the reverse mapping index list only the appropriate owner as the
owner of each block?
- Are none of the blocks claimed as free space?
- If these aren't file data blocks, are none of the blocks claimed as space
shared by different owners?
2. Btree blocks are cross-referenced as follows:
- Everything in class 1 above.
- If there's a parent node block, do the keys listed for this block match the
keyspace of this block?
- Do the sibling pointers point to valid blocks? Of the same level?
- Do the child pointers point to valid blocks? Of the next level down?
3. Free space btree records are cross-referenced as follows:
- Everything in class 1 and 2 above.
- Does the reverse mapping index list no owners of this space?
- Is this space not claimed by the inode index for inodes?
- Is it not mentioned by the reference count index?
- Is there a matching record in the other free space btree?
4. Inode btree records are cross-referenced as follows:
- Everything in class 1 and 2 above.
- Is there a matching record in free inode btree?
- Do cleared bits in the holemask correspond with inode clusters?
- Do set bits in the freemask correspond with inode records with zero link
count?
5. Inode records are cross-referenced as follows:
- Everything in class 1.
- Do all the fields that summarize information about the file forks actually
match those forks?
- Does each inode with zero link count correspond to a record in the free
inode btree?
6. File fork space mapping records are cross-referenced as follows:
- Everything in class 1 and 2 above.
- Is this space not mentioned by the inode btrees?
- If this is a CoW fork mapping, does it correspond to a CoW entry in the
reference count btree?
7. Reference count records are cross-referenced as follows:
- Everything in class 1 and 2 above.
- Within the space subkeyspace of the rmap btree (that is to say, all
records mapped to a particular space extent and ignoring the owner info),
are there the same number of reverse mapping records for each block as the
reference count record claims?
Proposed patchsets are the series to find gaps in
`refcount btree
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps>`_,
`inode btree
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps>`_, and
`rmap btree
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps>`_ records;
to find
`mergeable records
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records>`_;
and to
`improve cross referencing with rmap
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking>`_
before starting a repair.
Extended attribute structure 검사
1328-1373Extended attribute는 어떤 파일에도 데이터 조각을 붙일 수 있게 하는 key-value store를 구현합니다. Kernel과 userspace 모두 namespace 및 privilege 제한에 따라 key와 value에 접근할 수 있습니다. 이러한 데이터 조각은 보통 파일의 출처, security context, 사용자가 지정한 label, indexing 정보 등 파일에 관한 메타데이터입니다.
Name의 길이는 최대 255 byte이며 여러 namespace에 존재할 수 있습니다. Value의 크기는 최대 64KB입니다. 파일의 extended attribute는 attr fork가 매핑하는 블록에 저장됩니다. 이 매핑은 leaf block, remote value block 또는 dabtree block을 가리킵니다. Attribute fork의 block 0은 항상 구조의 최상위지만, 그 밖에는 세 블록 형식 모두 attr fork의 어느 offset에도 있을 수 있습니다. Leaf block에는 name과 value를 가리키는 attribute key record가 들어 있습니다. Name은 항상 같은 leaf block의 다른 위치에 저장됩니다. Filesystem block 크기의 3/4보다 작은 value도 같은 leaf block의 다른 위치에 저장됩니다. Leaf 안에 들어가기에는 너무 큰 value는 remote value block에 저장합니다. Leaf 정보가 filesystem block 하나를 넘으면 block 0을 root로 하는 dabtree를 만들어 attribute name의 hash를 attr fork의 leaf block에 매핑합니다.
Attr fork block 0: 항상 구조의 최상위→단일 block이면 leaf block에 attribute key record 저장→Name은 같은 leaf block 안의 다른 위치에 저장→Value가 filesystem block의 3/4보다 작으면 같은 leaf block에 저장→큰 value는 remote value block에 저장→Leaf 정보가 한 block을 넘으면 dabtree가 name hash를 leaf block에 매핑
Block 0에서 name과 value storage로 이어지는 관계를 구조화했습니다.
Attr block과 index block이 분리되어 있지 않으므로 extended attribute 구조를 검사하는 일은 단순하지 않습니다. Scrub은 attr fork가 매핑하는 각 블록을 읽고 non-leaf block을 무시해야 합니다.
단계검사
1Attr fork에 dabtree가 있으면 이를 순회하여 block에 irregularity가 없는지, dabtree mapping이 attr leaf block이 아닌 곳을 가리키지 않는지 확인합니다.
2Attr fork의 block을 순회하면서 leaf block을 찾습니다.
2.a각 leaf entry의 name에 invalid character가 없는지 검증합니다.
2.bAttr value를 읽습니다. 이 과정은 attr name으로 named lookup을 수행하여 dabtree가 올바른지 확인합니다. Value가 remote block에 저장되어 있다면 remote value block의 무결성도 검증합니다.
원문의 번호와 하위 단계를 그대로 보존합니다.
Checking Extended Attributes
````````````````````````````
Extended attributes implement a key-value store that enable fragments of data
to be attached to any file.
Both the kernel and userspace can access the keys and values, subject to
namespace and privilege restrictions.
Most typically these fragments are metadata about the file -- origins, security
contexts, user-supplied labels, indexing information, etc.
Names can be as long as 255 bytes and can exist in several different
namespaces.
Values can be as large as 64KB.
A file's extended attributes are stored in blocks mapped by the attr fork.
The mappings point to leaf blocks, remote value blocks, or dabtree blocks.
Block 0 in the attribute fork is always the top of the structure, but otherwise
each of the three types of blocks can be found at any offset in the attr fork.
Leaf blocks contain attribute key records that point to the name and the value.
Names are always stored elsewhere in the same leaf block.
Values that are less than 3/4 the size of a filesystem block are also stored
elsewhere in the same leaf block.
Remote value blocks contain values that are too large to fit inside a leaf.
If the leaf information exceeds a single filesystem block, a dabtree (also
rooted at block 0) is created to map hashes of the attribute names to leaf
blocks in the attr fork.
Checking an extended attribute structure is not so straightforward due to the
lack of separation between attr blocks and index blocks.
Scrub must read each block mapped by the attr fork and ignore the non-leaf
blocks:
1. Walk the dabtree in the attr fork (if present) to ensure that there are no
irregularities in the blocks or dabtree mappings that do not point to
attr leaf blocks.
2. Walk the blocks of the attr fork looking for leaf blocks.
For each entry inside a leaf:
a. Validate that the name does not contain invalid characters.
b. Read the attr value.
This performs a named lookup of the attr name to ensure the correctness
of the dabtree.
If the value is stored in a remote block, this also validates the
integrity of the remote value block.
Directory DAG와 세 partition 검사
1374-1434파일시스템 directory tree는 file이 node를 이루고 directory entry(dirent)가 edge를 이루는 directed acyclic graph 구조입니다. Directory는 255-byte sequence인 name을 inumber에 대응시키는 매핑 집합을 담는 특별한 종류의 파일입니다. 이 매핑을 directory entry, 줄여서 dirent라고 부릅니다. 각 directory file은 정확히 하나의 directory가 그 파일을 가리켜야 합니다. Root directory는 자기 자신을 가리킵니다. Directory entry는 어떤 형식의 파일도 가리킬 수 있습니다. Directory가 아닌 각 파일은 여러 directory가 가리킬 수 있습니다.
XFS에서 directory는 최대 세 개의 32GB partition을 포함하는 파일로 구현됩니다. 첫 번째 partition에는 directory entry data block이 있습니다. 각 data block에는 사용자가 제공한 name을 inumber 및 선택적인 file type과 연결하는 가변 크기 레코드가 들어 있습니다. Directory entry data가 block 하나를 넘어서면 post-EOF extent로 존재하는 두 번째 partition에 여유 공간 정보와 dirent name hash를 첫 번째 partition의 directory data block에 매핑하는 인덱스가 담긴 block을 채웁니다. 이 구조 덕분에 directory name lookup이 매우 빨라집니다. 두 번째 partition이 block 하나보다 커지면 더 빠른 확장을 위해 세 번째 partition에 여유 공간 정보의 선형 배열을 채웁니다. 여유 공간 정보가 분리된 뒤 두 번째 partition이 다시 block 하나를 넘도록 커지면 dabtree를 사용해 dirent name hash를 directory data block에 매핑합니다.
Partition내용조건·목적
1가변 크기 dirent record가 든 directory entry data block항상 기본 directory data를 저장
2Free-space 정보와 name-hash index, 필요하면 dabtreeDirent data가 한 block을 넘을 때 post-EOF extent에 생성되어 lookup을 가속
3Free-space 정보의 linear array두 번째 partition이 한 block을 넘을 때 빠른 expansion을 위해 생성
각 partition의 저장 내용과 생성 조건을 구분합니다.
Directory 검사는 비교적 단순합니다.
단계검사
1두 번째 partition에 dabtree가 있으면 이를 순회하여 block에 irregularity가 없는지, dabtree mapping이 dirent block이 아닌 곳을 가리키지 않는지 확인합니다.
2첫 번째 partition의 block을 순회하면서 directory entry를 찾습니다.
2.aName에 invalid character가 없는가?
2.bInumber가 실제로 할당된 inode와 대응하는가?
2.cChild inode의 link count가 0이 아닌가?
2.dDirent에 file type이 포함되어 있다면 inode의 type과 일치하는가?
2.eChild가 subdirectory라면 child의 dotdot pointer가 parent를 다시 가리키는가?
2.fDirectory에 두 번째 partition이 있다면 dirent name으로 named lookup을 수행하여 dabtree가 올바른지 확인합니다.
3세 번째 partition에 free-space list가 있으면 이를 순회하여 목록이 설명하는 여유 공간이 실제로 사용되지 않는지 확인합니다.
원문의 번호와 여섯 dirent 검사를 그대로 보존합니다.
`:ref:`의 `parents <dirparent>` 및 `file link counts <nlinks>`와 관련된 검사는 뒤 절에서 더 자세히 설명합니다.
Checking and Cross-Referencing Directories
``````````````````````````````````````````
The filesystem directory tree is a directed acylic graph structure, with files
constituting the nodes, and directory entries (dirents) constituting the edges.
Directories are a special type of file containing a set of mappings from a
255-byte sequence (name) to an inumber.
These are called directory entries, or dirents for short.
Each directory file must have exactly one directory pointing to the file.
A root directory points to itself.
Directory entries point to files of any type.
Each non-directory file may have multiple directories point to it.
In XFS, directories are implemented as a file containing up to three 32GB
partitions.
The first partition contains directory entry data blocks.
Each data block contains variable-sized records associating a user-provided
name with an inumber and, optionally, a file type.
If the directory entry data grows beyond one block, the second partition (which
exists as post-EOF extents) is populated with a block containing free space
information and an index that maps hashes of the dirent names to directory data
blocks in the first partition.
This makes directory name lookups very fast.
If this second partition grows beyond one block, the third partition is
populated with a linear array of free space information for faster
expansions.
If the free space has been separated and the second partition grows again
beyond one block, then a dabtree is used to map hashes of dirent names to
directory data blocks.
Checking a directory is pretty straightforward:
1. Walk the dabtree in the second partition (if present) to ensure that there
are no irregularities in the blocks or dabtree mappings that do not point to
dirent blocks.
2. Walk the blocks of the first partition looking for directory entries.
Each dirent is checked as follows:
a. Does the name contain no invalid characters?
b. Does the inumber correspond to an actual, allocated inode?
c. Does the child inode have a nonzero link count?
d. If a file type is included in the dirent, does it match the type of the
inode?
e. If the child is a subdirectory, does the child's dotdot pointer point
back to the parent?
f. If the directory has a second partition, perform a named lookup of the
dirent name to ensure the correctness of the dabtree.
3. Walk the free space list in the third partition (if present) to ensure that
the free spaces it describes are really unused.
Checking operations involving :ref:`parents <dirparent>` and
:ref:`file link counts <nlinks>` are discussed in more detail in later
sections.
Directory/attribute btree 검사
1435-1479앞 절에서 설명했듯이 directory/attribute btree(dabtree) index는 선형 스캔을 피하여 lookup 시간을 줄이기 위해 사용자가 제공한 name을 매핑합니다. 내부적으로는 name의 32-bit hash를 해당 file fork 안의 block offset에 매핑합니다.
Dabtree의 내부 구조는 고정 크기 메타데이터 레코드를 기록하는 btree와 매우 비슷합니다. 각 dabtree block에는 magic number, checksum, sibling pointer, UUID, tree level, log sequence number가 들어 있습니다. Leaf record와 node record의 형식은 같습니다. 각 entry는 계층에서 바로 아래 level을 가리키며, dabtree node record는 dabtree leaf block을 가리키고 dabtree leaf record는 fork의 다른 위치에 있는 non-dabtree block을 가리킵니다.
Dabtree의 검사와 cross-reference는 space btree에 수행하는 작업과 매우 비슷합니다.
Dabtree node record→다음 level dabtree block→Dabtree leaf record→Directory data 또는 attr leaf block→Name hash와 target content 일치 확인
Record type에 따라 마지막 target이 달라집니다.
번호검사 질문
1블록에 저장된 데이터 형식이 scrub이 예상한 형식과 일치하는가?
2블록이 읽기를 요청한 소유 구조에 속하는가?
3레코드가 블록 안에 들어맞는가?
4블록 안의 레코드에 명백한 손상이 없는가?
5Name hash가 올바른 순서로 정렬되어 있는가?
6Dabtree 안의 node pointer가 dabtree block에 유효한 fork offset을 가리키는가?
7Dabtree 안의 leaf pointer가 directory 또는 attr leaf block에 유효한 fork offset을 가리키는가?
8Child pointer가 leaf 방향을 가리키는가?
9Sibling pointer가 같은 level을 가로질러 가리키는가?
10각 dabtree node record에서 record key가 child dabtree block의 내용을 정확히 반영하는가?
11각 dabtree leaf record에서 record key가 directory 또는 attr block의 내용을 정확히 반영하는가?
원문의 열한 질문을 순서대로 보존합니다.
Checking Directory/Attribute Btrees
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As stated in previous sections, the directory/attribute btree (dabtree) index
maps user-provided names to improve lookup times by avoiding linear scans.
Internally, it maps a 32-bit hash of the name to a block offset within the
appropriate file fork.
The internal structure of a dabtree closely resembles the btrees that record
fixed-size metadata records -- each dabtree block contains a magic number, a
checksum, sibling pointers, a UUID, a tree level, and a log sequence number.
The format of leaf and node records are the same -- each entry points to the
next level down in the hierarchy, with dabtree node records pointing to dabtree
leaf blocks, and dabtree leaf records pointing to non-dabtree blocks elsewhere
in the fork.
Checking and cross-referencing the dabtree is very similar to what is done for
space btrees:
- Does the type of data stored in the block match what scrub is expecting?
- Does the block belong to the owning structure that asked for the read?
- Do the records fit within the block?
- Are the records contained inside the block free of obvious corruptions?
- Are the name hashes in the correct order?
- Do node pointers within the dabtree point to valid fork offsets for dabtree
blocks?
- Do leaf pointers within the dabtree point to valid fork offsets for directory
or attr leaf blocks?
- Do child pointers point towards the leaves?
- Do sibling pointers point across the same level?
- For each dabtree node record, does the record key accurate reflect the
contents of the child dabtree block?
- For each dabtree leaf record, does the record key accurate reflect the
contents of the directory or attr block?
세 summary counter class
1480-1497XFS는 사용 가능한 자원, quota 자원 사용량, 파일 link count라는 세 종류의 summary counter를 유지합니다.
이론적으로 사용 가능한 자원, 즉 data block, inode, realtime extent의 양은 파일시스템 전체를 순회하면 알아낼 수 있습니다. 그러나 그렇게 보고하면 매우 느리므로 transactional filesystem은 이 정보의 요약을 superblock에 유지할 수 있습니다. 이 값을 파일시스템 메타데이터와 cross-reference하는 작업은 각 AG의 free-space 및 inode metadata와 realtime bitmap을 순회하는 단순한 일이어야 하지만, 뒤의 `:ref:` `more detail <fscounters>`에서 설명할 복잡한 문제가 있습니다.
`:ref:` `Quota usage <quotacheck>`와 `file link count <nlinks>` 검사는 별도의 절이 필요할 만큼 충분히 복잡합니다.
종류요약 대상·cross-reference 원본
Available resourcesData block·inode·realtime extent; 각 AG의 free-space 및 inode metadata와 realtime bitmap
Quota resource usageQuota별 자원 사용량; 별도 `quotacheck` 절에서 검사
File link countFile을 가리키는 directory entry 수; 별도 `nlinks` 절에서 검사
요약 값과 원본 메타데이터를 연결합니다.
Cross-Referencing Summary Counters
``````````````````````````````````
XFS maintains three classes of summary counters: available resources, quota
resource usage, and file link counts.
In theory, the amount of available resources (data blocks, inodes, realtime
extents) can be found by walking the entire filesystem.
This would make for very slow reporting, so a transactional filesystem can
maintain summaries of this information in the superblock.
Cross-referencing these values against the filesystem metadata should be a
simple matter of walking the free space and inode metadata in each AG and the
realtime bitmap, but there are complications that will be discussed in
:ref:`more detail <fscounters>` later.
:ref:`Quota usage <quotacheck>` and :ref:`file link count <nlinks>`
checking are sufficiently complicated to warrant separate sections.
Repair 후 동일 check 재실행
1498-1508복구를 수행한 뒤에는 검사 코드를 두 번째로 실행하여 새 구조를 검증하고, health assessment 결과를 내부에 기록한 다음 호출 프로세스에 반환합니다. 이 단계는 시스템 관리자가 파일시스템 상태와 복구 작업의 진행 상황을 감시할 수 있게 하는 데 매우 중요합니다. 개발자에게는 online 및 offline 검사 도구의 오류 탐지와 수정 효과를 판단하는 유용한 수단입니다.
Replacement structure commit→같은 checker 재실행→Health result 내부 기록→Calling process에 결과 반환
수정 자체가 아니라 재검증 결과가 최종 health 상태를 정합니다.
Post-Repair Reverification
``````````````````````````
After performing a repair, the checking code is run a second time to validate
the new structure, and the results of the health assessment are recorded
internally and returned to the calling process.
This step is critical for enabling system administrator to monitor the status
of the filesystem and the progress of any repairs.
For developers, it is a useful means to judge the efficacy of error detection
and correction in the online and offline checking tools.
Transaction chain과 일시적 불일치
1509-1545복잡한 작업은 transaction chain을 사용하여 여러 per-AG 자료 구조를 수정할 수 있습니다. 일단 log에 commit된 이 chain은 처리 도중 시스템이 crash하면 log recovery 중 다시 시작됩니다. Chain 안의 transaction 사이에서는 AG header buffer의 lock이 풀리므로, online checking은 아직 진행 중인 chained operation과 조정하여 pending chain 때문에 생긴 불일치를 잘못 탐지하지 않아야 합니다. 또한 작업이 pending 상태일 때는 메타데이터가 서로 일시적으로 일관되지 않고 재구성도 불가능하므로 online repair를 실행해서는 안 됩니다.
AG metadata의 완전한 일관성을 요구하는 것은 online fsck뿐이며, online fsck는 파일시스템 변경 작업보다 비교적 드물게 실행됩니다. Online fsck는 다음과 같이 transaction chain과 조정합니다.
규칙동작
AG별 intent item count각 AG에 대해 그 AG를 대상으로 하는 intent item 수를 유지합니다. Chain에 새 item을 추가할 때 count를 증가시키고, 파일시스템이 AG header buffer를 잠가 작업을 마쳤을 때 count를 감소시킵니다.
AG 검사 시 quiesceOnline fsck가 AG를 조사하려면 AG header buffer를 잠가 그 AG를 수정하려는 모든 transaction chain을 quiesce합니다. Count가 0이면 검사를 진행합니다. 0이 아니면 buffer lock을 cycle하여 chain이 forward progress할 수 있게 합니다.
원문의 두 규칙과 count·lock 전환 조건을 그대로 보존합니다.
이 때문에 online fsck를 완료하는 데 오랜 시간이 걸릴 수 있지만, 일반 파일시스템 갱신은 background checking 작업보다 우선합니다. 이 상황을 발견한 과정은 다음 절의 `:ref:` `next section <chain_coordination>`에서 설명하고, 해결책의 세부 내용은 그 뒤의 `:ref:` `<intent_drains>`에서 설명합니다.
Eventual Consistency vs. Online Fsck
------------------------------------
Complex operations can make modifications to multiple per-AG data structures
with a chain of transactions.
These chains, once committed to the log, are restarted during log recovery if
the system crashes while processing the chain.
Because the AG header buffers are unlocked between transactions within a chain,
online checking must coordinate with chained operations that are in progress to
avoid incorrectly detecting inconsistencies due to pending chains.
Furthermore, online repair must not run when operations are pending because
the metadata are temporarily inconsistent with each other, and rebuilding is
not possible.
Only online fsck has this requirement of total consistency of AG metadata, and
should be relatively rare as compared to filesystem change operations.
Online fsck coordinates with transaction chains as follows:
* For each AG, maintain a count of intent items targeting that AG.
The count should be bumped whenever a new item is added to the chain.
The count should be dropped when the filesystem has locked the AG header
buffers and finished the work.
* When online fsck wants to examine an AG, it should lock the AG header
buffers to quiesce all transaction chains that want to modify that AG.
If the count is zero, proceed with the checking operation.
If it is nonzero, cycle the buffer locks to allow the chain to make forward
progress.
This may lead to online fsck taking a long time to complete, but regular
filesystem updates take precedence over background checking activity.
Details about the discovery of this situation are presented in the
:ref:`next section <chain_coordination>`, and details about the solution
are presented :ref:`after that<intent_drains>`.
.. _chain_coordination:
Fsstress가 발견한 false inconsistency
1546-1566Online scrubbing 개발이 한창 진행되던 중, fsstress test는 online fsck와 다른 writer thread가 만든 compound transaction chain 사이의 잘못된 상호작용을 찾아냈습니다. 이 상호작용 때문에 metadata inconsistency가 거짓으로 보고되었습니다. 이러한 보고의 근본 원인은 reverse mapping과 reflink가 도입될 때 deferred work item과 compound transaction chain의 적용 범위를 넓히면서 도입된 eventual consistency model입니다.
Transaction chain은 원래 file에서 space를 unmap할 때 deadlock을 피하려고 XFS에 추가되었습니다. Deadlock 회피 규칙에 따라 AG는 반드시 번호가 증가하는 순서로만 lock해야 합니다. 따라서 예를 들어 한 transaction에서 AG 7의 space extent를 해제한 다음, 이제 불필요해진 AG 3의 block mapping btree block을 해제하려고 시도할 수 없습니다. XFS는 이러한 deadlock을 피하기 위해 Extent Freeing Intent(EFI) log item을 만듭니다. 한 transaction에서 일정한 space를 해제하겠다는 의무를 commit하되, 실제 metadata update는 새 transaction으로 미룹니다. Transaction 순서는 다음과 같습니다.
Discovery of the Problem
````````````````````````
Midway through the development of online scrubbing, the fsstress tests
uncovered a misinteraction between online fsck and compound transaction chains
created by other writer threads that resulted in false reports of metadata
inconsistency.
The root cause of these reports is the eventual consistency model introduced by
the expansion of deferred work items and compound transaction chains when
reverse mapping and reflink were introduced.
Originally, transaction chains were added to XFS to avoid deadlocks when
unmapping space from files.
Deadlock avoidance rules require that AGs only be locked in increasing order,
which makes it impossible (say) to use a single transaction to free a space
extent in AG 7 and then try to free a now superfluous block mapping btree block
in AG 3.
To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
items to commit to freeing some space in one transaction while deferring the
actual metadata updates to a fresh transaction.
The transaction sequence looks like this:EFI와 EFD의 두 transaction
1567-15931. 첫 번째 transaction은 file의 block mapping structure를 물리적으로 갱신하여 btree block에서 mapping을 제거합니다. 그런 다음 space의 deferred freeing을 예약하는 action item을 in-memory transaction에 연결합니다. 구체적으로 각 transaction은 `struct xfs_defer_pending` object의 list를 유지하고, 각 object는 다시 `struct xfs_extent_free_item` object의 list를 유지합니다. 앞의 예에서 action item은 AG 7에서 unmap된 space와 AG 3의 block mapping btree(BMBT) block을 모두 해제하는 작업을 추적합니다. 이 방식으로 기록된 deferred free는 `struct xfs_extent_free_item` object에서 EFI log item을 만들어 transaction에 연결함으로써 log에 commit됩니다. Log가 disk에 영속화될 때 EFI item은 ondisk transaction record에 기록됩니다. EFI 하나에는 해제할 extent를 최대 16개까지 나열할 수 있으며, 모든 extent는 AG 순서로 정렬됩니다.
2. 두 번째 transaction은 AG 3의 free space btree를 물리적으로 갱신하여 이전 BMBT block을 해제하고, 이어서 AG 7의 free space btree를 두 번째로 물리 갱신하여 unmap된 file space를 해제합니다. 가능할 때 물리 update가 올바른 순서로 다시 배열된다는 점에 유의해야 합니다. 이 transaction에는 extent free done(EFD) log item이 연결됩니다. EFD에는 transaction #1에서 기록한 EFI를 가리키는 pointer가 있으므로, log recovery는 EFI를 replay해야 하는지 판별할 수 있습니다.
Transaction #1: file BMBT에서 mapping 제거→`struct xfs_defer_pending`와 `struct xfs_extent_free_item`에 AG 7·AG 3 free 예약→AG 순서로 정렬한 최대 16 extent를 EFI에 기록하고 disk에 영속화→Transaction #2: AG 3 BMBT block 해제 후 AG 7 file space 해제→EFD가 transaction #1의 EFI를 가리키도록 기록→EFI에 대응하는 EFD가 없으면 log recovery가 replay
두 transaction과 recovery 의무의 연결을 순서대로 보존합니다.
1. The first transaction contains a physical update to the file's block mapping
structures to remove the mapping from the btree blocks.
It then attaches to the in-memory transaction an action item to schedule
deferred freeing of space.
Concretely, each transaction maintains a list of ``struct
xfs_defer_pending`` objects, each of which maintains a list of ``struct
xfs_extent_free_item`` objects.
Returning to the example above, the action item tracks the freeing of both
the unmapped space from AG 7 and the block mapping btree (BMBT) block from
AG 3.
Deferred frees recorded in this manner are committed in the log by creating
an EFI log item from the ``struct xfs_extent_free_item`` object and
attaching the log item to the transaction.
When the log is persisted to disk, the EFI item is written into the ondisk
transaction record.
EFIs can list up to 16 extents to free, all sorted in AG order.
2. The second transaction contains a physical update to the free space btrees
of AG 3 to release the former BMBT block and a second physical update to the
free space btrees of AG 7 to release the unmapped file space.
Observe that the physical updates are resequenced in the correct order
when possible.
Attached to the transaction is a an extent free done (EFD) log item.
The EFD contains a pointer to the EFI logged in transaction #1 so that log
recovery can tell if the EFI needs to be replayed.
EFI without EFD의 crash recovery
1594-1603Transaction #1이 filesystem에 writeback된 뒤 #2가 commit되기 전에 system이 중단되면, filesystem metadata scan에서는 unmap된 space의 owner가 없는 것처럼 보여 metadata가 일관되지 않게 나타납니다. 다행히 log recovery가 이 불일치를 바로잡습니다. Recovery가 intent log item을 찾았지만 그에 대응하는 intent done item을 찾지 못하면, intent item의 incore state를 재구성하여 작업을 끝냅니다. 앞의 예에서는 recovery 단계를 완료하기 위해 복구된 EFI에 기술된 두 free를 log가 모두 replay해야 합니다.
If the system goes down after transaction #1 is written back to the filesystem
but before #2 is committed, a scan of the filesystem metadata would show
inconsistent filesystem metadata because there would not appear to be any owner
of the unmapped space.
Happily, log recovery corrects this inconsistency for us -- when recovery finds
an intent log item but does not find a corresponding intent done item, it will
reconstruct the incore state of the intent item and finish it.
In the example above, the log must replay both frees described in the recovered
EFI to complete the recovery phase.
Transaction chain의 세 가지 안전 조건
1604-1623XFS의 transaction chaining 전략에는 고려해야 할 미묘한 조건이 있습니다.
조건요구사항·효과
Log item 순서Transaction이 보유하지 않은 principal object와 충돌하지 않도록 log item을 올바른 순서로 추가해야 합니다. Unmap된 block의 모든 per-AG metadata update를 extent 해제라는 마지막 update보다 먼저 끝내야 하며, 그 마지막 update가 log에 commit되기 전에는 extent를 다시 할당해서는 안 됩니다.
Transaction 사이의 unlockChain의 각 transaction 사이에서 AG header buffer가 해제됩니다. 다른 thread가 AG의 중간 상태를 관찰할 수 있지만, 첫 번째 조건을 지키면 filesystem operation의 정확성에는 영향을 주지 않아야 합니다.
Unmount 시 flushFilesystem을 unmount하면 pending work가 모두 disk로 flush됩니다. 따라서 offline fsck는 deferred work item 처리 때문에 생기는 일시적 불일치를 보지 않습니다.
원문의 세 조건과 관찰 가능한 중간 상태를 분리합니다.
이와 같은 방식으로 XFS는 deadlock을 피하고 parallelism을 높이기 위해 eventual consistency의 한 형태를 사용합니다.
There are subtleties to XFS' transaction chaining strategy to consider:
* Log items must be added to a transaction in the correct order to prevent
conflicts with principal objects that are not held by the transaction.
In other words, all per-AG metadata updates for an unmapped block must be
completed before the last update to free the extent, and extents should not
be reallocated until that last update commits to the log.
* AG header buffers are released between each transaction in a chain.
This means that other threads can observe an AG in an intermediate state,
but as long as the first subtlety is handled, this should not affect the
correctness of filesystem operations.
* Unmounting the filesystem flushes all pending work to disk, which means that
offline fsck never sees the temporary inconsistencies caused by deferred
work item processing.
In this manner, XFS employs a form of eventual consistency to avoid deadlocks
and increase parallelism.
한 file mapping 변경의 update 증폭
1624-1656Reverse mapping과 reflink 기능을 설계할 때, file mapping operation 하나가 많은 작은 update로 폭증할 수 있으므로 단일 filesystem 변경에 필요한 모든 reverse mapping update를 하나의 transaction에 억지로 넣는 것은 현실적이지 않다고 판단했습니다.
번호의미군Update
1Block mappingBlock mapping update 자체
2Block mappingBlock mapping update에 대한 reverse mapping update
3Block mappingFreelist 수정
4Block mappingFreelist 수정에 대한 reverse mapping update
5BMBT shapeBlock mapping btree의 shape 변경
6BMBT shapeBtree update에 대한 reverse mapping update
7BMBT shapeFreelist 수정(두 번째)
8BMBT shapeFreelist 수정에 대한 reverse mapping update
9Reference countReference counting information update
10Reference countRefcount update에 대한 reverse mapping update
11Reference countFreelist 수정(세 번째)
12Reference countFreelist 수정에 대한 reverse mapping update
13Unmapped spaceUnmap되었고 다른 어떤 file도 소유하지 않는 space 해제
14Unmapped spaceFreelist 수정(네 번째)
15Unmapped spaceFreelist 수정에 대한 reverse mapping update
16BMBT spaceBlock mapping btree가 사용한 space 해제
17BMBT spaceFreelist 수정(다섯 번째)
18BMBT spaceFreelist 수정에 대한 reverse mapping update
원문의 18개 항목과 다섯 의미군을 순서대로 보존합니다.
Freelist fixup은 보통 transaction chain에서 AG마다 한 번을 넘겨 필요하지 않지만, space가 매우 부족하면 이론적으로 여러 번 필요할 수 있습니다. Copy-on-write update는 상황이 더 나쁩니다. Staging area에서 space를 제거할 때 한 번, 그 space를 file에 mapping할 때 다시 한 번 이 전체 작업을 수행해야 하기 때문입니다.
During the design phase of the reverse mapping and reflink features, it was
decided that it was impractical to cram all the reverse mapping updates for a
single filesystem change into a single transaction because a single file
mapping operation can explode into many small updates:
* The block mapping update itself
* A reverse mapping update for the block mapping update
* Fixing the freelist
* A reverse mapping update for the freelist fix
* A shape change to the block mapping btree
* A reverse mapping update for the btree update
* Fixing the freelist (again)
* A reverse mapping update for the freelist fix
* An update to the reference counting information
* A reverse mapping update for the refcount update
* Fixing the freelist (a third time)
* A reverse mapping update for the freelist fix
* Freeing any space that was unmapped and not owned by any other file
* Fixing the freelist (a fourth time)
* A reverse mapping update for the freelist fix
* Freeing the space used by the block mapping btree
* Fixing the freelist (a fifth time)
* A reverse mapping update for the freelist fix
Free list fixups are not usually needed more than once per AG per transaction
chain, but it is theoretically possible if space is very tight.
For copy-on-write updates this is even worse, because this must be done once to
remove the space from a staging area and again to map it into the file!
Deferred work와 reservation 절충
1657-1664이러한 update 폭증을 차분하게 처리하기 위해 XFS는 deferred work item의 사용 범위를 대부분의 reverse mapping update와 모든 refcount update로 넓혔습니다. 작업을 긴 small update chain으로 나누면 transaction reservation의 최악 크기는 줄어들지만, system의 eventual consistency 정도는 커집니다. 일반적으로 XFS는 서로 상황을 모르는 thread 사이에서 resource reuse conflict가 생기지 않도록 deferred work item의 순서를 신중하게 정하므로 이것이 문제가 되지는 않습니다.
To deal with this explosion in a calm manner, XFS expands its use of deferred
work items to cover most reverse mapping updates and all refcount updates.
This reduces the worst case size of transaction reservations by breaking the
work into a long chain of small updates, which increases the degree of eventual
consistency in the system.
Again, this generally isn't a problem because XFS orders its deferred work
items carefully to avoid resource reuse conflicts between unsuspecting threads.
Scrub가 관찰하는 false corruption
1665-1677그러나 online fsck는 이 규칙을 바꿉니다. Per-AG structure의 물리 update는 AG header buffer를 lock하여 조정하지만, transaction 사이에서는 buffer lock을 해제한다는 점을 기억해야 합니다. Scrub가 data structure의 resource와 lock을 획득한 뒤에는 lock을 놓지 않고 모든 validation 작업을 수행해야 합니다. Space btree의 main lock이 AG header buffer lock이라면, scrub는 다른 thread가 chain을 마치는 도중에 그 thread를 중단시킨 것일 수 있습니다. 예를 들어 copy-on-write를 수행하는 thread가 reverse mapping update는 끝냈지만 대응하는 refcount update는 아직 끝내지 않았다면, 두 AG btree는 scrub에 일관되지 않은 것처럼 보이고 corruption 관찰 결과가 기록됩니다. 이 관찰은 정확하지 않습니다. 이 상태에서 repair를 시도하면 결과는 치명적입니다.
However, online fsck changes the rules -- remember that although physical
updates to per-AG structures are coordinated by locking the buffers for AG
headers, buffer locks are dropped between transactions.
Once scrub acquires resources and takes locks for a data structure, it must do
all the validation work without releasing the lock.
If the main lock for a space btree is an AG header buffer lock, scrub may have
interrupted another thread that is midway through finishing a chain.
For example, if a thread performing a copy-on-write has completed a reverse
mapping update but not the corresponding refcount update, the two AG btrees
will appear inconsistent to scrub and an observation of corruption will be
recorded. This observation will not be correct.
If a repair is attempted in this state, the results will be catastrophic!
검토 후 거부된 세 가지 대안
1678-1706이 결함을 발견한 뒤 문제를 해결하기 위한 몇 가지 다른 방법을 검토했지만 모두 거부했습니다.
1. Allocation group에 더 높은 수준의 lock을 추가하고, writer thread가 변경을 시작하기 전에 그 상위 lock을 AG 순서로 획득하도록 요구하는 방법입니다. 전체 operation을 모의 실행하지 않고는 어떤 lock을 어떤 순서로 얻어야 하는지 판단하기 어렵기 때문에 실제 구현은 매우 어렵습니다. 필요한 lock을 찾아내려고 file operation을 dry run하면 filesystem이 매우 느려집니다.
2. Deferred work coordinator code가 같은 AG를 대상으로 연속해서 이어지는 intent item을 인식하게 하고, update 사이에서 transaction을 roll하는 동안 AG header buffer를 계속 lock하도록 하는 방법입니다. Coordinator는 실제 deferred work item과 느슨하게 결합되어 있을 뿐이므로 이 방법은 coordinator에 큰 복잡성을 더합니다. 또한 deferred work item이 새 deferred subtask를 만들 수 있지만 새 sibling task의 작업을 시작하기 전에 모든 subtask를 완료해야 하므로 문제를 해결하지 못합니다.
3. Online fsck가 scrub 중인 data structure를 보호하는 lock을 기다리는 모든 transaction을 순회하여 pending operation을 찾도록 하는 방법입니다. Checking과 repair operation은 평가할 때 이러한 pending operation을 반영해야 합니다. 이 해결책은 main filesystem에 극도로 침습적이므로 처음부터 채택할 수 없습니다.
Several other solutions to this problem were evaluated upon discovery of this
flaw and rejected:
1. Add a higher level lock to allocation groups and require writer threads to
acquire the higher level lock in AG order before making any changes.
This would be very difficult to implement in practice because it is
difficult to determine which locks need to be obtained, and in what order,
without simulating the entire operation.
Performing a dry run of a file operation to discover necessary locks would
make the filesystem very slow.
2. Make the deferred work coordinator code aware of consecutive intent items
targeting the same AG and have it hold the AG header buffers locked across
the transaction roll between updates.
This would introduce a lot of complexity into the coordinator since it is
only loosely coupled with the actual deferred work items.
It would also fail to solve the problem because deferred work items can
generate new deferred subtasks, but all subtasks must be complete before
work can start on a new sibling task.
3. Teach online fsck to walk all transactions waiting for whichever lock(s)
protect the data structure being scrubbed to look for pending operations.
The checking and repair operations must factor these pending operations into
the evaluations being performed.
This solution is a nonstarter because it is *extremely* invasive to the main
filesystem.
.. _intent_drains:
Atomic counter와 lock cycling
1707-1764Online fsck는 atomic intent item counter와 lock cycling을 사용하여 transaction chain과 조정합니다. Drain mechanism에는 두 가지 핵심 속성이 있습니다. 첫째, deferred work item을 transaction에 queue할 때 counter를 증가시키고, 연결된 intent done log item이 다른 transaction에 commit된 뒤 counter를 감소시킵니다. 둘째, AG header lock을 보유하지 않아도 deferred work를 transaction에 추가할 수 있지만, 물리 update와 intent done log item을 기록하기 위해 해당 AG header buffer를 lock하지 않고는 per-AG work item을 done으로 표시할 수 없습니다. 첫 번째 속성은 실행 중인 transaction chain에 scrub가 양보할 수 있게 하며, file operation에 이익을 주기 위해 online fsck의 우선순위를 명시적으로 낮춥니다. 두 번째 속성은 scrub가 conflict 가능성을 언제나 판별할 수 있게 하므로 scrub를 정확하게 조정하는 핵심입니다.
일반 filesystem code에서 drain은 다음과 같이 동작합니다.
1. 알맞은 subsystem function을 호출하여 deferred work item을 transaction에 추가→2. Function이 `xfs_defer_drain_bump`를 호출하여 counter 증가→3. Deferred item manager가 작업을 끝낼 때 `->finish_item` 호출→4. `->finish_item` 구현이 변경을 log하고 `xfs_defer_drain_drop`으로 sloppy counter를 감소시킨 뒤 drain waiter를 깨움→5. Subtransaction이 commit되면서 intent item과 연결된 resource의 lock 해제
Deferred item queue부터 subtransaction commit까지의 다섯 단계를 보존합니다.
Scrub에서 drain은 다음과 같이 동작합니다.
1. Scrub할 metadata와 연결된 resource를 lock; refcount btree scan은 AGI와 AGF header buffer를 lock→2. Counter가 0이고 `xfs_defer_drain_busy`가 false를 반환하면 진행 중인 chain이 없으므로 검사 진행→3. 그렇지 않으면 1단계에서 획득한 resource 해제→4. `xfs_defer_drain_intents`로 intent counter가 0이 되기를 기다리고, signal을 받지 않았다면 1단계로 복귀
Busy 상태에서는 resource를 놓고 count가 0이 될 때까지 기다립니다.
4단계에서 polling하지 않도록 drain은 scrub thread를 위한 waitqueue를 제공합니다. Intent count가 0으로 내려갈 때마다 이 thread들을 깨웁니다.
제안된 patchset은 `scrub intent drain series <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_입니다.
Intent Drains
`````````````
Online fsck uses an atomic intent item counter and lock cycling to coordinate
with transaction chains.
There are two key properties to the drain mechanism.
First, the counter is incremented when a deferred work item is *queued* to a
transaction, and it is decremented after the associated intent done log item is
*committed* to another transaction.
The second property is that deferred work can be added to a transaction without
holding an AG header lock, but per-AG work items cannot be marked done without
locking that AG header buffer to log the physical updates and the intent done
log item.
The first property enables scrub to yield to running transaction chains, which
is an explicit deprioritization of online fsck to benefit file operations.
The second property of the drain is key to the correct coordination of scrub,
since scrub will always be able to decide if a conflict is possible.
For regular filesystem code, the drain works as follows:
1. Call the appropriate subsystem function to add a deferred work item to a
transaction.
2. The function calls ``xfs_defer_drain_bump`` to increase the counter.
3. When the deferred item manager wants to finish the deferred work item, it
calls ``->finish_item`` to complete it.
4. The ``->finish_item`` implementation logs some changes and calls
``xfs_defer_drain_drop`` to decrease the sloppy counter and wake up any threads
waiting on the drain.
5. The subtransaction commits, which unlocks the resource associated with the
intent item.
For scrub, the drain works as follows:
1. Lock the resource(s) associated with the metadata being scrubbed.
For example, a scan of the refcount btree would lock the AGI and AGF header
buffers.
2. If the counter is zero (``xfs_defer_drain_busy`` returns false), there are no
chains in progress and the operation may proceed.
3. Otherwise, release the resources grabbed in step 1.
4. Wait for the intent counter to reach zero (``xfs_defer_drain_intents``), then go
back to step 1 unless a signal has been caught.
To avoid polling in step 4, the drain provides a waitqueue for scrub threads to
be woken up whenever the intent count drops to zero.
The proposed patchset is the
`scrub intent drain series
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_.
.. _jump_labels:
비활성 hook의 runtime 비용
1765-1786XFS의 online fsck는 일반 filesystem을 checking 및 repair code와 최대한 분리합니다. 그러나 intent drain과 나중에 나올 live update hook처럼 online fsck code가 filesystem의 나머지 부분에서 무슨 일이 일어나는지 알면 유용한 영역이 몇 군데 있습니다. Online fsck가 background에서 계속 실행될 것으로 예상하지 않으므로, online fsck가 kernel에 compile되어 있지만 userspace를 대신해 실제로 실행되고 있지 않을 때 이러한 hook이 부과하는 runtime overhead를 최소화하는 것이 매우 중요합니다. Writer thread의 hot path에서 data structure에 접근하려고 lock을 잡았는데 추가 작업이 필요하지 않다는 사실만 확인하는 것은 비쌉니다. 저자의 computer에서는 access마다 40-50ns의 overhead가 발생합니다. 다행히 kernel은 dynamic code patching을 지원하므로, online fsck가 실행되지 않을 때 XFS는 hook code로 가는 static branch를 `nop` sled로 바꿀 수 있습니다. 이 sled의 overhead는 instruction decoder가 sled를 건너뛰는 데 걸리는 시간뿐이며, 대략 1ns 미만이고 instruction fetch 외에는 memory에 접근하지 않습니다.
상태비용Memory·locking 특성
Writer hot-path lock 확인40-50ns/accessData structure 접근 후 추가 작업 없음
비활성 `nop` sled<1nsInstruction fetch 외 memory 접근 없음
Static key 전환~22000nsCPU hotplug lock 필요
원문의 세 비용을 표기 그대로 구분합니다.
Static Keys (aka Jump Label Patching)
`````````````````````````````````````
Online fsck for XFS separates the regular filesystem from the checking and
repair code as much as possible.
However, there are a few parts of online fsck (such as the intent drains, and
later, live update hooks) where it is useful for the online fsck code to know
what's going on in the rest of the filesystem.
Since it is not expected that online fsck will be constantly running in the
background, it is very important to minimize the runtime overhead imposed by
these hooks when online fsck is compiled into the kernel but not actively
running on behalf of userspace.
Taking locks in the hot path of a writer thread to access a data structure only
to find that no further action is necessary is expensive -- on the author's
computer, this have an overhead of 40-50ns per access.
Fortunately, the kernel supports dynamic code patching, which enables XFS to
replace a static branch to hook code with ``nop`` sleds when online fsck isn't
running.
This sled has an overhead of however long it takes the instruction decoder to
skip past the sled, which seems to be on the order of less than 1ns and
does not access memory outside of instruction fetching.
Static key 전환 비용과 locking 제약
1787-1799Online fsck가 static key를 enable하면 sled가 hook code를 호출하는 unconditional branch로 바뀝니다. 이 전환은 매우 비싸서 ~22000ns가 걸리지만, 비용은 online fsck를 호출한 program이 전부 부담합니다. 여러 thread가 동시에 online fsck에 들어가거나 여러 filesystem을 동시에 검사하면 이 비용을 분산할 수 있습니다. Branch 방향을 바꾸려면 CPU hotplug lock을 잡아야 하고 CPU initialization에는 memory allocation이 필요하므로, online fsck는 memory reclaim path가 접근할 수 있는 lock이나 resource를 보유한 상태에서 static key를 변경하지 않도록 주의해야 합니다. CPU hotplug lock의 contention을 최소화하려면 static key를 불필요하게 enable하거나 disable하지 않아야 합니다.
When online fsck enables the static key, the sled is replaced with an
unconditional branch to call the hook code.
The switchover is quite expensive (~22000ns) but is paid entirely by the
program that invoked online fsck, and can be amortized if multiple threads
enter online fsck at the same time, or if multiple filesystems are being
checked at the same time.
Changing the branch direction requires taking the CPU hotplug lock, and since
CPU initialization requires memory allocation, online fsck must be careful not
to change a static key while holding any locks or resources that could be
accessed in the memory reclaim paths.
To minimize contention on the CPU hotplug lock, care should be taken not to
enable or disable static keys unnecessarily.
Static key 사용 규칙 네 가지
1800-1826Static key는 `xfs_scrub`가 실행되지 않을 때 일반 filesystem operation의 hook overhead를 최소화하기 위한 것이므로, 의도한 사용 pattern은 다음과 같습니다.
위치규칙
Hook을 둔 XFS code기본값이 false인 static scope의 static key를 선언합니다. `DEFINE_STATIC_KEY_FALSE` macro가 이를 처리하며, static key 자체는 `static` variable로 선언해야 합니다.
일반 filesystem 분기Scrub에서만 사용하는 code의 호출 여부를 결정할 때 `static_branch_unlikely` predicate를 호출하여 static key가 enable되지 않았으면 scrub 전용 hook code를 피합니다.
Exported helper일반 filesystem은 `static_branch_inc`로 static key를 enable하고 `static_branch_dec`로 disable하는 helper function을 export해야 합니다. Wrapper function을 사용하면 kernel distributor가 build time에 online fsck를 끌 때 관련 code를 쉽게 compile out할 수 있습니다.
Scrub setupScrub 전용 XFS 기능을 켜려는 scrub function은 setup function에서 `xchk_fsgates_enable`을 호출하여 특정 hook을 enable해야 합니다. Memory reclaim이 사용하는 resource를 얻기 전에 수행해야 하며, caller는 static key가 gate하는 기능이 실제로 필요한지 확실히 판단해야 합니다. 이때 `TRY_HARDER` flag가 유용합니다.
원문의 네 항목과 symbol·호출 위치를 보존합니다.
Because static keys are intended to minimize hook overhead for regular
filesystem operations when xfs_scrub is not running, the intended usage
patterns are as follows:
- The hooked part of XFS should declare a static-scoped static key that
defaults to false.
The ``DEFINE_STATIC_KEY_FALSE`` macro takes care of this.
The static key itself should be declared as a ``static`` variable.
- When deciding to invoke code that's only used by scrub, the regular
filesystem should call the ``static_branch_unlikely`` predicate to avoid the
scrub-only hook code if the static key is not enabled.
- The regular filesystem should export helper functions that call
``static_branch_inc`` to enable and ``static_branch_dec`` to disable the
static key.
Wrapper functions make it easy to compile out the relevant code if the kernel
distributor turns off online fsck at build time.
- Scrub functions wanting to turn on scrub-only XFS functionality should call
the ``xchk_fsgates_enable`` from the setup function to enable a specific
hook.
This must be done before obtaining any resources that are used by memory
reclaim.
Callers had better be sure they really need the functionality gated by the
static key; the ``TRY_HARDER`` flag is useful here.
TRY_HARDER 재시작과 teardown
1827-1842Online scrub에는 모든 scrubber function을 위해 AGI와 AGF buffer의 locking을 처리하는 resource acquisition helper가 있으며, `xchk_perag_lock`이 그 예입니다. Scrub와 실행 중인 transaction 사이의 conflict를 탐지하면 intent가 완료되기를 기다리려고 합니다. Helper caller가 static key를 enable하지 않았다면 helper는 `-EDEADLOCK`을 반환하고, 그 결과 `TRY_HARDER` flag를 설정한 상태로 scrub를 다시 시작해야 합니다. Scrub setup function은 이 flag를 탐지하고 static key를 enable한 뒤 scrub를 다시 시도해야 합니다. Scrub teardown은 `xchk_fsgates_enable`로 얻은 모든 static key를 disable합니다.
`xchk_perag_lock`이 AGI·AGF resource를 획득→Scrub와 실행 중인 transaction 사이 conflict 탐지→Static key가 enable되지 않았으면 `-EDEADLOCK` 반환→`TRY_HARDER` flag를 설정하여 scrub restart→Setup에서 `xchk_fsgates_enable`로 static key enable 후 재시도→Teardown에서 획득한 모든 static key disable
Conflict 탐지부터 setup 재시도와 teardown까지의 흐름입니다.
자세한 내용은 kernel 문서 `Documentation/staging/static-keys.rst`를 참조하십시오.
Online scrub has resource acquisition helpers (e.g. ``xchk_perag_lock``) to
handle locking AGI and AGF buffers for all scrubber functions.
If it detects a conflict between scrub and the running transactions, it will
try to wait for intents to complete.
If the caller of the helper has not enabled the static key, the helper will
return -EDEADLOCK, which should result in the scrub being restarted with the
``TRY_HARDER`` flag set.
The scrub setup function should detect that flag, enable the static key, and
try the scrub again.
Scrub teardown disables all static keys obtained by ``xchk_fsgates_enable``.
For more information, please see the kernel documentation of
Documentation/staging/static-keys.rst.
.. _xfile:
Pageable kernel memory와 xfile
1843-1878일부 online checking function은 filesystem을 scan하여 ondisk metadata structure의 shadow copy를 memory에 만들고 두 copy를 비교하는 방식으로 동작합니다. Online repair가 metadata structure를 재구축하려면 새 structure를 disk에 영속화하기 전에 그 structure에 저장할 record set을 먼저 계산해야 합니다. 이상적으로 repair는 새 data structure를 도입하는 단 한 번의 atomic commit으로 완료되어야 합니다. 이러한 목표를 달성하려면 kernel은 filesystem의 올바른 동작을 요구하지 않는 장소에 많은 양의 정보를 수집해야 합니다.
Kernel memory는 다음 이유로 적합하지 않습니다.
한계결과
Contiguous allocationC array를 만들기 위한 연속 memory 영역 할당은 특히 32-bit system에서 매우 어렵습니다.
Linked list overheadRecord linked list는 double pointer overhead가 매우 크고 indexed lookup 가능성을 없앱니다.
Pinned memoryKernel memory는 pinned되어 있으므로 system을 OOM 상태로 몰아갈 수 있습니다.
Insufficient capacitySystem에 모든 정보를 staging할 만큼 충분한 memory가 없을 수 있습니다.
원문의 네 한계를 각각 보존합니다.
Online fsck는 어느 한 시점에도 전체 record set을 memory에 유지할 필요가 없으므로, 필요하면 개별 record를 page out할 수 있습니다. Online fsck 개발을 계속하면서 indexed data storage 기능도 매우 유용하다는 사실이 드러났습니다. 다행히 Linux kernel에는 이미 byte-addressable하고 pageable한 storage인 tmpfs가 있습니다. In-kernel graphics driver, 그중에서도 특히 i915는 항상 memory에 둘 필요가 없는 중간 data를 저장하기 위해 tmpfs file을 활용하므로 사용 선례도 이미 확립되어 있습니다. 이렇게 `xfile`이 탄생했습니다.
Pageable Kernel Memory
----------------------
Some online checking functions work by scanning the filesystem to build a
shadow copy of an ondisk metadata structure in memory and comparing the two
copies.
For online repair to rebuild a metadata structure, it must compute the record
set that will be stored in the new structure before it can persist that new
structure to disk.
Ideally, repairs complete with a single atomic commit that introduces
a new data structure.
To meet these goals, the kernel needs to collect a large amount of information
in a place that doesn't require the correct operation of the filesystem.
Kernel memory isn't suitable because:
* Allocating a contiguous region of memory to create a C array is very
difficult, especially on 32-bit systems.
* Linked lists of records introduce double pointer overhead which is very high
and eliminate the possibility of indexed lookups.
* Kernel memory is pinned, which can drive the system into OOM conditions.
* The system might not have sufficient memory to stage all the information.
At any given time, online fsck does not need to keep the entire record set in
memory, which means that individual records can be paged out if necessary.
Continued development of online fsck demonstrated that the ability to perform
indexed data storage would also be very useful.
Fortunately, the Linux kernel already has a facility for byte-addressable and
pageable storage: tmpfs.
In-kernel graphics drivers (most notably i915) take advantage of tmpfs files
to store intermediate data that doesn't need to be in memory at all times, so
that usage precedent is already established.
Hence, the ``xfile`` was born!xfile 이전 구현의 세 세대
1879-1893세대접근 방식실패 원인
첫 번째Record를 발견하는 즉시 새 btree에 삽입구축 도중 filesystem이 shutdown되면 recovery 완료 뒤 불완전하게 구축된 data structure가 live 상태가 될 수 있었습니다.
두 번째Half-rebuilt structure 문제를 피하려고 모든 것을 memory에 저장System memory를 자주 고갈시켜 OOM을 일으켰습니다.
세 번째OOM 문제를 해결하려고 linked list 사용List pointer의 memory overhead가 극심했습니다.
원문의 ASCII 상자를 세 구현 세대와 실패 원인으로 구조화했습니다.
+--------------------------------------------------------------------------+
| **Historical Sidebar**: |
+--------------------------------------------------------------------------+
| The first edition of online repair inserted records into a new btree as |
| it found them, which failed because filesystem could shut down with a |
| built data structure, which would be live after recovery finished. |
| |
| The second edition solved the half-rebuilt structure problem by storing |
| everything in memory, but frequently ran the system out of memory. |
| |
| The third edition solved the OOM problem by using linked lists, but the |
| memory overhead of the list pointers was extreme. |
+--------------------------------------------------------------------------+
xfile의 다섯 access use case
1894-1917Xfile의 의도한 사용 방식을 조사한 결과 다음 use case가 도출되었습니다.
번호형태예시
1Fixed-sized record의 arraySpace management btree, directory entry, extended attribute entry
2Fixed-sized record의 sparse arrayQuota와 link count
3Variable size의 large binary object(BLOB)Directory와 extended attribute의 name 및 value
4Memory에 btree stagingReverse mapping btree
5임의의 contentRealtime space management
원문의 다섯 use case와 예시를 순서대로 보존합니다.
처음 네 use case를 지원하기 위해 high-level data structure가 xfile을 감싸서 online fsck function 사이에 기능을 공유합니다. 이 절의 나머지 부분은 xfile이 이 다섯 high-level data structure 중 네 가지에 제공하는 interface를 설명합니다. 다섯 번째 use case는 `realtime summary <rtsummary>` case study에서 다룹니다.
xfile Access Models
```````````````````
A survey of the intended uses of xfiles suggested these use cases:
1. Arrays of fixed-sized records (space management btrees, directory and
extended attribute entries)
2. Sparse arrays of fixed-sized records (quotas and link counts)
3. Large binary objects (BLOBs) of variable sizes (directory and extended
attribute names and values)
4. Staging btrees in memory (reverse mapping btrees)
5. Arbitrary contents (realtime space management)
To support the first four use cases, high level data structures wrap the xfile
to share functionality between online fsck functions.
The rest of this section discusses the interfaces that the xfile presents to
four of those five higher level data structures.
The fifth use case is discussed in the :ref:`realtime summary <rtsummary>` case
study.
완전한 record의 load와 store
1918-1925XFS는 record 중심성이 매우 강하므로 완전한 record를 load하고 store하는 기능이 중요합니다. 이러한 use case를 지원하기 위해 `xfile_load`와 `xfile_store` function 한 쌍을 제공합니다. 이들은 object를 xfile에서 읽고 xfile에 영속화하며, 어떤 error든 out of memory error로 취급합니다. Online repair에서는 error condition을 이 방식으로 하나로 축약해도 괜찮습니다. 가능한 반응은 operation을 userspace로 abort하는 것뿐이기 때문입니다.
XFS is very record-based, which suggests that the ability to load and store
complete records is important.
To support these cases, a pair of ``xfile_load`` and ``xfile_store``
functions are provided to read and persist objects into an xfile that treat any
error as an out of memory error. For online repair, squashing error conditions
in this manner is an acceptable behavior because the only reaction is to abort
the operation back to userspace.
Folio direct access와 memory reclaim
1926-1942File access 관용구를 논하면서 "mmap은 어떻게 하는가?"라는 질문을 빼놓을 수 없습니다. Userspace code가 일반 memory를 다루듯 pointer로 storage에 직접 접근하면 편리합니다. Online fsck는 system을 OOM 상태로 몰아가서는 안 되므로 xfile은 memory reclamation에 응답해야 합니다. Tmpfs는 pagecache folio가 pinned 상태도 아니고 locked 상태도 아닐 때만 그 folio를 swap cache로 밀어낼 수 있으므로, xfile은 너무 많은 folio를 pin해서는 안 됩니다.
Xfile content에 단기간 직접 접근할 때는 pagecache folio를 lock하고 kernel address space에 mapping합니다. Object load와 store는 이 mechanism을 사용합니다. Folio lock은 오랫동안 보유해서는 안 되므로, 장기간 직접 접근할 때는 folio refcount를 증가시키고 kernel address space에 mapping한 뒤 folio lock을 해제합니다. 이러한 장기 사용자는 folio를 언제 해제할지 알 수 있도록 shrinker infrastructure에 hook하여 memory reclaim에 반드시 응답해야 합니다.
접근Folio 처리Reclaim 조건
Short-termPagecache folio를 lock하고 kernel address space에 mappingObject load·store가 사용하며 lock을 짧게 보유
Long-termFolio refcount 증가, mapping 후 folio lock 해제Shrinker에 hook하여 reclaim 요청 시 folio 해제
Lock 보유 기간과 reclaim 의무를 구분합니다.
However, no discussion of file access idioms is complete without answering the
question, "But what about mmap?"
It is convenient to access storage directly with pointers, just like userspace
code does with regular memory.
Online fsck must not drive the system into OOM conditions, which means that
xfiles must be responsive to memory reclamation.
tmpfs can only push a pagecache folio to the swap cache if the folio is neither
pinned nor locked, which means the xfile must not pin too many folios.
Short term direct access to xfile contents is done by locking the pagecache
folio and mapping it into kernel address space. Object load and store uses this
mechanism. Folio locks are not supposed to be held for long periods of time, so
long term direct access to xfile contents is done by bumping the folio refcount,
mapping it into kernel address space, and dropping the folio lock.
These long term users *must* be responsive to memory reclaim by hooking into
the shrinker infrastructure to know when to release folios.
Folio lease API와 사용처
1943-1948`xfile_get_folio`와 `xfile_put_folio` function은 xfile 일부를 backing하는 locked folio를 가져오고 해제하기 위해 제공됩니다. 이 folio lease function을 사용하는 code는 xfarray의 `sorting <xfarray_sort>` algorithm과 `in-memory btrees <xfbtree>`뿐입니다.
The ``xfile_get_folio`` and ``xfile_put_folio`` functions are provided to
retrieve the (locked) folio that backs part of an xfile and to release it.
The only code to use these folio lease functions are the xfarray
:ref:`sorting<xfarray_sort>` algorithms and the :ref:`in-memory
btrees<xfbtree>`.
xfile 보안과 access coordination
1949-1976보안상의 이유로 xfile은 kernel이 private하게 소유해야 합니다. Security system의 간섭을 막기 위해 `S_PRIVATE`로 표시하고, process file descriptor table에 절대로 mapping해서는 안 되며, xfile page도 userspace process에 절대로 mapping해서는 안 됩니다.
VFS와의 locking recursion 문제를 피하기 위해 shmfs file에 대한 모든 access는 page cache를 직접 조작하여 수행합니다. Xfile writer는 xfile address space의 `->write_begin`과 `->write_end` function을 호출하여 writable page를 얻고 caller buffer를 page에 복사한 뒤 page를 해제합니다. Xfile reader는 `shmem_read_mapping_page_gfp`를 호출해 page를 직접 얻은 다음 content를 caller buffer에 복사합니다. 즉 xfile은 dummy `struct kiocb`를 만들지 않고 inode lock과 freeze lock도 잡지 않기 위해 VFS read/write code path를 무시합니다. Tmpfs는 freeze할 수 없고 xfile은 userspace에 노출해서는 안 됩니다.
경로API·동작목적
Writer`->write_begin`·`->write_end`; writable page 획득, buffer 복사, page 해제Page cache 직접 조작
Reader`shmem_read_mapping_page_gfp`; page 획득 후 caller buffer로 복사Page cache 직접 조작
VFS bypassVFS read/write path를 사용하지 않음Dummy `struct kiocb`, inode lock, freeze lock 회피
Writer·reader와 VFS bypass의 API 및 목적을 구분합니다.
Repair를 staging하기 위해 여러 thread가 하나의 xfile을 공유한다면 caller가 access coordination을 위한 자체 lock을 제공해야 합니다. 예를 들어 scrub function이 scan 결과를 xfile에 저장하고 다른 thread가 scan된 data의 update를 제공해야 한다면, scrub function은 모든 thread가 공유할 lock을 제공해야 합니다.
xfile Access Coordination
`````````````````````````
For security reasons, xfiles must be owned privately by the kernel.
They are marked ``S_PRIVATE`` to prevent interference from the security system,
must never be mapped into process file descriptor tables, and their pages must
never be mapped into userspace processes.
To avoid locking recursion issues with the VFS, all accesses to the shmfs file
are performed by manipulating the page cache directly.
xfile writers call the ``->write_begin`` and ``->write_end`` functions of the
xfile's address space to grab writable pages, copy the caller's buffer into the
page, and release the pages.
xfile readers call ``shmem_read_mapping_page_gfp`` to grab pages directly
before copying the contents into the caller's buffer.
In other words, xfiles ignore the VFS read and write code paths to avoid
having to create a dummy ``struct kiocb`` and to avoid taking inode and
freeze locks.
tmpfs cannot be frozen, and xfiles must not be exposed to userspace.
If an xfile is shared between threads to stage repairs, the caller must provide
its own locks to coordinate access.
For example, if a scrub function stores scan results in an xfile and needs
other threads to provide updates to the scanned data, the scrub function must
provide a lock for all threads to share.
.. _xfarray:
Fixed-sized record를 위한 xfarray
1977-1998XFS에서 indexed space metadata의 각 유형, 즉 free space, inode, reference count, file fork space, reverse mapping은 고정 크기 record 집합으로 구성되며 classic B+ tree로 index됩니다. Directory에는 name을 가리키는 고정 크기 dirent record 집합이 있고, extended attribute에는 name과 value를 가리키는 고정 크기 attribute key 집합이 있습니다. Quota counter와 file link counter는 number로 record를 index합니다. Repair 중 scrub는 gathering 단계에서 새 record를 staging하고 btree building 단계에서 이를 가져와야 합니다.
Xfile의 read·write method를 직접 호출해도 이 요구사항을 충족할 수 있지만, array offset 계산, iterator function 제공, sparse record와 sorting 처리를 맡는 high-level abstraction이 있으면 caller가 더 간단해집니다. `xfarray` abstraction은 byte-accessible xfile 위에 fixed-size record를 위한 linear array를 제공합니다.
Arrays of Fixed-Sized Records
`````````````````````````````
In XFS, each type of indexed space metadata (free space, inodes, reference
counts, file fork space, and reverse mappings) consists of a set of fixed-size
records indexed with a classic B+ tree.
Directories have a set of fixed-size dirent records that point to the names,
and extended attributes have a set of fixed-size attribute keys that point to
names and values.
Quota counters and file link counters index records with numbers.
During a repair, scrub needs to stage new records during the gathering step and
retrieve them during the btree building step.
Although this requirement can be satisfied by calling the read and write
methods of the xfile directly, it is simpler for callers for there to be a
higher level abstraction to take care of computing array offsets, to provide
iterator functions, and to deal with sparse records and sorting.
The ``xfarray`` abstraction presents a linear array for fixed-size records atop
the byte-accessible xfile.
.. _xfarray_access_patterns:
세 가지 array access pattern
1999-2042Online fsck의 array access pattern은 대체로 세 범주로 나뉩니다. 모든 경우에 record iteration이 필요하다고 가정하며, iteration은 다음 절에서 다룹니다.
첫 번째 caller 유형은 position으로 index되는 record를 처리합니다. Record 사이에 gap이 있을 수 있고 collection 단계에서 하나의 record가 여러 번 update될 수 있습니다. 즉 이 caller는 sparse하고 linear address를 사용하는 table file을 원합니다. 대표 use case는 quota record와 file link count record입니다. `xfarray_load`와 `xfarray_store` function이 같은 이름의 xfile function을 감싸서 임의의 array index에 있는 element를 load하고 store합니다. Gap은 null record로 정의하고, null record는 모든 byte가 0인 sequence로 정의합니다. `xfarray_element_is_null`을 호출해 null record를 탐지합니다. 기존 record를 null로 만드는 `xfarray_unset`을 호출하거나, 해당 array index에 아무것도 store하지 않는 방식으로 null record를 만듭니다.
두 번째 caller 유형은 position으로 index되지 않고 record를 여러 번 update할 필요가 없는 record를 처리합니다. 대표 use case는 space btree와 key/value btree의 rebuild입니다. 이 caller는 array index를 신경 쓰지 않고 `xfarray_append` function으로 record를 array 끝에 추가할 수 있습니다. Btree data rebuild처럼 record를 특정 순서로 제시해야 하는 caller는 `xfarray_sort` function으로 record를 정렬할 수 있으며, 이 function은 뒤에서 설명합니다.
세 번째 caller 유형은 record 수를 세는 데 유용한 bag입니다. 대표 use case는 reverse mapping information에서 space extent reference count를 구성하는 작업입니다. Record를 어떤 순서로든 bag에 넣을 수 있고 언제든 제거할 수 있으며, record uniqueness 보장은 caller에게 맡깁니다. `xfarray_store_anywhere` function은 bag의 임의 null record slot에 record를 삽입하고, `xfarray_unset` function은 bag에서 record를 제거합니다.
유형Record 특성주요 API·use case
Positional sparse tablePosition index, gap 허용, 반복 update`xfarray_load`·`xfarray_store`·`xfarray_element_is_null`·`xfarray_unset`; quota·link count
Append and sortPosition index 없음, 반복 update 불필요`xfarray_append`·`xfarray_sort`; space·key/value btree rebuild
Counting bag순서 자유, 제거 가능, uniqueness는 caller 책임`xfarray_store_anywhere`·`xfarray_unset`; extent reference count
세 caller 유형과 대표 API·용도를 연결합니다.
제안된 patchset은 `big in-memory array <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array>`_입니다.
Array Access Patterns
^^^^^^^^^^^^^^^^^^^^^
Array access patterns in online fsck tend to fall into three categories.
Iteration of records is assumed to be necessary for all cases and will be
covered in the next section.
The first type of caller handles records that are indexed by position.
Gaps may exist between records, and a record may be updated multiple times
during the collection step.
In other words, these callers want a sparse linearly addressed table file.
The typical use case are quota records or file link count records.
Access to array elements is performed programmatically via ``xfarray_load`` and
``xfarray_store`` functions, which wrap the similarly-named xfile functions to
provide loading and storing of array elements at arbitrary array indices.
Gaps are defined to be null records, and null records are defined to be a
sequence of all zero bytes.
Null records are detected by calling ``xfarray_element_is_null``.
They are created either by calling ``xfarray_unset`` to null out an existing
record or by never storing anything to an array index.
The second type of caller handles records that are not indexed by position
and do not require multiple updates to a record.
The typical use case here is rebuilding space btrees and key/value btrees.
These callers can add records to the array without caring about array indices
via the ``xfarray_append`` function, which stores a record at the end of the
array.
For callers that require records to be presentable in a specific order (e.g.
rebuilding btree data), the ``xfarray_sort`` function can arrange the sorted
records; this function will be covered later.
The third type of caller is a bag, which is useful for counting records.
The typical use case here is constructing space extent reference counts from
reverse mapping information.
Records can be put in the bag in any order, they can be removed from the bag
at any time, and uniqueness of records is left to callers.
The ``xfarray_store_anywhere`` function is used to insert a record in any
null record slot in the bag; and the ``xfarray_unset`` function removes a
record from the bag.
The proposed patchset is the
`big in-memory array
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array>`_.
Array element iteration
2043-2076대부분의 xfarray 사용자는 array에 저장된 record를 iterate할 수 있어야 합니다. Caller는 다음과 같이 가능한 모든 array index를 probe할 수 있습니다.
.. code-block:: c
xfarray_idx_t i;
foreach_xfarray_idx(array, i) {
xfarray_load(array, i, &rec);
/* do something with rec */
}이 관용구를 사용하는 모든 code는 null record를 처리할 준비가 되어 있거나 null record가 없다는 사실을 이미 알고 있어야 합니다.
Sparse array를 iterate하려는 xfarray 사용자를 위해 `xfarray_iter` function은 `xfile_seek_data`를 호출하여 xfarray에서 한 번도 write되지 않은 index를 무시합니다. `xfile_seek_data`는 내부적으로 `SEEK_DATA`를 사용하여 memory page가 채워지지 않은 array 영역을 건너뜁니다. Page를 찾은 뒤에는 그 page에서 zero로 채워진 영역도 건너뜁니다.
.. code-block:: c
xfarray_idx_t i = XFARRAY_CURSOR_INIT;
while ((ret = xfarray_iter(array, &i, &rec)) == 1) {
/* do something with rec */
}경로탐색 방식Caller 의무
`foreach_xfarray_idx`가능한 모든 array index를 probe하고 `xfarray_load` 호출Null record 처리 또는 null이 없음을 사전에 보장
`xfarray_iter``xfile_seek_data`·`SEEK_DATA`로 미기록 page를 건너뛰고 page 내부 zero 영역도 skipReturn value 1인 동안 cursor를 이어서 처리
전체 index scan과 sparse iterator의 skip 규칙을 구분합니다.
Iterating Array Elements
^^^^^^^^^^^^^^^^^^^^^^^^
Most users of the xfarray require the ability to iterate the records stored in
the array.
Callers can probe every possible array index with the following:
.. code-block:: c
xfarray_idx_t i;
foreach_xfarray_idx(array, i) {
xfarray_load(array, i, &rec);
/* do something with rec */
}
All users of this idiom must be prepared to handle null records or must already
know that there aren't any.
For xfarray users that want to iterate a sparse array, the ``xfarray_iter``
function ignores indices in the xfarray that have never been written to by
calling ``xfile_seek_data`` (which internally uses ``SEEK_DATA``) to skip areas
of the array that are not populated with memory pages.
Once it finds a page, it will skip the zeroed areas of the page.
.. code-block:: c
xfarray_idx_t i = XFARRAY_CURSOR_INIT;
while ((ret = xfarray_iter(array, &i, &rec)) == 1) {
/* do something with rec */
}
.. _xfarray_sort:
Btree bulk loading을 위한 sorting
2077-2087Online repair의 네 번째 demonstration에서 community reviewer는 성능을 위해 record를 한 번에 하나씩 새 btree에 삽입하는 대신 record batch를 btree record block에 load해야 한다고 지적했습니다. XFS의 btree insertion code는 record의 올바른 순서를 유지할 책임이 있으므로, xfarray도 bulk loading 전에 record set을 sorting할 수 있어야 합니다.
Sorting Array Elements
^^^^^^^^^^^^^^^^^^^^^^
During the fourth demonstration of online repair, a community reviewer remarked
that for performance reasons, online repair ought to load batches of records
into btree record blocks instead of inserting records into a new btree one at a
time.
The btree insertion code in XFS is responsible for maintaining correct ordering
of the records, so naturally the xfarray must also support sorting the record
set prior to bulk loading.
Adaptive quicksort와 heapsort 결합
2088-2114Xfarray의 sorting algorithm은 Linux kernel에 맞게 조정한 adaptive quicksort와 heapsort subalgorithm의 조합이며, `Sedgewick <https://algs4.cs.princeton.edu/23quicksort/>`_ 및 `pdqsort <https://github.com/orlp/pdqsort>`_의 정신을 따릅니다. 합리적으로 짧은 시간 안에 record를 sorting하기 위해 `xfarray`는 quicksort가 제공하는 binary subpartitioning을 활용하면서도, 선택한 quicksort pivot이 나쁠 때 성능이 붕괴하는 상황에 대비해 heapsort도 사용합니다. 두 algorithm은 일반적으로 모두 O(n * lg(n))이지만 두 구현의 성능에는 큰 차이가 있습니다.
Linux kernel에는 이미 상당히 빠른 heapsort 구현이 있지만 일반 C array에서만 동작하므로 활용 범위가 제한됩니다. Xfarray는 이 구현을 두 핵심 위치에서 사용합니다.
사용처동작
Single-page subset하나의 xfile page가 backing하는 임의의 record subset을 sorting
Buffered subsetXfarray의 서로 떨어진 부분에서 소수의 record를 memory buffer로 load하고 그 buffer를 sorting
원문의 두 사용처와 입력 위치를 보존합니다.
즉 `xfarray`는 heapsort를 사용하여 quicksort의 nested recursion을 제한하고, 그 결과 quicksort의 최악 실행시간 동작을 완화합니다.
Case Study: Sorting xfarrays
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The sorting algorithm used in the xfarray is actually a combination of adaptive
quicksort and a heapsort subalgorithm in the spirit of
`Sedgewick <https://algs4.cs.princeton.edu/23quicksort/>`_ and
`pdqsort <https://github.com/orlp/pdqsort>`_, with customizations for the Linux
kernel.
To sort records in a reasonably short amount of time, ``xfarray`` takes
advantage of the binary subpartitioning offered by quicksort, but it also uses
heapsort to hedge against performance collapse if the chosen quicksort pivots
are poor.
Both algorithms are (in general) O(n * lg(n)), but there is a wide performance
gulf between the two implementations.
The Linux kernel already contains a reasonably fast implementation of heapsort.
It only operates on regular C arrays, which limits the scope of its usefulness.
There are two key places where the xfarray uses it:
* Sorting any record subset backed by a single xfile page.
* Loading a small number of xfarray records from potentially disparate parts
of the xfarray into a memory buffer, and sorting the buffer.
In other words, ``xfarray`` uses heapsort to constrain the nested recursion of
quicksort, thereby mitigating quicksort's worst runtime behavior.
Pivot 선택과 partition 최적화
2115-2150Quicksort pivot을 고르는 일은 까다롭습니다. 좋은 pivot은 sorting할 집합을 절반으로 나누어 O(n * lg(n)) 성능에 핵심적인 divide-and-conquer 동작을 이끌어냅니다. 나쁜 pivot은 subset을 거의 나누지 못해 O(n\ :sup:`2`) 실행시간으로 이어집니다. Xfarray sorting routine은 나쁜 pivot을 피하려고 record 9개를 memory buffer에 sample하고 kernel heapsort를 사용하여 아홉 값의 median을 찾습니다.
현대 quicksort 구현 대부분은 classic C array에서 pivot을 선택하기 위해 Tukey의 "ninther"를 사용합니다. 일반적인 ninther 구현은 서로 겹치지 않는 record triad 세 개를 선택해 각 triad를 sorting한 다음, 각 triad의 middle value를 다시 sorting하여 ninther value를 결정합니다. 그러나 앞에서 설명했듯 xfile access에는 무시할 수 없는 비용이 듭니다. 아홉 element를 memory buffer로 읽고 kernel의 in-memory heapsort를 buffer에 실행한 뒤 buffer의 4th element를 pivot으로 선택하는 방식이 훨씬 빠른 것으로 드러났습니다. Tukey ninther는 J. W. Tukey의 `The ninther, a technique for low-effort robust (resistant) location in large samples`, H. David 편집, *Contributions to Survey Sampling and Applied Statistics*, Academic Press, 1978, pp. 251–257에 설명되어 있습니다.
방식절차xfile 관점
Classic Tukey ninther서로 다른 triad 3개를 각각 sorting하고 각 middle value를 다시 sorting여러 xfile access가 필요
xfarray buffered sampleRecord 9개를 한 번에 memory buffer로 읽어 heapsort 후 4th element 선택Page access 비용을 줄여 더 빠름
Classic ninther와 xfarray의 buffered 변형을 비교합니다.
Quicksort partition은 상당히 교과서적인 방식입니다. Pivot을 기준으로 record subset을 재배열한 다음, 현재 stack frame과 다음 stack frame이 각각 pivot의 큰 절반과 작은 절반을 sorting하도록 설정합니다. 이 방식은 stack space 요구량을 log2(record count)로 유지합니다.
마지막 성능 최적화로 quicksort의 hi·lo scanning 단계는 map/unmap cycle을 줄이기 위해 조사한 xfile page를 가능한 한 오래 kernel에 mapping한 채 유지합니다. 놀랍게도 xfile page에 heapsort를 직접 적용한 효과까지 반영한 뒤에도 이 최적화는 전체 sorting 실행시간을 거의 절반만큼 추가로 줄입니다.
Choosing a quicksort pivot is a tricky business.
A good pivot splits the set to sort in half, leading to the divide and conquer
behavior that is crucial to O(n * lg(n)) performance.
A poor pivot barely splits the subset at all, leading to O(n\ :sup:`2`)
runtime.
The xfarray sort routine tries to avoid picking a bad pivot by sampling nine
records into a memory buffer and using the kernel heapsort to identify the
median of the nine.
Most modern quicksort implementations employ Tukey's "ninther" to select a
pivot from a classic C array.
Typical ninther implementations pick three unique triads of records, sort each
of the triads, and then sort the middle value of each triad to determine the
ninther value.
As stated previously, however, xfile accesses are not entirely cheap.
It turned out to be much more performant to read the nine elements into a
memory buffer, run the kernel's in-memory heapsort on the buffer, and choose
the 4th element of that buffer as the pivot.
Tukey's ninthers are described in J. W. Tukey, `The ninther, a technique for
low-effort robust (resistant) location in large samples`, in *Contributions to
Survey Sampling and Applied Statistics*, edited by H. David, (Academic Press,
1978), pp. 251–257.
The partitioning of quicksort is fairly textbook -- rearrange the record
subset around the pivot, then set up the current and next stack frames to
sort with the larger and the smaller halves of the pivot, respectively.
This keeps the stack space requirements to log2(record count).
As a final performance optimization, the hi and lo scanning phase of quicksort
keeps examined xfile pages mapped in the kernel for as long as possible to
reduce map/unmap cycles.
Surprisingly, this reduces overall sort runtime by nearly half again after
accounting for the application of heapsort directly onto xfile pages.
.. _xfblob:
가변 길이 object를 위한 xfblob
2151-2178Extended attribute와 directory는 record를 staging할 때 유한한 길이의 임의 byte sequence를 다뤄야 한다는 요구사항을 추가합니다. 각 directory entry record에는 entry name을 저장해야 하고, 각 extended attribute에는 attribute name과 value를 모두 저장해야 합니다. 이러한 name, key, value는 많은 memory를 소비할 수 있으므로 xfile 위에서 blob을 간단히 관리하도록 `xfblob` abstraction을 만들었습니다.
Blob array는 object를 가져오고 영속화하는 `xfblob_load`와 `xfblob_store` function을 제공합니다. Store function은 영속화한 object마다 magic cookie를 반환하고, caller는 나중에 이 cookie를 원문에 표기된 `xblob_load`에 전달하여 object를 다시 불러옵니다. `xfblob_free`는 특정 blob을 해제하며, compaction이 필요하지 않으므로 `xfblob_truncate`는 모든 blob을 해제합니다.
단계Function·값동작
Store`xfblob_store` → magic cookieObject를 영속화하고 이후 lookup에 쓸 cookie 반환
Load`xfblob_load`; 원문의 후속 표기는 `xblob_load`Cookie로 저장된 object를 다시 불러옴
Free one`xfblob_free`특정 blob 해제
Free all`xfblob_truncate`Compaction 없이 모든 blob 해제
원문에 명시된 blob operation과 cookie 흐름을 보존합니다.
Directory와 extended attribute repair의 자세한 내용은 atomic file content exchange를 다루는 뒤 절에서 설명합니다. 다만 이 repair function은 temporary ondisk file에 entry를 추가하기 전에 소수의 entry만 blob storage에 cache하므로 compaction이 필요하지 않습니다.
제안된 patchset은 `extended attribute repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series의 시작 부분에 있습니다.
Blob Storage
````````````
Extended attributes and directories add an additional requirement for staging
records: arbitrary byte sequences of finite length.
Each directory entry record needs to store entry name,
and each extended attribute needs to store both the attribute name and value.
The names, keys, and values can consume a large amount of memory, so the
``xfblob`` abstraction was created to simplify management of these blobs
atop an xfile.
Blob arrays provide ``xfblob_load`` and ``xfblob_store`` functions to retrieve
and persist objects.
The store function returns a magic cookie for every object that it persists.
Later, callers provide this cookie to the ``xblob_load`` to recall the object.
The ``xfblob_free`` function frees a specific blob, and the ``xfblob_truncate``
function frees them all because compaction is not needed.
The details of repairing directories and extended attributes will be discussed
in a subsequent section about atomic file content exchanges.
However, it should be noted that these repair functions only use blob storage
to cache a small number of entries before adding them to a temporary ondisk
file, which is why compaction is not required.
The proposed patchset is at the start of the
`extended attribute repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series.
In-memory B+tree와 live update 반영
2179-2220`Secondary metadata <secondary_metadata>` 장에서 설명했듯 secondary metadata의 checking과 repair에는 일반적으로 filesystem의 live metadata scan과 그 metadata를 update하는 writer thread 사이의 coordination이 필요합니다. Scan data를 최신 상태로 유지하려면 filesystem metadata update를 scan이 수집하는 data로 전달할 수 있어야 합니다.
한 방법은 concurrent update를 별도 log file에 append한 뒤 새 metadata를 disk에 쓰기 전에 적용하는 것입니다. 하지만 system의 나머지 부분이 매우 바쁘면 memory consumption이 제한 없이 증가합니다. 다른 방법은 side-log를 생략하고 filesystem의 live update를 scan data에 직접 commit하는 것입니다. 이 방식은 overhead가 더 큰 대신 maximum memory requirement가 낮습니다. 어느 전략이든 scan result를 보관하는 data structure는 좋은 성능을 위해 indexed access를 지원해야 합니다.
전략반영 방식Trade-off
Side-logConcurrent update를 별도 log file에 append하고 새 metadata의 disk write 전에 적용Busy system에서는 memory consumption이 제한 없이 증가할 수 있음
Direct commitFilesystem의 live update를 scan data에 직접 commitOverhead는 더 크지만 maximum memory requirement가 낮음
동시 update를 scan data에 반영하는 두 전략의 비용을 비교합니다.
두 전략 모두 indexed lookup이 필요하므로 online fsck는 live update를 scan data에 직접 commit하는 두 번째 전략을 사용합니다. Xfarray는 index가 없고 record ordering도 강제하지 않으므로 이 작업에 적합하지 않습니다. 다행히 XFS에는 ordered reverse mapping record를 만들고 유지하는 기존 rmap btree code가 있습니다. 이 btree를 memory에 만들 수 있으면 됩니다.
`xfile <xfile>` abstraction은 memory page를 regular file로 나타내므로 kernel은 byte-addressable 또는 block-addressable virtual address space를 필요에 따라 만들 수 있습니다. XFS buffer cache는 block-oriented address space에 대한 IO를 추상화하는 데 특화되어 있습니다. 따라서 buffer cache가 xfile과 interface하도록 조정하면 전체 btree library를 재사용할 수 있습니다. Xfile 위에 구축한 btree를 통틀어 `xfbtree`라고 하며, 이어지는 절에서 실제 동작을 설명합니다.
제안된 patchset은 `in-memory btree <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=in-memory-btrees>`_ series입니다.
.. _xfbtree:
In-Memory B+Trees
`````````````````
The chapter about :ref:`secondary metadata<secondary_metadata>` mentioned that
checking and repairing of secondary metadata commonly requires coordination
between a live metadata scan of the filesystem and writer threads that are
updating that metadata.
Keeping the scan data up to date requires the ability to propagate
metadata updates from the filesystem into the data being collected by the scan.
This *can* be done by appending concurrent updates into a separate log file and
applying them before writing the new metadata to disk, but this leads to
unbounded memory consumption if the rest of the system is very busy.
Another option is to skip the side-log and commit live updates from the
filesystem directly into the scan data, which trades more overhead for a lower
maximum memory requirement.
In both cases, the data structure holding the scan results must support indexed
access to perform well.
Given that indexed lookups of scan data is required for both strategies, online
fsck employs the second strategy of committing live updates directly into
scan data.
Because xfarrays are not indexed and do not enforce record ordering, they
are not suitable for this task.
Conveniently, however, XFS has a library to create and maintain ordered reverse
mapping records: the existing rmap btree code!
If only there was a means to create one in memory.
Recall that the :ref:`xfile <xfile>` abstraction represents memory pages as a
regular file, which means that the kernel can create byte or block addressable
virtual address spaces at will.
The XFS buffer cache specializes in abstracting IO to block-oriented address
spaces, which means that adaptation of the buffer cache to interface with
xfiles enables reuse of the entire btree library.
Btrees built atop an xfile are collectively known as ``xfbtrees``.
The next few sections describe how they actually work.
The proposed patchset is the
`in-memory btree
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=in-memory-btrees>`_
series.xfile을 buffer cache target으로 사용
2221-2239Xfile을 buffer cache target으로 지원하려면 두 가지를 수정해야 합니다. 첫째, 일반적으로 per-AG structure가 보유하는 `struct xfs_buf` rhashtable을 `struct xfs_buftarg` structure가 host할 수 있어야 합니다. 둘째, buffer `ioapply` function을 수정하여 cached page를 xfile에서 "read"하고 xfile로 다시 "write"해야 합니다.
변경대상역할
Hashtable hosting`struct xfs_buftarg` · `struct xfs_buf` rhashtable기존 per-AG 보관 위치 대신 buffer target이 rhashtable을 host
Cached-page IOBuffer `ioapply`Cached page를 xfile에서 "read"하고 xfile로 "write"
필요한 두 변경과 담당 역할을 구분합니다.
Xfile 자체는 locking을 제공하지 않으므로 개별 buffer에 대한 concurrent access는 `xfs_buf` lock으로 제어합니다. 이 adaptation을 적용하면 xfile-backed buffer cache 사용자와 disk-backed buffer cache 사용자가 정확히 같은 API를 사용합니다.
Xfile과 buffer cache가 page를 공유하지 않아 분리 구조는 memory usage를 늘립니다. 다만 이 특성은 언젠가 in-memory btree에 transactional update를 가능하게 할 수 있습니다. 현재로서는 새 code를 작성할 필요를 없애는 역할만 합니다.
Using xfiles as a Buffer Cache Target
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Two modifications are necessary to support xfiles as a buffer cache target.
The first is to make it possible for the ``struct xfs_buftarg`` structure to
host the ``struct xfs_buf`` rhashtable, because normally those are held by a
per-AG structure.
The second change is to modify the buffer ``ioapply`` function to "read" cached
pages from the xfile and "write" cached pages back to the xfile.
Multiple access to individual buffers is controlled by the ``xfs_buf`` lock,
since the xfile does not provide any locking on its own.
With this adaptation in place, users of the xfile-backed buffer cache use
exactly the same APIs as users of the disk-backed buffer cache.
The separation between xfile and buffer cache implies higher memory usage since
they do not share pages, but this property could some day enable transactional
updates to an in-memory btree.
Today, however, it simply eliminates the need for new code.
xfbtree 공간 관리
2240-2260Xfile의 space management는 매우 단순하여 btree block 하나가 memory page 하나와 같은 크기입니다. 이 block은 on-disk btree와 같은 header format을 사용하지만, in-memory block verifier는 xfile memory가 일반 DRAM보다 corruption에 더 취약하지 않다고 가정하여 checksum을 무시합니다. 여기서는 절대적인 memory efficiency보다 기존 code 재사용이 더 중요합니다.
Xfbtree를 backing하는 xfile의 맨 첫 block에는 header block이 들어갑니다. 이 header는 owner, height, root xfbtree block의 block number를 기술합니다.
Btree block을 allocate할 때는 `xfile_seek_data`로 file의 gap을 찾습니다. Gap이 없으면 xfile length를 늘려 하나를 만들고, `xfile_prealloc`으로 block space를 preallocate한 뒤 location을 반환합니다. Xfbtree block을 free할 때는 내부적으로 `FALLOC_FL_PUNCH_HOLE`을 사용하는 `xfile_discard`로 xfile에서 memory page를 제거합니다.
작업API·조건결과
Allocate`xfile_seek_data`; gap이 없으면 xfile length 확장 → `xfile_prealloc`Block space를 preallocate하고 location 반환
Free`xfile_discard` → `FALLOC_FL_PUNCH_HOLE`Xfile에서 해당 memory page 제거
Block allocation과 해제의 API 순서를 보존합니다.
Space Management with an xfbtree
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Space management for an xfile is very simple -- each btree block is one memory
page in size.
These blocks use the same header format as an on-disk btree, but the in-memory
block verifiers ignore the checksums, assuming that xfile memory is no more
corruption-prone than regular DRAM.
Reusing existing code here is more important than absolute memory efficiency.
The very first block of an xfile backing an xfbtree contains a header block.
The header describes the owner, height, and the block number of the root
xfbtree block.
To allocate a btree block, use ``xfile_seek_data`` to find a gap in the file.
If there are no gaps, create one by extending the length of the xfile.
Preallocate space for the block with ``xfile_prealloc``, and hand back the
location.
To free an xfbtree block, use ``xfile_discard`` (which internally uses
``FALLOC_FL_PUNCH_HOLE``) to remove the memory page from the xfile.
xfbtree 생성과 사용 절차
2261-2294Online fsck function이 xfbtree를 만들려면 다음 여섯 단계를 순서대로 수행합니다.
단계호출·대상동작
1`xfile_create`Xfile 생성
2`xfs_alloc_memory_buftarg`Xfile을 가리키는 buffer cache target structure 생성
3`xfbtree_init`; `struct xfbtree`; `xfs_rmapbt_mem_create`Buffer cache target, buffer ops, 기타 정보를 전달해 object를 initialize하고 initial root block을 기록; btree type별 wrapper가 필요한 argument를 제공
4`xfs_rmapbt_mem_cursor` 등 btree cursor creation functionXfbtree object를 해당 btree type의 cursor creation function에 전달
5Regular btree function; `xfs_rmap_*`; `next section <xfbtree_commit>`Cursor로 in-memory btree를 query·update하고 transaction에 log된 update 처리는 다음 절을 참조
6Btree cursor → xfbtree object → buffer target → xfile순서대로 delete·destroy·free·destroy하여 모든 resource 해제
원문의 1–6단계와 API·resource lifecycle을 그대로 연결합니다.
Populating an xfbtree
^^^^^^^^^^^^^^^^^^^^^
An online fsck function that wants to create an xfbtree should proceed as
follows:
1. Call ``xfile_create`` to create an xfile.
2. Call ``xfs_alloc_memory_buftarg`` to create a buffer cache target structure
pointing to the xfile.
3. Pass the buffer cache target, buffer ops, and other information to
``xfbtree_init`` to initialize the passed in ``struct xfbtree`` and write an
initial root block to the xfile.
Each btree type should define a wrapper that passes necessary arguments to
the creation function.
For example, rmap btrees define ``xfs_rmapbt_mem_create`` to take care of
all the necessary details for callers.
4. Pass the xfbtree object to the btree cursor creation function for the
btree type.
Following the example above, ``xfs_rmapbt_mem_cursor`` takes care of this
for callers.
5. Pass the btree cursor to the regular btree functions to make queries against
and to update the in-memory btree.
For example, a btree cursor for an rmap xfbtree can be passed to the
``xfs_rmap_*`` functions just like any other btree cursor.
See the :ref:`next section<xfbtree_commit>` for information on dealing with
xfbtree updates that are logged to a transaction.
6. When finished, delete the btree cursor, destroy the xfbtree object, free the
buffer target, and the destroy the xfile to release all resources.
Log된 xfbtree buffer의 commit
2295-2332Staging structure를 처리하기 위해 rmap btree code를 재사용하는 방식은 영리하지만, in-memory btree block storage의 ephemeral한 성격은 별도의 문제를 만듭니다. Log format은 data device 이외의 device update를 이해하지 못하므로 XFS transaction manager는 xfile-backed buffer의 buffer log item을 commit해서는 안 됩니다. Ephemeral xfbtree는 AIL이 log transaction을 filesystem에 checkpoint할 때에는 이미 존재하지 않을 가능성이 크고, log recovery 중에는 확실히 존재하지 않습니다.
따라서 transaction context에서 xfbtree를 update하는 code는 transaction을 commit하거나 cancel하기 전에 buffer log item을 transaction에서 제거하고 update를 backing xfile에 기록해야 합니다. `xfbtree_trans_commit`과 `xfbtree_trans_cancel`은 이 기능을 다음 순서로 구현합니다.
단계동작조건·결과
1Buffer가 xfile을 target으로 하는 각 buffer log item 탐색대상 log item 식별
2Log item의 dirty/ordered status 기록기존 상태 보존
3Buffer에서 log item detachTransaction과 xfile-backed buffer의 log 연결 제거
4Buffer를 special delwri list에 queueXfile write 준비
5Transaction dirty flag 정리Dirty log item이 3단계에서 detach한 항목뿐일 때 clear
6Delwri list submitUpdate를 commit하는 경우에만 변경을 xfile에 commit
원문의 1–6단계와 commit·cancel 분기를 순서대로 보존합니다.
이 방식으로 transaction에서 xfile logged buffer를 제거한 뒤에는 transaction을 정상적으로 commit하거나 cancel할 수 있습니다.
.. _xfbtree_commit:
Committing Logged xfbtree Buffers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Although it is a clever hack to reuse the rmap btree code to handle the staging
structure, the ephemeral nature of the in-memory btree block storage presents
some challenges of its own.
The XFS transaction manager must not commit buffer log items for buffers backed
by an xfile because the log format does not understand updates for devices
other than the data device.
An ephemeral xfbtree probably will not exist by the time the AIL checkpoints
log transactions back into the filesystem, and certainly won't exist during
log recovery.
For these reasons, any code updating an xfbtree in transaction context must
remove the buffer log items from the transaction and write the updates into the
backing xfile before committing or cancelling the transaction.
The ``xfbtree_trans_commit`` and ``xfbtree_trans_cancel`` functions implement
this functionality as follows:
1. Find each buffer log item whose buffer targets the xfile.
2. Record the dirty/ordered status of the log item.
3. Detach the log item from the buffer.
4. Queue the buffer to a special delwri list.
5. Clear the transaction dirty flag if the only dirty log items were the ones
that were detached in step 3.
6. Submit the delwri list to commit the changes to the xfile, if the updates
are being committed.
After removing xfile logged buffers from the transaction in this manner, the
transaction can be committed or cancelled.
Ondisk B+tree bulk loading
2333-2353초기 online repair는 새 btree를 만들고 observation을 하나씩 추가하여 새 btree structure를 구축했습니다. Btree를 record 하나씩 load하면 commit 전에 incore record를 sorting하지 않아도 된다는 작은 이점이 있었지만, 매우 느렸고 repair 도중 system이 중단되면 block을 leak했습니다. 또한 새 btree block의 loading factor를 repair가 제어할 수 없었습니다.
방식이점문제·개선
One record at a timeCommit 전에 incore record sorting 불필요매우 느림; 중단 시 block leak; loading factor 제어 불가
Bulk btree loadingRecord collection에서 btree index를 효율적으로 rebuild`xfs_repair`의 btree type별 copy-pasted 구현을 generic mechanism으로 통합
초기 개별 삽입과 bulk loading의 동작 차이를 정리합니다.
다행히 오래된 `xfs_repair` tool에는 record collection에서 btree index를 rebuild하는 더 효율적인 방법인 bulk btree loading이 있었습니다. 다만 `xfs_repair`에는 btree type마다 별도의 copy-pasted implementation이 있어 code 측면에서는 비효율적이었습니다.
Online fsck를 준비하면서 네 bulk loader를 각각 연구하고 note를 작성한 뒤, 네 구현을 하나의 generic btree bulk loading mechanism으로 refactor했습니다. 이어지는 내용은 당시 note를 새로 갱신하여 제시한 것입니다.
Bulk Loading of Ondisk B+Trees
------------------------------
As mentioned previously, early iterations of online repair built new btree
structures by creating a new btree and adding observations individually.
Loading a btree one record at a time had a slight advantage of not requiring
the incore records to be sorted prior to commit, but was very slow and leaked
blocks if the system went down during a repair.
Loading records one at a time also meant that repair could not control the
loading factor of the blocks in the new btree.
Fortunately, the venerable ``xfs_repair`` tool had a more efficient means for
rebuilding a btree index from a collection of records -- bulk btree loading.
This was implemented rather inefficiently code-wise, since ``xfs_repair``
had separate copy-pasted implementations for each btree type.
To prepare for online fsck, each of the four bulk loaders were studied, notes
were taken, and the four were refactored into a single generic btree bulk
loading mechanism.
Those notes in turn have been refreshed and are presented below.
Bulk loader geometry 계산
2354-2428Bulk loading의 zeroth step은 새 btree에 저장할 전체 record set을 모아 sorting하는 것입니다. 이어서 record set, btree type, load factor preference를 `xfs_btree_bload_compute_geometry`에 전달하여 btree shape를 계산합니다. 이 정보는 resource reservation에 필요합니다.
먼저 btree block과 block header의 size를 사용해 leaf block에 들어갈 minimum·maximum record 수를 계산합니다. Maximum은 `maxrecs = (block_size - header_size) / record_size`입니다. XFS design은 가능하면 btree block을 merge하도록 규정하므로 minimum은 `minrecs = maxrecs / 2`입니다.
원하는 loading factor는 minrecs 이상 maxrecs 이하여야 합니다. Minrecs를 고르면 block의 절반을 낭비하고, maxrecs를 고르면 rebuild 직후 각 leaf block에 record 하나만 추가해도 tree split이 발생하여 성능이 눈에 띄게 떨어집니다. 기본값은 maxrecs의 75%인 `default_load_factor = (maxrecs + minrecs) / 2`로, 즉각적인 split penalty 없이 합리적으로 compact한 structure를 만듭니다. Space가 부족하면 고갈을 피하려고 `leaf_load_factor`를 maxrecs로 설정합니다.
대상공식의미
Leaf maximum`maxrecs = (block_size - header_size) / record_size`Leaf block에 들어가는 maximum record 수
Leaf minimum`minrecs = maxrecs / 2`Merge 규칙에 따른 minimum record 수
Default leaf load`default_load_factor = (maxrecs + minrecs) / 2`Maxrecs의 75%; compact함과 split 여유의 균형
Selected leaf load`leaf_load_factor = enough space ? default_load_factor : maxrecs`Space 여유에 따라 기본값 또는 maximum 선택
Node capacity·load`maxrecs = (block_size - header_size) / (key_size + ptr_size)`; `minrecs = maxrecs / 2`; `node_load_factor = enough space ? default_load_factor : maxrecs`Key와 pointer의 합친 size를 node record size로 사용
Level block counts`leaf_blocks = ceil(record_count / leaf_load_factor)`; `n_blocks = (n == 0 ? leaf_blocks : node_blocks[n])`; `node_blocks[n + 1] = ceil(n_blocks / node_load_factor)`현재 level이 block 하나만 필요할 때까지 recursive하게 반복
Leaf·node load factor와 level별 block 수 계산식을 원문 그대로 보존합니다.
Leaf block 수를 구한 뒤, 아래 level을 가리키는 node block 수를 level마다 계산합니다. 현재 level에 block 하나만 필요할 때까지 전체 계산을 recursive하게 수행합니다.
Btree 형태Height필요 space·root 위치
AG-rooted`level + 1`각 level의 block 수를 모두 합산; 현재 level이 root level
Inode-rooted, top level이 inode fork에 들어가지 않음`level + 2`각 level block 수의 합; inode fork가 root block을 가리킴
Inode-rooted, top level이 inode fork에 들어감`level + 1`Root block을 inode에 저장; 각 level block 수 합보다 하나 적음
Root 위치에 따른 height와 예약 space 차이를 구분합니다.
마지막 inode-rooted 경우는 non-bmap btree가 inode에 root를 둘 수 있게 되는 future patchset에서만 관련되며, 여기에는 완전성을 위해 포함됐습니다.
Geometry Computation
````````````````````
The zeroth step of bulk loading is to assemble the entire record set that will
be stored in the new btree, and sort the records.
Next, call ``xfs_btree_bload_compute_geometry`` to compute the shape of the
btree from the record set, the type of btree, and any load factor preferences.
This information is required for resource reservation.
First, the geometry computation computes the minimum and maximum records that
will fit in a leaf block from the size of a btree block and the size of the
block header.
Roughly speaking, the maximum number of records is::
maxrecs = (block_size - header_size) / record_size
The XFS design specifies that btree blocks should be merged when possible,
which means the minimum number of records is half of maxrecs::
minrecs = maxrecs / 2
The next variable to determine is the desired loading factor.
This must be at least minrecs and no more than maxrecs.
Choosing minrecs is undesirable because it wastes half the block.
Choosing maxrecs is also undesirable because adding a single record to each
newly rebuilt leaf block will cause a tree split, which causes a noticeable
drop in performance immediately afterwards.
The default loading factor was chosen to be 75% of maxrecs, which provides a
reasonably compact structure without any immediate split penalties::
default_load_factor = (maxrecs + minrecs) / 2
If space is tight, the loading factor will be set to maxrecs to try to avoid
running out of space::
leaf_load_factor = enough space ? default_load_factor : maxrecs
Load factor is computed for btree node blocks using the combined size of the
btree key and pointer as the record size::
maxrecs = (block_size - header_size) / (key_size + ptr_size)
minrecs = maxrecs / 2
node_load_factor = enough space ? default_load_factor : maxrecs
Once that's done, the number of leaf blocks required to store the record set
can be computed as::
leaf_blocks = ceil(record_count / leaf_load_factor)
The number of node blocks needed to point to the next level down in the tree
is computed as::
n_blocks = (n == 0 ? leaf_blocks : node_blocks[n])
node_blocks[n + 1] = ceil(n_blocks / node_load_factor)
The entire computation is performed recursively until the current level only
needs one block.
The resulting geometry is as follows:
- For AG-rooted btrees, this level is the root level, so the height of the new
tree is ``level + 1`` and the space needed is the summation of the number of
blocks on each level.
- For inode-rooted btrees where the records in the top level do not fit in the
inode fork area, the height is ``level + 2``, the space needed is the
summation of the number of blocks on each level, and the inode fork points to
the root block.
- For inode-rooted btrees where the records in the top level can be stored in
the inode fork area, then the root block can be stored in the inode, the
height is ``level + 1``, and the space needed is one less than the summation
of the number of blocks on each level.
This only becomes relevant when non-bmap btrees gain the ability to root in
an inode, which is a future patchset and only included here for completeness.
새 B+tree block 예약
2429-2468Repair가 새 btree에 필요한 block 수를 알게 되면 free space information을 사용하여 그 block을 allocate합니다. 예약된 extent는 각각 btree builder state data가 별도로 추적합니다.
Crash resilience를 높이기 위해 reservation code는 각 space allocation과 같은 transaction에 Extent Freeing Intent(EFI) item도 log하고, 그 in-memory `struct xfs_extent_free_item` object를 space reservation에 attach합니다. System이 중단되면 log recovery는 완료되지 않은 EFI를 사용하여 쓰지 않은 예약 space를 free하므로 filesystem은 변경되지 않은 상태로 남습니다.
Btree builder가 reserved extent에서 btree용 block을 claim할 때마다 claimed space를 반영하도록 in-memory reservation을 update합니다. Block reservation은 사용 중인 EFI 수를 줄이기 위해 가능한 한 많은 contiguous space를 allocate하려고 합니다.
Repair가 새 btree block을 쓰는 동안 space reservation용 EFI는 ondisk log의 tail을 pin합니다. System의 다른 부분이 계속 바쁘면 log head가 pinned tail 쪽으로 밀릴 수 있으므로, filesystem livelock을 피하려면 EFI가 log tail을 너무 오래 pin해서는 안 됩니다.
단계동작복구·log 효과
1. ReserveFree space information으로 block allocate; extent별 builder state 추적새 btree용 space 확보
2. Attach EFI같은 transaction에 EFI log; `struct xfs_extent_free_item`을 reservation에 attachCrash 시 unfinished EFI가 unused space를 free
3. ClaimReserved extent에서 block을 claim하고 in-memory reservation update가능한 contiguous allocation으로 EFI 수 감소
4. Pin새 block write 중 EFI가 ondisk log tail을 pinBusy workload가 log head를 pinned tail 쪽으로 밀 수 있음
5. RelogDeferred ops의 dynamic relogging으로 log head에 old EFI용 EFD와 new EFI를 포함한 transaction commitOld EFI를 release하여 log가 계속 전진
EFI·EFD와 block claim의 상태 전이를 순서대로 보존합니다.
이 문제를 완화하기 위해 deferred ops mechanism의 dynamic relogging capability를 재사용합니다. Log head에서 old EFI용 EFD와 head의 new EFI를 포함한 transaction을 commit하면 log가 old EFI를 release하고 계속 앞으로 이동할 수 있습니다. EFI는 commit 및 `reaping <reaping>` phase에서도 역할을 하며 자세한 내용은 다음 절과 reaping 절에서 다룹니다.
제안된 patchset은 `bitmap rework <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework>`_와 `preparation for bulk loading btrees <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_입니다.
.. _newbt:
Reserving New B+Tree Blocks
```````````````````````````
Once repair knows the number of blocks needed for the new btree, it allocates
those blocks using the free space information.
Each reserved extent is tracked separately by the btree builder state data.
To improve crash resilience, the reservation code also logs an Extent Freeing
Intent (EFI) item in the same transaction as each space allocation and attaches
its in-memory ``struct xfs_extent_free_item`` object to the space reservation.
If the system goes down, log recovery will use the unfinished EFIs to free the
unused space, the free space, leaving the filesystem unchanged.
Each time the btree builder claims a block for the btree from a reserved
extent, it updates the in-memory reservation to reflect the claimed space.
Block reservation tries to allocate as much contiguous space as possible to
reduce the number of EFIs in play.
While repair is writing these new btree blocks, the EFIs created for the space
reservations pin the tail of the ondisk log.
It's possible that other parts of the system will remain busy and push the head
of the log towards the pinned tail.
To avoid livelocking the filesystem, the EFIs must not pin the tail of the log
for too long.
To alleviate this problem, the dynamic relogging capability of the deferred ops
mechanism is reused here to commit a transaction at the log head containing an
EFD for the old EFI and new EFI at the head.
This enables the log to release the old EFI to keep the log moving forwards.
EFIs have a role to play during the commit and reaping phases; please see the
next section and the section about :ref:`reaping<reaping>` for more details.
Proposed patchsets are the
`bitmap rework
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework>`_
and the
`preparation for bulk loading btrees
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_.
새 tree 작성과 commit 준비
2469-2565Btree builder인 `xfs_btree_bulkload`는 reserved list에서 block을 claim하고 새 btree block header를 쓴 뒤, 나머지 block을 record로 채우고 새 leaf block을 written block list에 추가합니다. 같은 level에 새 block을 추가할 때마다 sibling pointer도 설정하여 leaf block을 양방향으로 연결합니다.
Record leaf block 작성을 마치면 node block으로 이동합니다. Node block을 채울 때는 tree의 바로 아래 level에 있는 각 block을 순회하여 관련 key를 계산하고 parent node에 기록합니다. Root level에 도달하면 새 btree를 commit할 준비가 끝납니다.
Level구조Pointer·content
Root`root(PP)``root` → `node0`, `node1`
Node`node0(PP)` ↔ `node1(PP)``node0` → `leaf0`, `leaf1`; `node1` → `leaf2`, `leaf3`; node sibling 양방향 연결
Leaf`leaf0(RRR)` ↔ `leaf1(RRR)` ↔ `leaf2(RRR)` ↔ `leaf3(RRR)`각 leaf는 record를 보관하고 sibling pointer로 양방향 연결
원문의 leaf·node·root ASCII 그림을 동일한 계층과 연결 관계로 구조화했습니다.
새 btree commit의 첫 단계는 btree block을 disk에 synchronous하게 persist하는 것입니다. 최근에 free됐던 block을 새 btree block으로 다시 사용할 수 있으므로, builder는 새 block을 disk에 쓰기 전에 `xfs_buf_delwri_queue_here`로 stale buffer를 AIL list에서 제거해야 합니다. Block은 delwri list를 통해 IO용으로 queue되고 `xfs_buf_delwri_submit`으로 한 번의 큰 batch에 write됩니다.
새 block이 disk에 persist되면 control은 bulk loader를 호출한 개별 repair function으로 돌아갑니다. Repair function은 transaction에 새 root location을 log하고, 새 btree용 space reservation을 정리하고, old metadata block을 reap해야 합니다.
단계동작안전 조건
1새 btree root location commit새 tree 연결 지점 확정
2aBuilder가 소비한 모든 space에 Extent Freeing Done(EFD) item log새 EFD는 reservation에 attach된 EFI를 가리켜 log recovery가 새 block을 free하지 못하게 함
2bIncore reservation의 unclaimed portion마다 regular deferred extent free work item 생성Transaction chain 뒤에서 unused space를 free
2c2a·2b에서 log한 EFD와 EFI가 committing transaction reservation을 넘지 않도록 확인초과가 예상되면 `xrep_defer_finish`로 deferred work를 비우고 fresh transaction 획득
3Deferred work를 두 번째로 비움Commit을 끝내고 repair transaction 정리
원문의 1·2a·2b·2c·3 단계와 transaction 조건을 보존합니다.
2c와 3단계의 transaction rolling은 repair algorithm의 약점입니다. Log flush 뒤 reap 단계가 끝나기 전에 crash하면 space leak이 생길 수 있습니다. Online repair function은 각각 수천 개의 block freeing instruction을 수용할 수 있는 매우 큰 transaction을 사용하여 이 가능성을 줄입니다. 이후 old block을 reap하며, 자세한 내용은 bulk loading case study 뒤의 `reaping <reaping>` 절에서 설명합니다.
Writing the New Tree
````````````````````
This part is pretty simple -- the btree builder (``xfs_btree_bulkload``) claims
a block from the reserved list, writes the new btree block header, fills the
rest of the block with records, and adds the new leaf block to a list of
written blocks::
┌────┐
│leaf│
│RRR │
└────┘
Sibling pointers are set every time a new block is added to the level::
┌────┐ ┌────┐ ┌────┐ ┌────┐
│leaf│→│leaf│→│leaf│→│leaf│
│RRR │←│RRR │←│RRR │←│RRR │
└────┘ └────┘ └────┘ └────┘
When it finishes writing the record leaf blocks, it moves on to the node
blocks
To fill a node block, it walks each block in the next level down in the tree
to compute the relevant keys and write them into the parent node::
┌────┐ ┌────┐
│node│──────→│node│
│PP │←──────│PP │
└────┘ └────┘
↙ ↘ ↙ ↘
┌────┐ ┌────┐ ┌────┐ ┌────┐
│leaf│→│leaf│→│leaf│→│leaf│
│RRR │←│RRR │←│RRR │←│RRR │
└────┘ └────┘ └────┘ └────┘
When it reaches the root level, it is ready to commit the new btree!::
┌─────────┐
│ root │
│ PP │
└─────────┘
↙ ↘
┌────┐ ┌────┐
│node│──────→│node│
│PP │←──────│PP │
└────┘ └────┘
↙ ↘ ↙ ↘
┌────┐ ┌────┐ ┌────┐ ┌────┐
│leaf│→│leaf│→│leaf│→│leaf│
│RRR │←│RRR │←│RRR │←│RRR │
└────┘ └────┘ └────┘ └────┘
The first step to commit the new btree is to persist the btree blocks to disk
synchronously.
This is a little complicated because a new btree block could have been freed
in the recent past, so the builder must use ``xfs_buf_delwri_queue_here`` to
remove the (stale) buffer from the AIL list before it can write the new blocks
to disk.
Blocks are queued for IO using a delwri list and written in one large batch
with ``xfs_buf_delwri_submit``.
Once the new blocks have been persisted to disk, control returns to the
individual repair function that called the bulk loader.
The repair function must log the location of the new root in a transaction,
clean up the space reservations that were made for the new btree, and reap the
old metadata blocks:
1. Commit the location of the new btree root.
2. For each incore reservation:
a. Log Extent Freeing Done (EFD) items for all the space that was consumed
by the btree builder. The new EFDs must point to the EFIs attached to
the reservation to prevent log recovery from freeing the new blocks.
b. For unclaimed portions of incore reservations, create a regular deferred
extent free work item to be free the unused space later in the
transaction chain.
c. The EFDs and EFIs logged in steps 2a and 2b must not overrun the
reservation of the committing transaction.
If the btree loading code suspects this might be about to happen, it must
call ``xrep_defer_finish`` to clear out the deferred work and obtain a
fresh transaction.
3. Clear out the deferred work a second time to finish the commit and clean
the repair transaction.
The transaction rolling in steps 2c and 3 represent a weakness in the repair
algorithm, because a log flush and a crash before the end of the reap step can
result in space leaking.
Online repair functions minimize the chances of this occurring by using very
large transactions, which each can accommodate many thousands of block freeing
instructions.
Repair moves on to reaping the old blocks, which will be presented in a
subsequent :ref:`section<reaping>` after a few case studies of bulk loading.Case study: inode index rebuild
2566-2630Inode index btree rebuild의 상위 절차는 다음 일곱 단계입니다.
단계동작Finobt·결과
1Reverse mapping record를 순회해 inode chunk information으로 `struct xfs_inobt_rec` 생성; old inode btree block bitmap 생성재구축 record와 reap 대상 수집
2Record를 inode order로 xfarray에 append정렬된 staging array 구성
3`xfs_btree_bload_compute_geometry`로 inode btree block 수 계산Free space inode btree가 enabled면 finobt geometry도 다시 계산
4앞 단계에서 계산한 수의 block allocate두 tree 작성용 reservation 확보
5`xfs_btree_bload`로 xfarray record를 btree block에 쓰고 internal node block 생성Enabled면 다시 호출하여 finobt load
6새 btree root block location을 AGI에 commit하나 또는 여러 root 연결
71단계 bitmap으로 old btree block reap이전 index 제거
Inobt와 선택적 finobt를 함께 구축하는 원문의 1–7단계를 보존합니다.
Inode btree는 inumber를 관련 inode record의 ondisk location에 mapping하므로 reverse mapping information에서 rebuild할 수 있습니다. Owner가 `XFS_RMAP_OWN_INOBT`인 reverse mapping record는 old inode btree block location을 표시하고, owner가 `XFS_RMAP_OWN_INODES`인 각 record는 inode cluster buffer가 하나 이상 있는 location을 표시합니다. Cluster는 한 transaction에서 allocate하거나 free할 수 있는 ondisk inode의 최소 단위이며, 1 fs block 또는 inode 4개보다 작지 않습니다.
Owner표시 대상Rebuild 용도
`XFS_RMAP_OWN_INOBT`Old inode btree block locationReap용 bitmap 구성
`XFS_RMAP_OWN_INODES`하나 이상의 inode cluster buffer locationInode chunk record 재구성
두 owner가 가리키는 공간과 rebuild 용도를 구분합니다.
각 inode cluster가 나타내는 space에는 free space btree record와 reference count btree record가 없어야 합니다. 하나라도 있으면 space metadata inconsistency만으로 operation을 abort할 이유가 충분합니다. 그렇지 않으면 각 cluster buffer를 읽어 content가 ondisk inode처럼 보이는지 확인하고, `xfs_dinode.i_mode != 0`이면 allocated file, `xfs_dinode.i_mode == 0`이면 free inode로 판정합니다. 연속한 cluster buffer read 결과를 inumber keyspace의 연속 64개 number인 inode chunk record 하나를 채울 만큼 모으며, sparse chunk의 record에는 hole이 포함될 수 있습니다.
Repair function이 chunk 하나 분량의 data를 모으면 `xfarray_append`로 inode btree record를 xfarray에 추가합니다. Btree 생성 단계에서는 이 xfarray를 두 번 순회합니다. 첫 번째 순회는 모든 inode chunk record로 inode btree를 채우고, 두 번째 순회는 free non-sparse inode가 있는 chunk record로 free inode btree를 채웁니다. Inode btree record 수는 xfarray record 수와 같지만, free inode btree record count는 inode chunk record를 xfarray에 저장하는 동안 별도로 계산해야 합니다.
제안된 patchset은 `AG btree repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ series입니다.
Case Study: Rebuilding the Inode Index
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The high level process to rebuild the inode index btree is:
1. Walk the reverse mapping records to generate ``struct xfs_inobt_rec``
records from the inode chunk information and a bitmap of the old inode btree
blocks.
2. Append the records to an xfarray in inode order.
3. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
of blocks needed for the inode btree.
If the free space inode btree is enabled, call it again to estimate the
geometry of the finobt.
4. Allocate the number of blocks computed in the previous step.
5. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
generate the internal node blocks.
If the free space inode btree is enabled, call it again to load the finobt.
6. Commit the location of the new btree root block(s) to the AGI.
7. Reap the old btree blocks using the bitmap created in step 1.
Details are as follows.
The inode btree maps inumbers to the ondisk location of the associated
inode records, which means that the inode btrees can be rebuilt from the
reverse mapping information.
Reverse mapping records with an owner of ``XFS_RMAP_OWN_INOBT`` marks the
location of the old inode btree blocks.
Each reverse mapping record with an owner of ``XFS_RMAP_OWN_INODES`` marks the
location of at least one inode cluster buffer.
A cluster is the smallest number of ondisk inodes that can be allocated or
freed in a single transaction; it is never smaller than 1 fs block or 4 inodes.
For the space represented by each inode cluster, ensure that there are no
records in the free space btrees nor any records in the reference count btree.
If there are, the space metadata inconsistencies are reason enough to abort the
operation.
Otherwise, read each cluster buffer to check that its contents appear to be
ondisk inodes and to decide if the file is allocated
(``xfs_dinode.i_mode != 0``) or free (``xfs_dinode.i_mode == 0``).
Accumulate the results of successive inode cluster buffer reads until there is
enough information to fill a single inode chunk record, which is 64 consecutive
numbers in the inumber keyspace.
If the chunk is sparse, the chunk record may include holes.
Once the repair function accumulates one chunk's worth of data, it calls
``xfarray_append`` to add the inode btree record to the xfarray.
This xfarray is walked twice during the btree creation step -- once to populate
the inode btree with all inode chunk records, and a second time to populate the
free inode btree with records for chunks that have free non-sparse inodes.
The number of records for the inode btree is the number of xfarray records,
but the record count for the free inode btree has to be computed as inode chunk
records are stored in the xfarray.
The proposed patchset is the
`AG btree repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
series.
Case study: space reference count rebuild
2631-2723Reverse mapping record를 사용하여 reference count information을 rebuild합니다. Shared file data의 copy on write가 올바르게 동작하려면 reference count가 필요합니다. Reverse mapping entry를 physical block extent를 나타내는 rectangle로 생각하고 서로 겹쳐 놓으면, stack height가 변하는 모든 지점에서 reference count record가 시작하거나 끝나야 함을 알 수 있습니다. 즉 record emission stimulus는 level-triggered입니다.
요소구조의미
InputPhysical block extent를 나타내는 여러 reverse-mapping rectangle의 겹침각 위치의 stack height가 현재 reference count를 나타냄
TriggerExtent start 또는 end에서 stack height 변화Height가 바뀌는 모든 boundary가 record emission 지점
Output연속한 두 boundary 사이의 block range + 해당 stack heightBag size가 1보다 클 때 새 refcount record 생성
원문의 겹친 extent ASCII 그림을 입력·trigger·출력 관계로 구조화했습니다.
Ondisk reference count btree는 `refcount == 0`인 경우를 저장하지 않습니다. Free space btree가 이미 free block을 기록하기 때문입니다. Copy-on-write operation을 staging하는 extent만 `refcount == 1`인 record여야 합니다. Single-owner file block은 free space btree와 reference count btree 어느 쪽에도 기록되지 않습니다.
Refcount·상태기록 위치이유
`refcount == 0`Reference count btree에 저장하지 않음Free space btree가 free block을 기록
`refcount == 1`, CoW stagingReference count btree에 저장Copy-on-write staging extent를 추적
Single-owner file blockFree space·reference count btree 모두에 저장하지 않음Free하지 않고 shared도 아님
0과 1의 특수한 경우를 ownership과 staging 상태로 구분합니다.
Reference count btree rebuild의 상위 절차는 다음 일곱 단계입니다.
단계동작결과
1Reverse mapping이 둘 이상인 space마다 `struct xfs_refcount_irec`를 만들어 xfarray에 추가; `XFS_RMAP_OWN_COW` record도 추가Shared·CoW staging extent 수집; `XFS_RMAP_OWN_REFC` record로 old refcount btree block bitmap 생성
2Physical extent order로 sorting하고 CoW staging extent를 xfarray 끝에 배치Refcount btree record sorting order와 일치
3`xfs_btree_bload_compute_geometry`로 새 tree block 수 계산Geometry 확정
4계산한 수의 block allocate새 tree용 reservation 확보
5`xfs_btree_bload`로 xfarray record를 btree block에 쓰고 internal node block 생성새 refcount tree 작성
6새 btree root block location을 AGF에 commit새 index 연결
71단계 bitmap으로 old btree block reap이전 refcount tree 제거
Reverse mapping scan부터 old tree reap까지 원문의 1–7단계를 보존합니다.
세부 algorithm은 `xfs_repair`가 reverse mapping record에서 refcount information을 생성할 때 쓰는 것과 같습니다. Reverse mapping btree가 소진될 때까지 다음 record와 start block이 같은 모든 record를 bag에 넣고, bag이 빌 때까지 reference count가 다음으로 바뀌는 가장 낮은 block number를 찾습니다.
순서Bag·btree 동작Emission 조건
1Btree의 next record를 bag에 넣음새 sweep group 시작
2같은 starting block을 가진 모든 record를 bag에 추가현재 boundary의 active mapping 수 확정
3Next unprocessed mapping의 start 또는 bag에서 가장 짧은 mapping의 end 다음 block 중 낮은 위치 선택Reference count가 바뀌는 다음 block number 결정
4그 위치에서 끝나는 mapping을 모두 bag에서 제거하고, 그 위치에서 시작하는 mapping을 btree에서 모두 추가Boundary를 지나 active mapping set 갱신
5Bag size가 바뀌었고 1보다 크면 직전에 순회한 block range와 bag size로 새 refcount record 생성Level-triggered record emission
동일 start 수집과 다음 height-change boundary 계산을 순서대로 나타냅니다.
여기서 bag-like structure는 `xfarray access patterns <xfarray_access_patterns>` 절에서 설명한 type 2 xfarray입니다. Reverse mapping은 `xfarray_store_anywhere`로 bag에 추가하고 `xfarray_unset`으로 제거하며, bag member는 `xfarray_iter` loop로 검사합니다.
제안된 patchset은 `AG btree repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ series입니다.
Case Study: Rebuilding the Space Reference Counts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Reverse mapping records are used to rebuild the reference count information.
Reference counts are required for correct operation of copy on write for shared
file data.
Imagine the reverse mapping entries as rectangles representing extents of
physical blocks, and that the rectangles can be laid down to allow them to
overlap each other.
From the diagram below, it is apparent that a reference count record must start
or end wherever the height of the stack changes.
In other words, the record emission stimulus is level-triggered::
█ ███
██ █████ ████ ███ ██████
██ ████ ███████████ ████ █████████
████████████████████████████████ ███████████
^ ^ ^^ ^^ ^ ^^ ^^^ ^^^^ ^ ^^ ^ ^ ^
2 1 23 21 3 43 234 2123 1 01 2 3 0
The ondisk reference count btree does not store the refcount == 0 cases because
the free space btree already records which blocks are free.
Extents being used to stage copy-on-write operations should be the only records
with refcount == 1.
Single-owner file blocks aren't recorded in either the free space or the
reference count btrees.
The high level process to rebuild the reference count btree is:
1. Walk the reverse mapping records to generate ``struct xfs_refcount_irec``
records for any space having more than one reverse mapping and add them to
the xfarray.
Any records owned by ``XFS_RMAP_OWN_COW`` are also added to the xfarray
because these are extents allocated to stage a copy on write operation and
are tracked in the refcount btree.
Use any records owned by ``XFS_RMAP_OWN_REFC`` to create a bitmap of old
refcount btree blocks.
2. Sort the records in physical extent order, putting the CoW staging extents
at the end of the xfarray.
This matches the sorting order of records in the refcount btree.
3. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
of blocks needed for the new tree.
4. Allocate the number of blocks computed in the previous step.
5. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
generate the internal node blocks.
6. Commit the location of new btree root block to the AGF.
7. Reap the old btree blocks using the bitmap created in step 1.
Details are as follows; the same algorithm is used by ``xfs_repair`` to
generate refcount information from reverse mapping records.
- Until the reverse mapping btree runs out of records:
- Retrieve the next record from the btree and put it in a bag.
- Collect all records with the same starting block from the btree and put
them in the bag.
- While the bag isn't empty:
- Among the mappings in the bag, compute the lowest block number where the
reference count changes.
This position will be either the starting block number of the next
unprocessed reverse mapping or the next block after the shortest mapping
in the bag.
- Remove all mappings from the bag that end at this position.
- Collect all reverse mappings that start at this position from the btree
and put them in the bag.
- If the size of the bag changed and is greater than one, create a new
refcount record associating the block number range that we just walked to
the size of the bag.
The bag-like structure in this case is a type 2 xfarray as discussed in the
:ref:`xfarray access patterns<xfarray_access_patterns>` section.
Reverse mappings are added to the bag using ``xfarray_store_anywhere`` and
removed via ``xfarray_unset``.
Bag members are examined through ``xfarray_iter`` loops.
The proposed patchset is the
`AG btree repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
series.
Case study: file fork mapping index rebuild
2724-2764Data fork 또는 attr fork의 mapping btree rebuild는 다음 여덟 단계로 진행합니다.
단계동작분기·결과
1해당 inode·fork의 reverse mapping record에서 `struct xfs_bmbt_rec` 생성 후 xfarray에 append; `BMBT_BLOCK` record로 old bmap btree block bitmap 계산새 mapping record와 reap 대상 수집
2`xfs_btree_bload_compute_geometry`로 새 tree block 수 계산필요 geometry 확정
3Record를 file offset order로 sortingBmap btree order에 맞춤
4Extent record가 inode fork immediate area에 들어가는지 확인들어가면 immediate area에 commit하고 8단계로 이동
52단계에서 계산한 수의 block allocateBMBT 작성용 reservation 확보
6`xfs_btree_bload`로 xfarray record를 btree block에 쓰고 internal node block 생성새 mapping btree 작성
7새 btree root block을 inode fork immediate area에 commitBMBT root 연결
81단계 bitmap으로 old btree block reap이전 mapping index 제거
Immediate-area 단축 경로를 포함한 원문의 1–8단계를 보존합니다.
여기에는 세 가지 complication이 있습니다. 첫째, data fork와 attr fork가 둘 다 BMBT format이 아니라면 immediate area size를 조정하도록 fork offset을 옮길 수 있습니다. 둘째, fork mapping 수가 충분히 적으면 BMBT 대신 EXTENTS format을 사용할 수 있어 conversion이 필요할 수 있습니다. 셋째, delayed allocation extent를 건드리지 않도록 incore extent map을 주의 깊게 reload해야 합니다.
난점조건필요 처리
Fork offsetData·attr fork가 모두 BMBT format은 아님Immediate area size 조정을 위해 fork offset 이동 가능
Format selectionFork mapping 수가 충분히 적음BMBT 대신 EXTENTS format 사용을 위한 conversion 가능
Incore reloadDelayed allocation extent가 존재할 수 있음해당 extent를 방해하지 않도록 incore extent map을 신중히 reload
Fork layout과 incore state에 영향을 주는 조건을 구분합니다.
제안된 patchset은 `file mapping repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings>`_ series입니다.
Case Study: Rebuilding File Fork Mapping Indices
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The high level process to rebuild a data/attr fork mapping btree is:
1. Walk the reverse mapping records to generate ``struct xfs_bmbt_rec``
records from the reverse mapping records for that inode and fork.
Append these records to an xfarray.
Compute the bitmap of the old bmap btree blocks from the ``BMBT_BLOCK``
records.
2. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
of blocks needed for the new tree.
3. Sort the records in file offset order.
4. If the extent records would fit in the inode fork immediate area, commit the
records to that immediate area and skip to step 8.
5. Allocate the number of blocks computed in the previous step.
6. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
generate the internal node blocks.
7. Commit the new btree root block to the inode fork immediate area.
8. Reap the old btree blocks using the bitmap created in step 1.
There are some complications here:
First, it's possible to move the fork offset to adjust the sizes of the
immediate areas if the data and attr forks are not both in BMBT format.
Second, if there are sufficiently few fork mappings, it may be possible to use
EXTENTS format instead of BMBT, which may require a conversion.
Third, the incore extent map must be reloaded carefully to avoid disturbing
any delayed allocation extents.
The proposed patchset is the
`file mapping repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings>`_
series.
Old metadata block reaping
2765-2849Online fsck가 의심스러운 data structure를 대체할 새 structure를 만들 때마다 old structure에 속했던 block을 찾아 폐기하는 문제가 생깁니다. 이를 전혀 처리하지 않는 가장 게으른 방법은 filesystem에서 space가 leak되면서 service를 서서히 저하시킵니다. Offline repair는 지우지 않기로 한 file과 directory의 usage를 기록한 뒤 모든 space metadata를 rebuild하므로, 발견한 free space에 새 structure를 만들고 reaping 문제를 피할 수 있습니다.
방식처리결과
미처리Old structure block을 방치Space leak 누적으로 service degradation
Offline repair보존할 file·directory usage 기록 후 모든 space metadata rebuildDiscovered free space에 새 structure를 만들어 별도 reaping 불필요
Online fsckReverse mapping을 교차검증해 old structure block을 찾아 disposeLive filesystem에서 ownership 안전성을 확인하며 회수
미처리·offline repair·online fsck의 space 회수 방식을 비교합니다.
Online fsck는 repair 중 reverse mapping record에 크게 의존하여 해당 rmap owner가 소유하지만 실제로는 free인 space를 찾습니다. 다른 data structure도 같은 block을 소유한다고 생각할 수 있으므로, 예를 들어 crosslinked tree를 검출하려면 rmap record를 다른 rmap record와 cross-reference해야 합니다. 이런 block을 allocator가 다시 내주게 해서는 system이 consistency에 가까워지지 않습니다.
Space metadata에서 폐기할 extent 후보를 찾는 과정은 일반적으로 다음 1–3단계를 따릅니다.
단계Bitmap 동작결과
1보존해야 할 data structure가 사용하는 space bitmap 생성같은 rmap owner code로 rebuild하는 모든 object를 표시한다면 새 metadata용 space reservation도 사용 가능
2Reverse mapping data를 조사해 보존 metadata와 같은 `XFS_RMAP_OWN_*` number가 소유한 space bitmap 생성Owner가 주장하는 전체 범위 수집
3Bitmap disunion operator로 (2)에서 (1)을 뺌남은 set bit가 free 가능성이 있는 candidate extent
보존 space와 owner space의 disunion으로 후보를 계산합니다.
Extended attribute, directory, symbolic link, quota file, realtime bitmap 같은 file-based metadata는 temporary file에 붙인 새 structure를 만들고 file fork의 모든 mapping을 exchange하여 repair합니다. 그 뒤 old file fork의 mapping이 disposal candidate block이 됩니다.
Old extent를 실제로 dispose하는 과정은 다음 4–8단계입니다.
단계검사·동작결과
4Candidate extent 첫 block에서 repair 대상과 다른 rmap owner의 reverse mapping record 수 계산0이면 single owner라 free 가능; 0이 아니면 crosslinked라 free 금지
5다음 block부터 첫 block과 같은 zero/nonzero other-owner status가 이어지는 길이 계산동일 상태의 region 경계 확정
6Crosslinked region이면 repair 대상 structure의 reverse mapping entry 삭제Block은 free하지 않고 다음 region으로 이동
7Free할 region이면 buffer cache의 대응 buffer를 stale로 표시Log writeback 방지
8Region을 free다음 region으로 이동
첫 block의 다른 owner 수를 기준으로 region을 분류하고 처리합니다.
Transaction size는 유한하므로 reaping은 overrun을 피하도록 transaction을 주의 깊게 roll해야 합니다. Overrun은 더는 occupied되지 않은 space를 대신해 log한 EFI와 buffer invalidation용 log item에서 생깁니다. 또한 reaping 도중 crash하면 block이 leak될 수 있는 window이므로, online repair function은 매우 큰 transaction으로 발생 가능성을 줄입니다.
원인Log 소비
a더는 occupied되지 않은 space를 대신해 log한 EFI
bBuffer invalidation용 log item
원문에 열거된 두 log-space 소비원을 보존합니다.
제안된 patchset은 `preparation for bulk loading btrees <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_ series입니다.
.. _reaping:
Reaping Old Metadata Blocks
---------------------------
Whenever online fsck builds a new data structure to replace one that is
suspect, there is a question of how to find and dispose of the blocks that
belonged to the old structure.
The laziest method of course is not to deal with them at all, but this slowly
leads to service degradations as space leaks out of the filesystem.
Hopefully, someone will schedule a rebuild of the free space information to
plug all those leaks.
Offline repair rebuilds all space metadata after recording the usage of
the files and directories that it decides not to clear, hence it can build new
structures in the discovered free space and avoid the question of reaping.
As part of a repair, online fsck relies heavily on the reverse mapping records
to find space that is owned by the corresponding rmap owner yet truly free.
Cross referencing rmap records with other rmap records is necessary because
there may be other data structures that also think they own some of those
blocks (e.g. crosslinked trees).
Permitting the block allocator to hand them out again will not push the system
towards consistency.
For space metadata, the process of finding extents to dispose of generally
follows this format:
1. Create a bitmap of space used by data structures that must be preserved.
The space reservations used to create the new metadata can be used here if
the same rmap owner code is used to denote all of the objects being rebuilt.
2. Survey the reverse mapping data to create a bitmap of space owned by the
same ``XFS_RMAP_OWN_*`` number for the metadata that is being preserved.
3. Use the bitmap disunion operator to subtract (1) from (2).
The remaining set bits represent candidate extents that could be freed.
The process moves on to step 4 below.
Repairs for file-based metadata such as extended attributes, directories,
symbolic links, quota files and realtime bitmaps are performed by building a
new structure attached to a temporary file and exchanging all mappings in the
file forks.
Afterward, the mappings in the old file fork are the candidate blocks for
disposal.
The process for disposing of old extents is as follows:
4. For each candidate extent, count the number of reverse mapping records for
the first block in that extent that do not have the same rmap owner for the
data structure being repaired.
- If zero, the block has a single owner and can be freed.
- If not, the block is part of a crosslinked structure and must not be
freed.
5. Starting with the next block in the extent, figure out how many more blocks
have the same zero/nonzero other owner status as that first block.
6. If the region is crosslinked, delete the reverse mapping entry for the
structure being repaired and move on to the next region.
7. If the region is to be freed, mark any corresponding buffers in the buffer
cache as stale to prevent log writeback.
8. Free the region and move on.
However, there is one complication to this procedure.
Transactions are of finite size, so the reaping process must be careful to roll
the transactions to avoid overruns.
Overruns come from two sources:
a. EFIs logged on behalf of space that is no longer occupied
b. Log items for buffer invalidations
This is also a window in which a crash during the reaping process can leak
blocks.
As stated earlier, online repair functions use very large transactions to
minimize the chances of this occurring.
The proposed patchset is the
`preparation for bulk loading btrees
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_
series.Regular btree repair 후 reaping
2850-2876Old reference count btree와 inode btree는 special owner code가 있는 rmap record를 가지므로 가장 쉽게 reap할 수 있습니다. Refcount btree는 `XFS_RMAP_OWN_REFC`, inode와 free inode btree는 `XFS_RMAP_OWN_INOBT`를 사용합니다. Old btree block의 reap extent list를 만드는 개념적 절차는 다음과 같습니다.
단계동작Bitmap 결과
1관련 AGI/AGF header buffer를 lockAllocation과 free 방지
2Rebuild 중인 metadata structure의 rmap owner와 일치하는 각 reverse mapping record range를 bitmap에 set해당 owner가 주장하는 block 수집
3같은 rmap owner를 가진 current data structure를 순회하고 방문 block range를 bitmap에서 clear현재 reachable block 제외
4남은 set bit를 old data structure block candidate로 취급`(rmap_records_owned_by & ~blocks_reachable_by_walk)`가 free 가능 block
Owner bitmap에서 현재 reachable block을 빼는 원문의 1–4단계를 보존합니다.
일반적인 경우처럼 repair 내내 AGF lock을 유지할 수 있으면, 새 btree record를 생성하는 reverse mapping record walk와 동시에 2단계를 수행할 수 있습니다.
Case Study: Reaping After a Regular Btree Repair
````````````````````````````````````````````````
Old reference count and inode btrees are the easiest to reap because they have
rmap records with special owner codes: ``XFS_RMAP_OWN_REFC`` for the refcount
btree, and ``XFS_RMAP_OWN_INOBT`` for the inode and free inode btrees.
Creating a list of extents to reap the old btree blocks is quite simple,
conceptually:
1. Lock the relevant AGI/AGF header buffers to prevent allocation and frees.
2. For each reverse mapping record with an rmap owner corresponding to the
metadata structure being rebuilt, set the corresponding range in a bitmap.
3. Walk the current data structures that have the same rmap owner.
For each block visited, clear that range in the above bitmap.
4. Each set bit in the bitmap represents a block that could be a block from the
old data structures and hence is a candidate for reaping.
In other words, ``(rmap_records_owned_by & ~blocks_reachable_by_walk)``
are the blocks that might be freeable.
If it is possible to maintain the AGF lock throughout the repair (which is the
common case), then step 2 can be performed at the same time as the reverse
mapping record walk that creates the records for the new btree.
Free-space index rebuild
2877-2949Free space index rebuild의 상위 절차는 다음 일곱 단계입니다.
단계동작결과
1Reverse mapping btree의 gap에서 `struct xfs_alloc_rec_incore` record 생성명시되지 않은 free space 추론
2Record를 xfarray에 appendBulk-load input 구성
3`xfs_btree_bload_compute_geometry`로 각 새 tree의 block 수 계산두 index geometry 산정
4수집한 free space information에서 계산한 수의 block allocate새 tree block 확보
5`xfs_btree_bload`로 xfarray record를 쓰고 free-space-by-length index의 internal node 생성; 다시 호출해 free-space-by-block-number index 작성두 free space btree 구축
6새 btree root block location을 AGF에 commit두 새 index 연결
7Reverse mapping btree·새 free space btree·AGFL 어디에도 기록되지 않은 space를 찾아 old btree block reap이전 free space index 제거
By-length와 by-block-number 두 index를 만드는 원문의 1–7단계를 보존합니다.
Free space btree repair에는 regular btree repair보다 세 가지 핵심 complication이 있습니다.
난점원인처리·공식
1. Record inferenceFree space는 reverse mapping record에 명시적으로 추적되지 않음Reverse mapping btree keyspace의 physical-space component에 있는 gap에서 새 free space record 추론
2. Self-reservationCommon btree reservation code는 free space btree에서 새 block을 reserve하므로 free space btree 자체 repair에는 사용 불가Repair 내내 AGF buffer lock을 유지하고 수집한 free space로 새 block 공급; ondisk filesystem이 unowned로 보는 space라 reserved extent마다 EFI 불필요; reservation으로 record 수가 바뀌면 충분할 때까지 geometry 재계산; commit 때 reserved block rmap 생성·unused block 재삽입; deferred rmap/free operation으로 atomic transition
3. Reap discoveryAGFL, free space btree, rmap btree block이 모두 `XFS_RMAP_OWN_AG` ownership을 유지Rmap walk에서 `ag_owner_bitmap` 생성, current rmap btree·AGFL block으로 `rmap_agfl_bitmap` 유지; `(ag_owner_bitmap & ~rmap_agfl_bitmap)`으로 old free space btree extent 계산
Record 생성, block reservation, old-block reaping의 특수 처리를 구분합니다.
제안된 patchset은 `AG btree repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ series입니다.
Case Study: Rebuilding the Free Space Indices
`````````````````````````````````````````````
The high level process to rebuild the free space indices is:
1. Walk the reverse mapping records to generate ``struct xfs_alloc_rec_incore``
records from the gaps in the reverse mapping btree.
2. Append the records to an xfarray.
3. Use the ``xfs_btree_bload_compute_geometry`` function to compute the number
of blocks needed for each new tree.
4. Allocate the number of blocks computed in the previous step from the free
space information collected.
5. Use ``xfs_btree_bload`` to write the xfarray records to btree blocks and
generate the internal node blocks for the free space by length index.
Call it again for the free space by block number index.
6. Commit the locations of the new btree root blocks to the AGF.
7. Reap the old btree blocks by looking for space that is not recorded by the
reverse mapping btree, the new free space btrees, or the AGFL.
Repairing the free space btrees has three key complications over a regular
btree repair:
First, free space is not explicitly tracked in the reverse mapping records.
Hence, the new free space records must be inferred from gaps in the physical
space component of the keyspace of the reverse mapping btree.
Second, free space repairs cannot use the common btree reservation code because
new blocks are reserved out of the free space btrees.
This is impossible when repairing the free space btrees themselves.
However, repair holds the AGF buffer lock for the duration of the free space
index reconstruction, so it can use the collected free space information to
supply the blocks for the new free space btrees.
It is not necessary to back each reserved extent with an EFI because the new
free space btrees are constructed in what the ondisk filesystem thinks is
unowned space.
However, if reserving blocks for the new btrees from the collected free space
information changes the number of free space records, repair must re-estimate
the new free space btree geometry with the new record count until the
reservation is sufficient.
As part of committing the new btrees, repair must ensure that reverse mappings
are created for the reserved blocks and that unused reserved blocks are
inserted into the free space btrees.
Deferrred rmap and freeing operations are used to ensure that this transition
is atomic, similar to the other btree repair functions.
Third, finding the blocks to reap after the repair is not overly
straightforward.
Blocks for the free space btrees and the reverse mapping btrees are supplied by
the AGFL.
Blocks put onto the AGFL have reverse mapping records with the owner
``XFS_RMAP_OWN_AG``.
This ownership is retained when blocks move from the AGFL into the free space
btrees or the reverse mapping btrees.
When repair walks reverse mapping records to synthesize free space records, it
creates a bitmap (``ag_owner_bitmap``) of all the space claimed by
``XFS_RMAP_OWN_AG`` records.
The repair context maintains a second bitmap corresponding to the rmap btree
blocks and the AGFL blocks (``rmap_agfl_bitmap``).
When the walk is complete, the bitmap disunion operation ``(ag_owner_bitmap &
~rmap_agfl_bitmap)`` computes the extents that are used by the old free space
btrees.
These blocks can then be reaped using the methods outlined above.
The proposed patchset is the
`AG btree repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
series.Reverse-mapping btree repair 후 reaping
2950-2979Old reverse mapping btree는 repair 뒤 비교적 쉽게 reap할 수 있습니다. AGFL block, 두 free space btree의 block, reverse mapping btree block은 모두 owner가 `XFS_RMAP_OWN_AG`인 reverse mapping record를 갖습니다. `Live rebuilds of rmap data <rmap_repair>` case study에서 전체 수집·rebuild 과정을 설명하지만, 여기서 중요한 점은 새 rmap btree에 old rmap btree용 record가 없고 old btree block도 free space btree에 추적되지 않는다는 것입니다.
단계Bitmap 동작결과
1새 rmap btree record의 gap에 해당하는 bit set`new_rmapbt_gaps` 생성
2Free space btree extent와 current AGFL block에 해당하는 bit clear`agfl | bnobt_records` 제외
3`(new_rmapbt_gaps & ~(agfl | bnobt_records))` 계산위 reaping method로 처리할 candidate block
새 rmapbt gap에서 free·AGFL 범위를 제외하는 bitmap 공식을 구조화합니다.
Reverse mapping btree rebuild의 나머지 과정은 별도 `case study <rmap_repair>`에서 설명합니다.
제안된 patchset은 `AG btree repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ series입니다.
.. _rmap_reap:
Case Study: Reaping After Repairing Reverse Mapping Btrees
``````````````````````````````````````````````````````````
Old reverse mapping btrees are less difficult to reap after a repair.
As mentioned in the previous section, blocks on the AGFL, the two free space
btree blocks, and the reverse mapping btree blocks all have reverse mapping
records with ``XFS_RMAP_OWN_AG`` as the owner.
The full process of gathering reverse mapping records and building a new btree
are described in the case study of
:ref:`live rebuilds of rmap data <rmap_repair>`, but a crucial point from that
discussion is that the new rmap btree will not contain any records for the old
rmap btree, nor will the old btree blocks be tracked in the free space btrees.
The list of candidate reaping blocks is computed by setting the bits
corresponding to the gaps in the new rmap btree records, and then clearing the
bits corresponding to extents in the free space btrees and the current AGFL
blocks.
The result ``(new_rmapbt_gaps & ~(agfl | bnobt_records))`` are reaped using the
methods outlined above.
The rest of the process of rebuildng the reverse mapping btree is discussed
in a separate :ref:`case study<rmap_repair>`.
The proposed patchset is the
`AG btree repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
series.
AGFL rebuild
2980-2998Allocation group free block list(AGFL)은 다음 다섯 단계로 repair합니다.
단계동작결과
1Reverse mapping data가 `XFS_RMAP_OWN_AG` 소유라고 주장하는 모든 space bitmap 생성AG owner space 수집
2두 free space btree와 rmap btree가 사용하는 space 차감현재 metadata block 제외
3Reverse mapping data가 다른 owner 소유라고 주장하는 모든 space 차감Crosslinked block의 AGFL 재삽입 방지
4AGFL이 가득 차면 남은 block reap초과 candidate 회수
5다음 freelist fix operation이 list를 right-sizeAGFL 크기 정상화
Crosslinked block을 다시 AGFL에 넣지 않도록 subtraction 순서를 보존합니다.
자세한 내용은 `fs/xfs/scrub/agheader_repair.c <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/xfs/scrub/agheader_repair.c>`_를 참조합니다.
Case Study: Rebuilding the AGFL
```````````````````````````````
The allocation group free block list (AGFL) is repaired as follows:
1. Create a bitmap for all the space that the reverse mapping data claims is
owned by ``XFS_RMAP_OWN_AG``.
2. Subtract the space used by the two free space btrees and the rmap btree.
3. Subtract any space that the reverse mapping data claims is owned by any
other owner, to avoid re-adding crosslinked blocks to the AGFL.
4. Once the AGFL is full, reap any blocks leftover.
5. The next operation to fix the freelist will right-size the list.
See `fs/xfs/scrub/agheader_repair.c <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/xfs/scrub/agheader_repair.c>`_ for more details.
Inode record repair와 cache coherency
2999-3031Inode record에는 ondisk record인 dinode와 in-memory cached representation이 모두 있으므로 조심해서 처리해야 합니다. Filesystem이 in-memory representation을 load할 수 없을 만큼 ondisk metadata가 심하게 손상된 경우에만 ondisk metadata에 접근하지 않으면 cache coherency 문제가 발생할 가능성이 매우 큽니다. Online fsck가 손상된 file을 scrub하려고 open할 때는 in-memory representation 또는 ondisk location update를 막는 데 필요한 object lock 중 하나를 반환하는 specialized resource acquisition function을 사용해야 합니다.
상태허용 repair·검사다음 단계
In-core load 불가Inode cluster buffer verifier와 inode fork verifier가 잡은 문제만 ondisk inode buffer에서 수정`iget` 재시도
두 번째 `iget` 실패추가 ondisk repair 중단Repair 실패
In-memory representation load 성공Inode lock 후 comprehensive check·repair·optimization단순 attribute 제약 및 fork·block count 검증
Ondisk 최소 repair에서 in-core 종합 repair로 넘어가는 조건을 구분합니다.
Ondisk inode buffer에는 in-core structure를 load하는 데 필요한 repair만 해야 합니다. 즉 inode cluster buffer와 inode fork verifier가 잡은 문제를 고친 뒤 `iget` operation을 재시도하며, 두 번째 `iget`도 실패하면 repair가 실패한 것입니다.
In-memory representation을 load한 뒤에는 inode를 lock하고 comprehensive check, repair, optimization을 수행할 수 있습니다. 대부분의 inode attribute는 쉽게 검사·제한할 수 있거나 user-controlled arbitrary bit pattern이므로 고치기 쉽습니다. 반면 data·attr fork extent count와 file block count의 올바른 값은 fork traversal이 필요하여 더 복잡합니다. Traversal이 실패하면 field를 invalid 상태로 두고 fork fsck function이 실행되기를 기다립니다.
제안된 patchset은 `inode repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes>`_ series입니다.
Inode Record Repairs
--------------------
Inode records must be handled carefully, because they have both ondisk records
("dinodes") and an in-memory ("cached") representation.
There is a very high potential for cache coherency issues if online fsck is not
careful to access the ondisk metadata *only* when the ondisk metadata is so
badly damaged that the filesystem cannot load the in-memory representation.
When online fsck wants to open a damaged file for scrubbing, it must use
specialized resource acquisition functions that return either the in-memory
representation *or* a lock on whichever object is necessary to prevent any
update to the ondisk location.
The only repairs that should be made to the ondisk inode buffers are whatever
is necessary to get the in-core structure loaded.
This means fixing whatever is caught by the inode cluster buffer and inode fork
verifiers, and retrying the ``iget`` operation.
If the second ``iget`` fails, the repair has failed.
Once the in-memory representation is loaded, repair can lock the inode and can
subject it to comprehensive checks, repairs, and optimizations.
Most inode attributes are easy to check and constrain, or are user-controlled
arbitrary bit patterns; these are both easy to fix.
Dealing with the data and attr fork extent counts and the file block counts is
more complicated, because computing the correct value requires traversing the
forks, or if that fails, leaving the fields invalid and waiting for the fork
fsck functions to run.
The proposed patchset is the
`inode
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes>`_
repair series.
Quota record repair와 cache coherency
3032-3052Inode와 마찬가지로 quota record인 dquot에도 ondisk record와 in-memory representation이 모두 있어 같은 cache coherency 문제가 적용됩니다. 혼란스럽게도 XFS codebase에서는 두 representation을 모두 dquot이라고 부릅니다.
Ondisk quota record buffer에는 in-core structure를 load하는 데 필요한 repair만 해야 합니다. In-memory representation을 load한 뒤 검사해야 할 attribute는 명백히 잘못된 limit과 timer value뿐입니다.
Quota usage counter의 check와 repair는 `live quotacheck <quotacheck>` 절에서 별도로 설명합니다.
제안된 patchset은 `quota repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota>`_ series입니다.
Quota Record Repairs
--------------------
Similar to inodes, quota records ("dquots") also have both ondisk records and
an in-memory representation, and hence are subject to the same cache coherency
issues.
Somewhat confusingly, both are known as dquots in the XFS codebase.
The only repairs that should be made to the ondisk quota record buffers are
whatever is necessary to get the in-core structure loaded.
Once the in-memory representation is loaded, the only attributes needing
checking are obviously bad limits and timer values.
Quota usage counters are checked, repaired, and discussed separately in the
section about :ref:`live quotacheck <quotacheck>`.
The proposed patchset is the
`quota
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota>`_
repair series.
Freeze를 이용한 summary counter repair
3053-3152Filesystem summary counter는 free block, free inode, allocated inode 같은 filesystem resource의 availability를 추적합니다. Free space와 inode index를 순회해 계산할 수 있지만 느리기 때문에 XFS는 clean unmount 시 ondisk metadata를 반영해야 하는 copy를 ondisk superblock에 유지합니다. 성능을 위해 active transaction의 resource reservation에 핵심적인 incore copy도 유지합니다. Writer thread는 incore counter에서 worst-case resource quantity를 reserve하고 commit 때 사용하지 않은 양을 돌려주므로, superblock을 disk에 commit할 때만 serialize하면 됩니다.
XFS v5의 lazy superblock counter feature는 log recovery가 AG header에서 summary counter를 다시 계산하도록 하여 대부분의 transaction이 superblock을 건드릴 필요를 없앴습니다. XFS가 summary counter를 commit하는 때는 filesystem unmount뿐입니다. Contention을 더 줄이려고 incore counter를 percpu counter로 구현하여 각 CPU가 global incore counter에서 block batch를 받고 작은 allocation은 local batch에서 처리합니다.
이 고성능 구조 때문에 system 실행 중 percpu counter를 quiesce할 방법이 없어 online fsck의 check가 어렵습니다. Metadata walk로 올바른 counter 값을 계산할 수는 있지만 percpu counter 값을 stable하게 유지할 수 없어 walk가 끝날 때에는 값이 stale일 수 있습니다. 초기 online scrub는 incomplete scan flag를 userspace에 반환했지만 administrator에게 만족스러운 결과가 아닙니다. Repair하려면 metadata walk 동안 in-memory counter를 stabilize하고 정확한 값을 percpu counter에 install해야 합니다.
이를 위해 online fsck는 다른 program이 새 filesystem write를 시작하지 못하게 하고, background garbage collection thread를 disable하며, 기존 writer program이 kernel을 빠져나오기를 기다립니다. 그런 뒤 AG free space index, inode btree, realtime bitmap을 순회하여 네 summary counter의 올바른 값을 계산합니다. 이는 filesystem freeze와 비슷하지만 모든 요소가 필요하지는 않습니다.
항목Fscounter freeze 동작목적
Final freeze state`SB_FREEZE_COMPLETE`보다 하나 높은 값으로 설정다른 thread의 thaw와 다른 scrub thread의 fscounters freeze 시작 방지
LogLog를 quiesce하지 않음Counter check·repair에 불필요한 log 정지 회피
일반 VFS freeze와 달리 필요한 두 동작만 구분합니다.
이 code를 사용하면 summary counter를 check하고 correct하는 데 필요한 짧은 시간만 filesystem을 pause할 수 있습니다.
문제세부 내용영향
외부 thaw다른 program이 online fsck 모르게 filesystem을 unfreeze 가능잘못된 scan result와 repair
추가 lock·UAF다른 thaw를 막으려고 `freeze_fs()`를 감싸는 `->freeze_super` 추가; VFS `freeze_super`·`thaw_super`가 VFS superblock의 마지막 reference를 drop할 수 있음Underlying block device가 freeze한 동안 unmount되면 이후 access가 UAF bug; extra reference 해결책도 부적절
불필요한 log quiesceSummary counter check에는 log quiesce가 필요 없지만 VFS freeze가 수행Live fscounter fsck runtime 증가
잘못된 counter persistLog quiesce 중 XFS가 log cleaning의 일부로 possibly incorrect counter를 disk에 flush잘못된 값이 ondisk에 기록될 수 있음
VFS freeze bug`sync_filesystem`이 flush에 실패하고 error를 반환해도 freeze가 완료될 수 있었음Linux 5.17에서 수정
원문 ASCII sidebar의 다섯 문제와 결과를 구조화했습니다.
제안된 patchset은 `summary counter cleanup <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters>`_ series입니다.
.. _fscounters:
Freezing to Fix Summary Counters
--------------------------------
Filesystem summary counters track availability of filesystem resources such
as free blocks, free inodes, and allocated inodes.
This information could be compiled by walking the free space and inode indexes,
but this is a slow process, so XFS maintains a copy in the ondisk superblock
that should reflect the ondisk metadata, at least when the filesystem has been
unmounted cleanly.
For performance reasons, XFS also maintains incore copies of those counters,
which are key to enabling resource reservations for active transactions.
Writer threads reserve the worst-case quantities of resources from the
incore counter and give back whatever they don't use at commit time.
It is therefore only necessary to serialize on the superblock when the
superblock is being committed to disk.
The lazy superblock counter feature introduced in XFS v5 took this even further
by training log recovery to recompute the summary counters from the AG headers,
which eliminated the need for most transactions even to touch the superblock.
The only time XFS commits the summary counters is at filesystem unmount.
To reduce contention even further, the incore counter is implemented as a
percpu counter, which means that each CPU is allocated a batch of blocks from a
global incore counter and can satisfy small allocations from the local batch.
The high-performance nature of the summary counters makes it difficult for
online fsck to check them, since there is no way to quiesce a percpu counter
while the system is running.
Although online fsck can read the filesystem metadata to compute the correct
values of the summary counters, there's no way to hold the value of a percpu
counter stable, so it's quite possible that the counter will be out of date by
the time the walk is complete.
Earlier versions of online scrub would return to userspace with an incomplete
scan flag, but this is not a satisfying outcome for a system administrator.
For repairs, the in-memory counters must be stabilized while walking the
filesystem metadata to get an accurate reading and install it in the percpu
counter.
To satisfy this requirement, online fsck must prevent other programs in the
system from initiating new writes to the filesystem, it must disable background
garbage collection threads, and it must wait for existing writer programs to
exit the kernel.
Once that has been established, scrub can walk the AG free space indexes, the
inode btrees, and the realtime bitmap to compute the correct value of all
four summary counters.
This is very similar to a filesystem freeze, though not all of the pieces are
necessary:
- The final freeze state is set one higher than ``SB_FREEZE_COMPLETE`` to
prevent other threads from thawing the filesystem, or other scrub threads
from initiating another fscounters freeze.
- It does not quiesce the log.
With this code in place, it is now possible to pause the filesystem for just
long enough to check and correct the summary counters.
+--------------------------------------------------------------------------+
| **Historical Sidebar**: |
+--------------------------------------------------------------------------+
| The initial implementation used the actual VFS filesystem freeze |
| mechanism to quiesce filesystem activity. |
| With the filesystem frozen, it is possible to resolve the counter values |
| with exact precision, but there are many problems with calling the VFS |
| methods directly: |
| |
| - Other programs can unfreeze the filesystem without our knowledge. |
| This leads to incorrect scan results and incorrect repairs. |
| |
| - Adding an extra lock to prevent others from thawing the filesystem |
| required the addition of a ``->freeze_super`` function to wrap |
| ``freeze_fs()``. |
| This in turn caused other subtle problems because it turns out that |
| the VFS ``freeze_super`` and ``thaw_super`` functions can drop the |
| last reference to the VFS superblock, and any subsequent access |
| becomes a UAF bug! |
| This can happen if the filesystem is unmounted while the underlying |
| block device has frozen the filesystem. |
| This problem could be solved by grabbing extra references to the |
| superblock, but it felt suboptimal given the other inadequacies of |
| this approach. |
| |
| - The log need not be quiesced to check the summary counters, but a VFS |
| freeze initiates one anyway. |
| This adds unnecessary runtime to live fscounter fsck operations. |
| |
| - Quiescing the log means that XFS flushes the (possibly incorrect) |
| counters to disk as part of cleaning the log. |
| |
| - A bug in the VFS meant that freeze could complete even when |
| sync_filesystem fails to flush the filesystem and returns an error. |
| This bug was fixed in Linux 5.17. |
+--------------------------------------------------------------------------+
The proposed patchset is the
`summary counter cleanup
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters>`_
series.
전체 filesystem scan
3153-3171일부 metadata 유형은 filesystem의 모든 file을 순회하며 observation을 기록하고 이를 ondisk record와 비교해야만 검사할 수 있습니다. 다른 online repair와 마찬가지로, 관찰 결과를 replacement structure에 기록한 뒤 atomic하게 commit하여 repair합니다.
그러나 수천억 개 file을 검사하려고 filesystem 전체를 정지하면 downtime이 지나치게 길어지므로 실용적이지 않습니다. 따라서 online fsck에는 live filesystem에서 모든 file을 scan할 infrastructure가 필요하며, scan 중 data 수집을 어떻게 관리할지와 다른 thread가 만드는 변경을 어떻게 따라갈지라는 두 문제를 해결해야 합니다.
과제질문필요한 성질
Scan 관리Scrub은 data를 수집하는 동안 scan을 어떻게 관리하는가?진행 위치와 이미 방문한 범위를 안정적으로 추적
Concurrent update 반영다른 thread가 system에 만드는 변경을 scan이 어떻게 놓치지 않는가?방문 경계와 update 위치를 비교해 scan data에 포함할지 결정
전체 file walk를 실행 중인 filesystem과 조정하기 위해 풀어야 할 질문을 보존합니다.
Full Filesystem Scans
---------------------
Certain types of metadata can only be checked by walking every file in the
entire filesystem to record observations and comparing the observations against
what's recorded on disk.
Like every other type of online repair, repairs are made by writing those
observations to disk in a replacement structure and committing it atomically.
However, it is not practical to shut down the entire filesystem to examine
hundreds of billions of files because the downtime would be excessive.
Therefore, online fsck must build the infrastructure to manage a live scan of
all the files in the filesystem.
There are two questions that need to be solved to perform a live walk:
- How does scrub manage the scan while it is collecting data?
- How does the scan keep abreast of changes being made to the system by other
threads?
조정된 inode scan
3172-32521970년대 초기 Unix filesystem에서 각 directory entry는 index number인 *inumber*를 담았고, 이 값은 file attribute와 data block mapping을 설명하는 고정 크기 record인 *inode*의 ondisk array인 *itable*을 찾아가는 index였습니다. 이 체계는 J. Lions의 `"inode (5659)" <http://www.lemis.com/grog/Documentation/Lions/>`_, *Lions' Commentary on UNIX, 6th Edition* (University of New South Wales, Department of Computer Science, 1977년 11월), pp. 18-2와 D. Ritchie·K. Thompson의 `"Implementation of the File System" <https://archive.org/details/bstj57-6-1905/page/n8/mode/1up>`_, *The UNIX Time-Sharing System* (The Bell System Technical Journal, 1978년 7월), pp. 1913-4에 설명되어 있습니다.
XFS도 이 설계를 대부분 유지하지만, 이제 inumber는 data section filesystem의 전체 space를 대상으로 하는 search key입니다. Inumber는 64-bit integer로 표현할 수 있는 연속 keyspace를 이루지만 inode 자체는 그 안에 sparse하게 분포합니다. Scan은 `0x0`에서 시작해 `0xFFFFFFFFFFFFFFFF`에서 끝나도록 inumber keyspace를 선형으로 진행하므로, 진행 상태를 추적할 scan cursor object가 필요합니다.
Sparse keyspace를 다루는 scan cursor는 두 부분으로 구성됩니다. Examination cursor는 다음에 검사할 inode를 가리키고, visited inode cursor는 이미 방문한 keyspace 범위를 나타냅니다. 후자는 concurrent filesystem update를 scan data에 반영해야 하는지 판단하는 데 반드시 필요합니다.
Cursor추적 대상동시성 판단에서의 역할
Examination cursor다음에 검사할 inodeScanner가 확보해야 할 다음 incore inode 결정
Visited inode cursor이미 방문한 inumber keyspace의 마지막 위치Concurrent update가 완료 구간 뒤에서 발생했는지 판단
다음 검사 대상과 완료된 keyspace 경계를 분리해 추적합니다.
Scan cursor의 전진은 `xchk_iscan_iter`에 캡슐화된 다음 6단계 과정입니다.
단계동작보장·결과
1Visited inode cursor가 가리키는 inode가 속한 AG의 AGI buffer를 lockCursor 전진 중 이 AG의 inode allocate·free 방지
2Per-AG inode btree에서 방금 방문한 inode 다음의 inumber를 lookupKeyspace상 바로 인접하지 않을 수 있는 다음 allocated inode 탐색
3현재 AG에 inode가 더 없으면 examination cursor를 다음 AG의 keyspace 시작으로 옮기고 visited cursor를 현재 AG keyspace의 마지막 가능한 inode로 조정Segmented XFS inumber에서 다음 AG 직전까지 모두 방문했음을 표시; 미검사 AG가 있으면 AGI를 unlock하고 1단계로 반복하며, 없으면 두 cursor를 keyspace 끝으로 설정해 scan 완료
4현재 AG에 inode가 남았으면 examination cursor를 inode btree가 allocated로 표시한 다음 inode로 옮기고 visited cursor를 그 직전 inode로 조정AGI lock을 유지하므로 visited cursor가 방금 지난 keyspace에 새 inode가 생기지 않았음
5Examination cursor의 inumber에 해당하는 incore inode를 획득AGI lock을 여기까지 유지하여 전체 keyspace 전진이 안전했고, 다음 inode가 안정화되어 scan이 incore inode를 release할 때까지 filesystem에서 사라지지 않음을 보장
6AGI lock을 해제하고 incore inode를 caller에 반환Caller가 안정화된 다음 inode를 검사
AGI lock 아래에서 sparse inumber keyspace를 안전하게 건너는 원문의 1–6단계를 보존합니다.
핵심은 AGI buffer lock을 다음 incore inode 획득 시점까지 유지하는 것입니다. 이 잠금이 cursor가 sparse keyspace와 AG 경계를 건너는 동안 allocate·free를 막고, 이미 방문했다고 표시한 범위에 inode가 뒤늦게 나타나거나 반환할 inode가 사라지는 일을 방지합니다.
.. _iscan:
Coordinated Inode Scans
```````````````````````
In the original Unix filesystems of the 1970s, each directory entry contained
an index number (*inumber*) which was used as an index into on ondisk array
(*itable*) of fixed-size records (*inodes*) describing a file's attributes and
its data block mapping.
This system is described by J. Lions, `"inode (5659)"
<http://www.lemis.com/grog/Documentation/Lions/>`_ in *Lions' Commentary on
UNIX, 6th Edition*, (Dept. of Computer Science, the University of New South
Wales, November 1977), pp. 18-2; and later by D. Ritchie and K. Thompson,
`"Implementation of the File System"
<https://archive.org/details/bstj57-6-1905/page/n8/mode/1up>`_, from *The UNIX
Time-Sharing System*, (The Bell System Technical Journal, July 1978), pp.
1913-4.
XFS retains most of this design, except now inumbers are search keys over all
the space in the data section filesystem.
They form a continuous keyspace that can be expressed as a 64-bit integer,
though the inodes themselves are sparsely distributed within the keyspace.
Scans proceed in a linear fashion across the inumber keyspace, starting from
``0x0`` and ending at ``0xFFFFFFFFFFFFFFFF``.
Naturally, a scan through a keyspace requires a scan cursor object to track the
scan progress.
Because this keyspace is sparse, this cursor contains two parts.
The first part of this scan cursor object tracks the inode that will be
examined next; call this the examination cursor.
Somewhat less obviously, the scan cursor object must also track which parts of
the keyspace have already been visited, which is critical for deciding if a
concurrent filesystem update needs to be incorporated into the scan data.
Call this the visited inode cursor.
Advancing the scan cursor is a multi-step process encapsulated in
``xchk_iscan_iter``:
1. Lock the AGI buffer of the AG containing the inode pointed to by the visited
inode cursor.
This guarantee that inodes in this AG cannot be allocated or freed while
advancing the cursor.
2. Use the per-AG inode btree to look up the next inumber after the one that
was just visited, since it may not be keyspace adjacent.
3. If there are no more inodes left in this AG:
a. Move the examination cursor to the point of the inumber keyspace that
corresponds to the start of the next AG.
b. Adjust the visited inode cursor to indicate that it has "visited" the
last possible inode in the current AG's inode keyspace.
XFS inumbers are segmented, so the cursor needs to be marked as having
visited the entire keyspace up to just before the start of the next AG's
inode keyspace.
c. Unlock the AGI and return to step 1 if there are unexamined AGs in the
filesystem.
d. If there are no more AGs to examine, set both cursors to the end of the
inumber keyspace.
The scan is now complete.
4. Otherwise, there is at least one more inode to scan in this AG:
a. Move the examination cursor ahead to the next inode marked as allocated
by the inode btree.
b. Adjust the visited inode cursor to point to the inode just prior to where
the examination cursor is now.
Because the scanner holds the AGI buffer lock, no inodes could have been
created in the part of the inode keyspace that the visited inode cursor
just advanced.
5. Get the incore inode for the inumber of the examination cursor.
By maintaining the AGI buffer lock until this point, the scanner knows that
it was safe to advance the examination cursor across the entire keyspace,
and that it has stabilized this next inode so that it cannot disappear from
the filesystem until the scan releases the incore inode.
6. Drop the AGI lock and return the incore inode to the caller.Online fsck의 inode scan loop
3253-3288Online fsck function은 다음 절차로 filesystem의 모든 file을 scan합니다. 원문 단계 번호는 `1`, `2`, `8`이며 그대로 보존합니다.
단계호출·동작동시성 조건
1`xchk_iscan_start`를 호출하여 scan 시작Scan cursor 초기화
2`xchk_iscan_iter`로 cursor를 전진시켜 다음 inode 획득Inode가 반환되면 하위 a–d 수행
2.aScan 중 update를 막도록 inode lock검사 대상 안정화
2.bInode scanMetadata observation 수집
2.cInode lock을 유지한 채 `xchk_iscan_mark_visited`로 visited inode cursor가 이 inode를 가리키게 조정검사 완료 경계와 concurrent update 판단의 일관성 유지
2.dInode unlock 및 release다음 반복 허용
8`xchk_iscan_teardown` 호출Scan 완료 및 자원 정리
Cursor 시작·반복·종료와 inode lock 아래의 visited 표시 순서를 나타냅니다.
Caller가 사용할 incore inode를 가져오는 과정은 inode cache 때문에 미묘합니다. 첫째, inode metadata가 inode cache에 load할 수 있을 만큼 일관되어 있어야 합니다. 둘째, incore inode가 중간 상태에 걸려 있으면 scan coordinator가 AGI를 release하고 main filesystem을 push하여 inode를 다시 load 가능한 상태로 돌려야 합니다.
조건Coordinator 동작목적
Metadata consistencyInode cache에 load 가능한 수준인지 확인Caller가 incore inode를 사용할 수 있게 함
Intermediate stateAGI를 release하고 main filesystem을 pushInode를 load 가능한 상태로 복귀
Cache load와 intermediate state 처리 조건을 구분합니다.
제안된 patch는 `inode scanner <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_ series이며, 새 기능의 첫 사용자는 `online quotacheck <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_ series입니다.
Online fsck functions scan all files in the filesystem as follows:
1. Start a scan by calling ``xchk_iscan_start``.
2. Advance the scan cursor (``xchk_iscan_iter``) to get the next inode.
If one is provided:
a. Lock the inode to prevent updates during the scan.
b. Scan the inode.
c. While still holding the inode lock, adjust the visited inode cursor
(``xchk_iscan_mark_visited``) to point to this inode.
d. Unlock and release the inode.
8. Call ``xchk_iscan_teardown`` to complete the scan.
There are subtleties with the inode cache that complicate grabbing the incore
inode for the caller.
Obviously, it is an absolute requirement that the inode metadata be consistent
enough to load it into the inode cache.
Second, if the incore inode is stuck in some intermediate state, the scan
coordinator must release the AGI and push the main filesystem to get the inode
back into a loadable state.
The proposed patches are the
`inode scanner
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_
series.
The first user of the new functionality is the
`online quotacheck
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_
series.
Inode management와 resource ordering
3289-3356일반 filesystem code에서 allocated XFS incore inode reference는 기존 file의 incore context 생성에 metadata update가 필요하지 않으므로 항상 transaction context 밖에서 `xfs_iget`으로 얻습니다. 반면 file creation의 일부로 incore inode reference를 얻는 작업은 ondisk inode btree index update와 실제 ondisk inode initialization의 atomicity를 filesystem이 보장해야 하므로 transaction context 안에서 수행해야 합니다.
Incore inode reference는 항상 transaction context 밖에서 `xfs_irele`로 release합니다. Release 과정에서 ondisk update가 필요할 수 있는 활동이 있기 때문입니다.
활동조건필요한 ondisk 처리
Writeback`DONTCACHE` inode release 중 VFS가 writeback 시작을 결정Dirty state를 disk에 반영
Speculative preallocation미사용 speculative allocation 존재Reservation 해제
Unlinked file inactivationUnlinked file이 마지막 reference를 잃음Ondisk metadata에서 모든 resource를 release하고 inode free
Reference release가 유발할 수 있는 세 가지 ondisk update 활동을 보존합니다.
이 활동들을 통틀어 inode inactivation이라고 합니다. Inactivation은 dirty file page 전체의 writeback을 시작하는 VFS 부분과 XFS-specific information을 정리하고 unlinked inode를 free하는 XFS 부분으로 나뉩니다. Inode가 unlinked 상태이거나 file handle operation 뒤 unconnected 상태라면 kernel은 즉시 inactivation machinery에 넣습니다.
정상 update operation은 deadlock을 피하기 위해 다음 순서로 resource를 획득합니다.
순서Resource용도·조건
1Inode reference (`iget`)대상 inode 확보
2Filesystem freeze protection (`mnt_want_write_file`)Repair인 경우 write 보호
3Inode `IOLOCK` (VFS `i_rwsem`)File I/O 제어
4Inode `MMAPLOCK` (page cache `invalidate_lock`)Page cache mapping을 update할 수 있는 operation 제어
5Log feature enablement필요 log feature 활성화
6Transaction log space grantLog 공간 확보
7Data·realtime device의 transaction용 spaceBlock 공간 확보
8Incore dquot referenceFile repair 시 획득하되 lock하지 않음
9Inode `ILOCK`File metadata update 보호
10AG header buffer lock / realtime metadata inode `ILOCK`Allocation group 또는 realtime metadata 보호
11Realtime metadata buffer lock해당하는 경우 획득
12Extent mapping btree block해당하는 경우 획득
원문의 1–12단계 lock·reference·space 획득 순서를 그대로 보존합니다.
Resource는 흔히 역순으로 release하지만 반드시 그래야 하는 것은 아닙니다. Online fsck는 일반 XFS operation과 달리 lock ordering의 뒤 단계에서 보통 획득하는 object를 먼저 검사한 뒤, 더 앞 단계에서 획득해야 하는 object와 cross-reference하기로 결정할 수 있습니다. 이어지는 절들은 이런 순서 역전 상황에서 deadlock을 피하는 구체적인 방법을 설명합니다.
Inode Management
````````````````
In regular filesystem code, references to allocated XFS incore inodes are
always obtained (``xfs_iget``) outside of transaction context because the
creation of the incore context for an existing file does not require metadata
updates.
However, it is important to note that references to incore inodes obtained as
part of file creation must be performed in transaction context because the
filesystem must ensure the atomicity of the ondisk inode btree index updates
and the initialization of the actual ondisk inode.
References to incore inodes are always released (``xfs_irele``) outside of
transaction context because there are a handful of activities that might
require ondisk updates:
- The VFS may decide to kick off writeback as part of a ``DONTCACHE`` inode
release.
- Speculative preallocations need to be unreserved.
- An unlinked file may have lost its last reference, in which case the entire
file must be inactivated, which involves releasing all of its resources in
the ondisk metadata and freeing the inode.
These activities are collectively called inode inactivation.
Inactivation has two parts -- the VFS part, which initiates writeback on all
dirty file pages, and the XFS part, which cleans up XFS-specific information
and frees the inode if it was unlinked.
If the inode is unlinked (or unconnected after a file handle operation), the
kernel drops the inode into the inactivation machinery immediately.
During normal operation, resource acquisition for an update follows this order
to avoid deadlocks:
1. Inode reference (``iget``).
2. Filesystem freeze protection, if repairing (``mnt_want_write_file``).
3. Inode ``IOLOCK`` (VFS ``i_rwsem``) lock to control file IO.
4. Inode ``MMAPLOCK`` (page cache ``invalidate_lock``) lock for operations that
can update page cache mappings.
5. Log feature enablement.
6. Transaction log space grant.
7. Space on the data and realtime devices for the transaction.
8. Incore dquot references, if a file is being repaired.
Note that they are not locked, merely acquired.
9. Inode ``ILOCK`` for file metadata updates.
10. AG header buffer locks / Realtime metadata inode ILOCK.
11. Realtime metadata buffer locks, if applicable.
12. Extent mapping btree blocks, if applicable.
Resources are often released in the reverse order, though this is not required.
However, online fsck differs from regular XFS operations because it may examine
an object that normally is acquired in a later stage of the locking order, and
then decide to cross-reference the object with an object that is acquired
earlier in the order.
The next few sections detail the specific ways in which online fsck takes care
to avoid deadlocks.Scrub 중 iget과 irele
3357-3389Scrub operation을 대신해 수행하는 inode scan은 transaction context에서 실행되며, 이미 lock되어 transaction에 bind된 resource가 있을 수도 있습니다. `iget`은 기존 transaction context에서도 동작할 수 있으므로 큰 문제는 아니지만, 일반 filesystem lock order에서 inode reference보다 앞서는 모든 bound resource를 먼저 획득해야 합니다.
VFS `iput`이 다른 reference가 없는 linked inode를 받으면, 보통 memory 부족으로 inode를 free하기 전에 다른 process가 file을 다시 열 경우 시간을 절약하려고 inode를 LRU list에 넣습니다. Filesystem caller는 inode에 `DONTCACHE` flag를 설정해 LRU 과정을 생략하고 kernel이 inode를 즉시 inactivation machinery로 보내도록 할 수 있습니다.
과거에는 inode reference를 drop한 process가 언제나 inactivation을 직접 수행했습니다. Scrub이 이미 transaction을 보유할 수 있고 XFS가 nested transaction을 지원하지 않기 때문에 이는 문제가 됩니다. 반대로 scrub transaction이 없다면 사용하지 않는 inode를 즉시 drop해 cache pollution을 피하는 편이 바람직합니다.
상태`DONTCACHE` 처리목적
Scrub transaction 보유즉시 inactivation을 피하도록 flag 조정XFS가 지원하지 않는 nested transaction 방지
Scrub transaction 없음사용하지 않는 inode를 즉시 drop하도록 flag 조정Cache pollution 방지
Scrub transaction 유무에 따라 `DONTCACHE`를 조정하는 이유를 구분합니다.
이 미묘한 차이를 반영하기 위해 online fsck는 별도 `xchk_irele` function으로 `DONTCACHE` flag를 set 또는 clear하여 필요한 release behavior를 얻습니다. 제안된 patchset은 `scrub iget usage <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iget-fixes>`_와 `dir iget usage <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-dir-iget-fixes>`_ 수정입니다.
iget and irele During a Scrub
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
An inode scan performed on behalf of a scrub operation runs in transaction
context, and possibly with resources already locked and bound to it.
This isn't much of a problem for ``iget`` since it can operate in the context
of an existing transaction, as long as all of the bound resources are acquired
before the inode reference in the regular filesystem.
When the VFS ``iput`` function is given a linked inode with no other
references, it normally puts the inode on an LRU list in the hope that it can
save time if another process re-opens the file before the system runs out
of memory and frees it.
Filesystem callers can short-circuit the LRU process by setting a ``DONTCACHE``
flag on the inode to cause the kernel to try to drop the inode into the
inactivation machinery immediately.
In the past, inactivation was always done from the process that dropped the
inode, which was a problem for scrub because scrub may already hold a
transaction, and XFS does not support nesting transactions.
On the other hand, if there is no scrub transaction, it is desirable to drop
otherwise unused inodes immediately to avoid polluting caches.
To capture these nuances, the online fsck code has a separate ``xchk_irele``
function to set or clear the ``DONTCACHE`` flag to get the required release
behavior.
Proposed patchsets include fixing
`scrub iget usage
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iget-fixes>`_ and
`dir iget usage
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-dir-iget-fixes>`_.
Inode locking
3390-3424일반 filesystem code에서 VFS와 XFS는 여러 inode lock을 잘 알려진 순서로 획득합니다.
Lock·상황획득 순서추가 제약
여러 `IOLOCK`Directory tree update는 parent → child, 그 밖에는 `struct inode` object address의 숫자 순서ABBA를 피하는 전역 순서
Regular file `MMAPLOCK``IOLOCK` 뒤에 획득Page fault 중지
여러 `MMAPLOCK``struct address_space` object address의 숫자 순서동일 class lock ordering
`IOLOCK`·`MMAPLOCK`Transaction allocate 전에 획득기존 filesystem code 구조가 요구
여러 `ILOCK`Inumber 순서File metadata lock ordering
IOLOCK·MMAPLOCK·ILOCK의 ordering key와 transaction 경계를 보존합니다.
Coordinated inode scan에서는 이 관례를 그대로 따를 수 없습니다. Directory tree scanner는 현재 scan 중인 file의 `IOLOCK`을 보유한 상태에서 directory link 반대편 file의 `IOLOCK`을 얻어야 합니다. 손상된 directory tree에 cycle이 있으면 일반 inode locking function을 사용할 경우 `xfs_scrub`이 ABBA deadlock에 갇힐 수 있습니다.
해결 방법은 online fsck가 같은 class의 두 번째 lock을 얻을 때 언제나 trylock을 사용하는 것입니다. Trylock이 실패하면 scrub은 모든 inode lock을 drop하고 trylock loop로 필요한 resource 전체를 다시 획득합니다.
동작효과후속 의무
같은 class의 두 번째 lock에 trylockABBA deadlock 회피실패 시 전체 inode lock drop
Trylock loop로 모든 resource 재획득Pending fatal signal 검사 가능Filesystem deadlock과 unresponsive process 방지
Lock cycle 전후 scrub 대상 측정Unlock 동안 발생한 변경 검출변경에 맞춰 검사 결과를 재평가
Deadlock 회피와 재획득 뒤 일관성 검사를 함께 나타냅니다.
Trylock loop 덕분에 scrub은 pending fatal signal을 검사할 수 있지만, lock을 drop했다가 다시 얻는 동안 대상이 바뀔 수 있습니다. 따라서 online fsck는 lock cycle 전후에 scrub 중인 resource를 측정하고 변경을 검출해 그에 맞게 반응해야 합니다.
.. _ilocking:
Locking Inodes
^^^^^^^^^^^^^^
In regular filesystem code, the VFS and XFS will acquire multiple IOLOCK locks
in a well-known order: parent → child when updating the directory tree, and
in numerical order of the addresses of their ``struct inode`` object otherwise.
For regular files, the MMAPLOCK can be acquired after the IOLOCK to stop page
faults.
If two MMAPLOCKs must be acquired, they are acquired in numerical order of
the addresses of their ``struct address_space`` objects.
Due to the structure of existing filesystem code, IOLOCKs and MMAPLOCKs must be
acquired before transactions are allocated.
If two ILOCKs must be acquired, they are acquired in inumber order.
Inode lock acquisition must be done carefully during a coordinated inode scan.
Online fsck cannot abide these conventions, because for a directory tree
scanner, the scrub process holds the IOLOCK of the file being scanned and it
needs to take the IOLOCK of the file at the other end of the directory link.
If the directory tree is corrupt because it contains a cycle, ``xfs_scrub``
cannot use the regular inode locking functions and avoid becoming trapped in an
ABBA deadlock.
Solving both of these problems is straightforward -- any time online fsck
needs to take a second lock of the same class, it uses trylock to avoid an ABBA
deadlock.
If the trylock fails, scrub drops all inode locks and use trylock loops to
(re)acquire all necessary resources.
Trylock loops enable scrub to check for pending fatal signals, which is how
scrub avoids deadlocking the filesystem or becoming an unresponsive process.
However, trylock loops means that online fsck must be prepared to measure the
resource being scrubbed before and after the lock cycle to detect changes and
react accordingly.
Case study: directory parent 찾기
3425-3450Directory parent pointer repair를 예로 들면, online fsck는 directory의 dotdot dirent가 parent directory를 가리키는지와 그 parent가 child directory를 아래로 가리키는 dirent를 정확히 하나만 포함하는지를 검증해야 합니다.
관계검증 조건손상 판단
Child → parentChild directory의 dotdot dirent가 parent directory를 가리킴잘못된 parent pointer
Parent → childParent directory에 child를 가리키는 dirent가 정확히 하나누락 또는 중복 child entry
Child의 상향 link와 parent의 하향 link를 짝으로 검증합니다.
이 관계를 완전히 검증하고 가능하면 repair하려면 child를 lock한 채 filesystem의 모든 directory를 순회해야 하며, 동시에 directory tree update도 계속 일어납니다. Coordinated inode scan은 inode 하나도 놓치지 않고 filesystem을 순회할 방법을 제공합니다.
Child directory는 dotdot dirent update를 막기 위해 lock 상태를 유지합니다. Scanner가 prospective parent를 lock하지 못하면 child와 parent lock을 모두 drop한 뒤 다시 획득할 수 있습니다. Directory가 unlock된 동안 dotdot entry가 바뀌었다면 move 또는 rename operation이 child의 parentage를 변경한 것이므로 scan을 조기에 종료할 수 있습니다.
상황동작판정
Prospective parent lock 실패Child와 prospective parent를 모두 drop 후 relockABBA 회피
Relock 뒤 dotdot 유지Directory scan 계속같은 parentage에 대한 검증 유효
Relock 뒤 dotdot 변경Scan 조기 종료Move/rename이 parentage를 바꿨으므로 이전 observation 폐기
Unlock·relock 사이의 parentage 변경을 처리하는 흐름입니다.
제안된 patchset은 `directory repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_ series입니다.
.. _dirparent:
Case Study: Finding a Directory Parent
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Consider the directory parent pointer repair code as an example.
Online fsck must verify that the dotdot dirent of a directory points up to a
parent directory, and that the parent directory contains exactly one dirent
pointing down to the child directory.
Fully validating this relationship (and repairing it if possible) requires a
walk of every directory on the filesystem while holding the child locked, and
while updates to the directory tree are being made.
The coordinated inode scan provides a way to walk the filesystem without the
possibility of missing an inode.
The child directory is kept locked to prevent updates to the dotdot dirent, but
if the scanner fails to lock a parent, it can drop and relock both the child
and the prospective parent.
If the dotdot entry changes while the directory is unlocked, then a move or
rename operation must have changed the child's parentage, and the scan can
exit early.
The proposed patchset is the
`directory repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_
series.
Filesystem hooks
3451-3523Full filesystem scan 중 online fsck에 필요한 두 번째 지원 요소는 다른 thread가 filesystem에 만드는 update를 계속 통지받는 능력입니다. Dynamic environment에서는 과거 state와의 비교만으로 의미 있는 결과를 얻을 수 없습니다. Linux kernel의 filesystem hook과 :ref:`static key <jump_labels>` infrastructure가 online fsck로 하여금 일반 filesystem operation을 monitor할 수 있게 합니다.
Filesystem hook은 진행 중인 filesystem operation 정보를 downstream consumer에 전달하며, 여기서 consumer는 항상 online fsck function입니다. 여러 fsck function이 병렬로 실행될 수 있으므로 Linux notifier call chain으로 관심 있는 임의 개수의 fsck process에 update를 dispatch합니다. Call chain은 runtime에 구성할 수 있는 dynamic list이고, hook은 XFS module 내부 전용이므로 전달 정보에는 checking function이 observation을 갱신하는 데 꼭 필요한 내용만 포함됩니다.
현재 XFS hook 구현은 thread가 많은 workload에 미치는 영향을 줄이기 위해 SRCU notifier chain을 사용합니다. 일반 blocking notifier chain은 rwsem을 사용하며 single-threaded application에서는 overhead가 훨씬 낮아 보입니다. Blocking chain과 static key 조합이 더 빠를 가능성도 있으므로 이 부분은 추가 연구가 필요합니다.
Filesystem의 특정 지점에 hook을 설치하려면 다음 구성 요소가 필요합니다.
순서구성 요소요건
1`struct xfs_hooks` object잘 알려진 incore filesystem object처럼 편리한 위치에 embed
2Action definition각 hook마다 action code와 action context를 담는 structure 정의
3Typed provider wrapper`xfs_hooks`·`xfs_hook` object를 감싸는 적절한 function·struct를 제공해 type checking으로 올바른 사용 보장
4Regular filesystem callsiteAction code와 data structure로 `xfs_hooks_call` 호출; filesystem update가 transaction에 commit되는 지점과 인접하고 그보다 빠르지 않아야 함; 일반적으로 sleep 가능하고 memory reclaim·locking recursion에 취약하지 않아야 하나 정확한 요건은 caller·callee context에 의존
5Online fsck scan stateScan data structure, 접근 조정 lock, `struct xfs_hook` object 정의; scanner와 일반 filesystem code가 같은 순서로 resource 획득
6Hook callbackAction code와 data structure를 받는 C function 정의; update object가 이미 scan된 경우 hook 정보를 scan data에 적용
7Setup·enableScan 시작을 위해 inode를 unlock하기 전에 `xfs_hooks_setup`으로 `struct xfs_hook` 초기화 후 `xfs_hooks_add`로 hook 활성화
8DisableScan 완료 뒤 `xfs_hooks_del`로 hook 비활성화
Provider·callsite·online fsck consumer의 설치와 해제 요건을 원문 순서대로 보존합니다.
Complexity를 줄이기 위해 hook 수는 최소로 유지해야 합니다. Online fsck가 실행 중이 아닐 때 filesystem hook overhead를 거의 0으로 낮추기 위해 static key를 사용합니다.
.. _fshooks:
Filesystem Hooks
`````````````````
The second piece of support that online fsck functions need during a full
filesystem scan is the ability to stay informed about updates being made by
other threads in the filesystem, since comparisons against the past are useless
in a dynamic environment.
Two pieces of Linux kernel infrastructure enable online fsck to monitor regular
filesystem operations: filesystem hooks and :ref:`static keys<jump_labels>`.
Filesystem hooks convey information about an ongoing filesystem operation to
a downstream consumer.
In this case, the downstream consumer is always an online fsck function.
Because multiple fsck functions can run in parallel, online fsck uses the Linux
notifier call chain facility to dispatch updates to any number of interested
fsck processes.
Call chains are a dynamic list, which means that they can be configured at
run time.
Because these hooks are private to the XFS module, the information passed along
contains exactly what the checking function needs to update its observations.
The current implementation of XFS hooks uses SRCU notifier chains to reduce the
impact to highly threaded workloads.
Regular blocking notifier chains use a rwsem and seem to have a much lower
overhead for single-threaded applications.
However, it may turn out that the combination of blocking chains and static
keys are a more performant combination; more study is needed here.
The following pieces are necessary to hook a certain point in the filesystem:
- A ``struct xfs_hooks`` object must be embedded in a convenient place such as
a well-known incore filesystem object.
- Each hook must define an action code and a structure containing more context
about the action.
- Hook providers should provide appropriate wrapper functions and structs
around the ``xfs_hooks`` and ``xfs_hook`` objects to take advantage of type
checking to ensure correct usage.
- A callsite in the regular filesystem code must be chosen to call
``xfs_hooks_call`` with the action code and data structure.
This place should be adjacent to (and not earlier than) the place where
the filesystem update is committed to the transaction.
In general, when the filesystem calls a hook chain, it should be able to
handle sleeping and should not be vulnerable to memory reclaim or locking
recursion.
However, the exact requirements are very dependent on the context of the hook
caller and the callee.
- The online fsck function should define a structure to hold scan data, a lock
to coordinate access to the scan data, and a ``struct xfs_hook`` object.
The scanner function and the regular filesystem code must acquire resources
in the same order; see the next section for details.
- The online fsck code must contain a C function to catch the hook action code
and data structure.
If the object being updated has already been visited by the scan, then the
hook information must be applied to the scan data.
- Prior to unlocking inodes to start the scan, online fsck must call
``xfs_hooks_setup`` to initialize the ``struct xfs_hook``, and
``xfs_hooks_add`` to enable the hook.
- Online fsck must call ``xfs_hooks_del`` to disable the hook once the scan is
complete.
The number of hooks should be kept to a minimum to reduce complexity.
Static keys are used to reduce the overhead of filesystem hooks to nearly
zero when online fsck is not running.
Scan 중 live update
3524-3601Online fsck scanning code와 hook이 설치된 filesystem code의 실행 경로는 다음과 같습니다. 두 경로는 같은 inode lock과 같은 scan-data lock을 사용합니다.
경로실행 순서공유 지점
Filesystem update pathOther program → inode lock → AG header lock → filesystem function → notifier call chain → scrub hook function → scan data mutex → update scan dataScanner와 같은 inode lock 및 scan data mutex
Online scrub path`xfs_scrub` → inode scanner → scrub function → inode lock → scan data mutex → update scan dataUpdater와 같은 inode lock 및 scan data mutex
원문 ASCII 흐름을 update producer와 scanner가 공유 lock에서 만나는 구조로 다시 그렸습니다.
Checking code와 filesystem update code가 올바르게 상호작용하려면 다음 규칙을 따라야 합니다.
규칙요구 사항이유
동일 inode lockHook 대상 filesystem function은 notifier call chain 호출 전에 scrub scanner가 inode scan에 쓰는 것과 같은 lock 획득Observation과 update의 직렬화
Scan-data lockScanning function과 scrub hook function 모두 scan data lock을 획득해 접근 조정In-memory observation의 동시 update 보호
Visited gateUpdate inode가 이미 scan된 경우에만 live update를 observation에 추가; `xchk_iscan_want_live_update` helper predicate 사용아직 scan하지 않은 inode의 update를 중복 반영하지 않음
Caller state 불변Hook function은 실행 중 transaction을 포함한 caller state를 바꾸지 않고, hook 대상 filesystem function과 충돌할 resource를 획득하지 않음Recursion·deadlock·transaction corruption 방지
Abort escape다른 규칙을 깨뜨리지 않기 위해 hook function이 inode scan을 abort할 수 있음안전하게 처리할 수 없는 update에서 탈출
Lock ordering, visited 판단, callback 제약과 abort escape를 구분합니다.
Inode scan API는 다음 다섯 function으로 구성됩니다.
API역할반환·판단
`xchk_iscan_start`Scan 시작Cursor와 coordinator 초기화
`xchk_iscan_iter`Scan의 다음 inode reference 획득남은 inode가 없으면 0 반환
`xchk_iscan_want_live_update`Inode가 scan에서 이미 방문됐는지 판단Hook이 in-memory scan information을 update해야 하는지 결정하는 핵심 predicate
`xchk_iscan_mark_visited`Inode를 방문 완료로 표시Visited cursor 갱신
`xchk_iscan_teardown`Scan 종료Coordinator와 hook 관련 자원 정리
Start부터 teardown까지 각 API의 역할을 정리합니다.
이 기능 역시 `inode scanner <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_ series의 일부입니다.
.. _liveupdate:
Live Updates During a Scan
``````````````````````````
The code paths of the online fsck scanning code and the :ref:`hooked<fshooks>`
filesystem code look like this::
other program
↓
inode lock ←────────────────────┐
↓ │
AG header lock │
↓ │
filesystem function │
↓ │
notifier call chain │ same
↓ ├─── inode
scrub hook function │ lock
↓ │
scan data mutex ←──┐ same │
↓ ├─── scan │
update scan data │ lock │
↑ │ │
scan data mutex ←──┘ │
↑ │
inode lock ←────────────────────┘
↑
scrub function
↑
inode scanner
↑
xfs_scrub
These rules must be followed to ensure correct interactions between the
checking code and the code making an update to the filesystem:
- Prior to invoking the notifier call chain, the filesystem function being
hooked must acquire the same lock that the scrub scanning function acquires
to scan the inode.
- The scanning function and the scrub hook function must coordinate access to
the scan data by acquiring a lock on the scan data.
- Scrub hook function must not add the live update information to the scan
observations unless the inode being updated has already been scanned.
The scan coordinator has a helper predicate (``xchk_iscan_want_live_update``)
for this.
- Scrub hook functions must not change the caller's state, including the
transaction that it is running.
They must not acquire any resources that might conflict with the filesystem
function being hooked.
- The hook function can abort the inode scan to avoid breaking the other rules.
The inode scan APIs are pretty simple:
- ``xchk_iscan_start`` starts a scan
- ``xchk_iscan_iter`` grabs a reference to the next inode in the scan or
returns zero if there is nothing left to scan
- ``xchk_iscan_want_live_update`` to decide if an inode has already been
visited in the scan.
This is critical for hook functions to decide if they need to update the
in-memory scan information.
- ``xchk_iscan_mark_visited`` to mark an inode as having been visited in the
scan
- ``xchk_iscan_teardown`` to finish the scan
This functionality is also a part of the
`inode scanner
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_
series.
Case study: quota counter 검사
3602-3693Mount-time quotacheck와 online repair quotacheck를 비교하면 live update가 필요한 이유가 분명해집니다. Mount-time quotacheck는 concurrent operation과 경쟁하지 않으므로 다음 네 단계로 동작합니다.
단계동작결과
1모든 incore dquot를 load할 수 있을 만큼 ondisk dquot를 정리하고 ondisk buffer의 resource usage counter를 0으로 설정집계를 시작할 깨끗한 dquot state 마련
2Filesystem의 모든 inode를 순회하고 각 file의 resource usage를 incore dquot에 더함실제 usage 재계산
3각 incore dquot를 순회하고 flush 중이 아니면 backing ondisk buffer를 delayed-write (`delwri`) list에 추가Writeback 대상 수집
4Buffer list를 disk에 기록재계산한 quota counter 영속화
Incore dquot 집계와 delayed write를 이용하는 원문의 1–4단계를 보존합니다.
대부분의 online fsck function처럼 online quotacheck도 새로 수집한 metadata가 filesystem의 모든 state를 반영하기 전에는 일반 filesystem object에 쓸 수 없습니다. 따라서 file resource usage를 sparse `xfarray`로 구현한 shadow dquot index에 기록하고, scan이 끝난 뒤에만 real dquot에 씁니다.
Dquot contention을 줄이기 위해 quota resource usage update를 여러 phase로 처리하므로 transactional update의 live 반영은 더 복잡합니다.
단계동작Lock·transaction 효과
1관련 inode를 transaction에 join하고 lockFile update context 확정
2File에 붙은 각 dquot를 lock하고 resource usage에 quota reservation을 더해 transaction에 기록한 뒤 unlockReservation을 짧은 dquot lock 구간에서 반영
3Actual quota usage 변경을 transaction에서 추적Commit 전 usage delta 유지
4Transaction commit 때 각 dquot를 다시 lock하고 quota usage change를 log하며 unused reservation을 돌려준 뒤 unlock실제 usage와 reservation 정산
Inode join부터 commit 시 dquot 정산까지 원문의 1–4단계를 보존합니다.
Online quotacheck hook은 2단계와 4단계에 배치됩니다. 2단계 hook은 일반 code와 비슷하게 동작하는 transaction dquot context인 `dqtrx`의 shadow version을 만들고, 4단계 hook은 shadow `dqtrx` 변경을 shadow dquot에 commit합니다. 두 hook 모두 inode가 lock된 상태에서 호출되므로 live update가 inode scanner와 조정됩니다.
Hook 위치Shadow 동작조정 조건
Step 2Shadow transaction dquot context (`dqtrx`) 생성Inode lock 아래에서 reservation 변화 기록
Step 4Shadow `dqtrx` 변경을 shadow dquot에 commit같은 inode lock으로 inode scanner와 직렬화
Transaction의 reservation phase와 commit phase를 shadow state에 대응시킵니다.
Quotacheck scan은 coordinated inode scan을 설정한 뒤 다음 절차로 집계와 비교를 수행합니다.
단계동작Shadow state
1Coordinated inode scan 설정Live hook과 visited cursor 준비
2Iterator가 반환한 각 inode를 잡아 lock하고 data block·inode count·realtime block usage를 계산하여 해당 user·group·project id의 shadow dquot에 더한 뒤 unlock·releaseFile별 resource usage 집계
3System의 각 dquot를 잡아 lock하고 scan 및 live hook이 만든 shadow dquot와 비교실제 counter 오류 검출
Inode usage 수집과 system-wide dquot 비교를 순서대로 나타냅니다.
Live update 덕분에 어느 lock도 오래 유지하지 않고 모든 quota record를 순회할 수 있습니다. Repair가 필요하면 real dquot와 shadow dquot를 함께 lock하고 real resource count를 shadow dquot 값으로 설정합니다.
제안된 patchset은 `online quotacheck <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_ series입니다.
.. _quotacheck:
Case Study: Quota Counter Checking
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is useful to compare the mount time quotacheck code to the online repair
quotacheck code.
Mount time quotacheck does not have to contend with concurrent operations, so
it does the following:
1. Make sure the ondisk dquots are in good enough shape that all the incore
dquots will actually load, and zero the resource usage counters in the
ondisk buffer.
2. Walk every inode in the filesystem.
Add each file's resource usage to the incore dquot.
3. Walk each incore dquot.
If the incore dquot is not being flushed, add the ondisk buffer backing the
incore dquot to a delayed write (delwri) list.
4. Write the buffer list to disk.
Like most online fsck functions, online quotacheck can't write to regular
filesystem objects until the newly collected metadata reflect all filesystem
state.
Therefore, online quotacheck records file resource usage to a shadow dquot
index implemented with a sparse ``xfarray``, and only writes to the real dquots
once the scan is complete.
Handling transactional updates is tricky because quota resource usage updates
are handled in phases to minimize contention on dquots:
1. The inodes involved are joined and locked to a transaction.
2. For each dquot attached to the file:
a. The dquot is locked.
b. A quota reservation is added to the dquot's resource usage.
The reservation is recorded in the transaction.
c. The dquot is unlocked.
3. Changes in actual quota usage are tracked in the transaction.
4. At transaction commit time, each dquot is examined again:
a. The dquot is locked again.
b. Quota usage changes are logged and unused reservation is given back to
the dquot.
c. The dquot is unlocked.
For online quotacheck, hooks are placed in steps 2 and 4.
The step 2 hook creates a shadow version of the transaction dquot context
(``dqtrx``) that operates in a similar manner to the regular code.
The step 4 hook commits the shadow ``dqtrx`` changes to the shadow dquots.
Notice that both hooks are called with the inode locked, which is how the
live update coordinates with the inode scanner.
The quotacheck scan looks like this:
1. Set up a coordinated inode scan.
2. For each inode returned by the inode scan iterator:
a. Grab and lock the inode.
b. Determine that inode's resource usage (data blocks, inode counts,
realtime blocks) and add that to the shadow dquots for the user, group,
and project ids associated with the inode.
c. Unlock and release the inode.
3. For each dquot in the system:
a. Grab and lock the dquot.
b. Check the dquot against the shadow dquots created by the scan and updated
by the live hooks.
Live updates are key to being able to walk every quota record without
needing to hold any locks for a long duration.
If repairs are desired, the real and shadow dquots are locked and their
resource counts are set to the values in the shadow dquot.
The proposed patchset is the
`online quotacheck
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_
series.
Case study: file link count 검사
3694-3751File link count 검사도 live update hook을 사용합니다. Coordinated inode scanner로 filesystem의 모든 directory를 방문하고, file별 link count record는 inumber로 index한 sparse `xfarray`에 저장합니다. Scan phase에서 각 directory entry는 다음 observation data를 생성합니다.
단계Entry 조건증가하는 shadow counter
1Root directory의 dotdot (`'..'`) entryRoot의 dotdot이 self-referential이므로 directory의 parent link count 증가
2Subdirectory의 dotdot entryParent의 backref count 증가
3Dot도 dotdot도 아닌 entryTarget file의 parent count 증가
4Target이 subdirectoryParent의 child link count 증가
Dot·dotdot·subdirectory 여부에 따른 네 counter update를 보존합니다.
Link-count inode scanner와 live-update hook의 상호작용에서 핵심은 scan cursor가 어떤 *parent* directory를 scan했는지 추적한다는 점입니다. 따라서 `A → B` update에서 A가 아직 scan되지 않았다면 B가 이미 scan됐더라도 live update를 무시합니다. 또한 B를 가리키는 dotdot entry가 있는 subdirectory A는 child dotdot entry가 parent link count에 영향을 주므로 A의 shadow data에서 backref counter로 집계됩니다.
상황반영 여부이유
`A → B`, parent A 미방문·target B 방문무시A를 나중에 scan할 때 directory entry가 자연스럽게 집계됨
`A → B`, parent A 방문 완료반영이미 scan한 parent의 이후 change를 shadow observation에 적용해야 함
Subdirectory A의 dotdot이 B를 가리킴A의 shadow backref counter에 집계Child dotdot entry가 parent B의 link count에 영향
Target 방문 여부가 아니라 directory entry를 소유한 parent 방문 여부로 update 반영을 결정합니다.
Directory entry를 create, change, remove하는 operation은 `bumplink`와 `droplink`를 수반하므로 filesystem의 해당 모든 경로에 live-update hook을 신중히 배치합니다. 모든 file에서 올바른 link count는 parent 수와 child subdirectory 수의 합입니다. Non-directory에는 어떤 종류의 child도 없으며, backref information은 child subdirectory를 가리키는 link 수와 되돌아오는 dotdot entry 수의 불일치를 찾는 데 사용합니다.
대상올바른 link count추가 검사
DirectoryParent 수 + child subdirectory 수Child link와 dotdot backref 수의 일치
Non-directoryParent 수Child는 존재하지 않음
Backref shadow dataSubdirectory dotdot observation에서 계산Parent의 child-link count와 비교
Directory와 non-directory의 기대값 및 backref 교차검사를 구분합니다.
Scan 완료 뒤 inode와 shadow data를 모두 lock하고 link count를 비교합니다. 비교에는 두 번째 coordinated inode scan cursor를 사용하며, live update 덕분에 inode 사이에서 lock을 계속 유지하지 않고도 모든 inode를 순회할 수 있습니다. Repair가 필요하면 inode link count를 shadow information 값으로 설정합니다. Parent가 하나도 발견되지 않으면 file을 영원히 잃지 않도록 orphanage에 :ref:`reparent <orphanage>`해야 합니다.
제안된 patchset은 `file link count repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks>`_ series입니다.
.. _nlinks:
Case Study: File Link Count Checking
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File link count checking also uses live update hooks.
The coordinated inode scanner is used to visit all directories on the
filesystem, and per-file link count records are stored in a sparse ``xfarray``
indexed by inumber.
During the scanning phase, each entry in a directory generates observation
data as follows:
1. If the entry is a dotdot (``'..'``) entry of the root directory, the
directory's parent link count is bumped because the root directory's dotdot
entry is self referential.
2. If the entry is a dotdot entry of a subdirectory, the parent's backref
count is bumped.
3. If the entry is neither a dot nor a dotdot entry, the target file's parent
count is bumped.
4. If the target is a subdirectory, the parent's child link count is bumped.
A crucial point to understand about how the link count inode scanner interacts
with the live update hooks is that the scan cursor tracks which *parent*
directories have been scanned.
In other words, the live updates ignore any update about ``A → B`` when A has
not been scanned, even if B has been scanned.
Furthermore, a subdirectory A with a dotdot entry pointing back to B is
accounted as a backref counter in the shadow data for A, since child dotdot
entries affect the parent's link count.
Live update hooks are carefully placed in all parts of the filesystem that
create, change, or remove directory entries, since those operations involve
bumplink and droplink.
For any file, the correct link count is the number of parents plus the number
of child subdirectories.
Non-directories never have children of any kind.
The backref information is used to detect inconsistencies in the number of
links pointing to child subdirectories and the number of dotdot entries
pointing back.
After the scan completes, the link count of each file can be checked by locking
both the inode and the shadow data, and comparing the link counts.
A second coordinated inode scan cursor is used for comparisons.
Live updates are key to being able to walk every inode without needing to hold
any locks between inodes.
If repairs are desired, the inode's link count is set to the value in the
shadow information.
If no parents are found, the file must be :ref:`reparented <orphanage>` to the
orphanage to prevent the file from being lost forever.
The proposed patchset is the
`file link count repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks>`_
series.
Case study: reverse mapping record rebuild
3752-3835대부분의 repair function은 filesystem resource를 lock하고, 살아남은 ondisk metadata를 순회해 replacement metadata record를 찾고, 수집한 observation을 :ref:`in-memory array <xfarray>`에 저장하는 같은 pattern을 따릅니다. 이 방식의 주된 장점은 단순성과 modularity입니다. Code와 data가 scrub module 안에 완전히 들어가 main filesystem hook이 필요 없고, 보통 memory 사용도 가장 효율적입니다.
두 번째 장점은 atomicity입니다. Kernel이 structure가 손상됐다고 판단하면 repair와 metadata revalidation을 마칠 때까지 다른 thread가 metadata에 접근할 수 없습니다. Filesystem shard 내부 repair에서는 shard 일부를 고치는 동안 lock하는 지연보다 이 장점들이 더 큽니다.
장점구현 특성효과
Simplicity·modularityCode·data가 scrub module 내부에 있고 main filesystem hook 불필요Repair 코드 격리와 효율적인 memory 사용
AtomicityCorruption 판정부터 repair·revalidation 완료까지 다른 thread의 metadata 접근 차단중간 structure가 외부에 노출되지 않음
Scrub-local 구성과 lock 기반 atomicity를 구분합니다.
Reverse mapping btree repair는 모든 file의 모든 fork에 있는 모든 space mapping을 scan해야 하고 filesystem을 멈출 수 없으므로 이 표준 전략을 사용할 수 없습니다. 따라서 scrub과 repair 사이의 atomicity를 포기하고 :ref:`coordinated inode scanner <iscan>`, :ref:`live update hook <liveupdate>`, :ref:`in-memory rmap btree <xfbtree>`를 결합해 reverse mapping record scan을 완성합니다.
단계동작결과·조건
1Rmap record staging용 xfbtree 설정In-memory shadow index 준비
2Scrub 중 얻은 AGI·AGF buffer lock을 유지한 채 inode, btree, CoW staging extent, internal log를 포함한 모든 AG metadata의 reverse mapping 생성고정 AG metadata observation 수집
3Inode scanner 설정File fork walk 준비
4Repair 중인 AG의 rmap update에 hook 설치File scan 중 filesystem 나머지 부분의 rmapbt update를 live scan data에 전달
5Scan한 각 file의 두 fork에서 찾은 space mapping이 대상 AG와 일치하면 in-memory btree cursor를 만들고 rmap code로 record를 추가한 뒤 :ref:`special commit function <xfbtree_commit>`으로 xfbtree change를 xfile에 기록Scanner observation을 shadow btree에 commit
6Hook으로 받은 live update마다 owner가 이미 scan됐는지 판단하고, 그렇다면 in-memory btree cursor를 만들어 operation을 replay한 뒤 `xfbtree_commit`으로 xfile에 기록Caller state를 바꾸지 않도록 empty transaction으로 live update commit
7Inode scan 완료 뒤 새 scrub transaction을 만들고 두 AG header를 다시 lockInstall phase 진입
8Shadow btree의 rmap record 수로 새 btree geometry 계산필요 block 수 산정
9앞 단계에서 계산한 block 수 allocate새 ondisk rmapbt 공간 확보
10일반 btree bulk loading과 commit 수행새 rmap btree 설치
11:ref:`rmap btree repair 후 reap <rmap_reap>` case study 방식으로 old rmap btree block 회수이전 tree 제거
12더는 필요 없는 xfbtree freeStaging resource 정리
Live scan staging부터 old-tree reap과 xfbtree 해제까지 원문의 1–12단계를 보존합니다.
제안된 patchset은 `rmap repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree>`_ series입니다.
.. _rmap_repair:
Case Study: Rebuilding Reverse Mapping Records
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Most repair functions follow the same pattern: lock filesystem resources,
walk the surviving ondisk metadata looking for replacement metadata records,
and use an :ref:`in-memory array <xfarray>` to store the gathered observations.
The primary advantage of this approach is the simplicity and modularity of the
repair code -- code and data are entirely contained within the scrub module,
do not require hooks in the main filesystem, and are usually the most efficient
in memory use.
A secondary advantage of this repair approach is atomicity -- once the kernel
decides a structure is corrupt, no other threads can access the metadata until
the kernel finishes repairing and revalidating the metadata.
For repairs going on within a shard of the filesystem, these advantages
outweigh the delays inherent in locking the shard while repairing parts of the
shard.
Unfortunately, repairs to the reverse mapping btree cannot use the "standard"
btree repair strategy because it must scan every space mapping of every fork of
every file in the filesystem, and the filesystem cannot stop.
Therefore, rmap repair foregoes atomicity between scrub and repair.
It combines a :ref:`coordinated inode scanner <iscan>`, :ref:`live update hooks
<liveupdate>`, and an :ref:`in-memory rmap btree <xfbtree>` to complete the
scan for reverse mapping records.
1. Set up an xfbtree to stage rmap records.
2. While holding the locks on the AGI and AGF buffers acquired during the
scrub, generate reverse mappings for all AG metadata: inodes, btrees, CoW
staging extents, and the internal log.
3. Set up an inode scanner.
4. Hook into rmap updates for the AG being repaired so that the live scan data
can receive updates to the rmap btree from the rest of the filesystem during
the file scan.
5. For each space mapping found in either fork of each file scanned,
decide if the mapping matches the AG of interest.
If so:
a. Create a btree cursor for the in-memory btree.
b. Use the rmap code to add the record to the in-memory btree.
c. Use the :ref:`special commit function <xfbtree_commit>` to write the
xfbtree changes to the xfile.
6. For each live update received via the hook, decide if the owner has already
been scanned.
If so, apply the live update into the scan data:
a. Create a btree cursor for the in-memory btree.
b. Replay the operation into the in-memory btree.
c. Use the :ref:`special commit function <xfbtree_commit>` to write the
xfbtree changes to the xfile.
This is performed with an empty transaction to avoid changing the
caller's state.
7. When the inode scan finishes, create a new scrub transaction and relock the
two AG headers.
8. Compute the new btree geometry using the number of rmap records in the
shadow btree, like all other btree rebuilding functions.
9. Allocate the number of blocks computed in the previous step.
10. Perform the usual btree bulk loading and commit to install the new rmap
btree.
11. Reap the old rmap btree blocks as discussed in the case study about how
to :ref:`reap after rmap btree repair <rmap_reap>`.
12. Free the xfbtree now that it not needed.
The proposed patchset is the
`rmap repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree>`_
series.
Disk temporary file을 이용한 repair staging
3836-3939XFS는 directory, extended attribute, symbolic-link target, realtime volume의 free-space bitmap·summary information, quota record 등 많은 metadata를 file fork에 저장합니다. File fork는 memory management unit이 64-bit virtual address를 physical memory address에 mapping하는 것처럼 64-bit logical file-fork space extent를 physical storage extent에 mapping합니다.
따라서 directory·extended attribute 같은 file-based tree structure는 file-fork offset address space에 mapping된 block이 같은 address space의 다른 block을 가리키는 방식으로 구성됩니다. Bitmap·quota record 같은 linear structure는 file-fork offset address space에서 array element offset을 계산합니다.
File fork는 filesystem 전체만큼 큰 space를 소비할 수 있으므로 paging scheme이 있어도 repair를 memory에 staging할 수 없습니다. Online repair는 XFS filesystem 내부에 temporary file을 만들고 올바른 offset에 새 structure를 쓴 다음, file-fork mapping 전체와 그 contents를 atomic하게 exchange하여 repair를 commit합니다.
Repair가 끝나면 old fork를 필요에 따라 reap합니다. Reap 중 system이 내려가면 log recovery 때 `iunlink` code가 block을 삭제합니다. 단, temporary file을 안전하게 사용하려면 filesystem의 모든 space-usage information과 inode index가 반드시 일관되어야 합니다. 이 dependency 때문에 online repair는 ondisk space-usage information을 staging할 때만 pageable kernel memory를 사용할 수 있습니다.
Metadata file mapping을 temporary file과 exchange하려면 block header의 owner field가 temporary file이 아니라 repair 대상 file과 일치해야 합니다. 이를 위해 directory, extended-attribute, symbolic-link function은 caller가 owner number를 명시할 수 있도록 수정됐습니다.
Reaping에는 단점도 있습니다. Reap phase 중 crash가 발생하고 fork extent가 crosslinked 상태라면, `iunlink` processing이 space를 free할 때 extra reverse mapping을 발견하고 abort하므로 처리에 실패합니다.
Repair용 temporary file은 userspace의 `O_TMPFILE` file과 비슷합니다. Directory에 link되지 않고 마지막 reference를 잃으면 file 전체가 reap됩니다. 차이점은 kernel 밖의 access permission이 전혀 없어야 하고, handle로 열리지 않도록 특별히 표시해야 하며, directory tree에는 절대로 link하면 안 된다는 것입니다.
항목접근·문제결과
첫 번째 iteration손상 metadata block에서 salvage 가능한 data를 scan하고 file-fork extent를 reap한 뒤 같은 자리에 새 structure 구축앞에서 요구한 atomic repair requirement를 충족하지 못해 폐기
두 번째 iterationSalvage data로 fork의 높은 offset에 두 번째 structure를 만들고 old extent를 reap한 뒤 `COLLAPSE_RANGE`로 새 extent를 제자리로 이동다음 일곱 단점이 발생
1. Linear array offsetRegular filesystem code에 alternate copy 구축용 linear offset을 record-offset 계산에 적용하는 개념이 없음Array structure를 다른 fork 구간에 투명하게 만들기 어려움
2. Attr-fork capacityExtended attribute가 attr-fork offset address space 전체를 사용할 수 있음Alternate copy용 높은 offset 공간을 보장할 수 없음
3. Atomic commitOld structure를 완전히 대체하려면 log-assisted `COLLAPSE_RANGE`가 필요일반 range collapse로 atomic repair requirement 충족 불가
4. Pre-collapse crashSecondary tree 구축 뒤 range collapse 전에 crashFile fork에 unreachable block이 남아 손상을 더 혼란스럽게 만듦
5. Recovery-time reapRepair 후 block reaping 자체가 단순하지 않음Log recovery에서 재시작한 range-collapse operation으로 reap을 시작하기 매우 어려움
6. Header fork offsetDirectory-entry block과 quota record가 각 block header에 file-fork offset 기록Atomic range collapse가 모든 block header의 해당 field를 다시 써야 함
7. Graph pointer rewriteDirectory·extended-attribute btree index block마다 sibling·child block pointer 존재Graph structure 보존을 위해 많은 block을 반복 rewrite하므로 빠른 repair에 부적합
결론두 iteration의 한계를 종합Temporary-file staging 도입
원문 ASCII sidebar의 두 iteration과 두 번째 방식의 일곱 문제를 구조화했습니다.
Staging Repairs with Temporary Files on Disk
--------------------------------------------
XFS stores a substantial amount of metadata in file forks: directories,
extended attributes, symbolic link targets, free space bitmaps and summary
information for the realtime volume, and quota records.
File forks map 64-bit logical file fork space extents to physical storage space
extents, similar to how a memory management unit maps 64-bit virtual addresses
to physical memory addresses.
Therefore, file-based tree structures (such as directories and extended
attributes) use blocks mapped in the file fork offset address space that point
to other blocks mapped within that same address space, and file-based linear
structures (such as bitmaps and quota records) compute array element offsets in
the file fork offset address space.
Because file forks can consume as much space as the entire filesystem, repairs
cannot be staged in memory, even when a paging scheme is available.
Therefore, online repair of file-based metadata createas a temporary file in
the XFS filesystem, writes a new structure at the correct offsets into the
temporary file, and atomically exchanges all file fork mappings (and hence the
fork contents) to commit the repair.
Once the repair is complete, the old fork can be reaped as necessary; if the
system goes down during the reap, the iunlink code will delete the blocks
during log recovery.
**Note**: All space usage and inode indices in the filesystem *must* be
consistent to use a temporary file safely!
This dependency is the reason why online repair can only use pageable kernel
memory to stage ondisk space usage information.
Exchanging metadata file mappings with a temporary file requires the owner
field of the block headers to match the file being repaired and not the
temporary file.
The directory, extended attribute, and symbolic link functions were all
modified to allow callers to specify owner numbers explicitly.
There is a downside to the reaping process -- if the system crashes during the
reap phase and the fork extents are crosslinked, the iunlink processing will
fail because freeing space will find the extra reverse mappings and abort.
Temporary files created for repair are similar to ``O_TMPFILE`` files created
by userspace.
They are not linked into a directory and the entire file will be reaped when
the last reference to the file is lost.
The key differences are that these files must have no access permission outside
the kernel at all, they must be specially marked to prevent them from being
opened by handle, and they must never be linked into the directory tree.
+--------------------------------------------------------------------------+
| **Historical Sidebar**: |
+--------------------------------------------------------------------------+
| In the initial iteration of file metadata repair, the damaged metadata |
| blocks would be scanned for salvageable data; the extents in the file |
| fork would be reaped; and then a new structure would be built in its |
| place. |
| This strategy did not survive the introduction of the atomic repair |
| requirement expressed earlier in this document. |
| |
| The second iteration explored building a second structure at a high |
| offset in the fork from the salvage data, reaping the old extents, and |
| using a ``COLLAPSE_RANGE`` operation to slide the new extents into |
| place. |
| |
| This had many drawbacks: |
| |
| - Array structures are linearly addressed, and the regular filesystem |
| codebase does not have the concept of a linear offset that could be |
| applied to the record offset computation to build an alternate copy. |
| |
| - Extended attributes are allowed to use the entire attr fork offset |
| address space. |
| |
| - Even if repair could build an alternate copy of a data structure in a |
| different part of the fork address space, the atomic repair commit |
| requirement means that online repair would have to be able to perform |
| a log assisted ``COLLAPSE_RANGE`` operation to ensure that the old |
| structure was completely replaced. |
| |
| - A crash after construction of the secondary tree but before the range |
| collapse would leave unreachable blocks in the file fork. |
| This would likely confuse things further. |
| |
| - Reaping blocks after a repair is not a simple operation, and |
| initiating a reap operation from a restarted range collapse operation |
| during log recovery is daunting. |
| |
| - Directory entry blocks and quota records record the file fork offset |
| in the header area of each block. |
| An atomic range collapse operation would have to rewrite this part of |
| each block header. |
| Rewriting a single field in block headers is not a huge problem, but |
| it's something to be aware of. |
| |
| - Each block in a directory or extended attributes btree index contains |
| sibling and child block pointers. |
| Were the atomic commit to use a range collapse operation, each block |
| would have to be rewritten very carefully to preserve the graph |
| structure. |
| Doing this as part of a range collapse means rewriting a large number |
| of blocks repeatedly, which is not conducive to quick repairs. |
| |
| This lead to the introduction of temporary file staging. |
+--------------------------------------------------------------------------+
Temporary file 사용
3940-3978Online repair code는 `xrep_tempfile_create` function으로 filesystem 내부에 temporary file을 만들어야 합니다. 이 function은 inode를 allocate하고 incore inode를 private으로 표시한 뒤 scrub context에 attach합니다. 이 file은 userspace에서 숨겨지고 directory tree에 추가할 수 없으며 계속 private으로 유지해야 합니다.
Temporary file은 `IOLOCK`과 `ILOCK` 두 inode lock만 사용합니다. Data-fork block에는 userspace page fault가 절대 없어야 하므로 `MMAPLOCK`은 필요하지 않습니다. 일반 XFS file과 마찬가지로 file data access는 `IOLOCK`, file metadata access는 `ILOCK`으로 제어합니다. Scrub context가 temporary file과 lock state를 정리할 수 있도록 locking helper가 제공됩니다.
Lock사용 여부·역할이유
`IOLOCK`사용; file data access 제어Temporary file의 fork data 보호
`ILOCK`사용; file metadata access 제어Inode와 mapping metadata 보호
`MMAPLOCK`사용하지 않음Data-fork block에 userspace page fault가 없어야 함
필요한 두 lock과 사용하지 않는 lock을 구분합니다.
:ref:`inode locking <ilocking>` 절의 nested locking strategy를 따르기 위해 scrub function은 `xrep_tempfile_ilock*_nowait` lock helper를 사용하는 것이 권장됩니다.
Temporary file에는 다음 두 방법으로 data를 쓸 수 있습니다.
방법대상동작
`xrep_tempfile_copyin`Regular temporary fileXfile의 content로 temporary file 설정
Regular filesystem functionDirectory, symbolic link, extended attribute기존 metadata function으로 temporary file에 기록
Xfile copy와 일반 metadata function 사용을 구분합니다.
Temporary file에 올바른 data-file copy를 만든 뒤에는 repair 대상 file로 전달해야 하며, 이는 다음 절에서 설명합니다. 제안된 patch는 `repair temporary files <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-tempfiles>`_ series입니다.
Using a Temporary File
``````````````````````
Online repair code should use the ``xrep_tempfile_create`` function to create a
temporary file inside the filesystem.
This allocates an inode, marks the in-core inode private, and attaches it to
the scrub context.
These files are hidden from userspace, may not be added to the directory tree,
and must be kept private.
Temporary files only use two inode locks: the IOLOCK and the ILOCK.
The MMAPLOCK is not needed here, because there must not be page faults from
userspace for data fork blocks.
The usage patterns of these two locks are the same as for any other XFS file --
access to file data are controlled via the IOLOCK, and access to file metadata
are controlled via the ILOCK.
Locking helpers are provided so that the temporary file and its lock state can
be cleaned up by the scrub context.
To comply with the nested locking strategy laid out in the :ref:`inode
locking<ilocking>` section, it is recommended that scrub functions use the
xrep_tempfile_ilock*_nowait lock helpers.
Data can be written to a temporary file by two means:
1. ``xrep_tempfile_copyin`` can be used to set the contents of a regular
temporary file from an xfile.
2. The regular directory, symbolic link, and extended attribute functions can
be used to write to the temporary file.
Once a good copy of a data file has been constructed in a temporary file, it
must be conveyed to the file being repaired, which is the topic of the next
section.
The proposed patches are in the
`repair temporary files
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-tempfiles>`_
series.
Logged file-content exchange
3979-4032Repair가 새 data structure를 쓴 temporary file을 만든 뒤에는 변경을 기존 file에 commit해야 합니다. 두 file의 inumber를 교환할 수 없으므로 새 metadata가 old metadata를 대체해야 합니다. Extent 교환이 필요해 보이지만 file defragmentation tool `xfs_fsr`이 쓰는 기존 extent-swap code는 online repair에 충분하지 않습니다.
항목기존 전제·제약Online repair 문제
aRmapbt 사용 시 mapping을 한 번 교환할 때마다 reverse mapping information을 갱신하므로 transaction당 mapping 하나만 교환하고 각 transaction은 독립적여러 transaction에 걸친 전체 fork 교환의 진행 상태를 묶어 추적하지 못함
bOld defragmentation code는 extent fork 전체를 한 operation으로 swapOnline fsck에 핵심인 reverse mapping과 양립하지 않음
cDefragmentation 대상 두 file의 content가 동일하다고 가정중단된 불완전 exchange도 user-visible content 변화가 없다는 전제
dOnline repair의 두 file content는 정의상 동일하지 않음Directory·xattr의 user-visible content가 같아도 개별 block content는 크게 다를 수 있음
eOld file block이 다른 structure와 crosslinked일 수 있음Mid-repair crash 뒤 old block이 다시 나타나면 안 됨
Reverse mapping, content identity, crosslink recovery에 관한 원문의 a–e를 보존합니다.
이를 해결하기 위해 두 file range의 exchange 진행을 추적하는 새 deferred operation과 새 log-intent item type을 만듭니다. 새 exchange operation은 reverse-mapping extent-swap code가 쓰는 같은 transaction들을 chain으로 연결하되 intermediate progress를 log에 기록하여 crash 뒤 operation을 재시작할 수 있게 합니다.
새 기능은 file-contents exchange인 `xfs_exchrange` code이고, 내부 구현은 file-fork mapping을 교환하는 `xfs_exchmaps`입니다. 새 log item은 exchange 진행을 기록하여 일단 시작한 exchange가 interruption이 있어도 항상 완료되도록 보장합니다. Superblock의 새 incompatible feature flag `XFS_SB_FEAT_INCOMPAT_EXCHRANGE`는 old kernel이 이 새 log-item record를 replay하지 못하게 막습니다.
제안된 patchset은 `file contents exchange <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates>`_ series입니다.
Logged File Content Exchanges
-----------------------------
Once repair builds a temporary file with a new data structure written into
it, it must commit the new changes into the existing file.
It is not possible to swap the inumbers of two files, so instead the new
metadata must replace the old.
This suggests the need for the ability to swap extents, but the existing extent
swapping code used by the file defragmenting tool ``xfs_fsr`` is not sufficient
for online repair because:
a. When the reverse-mapping btree is enabled, the swap code must keep the
reverse mapping information up to date with every exchange of mappings.
Therefore, it can only exchange one mapping per transaction, and each
transaction is independent.
b. Reverse-mapping is critical for the operation of online fsck, so the old
defragmentation code (which swapped entire extent forks in a single
operation) is not useful here.
c. Defragmentation is assumed to occur between two files with identical
contents.
For this use case, an incomplete exchange will not result in a user-visible
change in file contents, even if the operation is interrupted.
d. Online repair needs to swap the contents of two files that are by definition
*not* identical.
For directory and xattr repairs, the user-visible contents might be the
same, but the contents of individual blocks may be very different.
e. Old blocks in the file may be cross-linked with another structure and must
not reappear if the system goes down mid-repair.
These problems are overcome by creating a new deferred operation and a new type
of log intent item to track the progress of an operation to exchange two file
ranges.
The new exchange operation type chains together the same transactions used by
the reverse-mapping extent swap code, but records intermedia progress in the
log so that operations can be restarted after a crash.
This new functionality is called the file contents exchange (xfs_exchrange)
code.
The underlying implementation exchanges file fork mappings (xfs_exchmaps).
The new log item records the progress of the exchange to ensure that once an
exchange begins, it will always run to completion, even there are
interruptions.
The new ``XFS_SB_FEAT_INCOMPAT_EXCHRANGE`` incompatible feature flag
in the superblock protects these new log item records from being replayed on
old kernels.
The proposed patchset is the
`file contents exchange
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates>`_
series.
Sidebar: log-incompatible feature flag 사용
4033-4074XFS v5부터 superblock의 `sb_features_log_incompat` field는 이 filesystem을 mount할 수 있는 모든 kernel이 읽을 수 있지는 않은 record가 log에 있음을 나타냅니다. 즉 log-incompat feature는 content를 이해하지 못하는 kernel로부터 log content를 보호합니다.
다른 superblock feature bit와 달리 log-incompat bit는 ephemeral합니다. 비어 있는 clean log는 보호할 필요가 없고, log content가 filesystem에 commit되면 unmount 과정이나 system idle 상태에서 log가 스스로 clean되기 때문입니다. Upper-level code가 transaction을 처리하는 동시에 log가 clean될 수 있으므로, log-incompatible feature를 사용할 예정임을 upper-level code가 log에 알려야 합니다.
Log는 feature마다 하나의 `struct rw_semaphore`를 사용해 incompatible feature access를 조정합니다. Log-cleaning code는 bit를 clear하려고 rwsem을 exclusive mode로 얻으며, lock 획득에 실패하면 feature bit를 set 상태로 둡니다.
단계동작순서·결과
1. WrapperFeature 지원 code가 log feature 획득용 wrapper function 제공직접 semaphore 조작을 캡슐화
2. Set bit`xfs_add_incompat_log_feature`로 primary superblock의 feature bit 설정Superblock update는 transactional
3. File-operation orderingIOLOCK과 MMAPLOCK 획득 뒤, 기능을 쓰는 transaction allocate 직전에 log assistance 획득Transaction이 log cleaning과 race하지 않게 함
4. Drop referenceTransaction 완료 뒤 `xlog_drop_incompat_feat` 호출Feature 사용 reference release
5. Clear bitLog가 clean해질 때까지 superblock bit 유지Recovery에 새 record가 남아 있는 동안 old kernel 차단
Feature 획득부터 clean-log clear까지 lock·transaction 순서를 보존합니다.
Log-assisted extended-attribute update와 file-content exchange는 모두 log-incompat feature를 사용하며 이 기능을 감싸는 convenience wrapper를 제공합니다.
+--------------------------------------------------------------------------+
| **Sidebar: Using Log-Incompatible Feature Flags** |
+--------------------------------------------------------------------------+
| Starting with XFS v5, the superblock contains a |
| ``sb_features_log_incompat`` field to indicate that the log contains |
| records that might not readable by all kernels that could mount this |
| filesystem. |
| In short, log incompat features protect the log contents against kernels |
| that will not understand the contents. |
| Unlike the other superblock feature bits, log incompat bits are |
| ephemeral because an empty (clean) log does not need protection. |
| The log cleans itself after its contents have been committed into the |
| filesystem, either as part of an unmount or because the system is |
| otherwise idle. |
| Because upper level code can be working on a transaction at the same |
| time that the log cleans itself, it is necessary for upper level code to |
| communicate to the log when it is going to use a log incompatible |
| feature. |
| |
| The log coordinates access to incompatible features through the use of |
| one ``struct rw_semaphore`` for each feature. |
| The log cleaning code tries to take this rwsem in exclusive mode to |
| clear the bit; if the lock attempt fails, the feature bit remains set. |
| The code supporting a log incompat feature should create wrapper |
| functions to obtain the log feature and call |
| ``xfs_add_incompat_log_feature`` to set the feature bits in the primary |
| superblock. |
| The superblock update is performed transactionally, so the wrapper to |
| obtain log assistance must be called just prior to the creation of the |
| transaction that uses the functionality. |
| For a file operation, this step must happen after taking the IOLOCK |
| and the MMAPLOCK, but before allocating the transaction. |
| When the transaction is complete, the ``xlog_drop_incompat_feat`` |
| function is called to release the feature. |
| The feature bit will not be cleared from the superblock until the log |
| becomes clean. |
| |
| Log-assisted extended attribute updates and file content exchanges bothe |
| use log incompat features and provide convenience wrappers around the |
| functionality. |
+--------------------------------------------------------------------------+
Logged file-content exchange 동작 원리
4075-4187두 file fork 사이에서 content를 exchange하는 일은 복잡합니다. 목표는 두 file-fork offset range의 mapping 전체를 교환하는 것입니다. 각 fork에는 extent mapping이 많을 수 있고 mapping edge가 서로 정렬되어 있지 않을 수 있습니다. Exchange 뒤에는 file size·inode flag 교환이나 fork data의 local format 변환 같은 추가 update도 필요할 수 있습니다.
새 deferred exchange-mapping work item은 `struct xfs_exchmaps_intent`로 표현되며, 원문의 C structure는 아래 영어 원문 block에 그대로 보존됩니다.
Field groupField역할
Participating inodes`xmi_ip1`, `xmi_ip2`Exchange에 참여하는 두 XFS inode
Range cursor`xmi_startoff1`, `xmi_startoff2`, `xmi_blockcount`두 logical fork-offset range와 남은 block 수
Final sizes`xmi_isize1`, `xmi_isize2`음수가 아니면 operation 뒤 설정할 file size
Behavior`xmi_flags``XFS_EXCHMAPS_*` log-operation flag; attr fork 선택과 exchange 후 작업 지정
두 inode, 두 logical range, 완료 후 size와 behavior flag를 field group별로 정리합니다.
새 log-intent item은 `(inode1, startoff1, blockcount)`와 `(inode2, startoff2, blockcount)` 두 logical fork-offset range를 추적할 정보를 담습니다. 각 step은 한 file에서 다른 file로 가능한 가장 큰 range mapping을 교환합니다. Step이 끝날 때마다 두 startoff를 늘리고 blockcount를 줄여 진행 상태를 반영합니다. Data fork가 대상이면 두 isize field로 operation 끝에서 file size를 교환합니다.
Exchange 시작 뒤 operation sequence는 다음과 같습니다.
단계동작진행·log 효과
1File-mapping exchange용 deferred work item 생성; 처음에는 교환할 file block range 전체 포함전체 작업 범위 초기화
2`xfs_defer_finish`로 exchange 처리; scrub에서는 `xrep_tempexch_contents`가 캡슐화Deferred mapping-exchange work item용 extent-swap intent item을 transaction에 log
3.a`xmi_startoff1`·`xmi_startoff2`부터 두 range의 block map을 읽고 한 step에서 교환할 최장 extent 계산; 두 mapping의 `br_blockcount` 중 작은 값 사용; 적어도 하나가 written block을 포함할 때까지 fork 진행Mutual hole, unwritten extent, 같은 physical space mapping은 교환하지 않음; file 1 mapping은 `map1`, file 2 mapping은 `map2`
3.bFile 1에서 `map1`을 unmap하는 deferred block-mapping update 생성첫 mapping 분리 예약
3.cFile 2에서 `map2`를 unmap하는 deferred block-mapping update 생성둘째 mapping 분리 예약
3.dFile 2에 `map1`을 map하는 deferred update 생성첫 mapping을 반대 file에 연결
3.eFile 1에 `map2`를 map하는 deferred update 생성둘째 mapping을 반대 file에 연결
3.f두 file의 block·quota·extent-count update를 logAccounting 변경 영속화
3.g필요하면 어느 file이든 ondisk size 확장교환 mapping을 포함하도록 size 보정
3.h3단계 시작 때 읽은 mapping-exchange intent용 done log item 기록현재 intent 처리 완료 표시
3.i방금 처리한 file range 양을 `(map1.br_startoff + map1.br_blockcount - xmi_startoff1)`로 계산3a에서 hole을 건너뛴 양까지 cursor 전진량에 포함
3.j계산한 block 수만큼 `xmi_startoff1`·`xmi_startoff2`를 늘리고 `xmi_blockcount`를 같은 양만큼 감소Work-item cursor 전진
3.k전진한 work-item state를 반영한 새 mapping-exchange intent log item 기록Crash recovery 재시작 지점 갱신
3.lDeferred-operation manager에 `EAGAIN` 반환Manager가 3b–3e의 deferred work를 완료한 뒤 3단계 처음으로 반복
4모든 `xmi_blockcount`를 처리한 뒤 post-processing 수행후속 절에서 설명하는 size·format 등 마무리
원문의 1–4단계와 반복되는 3a–3l을 모두 보존합니다.
Operation 중 filesystem이 내려가면 log recovery가 가장 최근의 미완료 mapping-exchange intent item을 찾아 그 지점부터 재시작합니다. 이 방식으로 atomic file-mapping exchange는 외부 observer가 old broken structure 또는 new structure 중 하나만 보고 둘이 섞인 상태는 절대 보지 않도록 보장합니다.
Mechanics of a Logged File Content Exchange
```````````````````````````````````````````
Exchanging contents between file forks is a complex task.
The goal is to exchange all file fork mappings between two file fork offset
ranges.
There are likely to be many extent mappings in each fork, and the edges of
the mappings aren't necessarily aligned.
Furthermore, there may be other updates that need to happen after the exchange,
such as exchanging file sizes, inode flags, or conversion of fork data to local
format.
This is roughly the format of the new deferred exchange-mapping work item:
.. code-block:: c
struct xfs_exchmaps_intent {
/* Inodes participating in the operation. */
struct xfs_inode *xmi_ip1;
struct xfs_inode *xmi_ip2;
/* File offset range information. */
xfs_fileoff_t xmi_startoff1;
xfs_fileoff_t xmi_startoff2;
xfs_filblks_t xmi_blockcount;
/* Set these file sizes after the operation, unless negative. */
xfs_fsize_t xmi_isize1;
xfs_fsize_t xmi_isize2;
/* XFS_EXCHMAPS_* log operation flags */
uint64_t xmi_flags;
};
The new log intent item contains enough information to track two logical fork
offset ranges: ``(inode1, startoff1, blockcount)`` and ``(inode2, startoff2,
blockcount)``.
Each step of an exchange operation exchanges the largest file range mapping
possible from one file to the other.
After each step in the exchange operation, the two startoff fields are
incremented and the blockcount field is decremented to reflect the progress
made.
The flags field captures behavioral parameters such as exchanging attr fork
mappings instead of the data fork and other work to be done after the exchange.
The two isize fields are used to exchange the file sizes at the end of the
operation if the file data fork is the target of the operation.
When the exchange is initiated, the sequence of operations is as follows:
1. Create a deferred work item for the file mapping exchange.
At the start, it should contain the entirety of the file block ranges to be
exchanged.
2. Call ``xfs_defer_finish`` to process the exchange.
This is encapsulated in ``xrep_tempexch_contents`` for scrub operations.
This will log an extent swap intent item to the transaction for the deferred
mapping exchange work item.
3. Until ``xmi_blockcount`` of the deferred mapping exchange work item is zero,
a. Read the block maps of both file ranges starting at ``xmi_startoff1`` and
``xmi_startoff2``, respectively, and compute the longest extent that can
be exchanged in a single step.
This is the minimum of the two ``br_blockcount`` s in the mappings.
Keep advancing through the file forks until at least one of the mappings
contains written blocks.
Mutual holes, unwritten extents, and extent mappings to the same physical
space are not exchanged.
For the next few steps, this document will refer to the mapping that came
from file 1 as "map1", and the mapping that came from file 2 as "map2".
b. Create a deferred block mapping update to unmap map1 from file 1.
c. Create a deferred block mapping update to unmap map2 from file 2.
d. Create a deferred block mapping update to map map1 into file 2.
e. Create a deferred block mapping update to map map2 into file 1.
f. Log the block, quota, and extent count updates for both files.
g. Extend the ondisk size of either file if necessary.
h. Log a mapping exchange done log item for th mapping exchange intent log
item that was read at the start of step 3.
i. Compute the amount of file range that has just been covered.
This quantity is ``(map1.br_startoff + map1.br_blockcount -
xmi_startoff1)``, because step 3a could have skipped holes.
j. Increase the starting offsets of ``xmi_startoff1`` and ``xmi_startoff2``
by the number of blocks computed in the previous step, and decrease
``xmi_blockcount`` by the same quantity.
This advances the cursor.
k. Log a new mapping exchange intent log item reflecting the advanced state
of the work item.
l. Return the proper error code (EAGAIN) to the deferred operation manager
to inform it that there is more work to be done.
The operation manager completes the deferred work in steps 3b-3e before
moving back to the start of step 3.
4. Perform any post-processing.
This will be discussed in more detail in subsequent sections.
If the filesystem goes down in the middle of an operation, log recovery will
find the most recent unfinished mapping exchange log intent item and restart
from there.
This is how atomic file mapping exchanges guarantees that an outside observer
will either see the old broken structure or the new one, and never a mismash of
both.
File-content exchange 준비
4188-4218Atomic file-mapping exchange를 시작하기 전에 몇 가지 준비가 필요합니다. Regular file은 operation 시작 전에 page cache를 disk로 flush하고 direct-I/O write를 quiesce해야 합니다.
다른 filesystem operation과 마찬가지로 file-mapping exchange는 두 file을 대신해 소비할 수 있는 disk space와 quota의 최대량을 계산하고 그만큼 resource를 reserve해야 합니다. Metadata를 dirty하기 시작한 뒤 복구할 수 없는 out-of-space failure가 발생하는 일을 막기 위해서입니다. Preparation은 두 file range를 scan하여 다음 항목을 추정합니다.
항목추정 대상보장
1Fork mapping 반복 update에 필요한 data-device blockTransaction 진행 중 metadata block 부족 방지
2두 file의 data·realtime block count 변화Block accounting 정확성
3두 file의 quota-id set이 다를 때 quota usage 증가량User·group·project quota reservation
4각 file에 추가될 extent mapping 수Fork가 지원하는 최대 mapping 수 초과 방지
5Partially written realtime extent 존재 여부Operation이 끝까지 완료되지 않아도 user가 realtime volume의 서로 다른 extent로 mapping되는 realtime file extent에 접근하지 못하게 함
반복 mapping update와 accounting에 필요한 다섯 가지 추정치를 보존합니다.
정확한 추정은 exchange runtime을 늘리지만 올바른 accounting에 필수입니다. Filesystem이 free space를 완전히 소진해서는 안 되고, mapping exchange가 fork가 지원할 수 있는 수보다 많은 extent mapping을 추가해서도 안 됩니다. 일반 user는 quota limit을 따라야 하지만 metadata repair는 다른 곳의 inconsistent metadata를 해결하기 위해 quota를 초과할 수 있습니다.
Preparation for File Content Exchanges
``````````````````````````````````````
There are a few things that need to be taken care of before initiating an
atomic file mapping exchange operation.
First, regular files require the page cache to be flushed to disk before the
operation begins, and directio writes to be quiesced.
Like any filesystem operation, file mapping exchanges must determine the
maximum amount of disk space and quota that can be consumed on behalf of both
files in the operation, and reserve that quantity of resources to avoid an
unrecoverable out of space failure once it starts dirtying metadata.
The preparation step scans the ranges of both files to estimate:
- Data device blocks needed to handle the repeated updates to the fork
mappings.
- Change in data and realtime block counts for both files.
- Increase in quota usage for both files, if the two files do not share the
same set of quota ids.
- The number of extent mappings that will be added to each file.
- Whether or not there are partially written realtime extents.
User programs must never be able to access a realtime file extent that maps
to different extents on the realtime volume, which could happen if the
operation fails to run to completion.
The need for precise estimation increases the run time of the exchange
operation, but it is very important to maintain correct accounting.
The filesystem must not run completely out of free space, nor can the mapping
exchange ever add more extent mappings to a fork than it can support.
Regular users are required to abide the quota limits, though metadata repairs
may exceed quota to resolve inconsistent metadata elsewhere.
Metadata file-content exchange 특례
4219-4262Extended attribute, symbolic link, directory는 fork format을 `local`로 설정해 fork를 literal data-storage area로 사용할 수 있습니다. Metadata repair는 fork format 조합에 따라 추가 단계를 수행합니다.
상태Exchange 방식Transaction 조건
두 fork 모두 local이고 fork area가 충분히 큼Incore fork content를 copy하고 두 fork를 log한 뒤 commitSingle transaction으로 가능하므로 atomic file-mapping exchange 불필요
두 fork 모두 block mapping일반 atomic file-mapping exchange 사용Logged mapping exchange 수행
한 fork만 localLocal-format content를 block으로 변환한 뒤 regular atomic mapping exchange; special flag로 transaction을 한 번 더 roll하여 두 번째 file fork를 local format으로 복원Block 변환은 initial mapping-exchange intent log item을 기록하는 같은 transaction에서 수행; ILOCK drop 즉시 두 번째 file 사용 가능
두 fork의 local/block 조합에 따른 commit 방식을 구분합니다.
Extended attribute와 directory는 모든 block에 owning inode를 stamp하지만 buffer verifier는 실제로 inode number를 검사하지 않습니다. 검증이 없더라도 referential integrity는 유지해야 하므로 mapping exchange 전에 online repair가 새 data structure의 모든 block을 repair 대상 file의 owner field로 만듭니다.
Exchange 성공 뒤 repair operation은 각 old-fork mapping을 표준 :ref:`file extent reaping <reaping>` mechanism으로 처리해 old fork block을 reap해야 합니다. Reap 중 filesystem이 내려가면 recovery 끝의 `iunlink` processing이 temporary file과 아직 reap하지 않은 block을 free합니다. 다만 이 `iunlink` 경로는 online repair의 cross-link detection을 생략하므로 완전히 foolproof하지는 않습니다.
Special Features for Exchanging Metadata File Contents
``````````````````````````````````````````````````````
Extended attributes, symbolic links, and directories can set the fork format to
"local" and treat the fork as a literal area for data storage.
Metadata repairs must take extra steps to support these cases:
- If both forks are in local format and the fork areas are large enough, the
exchange is performed by copying the incore fork contents, logging both
forks, and committing.
The atomic file mapping exchange mechanism is not necessary, since this can
be done with a single transaction.
- If both forks map blocks, then the regular atomic file mapping exchange is
used.
- Otherwise, only one fork is in local format.
The contents of the local format fork are converted to a block to perform the
exchange.
The conversion to block format must be done in the same transaction that
logs the initial mapping exchange intent log item.
The regular atomic mapping exchange is used to exchange the metadata file
mappings.
Special flags are set on the exchange operation so that the transaction can
be rolled one more time to convert the second file's fork back to local
format so that the second file will be ready to go as soon as the ILOCK is
dropped.
Extended attributes and directories stamp the owning inode into every block,
but the buffer verifiers do not actually check the inode number!
Although there is no verification, it is still important to maintain
referential integrity, so prior to performing the mapping exchange, online
repair builds every block in the new data structure with the owner field of the
file being repaired.
After a successful exchange operation, the repair operation must reap the old
fork blocks by processing each fork mapping through the standard :ref:`file
extent reaping <reaping>` mechanism that is done post-repair.
If the filesystem should go down during the reap part of the repair, the
iunlink processing at the end of recovery will free both the temporary file and
whatever blocks were not reaped.
However, this iunlink processing omits the cross-link detection of online
repair, and is not completely foolproof.
Temporary-file content 교환
4263-4284Metadata file repair를 위한 online repair 절차는 다음 여섯 단계입니다.
단계동작결과
1Temporary repair file 생성새 metadata 대상 inode 확보
2Staging data로 temporary file에 새 content 기록; repair 대상과 같은 fork에 기록교환할 올바른 structure 구축
3Scrub transaction commitExchange resource estimation이 transaction reservation보다 먼저 끝나야 하는 순서 보장
4`xrep_tempexch_trans_alloc`으로 적절한 resource reservation과 lock을 가진 새 scrub transaction allocate하고 exchange 세부 정보를 `struct xfs_exchmaps_req`에 채움교환 실행 context 준비
5`xrep_tempexch_contents` 호출File contents exchange 수행
6Transaction commitRepair 완료
생성·staging·reservation·exchange·commit의 원문 1–6단계를 보존합니다.
Exchanging Temporary File Contents
``````````````````````````````````
To repair a metadata file, online repair proceeds as follows:
1. Create a temporary repair file.
2. Use the staging data to write out new contents into the temporary repair
file.
The same fork must be written to as is being repaired.
3. Commit the scrub transaction, since the exchange resource estimation step
must be completed before transaction reservations are made.
4. Call ``xrep_tempexch_trans_alloc`` to allocate a new scrub transaction with
the appropriate resource reservations, locks, and fill out a ``struct
xfs_exchmaps_req`` with the details of the exchange operation.
5. Call ``xrep_tempexch_contents`` to exchange the contents.
6. Commit the transaction to complete the repair.
Case study: realtime summary file repair
4285-4330XFS filesystem의 realtime section은 Unix FFS와 비슷하게 bitmap으로 free space를 추적합니다. Bitmap의 각 bit는 filesystem block size의 배수이며 4KiB에서 1GiB 사이 크기인 realtime extent 하나를 나타냅니다.
Realtime summary file은 특정 size의 free extent 수를 그 extent가 시작되는 realtime free-space bitmap block offset에 index합니다. 즉 data section의 free-space-by-count btree인 `cntbt`와 비슷하게 allocator가 length별 free extent를 찾도록 돕습니다.
Summary file 자체는 block header나 checksum이 없는 flat file입니다. `log2(total rt extents)`개 section으로 나뉘며, 각 section은 rt bitmap block 수와 맞는 수의 32-bit counter를 담습니다. 각 counter는 해당 bitmap block에서 시작하면서 power-of-two allocation request를 만족할 수 있는 free extent 수를 기록합니다.
요소표현의미
Realtime bitmap bitRealtime extent 하나; 4KiB–1GiB 범위의 filesystem-block 배수Free/allocated 상태
Summary partition`log2(total rt extents)` sectionPower-of-two extent-size class별 영역
32-bit counter각 rt-bitmap block과 대응그 block에서 시작해 해당 size request를 만족하는 free extent 수
Bitmap bit, summary section, counter 의미를 계층별로 구분합니다.
단계동작결과
1Realtime bitmap file과 summary file의 `ILOCK` 모두 획득두 metadata file 안정화
2.aBitmap의 각 free-space extent를 나타내는 summary-file counter 위치 계산Expected counter index 결정
2.bXfile에서 counter 읽기현재 재계산 값 획득
2.cCounter를 증가시켜 xfile에 다시 쓰기Expected summary 누적
3Xfile content와 ondisk file 비교Summary inconsistency 검출
Bitmap의 free extent에서 xfile counter를 재구성한 뒤 ondisk file과 비교합니다.
Summary file을 repair하려면 xfile content를 temporary file에 쓰고 atomic mapping exchange로 새 content를 commit한 뒤 temporary file을 reap합니다. 제안된 patchset은 `realtime summary repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary>`_ series입니다.
.. _rtsummary:
Case Study: Repairing the Realtime Summary File
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In the "realtime" section of an XFS filesystem, free space is tracked via a
bitmap, similar to Unix FFS.
Each bit in the bitmap represents one realtime extent, which is a multiple of
the filesystem block size between 4KiB and 1GiB in size.
The realtime summary file indexes the number of free extents of a given size to
the offset of the block within the realtime free space bitmap where those free
extents begin.
In other words, the summary file helps the allocator find free extents by
length, similar to what the free space by count (cntbt) btree does for the data
section.
The summary file itself is a flat file (with no block headers or checksums!)
partitioned into ``log2(total rt extents)`` sections containing enough 32-bit
counters to match the number of blocks in the rt bitmap.
Each counter records the number of free extents that start in that bitmap block
and can satisfy a power-of-two allocation request.
To check the summary file against the bitmap:
1. Take the ILOCK of both the realtime bitmap and summary files.
2. For each free space extent recorded in the bitmap:
a. Compute the position in the summary file that contains a counter that
represents this free extent.
b. Read the counter from the xfile.
c. Increment it, and write it back to the xfile.
3. Compare the contents of the xfile against the ondisk file.
To repair the summary file, write the xfile contents into the temporary file
and use atomic mapping exchange to commit the new contents.
The temporary file is then reaped.
The proposed patchset is the
`realtime summary repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary>`_
series.
Case study: extended attribute salvage
4331-4376XFS에서 extended attribute는 namespace가 있는 name-value store로 구현됩니다. Value size는 64KiB로 제한되지만 name 수에는 제한이 없습니다.
Attribute fork는 partition되지 않으므로 attribute structure root는 항상 logical block 0에 있지만 attribute leaf block, `dabtree` index block, remote-value block은 서로 섞여 있습니다. Leaf block은 user-provided name과 value를 연결하는 variable-size record를 담습니다. 한 block보다 큰 value는 별도 extent에 allocate하여 기록하고, leaf information이 single block을 넘으면 빠른 lookup을 위해 attribute-name hash를 entry에 mapping하는 directory/attribute btree인 `dabtree`를 만듭니다.
단계동작Staging·commit 결과
1Repair 대상 file의 attr-fork mapping을 순회해 attribute leaf block 탐색; 각 leaf에서 candidate key를 찾고 name 문제를 검사해 문제가 있으면 무시; value retrieval이 성공하면 name과 value를 staging xfarray·xfblob에 추가Salvage 가능한 name-value pair 수집
2Xfarray·xfblob memory usage가 threshold를 넘거나 더 볼 attr-fork block이 없으면 file unlock 후 staged extended attribute를 temporary file에 추가Bounded-memory batch를 temporary structure에 기록
3Atomic file-mapping exchange로 new·old extended-attribute structure 교환Old attribute block이 temporary file에 연결됨
4Temporary file reapOld attribute block 회수
Candidate key 수집부터 atomic exchange와 reap까지 원문의 1–4단계를 보존합니다.
제안된 patchset은 `extended attribute repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series입니다.
Case Study: Salvaging Extended Attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In XFS, extended attributes are implemented as a namespaced name-value store.
Values are limited in size to 64KiB, but there is no limit in the number of
names.
The attribute fork is unpartitioned, which means that the root of the attribute
structure is always in logical block zero, but attribute leaf blocks, dabtree
index blocks, and remote value blocks are intermixed.
Attribute leaf blocks contain variable-sized records that associate
user-provided names with the user-provided values.
Values larger than a block are allocated separate extents and written there.
If the leaf information expands beyond a single block, a directory/attribute
btree (``dabtree``) is created to map hashes of attribute names to entries
for fast lookup.
Salvaging extended attributes is done as follows:
1. Walk the attr fork mappings of the file being repaired to find the attribute
leaf blocks.
When one is found,
a. Walk the attr leaf block to find candidate keys.
When one is found,
1. Check the name for problems, and ignore the name if there are.
2. Retrieve the value.
If that succeeds, add the name and value to the staging xfarray and
xfblob.
2. If the memory usage of the xfarray and xfblob exceed a certain amount of
memory or there are no more attr fork blocks to examine, unlock the file and
add the staged extended attributes to the temporary file.
3. Use atomic file mapping exchange to exchange the new and old extended
attribute structures.
The old attribute blocks are now attached to the temporary file.
4. Reap the temporary file.
The proposed patchset is the
`extended attribute repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_
series.
Directory repair와 salvage
4377-4455현재 filesystem feature만으로 directory를 고치기는 어렵습니다. Directory entry에는 redundancy가 없기 때문입니다. Offline repair tool은 모든 inode를 scan해 link count가 0이 아닌 file을 찾고, 모든 directory를 scan해 linked file의 parentage를 확립합니다. 손상된 file과 directory는 제거하고 parent가 없는 file은 `/lost+found`로 옮기며, 어떤 data도 salvage하려 하지 않습니다.
현재 online repair가 할 수 있는 최선은 directory data block을 읽어 그럴듯해 보이는 dirent를 salvage하고, link count를 바로잡고, orphan을 directory tree로 되돌리는 것입니다. :ref:`file link count fsck <nlinks>` code가 link-count repair와 orphan의 `/lost+found` 이동을 담당합니다.
방식처리Salvage·복구
Offline repairNonzero link-count inode와 모든 directory를 scan해 parentage 확립; damaged file·directory 제거Data salvage 없이 parent 없는 file을 `/lost+found`로 이동
Online repairPlausible dirent salvage, link-count correction, orphan reparentLive filesystem에서 recoverable entry를 최대한 보존
Redundancy가 없는 dirent를 처리하는 두 방식의 한계를 구분합니다.
Extended attribute와 달리 directory block은 모두 같은 size이므로 directory salvage는 다음 다섯 단계로 비교적 단순하게 진행됩니다.
단계동작결과
1Directory parent 탐색; dotdot entry를 읽을 수 있으면 alleged parent에 repair 대상 directory를 가리키는 child entry가 있는지 확인하고, 그렇지 않으면 filesystem 전체를 순회Parent identity 확립
2Directory data fork의 첫 partition을 순회해 entry data block 탐색; 각 candidate entry name을 검사해 문제가 있으면 무시하고, inumber를 얻어 inode grab에 성공하면 name·inode number·file type을 staging xfarray·xblob에 추가Salvage 가능한 dirent 수집
3Xfarray·xfblob memory usage가 threshold를 넘거나 더 볼 data block이 없으면 directory unlock 후 staged dirent를 temporary directory에 추가하고 staging file truncateBounded-memory batch 기록과 재사용
4Atomic file-mapping exchange로 new·old directory structure 교환Old directory block이 temporary file에 연결
5Temporary file reap이전 directory structure 회수
Parent 확인부터 atomic exchange와 temporary-file reap까지 원문의 1–5단계를 보존합니다.
Directory rebuild 뒤 repair는 dentry cache를 재검증해야 하며, 이론적으로 directory의 모든 cached dentry를 scan해 다음 세 상태 중 하나인지 확인해야 합니다.
상태Ondisk 관계처리
1. ValidCached dentry가 새 directory의 ondisk dirent를 반영Cache 유지
2. Stale·purgeable새 directory에 대응 ondisk dirent가 없고 cache에서 purge 가능Dentry purge
3. Stale·unpurgeableOndisk dirent가 더는 없지만 dentry를 purge할 수 없음문제 상태
새 ondisk directory와 cache entry의 관계를 세 경우로 구분합니다.
하지만 현재 dentry-cache design은 특정 directory의 모든 child dentry를 순회할 방법을 제공하지 않아 어려운 문제이며 알려진 해법이 없습니다. 제안된 patchset은 `directory repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_ series입니다.
Fixing Directories
------------------
Fixing directories is difficult with currently available filesystem features,
since directory entries are not redundant.
The offline repair tool scans all inodes to find files with nonzero link count,
and then it scans all directories to establish parentage of those linked files.
Damaged files and directories are zapped, and files with no parent are
moved to the ``/lost+found`` directory.
It does not try to salvage anything.
The best that online repair can do at this time is to read directory data
blocks and salvage any dirents that look plausible, correct link counts, and
move orphans back into the directory tree.
The salvage process is discussed in the case study at the end of this section.
The :ref:`file link count fsck <nlinks>` code takes care of fixing link counts
and moving orphans to the ``/lost+found`` directory.
Case Study: Salvaging Directories
`````````````````````````````````
Unlike extended attributes, directory blocks are all the same size, so
salvaging directories is straightforward:
1. Find the parent of the directory.
If the dotdot entry is not unreadable, try to confirm that the alleged
parent has a child entry pointing back to the directory being repaired.
Otherwise, walk the filesystem to find it.
2. Walk the first partition of data fork of the directory to find the directory
entry data blocks.
When one is found,
a. Walk the directory data block to find candidate entries.
When an entry is found:
i. Check the name for problems, and ignore the name if there are.
ii. Retrieve the inumber and grab the inode.
If that succeeds, add the name, inode number, and file type to the
staging xfarray and xblob.
3. If the memory usage of the xfarray and xfblob exceed a certain amount of
memory or there are no more directory data blocks to examine, unlock the
directory and add the staged dirents into the temporary directory.
Truncate the staging files.
4. Use atomic file mapping exchange to exchange the new and old directory
structures.
The old directory blocks are now attached to the temporary file.
5. Reap the temporary file.
**Future Work Question**: Should repair revalidate the dentry cache when
rebuilding a directory?
*Answer*: Yes, it should.
In theory it is necessary to scan all dentry cache entries for a directory to
ensure that one of the following apply:
1. The cached dentry reflects an ondisk dirent in the new directory.
2. The cached dentry no longer has a corresponding ondisk dirent in the new
directory and the dentry can be purged from the cache.
3. The cached dentry no longer has an ondisk dirent but the dentry cannot be
purged.
This is the problem case.
Unfortunately, the current dentry cache design doesn't provide a means to walk
every child dentry of a specific directory, which makes this a hard problem.
There is no known solution.
The proposed patchset is the
`directory repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_
series.
Parent pointers
4456-4571Parent pointer는 root부터 directory tree를 순회하지 않고도 file의 parent directory를 찾게 하는 file metadata입니다. 과거 reverse-space-mapping information의 부재가 filesystem space metadata reconstruction을 방해했던 것처럼 parent pointer가 없으면 directory tree reconstruction도 제약됩니다. Parent-pointer feature는 전체 directory reconstruction을 가능하게 합니다.
XFS parent pointer는 parent directory의 대응 directory entry를 식별할 정보를 담습니다. Child file은 extended attribute에 ``(dirent_name) → (parent_inum, parent_gen)`` 형태로 parent pointer를 저장합니다. Directory checker는 각 dirent target에도 그 dirent를 되가리키는 parent pointer가 있는지 확인할 수 있고, 반대로 각 parent pointer target이 directory이며 그 pointer와 일치하는 dirent를 포함하는지도 확인할 수 있습니다. Online·offline repair 모두 이 양방향 전략을 사용할 수 있습니다.
항목설계·문제결과·판정
초기 SGI 제안10여 년 전 parent→child link마다 child의 extended attribute에 parent 식별 정보를 mirrorLinux XFS에 merge되지 않음
초기 결함 12000년대 후반 XFS에 directory-tree strong referential integrity infrastructure가 없어 forward-link 변경 뒤 reverse-link 변경을 보장하지 못함양방향 link가 쉽게 불일치
초기 결함 2Referential integrity가 offline repair에 통합되지 않았고 mounted filesystem에서 kernel·inode lock 없이 검사·repair동시 접근과 제대로 조정됐는지 불명확
초기 결함 3Extended attribute에 parent의 directory-entry name을 기록하지 않음Directory tree reconnect에 사용 불가
초기 결함 4Extended-attribute fork가 65,536 extent만 지원최대 file-link count 전에 parent-pointer attribute 생성 실패 가능
두 번째 구현: log intentAllison Henderson가 2022년에 extended-attribute structure의 physical manipulation을 추적하는 log-intent item 도입Dirent update와 parent-pointer update를 같은 transaction에 commit하여 referential-integrity 문제 해결
두 번째 구현: extent capacityChandan Babu가 data·attribute fork의 maximum extent count 증가어떤 file의 maximum hardlink count도 처리할 만큼 xattr structure 확장 가능
초기 두 번째 format``(parent_inum, parent_gen, dirent_pos) → (dirent_name)``Directory reconstruction 때 repair tool이 `dirent_pos` 일치를 유지할 필요를 없애도록 개발 중 변경
대안 1`dirent_pos`를 advisory로 지정; 나머지 세 값으로 parent entry 탐색Repair 중 indexed-key lookup 불가능
대안 2지정 offset에 directory entry 생성 허용Free-space conflict로 dirent creation 실패 위험; append 후 xattr key update와 dabtree reindex로 해결할 수 있으나 parent directory lock 유지 필요
대안 3대안 2와 같되 old parent-pointer entry를 제거하고 new entry를 atomic하게 추가Position 변경을 양방향 xattr update로 처리
대안 4``(parent_inum, name) → (parent_gen)``필요한 attr-name uniqueness를 제공하고 dirent position update 불필요; 263-byte attr name 지원을 위한 xattr code 변경 필요
대안 5``(parent_inum, hash(name)) → (name, parent_gen)``; collision-resistant hash로 `sha256` 예시Attr-name uniqueness 제공; 247 byte보다 짧은 name은 직접 저장 가능
대안 6``(dirent_name) → (parent_ino, parent_gen)``; nested name hashing 불필요같은 inode에 같은 filename으로 여러 hardlink가 있을 때 hashed-xattr lookup 성능 문제가 있어 parent inumber를 hash index에 XOR
최종 결정대안 6이 가장 compact하고 performantParent pointer용 새 hash function 설계
초기 SGI 구현의 결함, 두 번째 구현의 개선, dirent-position 문제의 여섯 대안을 원문 순서대로 구조화했습니다.
원래 parent-pointer design은 filesystem repair가 의존하기에는 너무 불안정했습니다. Allison Henderson, Chandan Babu, Catherine Hoang이 첫 구현의 결함을 해결하는 두 번째 구현을 진행했고, 최종적으로 directory-entry name을 key로 삼는 대안 6과 parent-inumber-aware hash가 선택됐습니다.
Parent Pointers
```````````````
A parent pointer is a piece of file metadata that enables a user to locate the
file's parent directory without having to traverse the directory tree from the
root.
Without them, reconstruction of directory trees is hindered in much the same
way that the historic lack of reverse space mapping information once hindered
reconstruction of filesystem space metadata.
The parent pointer feature, however, makes total directory reconstruction
possible.
XFS parent pointers contain the information needed to identify the
corresponding directory entry in the parent directory.
In other words, child files use extended attributes to store pointers to
parents in the form ``(dirent_name) → (parent_inum, parent_gen)``.
The directory checking process can be strengthened to ensure that the target of
each dirent also contains a parent pointer pointing back to the dirent.
Likewise, each parent pointer can be checked by ensuring that the target of
each parent pointer is a directory and that it contains a dirent matching
the parent pointer.
Both online and offline repair can use this strategy.
+--------------------------------------------------------------------------+
| **Historical Sidebar**: |
+--------------------------------------------------------------------------+
| Directory parent pointers were first proposed as an XFS feature more |
| than a decade ago by SGI. |
| Each link from a parent directory to a child file is mirrored with an |
| extended attribute in the child that could be used to identify the |
| parent directory. |
| Unfortunately, this early implementation had major shortcomings and was |
| never merged into Linux XFS: |
| |
| 1. The XFS codebase of the late 2000s did not have the infrastructure to |
| enforce strong referential integrity in the directory tree. |
| It did not guarantee that a change in a forward link would always be |
| followed up with the corresponding change to the reverse links. |
| |
| 2. Referential integrity was not integrated into offline repair. |
| Checking and repairs were performed on mounted filesystems without |
| taking any kernel or inode locks to coordinate access. |
| It is not clear how this actually worked properly. |
| |
| 3. The extended attribute did not record the name of the directory entry |
| in the parent, so the SGI parent pointer implementation cannot be |
| used to reconnect the directory tree. |
| |
| 4. Extended attribute forks only support 65,536 extents, which means |
| that parent pointer attribute creation is likely to fail at some |
| point before the maximum file link count is achieved. |
| |
| The original parent pointer design was too unstable for something like |
| a file system repair to depend on. |
| Allison Henderson, Chandan Babu, and Catherine Hoang are working on a |
| second implementation that solves all shortcomings of the first. |
| During 2022, Allison introduced log intent items to track physical |
| manipulations of the extended attribute structures. |
| This solves the referential integrity problem by making it possible to |
| commit a dirent update and a parent pointer update in the same |
| transaction. |
| Chandan increased the maximum extent counts of both data and attribute |
| forks, thereby ensuring that the extended attribute structure can grow |
| to handle the maximum hardlink count of any file. |
| |
| For this second effort, the ondisk parent pointer format as originally |
| proposed was ``(parent_inum, parent_gen, dirent_pos) → (dirent_name)``. |
| The format was changed during development to eliminate the requirement |
| of repair tools needing to ensure that the ``dirent_pos`` field always |
| matched when reconstructing a directory. |
| |
| There were a few other ways to have solved that problem: |
| |
| 1. The field could be designated advisory, since the other three values |
| are sufficient to find the entry in the parent. |
| However, this makes indexed key lookup impossible while repairs are |
| ongoing. |
| |
| 2. We could allow creating directory entries at specified offsets, which |
| solves the referential integrity problem but runs the risk that |
| dirent creation will fail due to conflicts with the free space in the |
| directory. |
| |
| These conflicts could be resolved by appending the directory entry |
| and amending the xattr code to support updating an xattr key and |
| reindexing the dabtree, though this would have to be performed with |
| the parent directory still locked. |
| |
| 3. Same as above, but remove the old parent pointer entry and add a new |
| one atomically. |
| |
| 4. Change the ondisk xattr format to |
| ``(parent_inum, name) → (parent_gen)``, which would provide the attr |
| name uniqueness that we require, without forcing repair code to |
| update the dirent position. |
| Unfortunately, this requires changes to the xattr code to support |
| attr names as long as 263 bytes. |
| |
| 5. Change the ondisk xattr format to ``(parent_inum, hash(name)) → |
| (name, parent_gen)``. |
| If the hash is sufficiently resistant to collisions (e.g. sha256) |
| then this should provide the attr name uniqueness that we require. |
| Names shorter than 247 bytes could be stored directly. |
| |
| 6. Change the ondisk xattr format to ``(dirent_name) → (parent_ino, |
| parent_gen)``. This format doesn't require any of the complicated |
| nested name hashing of the previous suggestions. However, it was |
| discovered that multiple hardlinks to the same inode with the same |
| filename caused performance problems with hashed xattr lookups, so |
| the parent inumber is now xor'd into the hash index. |
| |
| In the end, it was decided that solution #6 was the most compact and the |
| most performant. A new hash function was designed for parent pointers. |
+--------------------------------------------------------------------------+
Case study: parent pointer로 directory repair
4572-4619Directory rebuild는 :ref:`coordinated inode scan <iscan>`과 :ref:`directory-entry live-update hook <liveupdate>`을 다음과 같이 사용합니다.
단계동작Staging·commit 의미
1새 directory structure 생성용 temporary directory, entry name 저장용 xfblob, directory update의 fixed-size field ``(child inumber, add vs. remove, name cookie, ftype)``를 보관할 xfarray 설정Variable name과 fixed update를 분리해 staging
2Inode scanner 설정 후 directory-entry code에 hook을 설치해 directory operation update 수신Live update channel 활성화
3Scan한 각 file에서 찾은 parent pointer가 대상 directory를 참조하는지 판단; 맞으면 name은 xfblob, 이 dirent의 `addname` entry는 xfarray에 저장; file scan 완료 또는 kernel memory threshold 초과 시 temporary directory로 flushParent-pointer observation을 bounded-memory batch로 새 directory에 반영
4Hook으로 받은 각 live directory update에서 child가 이미 scan됐는지 판단; scan됐다면 parent-pointer name과 `addname` 또는 `removename` entry를 xfblob·xfarray에 보관Hook은 filesystem metadata를 수정할 수 없어 temporary directory에 직접 쓰지 않고 scanner thread가 staged update를 적용
5Scan 완료 뒤 xfarray의 남은 staged entry replayLive update backlog 반영
6Temporary directory와 repair 대상 directory의 content를 atomic하게 exchangeTemporary directory가 damaged old directory structure를 받음
7Temporary directory reapOld structure 회수
Temporary directory staging부터 damaged structure reap까지 원문의 1–7단계를 보존합니다.
제안된 patchset은 `parent pointers directory repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_ series입니다.
Case Study: Repairing Directories with Parent Pointers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Directory rebuilding uses a :ref:`coordinated inode scan <iscan>` and
a :ref:`directory entry live update hook <liveupdate>` as follows:
1. Set up a temporary directory for generating the new directory structure,
an xfblob for storing entry names, and an xfarray for stashing the fixed
size fields involved in a directory update: ``(child inumber, add vs.
remove, name cookie, ftype)``.
2. Set up an inode scanner and hook into the directory entry code to receive
updates on directory operations.
3. For each parent pointer found in each file scanned, decide if the parent
pointer references the directory of interest.
If so:
a. Stash the parent pointer name and an addname entry for this dirent in the
xfblob and xfarray, respectively.
b. When finished scanning that file or the kernel memory consumption exceeds
a threshold, flush the stashed updates to the temporary directory.
4. For each live directory update received via the hook, decide if the child
has already been scanned.
If so:
a. Stash the parent pointer name an addname or removename entry for this
dirent update in the xfblob and xfarray for later.
We cannot write directly to the temporary directory because hook
functions are not allowed to modify filesystem metadata.
Instead, we stash updates in the xfarray and rely on the scanner thread
to apply the stashed updates to the temporary directory.
5. When the scan is complete, replay any stashed entries in the xfarray.
6. When the scan is complete, atomically exchange the contents of the temporary
directory and the directory being repaired.
The temporary directory now contains the damaged directory structure.
7. Reap the temporary directory.
The proposed patchset is the
`parent pointers directory repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_
series.
Case study: parent pointer repair
4620-4669File의 parent-pointer information을 online으로 reconstruct하는 과정은 directory reconstruction과 비슷하지만, directory entry를 scan하여 새 extended-attribute structure를 만듭니다.
단계동작Staging·commit 의미
1새 extended-attribute structure 생성용 temporary file, parent-pointer name 저장용 xfblob, fixed-size field ``(parent inumber, parent generation, add vs. remove, name cookie)``를 보관할 xfarray 설정Variable name과 parent identity update 분리
2Inode scanner 설정 후 directory-entry code에 hook을 설치해 directory operation update 수신Live update channel 활성화
3Scan한 각 directory의 dirent가 대상 file을 참조하는지 판단; 맞으면 dirent name은 xfblob, `addpptr` entry는 xfarray에 저장; directory scan 완료 또는 kernel memory threshold 초과 시 temporary file로 flushForward dirent에서 reverse parent pointer 재구성
4Hook으로 받은 각 live directory update에서 parent가 이미 scan됐는지 판단; scan됐다면 dirent name과 `addpptr` 또는 `removepptr` entry를 xfblob·xfarray에 보관Hook은 filesystem metadata를 수정할 수 없으므로 scanner thread가 staged parent-pointer update를 temporary file에 적용
5Scan 완료 뒤 xfarray의 남은 staged entry replayLive update backlog 반영
6Parent pointer가 아닌 extended attribute를 모두 temporary file에 copy기존 non-parent xattr 보존
7Temporary file과 repair 대상 file의 attribute-fork mapping을 atomic하게 exchangeTemporary file이 damaged old extended-attribute structure를 받음
8Temporary file reapOld xattr structure 회수
Temporary xattr file staging부터 attr-fork exchange와 reap까지 원문의 1–8단계를 보존합니다.
제안된 patchset은 `parent pointers repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_ series입니다.
Case Study: Repairing Parent Pointers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Online reconstruction of a file's parent pointer information works similarly to
directory reconstruction:
1. Set up a temporary file for generating a new extended attribute structure,
an xfblob for storing parent pointer names, and an xfarray for stashing the
fixed size fields involved in a parent pointer update: ``(parent inumber,
parent generation, add vs. remove, name cookie)``.
2. Set up an inode scanner and hook into the directory entry code to receive
updates on directory operations.
3. For each directory entry found in each directory scanned, decide if the
dirent references the file of interest.
If so:
a. Stash the dirent name and an addpptr entry for this parent pointer in the
xfblob and xfarray, respectively.
b. When finished scanning the directory or the kernel memory consumption
exceeds a threshold, flush the stashed updates to the temporary file.
4. For each live directory update received via the hook, decide if the parent
has already been scanned.
If so:
a. Stash the dirent name and an addpptr or removepptr entry for this dirent
update in the xfblob and xfarray for later.
We cannot write parent pointers directly to the temporary file because
hook functions are not allowed to modify filesystem metadata.
Instead, we stash updates in the xfarray and rely on the scanner thread
to apply the stashed parent pointer updates to the temporary file.
5. When the scan is complete, replay any stashed entries in the xfarray.
6. Copy all non-parent pointer extended attributes to the temporary file.
7. When the scan is complete, atomically exchange the mappings of the attribute
forks of the temporary file and the file being repaired.
The temporary file now contains the damaged extended attribute structure.
8. Reap the temporary file.
The proposed patchset is the
`parent pointers repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_
series.
Digression: offline parent-pointer 검사
4670-4762Offline repair에서는 directory-tree connectivity check보다 훨씬 전에 corrupt file을 지우므로 parent-pointer 검사가 다르게 동작합니다. 따라서 기존 connectivity check에 다음 second pass를 추가합니다.
단계동작자료 구조·정렬
1Surviving file set을 확정한 phase 6 뒤 filesystem 각 AG의 surviving directory 순회기존 connectivity check walk 재사용
2각 dirent name이 xfblob에 있으면 기존 cookie 사용, 없으면 name을 기록하고 cookie 기억; ``(child_ag_inum, parent_inum, parent_gen, name_hash, name_len, name_cookie)`` tuple을 per-AG in-memory slab에 저장`name_hash`는 parent-pointer xattr용 special hash가 아니라 일반 directory-entry name hash
3각 AG에서 per-AG tuple을 `child_ag_inum`, `parent_inum`, `name_hash`, `name_cookie` 순으로 sort; 각 inode의 ondisk parent pointer를 validate해 ``(parent_inum, parent_gen, name_hash, name_len, name_cookie)`` per-file tuple을 만들고 `parent_inum`, `name_hash`, `name_cookie` 순으로 sort; 두 slab cursor를 lockstep 비교Dirent observation과 parent-pointer record의 merge validation
4현재 방식대로 link-count 검사로 이동Parent-pointer second pass 완료
Surviving dirent 수집부터 tuple merge와 link-count 검사까지 원문의 1–4단계를 보존합니다.
Name-to-cookie mapping이 unique해야 name deduplication으로 memory usage를 줄이고 parent-pointer index용 stable sort key를 만들 수 있습니다. 특히 한 directory에 같은 file을 가리키며 모든 name hash가 같은 여러 hardlink가 있는 드문 경우에도 name마다 하나의 `name_cookie`가 있어야 올바르게 처리할 수 있습니다.
비교 결과의미Repair·cursor 전진
Per-AG cursor key가 더 낮음Dirent는 있지만 parent pointer가 누락Inode에 parent pointer를 추가하고 per-AG cursor 전진
Per-file cursor key가 더 낮음대응 dirent가 없는 dangling parent pointerInode에서 parent pointer를 제거하고 per-file cursor 전진
두 cursor key 동일같은 parent pointer를 양쪽에서 확인필요하면 `parent_gen` update 후 두 cursor 모두 전진
세 가지 keyspace ordering 결과에 따른 repair 동작을 구분합니다.
각 ondisk parent pointer validation이 실패하면 그 file의 다음 pointer로 넘어갑니다. 제안된 patchset은 `offline parent pointers repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fsck>`_ series입니다.
Digression: Offline Checking of Parent Pointers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Examining parent pointers in offline repair works differently because corrupt
files are erased long before directory tree connectivity checks are performed.
Parent pointer checks are therefore a second pass to be added to the existing
connectivity checks:
1. After the set of surviving files has been established (phase 6),
walk the surviving directories of each AG in the filesystem.
This is already performed as part of the connectivity checks.
2. For each directory entry found,
a. If the name has already been stored in the xfblob, then use that cookie
and skip the next step.
b. Otherwise, record the name in an xfblob, and remember the xfblob cookie.
Unique mappings are critical for
1. Deduplicating names to reduce memory usage, and
2. Creating a stable sort key for the parent pointer indexes so that the
parent pointer validation described below will work.
c. Store ``(child_ag_inum, parent_inum, parent_gen, name_hash, name_len,
name_cookie)`` tuples in a per-AG in-memory slab. The ``name_hash``
referenced in this section is the regular directory entry name hash, not
the specialized one used for parent pointer xattrs.
3. For each AG in the filesystem,
a. Sort the per-AG tuple set in order of ``child_ag_inum``, ``parent_inum``,
``name_hash``, and ``name_cookie``.
Having a single ``name_cookie`` for each ``name`` is critical for
handling the uncommon case of a directory containing multiple hardlinks
to the same file where all the names hash to the same value.
b. For each inode in the AG,
1. Scan the inode for parent pointers.
For each parent pointer found,
a. Validate the ondisk parent pointer.
If validation fails, move on to the next parent pointer in the
file.
b. If the name has already been stored in the xfblob, then use that
cookie and skip the next step.
c. Record the name in a per-file xfblob, and remember the xfblob
cookie.
d. Store ``(parent_inum, parent_gen, name_hash, name_len,
name_cookie)`` tuples in a per-file slab.
2. Sort the per-file tuples in order of ``parent_inum``, ``name_hash``,
and ``name_cookie``.
3. Position one slab cursor at the start of the inode's records in the
per-AG tuple slab.
This should be trivial since the per-AG tuples are in child inumber
order.
4. Position a second slab cursor at the start of the per-file tuple slab.
5. Iterate the two cursors in lockstep, comparing the ``parent_ino``,
``name_hash``, and ``name_cookie`` fields of the records under each
cursor:
a. If the per-AG cursor is at a lower point in the keyspace than the
per-file cursor, then the per-AG cursor points to a missing parent
pointer.
Add the parent pointer to the inode and advance the per-AG
cursor.
b. If the per-file cursor is at a lower point in the keyspace than
the per-AG cursor, then the per-file cursor points to a dangling
parent pointer.
Remove the parent pointer from the inode and advance the per-file
cursor.
c. Otherwise, both cursors point at the same parent pointer.
Update the parent_gen component if necessary.
Advance both cursors.
4. Move on to examining link counts, as we do today.
The proposed patchset is the
`offline parent pointers repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fsck>`_
series.
Offline parent pointer 기반 directory rebuild 제안
4763-4787Offline repair에서 parent pointer로 directory를 rebuild하는 일은 매우 어렵습니다. `xfs_repair`가 현재 phase 3과 4에서 두 번의 single-pass filesystem scan으로 어느 file이 제거할 만큼 손상됐는지 결정하기 때문입니다. 이 scan을 다음 multi-pass 방식으로 바꿔야 합니다.
Pass동작제약·결과
1현재와 비슷하게 corrupt inode·fork·attribute 제거; corrupt directory는 표시만 하고 제거하지 않음Rebuild 후보 보존
2첫 pass에서 corrupt로 표시한 directory를 가리키는 parent pointer 기록Phase 4가 directory도 제거할 수 있다면 duplicate-block phase 4 scan 뒤에 수행해야 할 수 있음
3Corrupt directory를 empty shortform directory로 reset아직 free-space metadata가 보장되지 않아 libxfs directory-building code 사용 불가
4Space metadata가 rebuild된 phase 6 시작에 pass 2의 parent-pointer information으로 dirent를 reconstruct하여 빈 directory에 추가Directory tree 복구
Corrupt inode 제거부터 phase 6 dirent reconstruction까지 네 pass를 보존합니다.
이 code는 아직 구현되지 않았습니다.
Rebuilding directories from parent pointers in offline repair would be very
challenging because xfs_repair currently uses two single-pass scans of the
filesystem during phases 3 and 4 to decide which files are corrupt enough to be
zapped.
This scan would have to be converted into a multi-pass scan:
1. The first pass of the scan zaps corrupt inodes, forks, and attributes
much as it does now.
Corrupt directories are noted but not zapped.
2. The next pass records parent pointers pointing to the directories noted
as being corrupt in the first pass.
This second pass may have to happen after the phase 4 scan for duplicate
blocks, if phase 4 is also capable of zapping directories.
3. The third pass resets corrupt directories to an empty shortform directory.
Free space metadata has not been ensured yet, so repair cannot yet use the
directory building code in libxfs.
4. At the start of phase 6, space metadata have been rebuilt.
Use the parent pointer information recorded during step 2 to reconstruct
the dirents and add them to the now-empty directories.
This code has not yet been constructed.
Case study: directory tree structure
4788-4911Filesystem directory tree는 directed acyclic graph여야 합니다. 그러나 graph의 각 node가 자체 lock을 가진 별도 `xfs_inode` object이므로 tree 속성을 검증하기 어렵습니다. Non-directory는 여러 parent를 가질 수 있고 child는 가질 수 없으므로 directory만 scan하면 됩니다. Directory는 일반적으로 filesystem file의 5–10%여서 작업량이 크게 줄어듭니다.
Directory tree를 freeze할 수 있다면 root에서 아래로 depth-first 또는 breadth-first search를 하며 찾은 directory bit를 bitmap에 표시해 cycle과 disconnected region을 쉽게 찾을 수 있습니다. 이미 set된 bit를 다시 set하려 하면 cycle이고, scan 뒤 marked-inode bitmap과 inode-allocation bitmap을 XOR하면 disconnected inode가 드러납니다. 하지만 live filesystem에서는 subtree update가 scanner wavefront를 가로질러 이동할 수 있어 이 bitmap algorithm을 적용할 수 없습니다.
Parent pointer는 tree structure를 incremental하게 검증할 수 있게 합니다. 한 thread가 filesystem 전체를 scan하는 대신 여러 thread가 개별 subdirectory에서 root를 향해 위로 걸어갑니다. 이를 위해 모든 dirent와 parent pointer가 내부적으로 일관되어야 하고, 각 dirent에 parent pointer가 있으며 모든 directory link count가 정확해야 합니다.
Scanner가 child `IOLOCK`을 보유한 채 alleged parent의 `IOLOCK`을 얻어 두 directory의 이동을 막는 방식은 VFS가 subdirectory move 때 child `IOLOCK`을 잡지 않으므로 사용할 수 없습니다. 대신 scanner는 두 `ILOCK`을 획득하고 dirent-update hook을 설치하여 parent→child 관계의 변경을 검출합니다.
단계동작판정·후속
1.a.1각 subdirectory parent pointer마다 path object를 만들고 path bitmap에 subdirectory inumber 표시Path-local visited set 초기화
1.a.2Parent-pointer name과 inode number를 path structure에 기록현재 edge 보존
1.a.3Alleged parent가 scrub 중인 subdirectory 자체인지 확인같으면 cycle; path를 deletion으로 표시하고 다음 parent pointer로 이동
1.a.4Path bitmap에 alleged-parent inumber 표시 시도Bit가 이미 set이면 directory-tree cycle; path를 cycle로 표시하고 다음 parent pointer로 이동
1.a.5Alleged parent loadLinked directory가 아니면 parent-pointer information이 inconsistent하므로 scan abort
1.a.6Alleged ancestor의 각 parent pointer를 조사; 해당 level parent가 아직 없으면 name·inumber 기록, parent가 둘 이상이면 path corrupt; 선택한 ancestor에 대해 1.a.3–1.a.6 반복Root 도달 또는 parent 없음까지 upward walk
1.a.7Walk가 root directory에서 끝남Path를 `ok`로 표시
1.a.8Root에 닿지 않고 walk 종료Path를 `disconnected`로 표시
2Directory-entry update hook이 trigger되면 이미 찾은 모든 path 검사Entry가 path 일부와 일치하면 path와 scan을 stale로 표시; scanner가 모든 scan data를 삭제하고 처음부터 재시작
각 subdirectory의 parent pointer에서 root까지 올라가며 path 상태를 판정하는 원문의 scan 절차를 보존합니다.
단계조건Repair
1Target subdirectory의 각 path 분류Corrupt·cycle path는 suspect, deletion 표시 path는 bad, root 도달 path는 good으로 count
2Subdirectory가 root이거나 link count가 0Immediate parent의 incoming directory entry를 모두 삭제하고 완료
3Path가 정확히 하나Dotdot entry를 그 parent로 설정하고 종료
4Good path가 하나 이상Immediate parent의 다른 incoming directory entry를 모두 삭제
5Good path가 없고 suspect path가 둘 이상Immediate parent의 다른 incoming directory entry를 모두 삭제
6Path가 0개Lost and found에 연결
Path 분류부터 incoming-entry 삭제와 lost-and-found 연결까지 원문의 1–6단계를 보존합니다.
제안된 patch는 `directory tree repair <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-directory-tree>`_ series입니다.
.. _dirtree:
Case Study: Directory Tree Structure
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As mentioned earlier, the filesystem directory tree is supposed to be a
directed acylic graph structure.
However, each node in this graph is a separate ``xfs_inode`` object with its
own locks, which makes validating the tree qualities difficult.
Fortunately, non-directories are allowed to have multiple parents and cannot
have children, so only directories need to be scanned.
Directories typically constitute 5-10% of the files in a filesystem, which
reduces the amount of work dramatically.
If the directory tree could be frozen, it would be easy to discover cycles and
disconnected regions by running a depth (or breadth) first search downwards
from the root directory and marking a bitmap for each directory found.
At any point in the walk, trying to set an already set bit means there is a
cycle.
After the scan completes, XORing the marked inode bitmap with the inode
allocation bitmap reveals disconnected inodes.
However, one of online repair's design goals is to avoid locking the entire
filesystem unless it's absolutely necessary.
Directory tree updates can move subtrees across the scanner wavefront on a live
filesystem, so the bitmap algorithm cannot be applied.
Directory parent pointers enable an incremental approach to validation of the
tree structure.
Instead of using one thread to scan the entire filesystem, multiple threads can
walk from individual subdirectories upwards towards the root.
For this to work, all directory entries and parent pointers must be internally
consistent, each directory entry must have a parent pointer, and the link
counts of all directories must be correct.
Each scanner thread must be able to take the IOLOCK of an alleged parent
directory while holding the IOLOCK of the child directory to prevent either
directory from being moved within the tree.
This is not possible since the VFS does not take the IOLOCK of a child
subdirectory when moving that subdirectory, so instead the scanner stabilizes
the parent -> child relationship by taking the ILOCKs and installing a dirent
update hook to detect changes.
The scanning process uses a dirent hook to detect changes to the directories
mentioned in the scan data.
The scan works as follows:
1. For each subdirectory in the filesystem,
a. For each parent pointer of that subdirectory,
1. Create a path object for that parent pointer, and mark the
subdirectory inode number in the path object's bitmap.
2. Record the parent pointer name and inode number in a path structure.
3. If the alleged parent is the subdirectory being scrubbed, the path is
a cycle.
Mark the path for deletion and repeat step 1a with the next
subdirectory parent pointer.
4. Try to mark the alleged parent inode number in a bitmap in the path
object.
If the bit is already set, then there is a cycle in the directory
tree.
Mark the path as a cycle and repeat step 1a with the next subdirectory
parent pointer.
5. Load the alleged parent.
If the alleged parent is not a linked directory, abort the scan
because the parent pointer information is inconsistent.
6. For each parent pointer of this alleged ancestor directory,
a. Record the parent pointer name and inode number in the path object
if no parent has been set for that level.
b. If an ancestor has more than one parent, mark the path as corrupt.
Repeat step 1a with the next subdirectory parent pointer.
c. Repeat steps 1a3-1a6 for the ancestor identified in step 1a6a.
This repeats until the directory tree root is reached or no parents
are found.
7. If the walk terminates at the root directory, mark the path as ok.
8. If the walk terminates without reaching the root, mark the path as
disconnected.
2. If the directory entry update hook triggers, check all paths already found
by the scan.
If the entry matches part of a path, mark that path and the scan stale.
When the scanner thread sees that the scan has been marked stale, it deletes
all scan data and starts over.
Repairing the directory tree works as follows:
1. Walk each path of the target subdirectory.
a. Corrupt paths and cycle paths are counted as suspect.
b. Paths already marked for deletion are counted as bad.
c. Paths that reached the root are counted as good.
2. If the subdirectory is either the root directory or has zero link count,
delete all incoming directory entries in the immediate parents.
Repairs are complete.
3. If the subdirectory has exactly one path, set the dotdot entry to the
parent and exit.
4. If the subdirectory has at least one good path, delete all the other
incoming directory entries in the immediate parents.
5. If the subdirectory has no good paths and more than one suspect path, delete
all the other incoming directory entries in the immediate parents.
6. If the subdirectory has zero paths, attach it to the lost and found.
The proposed patches are in the
`directory tree repair
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-directory-tree>`_
series.
Orphanage
4912-4980Filesystem은 file을 directed graph, 가능하면 acyclic graph인 tree로 제시합니다. Root는 directory이고 각 directory entry는 아래쪽의 subdirectory 또는 non-directory file을 가리킵니다. Directory graph pointer가 끊어지면 disconnected graph가 생겨 regular path resolution으로 file에 접근할 수 없게 됩니다.
Parent pointer가 없더라도 directory-parent-pointer online scrub은 child로 돌아오는 link가 없는 parent를 가리키는 dotdot entry를 검출할 수 있고, file-link-count checker는 filesystem의 어느 directory도 가리키지 않는 file을 찾을 수 있습니다. 그런 file의 link count가 양수이면 orphan입니다.
Parent pointer가 있으면 parent pointer를 scan해 directory를 rebuild하고 directory를 scan해 parent pointer를 rebuild할 수 있으므로 `/lost+found`로 가는 file 수를 줄일 수 있습니다.
Orphan은 directory tree에 다시 연결해야 합니다. Offline fsck는 orphanage 역할의 `/lost+found` directory를 만들고 inumber를 name으로 사용해 orphan file을 link합니다. File을 orphanage로 reparent해도 permission이나 ACL은 reset하지 않습니다.
Kernel에서는 이 과정이 userspace보다 복잡합니다. Directory·file-link-count repair setup function은 일반 directory-tree modification처럼 필요한 security attribute와 dentry-cache entry를 모두 갖춘 orphanage directory를 regular VFS mechanism으로 만들어야 합니다.
단계API·동작Resource·결과
1Scrub setup 시작에 `xrep_orphanage_try_create` 호출Lost-and-found 존재를 보장하려 시도하고 orphanage directory를 scrub context에 attach
2File reconnect를 결정하면 orphanage와 reattach 대상 file의 `IOLOCK`을 모두 획득; `xrep_orphanage_iolock_two` 사용앞서 설명한 inode-locking strategy 준수
3`xrep_adoption_trans_alloc`으로 repair transaction resource reserveAdoption용 transaction 준비
4`xrep_orphanage_compute_name`으로 orphanage 안의 새 name 계산Inumber 기반 destination name 확정
5실제 adoption이면 `xrep_adoption_reparent` 호출Orphan file을 lost-and-found로 reparent하고 dentry cache invalidate
6`xrep_adoption_finish`로 filesystem update commit, orphanage `ILOCK` release, scrub transaction 정리; `xrep_adoption_commit`으로 update와 scrub transaction commit정상 완료
7Runtime error 시 `xrep_adoption_cancel` 호출모든 resource release
Scrub setup에서 생성 확인부터 commit·cancel까지 원문의 1–7단계를 보존합니다.
제안된 patch는 `orphanage adoption <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-orphanage>`_ series입니다.
.. _orphanage:
The Orphanage
-------------
Filesystems present files as a directed, and hopefully acyclic, graph.
In other words, a tree.
The root of the filesystem is a directory, and each entry in a directory points
downwards either to more subdirectories or to non-directory files.
Unfortunately, a disruption in the directory graph pointers result in a
disconnected graph, which makes files impossible to access via regular path
resolution.
Without parent pointers, the directory parent pointer online scrub code can
detect a dotdot entry pointing to a parent directory that doesn't have a link
back to the child directory and the file link count checker can detect a file
that isn't pointed to by any directory in the filesystem.
If such a file has a positive link count, the file is an orphan.
With parent pointers, directories can be rebuilt by scanning parent pointers
and parent pointers can be rebuilt by scanning directories.
This should reduce the incidence of files ending up in ``/lost+found``.
When orphans are found, they should be reconnected to the directory tree.
Offline fsck solves the problem by creating a directory ``/lost+found`` to
serve as an orphanage, and linking orphan files into the orphanage by using the
inumber as the name.
Reparenting a file to the orphanage does not reset any of its permissions or
ACLs.
This process is more involved in the kernel than it is in userspace.
The directory and file link count repair setup functions must use the regular
VFS mechanisms to create the orphanage directory with all the necessary
security attributes and dentry cache entries, just like a regular directory
tree modification.
Orphaned files are adopted by the orphanage as follows:
1. Call ``xrep_orphanage_try_create`` at the start of the scrub setup function
to try to ensure that the lost and found directory actually exists.
This also attaches the orphanage directory to the scrub context.
2. If the decision is made to reconnect a file, take the IOLOCK of both the
orphanage and the file being reattached.
The ``xrep_orphanage_iolock_two`` function follows the inode locking
strategy discussed earlier.
3. Use ``xrep_adoption_trans_alloc`` to reserve resources to the repair
transaction.
4. Call ``xrep_orphanage_compute_name`` to compute the new name in the
orphanage.
5. If the adoption is going to happen, call ``xrep_adoption_reparent`` to
reparent the orphaned file into the lost and found and invalidate the dentry
cache.
6. Call ``xrep_adoption_finish`` to commit any filesystem updates, release the
orphanage ILOCK, and clean the scrub transaction. Call
``xrep_adoption_commit`` to commit the updates and the scrub transaction.
7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all
resources.
The proposed patches are in the
`orphanage adoption
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-orphanage>`_
series.
사용자 공간 알고리즘과 자료 구조: 메타데이터 검사
4981-5054이 절에서는 사용자 공간 프로그램 `xfs_scrub`의 핵심 알고리즘과 자료 구조를 설명합니다. 이들은 kernel의 metadata 검사와 repair를 구동하고, file data를 검증하며, 그 밖의 잠재적인 문제를 찾을 수 있게 합니다.
앞에서 설명한 :ref:`fsck 작업 단계<scrubphases>`는 XFS가 1993년에 처음 설계될 때부터 갖고 있던 data dependency에서 자연스럽게 도출됩니다. XFS metadata의 의존 관계는 다음과 같은 아홉 그룹으로 나뉩니다.
그룹대상일관성을 먼저 보장해야 하는 구조
aFilesystem summary countInode index, allocation-group space btree, realtime-volume space information
bQuota resource countQuota file의 data fork, inode index, inode record, filesystem에 있는 모든 file의 fork
cNaming hierarchy와 file link countDirectory와 extended-attribute structure
dDirectory, extended attribute, file dataDirectory 및 extended-attribute data를 physical storage media에 mapping하는 file fork
eFile forkInode record, allocation group과 realtime volume의 space-metadata index. Quota 및 realtime metadata file도 포함
fInode recordInode metadata index
gRealtime space metadataRealtime metadata inode의 inode record와 data fork
hAllocation-group metadata indexAG header와 모든 AG metadata btree 사이의 일관성. Free-space, inode, reference-count, reverse-mapping btree를 포함
i`xfs_scrub` 실행 기반Filesystem이 mount되어 있고 kernel이 online-fsck 기능을 지원해야 함
상위 집계와 namespace부터 inode 및 allocation-group 기반 구조까지, 각 그룹이 일관성을 의존하는 하위 metadata를 원문 순서대로 정리합니다.
따라서 metadata dependency graph는 `xfs_scrub`의 검사 작업을 scheduling하는 편리한 모델입니다. Program flow 자체가 그룹 사이의 data dependency를 강제하도록 각 단계가 다음 순서로 구성됩니다.
Phase검사 작업의존성·병렬성
1주어진 path가 XFS filesystem에 mapping되는지 확인하고 kernel의 scrubbing capability를 탐지그룹 (i) 검증
2그룹 (g)와 (h)를 scrubThreaded workqueue에서 병렬 실행
3Inode를 병렬 scan하고 각 inode에서 그룹 (f), (e), (d)를 이 순서로 검사Inode 내부 순서는 dependency에 따라 직렬
4그룹 (i)부터 (d)까지 모든 항목을 repairPhase 5와 6이 신뢰성 있게 실행될 기반 마련
5그룹 (b)와 (c)를 병렬 검사한 뒤 name 검사로 진행Quota와 naming hierarchy를 먼저 검증
6검증할 file-data block을 찾고 읽은 뒤, 어떤 file의 어느 block이 영향을 받았는지 보고그룹 (i)부터 (b)까지의 결과에 의존
7Filesystem summary count인 그룹 (a)를 검사그 밖의 모든 항목을 검증한 뒤 마지막으로 실행
각 phase가 검증하거나 의존하는 그룹과 병렬 처리 범위를 원문 순서대로 보존합니다.
이 구조에서 그룹 사이의 data dependency는 별도의 임의 규칙이 아니라 program flow의 단계 구성 자체로 강제된다는 점에 주목해야 합니다.
6. Userspace Algorithms and Data Structures
===========================================
This section discusses the key algorithms and data structures of the userspace
program, ``xfs_scrub``, that provide the ability to drive metadata checks and
repairs in the kernel, verify file data, and look for other potential problems.
.. _scrubcheck:
Checking Metadata
-----------------
Recall the :ref:`phases of fsck work<scrubphases>` outlined earlier.
That structure follows naturally from the data dependencies designed into the
filesystem from its beginnings in 1993.
In XFS, there are several groups of metadata dependencies:
a. Filesystem summary counts depend on consistency within the inode indices,
the allocation group space btrees, and the realtime volume space
information.
b. Quota resource counts depend on consistency within the quota file data
forks, inode indices, inode records, and the forks of every file on the
system.
c. The naming hierarchy depends on consistency within the directory and
extended attribute structures.
This includes file link counts.
d. Directories, extended attributes, and file data depend on consistency within
the file forks that map directory and extended attribute data to physical
storage media.
e. The file forks depends on consistency within inode records and the space
metadata indices of the allocation groups and the realtime volume.
This includes quota and realtime metadata files.
f. Inode records depends on consistency within the inode metadata indices.
g. Realtime space metadata depend on the inode records and data forks of the
realtime metadata inodes.
h. The allocation group metadata indices (free space, inodes, reference count,
and reverse mapping btrees) depend on consistency within the AG headers and
between all the AG metadata btrees.
i. ``xfs_scrub`` depends on the filesystem being mounted and kernel support
for online fsck functionality.
Therefore, a metadata dependency graph is a convenient way to schedule checking
operations in the ``xfs_scrub`` program:
- Phase 1 checks that the provided path maps to an XFS filesystem and detect
the kernel's scrubbing abilities, which validates group (i).
- Phase 2 scrubs groups (g) and (h) in parallel using a threaded workqueue.
- Phase 3 scans inodes in parallel.
For each inode, groups (f), (e), and (d) are checked, in that order.
- Phase 4 repairs everything in groups (i) through (d) so that phases 5 and 6
may run reliably.
- Phase 5 starts by checking groups (b) and (c) in parallel before moving on
to checking names.
- Phase 6 depends on groups (i) through (b) to find file data blocks to verify,
to read them, and to report which blocks of which files are affected.
- Phase 7 checks group (a), having validated everything else.
Notice that the data dependencies between groups are enforced by the structure
of the program flow.
병렬 inode scan
5055-5101XFS filesystem에는 수억 개의 inode가 쉽게 존재할 수 있습니다. XFS는 대규모 고성능 storage 설치 환경을 대상으로 하므로, 특히 사용자가 command line에서 직접 실행했을 때 전체 시간을 줄이려면 inode를 병렬로 scrub하는 편이 바람직합니다. 이를 위해서는 thread의 작업량을 가능한 한 고르게 유지하는 세심한 scheduling이 필요합니다.
초기 `xfs_scrub` inode scanner는 단순하게 workqueue 하나를 만들고 AG마다 workqueue item 하나를 배정했습니다. 각 item은 `XFS_IOC_INUMBERS`로 inode btree를 순회해 inode chunk를 찾은 뒤, `XFS_IOC_BULKSTAT`을 호출해 file handle을 구성하는 데 필요한 정보를 모았습니다. 그런 다음 file handle을 각 inode의 metadata object마다 scrub item을 생성하는 함수에 전달했습니다.
Filesystem의 AG 하나에는 소수의 큰 sparse file만 있고 나머지 AG에는 작은 file이 많이 들어 있으면, 이 단순한 알고리즘은 phase 3에서 thread balancing 문제를 일으킵니다. Inode scan dispatch 단위가 충분히 세밀하지 않았기 때문입니다. 개별 inode 단위로 dispatch하거나, memory 사용량을 제한하려면 inode btree record 단위로 dispatch해야 했습니다.
Dave Chinner가 제안한 사용자 공간 bounded workqueue 덕분에 `xfs_scrub`은 두 번째 workqueue를 추가하는 방식으로 이 문제를 완화합니다. 첫 번째 queue는 이전과 마찬가지로 AG마다 item 하나를 받아 INUMBERS로 inode btree chunk를 찾고, 두 번째 queue에는 실행을 기다릴 수 있는 item 수의 상한을 설정합니다.
구성 요소입력·작업부하 제어
첫 번째 workqueueAG마다 item 하나를 받아 INUMBERS로 inode btree chunk 검색발견한 각 chunk를 두 번째 workqueue에 enqueue
두 번째 bounded workqueueBULKSTAT 조회, file handle 생성, 각 inode metadata object의 scrub item 생성대기 item 수에 상한을 둠
Workqueue add 함수두 번째 queue가 너무 가득 차면 첫 번째 queue의 worker를 blockBacklog가 줄어들 때까지 producer를 늦춰 memory와 부하를 제한
검색과 실제 inode 처리를 분리하고 bounded queue로 producer에 backpressure를 거는 구조입니다.
이 방식이 balancing 문제를 완전히 해결하지는 않지만, 더 시급한 문제로 넘어갈 수 있을 만큼 불균형을 줄입니다.
제안된 patchset은 scrub `performance tweaks <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-performance-tweaks>`_ series와 `inode scan rebalance <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance>`_ series입니다.
Parallel Inode Scans
--------------------
An XFS filesystem can easily contain hundreds of millions of inodes.
Given that XFS targets installations with large high-performance storage,
it is desirable to scrub inodes in parallel to minimize runtime, particularly
if the program has been invoked manually from a command line.
This requires careful scheduling to keep the threads as evenly loaded as
possible.
Early iterations of the ``xfs_scrub`` inode scanner naïvely created a single
workqueue and scheduled a single workqueue item per AG.
Each workqueue item walked the inode btree (with ``XFS_IOC_INUMBERS``) to find
inode chunks and then called bulkstat (``XFS_IOC_BULKSTAT``) to gather enough
information to construct file handles.
The file handle was then passed to a function to generate scrub items for each
metadata object of each inode.
This simple algorithm leads to thread balancing problems in phase 3 if the
filesystem contains one AG with a few large sparse files and the rest of the
AGs contain many smaller files.
The inode scan dispatch function was not sufficiently granular; it should have
been dispatching at the level of individual inodes, or, to constrain memory
consumption, inode btree records.
Thanks to Dave Chinner, bounded workqueues in userspace enable ``xfs_scrub`` to
avoid this problem with ease by adding a second workqueue.
Just like before, the first workqueue is seeded with one workqueue item per AG,
and it uses INUMBERS to find inode btree chunks.
The second workqueue, however, is configured with an upper bound on the number
of items that can be waiting to be run.
Each inode btree chunk found by the first workqueue's workers are queued to the
second workqueue, and it is this second workqueue that queries BULKSTAT,
creates a file handle, and passes it to a function to generate scrub items for
each metadata object of each inode.
If the second workqueue is too full, the workqueue add function blocks the
first workqueue's workers until the backlog eases.
This doesn't completely solve the balancing problem, but reduces it enough to
move on to more pressing issues.
The proposed patchsets are the scrub
`performance tweaks
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-performance-tweaks>`_
and the
`inode scan rebalance
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance>`_
series.
Repair scheduling
5102-5195Phase 2에서는 AGI header나 inode btree에서 보고된 corruption과 inconsistency를 즉시 repair합니다. Phase 3이 scan할 inode를 찾으려면 inode index가 올바르게 동작해야 하기 때문입니다. 실패한 repair는 phase 4로 다시 scheduling하며, 그 밖의 space metadata에서 보고된 문제도 phase 4로 미룹니다. Optimization 기회는 어디에서 발견되었는지와 관계없이 항상 phase 4로 미룹니다.
Phase 3에서는 phase 2에서 모든 space metadata가 검증되었다면 file metadata 어느 부분에서 보고된 corruption과 inconsistency도 즉시 repair합니다. 실패했거나 즉시 처리할 수 없는 repair는 phase 4로 scheduling합니다.
초기 `xfs_scrub` 설계에서는 repair가 매우 드물 것으로 예상했기 때문에 kernel과 통신하는 `struct xfs_scrub_metadata` object를 repair scheduling의 기본 object로도 사용할 수 있다고 생각했습니다. 그러나 filesystem object 하나에 적용할 수 있는 optimization 수가 늘면서, 해당 object에 가능한 모든 repair를 repair item 하나로 추적하는 편이 memory 효율이 훨씬 좋아졌습니다. 각 repair item은 AG, metadata file, 개별 inode 또는 summary information class처럼 lock할 수 있는 object 하나를 나타냅니다.
Phase 4는 가능한 한 빠르게 많은 repair 작업을 scheduling합니다. 앞서 설명한 :ref:`data dependency <scrubcheck>`가 그대로 적용되므로, `xfs_scrub`은 phase 3에서 scheduling한 작업을 시도하기 전에 phase 2에서 scheduling한 repair를 먼저 완료하려고 해야 합니다. Repair는 다음 반복 절차로 진행됩니다.
단계작업계속 조건·결과
1사용자가 원하는 만큼 CPU를 바쁘게 유지할 worker 수와 workqueue로 repair round 시작병렬 repair round 초기화
1.a.iPhase 2에서 queue한 각 repair item에 대해, 해당 filesystem object에 열거된 모든 항목을 repair하도록 kernel에 요청Phase 2 dependency 우선
1.a.iiKernel이 이 object에 필요한 repair 수를 줄이는 진전을 냈는지 기록Round progress 추적
1.a.iii더는 repair가 필요하지 않으면 object와 연관된 모든 metadata를 재검증성공하면 item을 버리고, 실패하면 추가 repair를 위해 다시 queue
1.bRepair가 하나라도 이루어졌으면 1.a로 돌아가 모든 phase 2 item 재시도Phase 2 item이 수렴할 때까지 반복
1.c.iPhase 3에서 queue한 각 repair item에 대해, 해당 filesystem object에 열거된 모든 항목을 repair하도록 kernel에 요청Phase 2 처리 뒤 phase 3 진행
1.c.iiKernel이 이 object에 필요한 repair 수를 줄이는 진전을 냈는지 기록Round progress 추적
1.c.iii더는 repair가 필요하지 않으면 연관 metadata를 모두 재검증성공하면 item을 버리고, 실패하면 다시 queue
1.dRepair가 하나라도 이루어졌으면 1.c로 돌아가 모든 phase 3 item 재시도Phase 3 item이 수렴할 때까지 반복
21단계에서 어떤 종류든 repair 진전이 있었으면 1단계로 돌아가 새 repair round 시작Phase 2와 3 전체를 다시 순환
3남은 repair item을 모두 한 번 더 직렬 실행마지막 기회이므로 성공하지 못한 repair를 보고
Phase 2 item을 먼저 수렴시키고 phase 3 item을 처리한 뒤, 진전이 있는 동안 전체 round를 반복하는 원문의 중첩 순서를 펼쳐 보입니다.
Phase 5와 7에서 발견한 corruption과 inconsistency는 즉시 repair합니다. Phase 6에서 보고된 손상된 file-data block은 filesystem이 복구할 수 없습니다.
제안된 patchset은 `repair warning improvements <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings>`_, `repair data dependency <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps>`_ 및 `object tracking <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking>`_ refactoring, 그리고 `repair scheduling <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling>`_ improvement series입니다.
.. _scrubrepair:
Scheduling Repairs
------------------
During phase 2, corruptions and inconsistencies reported in any AGI header or
inode btree are repaired immediately, because phase 3 relies on proper
functioning of the inode indices to find inodes to scan.
Failed repairs are rescheduled to phase 4.
Problems reported in any other space metadata are deferred to phase 4.
Optimization opportunities are always deferred to phase 4, no matter their
origin.
During phase 3, corruptions and inconsistencies reported in any part of a
file's metadata are repaired immediately if all space metadata were validated
during phase 2.
Repairs that fail or cannot be repaired immediately are scheduled for phase 4.
In the original design of ``xfs_scrub``, it was thought that repairs would be
so infrequent that the ``struct xfs_scrub_metadata`` objects used to
communicate with the kernel could also be used as the primary object to
schedule repairs.
With recent increases in the number of optimizations possible for a given
filesystem object, it became much more memory-efficient to track all eligible
repairs for a given filesystem object with a single repair item.
Each repair item represents a single lockable object -- AGs, metadata files,
individual inodes, or a class of summary information.
Phase 4 is responsible for scheduling a lot of repair work in as quick a
manner as is practical.
The :ref:`data dependencies <scrubcheck>` outlined earlier still apply, which
means that ``xfs_scrub`` must try to complete the repair work scheduled by
phase 2 before trying repair work scheduled by phase 3.
The repair process is as follows:
1. Start a round of repair with a workqueue and enough workers to keep the CPUs
as busy as the user desires.
a. For each repair item queued by phase 2,
i. Ask the kernel to repair everything listed in the repair item for a
given filesystem object.
ii. Make a note if the kernel made any progress in reducing the number
of repairs needed for this object.
iii. If the object no longer requires repairs, revalidate all metadata
associated with this object.
If the revalidation succeeds, drop the repair item.
If not, requeue the item for more repairs.
b. If any repairs were made, jump back to 1a to retry all the phase 2 items.
c. For each repair item queued by phase 3,
i. Ask the kernel to repair everything listed in the repair item for a
given filesystem object.
ii. Make a note if the kernel made any progress in reducing the number
of repairs needed for this object.
iii. If the object no longer requires repairs, revalidate all metadata
associated with this object.
If the revalidation succeeds, drop the repair item.
If not, requeue the item for more repairs.
d. If any repairs were made, jump back to 1c to retry all the phase 3 items.
2. If step 1 made any repair progress of any kind, jump back to step 1 to start
another round of repair.
3. If there are items left to repair, run them all serially one more time.
Complain if the repairs were not successful, since this is the last chance
to repair anything.
Corruptions and inconsistencies encountered during phases 5 and 7 are repaired
immediately.
Corrupt file data blocks reported by phase 6 cannot be recovered by the
filesystem.
The proposed patchsets are the
`repair warning improvements
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings>`_,
refactoring of the
`repair data dependency
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps>`_
and
`object tracking
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking>`_,
and the
`repair scheduling
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling>`_
improvement series.
혼동 가능한 Unicode sequence 이름 검사
5196-5269`xfs_scrub`이 phase 4가 끝날 때까지 filesystem metadata를 성공적으로 검증하면, phase 5로 넘어가 filesystem에서 의심스러워 보이는 이름을 검사합니다. 검사 대상은 filesystem label, directory entry의 이름, extended attribute의 이름입니다. 대부분의 Unix filesystem과 마찬가지로 XFS가 이름 내용에 부과하는 제약은 매우 적습니다.
이름 영역금지 항목저장 의미
Directory entrySlash와 null byte이름 길이를 ondisk에 명시적으로 저장
Userspace-visible extended attributeNull byteAttribute key의 길이를 ondisk에 명시적으로 저장
Filesystem labelNull byteNull을 허용하지 않음
Directory entry와 attribute keyNull이 name terminator라는 해석명시적인 길이를 사용하므로 null은 이름 종결자가 아님
Directory entry, userspace-visible extended attribute, filesystem label에 적용되는 금지 byte와 ondisk 길이 규칙입니다.
이 절에서 `naming domain`은 이름이 함께 제시되는 모든 장소를 뜻합니다. 예를 들면 directory 안의 모든 이름 또는 file의 모든 attribute가 각각 하나의 naming domain입니다.
Unix의 이름 제약은 매우 관대하지만, 현대 Linux system의 program은 국제 언어를 지원하기 위해 Unicode character code point를 사용합니다. Kernel이 null-terminated name을 기대하므로 이 program들은 C library와 통신할 때 일반적으로 code point를 UTF-8로 encoding합니다. 따라서 보통 XFS filesystem에서 발견되는 이름은 실제로 UTF-8로 encoding된 Unicode data입니다.
범주원문 사례표시 결과·위험
서로 다른 문자 체계의 동형 문자`Cyrillic Small Letter A` U+0430 `а`와 `Latin Small Letter A` U+0061 `a`두 문자가 흔히 동일하게 표시됨
여러 구성 방식`Angstrom Sign` U+212B `Å` 또는 `Latin Capital Letter A` U+0041 `A` 뒤의 `Combining Ring Above` U+030A `◌̊`두 sequence가 동일하게 표시됨
표시 방향 제어`Right-to-Left Override` U+202E가 포함된 `moo\\xe2\\x80\\xaegnp.txt`일부 program이 `mootxt.png`처럼 표시할 수 있음
보이지 않는 공백File name 안의 `Zero Width Space` U+200B해당 문자가 없는 이름과 동일하게 표시됨
서로 다른 byte sequence가 같은 모양으로 rendering되어 사용자를 혼동시킬 수 있는 네 범주를 원문의 code point와 함께 보존합니다.
Naming domain 안의 두 이름이 서로 다른 byte sequence를 갖지만 동일하게 표시된다면 사용자는 혼동할 수 있습니다. 상위 encoding scheme에 관여하지 않는 kernel은 이를 허용하며, 대부분의 filesystem driver는 VFS에서 받은 이름의 byte sequence를 그대로 영속화합니다.
혼동 가능한 이름을 탐지하는 기법은 `Unicode Security Mechanisms <https://unicode.org/reports/tr39/>`_ 문서의 4절과 5절에 자세히 설명되어 있습니다. `xfs_scrub`은 system에서 UTF-8 encoding 사용을 감지하면 Unicode normalization form NFD와 `libicu <https://github.com/unicode-org/icu>`_의 confusable-name detection component를 함께 사용해, directory 안이나 file의 extended attribute 안에서 서로 혼동될 수 있는 이름을 찾습니다. 또한 control character, non-rendering character, bidirectional character 혼합도 검사하며, 이 잠재적 문제를 모두 phase 5에서 system administrator에게 보고합니다.
Checking Names for Confusable Unicode Sequences
-----------------------------------------------
If ``xfs_scrub`` succeeds in validating the filesystem metadata by the end of
phase 4, it moves on to phase 5, which checks for suspicious looking names in
the filesystem.
These names consist of the filesystem label, names in directory entries, and
the names of extended attributes.
Like most Unix filesystems, XFS imposes the sparest of constraints on the
contents of a name:
- Slashes and null bytes are not allowed in directory entries.
- Null bytes are not allowed in userspace-visible extended attributes.
- Null bytes are not allowed in the filesystem label.
Directory entries and attribute keys store the length of the name explicitly
ondisk, which means that nulls are not name terminators.
For this section, the term "naming domain" refers to any place where names are
presented together -- all the names in a directory, or all the attributes of a
file.
Although the Unix naming constraints are very permissive, the reality of most
modern-day Linux systems is that programs work with Unicode character code
points to support international languages.
These programs typically encode those code points in UTF-8 when interfacing
with the C library because the kernel expects null-terminated names.
In the common case, therefore, names found in an XFS filesystem are actually
UTF-8 encoded Unicode data.
To maximize its expressiveness, the Unicode standard defines separate control
points for various characters that render similarly or identically in writing
systems around the world.
For example, the character "Cyrillic Small Letter A" U+0430 "а" often renders
identically to "Latin Small Letter A" U+0061 "a".
The standard also permits characters to be constructed in multiple ways --
either by using a defined code point, or by combining one code point with
various combining marks.
For example, the character "Angstrom Sign U+212B "Å" can also be expressed
as "Latin Capital Letter A" U+0041 "A" followed by "Combining Ring Above"
U+030A "◌̊".
Both sequences render identically.
Like the standards that preceded it, Unicode also defines various control
characters to alter the presentation of text.
For example, the character "Right-to-Left Override" U+202E can trick some
programs into rendering "moo\\xe2\\x80\\xaegnp.txt" as "mootxt.png".
A second category of rendering problems involves whitespace characters.
If the character "Zero Width Space" U+200B is encountered in a file name, the
name will render identically to a name that does not have the zero width
space.
If two names within a naming domain have different byte sequences but render
identically, a user may be confused by it.
The kernel, in its indifference to upper level encoding schemes, permits this.
Most filesystem drivers persist the byte sequence names that are given to them
by the VFS.
Techniques for detecting confusable names are explained in great detail in
sections 4 and 5 of the
`Unicode Security Mechanisms <https://unicode.org/reports/tr39/>`_
document.
When ``xfs_scrub`` detects UTF-8 encoding in use on a system, it uses the
Unicode normalization form NFD in conjunction with the confusable name
detection component of
`libicu <https://github.com/unicode-org/icu>`_
to identify names with a directory or within a file's extended attributes that
could be confused for each other.
Names are also checked for control characters, non-rendering characters, and
mixing of bidirectional characters.
All of these potential issues are reported to the system administrator during
phase 5.File-data extent의 media verification
5270-5293System administrator는 모든 file-data block의 media scan을 시작하도록 선택할 수 있습니다. 이 scan은 summary counter를 제외한 모든 filesystem metadata를 검증한 뒤 phase 6에서 실행됩니다.
단계동작목적·기준
1. Space-map scan`FS_IOC_GETFSMAP`을 호출해 filesystem space map을 scanFile data-fork extent에 할당된 영역 검색
2. 작은 gap 병합Data-fork extent 사이의 64KiB보다 작은 gap을 data-fork extent처럼 취급Command setup overhead 감소
3. Verification request모은 영역이 32MiB보다 커지면 raw block device에 direct-I/O read 전송Disk에 media verification 요청
4. 실패 범위 축소Verification read 실패 시 single-block read로 재시도하고 media의 구체적인 실패 영역 기록손상 위치를 block 단위로 한정
5. 손실 역추적요청을 모두 발행한 뒤 space-mapping ioctl로 기록된 media error를 metadata structure에 다시 mapping손실된 구조를 보고
6. 사용자 친화적 보고File 소유 block의 media error는 parent pointer로 inode number에서 file path 구성관리자에게 영향받은 file을 경로로 표시
Filesystem space map에서 file-data extent를 모아 raw block device를 읽고, 실패 범위를 다시 metadata와 file path로 연결하는 절차입니다.
이 과정은 검증 읽기 자체와 손상 소유자 식별을 분리합니다. 먼저 raw device에서 실제 media failure를 좁힌 다음, 최신 space map과 parent-pointer 정보를 사용해 어떤 metadata 또는 file이 손실되었는지 보고합니다.
Media Verification of File Data Extents
---------------------------------------
The system administrator can elect to initiate a media scan of all file data
blocks.
This scan after validation of all filesystem metadata (except for the summary
counters) as phase 6.
The scan starts by calling ``FS_IOC_GETFSMAP`` to scan the filesystem space map
to find areas that are allocated to file data fork extents.
Gaps between data fork extents that are smaller than 64k are treated as if
they were data fork extents to reduce the command setup overhead.
When the space map scan accumulates a region larger than 32MB, a media
verification request is sent to the disk as a directio read of the raw block
device.
If the verification read fails, ``xfs_scrub`` retries with single-block reads
to narrow down the failure to the specific region of the media and recorded.
When it has finished issuing verification requests, it again uses the space
mapping ioctl to map the recorded media errors back to metadata structures
and report what has been lost.
For media errors in blocks owned by files, parent pointers can be used to
construct file paths from inode numbers for user-friendly reporting.
결론과 향후 작업
5294-5305이 문서의 설계를 따라온 독자가 이제 XFS가 metadata index를 online으로 rebuild하는 방식과 filesystem 사용자가 그 기능과 상호 작용하는 방식을 어느 정도 익혔기를 바랍니다.
작업 범위는 벅찰 만큼 크지만, 이 안내서가 code reader에게 무엇이 구축되었고 누구를 위해 구축되었으며 왜 그렇게 설계되었는지를 더 쉽게 이해하도록 돕기를 바랍니다. 질문이 있으면 XFS mailing list로 문의할 수 있습니다.
7. Conclusion and Future Work
=============================
It is hoped that the reader of this document has followed the designs laid out
in this document and now has some familiarity with how XFS performs online
rebuilding of its metadata indices, and how filesystem users can interact with
that functionality.
Although the scope of this work is daunting, it is hoped that this guide will
make it easier for code readers to understand what has been built, for whom it
has been built, and why.
Please feel free to contact the XFS mailing list with questions.
XFS_IOC_EXCHANGE_RANGE와 일반 사용자 file의 content exchange
5306-5374앞에서 설명했듯이 atomic file-mapping exchange mechanism의 두 번째 frontend는 userspace program이 file update를 atomic하게 commit할 수 있게 하는 새 ioctl입니다. 이 frontend는 여러 해 동안 review를 받아 왔지만, online repair에 필요한 개선과 customer demand 부족 때문에 제안이 강하게 추진되지는 않았습니다.
XFS는 오래전부터 file 사이에서 extent를 swap할 수 있었고, 이 기능은 거의 전적으로 `xfs_fsr`의 file defragmentation에 사용되었습니다. 최초 형태인 fork-swap mechanism은 두 inode fork의 immediate area에 있는 raw byte를 맞바꾸어 data fork 전체 content를 exchange했습니다. Self-describing metadata가 도입된 XFS v5에서는 log recovery 중 BMBT block의 owner field를 계속 rewrite하도록 기존 mechanism에 log 지원이 추가되었습니다.
이후 reverse-mapping btree가 XFS에 추가되자 fork mapping과 reverse-mapping index의 일관성을 유지하려면 deferred bmap 및 rmap operation으로 mapping을 하나씩 swap하는 iterative mechanism이 필요했습니다. 새 tracking item을 제외하면 이 mechanism은 앞 절차의 2~3단계와 동일합니다. Atomic file-mapping exchange는 완전히 새로운 발명이라기보다 기존 mechanism의 발전형입니다. File defragmentation이라는 좁은 사례에서는 두 file의 content가 동일해야 하므로 recovery 보장의 이점도 크지 않습니다.
Atomic file-content exchange는 crash 후에도 caller가 old content와 new content가 섞인 상태를 보지 않도록 보장하고, 임의의 두 file-fork range에 동작할 수 있으므로 기존 swapext 구현보다 훨씬 유연합니다. 이 유연성은 다음 사용 사례를 가능하게 합니다.
사용 사례Userspace 절차Atomic 조건·API
File write의 atomic commitUpdate할 file을 열고 temporary file을 만든 뒤 file-clone operation으로 원본 content를 temporary file에 reflink합니다. 원본에 쓸 update를 temporary file에 기록한 후 content를 exchange합니다.`XFS_IOC_EXCHANGE_RANGE`가 update 전부를 원본에 commit하거나 하나도 commit하지 않음
Transactional file updateReflink 전에 원본 file의 modification timestamp와 change timestamp를 snapshot하고, commit할 때 두 timestamp를 atomic mapping-exchange argument로 kernel에 전달합니다.제공된 timestamp가 원본 file과 일치할 때만 kernel이 commit하며 `XFS_IOC_COMMIT_RANGE`를 사용. :ref:`exchrange_if_unchanged`
Atomic block-device write emulationFilesystem block size와 같은 logical sector size의 block device를 export해 write alignment를 강제하고, 모든 write를 temporary file에 staging한 뒤 mapping exchange를 호출합니다.Temporary file의 hole을 무시하는 flag로 software atomic device write를 흉내 내며 임의의 scattered write를 지원
Temporary file staging과 mapping exchange를 이용하는 세 workflow의 commit 조건과 ioctl·flag를 보존합니다.
XFS_IOC_EXCHANGE_RANGE
----------------------
As discussed earlier, a second frontend to the atomic file mapping exchange
mechanism is a new ioctl call that userspace programs can use to commit updates
to files atomically.
This frontend has been out for review for several years now, though the
necessary refinements to online repair and lack of customer demand mean that
the proposal has not been pushed very hard.
File Content Exchanges with Regular User Files
``````````````````````````````````````````````
As mentioned earlier, XFS has long had the ability to swap extents between
files, which is used almost exclusively by ``xfs_fsr`` to defragment files.
The earliest form of this was the fork swap mechanism, where the entire
contents of data forks could be exchanged between two files by exchanging the
raw bytes in each inode fork's immediate area.
When XFS v5 came along with self-describing metadata, this old mechanism grew
some log support to continue rewriting the owner fields of BMBT blocks during
log recovery.
When the reverse mapping btree was later added to XFS, the only way to maintain
the consistency of the fork mappings with the reverse mapping index was to
develop an iterative mechanism that used deferred bmap and rmap operations to
swap mappings one at a time.
This mechanism is identical to steps 2-3 from the procedure above except for
the new tracking items, because the atomic file mapping exchange mechanism is
an iteration of an existing mechanism and not something totally novel.
For the narrow case of file defragmentation, the file contents must be
identical, so the recovery guarantees are not much of a gain.
Atomic file content exchanges are much more flexible than the existing swapext
implementations because it can guarantee that the caller never sees a mix of
old and new contents even after a crash, and it can operate on two arbitrary
file fork ranges.
The extra flexibility enables several new use cases:
- **Atomic commit of file writes**: A userspace process opens a file that it
wants to update.
Next, it opens a temporary file and calls the file clone operation to reflink
the first file's contents into the temporary file.
Writes to the original file should instead be written to the temporary file.
Finally, the process calls the atomic file mapping exchange system call
(``XFS_IOC_EXCHANGE_RANGE``) to exchange the file contents, thereby
committing all of the updates to the original file, or none of them.
.. _exchrange_if_unchanged:
- **Transactional file updates**: The same mechanism as above, but the caller
only wants the commit to occur if the original file's contents have not
changed.
To make this happen, the calling process snapshots the file modification and
change timestamps of the original file before reflinking its data to the
temporary file.
When the program is ready to commit the changes, it passes the timestamps
into the kernel as arguments to the atomic file mapping exchange system call.
The kernel only commits the changes if the provided timestamps match the
original file.
A new ioctl (``XFS_IOC_COMMIT_RANGE``) is provided to perform this.
- **Emulation of atomic block device writes**: Export a block device with a
logical sector size matching the filesystem block size to force all writes
to be aligned to the filesystem block size.
Stage all writes to a temporary file, and when that is complete, call the
atomic file mapping exchange system call with a flag to indicate that holes
in the temporary file should be ignored.
This emulates an atomic device write in software, and can support arbitrary
scattered writes.
Vectorized scrub
5375-5403앞에서 설명한 repair item :ref:`refactoring <scrubrepair>`은 vectorized scrub system call을 가능하게 한 촉매였습니다. 2018년 이후 일부 system에서는 speculative-execution attack의 영향을 완화하기 위해 kernel call 비용이 크게 증가했습니다. 이에 program 작성자는 execution path가 security boundary를 넘는 횟수를 줄이도록 system call을 가능한 한 적게 사용하게 되었습니다.
단계주체동작
1. Object 지정Userspace검사할 filesystem object의 identity를 kernel에 전달
2. Plan 구성Userspace해당 object에 실행할 scrub type 목록과 선택한 type 사이 data dependency의 단순 표현을 전달
3. Plan 실행KernelCorruption 때문에 만족할 수 없는 dependency를 만날 때까지 caller plan을 가능한 만큼 실행
4. 진전 보고Kernel완료한 작업량을 userspace에 반환
Userspace가 object와 scrub plan을 한 번에 전달하고 kernel이 dependency가 허용하는 만큼 실행한 뒤 진전량을 돌려주는 흐름입니다.
향후 `io_uring`이 이 기능을 충분히 흡수해 online fsck가 XFS 전용 vectored-scrub system call을 별도로 추가하지 않고 이를 사용할 수 있기를 기대합니다.
관련 patchset은 `kernel vectorized scrub <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=vectorized-scrub>`_ 및 `userspace vectorized scrub <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=vectorized-scrub>`_ series입니다.
Vectorized Scrub
----------------
As it turns out, the :ref:`refactoring <scrubrepair>` of repair items mentioned
earlier was a catalyst for enabling a vectorized scrub system call.
Since 2018, the cost of making a kernel call has increased considerably on some
systems to mitigate the effects of speculative execution attacks.
This incentivizes program authors to make as few system calls as possible to
reduce the number of times an execution path crosses a security boundary.
With vectorized scrub, userspace pushes to the kernel the identity of a
filesystem object, a list of scrub types to run against that object, and a
simple representation of the data dependencies between the selected scrub
types.
The kernel executes as much of the caller's plan as it can until it hits a
dependency that cannot be satisfied due to a corruption, and tells userspace
how much was accomplished.
It is hoped that ``io_uring`` will pick up enough of this functionality that
online fsck can use that instead of adding a separate vectored scrub system
call to XFS.
The relevant patchsets are the
`kernel vectorized scrub
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=vectorized-scrub>`_
and
`userspace vectorized scrub
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=vectorized-scrub>`_
series.
Scrub의 quality-of-service 목표
5404-5418Online fsck code의 심각한 단점 하나는 resource lock을 보유한 채 kernel에서 보낼 수 있는 시간이 사실상 무제한이라는 점입니다. Userspace는 process에 fatal signal을 보내 `xfs_scrub`이 적절한 중단 지점에 도달하면 종료하게 할 수 있지만, kernel에 time budget을 제공할 방법은 없습니다.
Scrub codebase에는 fatal signal을 탐지하는 helper가 있으므로 userspace가 scrub 또는 repair operation의 timeout을 지정하고 budget을 초과하면 operation을 abort하게 만드는 작업 자체는 지나치게 어렵지 않을 것입니다.
그러나 대부분의 repair function은 ondisk metadata를 건드리기 시작하면 operation을 깨끗하게 cancel할 수 없다는 특성이 있습니다. 그 지점을 지나면 QoS timeout은 더 이상 유용하지 않습니다.
Quality of Service Targets for Scrub
------------------------------------
One serious shortcoming of the online fsck code is that the amount of time that
it can spend in the kernel holding resource locks is basically unbounded.
Userspace is allowed to send a fatal signal to the process which will cause
``xfs_scrub`` to exit when it reaches a good stopping point, but there's no way
for userspace to provide a time budget to the kernel.
Given that the scrub codebase has helpers to detect fatal signals, it shouldn't
be too much work to allow userspace to specify a timeout for a scrub/repair
operation and abort the operation if it exceeds budget.
However, most repair functions have the property that once they begin to touch
ondisk metadata, the operation cannot be cancelled cleanly, after which a QoS
timeout is no longer useful.
Free space defragmentation
5419-5495여러 해 동안 많은 XFS 사용자는 filesystem 아래 physical storage의 일부를 비워 연속된 free-space chunk로 만드는 program을 요청해 왔습니다. 이 free-space defragmenter를 줄여서 `clearspace`라고 부릅니다.
구성 요소Interface역할
Reverse-mapping index 읽기`FS_IOC_GETFSMAP` ioctlUserspace에서 대상 영역의 metadata와 file-data mapping 검색
Free space를 file에 mapping새 fallocate mode `FALLOC_FL_MAP_FREE_SPACE`영역의 free space를 할당해 `space collector` file에 mapping
Metadata relocation강제 online repair대상 영역의 metadata structure를 다른 위치에 rebuild
Reverse mapping 탐색, free-space 수집, metadata 이동을 각각 담당하는 기존·신규 interface입니다.
Physical storage 영역에서 metadata를 모두 비우기 위해 `clearspace`는 새 fallocate map-freespace call로 그 영역의 모든 free space를 space-collector file에 mapping합니다. 다음으로 `GETFSMAP`을 통해 영역 안의 모든 metadata block을 찾고 해당 data structure에 forced-repair request를 발행합니다. 보통 metadata는 비우는 영역 밖에서 rebuild됩니다. 각 relocation 뒤에는 `map free space` 함수를 다시 호출해 대상 영역에서 새로 해제된 공간을 수집합니다.
File data를 모두 비우려면 FSMAP information으로 관련 file-data block을 찾습니다. 적절한 target을 식별한 뒤 file의 해당 부분에 `FICLONERANGE`를 사용해 physical space를 dummy file과 공유하려고 시도합니다. Extent를 clone하면 원래 owner는 content를 overwrite할 수 없고 변경은 copy-on-write로 다른 곳에 기록됩니다. `clearspace`는 비우지 않는 영역에 frozen extent의 자체 copy를 만든 다음 `FIEDEUPRANGE` 또는 :ref:`atomic file content exchange <exchrange_if_unchanged>`로 target file의 data-extent mapping을 비우는 영역 밖으로 바꿉니다. 다른 mapping을 모두 옮기면 그 공간을 space-collector file에 reflink해 사용할 수 없게 만듭니다.
공유도가 높은 physical storage를 비울 때는 기존 sharing factor를 유지하는 것이 매우 바람직합니다. Operation 뒤의 sharing factor를 최대화하려면 이런 extent를 먼저 옮겨야 합니다. 이를 원활히 수행하려면 reference-count information을 userspace에 보고하는 새 ioctl `FS_IOC_GETREFCOUNTS`가 필요합니다. Refcount가 노출되면 `clearspace`는 filesystem에서 가장 길고 가장 많이 공유된 data extent를 빠르게 찾아 먼저 처리할 수 있습니다.
질문제안된 답남은 문제
Filesystem이 inode chunk를 어떻게 이동할 수 있는가?Dave Chinner의 prototype은 old content로 새 file을 만들고 filesystem을 lock 없이 순회하며 directory entry를 갱신합니다. Inode-remapping table을 jump label 뒤에 숨기고 kernel의 directory-entry update 순회를 추적하는 log item을 둘 수 있습니다.Filesystem이 내려가면 완료할 수 없고 kernel은 open file을 revoke할 수 없음
Static key로 XFS file의 `revoke()` 지원 비용을 줄일 수 있는가?가능합니다. 첫 revocation 전까지 bailout code를 call path에 전혀 넣지 않아도 됩니다.첫 revocation 이후 경로 전환 필요
Inode chunk 이동과 XFS file revoke 지원에 대한 원문의 질문·답변과 남은 제약입니다.
관련 patchset은 `kernel freespace defrag <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace>`_ 및 `userspace freespace defrag <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace>`_ series입니다.
Defragmenting Free Space
------------------------
Over the years, many XFS users have requested the creation of a program to
clear a portion of the physical storage underlying a filesystem so that it
becomes a contiguous chunk of free space.
Call this free space defragmenter ``clearspace`` for short.
The first piece the ``clearspace`` program needs is the ability to read the
reverse mapping index from userspace.
This already exists in the form of the ``FS_IOC_GETFSMAP`` ioctl.
The second piece it needs is a new fallocate mode
(``FALLOC_FL_MAP_FREE_SPACE``) that allocates the free space in a region and
maps it to a file.
Call this file the "space collector" file.
The third piece is the ability to force an online repair.
To clear all the metadata out of a portion of physical storage, clearspace
uses the new fallocate map-freespace call to map any free space in that region
to the space collector file.
Next, clearspace finds all metadata blocks in that region by way of
``GETFSMAP`` and issues forced repair requests on the data structure.
This often results in the metadata being rebuilt somewhere that is not being
cleared.
After each relocation, clearspace calls the "map free space" function again to
collect any newly freed space in the region being cleared.
To clear all the file data out of a portion of the physical storage, clearspace
uses the FSMAP information to find relevant file data blocks.
Having identified a good target, it uses the ``FICLONERANGE`` call on that part
of the file to try to share the physical space with a dummy file.
Cloning the extent means that the original owners cannot overwrite the
contents; any changes will be written somewhere else via copy-on-write.
Clearspace makes its own copy of the frozen extent in an area that is not being
cleared, and uses ``FIEDEUPRANGE`` (or the :ref:`atomic file content exchanges
<exchrange_if_unchanged>` feature) to change the target file's data extent
mapping away from the area being cleared.
When all other mappings have been moved, clearspace reflinks the space into the
space collector file so that it becomes unavailable.
There are further optimizations that could apply to the above algorithm.
To clear a piece of physical storage that has a high sharing factor, it is
strongly desirable to retain this sharing factor.
In fact, these extents should be moved first to maximize sharing factor after
the operation completes.
To make this work smoothly, clearspace needs a new ioctl
(``FS_IOC_GETREFCOUNTS``) to report reference count information to userspace.
With the refcount information exposed, clearspace can quickly find the longest,
most shared data extents in the filesystem, and target them first.
**Future Work Question**: How might the filesystem move inode chunks?
*Answer*: To move inode chunks, Dave Chinner constructed a prototype program
that creates a new file with the old contents and then locklessly runs around
the filesystem updating directory entries.
The operation cannot complete if the filesystem goes down.
That problem isn't totally insurmountable: create an inode remapping table
hidden behind a jump label, and a log item that tracks the kernel walking the
filesystem to update directory entries.
The trouble is, the kernel can't do anything about open files, since it cannot
revoke them.
**Future Work Question**: Can static keys be used to minimize the cost of
supporting ``revoke()`` on XFS files?
*Answer*: Yes.
Until the first revocation, the bailout code need not be in the call path at
all.
The relevant patchsets are the
`kernel freespace defrag
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace>`_
and
`userspace freespace defrag
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace>`_
series.
Filesystem 축소
5496-5503Filesystem 끝부분을 제거하는 일은 끝에 있는 data와 metadata를 대피시키고 해제된 공간을 shrink code에 넘기는 단순한 작업이어야 합니다. 이를 위해서는 filesystem 끝의 공간을 비워야 하며, 바로 free-space defragmentation의 사용 사례입니다.
Shrinking Filesystems
---------------------
Removing the end of the filesystem ought to be a simple matter of evacuating
the data and metadata at the end of the filesystem, and handing the freed space
to the shrink code.
That requires an evacuation of the space at end of the filesystem, which is a
use of free space defragmentation!
요약·해설
xfs-online-fsck-design.rst:1-5503이 문서는 mount된 XFS filesystem을 중단하지 않고 검사하고 복구하기 위한 전체 설계를 설명합니다. Self-describing metadata와 reverse mapping을 기반으로 corruption을 찾고, xfile 계열 임시 저장소와 bulk-loading 기법으로 replacement metadata를 구성한 뒤 transaction과 log intent를 통해 atomic하게 교체합니다.
Kernel 쪽에서는 lock ordering, live-update hook, quota·link-count·parent-pointer 관찰, tempfile content exchange와 orphanage adoption을 다룹니다. Userspace xfs_scrub 쪽에서는 metadata dependency에 따른 phase scheduling, 병렬 inode scan, 반복 repair convergence, Unicode 이름 검사와 media verification을 설명합니다.
문서 전체를 검사 기반, replacement 구축, atomic commit, userspace orchestration의 네 축으로 요약합니다.