Tainted kernels

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

요약과 해설

tainted-kernels.rst:1-191

taint는 즉시 장애를 뜻하지 않고 이후 조사에서 kernel 신뢰성과 재현 조건을 판단하는 이력입니다. 원인 module을 제거해도 flag가 남으므로 bug report에서는 `/proc/sys/kernel/tainted` 값과 log의 letter를 함께 기록하고 가능한 경우 untainted kernel에서 재현해야 합니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기

1 Tainted kernels

2 ---------------

4 The kernel will mark itself as 'tainted' when something occurs that might be

5 relevant later when investigating problems. Don't worry too much about this,

6 most of the time it's not a problem to run a tainted kernel; the information is

7 mainly of interest once someone wants to investigate some problem, as its real

8 cause might be the event that got the kernel tainted. That's why bug reports

9 from tainted kernels will often be ignored by developers, hence try to reproduce

10 problems with an untainted kernel.

12 Note the kernel will remain tainted even after you undo what caused the taint

13 (i.e. unload a proprietary kernel module), to indicate the kernel remains not

14 trustworthy. That's also why the kernel will print the tainted state when it

15 notices an internal problem (a 'kernel bug'), a recoverable error

16 ('kernel oops') or a non-recoverable error ('kernel panic') and writes debug

17 information about this to the logs ``dmesg`` outputs. It's also possible to

18 check the tainted state at runtime through a file in ``/proc/``.

21 Tainted flag in bugs, oops or panics messages

22 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

24 You find the tainted state near the top in a line starting with 'CPU:'; if or

25 why the kernel was tainted is shown after the Process ID ('PID:') and a shortened

26 name of the command ('Comm:') that triggered the event::

28 BUG: unable to handle kernel NULL pointer dereference at 0000000000000000

29 Oops: 0002 [#1] SMP PTI

30 CPU: 0 PID: 4424 Comm: insmod Tainted: P W O 4.20.0-0.rc6.fc30 #1

31 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011

32 RIP: 0010:my_oops_init+0x13/0x1000 [kpanic]

33 [...]

35 You'll find a 'Not tainted: ' there if the kernel was not tainted at the

36 time of the event; if it was, then it will print 'Tainted: ' and characters

37 either letters or blanks. In the example above it looks like this::

39 Tainted: P W O

41 The meaning of those characters is explained in the table below. In this case

42 the kernel got tainted earlier because a proprietary Module (``P``) was loaded,

43 a warning occurred (``W``), and an externally-built module was loaded (``O``).

44 To decode other letters use the table below.

47 Decoding tainted state at runtime

48 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

50 At runtime, you can query the tainted state by reading

51 ``cat /proc/sys/kernel/tainted``. If that returns ``0``, the kernel is not

52 tainted; any other number indicates the reasons why it is. The easiest way to

53 decode that number is the script ``tools/debugging/kernel-chktaint``, which your

54 distribution might ship as part of a package called ``linux-tools`` or

55 ``kernel-tools``; if it doesn't, you can download the script from

`git.kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/debugging/kernel-chktaint>`_

57 and execute it with ``sh kernel-chktaint``, which would print something like

58 this on the machine that had the statements in the logs that were quoted earlier::

60 Kernel is Tainted for following reasons:

61 * Proprietary module was loaded (#0)

62 * Kernel issued warning (#9)

63 * Externally-built ('out-of-tree') module was loaded (#12)

64 See Documentation/admin-guide/tainted-kernels.rst in the Linux kernel or

65 https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html for

66 a more details explanation of the various taint flags.

67 Raw taint value as int/string: 4609/'P W O '

69 You can try to decode the number yourself. That's easy if there was only one

70 reason that got your kernel tainted, as in this case you can find the number

71 with the table below. If there were multiple reasons you need to decode the

72 number, as it is a bitfield, where each bit indicates the absence or presence of

73 a particular type of taint. It's best to leave that to the aforementioned

74 script, but if you need something quick you can use this shell command to check

75 which bits are set::

77 $ for i in $(seq 18); do echo $(($i-1)) $(($(cat /proc/sys/kernel/tainted)>>($i-1)&1));done

79 Table for decoding tainted state

80 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

82 === === ====== ========================================================

83 Bit Log Number Reason that got the kernel tainted

84 === === ====== ========================================================

85 0 G/P 1 proprietary module was loaded

86 1 _/F 2 module was force loaded

87 2 _/S 4 kernel running on an out of specification system

88 3 _/R 8 module was force unloaded

89 4 _/M 16 processor reported a Machine Check Exception (MCE)

90 5 _/B 32 bad page referenced or some unexpected page flags

91 6 _/U 64 taint requested by userspace application

92 7 _/D 128 kernel died recently, i.e. there was an OOPS or BUG

93 8 _/A 256 ACPI table overridden by user

94 9 _/W 512 kernel issued warning

95 10 _/C 1024 staging driver was loaded

96 11 _/I 2048 workaround for bug in platform firmware applied

97 12 _/O 4096 externally-built ("out-of-tree") module was loaded

98 13 _/E 8192 unsigned module was loaded

99 14 _/L 16384 soft lockup occurred

100 15 _/K 32768 kernel has been live patched

101 16 _/X 65536 auxiliary taint, defined for and used by distros

102 17 _/T 131072 kernel was built with the struct randomization plugin

103 18 _/N 262144 an in-kernel test has been run

104 19 _/J 524288 userspace used a mutating debug operation in fwctl

105 === === ====== ========================================================

106

107 Note: The character ``_`` is representing a blank in this table to make reading

108 easier.

109

110 More detailed explanation for tainting

111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

112

113 0) ``G`` if all modules loaded have a GPL or compatible license, ``P`` if

114 any proprietary module has been loaded. Modules without a

115 MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by

116 insmod as GPL compatible are assumed to be proprietary.

117

118 1) ``F`` if any module was force loaded by ``insmod -f``, ``' '`` if all

119 modules were loaded normally.

120

121 2) ``S`` if the kernel is running on a processor or system that is out of

122 specification: hardware has been put into an unsupported configuration,

123 therefore proper execution cannot be guaranteed.

124 Kernel will be tainted if, for example:

125

126 - on x86: PAE is forced through forcepae on intel CPUs (such as Pentium M)

127 which do not report PAE but may have a functional implementation, an SMP

128 kernel is running on non officially capable SMP Athlon CPUs, MSRs are

129 being poked at from userspace.

130 - on arm: kernel running on certain CPUs (such as Keystone 2) without

131 having certain kernel features enabled.

132 - on arm64: there are mismatched hardware features between CPUs, the

133 bootloader has booted CPUs in different modes.

134 - certain drivers are being used on non supported architectures (such as

135 scsi/snic on something else than x86_64, scsi/ips on non

136 x86/x86_64/itanium, have broken firmware settings for the

137 irqchip/irq-gic on arm64 ...).

138 - x86/x86_64: Microcode late loading is dangerous and will result in

139 tainting the kernel. It requires that all CPUs rendezvous to make sure

140 the update happens when the system is as quiescent as possible. However,

141 a higher priority MCE/SMI/NMI can move control flow away from that

142 rendezvous and interrupt the update, which can be detrimental to the

143 machine.

144

145 3) ``R`` if a module was force unloaded by ``rmmod -f``, ``' '`` if all

146 modules were unloaded normally.

147

148 4) ``M`` if any processor has reported a Machine Check Exception,

149 ``' '`` if no Machine Check Exceptions have occurred.

150

151 5) ``B`` If a page-release function has found a bad page reference or some

152 unexpected page flags. This indicates a hardware problem or a kernel bug;

153 there should be other information in the log indicating why this tainting

154 occurred.

155

156 6) ``U`` if a user or user application specifically requested that the

157 Tainted flag be set, ``' '`` otherwise.

158

159 7) ``D`` if the kernel has died recently, i.e. there was an OOPS or BUG.

160

161 8) ``A`` if an ACPI table has been overridden.

162

163 9) ``W`` if a warning has previously been issued by the kernel.

164 (Though some warnings may set more specific taint flags.)

165

166 10) ``C`` if a staging driver has been loaded.

167

168 11) ``I`` if the kernel is working around a severe bug in the platform

169 firmware (BIOS or similar).

170

171 12) ``O`` if an externally-built ("out-of-tree") module has been loaded.

172

173 13) ``E`` if an unsigned module has been loaded in a kernel supporting

174 module signature.

175

176 14) ``L`` if a soft lockup has previously occurred on the system.

177

178 15) ``K`` if the kernel has been live patched.

179

180 16) ``X`` Auxiliary taint, defined for and used by Linux distributors.

181

182 17) ``T`` Kernel was build with the randstruct plugin, which can intentionally

183 produce extremely unusual kernel structure layouts (even performance

184 pathological ones), which is important to know when debugging. Set at

185 build time.

186

187 18) ``N`` if an in-kernel test, such as a KUnit test, has been run.

188

189 19) ``J`` if userpace opened /dev/fwctl/* and performed a FWTCL_RPC_DEBUG_WRITE

190 to use the devices debugging features. Device debugging features could

191 cause the device to malfunction in undefined ways.

192

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

1-20

나중의 문제 조사와 관련될 수 있는 사건이 생기면 kernel은 자신을 `tainted` 상태로 표시합니다. 대부분 tainted kernel을 실행하는 것 자체는 문제가 아니지만, 실제 문제 원인이 taint를 만든 사건일 수 있어 조사할 때 중요합니다. 개발자는 tainted kernel의 bug report를 무시하는 경우가 많으므로 untainted kernel에서 재현하는 것이 좋습니다.

proprietary module을 unload하는 등 원인을 되돌려도 kernel은 신뢰할 수 없는 상태가 지속됨을 나타내기 위해 taint를 유지합니다. kernel bug, recoverable kernel oops, non-recoverable kernel panic 때 `dmesg`가 출력하는 log에 tainted state를 표시하며 runtime에는 `/proc/` file에서도 확인할 수 있습니다.

Tainted flag in bugs, oops or panics messages

21-46

tainted state는 message 상단의 `CPU:`로 시작하는 줄에서 찾습니다. 사건을 trigger한 Process ID `PID:`와 command 축약 이름 `Comm:` 뒤에 표시됩니다.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
Oops: 0002 [#1] SMP PTI
CPU: 0 PID: 4424 Comm: insmod Tainted: P        W  O      4.20.0-0.rc6.fc30 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:my_oops_init+0x13/0x1000 [kpanic]
[...]

untainted면 `Not tainted:`가, tainted면 `Tainted:` 뒤에 letter 또는 blank가 나옵니다. 위 예의 축약 표시는 다음과 같습니다.

Tainted: P        W  O

이 예에서는 proprietary Module 때문에 `P`, warning 때문에 `W`, externally-built module 때문에 `O`가 설정됐습니다.

Decoding tainted state at runtime

47-78

runtime에는 `cat /proc/sys/kernel/tainted`로 조회합니다. 결과가 `0`이면 untainted이고 다른 숫자는 하나 이상의 원인을 뜻합니다.

가장 쉬운 해석 방법은 `tools/debugging/kernel-chktaint` script입니다. distribution의 `linux-tools` 또는 `kernel-tools` package에 포함될 수 있으며, 없다면 git.kernel.org에서 내려받아 `sh kernel-chktaint`로 실행할 수 있습니다.

Kernel is Tainted for following reasons:
 * Proprietary module was loaded (#0)
 * Kernel issued warning (#9)
 * Externally-built ('out-of-tree') module was loaded  (#12)
See Documentation/admin-guide/tainted-kernels.rst in the Linux kernel or
 https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html for
 a more details explanation of the various taint flags.
Raw taint value as int/string: 4609/'P        W  O     '

원인이 하나면 아래 표의 number로 직접 찾을 수 있지만 여러 원인은 각 bit가 taint 유형의 존재 여부를 나타내는 bitfield이므로 script 사용이 좋습니다. 빠르게 set bit를 확인하려면 다음 shell command를 사용할 수 있습니다.

$ for i in $(seq 18); do echo $(($i-1)) $(($(cat /proc/sys/kernel/tainted)>>($i-1)&1));done

Table for decoding tainted state

79-109

Bit	Log	Number	kernel taint 원인
0	G/P	1	proprietary module load
1	_/F	2	module force load
2	_/S	4	specification을 벗어난 system에서 kernel 실행
3	_/R	8	module force unload
4	_/M	16	processor가 Machine Check Exception(MCE) 보고
5	_/B	32	bad page reference 또는 예상하지 못한 page flag
6	_/U	64	userspace application이 taint 요청
7	_/D	128	최근 OOPS 또는 BUG로 kernel die
8	_/A	256	사용자가 ACPI table override
9	_/W	512	kernel warning
10	_/C	1024	staging driver load
11	_/I	2048	platform firmware bug workaround 적용
12	_/O	4096	externally-built out-of-tree module load
13	_/E	8192	unsigned module load
14	_/L	16384	soft lockup 발생
15	_/K	32768	kernel live patch 적용
16	_/X	65536	distribution이 정의·사용하는 auxiliary taint
17	_/T	131072	struct randomization plugin으로 kernel build
18	_/N	262144	in-kernel test 실행
19	_/J	524288	userspace가 fwctl mutating debug operation 사용

표에서 `_` 문자는 읽기 쉽게 blank를 나타냅니다.

More detailed explanation for tainting

110-191

Bit	표시	상세 의미
0	`G`/`P`	모든 module이 GPL compatible이면 `G`; proprietary module이 하나라도 있으면 `P`. `MODULE_LICENSE`가 없거나 insmod가 GPL compatible로 인식하지 못하면 proprietary로 간주
1	`F`/`' '`	`insmod -f`로 force load된 module이 있으면 `F`, 모두 정상 load면 blank
2	`S`	processor/system이 specification 밖의 unsupported configuration에서 실행되어 정상 동작을 보장할 수 없음
3	`R`/`' '`	`rmmod -f`로 force unload한 module이 있으면 `R`, 모두 정상 unload면 blank
4	`M`/`' '`	processor가 Machine Check Exception을 보고했으면 `M`
5	`B`	page-release function이 bad page reference 또는 예상하지 못한 page flag 발견. hardware 문제나 kernel bug를 뜻하며 log에 추가 정보가 있어야 함
6	`U`/`' '`	user 또는 application이 Tainted flag 설정을 명시적으로 요청
7	`D`	최근 OOPS 또는 BUG로 kernel die
8	`A`	ACPI table override
9	`W`	kernel warning 발생. 일부 warning은 더 구체적인 flag도 설정
10	`C`	staging driver load
11	`I`	BIOS 같은 platform firmware의 심각한 bug를 kernel이 우회
12	`O`	externally-built out-of-tree module load
13	`E`	module signature를 지원하는 kernel에 unsigned module load
14	`L`	soft lockup 발생
15	`K`	kernel live patch
16	`X`	Linux distributor가 정의·사용하는 auxiliary taint
17	`T`	비정상적이거나 성능에 병적인 layout도 의도적으로 만들 수 있는 randstruct plugin으로 build. build time에 설정
18	`N`	KUnit 같은 in-kernel test 실행
19	`J`	userspace가 `/dev/fwctl/*`를 열고 `FWTCL_RPC_DEBUG_WRITE`로 device debug 기능 사용. device가 정의되지 않은 방식으로 malfunction할 수 있음

`S`의 예로는 x86에서 Intel Pentium M에 `forcepae`, 비공식 SMP Athlon에 SMP kernel, userspace MSR 조작이 있습니다. arm의 Keystone 2 같은 특정 CPU에서 필요한 kernel 기능 누락, arm64 CPU 사이 hardware feature 불일치나 서로 다른 boot mode도 포함합니다.

지원하지 않는 architecture에서 특정 driver 사용, 예를 들어 x86_64 외의 `scsi/snic`, x86/x86_64/itanium 외의 `scsi/ips`, arm64 `irqchip/irq-gic`의 잘못된 firmware 설정도 해당합니다.

x86/x86_64의 Microcode late loading도 위험하여 taint를 설정합니다. update 때 모든 CPU가 가능한 한 quiescent하게 rendezvous해야 하지만 더 높은 priority의 MCE/SMI/NMI가 control flow를 옮겨 update를 방해하면 machine에 해로울 수 있습니다.