요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
.. [see the bottom of this file for redistribution information]
Reporting regressions
+++++++++++++++++++++
"*We don't cause regressions*" is the first rule of Linux kernel development;
Linux founder and lead developer Linus Torvalds established it himself and
ensures it's obeyed.
This document describes what the rule means for users and how the Linux kernel's
development model ensures to address all reported regressions; aspects relevant
for kernel developers are left to Documentation/process/handling-regressions.rst.
The important bits (aka "TL;DR")
================================
#. It's a regression if something running fine with one Linux kernel works worse
or not at all with a newer version. Note, the newer kernel has to be compiled
using a similar configuration; the detailed explanations below describes this
and other fine print in more detail.
#. Report your issue as outlined in Documentation/admin-guide/reporting-issues.rst,
it already covers all aspects important for regressions and repeated
below for convenience. Two of them are important: start your report's subject
with "[REGRESSION]" and CC or forward it to `the regression mailing list
<https://lore.kernel.org/regressions/>`_ (regressions@lists.linux.dev).
#. Optional, but recommended: when sending or forwarding your report, make the
Linux kernel regression tracking bot "regzbot" track the issue by specifying
when the regression started like this::
#regzbot introduced: v5.13..v5.14-rc1
All the details on Linux kernel regressions relevant for users
==============================================================
The important basics
--------------------
What is a "regression" and what is the "no regressions" rule?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It's a regression if some application or practical use case running fine with
one Linux kernel works worse or not at all with a newer version compiled using a
similar configuration. The "no regressions" rule forbids this to take place; if
it happens by accident, developers that caused it are expected to quickly fix
the issue.
It thus is a regression when a WiFi driver from Linux 5.13 works fine, but with
5.14 doesn't work at all, works significantly slower, or misbehaves somehow.
It's also a regression if a perfectly working application suddenly shows erratic
behavior with a newer kernel version; such issues can be caused by changes in
procfs, sysfs, or one of the many other interfaces Linux provides to userland
software. But keep in mind, as mentioned earlier: 5.14 in this example needs to
be built from a configuration similar to the one from 5.13. This can be achieved
using ``make olddefconfig``, as explained in more detail below.
Note the "practical use case" in the first sentence of this section: developers
despite the "no regressions" rule are free to change any aspect of the kernel
and even APIs or ABIs to userland, as long as no existing application or use
case breaks.
Also be aware the "no regressions" rule covers only interfaces the kernel
provides to the userland. It thus does not apply to kernel-internal interfaces
like the module API, which some externally developed drivers use to hook into
the kernel.
How do I report a regression?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Just report the issue as outlined in
Documentation/admin-guide/reporting-issues.rst, it already describes the
important points. The following aspects outlined there are especially relevant
for regressions:
* When checking for existing reports to join, also search the `archives of the
Linux regressions mailing list <https://lore.kernel.org/regressions/>`_ and
`regzbot's web-interface <https://linux-regtracking.leemhuis.info/regzbot/>`_.
* Start your report's subject with "[REGRESSION]".
* In your report, clearly mention the last kernel version that worked fine and
the first broken one. Ideally try to find the exact change causing the
regression using a bisection, as explained below in more detail.
* Remember to let the Linux regressions mailing list
(regressions@lists.linux.dev) know about your report:
* If you report the regression by mail, CC the regressions list.
* If you report your regression to some bug tracker, forward the submitted
report by mail to the regressions list while CCing the maintainer and the
mailing list for the subsystem in question.
If it's a regression within a stable or longterm series (e.g.
v5.15.3..v5.15.5), remember to CC the `Linux stable mailing list
<https://lore.kernel.org/stable/>`_ (stable@vger.kernel.org).
In case you performed a successful bisection, add everyone to the CC the
culprit's commit message mentions in lines starting with "Signed-off-by:".
When CCing for forwarding your report to the list, consider directly telling the
aforementioned Linux kernel regression tracking bot about your report. To do
that, include a paragraph like this in your mail::
#regzbot introduced: v5.13..v5.14-rc1
Regzbot will then consider your mail a report for a regression introduced in the
specified version range. In above case Linux v5.13 still worked fine and Linux
v5.14-rc1 was the first version where you encountered the issue. If you
performed a bisection to find the commit that caused the regression, specify the
culprit's commit-id instead::
#regzbot introduced: 1f2e3d4c5d
Placing such a "regzbot command" is in your interest, as it will ensure the
report won't fall through the cracks unnoticed. If you omit this, the Linux
kernel's regressions tracker will take care of telling regzbot about your
regression, as long as you send a copy to the regressions mailing lists. But the
regression tracker is just one human which sometimes has to rest or occasionally
might even enjoy some time away from computers (as crazy as that might sound).
Relying on this person thus will result in an unnecessary delay before the
regressions becomes mentioned `on the list of tracked and unresolved Linux
kernel regressions <https://linux-regtracking.leemhuis.info/regzbot/>`_ and the
weekly regression reports sent by regzbot. Such delays can result in Linus
Torvalds being unaware of important regressions when deciding between "continue
development or call this finished and release the final?".
Are really all regressions fixed?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Nearly all of them are, as long as the change causing the regression (the
"culprit commit") is reliably identified. Some regressions can be fixed without
this, but often it's required.
Who needs to find the root cause of a regression?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Developers of the affected code area should try to locate the culprit on their
own. But for them that's often impossible to do with reasonable effort, as quite
a lot of issues only occur in a particular environment outside the developer's
reach -- for example, a specific hardware platform, firmware, Linux distro,
system's configuration, or application. That's why in the end it's often up to
the reporter to locate the culprit commit; sometimes users might even need to
run additional tests afterwards to pinpoint the exact root cause. Developers
should offer advice and reasonably help where they can, to make this process
relatively easy and achievable for typical users.
How can I find the culprit?
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform a bisection, as roughly outlined in
Documentation/admin-guide/reporting-issues.rst and described in more detail by
Documentation/admin-guide/bug-bisect.rst. It might sound like a lot of work, but
in many cases finds the culprit relatively quickly. If it's hard or
time-consuming to reliably reproduce the issue, consider teaming up with other
affected users to narrow down the search range together.
Who can I ask for advice when it comes to regressions?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Send a mail to the regressions mailing list (regressions@lists.linux.dev) while
CCing the Linux kernel's regression tracker (regressions@leemhuis.info); if the
issue might better be dealt with in private, feel free to omit the list.
Additional details about regressions
------------------------------------
What is the goal of the "no regressions" rule?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Users should feel safe when updating kernel versions and not have to worry
something might break. This is in the interest of the kernel developers to make
updating attractive: they don't want users to stay on stable or longterm Linux
series that are either abandoned or more than one and a half years old. That's
in everybody's interest, as `those series might have known bugs, security
issues, or other problematic aspects already fixed in later versions
<http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/>`_.
Additionally, the kernel developers want to make it simple and appealing for
users to test the latest pre-release or regular release. That's also in
everybody's interest, as it's a lot easier to track down and fix problems, if
they are reported shortly after being introduced.
Is the "no regressions" rule really adhered in practice?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It's taken really seriously, as can be seen by many mailing list posts from
Linux creator and lead developer Linus Torvalds, some of which are quoted in
Documentation/process/handling-regressions.rst.
Exceptions to this rule are extremely rare; in the past developers almost always
turned out to be wrong when they assumed a particular situation was warranting
an exception.
Who ensures the "no regressions" rule is actually followed?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The subsystem maintainers should take care of that, which are watched and
supported by the tree maintainers -- e.g. Linus Torvalds for mainline and
Greg Kroah-Hartman et al. for various stable/longterm series.
All of them are helped by people trying to ensure no regression report falls
through the cracks. One of them is Thorsten Leemhuis, who's currently acting as
the Linux kernel's "regressions tracker"; to facilitate this work he relies on
regzbot, the Linux kernel regression tracking bot. That's why you want to bring
your report on the radar of these people by CCing or forwarding each report to
the regressions mailing list, ideally with a "regzbot command" in your mail to
get it tracked immediately.
How quickly are regressions normally fixed?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Developers should fix any reported regression as quickly as possible, to provide
affected users with a solution in a timely manner and prevent more users from
running into the issue; nevertheless developers need to take enough time and
care to ensure regression fixes do not cause additional damage.
The answer thus depends on various factors like the impact of a regression, its
age, or the Linux series in which it occurs. In the end though, most regressions
should be fixed within two weeks.
Is it a regression, if the issue can be avoided by updating some software?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Almost always: yes. If a developer tells you otherwise, ask the regression
tracker for advice as outlined above.
Is it a regression, if a newer kernel works slower or consumes more energy?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, but the difference has to be significant. A five percent slow-down in a
micro-benchmark thus is unlikely to qualify as regression, unless it also
influences the results of a broad benchmark by more than one percent. If in
doubt, ask for advice.
Is it a regression, if an external kernel module breaks when updating Linux?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
No, as the "no regression" rule is about interfaces and services the Linux
kernel provides to the userland. It thus does not cover building or running
externally developed kernel modules, as they run in kernel-space and hook into
the kernel using internal interfaces occasionally changed.
How are regressions handled that are caused by security fixes?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In extremely rare situations security issues can't be fixed without causing
regressions; those fixes are given way, as they are the lesser evil in the end.
Luckily this middling almost always can be avoided, as key developers for the
affected area and often Linus Torvalds himself try very hard to fix security
issues without causing regressions.
If you nevertheless face such a case, check the mailing list archives if people
tried their best to avoid the regression. If not, report it; if in doubt, ask
for advice as outlined above.
What happens if fixing a regression is impossible without causing another?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sadly these things happen, but luckily not very often; if they occur, expert
developers of the affected code area should look into the issue to find a fix
that avoids regressions or at least their impact. If you run into such a
situation, do what was outlined already for regressions caused by security
fixes: check earlier discussions if people already tried their best and ask for
advice if in doubt.
A quick note while at it: these situations could be avoided, if people would
regularly give mainline pre-releases (say v5.15-rc1 or -rc3) from each
development cycle a test run. This is best explained by imagining a change
integrated between Linux v5.14 and v5.15-rc1 which causes a regression, but at
the same time is a hard requirement for some other improvement applied for
5.15-rc1. All these changes often can simply be reverted and the regression thus
solved, if someone finds and reports it before 5.15 is released. A few days or
weeks later this solution can become impossible, as some software might have
started to rely on aspects introduced by one of the follow-up changes: reverting
all changes would then cause a regression for users of said software and thus is
out of the question.
Is it a regression, if some feature I relied on was removed months ago?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is, but often it's hard to fix such regressions due to the aspects outlined
in the previous section. It hence needs to be dealt with on a case-by-case
basis. This is another reason why it's in everybody's interest to regularly test
mainline pre-releases.
Does the "no regression" rule apply if I seem to be the only affected person?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It does, but only for practical usage: the Linux developers want to be free to
remove support for hardware only to be found in attics and museums anymore.
Note, sometimes regressions can't be avoided to make progress -- and the latter
is needed to prevent Linux from stagnation. Hence, if only very few users seem
to be affected by a regression, it for the greater good might be in their and
everyone else's interest to lettings things pass. Especially if there is an
easy way to circumvent the regression somehow, for example by updating some
software or using a kernel parameter created just for this purpose.
Does the regression rule apply for code in the staging tree as well?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Not according to the `help text for the configuration option covering all
staging code <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/Kconfig>`_,
which since its early days states::
Please note that these drivers are under heavy development, may or
may not work, and may contain userspace interfaces that most likely
will be changed in the near future.
The staging developers nevertheless often adhere to the "no regressions" rule,
but sometimes bend it to make progress. That's for example why some users had to
deal with (often negligible) regressions when a WiFi driver from the staging
tree was replaced by a totally different one written from scratch.
Why do later versions have to be "compiled with a similar configuration"?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Because the Linux kernel developers sometimes integrate changes known to cause
regressions, but make them optional and disable them in the kernel's default
configuration. This trick allows progress, as the "no regressions" rule
otherwise would lead to stagnation.
Consider for example a new security feature blocking access to some kernel
interfaces often abused by malware, which at the same time are required to run a
few rarely used applications. The outlined approach makes both camps happy:
people using these applications can leave the new security feature off, while
everyone else can enable it without running into trouble.
How to create a configuration similar to the one of an older kernel?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Start your machine with a known-good kernel and configure the newer Linux
version with ``make olddefconfig``. This makes the kernel's build scripts pick
up the configuration file (the ".config" file) from the running kernel as base
for the new one you are about to compile; afterwards they set all new
configuration options to their default value, which should disable new features
that might cause regressions.
Can I report a regression I found with pre-compiled vanilla kernels?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You need to ensure the newer kernel was compiled with a similar configuration
file as the older one (see above), as those that built them might have enabled
some known-to-be incompatible feature for the newer kernel. If in doubt, report
the matter to the kernel's provider and ask for advice.
More about regression tracking with "regzbot"
---------------------------------------------
What is regression tracking and why should I care about it?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Rules like "no regressions" need someone to ensure they are followed, otherwise
they are broken either accidentally or on purpose. History has shown this to be
true for Linux kernel development as well. That's why Thorsten Leemhuis, the
Linux Kernel's regression tracker, and some people try to ensure all regression
are fixed by keeping an eye on them until they are resolved. Neither of them are
paid for this, that's why the work is done on a best effort basis.
Why and how are Linux kernel regressions tracked using a bot?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tracking regressions completely manually has proven to be quite hard due to the
distributed and loosely structured nature of Linux kernel development process.
That's why the Linux kernel's regression tracker developed regzbot to facilitate
the work, with the long term goal to automate regression tracking as much as
possible for everyone involved.
Regzbot works by watching for replies to reports of tracked regressions.
Additionally, it's looking out for posted or committed patches referencing such
reports with "Link:" tags; replies to such patch postings are tracked as well.
Combined this data provides good insights into the current state of the fixing
process.
How to see which regressions regzbot tracks currently?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Check out `regzbot's web-interface <https://linux-regtracking.leemhuis.info/regzbot/>`_.
What kind of issues are supposed to be tracked by regzbot?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The bot is meant to track regressions, hence please don't involve regzbot for
regular issues. But it's okay for the Linux kernel's regression tracker if you
involve regzbot to track severe issues, like reports about hangs, corrupted
data, or internal errors (Panic, Oops, BUG(), warning, ...).
How to change aspects of a tracked regression?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By using a 'regzbot command' in a direct or indirect reply to the mail with the
report. The easiest way to do that: find the report in your "Sent" folder or the
mailing list archive and reply to it using your mailer's "Reply-all" function.
In that mail, use one of the following commands in a stand-alone paragraph (IOW:
use blank lines to separate one or multiple of these commands from the rest of
the mail's text).
* Update when the regression started to happen, for example after performing a
bisection::
#regzbot introduced: 1f2e3d4c5d
* Set or update the title::
#regzbot title: foo
* Monitor a discussion or bugzilla.kernel.org ticket where additions aspects of
the issue or a fix are discussed:::
#regzbot monitor: https://lore.kernel.org/r/30th.anniversary.repost@klaava.Helsinki.FI/
#regzbot monitor: https://bugzilla.kernel.org/show_bug.cgi?id=123456789
* Point to a place with further details of interest, like a mailing list post
or a ticket in a bug tracker that are slightly related, but about a different
topic::
#regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=123456789
* Mark a regression as invalid::
#regzbot invalid: wasn't a regression, problem has always existed
Regzbot supports a few other commands primarily used by developers or people
tracking regressions. They and more details about the aforementioned regzbot
commands can be found in the `getting started guide
<https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md>`_ and
the `reference documentation <https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md>`_
for regzbot.
..
end-of-content
..
This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
of the file. If you want to distribute this text under CC-BY-4.0 only,
please use "The Linux kernel developers" for author attribution and link
this as source:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-regressions.rst
..
Note: Only the content of this RST file as found in the Linux kernel sources
is available under CC-BY-4.0, as versions of this text that were processed
(for example by the kernel's build system) might contain content taken from
files which use a more restrictive license.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
핵심 사항(TL;DR)
1-36이 문서는 `(GPL-2.0+ OR CC-BY-4.0)` 조건으로 제공됩니다. `We don't cause regressions`는 Linux kernel 개발의 첫 번째 규칙이며 Linux 창립자이자 lead developer인 Linus Torvalds가 직접 세우고 준수를 확인합니다.
이 규칙이 사용자에게 무엇을 의미하고 Linux kernel 개발 model이 보고된 regression을 어떻게 다루는지 설명합니다. Kernel developer에게 필요한 내용은 `Documentation/process/handling-regressions.rst`에 있습니다.
어떤 Linux kernel에서 정상 동작하던 것이 더 새 버전에서 나빠지거나 전혀 동작하지 않으면 regression입니다. 새 kernel은 이전 것과 유사한 configuration으로 compile해야 하며 세부 조건은 뒤에서 설명합니다.
`Documentation/admin-guide/reporting-issues.rst`의 절차로 보고합니다. 특히 subject를 `[REGRESSION]`으로 시작하고 regression mailing list `https://lore.kernel.org/regressions/`의 `regressions@lists.linux.dev`를 CC하거나 보고서를 그곳으로 전달해야 합니다.
선택 사항이지만 권장하는 방법은 report를 보내거나 forward할 때 Linux kernel regression tracking bot인 `regzbot`에 regression이 시작된 범위를 다음과 같이 알려 tracking하게 하는 것입니다.
#regzbot introduced: v5.13..v5.14-rc1Regression 정의, 보고, culprit와 도움
37-171이전 Linux에서 잘 동작하던 application 또는 실제 사용 사례가 유사한 configuration으로 compile한 새 version에서 나빠지거나 동작하지 않으면 regression입니다. `no regressions` 규칙은 이를 금지하며 실수로 발생하면 원인 변경을 만든 개발자가 빠르게 고쳐야 합니다.
Linux 5.13의 WiFi driver가 정상인데 5.14에서 전혀 동작하지 않거나 크게 느려지거나 잘못 동작하면 regression입니다. 정상 application이 새 kernel에서 이상해지는 것도 regression이며 procfs, sysfs 등 userland interface 변경이 원인일 수 있습니다.
새 kernel은 이전 configuration과 유사해야 합니다. 아래에서 자세히 설명하는 `make olddefconfig`로 이를 만들 수 있습니다.
규칙의 핵심은 `practical use case`입니다. 기존 application이나 실제 사용을 깨뜨리지 않는 한 개발자는 kernel의 어떤 부분과 userland API·ABI도 바꿀 수 있습니다. 이 규칙은 kernel이 userland에 제공하는 interface에만 적용되며 외부 driver가 사용하는 module API 같은 kernel 내부 interface에는 적용되지 않습니다.
보고는 `Documentation/admin-guide/reporting-issues.rst`를 따릅니다. 기존 보고를 찾을 때 Linux regressions list archive와 regzbot web interface `https://linux-regtracking.leemhuis.info/regzbot/`도 검색합니다.
Subject를 `[REGRESSION]`으로 시작하고 마지막 정상 kernel과 첫 broken kernel을 명확히 적습니다. 가능하면 bisection으로 정확한 원인 변경을 찾습니다.
Mail 보고에는 `regressions@lists.linux.dev`를 CC합니다. Bug tracker에 보고했다면 subsystem maintainer와 mailing list를 CC하면서 submitted report를 regression list로 mail forward합니다.
v5.15.3에서 v5.15.5처럼 stable 또는 longterm series 내부 regression이면 Linux stable list `stable@vger.kernel.org`도 CC합니다. Bisection에 성공했다면 culprit commit message의 `Signed-off-by:` 줄에 있는 모든 사람을 CC합니다.
Report를 regression list에 CC하거나 forward할 때 다음 문단으로 regzbot에 도입 범위를 직접 알려주는 것이 좋습니다.
#regzbot introduced: v5.13..v5.14-rc1이 명령은 v5.13이 정상이고 v5.14-rc1이 처음 문제를 보인 버전이라는 뜻입니다. Bisection으로 culprit를 찾았다면 범위 대신 commit-id를 지정합니다.
#regzbot introduced: 1f2e3d4c5dRegzbot command를 넣으면 보고가 눈에 띄지 않은 채 빠질 가능성을 줄입니다. 생략해도 regression list로 사본을 보내면 human regression tracker가 등록하지만 한 사람에게 의존하므로 tracked unresolved list와 주간 보고에 나타나는 시점이 늦어질 수 있습니다.
이 지연은 Linus Torvalds가 개발을 계속할지 final release를 낼지 결정할 때 중요한 regression을 알지 못하게 할 수 있습니다.
Culprit commit을 신뢰성 있게 식별하면 거의 모든 regression이 고쳐집니다. Culprit 없이 고칠 수 있는 경우도 있지만 자주 필요합니다.
영향받은 code 영역 개발자가 culprit를 찾으려 노력해야 하지만 특정 hardware, firmware, distribution, system configuration, application에서만 나타나는 문제는 개발자 환경 밖이라 합리적인 노력으로 재현하기 어렵습니다. 결국 reporter가 culprit를 찾아야 할 때가 많고 root cause를 좁히기 위한 추가 시험도 필요할 수 있습니다.
개발자는 일반 사용자가 수행할 수 있도록 가능한 조언과 도움을 제공해야 합니다. Culprit는 `Documentation/admin-guide/reporting-issues.rst`의 개요와 `Documentation/admin-guide/bug-bisect.rst`의 상세 절차에 따라 bisection으로 찾습니다. 재현이 어렵거나 오래 걸리면 다른 영향받은 사용자와 범위를 나눠 좁힐 수 있습니다.
Regression 조언이 필요하면 `regressions@lists.linux.dev`로 mail을 보내고 Linux kernel regression tracker `regressions@leemhuis.info`를 CC합니다. Private로 다루는 편이 나은 문제라면 공개 list를 생략할 수 있습니다.
판정에서 culprit 확인과 regzbot tracking까지의 흐름입니다.
No regressions 규칙의 목적, 집행과 수정 시한
172-228사용자는 kernel update 때 무언가 깨질까 걱정하지 않아야 합니다. 개발자는 update를 매력적으로 만들어 사용자가 abandoned되거나 1년 반 넘게 오래된 stable·longterm series에 머물지 않게 하려 합니다. 오래된 계열에는 새 version에서 이미 고친 bug, security issue, 기타 문제가 남아 있을 수 있습니다. 관련 설명은 `http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/`에 있습니다.
최신 pre-release와 regular release를 사용자가 쉽게 시험하도록 하는 것도 목적입니다. 도입 직후 보고된 문제는 원인을 찾고 고치기 훨씬 쉽기 때문에 모두에게 이익입니다.
`no regressions` 규칙은 실제로 매우 진지하게 적용되며 Linus Torvalds의 여러 mailing list 글에서도 확인할 수 있습니다. 일부 인용은 `Documentation/process/handling-regressions.rst`에 있습니다. 예외는 극히 드물고 과거에 예외라고 판단한 개발자가 잘못된 경우가 거의 대부분이었습니다.
Subsystem maintainer가 규칙 준수를 책임지고 tree maintainer가 감시·지원합니다. Mainline은 Linus Torvalds, 여러 stable·longterm series는 Greg Kroah-Hartman 등이 맡습니다.
Regression report가 빠지지 않게 돕는 사람도 있습니다. Thorsten Leemhuis는 현재 Linux kernel regression tracker로 활동하며 regzbot을 사용합니다. 모든 report를 regression list에 CC 또는 forward하고 가능하면 regzbot command를 넣어 즉시 tracking하게 하는 이유입니다.
보고된 regression은 영향받은 사용자에게 빨리 해결책을 주고 더 많은 피해를 막도록 가능한 빠르게 고쳐야 합니다. 다만 fix가 추가 손상을 만들지 않게 충분한 시간과 주의를 들여야 합니다.
실제 시한은 영향, 나이, 발생한 Linux series 등에 달렸지만 대부분의 regression은 2주 안에 수정되어야 합니다.
Software update, 성능, 외부 module, 보안과 기타 경계 사례
229-322일부 software를 update해 피할 수 있는 문제도 거의 항상 regression입니다. 개발자가 아니라고 하면 앞 절차대로 regression tracker에게 조언을 구합니다.
새 kernel이 느리거나 energy를 더 쓰는 것도 차이가 유의미하면 regression입니다. micro-benchmark가 5% 느린 정도는 넓은 benchmark 결과에도 1% 넘는 영향을 주지 않는 한 보통 해당하지 않습니다. 의심되면 조언을 구합니다.
외부 kernel module이 Linux update 뒤 build되거나 실행되지 않는 문제는 regression이 아닙니다. 규칙은 Linux kernel이 userland에 제공하는 interface와 service에 관한 것이며, external module은 kernel-space에서 변경될 수 있는 내부 interface에 연결되기 때문입니다.
극히 드물게 security issue를 regression 없이 고칠 수 없으면 security fix가 더 작은 악으로 우선됩니다. 보통 관련 핵심 개발자와 Linus Torvalds가 regression 없는 방법을 찾으려 노력해 이런 절충을 피합니다. 해당 사례를 만나면 archive에서 충분한 회피 노력이 있었는지 확인하고 아니라면 보고하며 의심되면 조언을 구합니다.
한 regression을 고치면 다른 regression이 생기는 경우에도 해당 영역 expert가 둘 다 피하거나 영향을 최소화할 fix를 찾아야 합니다. Security fix 사례처럼 이전 논의를 확인하고 필요하면 조언을 구합니다.
이런 충돌은 각 development cycle의 mainline pre-release, 예를 들어 v5.15-rc1이나 -rc3을 정기적으로 시험하면 줄일 수 있습니다. v5.14와 v5.15-rc1 사이 변경이 regression을 만들면서 다른 개선의 필수 조건이 됐더라도 v5.15 release 전에 발견하면 관련 변경을 모두 revert할 수 있습니다.
며칠이나 몇 주 뒤에는 follow-up 변경에 software가 의존하기 시작해 모두 revert하면 그 software 사용자에게 새 regression이 생길 수 있습니다. 그러면 단순 revert가 불가능해집니다.
몇 달 전에 제거된 의존 기능도 regression이지만 늦게 발견하면 앞 이유로 고치기 어려워 case-by-case로 다룹니다. 이것도 mainline pre-release 정기 시험이 중요한 이유입니다.
영향받은 사람이 한 명뿐이어도 실제 사용이라면 규칙이 적용됩니다. 다만 attic이나 museum에서만 볼 hardware 지원을 제거할 자유도 필요합니다. Linux가 정체되지 않으려면 매우 적은 사용자만 영향받고 software update나 전용 kernel parameter로 쉽게 우회할 수 있는 경우 더 큰 이익을 위해 허용될 수도 있습니다.
Staging tree code에는 `https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/Kconfig`의 configuration help text에 적힌 대로 규칙이 적용되지 않습니다.
Please note that these drivers are under heavy development, may or
may not work, and may contain userspace interfaces that most likely
will be changed in the near future.Staging 개발자도 흔히 규칙을 따르지만 발전을 위해 완화할 때가 있습니다. Staging WiFi driver가 처음부터 새로 쓴 전혀 다른 driver로 교체되면서 대체로 사소한 regression이 생기는 사례가 있습니다.
유사한 configuration이 필요한 이유와 생성 방법
323-355개발자는 regression을 만든다고 알려진 변경을 optional로 통합하고 kernel default configuration에서는 끌 수 있습니다. 이 방법은 `no regressions` 규칙이 발전을 막는 것을 피합니다.
예를 들어 malware가 악용하는 kernel interface 접근을 차단하는 새 security 기능이 드물게 쓰는 application 실행에도 필요한 interface를 막을 수 있습니다. 해당 application 사용자는 기능을 끄고 다른 사용자는 켜는 방식으로 양쪽 요구를 만족합니다.
Known-good kernel로 machine을 boot한 뒤 새 Linux에서 `make olddefconfig`를 실행합니다. Build script는 실행 중인 kernel의 `.config`를 새 build의 기반으로 읽고 새 configuration option을 default로 지정합니다. Default는 regression을 만들 수 있는 새 기능을 꺼야 합니다.
Pre-compiled vanilla kernel에서 찾은 regression도 보고할 수 있지만 새 kernel이 이전과 유사한 configuration으로 build됐는지 확인해야 합니다. Provider가 새 kernel에 알려진 incompatible 기능을 켰을 수 있으므로 의심되면 provider에게 보고하고 조언을 구합니다.
Regzbot regression tracking
356-438`no regressions` 같은 규칙은 누군가 준수를 확인하지 않으면 실수나 고의로 깨질 수 있습니다. Linux에서도 마찬가지였기 때문에 regression tracker Thorsten Leemhuis와 몇몇 사람이 해결될 때까지 regression을 지켜봅니다. 이 일은 유급이 아니어서 best effort로 수행됩니다.
Linux 개발 과정은 분산되고 느슨해 완전한 수동 tracking이 어렵습니다. Regression tracker는 장기적으로 관련된 모두에게 가능한 많이 자동화하려고 regzbot을 만들었습니다.
Regzbot은 tracked report의 reply를 관찰하고 해당 report를 `Link:` tag로 참조하는 posted 또는 committed patch와 그 patch 논의도 추적합니다. 이를 합치면 fix 진행 상태를 파악할 수 있습니다.
현재 tracking 중인 regression은 `https://linux-regtracking.leemhuis.info/regzbot/` web interface에서 확인합니다.
Regzbot은 regression용이므로 일반 issue에는 사용하지 않습니다. 다만 hang, corrupted data, internal error인 Panic, Oops, `BUG()`, warning 같은 severe issue를 tracking하는 것은 regression tracker 관점에서 허용됩니다.
Tracked regression 속성을 바꾸려면 report mail에 직접 또는 간접 reply하면서 독립된 문단에 regzbot command를 씁니다. Sent folder나 mailing list archive에서 report를 찾아 `Reply-all`하는 것이 가장 쉽습니다.
Bisection 뒤 도입 commit을 갱신하는 예는 다음과 같습니다.
#regzbot introduced: 1f2e3d4c5d제목을 지정하거나 갱신하는 명령은 다음과 같습니다.
#regzbot title: fooIssue의 추가 측면이나 fix가 논의되는 mailing list thread 또는 `bugzilla.kernel.org` ticket을 monitor할 수 있습니다.
#regzbot monitor: https://lore.kernel.org/r/30th.anniversary.repost@klaava.Helsinki.FI/
#regzbot monitor: https://bugzilla.kernel.org/show_bug.cgi?id=123456789약간 관련됐지만 다른 주제인 post나 tracker ticket을 추가 정보 link로 지정할 수 있습니다.
#regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=123456789Regression이 아니었다면 invalid로 표시할 수 있습니다.
#regzbot invalid: wasn't a regression, problem has always existed주로 developer와 regression tracking 담당자가 쓰는 다른 command도 있습니다. 자세한 내용은 regzbot getting started guide `https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md`와 reference `https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md`를 참조합니다.
재배포 정보
439-451이 text는 파일 상단에 명시된 대로 `GPL-2.0+` 또는 `CC-BY-4.0`으로 제공됩니다. `CC-BY-4.0`만으로 배포하려면 author attribution을 `The Linux kernel developers`로 쓰고 `https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-regressions.rst`를 source로 link합니다.
`CC-BY-4.0`은 Linux kernel source에 있는 이 RST file 내용에만 적용됩니다. Kernel build system 등으로 처리된 version은 더 제한적인 license를 쓰는 다른 file 내용을 포함할 수 있습니다.
요약과 해설
reporting-regressions.rst:1-451Regression은 단순한 새 bug가 아니라 이전 kernel에서 정상인 실제 사용이 유사한 설정의 새 kernel에서 나빠진 경우입니다. 빠른 해결을 위해 마지막 정상 버전과 첫 broken 버전을 제시하고 regression list에 공개적으로 알려야 합니다.
가능하면 bisection으로 culprit를 찾고 regzbot command로 tracking을 시작합니다. 이 정보는 maintainer와 Linus Torvalds가 regression 상태를 놓치지 않고 release 판단과 fix 우선순위에 반영하도록 돕습니다.