Automated testing of the DRM subsystem

공유 DRM 자동화 시험의 목적

Core 또는 driver 변경이 regression을 만들지 않았는지 확인하려면 다양한 hardware configuration을 시험해야 하므로 시간이 많이 듭니다. 시험에 관심 있는 각 사람이 상당한 양의 hardware를 직접 구입하고 유지하는 것도 현실적이지 않습니다.

또한 maintainer가 regression을 찾아 되돌려 알릴 때까지 기다리기보다 developer가 자신의 code에서 regression을 직접 확인하는 편이 바람직합니다.

gitlab.freedesktop.org에는 Mesa를 자동 시험하는 facility가 있으며 DRM subsystem 시험에도 사용할 수 있습니다. 이 문서는 공유 infrastructure를 이용해 시험 시간과 노력을 줄이는 방법을 설명합니다.

.. SPDX-License-Identifier: GPL-2.0+

=========================================
Automated testing of the DRM subsystem
=========================================

Introduction
============

Making sure that changes to the core or drivers don't introduce regressions can
be very time-consuming when lots of different hardware configurations need to
be tested. Moreover, it isn't practical for each person interested in this
testing to have to acquire and maintain what can be a considerable amount of
hardware.

Also, it is desirable for developers to check for regressions in their code by
themselves, instead of relying on the maintainers to find them and then
reporting back.

There are facilities in gitlab.freedesktop.org to automatically test Mesa that
can be used as well for testing the DRM subsystem. This document explains how
people interested in testing it can use this shared infrastructure to save
quite some time and effort.

Root CI 설정과 공통 test list

`drivers/gpu/drm/ci/gitlab-ci.yml`은 GitLab CI의 root configuration file입니다. 사용해야 할 script의 특정 version을 지정하며 pipeline 동작을 바꾸는 변수도 제공합니다.

`IGT_VERSION`의 revision은 `https://gitlab.freedesktop.org/drm/igt-gpu-tools` 프로젝트를 기준으로 합니다.

`drivers/gpu/drm/ci/testlist.txt`에는 모든 driver에서 실행할 IGT test가 들어 있습니다. 다만 해당 driver의 `*-skips.txt`에 기록된 test는 실행하지 않습니다.

Relevant files
==============

drivers/gpu/drm/ci/gitlab-ci.yml
--------------------------------

This is the root configuration file for GitLab CI. Among other less interesting
bits, it specifies the specific version of the scripts to be used. There are
some variables that can be modified to change the behavior of the pipeline:

DRM_CI_PROJECT_PATH
    Repository that contains the Mesa software infrastructure for CI

DRM_CI_COMMIT_SHA
    A particular revision to use from that repository

UPSTREAM_REPO
    URL to git repository containing the target branch

TARGET_BRANCH
    Branch to which this branch is to be merged into

IGT_VERSION
    Revision of igt-gpu-tools being used, from
    https://gitlab.freedesktop.org/drm/igt-gpu-tools

drivers/gpu/drm/ci/testlist.txt
-------------------------------

IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
file, see below).

Known failure, flake, skip 파일

`drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt`는 특정 hardware revision의 해당 driver에서 알려진 failure를 나열합니다.

`${DRIVER_NAME}-${HW_REVISION}-flakes.txt`는 신뢰성 없이 동작하는 것으로 알려진 test를 나열합니다. 이 test들은 계속 실행되지만 결과와 관계없이 job failure를 일으키지 않습니다.

새 flake entry에는 affected driver author에게 bug를 보고한 email 링크 또는 관련 GitLab issue 링크가 반드시 있어야 합니다. Board name 또는 Device Tree name, 처음 영향을 받은 kernel version, 시험에 사용한 IGT version, 대략적인 failure rate도 포함해야 합니다.

# Bug Report: $LORE_URL_OR_GITLAB_ISSUE
# Board Name: broken-board.dtb
# Linux Version: 6.6-rc1
# IGT Version: 1.28-gd2af13d9f
# Failure Rate: 100
flaky-test

`${DRIVER_NAME}-${HW_REVISION}-skips.txt`는 특정 hardware revision의 driver에서 실행하지 않을 test를 나열합니다. 보통 machine hang, OOM, 지나치게 긴 실행 시간 등으로 전체 test list 실행을 방해하는 test입니다.

drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
----------------------------------------------------------

Lists the known failures for a given driver on a specific hardware revision.

drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
-----------------------------------------------------------

Lists the tests that for a given driver on a specific hardware revision are
known to behave unreliably. These tests won't cause a job to fail regardless of
the result. They will still be run.

Each new flake entry must be associated with a link to the email reporting the
bug to the author of the affected driver or the relevant GitLab issue. The entry
must also include the board name or Device Tree name, the first kernel version
affected, the IGT version used for tests, and an approximation of the failure rate.

They should be provided under the following format::

  # Bug Report: $LORE_URL_OR_GITLAB_ISSUE
  # Board Name: broken-board.dtb
  # Linux Version: 6.6-rc1
  # IGT Version: 1.28-gd2af13d9f
  # Failure Rate: 100
  flaky-test

Use the appropriate link below to create a GitLab issue:
amdgpu driver: https://gitlab.freedesktop.org/drm/amd/-/issues
i915 driver: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues
msm driver: https://gitlab.freedesktop.org/drm/msm/-/issues
xe driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues

drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
-----------------------------------------------------------

Lists the tests that won't be run for a given driver on a specific hardware
revision. These are usually tests that interfere with the running of the test
list due to hanging the machine, causing OOM, taking too long, etc.

자신의 tree에서 자동 시험 활성화

자동 시험을 활성화하려면 gitlab.freedesktop.org에 Linux tree를 만들고 repository CI/CD 설정에서 configuration file을 `.gitlab-ci.yml` 대신 `drivers/gpu/drm/ci/gitlab-ci.yml`로 바꿉니다.

CI 실행 권한을 얻도록 `drm/ci-ok` group 추가를 요청합니다. 그 다음 repository에 push하면 CI pipeline이 생성되고 여러 job이 실행됩니다.

Pipeline이 끝나면 regression이 발견되지 않은 한 모든 job이 green이어야 합니다. Pipeline warning은 시험 중 lockdep issue가 검출됐음을 나타냅니다.

How to enable automated testing on your tree
============================================

1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
yet

2. In your kernel repo's configuration (eg.
https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
CI/CD configuration file from .gitlab-ci.yml to
drivers/gpu/drm/ci/gitlab-ci.yml.

3. Request to be added to the drm/ci-ok group so that your user has the
necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok

4. Next time you push to this repository, you will see a CI pipeline being
created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)

5. The various jobs will be run and when the pipeline is finished, all jobs
should be green unless a regression has been found.

6. Warnings in the pipeline indicate that lockdep
(see Documentation/locking/lockdep-design.rst) issues have been detected
during the tests.

Expectation 갱신과 coverage 확장

Code 변경으로 test가 고쳐졌다면, 그 변경의 영향을 받은 각 test platform에 대해 하나 이상의 `drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt` 파일에서 해당 줄을 제거해야 합니다.

Reliability 문제 해결 등으로 더 많은 test를 실행할 수 있게 됐다면 flakes 또는 skips list에서 해당 test를 제거할 수 있습니다. 알려진 failure가 있다면 이어서 expected result도 맞게 조정합니다.

사용하는 IGT version을 갱신해야 한다면, 예를 들어 IGT에 test를 더 추가했다면 `gitlab-ci.yml` 맨 위의 `IGT_VERSION` 변수를 갱신합니다.

How to update test expectations
===============================

If your changes to the code fix any tests, you will have to remove one or more
lines from one or more of the files in
drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
affected by the change.


How to expand coverage
======================

If your code changes make it possible to run more tests (by solving reliability
issues, for example), you can remove tests from the flakes and/or skips lists,
and then the expected results if there are any known failures.

If there is a need for updating the version of IGT being used (maybe you have
added more tests to it), update the IGT_VERSION variable at the top of the
gitlab-ci.yml file.

CI script 변경과 external fix 병합

drm-ci repository의 script 변경을 시험하려면 `drivers/gpu/drm/ci/gitlab-ci.yml`에서 `DRM_CI_PROJECT_PATH`와 `DRM_CI_COMMIT_SHA`를 자신의 project fork와 revision에 맞춥니다. 예시는 `janedoe/drm-ci`이며 fork는 gitlab.freedesktop.org에 있어야 합니다.

다른 tree의 regression 때문에 현재 tree의 변경을 시험할 수 없는 경우가 자주 있습니다. 필요한 fix는 target tree에서 `${TARGET_BRANCH}-external-fixes`라는 branch에 두면 build job 중 자동으로 merge됩니다.

Pipeline이 merge request 안에서 실행되지 않고 local tree에도 같은 이름의 branch가 있으면, 그 local branch의 commit도 함께 merge됩니다.

How to test your changes to the scripts
=======================================

For testing changes to the scripts in the drm-ci repo, change the
DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.


How to incorporate external fixes in your testing
=================================================

Often, regressions in other trees will prevent testing changes local to the
tree under test. These fixes will be automatically merged in during the build
jobs from a branch in the target tree that is named as
${TARGET_BRANCH}-external-fixes.

If the pipeline is not in a merge request and a branch with the same name
exists in the local tree, commits from that branch will be merged in as well.

중단된 automated testing lab 처리

Hardware farm 중단 때문에 원래 통과해야 할 pipeline이 실패한다면, 해당 farm으로 제출될 모든 job을 비활성화할 수 있습니다.

비활성화 상태는 `https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml` 파일을 편집하여 설정합니다.

How to deal with automated testing labs that may be down
========================================================

If a hardware farm is down and thus causing pipelines to fail that would
otherwise pass, one can disable all jobs that would be submitted to that farm
by editing the file at
https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.