Event Tracing

1. 소개

이 문서는 Theodore Ts'o가 작성했고 Li Zefan과 Tom Zanussi가 갱신했다.

event tracing infrastructure를 사용하면 probe function을 등록하는 custom kernel module을 따로 만들지 않고도 tracepoint를 활용할 수 있다. tracepoint 자체에 대한 설명은 `Documentation/trace/tracepoints.rst`를 참조한다.

모든 tracepoint를 event tracing system으로 추적할 수 있는 것은 아니다. kernel developer가 tracing information을 tracing buffer에 어떤 형식으로 저장하고 어떻게 출력할지를 정의하는 code snippet을 제공해야 한다.

=============
Event Tracing
=============

:Author: Theodore Ts'o
:Updated: Li Zefan and Tom Zanussi

1. Introduction
===============

Tracepoints (see Documentation/trace/tracepoints.rst) can be used
without creating custom kernel modules to register probe functions
using the event tracing infrastructure.

Not all tracepoints can be traced using the event tracing system;
the kernel developer must provide code snippets which define how the
tracing information is saved into the tracing buffer, and how the
tracing information should be printed.

2.1 set_event interface

추적할 수 있는 event 목록은 `/sys/kernel/tracing/available_events`에서 확인한다.

`sched_wakeup` 같은 특정 event를 활성화하려면 event 이름을 `/sys/kernel/tracing/set_event`에 쓴다. 기존 선택을 유지하면서 추가하려면 반드시 `>>`를 사용해야 한다. `>`를 쓰면 먼저 모든 event가 비활성화된다.

event를 비활성화하려면 이름 앞에 느낌표를 붙여 `set_event`에 추가한다. 빈 줄을 쓰면 모든 event가 비활성화되고, `*:*` 또는 `*:`를 쓰면 모든 event가 활성화된다.

event는 `ext4`, `irq`, `sched` 같은 subsystem으로 구성된다. 완전한 event 이름은 `<subsystem>:<event>`이며 subsystem 이름은 생략할 수 있지만 `available_events`에는 함께 표시된다. `<subsystem>:*`는 해당 subsystem의 모든 event를 뜻하므로 `irq:*`는 모든 IRQ event를 활성화한다.

`set_event`는 특정 module과 연관된 event만 선택할 수도 있다. `:mod:<module>`은 해당 module의 모든 event를 활성화한다. module이 아직 load되지 않았다면 문자열을 보관했다가 이름이 일치하는 module이 load될 때 적용한다.

`:mod:` 앞의 `<match>`는 module이 만드는 특정 system 또는 event를 고르는 pattern으로 해석된다. `*`는 모든 event와 일치하고, `<system>:<event>:mod:<module>`은 지정한 system 안의 event만 선택한다. 이때 `<event>`가 `*`이면 그 module이 제공하는 해당 system의 모든 event와 일치한다.

2. Using Event Tracing
======================

2.1 Via the 'set_event' interface
---------------------------------

The events which are available for tracing can be found in the file
/sys/kernel/tracing/available_events.

To enable a particular event, such as 'sched_wakeup', simply echo it
to /sys/kernel/tracing/set_event. For example::

	# echo sched_wakeup >> /sys/kernel/tracing/set_event

.. Note:: '>>' is necessary, otherwise it will firstly disable all the events.

To disable an event, echo the event name to the set_event file prefixed
with an exclamation point::

	# echo '!sched_wakeup' >> /sys/kernel/tracing/set_event

To disable all events, echo an empty line to the set_event file::

	# echo > /sys/kernel/tracing/set_event

To enable all events, echo ``*:*`` or ``*:`` to the set_event file::

	# echo *:* > /sys/kernel/tracing/set_event

The events are organized into subsystems, such as ext4, irq, sched,
etc., and a full event name looks like this: <subsystem>:<event>.  The
subsystem name is optional, but it is displayed in the available_events
file.  All of the events in a subsystem can be specified via the syntax
``<subsystem>:*``; for example, to enable all irq events, you can use the
command::

	# echo 'irq:*' > /sys/kernel/tracing/set_event

The set_event file may also be used to enable events associated to only
a specific module::

	# echo ':mod:<module>' > /sys/kernel/tracing/set_event

Will enable all events in the module ``<module>``.  If the module is not yet
loaded, the string will be saved and when a module is that matches ``<module>``
is loaded, then it will apply the enabling of events then.

The text before ``:mod:`` will be parsed to specify specific events that the
module creates::

	# echo '<match>:mod:<module>' > /sys/kernel/tracing/set_event

The above will enable any system or event that ``<match>`` matches. If
``<match>`` is ``"*"`` then it will match all events.

To enable only a specific event within a system::

	# echo '<system>:<event>:mod:<module>' > /sys/kernel/tracing/set_event

If ``<event>`` is ``"*"`` then it will match all events within the system
for a given module.

2.2 enable toggle

사용 가능한 event는 `/sys/kernel/tracing/events/` directory hierarchy에도 나열된다. 개별 event의 `enable`에 `1`을 쓰면 활성화되고 `0`을 쓰면 비활성화된다.

`/sys/kernel/tracing/events/sched/enable`에 `1`을 쓰면 `sched` subsystem의 모든 event가 활성화된다. 최상위 `/sys/kernel/tracing/events/enable`에 `1`을 쓰면 모든 event가 활성화된다.

`enable` 파일을 읽었을 때 `0`은 이 파일이 관할하는 모든 event가 꺼졌음을, `1`은 모두 켜졌음을 뜻한다. `X`는 활성화와 비활성화가 섞였다는 뜻이고, `?`는 이 파일의 영향을 받는 event가 없다는 뜻이다.

2.2 Via the 'enable' toggle
---------------------------

The events available are also listed in /sys/kernel/tracing/events/ hierarchy
of directories.

To enable event 'sched_wakeup'::

	# echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable

To disable it::

	# echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable

To enable all events in sched subsystem::

	# echo 1 > /sys/kernel/tracing/events/sched/enable

To enable all events::

	# echo 1 > /sys/kernel/tracing/events/enable

When reading one of these enable files, there are four results:

 - 0 - all events this file affects are disabled
 - 1 - all events this file affects are enabled
 - X - there is a mixture of events enabled and disabled
 - ? - this file does not affect any event

2.3 boot option

early boot debugging에는 `trace_event=[event-list]` boot option을 사용한다. `event-list`는 쉼표로 구분한 event 목록이며 각 event의 형식은 2.1절의 `set_event` 형식을 따른다.

2.3 Boot option
---------------

In order to facilitate early boot debugging, use boot option::

	trace_event=[event-list]

event-list is a comma separated list of events. See section 2.1 for event
format.

3. event-enabled tracepoint 정의

event tracing을 지원하는 tracepoint를 정의하는 실제 예제는 `samples/trace_events`에서 확인한다.

3. Defining an event-enabled tracepoint
=======================================

See The example provided in samples/trace_events

4. Event format

각 trace event에는 기록된 event의 field를 설명하는 `format` 파일이 연결돼 있다. 이 정보는 binary trace stream을 parse하는 데 쓰이며, 5절의 event filter에서 사용할 field 이름도 여기서 찾는다.

`format` 파일에는 text mode 출력에 사용할 format string, profiling에서 쓰는 event 이름과 ID도 표시된다.

모든 event에는 `common_` prefix가 붙은 공통 field 집합이 있다. 나머지 field는 event마다 다르며 해당 event의 `TRACE_EVENT` 정의에 선언된 field와 대응한다.

각 field는 `field:field-type field-name; offset:N; size:N;` 형식이다. `offset`은 trace record 안에서 field가 시작하는 위치이고, `size`는 data item의 byte 단위 크기다.

예시의 `sched_wakeup` event에는 10개 field가 있다. 앞의 5개는 common field이고 뒤의 5개는 event 전용 field다. `comm`만 string이고 나머지는 numeric field이며, 이 type 차이는 event filtering에서 중요하다.

4. Event formats
================

Each trace event has a 'format' file associated with it that contains
a description of each field in a logged event.  This information can
be used to parse the binary trace stream, and is also the place to
find the field names that can be used in event filters (see section 5).

It also displays the format string that will be used to print the
event in text mode, along with the event name and ID used for
profiling.

Every event has a set of ``common`` fields associated with it; these are
the fields prefixed with ``common_``.  The other fields vary between
events and correspond to the fields defined in the TRACE_EVENT
definition for that event.

Each field in the format has the form::

     field:field-type field-name; offset:N; size:N;

where offset is the offset of the field in the trace record and size
is the size of the data item, in bytes.

For example, here's the information displayed for the 'sched_wakeup'
event::

	# cat /sys/kernel/tracing/events/sched/sched_wakeup/format

	name: sched_wakeup
	ID: 60
	format:
		field:unsigned short common_type;	offset:0;	size:2;
		field:unsigned char common_flags;	offset:2;	size:1;
		field:unsigned char common_preempt_count;	offset:3;	size:1;
		field:int common_pid;	offset:4;	size:4;
		field:int common_tgid;	offset:8;	size:4;

		field:char comm[TASK_COMM_LEN];	offset:12;	size:16;
		field:pid_t pid;	offset:28;	size:4;
		field:int prio;	offset:32;	size:4;
		field:int success;	offset:36;	size:4;
		field:int cpu;	offset:40;	size:4;

	print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
		   REC->prio, REC->success, REC->cpu

This event contains 10 fields, the first 5 common and the remaining 5
event-specific.  All the fields for this event are numeric, except for
'comm' which is a string, a distinction important for event filtering.

5. Event filtering

trace event에는 boolean `filter expression`을 연결해 kernel 안에서 거를 수 있다. event가 trace buffer에 기록되는 즉시 해당 event type에 연결된 expression과 field 값이 비교된다.

field 값이 filter와 일치하면 trace output에 나타나고, 일치하지 않으면 버려진다. filter가 없는 event는 모든 값과 일치하며 이것이 기본 상태다.

5. Event filtering
==================

Trace events can be filtered in the kernel by associating boolean
'filter expressions' with them.  As soon as an event is logged into
the trace buffer, its fields are checked against the filter expression
associated with that event type.  An event with field values that
'match' the filter will appear in the trace output, and an event whose
values don't match will be discarded.  An event with no filter
associated with it matches everything, and is the default when no
filter has been set for an event.

5.1 Filter expression 문법

filter expression은 하나 이상의 predicate로 이루어지며 `&&`와 `||` logical operator로 결합할 수 있다. predicate는 기록된 event field와 constant value를 비교해 일치하면 1, 일치하지 않으면 0을 반환하는 `field-name relational-operator value` 절이다.

괄호로 원하는 logical grouping을 만들 수 있다. shell이 operator를 metacharacter로 해석하지 않게 하려면 expression을 double quote로 감싼다. filter에 사용할 field 이름은 4절에서 설명한 event의 `format` 파일에서 찾는다.

사용 가능한 relational operator는 field type에 따라 다르다. numeric field에는 `==`, `!=`, `<`, `<=`, `>`, `>=`, `&`를 쓸 수 있고, string field에는 `==`, `!=`, `~`를 쓸 수 있다.

glob operator `~`는 wildcard `*`, `?`와 character class를 받는다. 따라서 `*sh`, `sh*`, `*sh*`, `ba*sh`처럼 앞·뒤·중간을 pattern으로 비교할 수 있다.

`sys_enter_openat`의 `filename`처럼 user space를 가리키는 pointer field를 검사할 때는 field 이름 뒤에 `.ustring`을 붙인다. 그래야 kernel이 pointer가 가리키는 user memory를 가져와야 한다는 사실을 알 수 있다.

long 크기의 값을 function address로 해석하려면 `.function` postfix를 붙여 function 이름과 비교할 수 있다. `call_site.function == security_prepare_creds`는 `call_site`가 `security_prepare_creds` 시작 주소 이상이고 끝 주소 미만일 때 참이다. `.function`은 long 크기 값에만 붙일 수 있고 `==` 또는 `!=`로만 비교한다.

cpumask field 또는 CPU 번호를 encode한 scalar field는 cpulist 형식의 `CPUS{$cpulist}`와 비교할 수 있다. cpumask filter는 intersection `&`, equality `==`, inequality `!=`를 지원한다. `target_cpu & CPUS{17-42}`는 `target_cpu`가 지정한 mask에 포함된 event를 선택한다.

5.1 Expression syntax
---------------------

A filter expression consists of one or more 'predicates' that can be
combined using the logical operators '&&' and '||'.  A predicate is
simply a clause that compares the value of a field contained within a
logged event with a constant value and returns either 0 or 1 depending
on whether the field value matched (1) or didn't match (0)::

	  field-name relational-operator value

Parentheses can be used to provide arbitrary logical groupings and
double-quotes can be used to prevent the shell from interpreting
operators as shell metacharacters.

The field-names available for use in filters can be found in the
'format' files for trace events (see section 4).

The relational-operators depend on the type of the field being tested:

The operators available for numeric fields are:

==, !=, <, <=, >, >=, &

And for string fields they are:

==, !=, ~

The glob (~) accepts a wild card character (\*,?) and character classes
([). For example::

  prev_comm ~ "*sh"
  prev_comm ~ "sh*"
  prev_comm ~ "*sh*"
  prev_comm ~ "ba*sh"

If the field is a pointer that points into user space (for example
"filename" from sys_enter_openat), then you have to append ".ustring" to the
field name::

  filename.ustring ~ "password"

As the kernel will have to know how to retrieve the memory that the pointer
is at from user space.

You can convert any long type to a function address and search by function name::

  call_site.function == security_prepare_creds

The above will filter when the field "call_site" falls on the address within
"security_prepare_creds". That is, it will compare the value of "call_site" and
the filter will return true if it is greater than or equal to the start of
the function "security_prepare_creds" and less than the end of that function.

The ".function" postfix can only be attached to values of size long, and can only
be compared with "==" or "!=".

Cpumask fields or scalar fields that encode a CPU number can be filtered using
a user-provided cpumask in cpulist format. The format is as follows::

  CPUS{$cpulist}

Operators available to cpumask filtering are:

& (intersection), ==, !=

For example, this will filter events that have their .target_cpu field present
in the given cpumask::

  target_cpu & CPUS{17-42}

5.2 Filter 설정

개별 event의 filter는 해당 event directory의 `filter` 파일에 expression을 써서 설정한다. 예를 들어 `sched_wakeup`에서 `common_preempt_count > 4`를 쓰면 그 조건에 맞는 record만 남는다.

여러 조건은 괄호와 logical operator로 조합할 수 있다. 예시의 signal filter는 `sig`가 10 이상 15 미만이거나 17이면서 process 이름이 `bash`가 아닌 event를 선택한다.

expression에 오류가 있으면 쓰기 동작이 `Invalid argument`로 실패한다. 이후 `filter` 파일을 읽으면 잘못된 문자열과 `parse_error`를 확인할 수 있다. 예시에서는 존재하지 않는 `dsig`를 사용해 `Field not found`가 출력된다.

현재 오류를 가리키는 caret `^`는 실제 오류 위치와 관계없이 filter 문자열 시작에 표시된다. 위치 표시는 정확하지 않아도 함께 출력되는 오류 메시지는 원인을 찾는 데 사용할 수 있다.

5.2 Setting filters
-------------------

A filter for an individual event is set by writing a filter expression
to the 'filter' file for the given event.

For example::

	# cd /sys/kernel/tracing/events/sched/sched_wakeup
	# echo "common_preempt_count > 4" > filter

A slightly more involved example::

	# cd /sys/kernel/tracing/events/signal/signal_generate
	# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter

If there is an error in the expression, you'll get an 'Invalid
argument' error when setting it, and the erroneous string along with
an error message can be seen by looking at the filter e.g.::

	# cd /sys/kernel/tracing/events/signal/signal_generate
	# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
	-bash: echo: write error: Invalid argument
	# cat filter
	((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
	^
	parse_error: Field not found

Currently the caret ('^') for an error always appears at the beginning of
the filter string; the error message should still be useful though
even without more accurate position info.

5.2.1 Filter 제한

ring buffer 안의 string이 아니라 kernel 또는 user space memory를 가리키는 `char *` pointer에 string filter를 적용하면, 안전을 위해 최대 1024 byte만 temporary buffer로 복사해 비교한다.

접근해서는 안 되는 memory를 가리켜 복사 중 fault가 발생하면 string 비교 결과는 불일치로 처리된다.

5.2.1 Filter limitations
------------------------

If a filter is placed on a string pointer ``(char *)`` that does not point
to a string on the ring buffer, but instead points to kernel or user space
memory, then, for safety reasons, at most 1024 bytes of the content is
copied onto a temporary buffer to do the compare. If the copy of the memory
faults (the pointer points to memory that should not be accessed), then the
string compare will be treated as not matching.

5.3 Filter 제거

개별 event의 filter를 지우려면 그 event의 `filter` 파일에 `0`을 쓴다. subsystem의 모든 event filter를 지우려면 subsystem root의 `filter` 파일에 `0`을 쓴다.

5.3 Clearing filters
--------------------

To clear the filter for an event, write a '0' to the event's filter
file.

To clear the filters for all events in a subsystem, write a '0' to the
subsystem's filter file.

5.4 Subsystem filter

subsystem root의 `filter` 파일에 expression을 쓰면 그 subsystem의 모든 event filter를 한 묶음으로 설정하거나 제거할 수 있다.

하지만 subsystem filter가 참조한 field를 어떤 event가 갖고 있지 않거나 다른 이유로 filter를 적용할 수 없으면, 그 event는 이전 filter 설정을 유지한다. 그 결과 사용자가 동일한 filter가 적용됐다고 생각해도 실제로는 서로 다른 filter가 섞여 혼란스러운 trace output이 생길 수 있다.

모든 event에 확실히 전파되는 것은 common field만 참조하는 filter다.

`sched` subsystem root에 `0`을 쓰면 `sched_switch`와 `sched_wakeup`의 filter가 모두 `none`이 된다. `common_pid == 0`처럼 common field만 사용하면 두 event 모두 같은 filter를 갖는다.

반면 `prev_pid == 0`은 `prev_pid` field가 있는 event에만 적용된다. 예시에서 `sched_switch`는 새 filter를 받지만 `sched_wakeup`은 이전의 `common_pid == 0`을 그대로 유지한다.

5.4 Subsystem filters
---------------------

For convenience, filters for every event in a subsystem can be set or
cleared as a group by writing a filter expression into the filter file
at the root of the subsystem.  Note however, that if a filter for any
event within the subsystem lacks a field specified in the subsystem
filter, or if the filter can't be applied for any other reason, the
filter for that event will retain its previous setting.  This can
result in an unintended mixture of filters which could lead to
confusing (to the user who might think different filters are in
effect) trace output.  Only filters that reference just the common
fields can be guaranteed to propagate successfully to all events.

Here are a few subsystem filter examples that also illustrate the
above points:

Clear the filters on all events in the sched subsystem::

	# cd /sys/kernel/tracing/events/sched
	# echo 0 > filter
	# cat sched_switch/filter
	none
	# cat sched_wakeup/filter
	none

Set a filter using only common fields for all events in the sched
subsystem (all events end up with the same filter)::

	# cd /sys/kernel/tracing/events/sched
	# echo common_pid == 0 > filter
	# cat sched_switch/filter
	common_pid == 0
	# cat sched_wakeup/filter
	common_pid == 0

Attempt to set a filter using a non-common field for all events in the
sched subsystem (all events but those that have a prev_pid field retain
their old filters)::

	# cd /sys/kernel/tracing/events/sched
	# echo prev_pid == 0 > filter
	# cat sched_switch/filter
	prev_pid == 0
	# cat sched_wakeup/filter
	common_pid == 0

5.5 PID filtering

최상위 `events` directory와 같은 위치의 `set_event_pid` 파일은 여기에 나열되지 않은 PID의 task에서 발생한 모든 event를 제외한다.

현재 shell의 PID인 `$$`를 `set_event_pid`에 쓰고 event를 활성화하면 현재 task의 event만 추적한다. 기존 PID를 잃지 않고 더 추가하려면 `>>`로 여러 PID를 덧붙인다.

5.5 PID filtering
-----------------

The set_event_pid file in the same directory as the top events directory
exists, will filter all events from tracing any task that does not have the
PID listed in the set_event_pid file.
::

	# cd /sys/kernel/tracing
	# echo $$ > set_event_pid
	# echo 1 > events/enable

Will only trace events for the current task.

To add more PIDs without losing the PIDs already included, use '>>'.
::

	# echo 123 244 1 >> set_event_pid

6. Event trigger

trace event에는 조건부로 trigger command를 실행하게 할 수 있다. 다른 trace event를 켜거나 끄는 동작, event가 발생할 때 stack trace를 남기는 동작 등이 여기에 해당한다.

trigger가 연결된 trace event가 호출되면 그 event에 연결된 command 집합이 실행된다. 각 trigger에는 5절과 같은 event filter를 추가할 수 있으며, event가 filter를 통과할 때만 command가 실행된다. filter가 없으면 항상 통과한다.

특정 event의 trigger는 해당 event의 `trigger` 파일에 expression을 써서 추가하거나 제거한다. 개별 command의 제한을 지키는 범위에서 한 event에 여러 trigger를 연결할 수 있다.

event trigger는 `soft` mode 위에 구현된다. 하나 이상의 trigger가 연결되면 event 자체가 명시적으로 활성화되지 않았더라도 tracepoint는 활성 상태가 되어 호출되지만, event가 실제로 enabled가 아니면 record는 trace되지 않는다. 이 구조 덕분에 비활성 event에서도 trigger를 실행하고 기존 event filter를 조건 판정에 재사용할 수 있다.

trigger 문법은 `Documentation/trace/ftrace.rst`의 `set_ftrace_filter` filter command 문법을 대략 바탕으로 하지만 차이가 크고 구현도 직접 연결돼 있지 않으므로 둘을 일반화해 동일하게 취급하면 안 된다.

`trace_marker`에 쓰는 동작도 `/sys/kernel/tracing/events/ftrace/print/trigger`에 설정한 trigger를 실행할 수 있다.

6. Event triggers
=================

Trace events can be made to conditionally invoke trigger 'commands'
which can take various forms and are described in detail below;
examples would be enabling or disabling other trace events or invoking
a stack trace whenever the trace event is hit.  Whenever a trace event
with attached triggers is invoked, the set of trigger commands
associated with that event is invoked.  Any given trigger can
additionally have an event filter of the same form as described in
section 5 (Event filtering) associated with it - the command will only
be invoked if the event being invoked passes the associated filter.
If no filter is associated with the trigger, it always passes.

Triggers are added to and removed from a particular event by writing
trigger expressions to the 'trigger' file for the given event.

A given event can have any number of triggers associated with it,
subject to any restrictions that individual commands may have in that
regard.

Event triggers are implemented on top of "soft" mode, which means that
whenever a trace event has one or more triggers associated with it,
the event is activated even if it isn't actually enabled, but is
disabled in a "soft" mode.  That is, the tracepoint will be called,
but just will not be traced, unless of course it's actually enabled.
This scheme allows triggers to be invoked even for events that aren't
enabled, and also allows the current event filter implementation to be
used for conditionally invoking triggers.

The syntax for event triggers is roughly based on the syntax for
set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'
section of Documentation/trace/ftrace.rst), but there are major
differences and the implementation isn't currently tied to it in any
way, so beware about making generalizations between the two.

.. Note::
     Writing into trace_marker (See Documentation/trace/ftrace.rst)
     can also enable triggers that are written into
     /sys/kernel/tracing/events/ftrace/print/trigger

6.1 Trigger expression 문법

trigger는 `command[:count] [if filter]`를 `trigger` 파일에 써서 추가한다. 같은 command 앞에 `!`를 붙여 쓰면 제거한다.

trigger를 제거할 때 `[if filter]` 부분은 command 일치 판정에 사용되지 않는다. 따라서 `!` command에서 filter를 생략해도 filter까지 적은 경우와 같은 trigger를 제거한다.

filter 문법은 앞의 Event filtering 절과 같다. 현재 `trigger` 파일에 `>`로 쓰는 동작은 trigger 하나를 추가하거나 제거하며 실제로 `>>`처럼 동작한다. 명시적인 `>>` 지원이나 파일 truncate로 모든 trigger를 지우는 기능은 없으므로 추가한 각 trigger를 `!`로 제거해야 한다.

6.1 Expression syntax
---------------------

Triggers are added by echoing the command to the 'trigger' file::

  # echo 'command[:count] [if filter]' > trigger

Triggers are removed by echoing the same command but starting with '!'
to the 'trigger' file::

  # echo '!command[:count] [if filter]' > trigger

The [if filter] part isn't used in matching commands when removing, so
leaving that off in a '!' command will accomplish the same thing as
having it in.

The filter syntax is the same as that described in the 'Event
filtering' section above.

For ease of use, writing to the trigger file using '>' currently just
adds or removes a single trigger and there's no explicit '>>' support
('>' actually behaves like '>>') or truncation support to remove all
triggers (you have to use '!' for each one added.)

6.2 지원되는 trigger command

`enable_event`와 `disable_event`는 triggering event가 발생할 때 다른 trace event를 활성화하거나 비활성화한다. 이 command를 등록하면 대상 event는 soft mode로 활성화되어 tracepoint는 호출되지만 record는 아직 기록되지 않는다. trigger가 존재하는 동안 대상 tracepoint는 이 상태를 유지한다.

예시에서 read system call 진입 event의 `enable_event:kmem:kmalloc:1`은 `kmalloc` tracing을 한 번만 활성화한다. read system call 종료 event의 `disable_event:kmem:kmalloc`은 종료 때마다 `kmalloc` tracing을 중지한다. 형식은 `enable_event:<system>:<event>[:count]`와 `disable_event:<system>:<event>[:count]`이며, 제거할 때 앞에 `!`를 붙인다.

한 triggering event에는 여러 enable/disable trigger를 둘 수 있어 `sys_enter_read`가 `kmem:kmalloc`과 `sched:sched_switch`를 함께 켤 수 있다. 그러나 동일한 triggered event를 대상으로 하는 trigger는 하나만 둘 수 있다. 같은 `kmem:kmalloc`에 count나 filter만 다른 두 version을 따로 둘 수 없고, 필요하면 하나의 filter로 결합해야 한다.

`stacktrace`는 triggering event가 발생할 때 trace buffer에 stack trace를 dump한다. count가 없으면 매번 실행되고, `stacktrace:5 if bytes_req >= 65536`은 64 KiB 이상 `kmalloc` 요청 중 처음 5번만 기록한다. 제거할 때 filter는 생략할 수 있으며 triggering event 하나당 stacktrace trigger는 하나만 허용된다.

`snapshot`은 triggering event가 발생할 때 snapshot을 만든다. `block_unplug`에서 `nr_rq > 1`일 때 실행하도록 설정하면 그 시점에 trace 중이던 event나 function을 snapshot trace buffer가 보존한다. `snapshot:1`은 한 번만 실행한다. triggering event 하나당 snapshot trigger도 하나만 허용된다.

`traceon`과 `traceoff`는 지정한 event가 발생할 때 tracing을 켜거나 끈다. parameter는 실행 횟수를 정하고 생략하면 제한이 없다. `traceoff:1 if nr_rq > 1`은 조건을 처음 만족할 때 tracing을 멈추므로, trace buffer에서 trigger까지 이어진 event sequence를 조사할 수 있다. triggering event 하나에는 traceon 또는 traceoff trigger 하나만 둘 수 있다.

`hist`는 하나 이상의 trace event format field 또는 stack trace를 key로 삼아 event hit를 hash table에 집계한다. 하나 이상의 field에서 계산한 running total과 event count인 `hitcount`도 함께 누적할 수 있다. 자세한 설명과 예제는 `Documentation/trace/histogram.rst`를 참조한다.

6.2 Supported trigger commands
------------------------------

The following commands are supported:

- enable_event/disable_event

  These commands can enable or disable another trace event whenever
  the triggering event is hit.  When these commands are registered,
  the other trace event is activated, but disabled in a "soft" mode.
  That is, the tracepoint will be called, but just will not be traced.
  The event tracepoint stays in this mode as long as there's a trigger
  in effect that can trigger it.

  For example, the following trigger causes kmalloc events to be
  traced when a read system call is entered, and the :1 at the end
  specifies that this enablement happens only once::

	  # echo 'enable_event:kmem:kmalloc:1' > \
	      /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger

  The following trigger causes kmalloc events to stop being traced
  when a read system call exits.  This disablement happens on every
  read system call exit::

	  # echo 'disable_event:kmem:kmalloc' > \
	      /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger

  The format is::

      enable_event:<system>:<event>[:count]
      disable_event:<system>:<event>[:count]

  To remove the above commands::

	  # echo '!enable_event:kmem:kmalloc:1' > \
	      /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger

	  # echo '!disable_event:kmem:kmalloc' > \
	      /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger

  Note that there can be any number of enable/disable_event triggers
  per triggering event, but there can only be one trigger per
  triggered event. e.g. sys_enter_read can have triggers enabling both
  kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc
  versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if
  bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they
  could be combined into a single filter on kmem:kmalloc though).

- stacktrace

  This command dumps a stacktrace in the trace buffer whenever the
  triggering event occurs.

  For example, the following trigger dumps a stacktrace every time the
  kmalloc tracepoint is hit::

	  # echo 'stacktrace' > \
		/sys/kernel/tracing/events/kmem/kmalloc/trigger

  The following trigger dumps a stacktrace the first 5 times a kmalloc
  request happens with a size >= 64K::

	  # echo 'stacktrace:5 if bytes_req >= 65536' > \
		/sys/kernel/tracing/events/kmem/kmalloc/trigger

  The format is::

      stacktrace[:count]

  To remove the above commands::

	  # echo '!stacktrace' > \
		/sys/kernel/tracing/events/kmem/kmalloc/trigger

	  # echo '!stacktrace:5 if bytes_req >= 65536' > \
		/sys/kernel/tracing/events/kmem/kmalloc/trigger

  The latter can also be removed more simply by the following (without
  the filter)::

	  # echo '!stacktrace:5' > \
		/sys/kernel/tracing/events/kmem/kmalloc/trigger

  Note that there can be only one stacktrace trigger per triggering
  event.

- snapshot

  This command causes a snapshot to be triggered whenever the
  triggering event occurs.

  The following command creates a snapshot every time a block request
  queue is unplugged with a depth > 1.  If you were tracing a set of
  events or functions at the time, the snapshot trace buffer would
  capture those events when the trigger event occurred::

	  # echo 'snapshot if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

  To only snapshot once::

	  # echo 'snapshot:1 if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

  To remove the above commands::

	  # echo '!snapshot if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

	  # echo '!snapshot:1 if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

  Note that there can be only one snapshot trigger per triggering
  event.

- traceon/traceoff

  These commands turn tracing on and off when the specified events are
  hit. The parameter determines how many times the tracing system is
  turned on and off. If unspecified, there is no limit.

  The following command turns tracing off the first time a block
  request queue is unplugged with a depth > 1.  If you were tracing a
  set of events or functions at the time, you could then examine the
  trace buffer to see the sequence of events that led up to the
  trigger event::

	  # echo 'traceoff:1 if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

  To always disable tracing when nr_rq  > 1::

	  # echo 'traceoff if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

  To remove the above commands::

	  # echo '!traceoff:1 if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

	  # echo '!traceoff if nr_rq > 1' > \
		/sys/kernel/tracing/events/block/block_unplug/trigger

  Note that there can be only one traceon or traceoff trigger per
  triggering event.

- hist

  This command aggregates event hits into a hash table keyed on one or
  more trace event format fields (or stacktrace) and a set of running
  totals derived from one or more trace event format fields and/or
  event counts (hitcount).

  See Documentation/trace/histogram.rst for details and examples.

7. Kernel 내부 trace event API

대부분은 trace event command-line interface만으로 충분하다. 하지만 단순한 command expression 연결로 표현하기 어려운 관계가 필요하거나 command 집합 구성이 지나치게 번거로운 application도 있다.

예를 들어 scheduler가 잘못된 kernel state에 들어갔는지 감지하는 in-kernel state machine을 유지하려면 application이 trace stream을 계속 듣고 상태를 갱신해야 할 수 있다.

trace event subsystem은 module과 다른 kernel code가 사용자 정의 synthetic event를 필요할 때 생성하는 in-kernel API를 제공한다. 이 event는 기존 trace stream을 보강하거나 중요한 특정 상태의 발생을 알리는 데 쓸 수 있다.

kprobe와 kretprobe event를 만드는 유사한 in-kernel API도 있다. synthetic event API와 kprobe/kretprobe API는 더 낮은 단계의 `dynevent_cmd` event command API 위에 구현된다. 이 low-level API는 특수 application이나 다른 high-level trace event API의 기반으로 직접 사용할 수도 있다.

이 절은 synthetic event definition의 동적 생성, kprobe·kretprobe event definition의 동적 생성, kernel code에서 synthetic event tracing, low-level `dynevent_cmd` API를 설명한다.

7. In-kernel trace event API
============================

In most cases, the command-line interface to trace events is more than
sufficient.  Sometimes, however, applications might find the need for
more complex relationships than can be expressed through a simple
series of linked command-line expressions, or putting together sets of
commands may be simply too cumbersome.  An example might be an
application that needs to 'listen' to the trace stream in order to
maintain an in-kernel state machine detecting, for instance, when an
illegal kernel state occurs in the scheduler.

The trace event subsystem provides an in-kernel API allowing modules
or other kernel code to generate user-defined 'synthetic' events at
will, which can be used to either augment the existing trace stream
and/or signal that a particular important state has occurred.

A similar in-kernel API is also available for creating kprobe and
kretprobe events.

Both the synthetic event and k/ret/probe event APIs are built on top
of a lower-level "dynevent_cmd" event command API, which is also
available for more specialized applications, or as the basis of other
higher-level trace event APIs.

The API provided for these purposes is describe below and allows the
following:

  - dynamically creating synthetic event definitions
  - dynamically creating kprobe and kretprobe event definitions
  - tracing synthetic events from in-kernel code
  - the low-level "dynevent_cmd" API

7.1 Synthetic event definition 동적 생성

kernel module이나 다른 kernel code에서 synthetic event를 만드는 방법은 두 가지다.

첫 번째 방법은 `synth_event_create()` 한 번으로 event를 만든다. event 이름과 field를 정의한 array를 전달하며, 성공하면 호출 직후 해당 이름과 field를 가진 synthetic event가 존재한다. 예시에서는 `sched_fields`와 그 원소 수, `THIS_MODULE`을 전달해 `schedtest`를 만든다.

`sched_fields`는 `struct synth_field_desc` array이며 각 원소가 type과 name으로 event field 하나를 설명한다. 사용할 수 있는 type은 `synth_field_size()`를 참조한다.

field name에 `[n]`이 있으면 static array로 간주한다. subscript 없는 `[]`가 있으면 dynamic array이며 실제 array를 담는 데 필요한 공간만 event에 사용한다.

event 공간은 field 값을 넣기 전에 예약하므로 dynamic array를 사용하면 아래에서 설명하는 piecewise in-kernel API는 사용할 수 없다. piecewise가 아닌 다른 in-kernel API는 dynamic array를 지원한다.

module 안에서 event를 만들 때는 module pointer를 `synth_event_create()`에 전달해야 한다. 그래야 module 제거 후 trace buffer에 읽을 수 없는 event가 남지 않는다. 생성이 끝나면 event object로 새 event를 기록할 준비가 된다.

두 번째 방법은 여러 단계로 event를 만든다. 미리 field array를 만들고 채울 필요 없이 동적으로 구성할 수 있다.

먼저 `synth_event_gen_cmd_start()` 또는 `synth_event_gen_cmd_array_start()`로 비어 있거나 일부만 정의된 synthetic event를 시작한다. 전자는 event 이름과 하나 이상의 `type`, `field_name` 쌍을 받고, 후자는 event 이름과 `struct synth_field_desc` array를 받는다. 시작 전에 `synth_event_cmd_init()`으로 `dynevent_cmd` object와 command buffer를 초기화해야 한다.

event object가 생성되면 `synth_event_add_field()`에 command object, field type, field name을 전달해 field를 하나씩 추가한다. 지원 type은 `synth_field_size()`에서 확인하며 `[n]`이 들어간 이름은 array다.

`synth_event_add_fields()`에는 `synth_field_desc` array와 개수를 전달해 여러 field를 한 번에 추가할 수 있다. 이미 `type field_name` 형태의 문자열이 있다면 `synth_event_add_field_str()`로 그대로 추가할 수 있고 함수가 끝에 `;`도 자동으로 붙인다.

모든 field를 추가한 뒤 `synth_event_gen_cmd_end()`를 호출해 event를 finalize하고 등록한다. 이 시점부터 새 event를 tracing에 사용할 수 있다.

7.1 Dynamically creating synthetic event definitions
----------------------------------------------------

There are a couple ways to create a new synthetic event from a kernel
module or other kernel code.

The first creates the event in one step, using synth_event_create().
In this method, the name of the event to create and an array defining
the fields is supplied to synth_event_create().  If successful, a
synthetic event with that name and fields will exist following that
call.  For example, to create a new "schedtest" synthetic event::

  ret = synth_event_create("schedtest", sched_fields,
                           ARRAY_SIZE(sched_fields), THIS_MODULE);

The sched_fields param in this example points to an array of struct
synth_field_desc, each of which describes an event field by type and
name::

  static struct synth_field_desc sched_fields[] = {
        { .type = "pid_t",              .name = "next_pid_field" },
        { .type = "char[16]",           .name = "next_comm_field" },
        { .type = "u64",                .name = "ts_ns" },
        { .type = "u64",                .name = "ts_ms" },
        { .type = "unsigned int",       .name = "cpu" },
        { .type = "char[64]",           .name = "my_string_field" },
        { .type = "int",                .name = "my_int_field" },
  };

See synth_field_size() for available types.

If field_name contains [n], the field is considered to be a static array.

If field_names contains[] (no subscript), the field is considered to
be a dynamic array, which will only take as much space in the event as
is required to hold the array.

Because space for an event is reserved before assigning field values
to the event, using dynamic arrays implies that the piecewise
in-kernel API described below can't be used with dynamic arrays.  The
other non-piecewise in-kernel APIs can, however, be used with dynamic
arrays.

If the event is created from within a module, a pointer to the module
must be passed to synth_event_create().  This will ensure that the
trace buffer won't contain unreadable events when the module is
removed.

At this point, the event object is ready to be used for generating new
events.

In the second method, the event is created in several steps.  This
allows events to be created dynamically and without the need to create
and populate an array of fields beforehand.

To use this method, an empty or partially empty synthetic event should
first be created using synth_event_gen_cmd_start() or
synth_event_gen_cmd_array_start().  For synth_event_gen_cmd_start(),
the name of the event along with one or more pairs of args each pair
representing a 'type field_name;' field specification should be
supplied.  For synth_event_gen_cmd_array_start(), the name of the
event along with an array of struct synth_field_desc should be
supplied. Before calling synth_event_gen_cmd_start() or
synth_event_gen_cmd_array_start(), the user should create and
initialize a dynevent_cmd object using synth_event_cmd_init().

For example, to create a new "schedtest" synthetic event with two
fields::

  struct dynevent_cmd cmd;
  char *buf;

  /* Create a buffer to hold the generated command */
  buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);

  /* Before generating the command, initialize the cmd object */
  synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);

  ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
                                  "pid_t", "next_pid_field",
                                  "u64", "ts_ns");

Alternatively, using an array of struct synth_field_desc fields
containing the same information::

  ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
                                        fields, n_fields);

Once the synthetic event object has been created, it can then be
populated with more fields.  Fields are added one by one using
synth_event_add_field(), supplying the dynevent_cmd object, a field
type, and a field name.  For example, to add a new int field named
"intfield", the following call should be made::

  ret = synth_event_add_field(&cmd, "int", "intfield");

See synth_field_size() for available types. If field_name contains [n]
the field is considered to be an array.

A group of fields can also be added all at once using an array of
synth_field_desc with add_synth_fields().  For example, this would add
just the first four sched_fields::

  ret = synth_event_add_fields(&cmd, sched_fields, 4);

If you already have a string of the form 'type field_name',
synth_event_add_field_str() can be used to add it as-is; it will
also automatically append a ';' to the string.

Once all the fields have been added, the event should be finalized and
registered by calling the synth_event_gen_cmd_end() function::

  ret = synth_event_gen_cmd_end(&cmd);

At this point, the event object is ready to be used for tracing new
events.

7.2 Kernel code에서 synthetic event tracing

synthetic event를 기록하는 방법도 여러 가지다. 첫 번째는 variable argument를 받는 `synth_event_trace()` 또는 value array를 받는 `synth_event_trace_array()`로 한 번에 모든 field를 기록하는 방법이다.

두 번째는 미리 value array나 argument list를 만들지 않고 `synth_event_trace_start()`와 `synth_event_trace_end()` 사이에서 `synth_event_add_next_val()` 또는 `synth_event_add_val()`로 값을 하나씩 넣는 방법이다.

7.2 Tracing synthetic events from in-kernel code
------------------------------------------------

To trace a synthetic event, there are several options.  The first
option is to trace the event in one call, using synth_event_trace()
with a variable number of values, or synth_event_trace_array() with an
array of values to be set.  A second option can be used to avoid the
need for a pre-formed array of values or list of arguments, via
synth_event_trace_start() and synth_event_trace_end() along with
synth_event_add_next_val() or synth_event_add_val() to add the values
piecewise.

7.2.1 Synthetic event를 한 번에 기록

synthetic event의 모든 field를 한 번에 기록하려면 `synth_event_trace()` 또는 `synth_event_trace_array()`를 사용한다.

`synth_event_trace()`에는 synthetic event를 나타내는 `trace_event_file`, field마다 하나씩의 variable `u64` argument, 전달하는 값의 개수를 넘긴다. `trace_event_file`은 `trace_get_event_file()`에 synthetic event 이름, system 이름 `synthetic`, trace instance 이름을 주어 얻는다. global trace array를 사용하면 instance 이름은 `NULL`이다.

모든 값은 `u64`로 cast해야 한다. string 값은 `u64`로 cast한 string pointer이며, 함수는 이 pointer를 이용해 event에 예약된 string 공간으로 내용을 복사한다.

`synth_event_trace_array()`도 같은 결과를 내지만 field마다 하나씩의 `u64` 값을 담은 array를 받는다. array 원소 수는 synthetic event의 field 수와 같아야 하고 원소 순서도 field 순서와 정확히 같아야 한다. string pointer 처리 규칙도 동일하다.

synthetic event를 기록하려면 `trace_event_file` pointer가 필요하다. `trace_get_event_file()`은 지정한 trace instance에서 file을 찾는 동시에 그 instance가 사라지지 않도록 붙잡는다.

event가 어떤 방식으로든 활성화돼 있지 않으면 trace buffer에 나타나지 않는다. kernel에서 활성화하려면 `trace_array_set_clr_event()`를 사용한다. 이 함수는 synthetic event 전용이 아니므로 system 이름 `synthetic`을 명시해야 한다. 마지막 boolean이 `true`면 활성화하고 `false`면 비활성화한다.

활성화 뒤 `synth_event_trace_array()`를 호출하면 event가 trace buffer에 기록된다.

synthetic event를 제거할 때는 먼저 비활성화하고 `trace_put_event_file()`로 trace instance 참조를 반환한다. 이 단계가 성공하면 `synth_event_delete()`로 event를 삭제한다.

7.2.1 Tracing a synthetic event all at once
-------------------------------------------

To trace a synthetic event all at once, the synth_event_trace() or
synth_event_trace_array() functions can be used.

The synth_event_trace() function is passed the trace_event_file
representing the synthetic event (which can be retrieved using
trace_get_event_file() using the synthetic event name, "synthetic" as
the system name, and the trace instance name (NULL if using the global
trace array)), along with an variable number of u64 args, one for each
synthetic event field, and the number of values being passed.

So, to trace an event corresponding to the synthetic event definition
above, code like the following could be used::

  ret = synth_event_trace(create_synth_test, 7, /* number of values */
                          444,             /* next_pid_field */
                          (u64)"clackers", /* next_comm_field */
                          1000000,         /* ts_ns */
                          1000,            /* ts_ms */
                          smp_processor_id(),/* cpu */
                          (u64)"Thneed",   /* my_string_field */
                          999);            /* my_int_field */

All vals should be cast to u64, and string vals are just pointers to
strings, cast to u64.  Strings will be copied into space reserved in
the event for the string, using these pointers.

Alternatively, the synth_event_trace_array() function can be used to
accomplish the same thing.  It is passed the trace_event_file
representing the synthetic event (which can be retrieved using
trace_get_event_file() using the synthetic event name, "synthetic" as
the system name, and the trace instance name (NULL if using the global
trace array)), along with an array of u64, one for each synthetic
event field.

To trace an event corresponding to the synthetic event definition
above, code like the following could be used::

  u64 vals[7];

  vals[0] = 777;                  /* next_pid_field */
  vals[1] = (u64)"tiddlywinks";   /* next_comm_field */
  vals[2] = 1000000;              /* ts_ns */
  vals[3] = 1000;                 /* ts_ms */
  vals[4] = smp_processor_id();   /* cpu */
  vals[5] = (u64)"thneed";        /* my_string_field */
  vals[6] = 398;                  /* my_int_field */

The 'vals' array is just an array of u64, the number of which must
match the number of field in the synthetic event, and which must be in
the same order as the synthetic event fields.

All vals should be cast to u64, and string vals are just pointers to
strings, cast to u64.  Strings will be copied into space reserved in
the event for the string, using these pointers.

In order to trace a synthetic event, a pointer to the trace event file
is needed.  The trace_get_event_file() function can be used to get
it - it will find the file in the given trace instance (in this case
NULL since the top trace array is being used) while at the same time
preventing the instance containing it from going away::

       schedtest_event_file = trace_get_event_file(NULL, "synthetic",
                                                   "schedtest");

Before tracing the event, it should be enabled in some way, otherwise
the synthetic event won't actually show up in the trace buffer.

To enable a synthetic event from the kernel, trace_array_set_clr_event()
can be used (which is not specific to synthetic events, so does need
the "synthetic" system name to be specified explicitly).

To enable the event, pass 'true' to it::

       trace_array_set_clr_event(schedtest_event_file->tr,
                                 "synthetic", "schedtest", true);

To disable it pass false::

       trace_array_set_clr_event(schedtest_event_file->tr,
                                 "synthetic", "schedtest", false);

Finally, synth_event_trace_array() can be used to actually trace the
event, which should be visible in the trace buffer afterwards::

       ret = synth_event_trace_array(schedtest_event_file, vals,
                                     ARRAY_SIZE(vals));

To remove the synthetic event, the event should be disabled, and the
trace instance should be 'put' back using trace_put_event_file()::

       trace_array_set_clr_event(schedtest_event_file->tr,
                                 "synthetic", "schedtest", false);
       trace_put_event_file(schedtest_event_file);

If those have been successful, synth_event_delete() can be called to
remove the event::

       ret = synth_event_delete("schedtest");

7.2.2 Synthetic event를 단계별로 기록

piecewise 방식은 `synth_event_trace_start()`로 synthetic event trace를 연다. synthetic event의 `trace_event_file`과 `struct synth_event_trace_state` pointer를 전달한다. state object는 사용 전에 zero로 초기화되며 이후 호출 사이의 상태를 유지한다.

event를 연다는 것은 trace buffer에 필요한 공간을 예약한다는 뜻이다. 이후 개별 field 값을 넣는 방법은 event field 순서대로 연속 대입하는 방법과 field 이름으로 찾아 대입하는 방법 두 가지다.

순서대로 넣는 `synth_event_add_next_val()`은 lookup이 필요 없다. 매 호출에 같은 trace state와 다음 field 값을 전달하며, 값을 넣을 때마다 cursor가 다음 field로 이동한다. 모든 field를 정의 순서대로 끝까지 채워야 한다.

`synth_event_add_val()`은 같은 trace state에 field 이름과 값을 전달하므로 원하는 순서로 대입할 수 있다. 대신 field마다 name lookup 비용이 든다. 선택 기준은 대입 순서의 유연성과 lookup 비용 사이의 tradeoff다.

event 한 건을 기록하는 동안 `synth_event_add_next_val()`과 `synth_event_add_val()`을 섞어 사용할 수 없다. 둘 중 하나만 선택해야 한다.

event는 `synth_event_trace_end()`로 닫아야 실제로 trace된다. 이전 add 호출 중 잘못된 field 이름 등의 이유로 실패한 호출이 있어도 마지막에는 반드시 `synth_event_trace_end()`를 호출해야 한다.

7.2.2 Tracing a synthetic event piecewise
-----------------------------------------

To trace a synthetic using the piecewise method described above, the
synth_event_trace_start() function is used to 'open' the synthetic
event trace::

       struct synth_event_trace_state trace_state;

       ret = synth_event_trace_start(schedtest_event_file, &trace_state);

It's passed the trace_event_file representing the synthetic event
using the same methods as described above, along with a pointer to a
struct synth_event_trace_state object, which will be zeroed before use and
used to maintain state between this and following calls.

Once the event has been opened, which means space for it has been
reserved in the trace buffer, the individual fields can be set.  There
are two ways to do that, either one after another for each field in
the event, which requires no lookups, or by name, which does.  The
tradeoff is flexibility in doing the assignments vs the cost of a
lookup per field.

To assign the values one after the other without lookups,
synth_event_add_next_val() should be used.  Each call is passed the
same synth_event_trace_state object used in the synth_event_trace_start(),
along with the value to set the next field in the event.  After each
field is set, the 'cursor' points to the next field, which will be set
by the subsequent call, continuing until all the fields have been set
in order.  The same sequence of calls as in the above examples using
this method would be (without error-handling code)::

       /* next_pid_field */
       ret = synth_event_add_next_val(777, &trace_state);

       /* next_comm_field */
       ret = synth_event_add_next_val((u64)"slinky", &trace_state);

       /* ts_ns */
       ret = synth_event_add_next_val(1000000, &trace_state);

       /* ts_ms */
       ret = synth_event_add_next_val(1000, &trace_state);

       /* cpu */
       ret = synth_event_add_next_val(smp_processor_id(), &trace_state);

       /* my_string_field */
       ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);

       /* my_int_field */
       ret = synth_event_add_next_val(395, &trace_state);

To assign the values in any order, synth_event_add_val() should be
used.  Each call is passed the same synth_event_trace_state object used in
the synth_event_trace_start(), along with the field name of the field
to set and the value to set it to.  The same sequence of calls as in
the above examples using this method would be (without error-handling
code)::

       ret = synth_event_add_val("next_pid_field", 777, &trace_state);
       ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
                                 &trace_state);
       ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
       ret = synth_event_add_val("ts_ms", 1000, &trace_state);
       ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
       ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
                                 &trace_state);
       ret = synth_event_add_val("my_int_field", 3999, &trace_state);

Note that synth_event_add_next_val() and synth_event_add_val() are
incompatible if used within the same trace of an event - either one
can be used but not both at the same time.

Finally, the event won't be actually traced until it's 'closed',
which is done using synth_event_trace_end(), which takes only the
struct synth_event_trace_state object used in the previous calls::

       ret = synth_event_trace_end(&trace_state);

Note that synth_event_trace_end() must be called at the end regardless
of whether any of the add calls failed (say due to a bad field name
being passed in).

7.3 Kprobe·kretprobe event definition 동적 생성

kernel code에서 kprobe 또는 kretprobe trace event를 만들려면 `kprobe_event_gen_cmd_start()`나 `kretprobe_event_gen_cmd_start()`를 사용한다.

kprobe event는 `kprobe_event_gen_cmd_start()`로 비어 있거나 일부만 채운 event를 시작한다. event 이름, probe 위치, probe field를 나타내는 하나 이상의 argument를 전달한다. 시작 전에 `kprobe_event_cmd_init()`으로 `dynevent_cmd` object와 command buffer를 초기화해야 한다.

예시는 `do_sys_open` 위치에 `gen_kprobe_test`를 만들고 `dfd=%ax`, `filename=%dx` 두 field를 처음부터 정의한다.

event object를 만든 뒤 `kprobe_event_add_fields()`에 command object와 variable argument 형태의 probe field를 넘겨 field를 더 추가한다. 예시에서는 `flags=%cx`와 `mode=+4($stack)`을 추가한다.

모든 field를 추가하면 시작한 종류에 따라 `kprobe_event_gen_cmd_end()` 또는 `kretprobe_event_gen_cmd_end()`로 finalize하고 등록한다. 이후 새 event로 trace를 기록할 수 있다.

kretprobe event도 `kretprobe_event_gen_cmd_start()`에 probe 이름, 위치, `$retval` 같은 추가 parameter를 전달해 같은 방식으로 만든다.

새 kprobe event를 활성화하려면 `trace_get_event_file(NULL, "kprobes", event_name)`으로 file을 얻고 `trace_array_set_clr_event()`에 system `kprobes`와 `true`를 전달한다. 사용을 마치면 `trace_put_event_file()`로 file 참조를 반환하고 `kprobe_event_delete()`로 event를 삭제한다.

7.3 Dynamically creating kprobe and kretprobe event definitions
---------------------------------------------------------------

To create a kprobe or kretprobe trace event from kernel code, the
kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()
functions can be used.

To create a kprobe event, an empty or partially empty kprobe event
should first be created using kprobe_event_gen_cmd_start().  The name
of the event and the probe location should be specified along with one
or args each representing a probe field should be supplied to this
function.  Before calling kprobe_event_gen_cmd_start(), the user
should create and initialize a dynevent_cmd object using
kprobe_event_cmd_init().

For example, to create a new "schedtest" kprobe event with two fields::

  struct dynevent_cmd cmd;
  char *buf;

  /* Create a buffer to hold the generated command */
  buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);

  /* Before generating the command, initialize the cmd object */
  kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);

  /*
   * Define the gen_kprobe_test event with the first 2 kprobe
   * fields.
   */
  ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
                                   "dfd=%ax", "filename=%dx");

Once the kprobe event object has been created, it can then be
populated with more fields.  Fields can be added using
kprobe_event_add_fields(), supplying the dynevent_cmd object along
with a variable arg list of probe fields.  For example, to add a
couple additional fields, the following call could be made::

  ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");

Once all the fields have been added, the event should be finalized and
registered by calling the kprobe_event_gen_cmd_end() or
kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe
or kretprobe command was started::

  ret = kprobe_event_gen_cmd_end(&cmd);

or::

  ret = kretprobe_event_gen_cmd_end(&cmd);

At this point, the event object is ready to be used for tracing new
events.

Similarly, a kretprobe event can be created using
kretprobe_event_gen_cmd_start() with a probe name and location and
additional params such as $retval::

  ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
                                      "do_sys_open", "$retval");

Similar to the synthetic event case, code like the following can be
used to enable the newly created kprobe event::

  gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");

  ret = trace_array_set_clr_event(gen_kprobe_test->tr,
                                  "kprobes", "gen_kprobe_test", true);

Finally, also similar to synthetic events, the following code can be
used to give the kprobe event file back and delete the event::

  trace_put_event_file(gen_kprobe_test);

  ret = kprobe_event_delete("gen_kprobe_test");

7.4 Low-level dynevent_cmd API

kernel 내부 synthetic event와 kprobe interface는 모두 더 낮은 단계의 `dynevent_cmd` interface 위에 구현된다. 이 interface는 synthetic·kprobe interface 같은 high-level API의 기반이며, 두 구현을 사용 예제로 참고할 수 있다.

핵심은 trace event command를 만드는 범용 layer를 제공하는 것이다. 생성된 command string은 trace event subsystem에 이미 존재하는 command parser와 event creation code로 전달되어 대응하는 trace event를 만든다.

high-level interface는 `struct dynevent_cmd` object를 만들고 `dynevent_arg_add()`와 `dynevent_arg_pair_add()`로 command string을 조립한 뒤 `dynevent_create()`로 실행한다.

첫 단계는 `dynevent_cmd` instance와 buffer를 만들고 초기화하는 것이다. 초기화에는 사용자 제공 buffer와 길이, dynevent type ID, event 전용 `run_command()` callback이 필요하다.

`MAX_DYNEVENT_CMD_LEN`은 이 buffer 크기로 사용할 수 있다. 2 KiB라 stack에 편하게 두기에는 대체로 크므로 예시처럼 동적으로 할당한다. type ID는 뒤의 API 호출이 올바른 command type을 대상으로 하는지 검사하는 데 쓰이고, callback은 event 전용 command를 실제로 실행한다.

초기화가 끝나면 argument 추가 함수를 연속 호출해 command string을 만든다.

단일 argument는 `struct dynevent_arg`를 `dynevent_arg_init()`으로 초기화하고 문자열을 지정한 뒤 `dynevent_arg_add()`로 추가한다. 예시의 `NULL`, `0` 초기화 parameter는 optional sanity-check function도 없고 argument 끝에 붙일 separator도 없다는 뜻이다. 문자열은 whitespace로 구분된 argument로 command에 이어 붙는다.

두 component를 한 단위로 결합하는 argument에는 `struct dynevent_arg_pair`를 쓴다. `type field_name;`이나 `flags=%cx` 같은 expression이 예다. `dynevent_arg_pair_init()`에는 argument를 검사할 callback, 두 component 사이 operator, 끝에 붙일 separator를 지정한다.

예시의 callback은 pair 어느 쪽도 `NULL`이 아닌지 같은 조건을 검사할 수 있다. component 사이 operator는 없고 pair 끝에는 `;`가 붙는다. `lhs`와 `rhs`를 지정한 뒤 `dynevent_arg_pair_add()`로 command에 추가한다.

공백, delimiter, argument check 없이 문자열을 그대로 붙이려면 `dynevent_str_add()`를 사용한다.

완성 문자열이 `cmd->maxlen`을 넘지 않는 동안 `dynevent_*_add()`를 필요한 만큼 호출할 수 있다. 모든 argument를 넣은 뒤 `dynevent_create()`를 호출해 command를 실행한다. 반환 값이 0이면 dynamic event가 생성되어 사용할 준비가 된 것이다. 세부 계약은 각 `dynevent_cmd` 함수 정의를 참조한다.

7.4 The "dynevent_cmd" low-level API
------------------------------------

Both the in-kernel synthetic event and kprobe interfaces are built on
top of a lower-level "dynevent_cmd" interface.  This interface is
meant to provide the basis for higher-level interfaces such as the
synthetic and kprobe interfaces, which can be used as examples.

The basic idea is simple and amounts to providing a general-purpose
layer that can be used to generate trace event commands.  The
generated command strings can then be passed to the command-parsing
and event creation code that already exists in the trace event
subsystem for creating the corresponding trace events.

In a nutshell, the way it works is that the higher-level interface
code creates a struct dynevent_cmd object, then uses a couple
functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up
a command string, which finally causes the command to be executed
using the dynevent_create() function.  The details of the interface
are described below.

The first step in building a new command string is to create and
initialize an instance of a dynevent_cmd.  Here, for instance, we
create a dynevent_cmd on the stack and initialize it::

  struct dynevent_cmd cmd;
  char *buf;
  int ret;

  buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);

  dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
                    foo_event_run_command);

The dynevent_cmd initialization needs to be given a user-specified
buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
for this purpose - at 2k it's generally too big to be comfortably put
on the stack, so is dynamically allocated), a dynevent type id, which
is meant to be used to check that further API calls are for the
correct command type, and a pointer to an event-specific run_command()
callback that will be called to actually execute the event-specific
command function.

Once that's done, the command string can by built up by successive
calls to argument-adding functions.

To add a single argument, define and initialize a struct dynevent_arg
or struct dynevent_arg_pair object.  Here's an example of the simplest
possible arg addition, which is simply to append the given string as
a whitespace-separated argument to the command::

  struct dynevent_arg arg;

  dynevent_arg_init(&arg, NULL, 0);

  arg.str = name;

  ret = dynevent_arg_add(cmd, &arg);

The arg object is first initialized using dynevent_arg_init() and in
this case the parameters are NULL or 0, which means there's no
optional sanity-checking function or separator appended to the end of
the arg.

Here's another more complicated example using an 'arg pair', which is
used to create an argument that consists of a couple components added
together as a unit, for example, a 'type field_name;' arg or a simple
expression arg e.g. 'flags=%cx'::

  struct dynevent_arg_pair arg_pair;

  dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');

  arg_pair.lhs = type;
  arg_pair.rhs = name;

  ret = dynevent_arg_pair_add(cmd, &arg_pair);

Again, the arg_pair is first initialized, in this case with a callback
function used to check the sanity of the args (for example, that
neither part of the pair is NULL), along with a character to be used
to add an operator between the pair (here none) and a separator to be
appended onto the end of the arg pair (here ';').

There's also a dynevent_str_add() function that can be used to simply
add a string as-is, with no spaces, delimiters, or arg check.

Any number of dynevent_*_add() calls can be made to build up the string
(until its length surpasses cmd->maxlen).  When all the arguments have
been added and the command string is complete, the only thing left to
do is run the command, which happens by simply calling
dynevent_create()::

  ret = dynevent_create(&cmd);

At that point, if the return value is 0, the dynamic event has been
created and is ready to use.

See the dynevent_cmd function definitions themselves for the details
of the API.