Syzkaller

Syzkaller is a coverage-guided, kernel fuzzer for Linux and other OSes such as BSDs and Windows. It provides a framework for running many VMs with a kernel fuzzer inside. When crashes are detected, it will automatically “minimize” the crashing input, producing a C or Syzprog program as a PoC that triggers the crash.

Syzkaller can be customized to fuzz specific syscalls if you want to target a region of the kernel. In this post, we will explore general fuzzing of the whole Linux kernel.

A high-level diagram of Syzkaller is as follows:

syzkaller

sys-manager runs on the host system, orchistrating the launching of VMs and controlling the fuzzers inside of them, while maintaining a database of crashes and corpuses (inputs which drive the target program to different states). Inside the VMs, sys-fuzzer and sys-executor run, invoking syscalls with fuzzed inputs and recording the coverage information emitted from the kernel via KCOV. Monitoring the coverage information is critical as it provides feedback to the fuzzer on whether or not it’s driving the kernel to new states in the code paths.

Once a crash is found, Syzkaller works to reproduce and “minimize” it by finding the simplest possible sequence of events to trigger it, emitting a Sysprog/C file to trigger it.

ASAN

AddressSanitizer (or ASAN) is a memory error detector from the Clang framework which is added into programs via compiler instrumentation. It can detect many types of memory bugs such as out-of-bounds access, use-after-free, and double-free. Syskaller uses it to identify vulnerabilities in the kernel at runtime with minimal performance impact. ASAN works by tracking all memory allocations and frees to determine when bugs occur in the dynamic memory lifecycle.

Setup

First, download and build syzkaller:

git clone https://github.com/google/syzkaller
cd syzkaller
make

Next, build a version of the Linux kernel and make sure the following options are enabled in the kernel configuration:

CONFIG_KCOV=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_CONFIGFS_FS=y
CONFIG_DEBUG_INFO=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_BLK=y
CONFIG_DEBUG_FS=y

Now it is time to build a root image for our fuzzing VM containing an SSH key syzkaller will use to communicate with the VM

tools/create-image.sh

Finally, create a configuration file for syzkaller. The more CPU/memory available your system has, the greater you can increase the procs/count/cpu fields. Monitor the host’s CPU consumption and scale accordingly - oversaturating the CPU can hurt the performance which can be monitored in the web UI under the “execs/sec” field.

{
  "target": "linux/amd64",
  "http": "127.0.0.1:56741",
  "workdir": "/path/to/workdir",
  "kernel_obj": "/path/to/linux",
  "image": "/path/to/disk.img",
  "sshkey": "/path/to/ssh-key",
  "syzkaller": "/path/to/syzkaller",
  "procs": 4,
  "type": "qemu",
  "vm": {
    "count": 2,
    "kernel": "/path/to/linux/arch/x86/boot/bzImage",
    "cpu": 2,
    "mem": 2048
  }
}

Running

Syzkaller is ready to run, launch via

bin/syz-manager -config=my.cfg

And monitor it with the web UI at

http://127.0.0.1:56741

As time passes (be patient!), crashes will be recorded in the UI. It is common to have a lot of noisy crashes which are meaningless and are prefixed with “SYZFAIL” and “INFO”. What is important is to look at the “Rank” column, looking for crashes with a high score (> 15).

Additionally, you can monitor how well the fuzzer is doing by inspecting the “coverage” tab which approximates what percentage of the codebase is being reached by the fuzzer. Large amounts (like 80%) are rare, but if it’s stuck around 5% then the fuzzer is not effectively exploring the code paths in the kernel.

Syzprog Output

If you get a crash, Syzkaller will log the crashing inputs in the form of a ‘Syzprog’. Syzprog is a high-level script which is generated by Syzkaller to show the syscalls and their arguments used during fuzzing. It’s a simpler representation of what is going on compared to a full C program.

r0 = openat(0xffffffffffffff9c, & (0x7f0000000000)='test_file\x00', 0x42, 0x1b6)
write(r0, &(0x7f0000000100)="A" x 50, 0x32)
close(r0)
  1. r0 = openat(...): The openat() syscall is invoked, returning the file descriptor to the variable r0.
    • The syscall is defined as: int openat(int dirfd, const char *pathname, int flags, mode_t mode)
      • dirfd: Address of 0xffffffffffffff9c containing the directory file descriptor
      • pathname: A buffer created at 0x7f0000000000 which contains the string text_file\x00
      • flags: 0x42
      • mode: 0x1b6
  2. write(...): Here we write to the r0 fd returned from the previous line
    • Syscall definition: ssize_t write(int fd, const void buf[.count], size_t count)
      • fd: The r0 variable from earlier
      • buf: &(0x7f0000000100)="A" x 50 The buffer is an address of a memory region containing 50 ‘A’ characters
  3. close(r0): This closes the file with the r0 fd

If you want to generate a C program from Syzprog, you can use bin/sys-prog2c -prog <sysprog file> to generate it.

Testing Vulnerability

Launch test VM:

qemu-system-x86_64 -kernel linux-6.19.3/arch/x86/boot/bzImage \
    -append "console=ttyS0 root=/dev/sda debug net.ifnames=0 earlyprintk=serial slub_debug=QUZ" \
    -hda trixie.img -netdev user,id=net0,hostfwd=tcp::10021-:22 -device e1000,netdev=net0 \
    -enable-kvm -nogrpahic -m 4G -smp 4

Compile exploit:

cd /syzkaller/workdir/crashes/af353992365be0f90f455ddd2f634ef3f7b7fa47
gcc repro.cprog -o exploit
scp -P 10021 -i trixie.id_rsa exploit [email protected]:

Login to VM & run:

ssh -i trixie.id_rsa -p 10021 [email protected]
./exploit

Sometimes it’s useful to run in multi-shot mode where KASAN won’t exit on the first crash:

qemu-system-x86_64 -kernel linux-6.19.3/arch/x86/boot/bzImage -append \
    "kasan_multi_shot kasan.fault=report panic_on_warn=0 console=ttyS0 root=/dev/sda debug net.ifnames=0 earlyprintk=serial slub_debug=QUZ" \
    -hda trixie.img -netdev user,id=net0,hostfwd=tcp::10021-:22 -device e1000,netdev=net0 -enable-kvm -nographic -m 4G -smp 4

Bugs Found

Use-After-Free in perf_event struct

Patched in https://github.com/torvalds/linux/commit/77de62ad3de3

[   24.781312] BUG: KASAN: slab-use-after-free in task_work_run+0x181/0x230
[   24.781589] Read of size 8 at addr ffff8881107bdd90 by task a.out/5185
[   24.781908] Call Trace:
[   24.781910]  <TASK>
[   24.781912]  dump_stack_lvl+0x97/0xe0
[   24.781920]  print_report+0x170/0x4f3
[   24.781929]  ? __pfx__raw_spin_lock_irqsave+0x10/0x10
[   24.781938]  ? __virt_addr_valid+0x2d1/0x3d0
[   24.781944]  kasan_report+0xda/0x110
[   24.781952]  ? task_work_run+0x181/0x230
[   24.781957]  ? task_work_run+0x181/0x230
[   24.781962]  task_work_run+0x181/0x230
[   24.781967]  ? __pfx_task_work_run+0x10/0x10
...
[   24.789342] Allocated by task 5185 on cpu 0 at 24.765520s:
[   24.789552]  kasan_save_stack+0x30/0x50
[   24.789700]  kasan_save_track+0x17/0x60
[   24.789855]  __kasan_slab_alloc+0x63/0x80
[   24.790012]  kmem_cache_alloc_node_noprof+0x139/0x510
[   24.790211]  perf_event_alloc.part.0+0x125/0x4980
[   24.790498]  __do_sys_perf_event_open+0x620/0x25d0
[   24.790684]  do_syscall_64+0xa4/0x310
[   24.790831]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
...
[   24.791089] Freed by task 5194 on cpu 0 at 24.780446s:
[   24.791291]  kasan_save_stack+0x30/0x50
[   24.791463]  kasan_save_track+0x17/0x60
[   24.791610]  kasan_save_free_info+0x3b/0x70
[   24.791774]  __kasan_slab_free+0x47/0x70
[   24.791927]  kmem_cache_free+0xe8/0x4c0
[   24.792075]  rcu_core+0x5e6/0x19d0
[   24.792211]  handle_softirqs+0x17e/0x480
...
[   24.799763] The buggy address belongs to the object at ffff8881107bda08
[   24.799763]  which belongs to the cache perf_event of size 1136
[   24.800240] The buggy address is located 904 bytes inside of
[   24.800240]  freed 1136-byte region [ffff8881107bda08, ffff8881107bde78)
Show full crash log
[   24.781032] ==================================================================
[   24.781312] BUG: KASAN: slab-use-after-free in task_work_run+0x181/0x230
[   24.781589] Read of size 8 at addr ffff8881107bdd90 by task a.out/5185
[   24.781831] 
[   24.781898] CPU: 0 UID: 0 PID: 5185 Comm: a.out Not tainted 6.19.3 #1 PREEMPT(voluntary) 
[   24.781905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.17.0-2-2 04/01/2014
[   24.781908] Call Trace:
[   24.781910]  <TASK>
[   24.781912]  dump_stack_lvl+0x97/0xe0
[   24.781920]  print_report+0x170/0x4f3
[   24.781929]  ? __pfx__raw_spin_lock_irqsave+0x10/0x10
[   24.781938]  ? __virt_addr_valid+0x2d1/0x3d0
[   24.781944]  kasan_report+0xda/0x110
[   24.781952]  ? task_work_run+0x181/0x230
[   24.781957]  ? task_work_run+0x181/0x230
[   24.781962]  task_work_run+0x181/0x230
[   24.781967]  ? __pfx_task_work_run+0x10/0x10
[   24.781972]  ? exit_fs+0x133/0x190
[   24.781979]  do_exit+0x7dc/0x2640
[   24.781985]  ? proc_coredump_connector+0x2d3/0x370
[   24.781992]  ? __pfx_proc_coredump_connector+0x10/0x10
[   24.781999]  ? __pfx_do_exit+0x10/0x10
[   24.782005]  do_group_exit+0xc9/0x270
[   24.782012]  get_signal+0x1aaa/0x1d20
[   24.782021]  ? __pfx_send_sig_perf+0x10/0x10
[   24.782028]  ? __pfx_get_signal+0x10/0x10
[   24.782037]  arch_do_signal_or_restart+0x93/0x770
[   24.782046]  ? __pfx_arch_do_signal_or_restart+0x10/0x10
[   24.782056]  exit_to_user_mode_loop+0x65/0x420
[   24.782065]  do_syscall_64+0x2ed/0x310
[   24.782072]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   24.782078] RIP: 0033:0x7fd0b3f86779
[   24.782083] Code: Unable to access opcode bytes at 0x7fd0b3f8674f.
[   24.782085] RSP: 002b:00007fff8ffa3a68 EFLAGS: 00000297 ORIG_RAX: 000000000000012a
[   24.782091] RAX: 0000000000000004 RBX: 00007fff8ffa3e38 RCX: 00007fd0b3f86779
[   24.782095] RDX: ffefffffffffffff RSI: 0000000000000000 RDI: 0000200000000100
[   24.782098] RBP: 00007fff8ffa3c80 R08: 0000000000000008 R09: 0000559075e41d58
[   24.782102] R10: ffffffffffffffff R11: 0000000000000297 R12: 0000000000000000
[   24.782106] R13: 00007fff8ffa3e48 R14: 00007fd0b40b0000 R15: 0000559075e41d58
[   24.782110]  </TASK>
[   24.782112] 
[   24.789342] Allocated by task 5185 on cpu 0 at 24.765520s:
[   24.789552]  kasan_save_stack+0x30/0x50
[   24.789700]  kasan_save_track+0x17/0x60
[   24.789855]  __kasan_slab_alloc+0x63/0x80
[   24.790012]  kmem_cache_alloc_node_noprof+0x139/0x510
[   24.790211]  perf_event_alloc.part.0+0x125/0x4980
[   24.790498]  __do_sys_perf_event_open+0x620/0x25d0
[   24.790684]  do_syscall_64+0xa4/0x310
[   24.790831]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   24.791025] 
[   24.791089] Freed by task 5194 on cpu 0 at 24.780446s:
[   24.791291]  kasan_save_stack+0x30/0x50
[   24.791463]  kasan_save_track+0x17/0x60
[   24.791610]  kasan_save_free_info+0x3b/0x70
[   24.791774]  __kasan_slab_free+0x47/0x70
[   24.791927]  kmem_cache_free+0xe8/0x4c0
[   24.792075]  rcu_core+0x5e6/0x19d0
[   24.792211]  handle_softirqs+0x17e/0x480
[   24.792371]  do_softirq+0x3d/0x60
[   24.792502]  __local_bh_enable_ip+0x5d/0x60
[   24.792663]  copy_fpstate_to_sigframe+0x291/0x980
[   24.792847]  get_sigframe+0x3f3/0x950
[   24.792994]  x64_setup_rt_frame+0x133/0xcd0
[   24.793156]  arch_do_signal_or_restart+0x58c/0x770
[   24.793340]  irqentry_exit+0x108/0x470
[   24.793497]  asm_exc_page_fault+0x26/0x30
[   24.793650] 
[   24.793716] Last potentially related work creation:
[   24.793896]  kasan_save_stack+0x30/0x50
[   24.794044]  kasan_record_aux_stack+0x8c/0xa0
[   24.794217]  __call_rcu_common.constprop.0+0x72/0x9a0
[   24.794512]  perf_pending_task+0x14d/0x410
[   24.794674]  task_work_run+0x14f/0x230
[   24.794819]  do_exit+0x7dc/0x2640
[   24.794949]  do_group_exit+0xc9/0x270
[   24.795090]  get_signal+0x1aaa/0x1d20
[   24.795238]  arch_do_signal_or_restart+0x93/0x770
[   24.795427]  exit_to_user_mode_loop+0x65/0x420
[   24.795601]  do_syscall_64+0x2ed/0x310
[   24.795751]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   24.795945] 
[   24.796011] Second to last potentially related work creation:
[   24.796226]  kasan_save_stack+0x30/0x50
[   24.796481]  kasan_record_aux_stack+0x8c/0xa0
[   24.796656]  task_work_add+0x24a/0x350
[   24.796807]  __perf_event_overflow+0x6c1/0xa30
[   24.796979]  perf_swevent_event+0x1d7/0x400
[   24.797143]  perf_tp_event+0x49a/0x9a0
[   24.797294]  perf_trace_run_bpf_submit+0xe3/0x190
[   24.797599]  perf_trace_kmem_cache_alloc+0x15b/0x1b0
[   24.797790]  kmem_cache_alloc_noprof+0x2dc/0x4d0
[   24.797969]  __alloc_object+0x27/0x1e0
[   24.798116]  __create_object+0x21/0x90
[   24.798264]  kmem_cache_alloc_noprof+0x431/0x4d0
[   24.798547]  prepare_creds+0x2c/0x700
[   24.798693]  vfs_coredump+0x317/0x5500
[   24.798839]  get_signal+0x1b0c/0x1d20
[   24.798985]  arch_do_signal_or_restart+0x93/0x770
[   24.799175]  exit_to_user_mode_loop+0x65/0x420
[   24.799355]  do_syscall_64+0x2ed/0x310
[   24.799503]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   24.799698] 
[   24.799763] The buggy address belongs to the object at ffff8881107bda08
[   24.799763]  which belongs to the cache perf_event of size 1136
[   24.800240] The buggy address is located 904 bytes inside of
[   24.800240]  freed 1136-byte region [ffff8881107bda08, ffff8881107bde78)
[   24.800686] 
[   24.800749] The buggy address belongs to the physical page:
[   24.800954] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x1107b8
[   24.801253] head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[   24.801560] flags: 0x200000000000040(head|node=0|zone=2)
[   24.801759] page_type: f5(slab)
[   24.801886] raw: 0200000000000040 ffff888100050dc0 ffffea00001e8010 ffffea000405ea10
[   24.802169] raw: 0000000000000000 0000000000190019 00000000f5000000 0000000000000000
[   24.802576] head: 0200000000000040 ffff888100050dc0 ffffea00001e8010 ffffea000405ea10
[   24.802869] head: 0000000000000000 0000000000190019 00000000f5000000 0000000000000000
[   24.803162] head: 0200000000000003 ffffea000441ee01 00000000ffffffff 00000000ffffffff
[   24.803470] head: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[   24.803755] page dumped because: kasan: bad access detected
[   24.803968] 
[   24.804034] Memory state around the buggy address:
[   24.804218]  ffff8881107bdc80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   24.804495]  ffff8881107bdd00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   24.804766] >ffff8881107bdd80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   24.805034]                          ^
[   24.805183]  ffff8881107bde00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fc
[   24.805483]  ffff8881107bde80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[   24.805753] ==================================================================
[   24.806055] Disabling lock debugging due to kernel taint
[   24.806258] ==================================================================

Per the KASAN logs, there is a read of 8 bytes in task_work_run, accessing 904 bytes inside of the perf_event object which seems to be allocated here: perf_event_alloc and freed in rcu_core.

We can decode the stack trace to get line numbers with

$ linux-6.19.3/scripts/faddr2line linux-6.19.3/vmlinux task_work_run+0x181
task_work_run+0x181
task_work_run at kernel/task_work.c:214

To find what is 904 bytes inside of the perf_event struct and determine what variable inside is being accessed, we can use pahole which shows us it is the callback_head member:

$ pahole -C perf_event linux-6.19.3/kernel/events/core.o
904 bytes is: `struct callback_head       pending_task __attribute__((__aligned__(8))); /*   904    16 */`

callback_head is defined as a struct containing a member which is a function pointer (interesting!)

struct callback_head {
	struct callback_head *next;
	void (*func)(struct callback_head *head);
} __attribute__((aligned(sizeof(void *))));

Given the above, examining a snippet of task_work_run shows multiple invalid accesses of the freed memory, first a read of 8 bytes (64-bit pointer) at next = work->next triggering our KASAN crash. However, looking at the next line, we can see a call of a function pointer work->func() on freed memory!

void task_work_run(void) {
	struct task_struct *task = current;
	struct callback_head *work, *head, *next; // *work is our freed structure
	...
		do {
			next = work->next; // Access the already-freed memory (UAF)
			work->func(work); // Call function pointer on freed memory!
			work = next;
			cond_resched();
		} while (work);
}

This is particularly dangerous as this UaF potentially could be exploited by overwriting the function pointer from the freed callback_head struct and waiting for it to be called inside task_work_run.

Syzprog & PoC

The emitted syscall sequence needed to trigger this consists of perf_event_open calls with prlimit64 and mprotect. It does not trigger immediately as it is exploiting a race condition in which the memory is freed and then accessed again.

# {Threaded:false Repeat:true RepeatTimes:0 Procs:16 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:false IEEE802154:false Sysctl:true Swap:false UseTmpDir:true HandleSegv:true Trace:false CallComments:true LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}
perf_event_open(&(0x7f0000000100)={0x9, 0x80, 0x0, 0x3, 0x6, 0x3, 0x0, 0x4, 0x8a759, 0x1, 0x0, 0x0, 0x1, 0x0, 0x1, 0x0, 0x1, 0x0, 0x1, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0, 0x1, 0x0, 0x1, 0x1, 0x0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x1, 0x1, 0x1, 0x0, 0x1, 0x1, 0x0, 0x100065, 0x4, @perf_bp={0x0, 0x5}, 0x1002, 0x3, 0x8, 0x1, 0x1b, 0x9, 0x8, 0x0, 0x0, 0x0, 0xce}, 0x0, 0x1, 0xffffffffffffffff, 0x9)
prlimit64(0x0, 0x6, &(0x7f0000000100)={0x100000000002, 0x13f}, 0x0)
mprotect(&(0x7f0000000000/0x2000)=nil, 0x2000, 0xc)
perf_event_open(&(0x7f0000000100)={0x5, 0x80, 0x0, 0x5, 0x5, 0x6, 0x0, 0x3fffffff, 0x91026, 0xa, 0x0, 0x0, 0x1, 0x0, 0x1, 0x0, 0x1, 0x0, 0x1, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0, 0x1, 0x0, 0x1, 0x1, 0x0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x1, 0x1, 0x1, 0x0, 0x1, 0x1, 0x0, 0xed3, 0x4, @perf_bp={0x0, 0x3}, 0x1082, 0x2, 0x45, 0x1, 0x1000000000016, 0x9, 0x9, 0x0, 0xf2c, 0x0, 0xd3}, 0x0, 0xffefffffffffffff, 0xffffffffffffffff, 0x8) (rerun: 32)

Source code to trigger the vulnerability generated with Syzkaller is here: perf_event_uaf_poc.c

Other Crashes

When running KASAN in multi-shot mode, many more UAFs were found from the same vulnerability and are listed below.

Show all other UAF locations
event_sched_in+0x761/0x8b0:
event_sched_in at kernel/events/core.c:2716
event_sched_in+0x76e/0x8b0:
event_sched_in at kernel/events/core.c:2719 (discriminator 16)
event_sched_in+0x7a0/0x8b0:
event_sched_in at kernel/events/core.c:2712
event_sched_in+0x7ad/0x8b0:
event_sched_in at kernel/events/core.c:2757
event_sched_in+0x7c4/0x8b0:
event_sched_in at kernel/events/core.c:2736
event_sched_in+0x7fc/0x8b0:
event_sched_in at kernel/events/core.c:2733
event_sched_in+0x806/0x8b0:
event_sched_in at kernel/events/core.c:2708
event_sched_in+0x813/0x8b0:
event_sched_in at kernel/events/core.c:2740
event_sched_in+0x84b/0x8b0:
event_sched_in at kernel/events/core.c:2753
event_sched_in+0x855/0x8b0:
is_software_event at include/linux/perf_event.h:1548
(inlined by) event_sched_in at kernel/events/core.c:2747
event_sched_in+0x85f/0x8b0:
is_event_in_freq_mode at kernel/events/core.c:2355
(inlined by) event_sched_in at kernel/events/core.c:2749
event_sched_in+0x8a0/0x8b0:
is_event_in_freq_mode at kernel/events/core.c:2355 (discriminator 1)
(inlined by) event_sched_in at kernel/events/core.c:2749 (discriminator 1)
merge_sched_in+0xbe6/0x1050:
merge_sched_in at kernel/events/core.c:4000
merge_sched_in+0xbf0/0x1050:
event_filter_match at kernel/events/core.c:2349
(inlined by) merge_sched_in at kernel/events/core.c:4003
merge_sched_in+0xbfa/0x1050:
group_can_go_on at kernel/events/core.c:2819
(inlined by) merge_sched_in at kernel/events/core.c:4006
merge_sched_in+0xde6/0x1050:
group_can_go_on at kernel/events/core.c:2813
(inlined by) merge_sched_in at kernel/events/core.c:4006
merge_sched_in+0xe04/0x1050:
event_filter_match at kernel/events/core.c:2350
(inlined by) merge_sched_in at kernel/events/core.c:4003
merge_sched_in+0xe0e/0x1050:
merge_sched_in at kernel/events/core.c:3997
merge_sched_in+0xf59/0x1050:
group_sched_in at kernel/events/core.c:2779 (discriminator 1)
(inlined by) merge_sched_in at kernel/events/core.c:4007 (discriminator 1)
merge_sched_in+0xfad/0x1050:
group_sched_in at kernel/events/core.c:2779
(inlined by) merge_sched_in at kernel/events/core.c:4007
perf_event_groups_first+0x2bf/0x360:
perf_event_groups_cmp at kernel/events/core.c:1683
(inlined by) __group_cmp at kernel/events/core.c:1753
(inlined by) rb_find_first at include/linux/rbtree.h:423
(inlined by) perf_event_groups_first at kernel/events/core.c:1833
perf_event_groups_first+0x2c9/0x360:
rb_find_first at include/linux/rbtree.h:430
(inlined by) perf_event_groups_first at kernel/events/core.c:1833
perf_event_groups_first+0x2d0/0x360:
rb_find_first at include/linux/rbtree.h:428
(inlined by) perf_event_groups_first at kernel/events/core.c:1833
perf_event_groups_first+0x30b/0x360:
perf_event_groups_cmp at kernel/events/core.c:1689
(inlined by) __group_cmp at kernel/events/core.c:1753
(inlined by) rb_find_first at include/linux/rbtree.h:423
(inlined by) perf_event_groups_first at kernel/events/core.c:1833
perf_event_groups_first+0x315/0x360:
event_cgroup at kernel/events/core.c:1665
(inlined by) perf_event_groups_cmp at kernel/events/core.c:1697
(inlined by) __group_cmp at kernel/events/core.c:1753
(inlined by) rb_find_first at include/linux/rbtree.h:423
(inlined by) perf_event_groups_first at kernel/events/core.c:1833
perf_event_set_state+0x164/0x1a0:
perf_event_set_state at kernel/events/core.c:751
perf_event_set_state+0x175/0x1a0:
perf_event_set_state at kernel/events/core.c:762 (discriminator 2)
perf_event_update_time+0x2f4/0x3a0:
perf_event_update_time at kernel/events/core.c:737
perf_event_update_time+0x2fb/0x3a0:
perf_event_time at kernel/events/core.c:1580
(inlined by) perf_event_update_time at kernel/events/core.c:733
perf_event_update_time+0x305/0x3a0:
__perf_effective_state at kernel/events/core.c:708
(inlined by) __perf_update_times at kernel/events/core.c:719
(inlined by) perf_event_update_time at kernel/events/core.c:735
perf_event_update_time+0x30f/0x3a0:
__perf_effective_state at kernel/events/core.c:710
(inlined by) __perf_update_times at kernel/events/core.c:719
(inlined by) perf_event_update_time at kernel/events/core.c:735
perf_event_update_time+0x319/0x3a0:
is_cgroup_event at kernel/events/core.c:842
(inlined by) perf_event_time at kernel/events/core.c:1585
(inlined by) perf_event_update_time at kernel/events/core.c:733
perf_event_update_time+0x323/0x3a0:
__perf_effective_state at kernel/events/core.c:713
(inlined by) __perf_update_times at kernel/events/core.c:719
(inlined by) perf_event_update_time at kernel/events/core.c:735
perf_event_update_time+0x330/0x3a0:
__perf_update_times at kernel/events/core.c:720
(inlined by) perf_event_update_time at kernel/events/core.c:735
perf_event_update_time+0x33a/0x3a0:
__perf_update_times at kernel/events/core.c:722
(inlined by) perf_event_update_time at kernel/events/core.c:735
perf_log_itrace_start+0x590/0x620:
perf_log_itrace_start at kernel/events/core.c:10187
perf_log_itrace_start+0x59a/0x620:
perf_log_itrace_start at kernel/events/core.c:10184
perf_pending_task+0x16b/0x410:
arch_atomic64_dec_and_test at arch/x86/include/asm/atomic64_64.h:61 (discriminator 1)
(inlined by) raw_atomic64_dec_and_test at include/linux/atomic/atomic-arch-fallback.h:4404 (discriminator 1)
(inlined by) raw_atomic_long_dec_and_test at include/linux/atomic/atomic-long.h:1571 (discriminator 1)
(inlined by) atomic_long_dec_and_test at include/linux/atomic/atomic-instrumented.h:4540 (discriminator 1)
(inlined by) put_event at kernel/events/core.c:5751 (discriminator 1)
(inlined by) perf_pending_task at kernel/events/core.c:7374 (discriminator 1)
perf_pending_task+0x366/0x410:
perf_pending_task at kernel/events/core.c:7369
perf_swevent_set_period+0x1b1/0x1e0:
perf_swevent_set_period at kernel/events/core.c:10511
perf_swevent_set_period+0x1be/0x1e0:
arch_atomic64_read at arch/x86/include/asm/atomic64_64.h:15
(inlined by) raw_atomic64_read at include/linux/atomic/atomic-arch-fallback.h:2583
(inlined by) raw_atomic_long_read at include/linux/atomic/atomic-long.h:38
(inlined by) atomic_long_read at include/linux/atomic/atomic-instrumented.h:3189
(inlined by) perf_swevent_set_period at kernel/events/core.c:10517
perf_swevent_set_period+0x1c8/0x1e0:
perf_swevent_set_period at kernel/events/core.c:10515
perf_swevent_set_period+0xd1/0x1e0:
arch_atomic64_read at arch/x86/include/asm/atomic64_64.h:15
(inlined by) raw_atomic64_read at include/linux/atomic/atomic-arch-fallback.h:2583
(inlined by) raw_atomic_long_read at include/linux/atomic/atomic-long.h:38
(inlined by) atomic_long_read at include/linux/atomic/atomic-instrumented.h:3189
(inlined by) perf_swevent_set_period at kernel/events/core.c:10517
perf_trace_add+0x2dd/0x370:
perf_trace_add at kernel/trace/trace_event_perf.c:354
perf_trace_add+0x2fb/0x370:
is_sampling_event at include/linux/perf_event.h:1540
(inlined by) perf_trace_add at kernel/trace/trace_event_perf.c:360
perf_trace_add+0x31a/0x370:
perf_trace_add at kernel/trace/trace_event_perf.c:361
perf_trace_add+0x321/0x370:
perf_trace_add at kernel/trace/trace_event_perf.c:361
perf_trace_add+0x332/0x370:
hlist_add_head_rcu at include/linux/rculist.h:648
(inlined by) perf_trace_add at kernel/trace/trace_event_perf.c:379
perf_trace_add+0x344/0x370:
hlist_add_head_rcu at include/linux/rculist.h:649 (discriminator 2)
(inlined by) perf_trace_add at kernel/trace/trace_event_perf.c:379 (discriminator 2)
task_work_run+0x181/0x230:
task_work_run at kernel/task_work.c:232
task_work_run+0x209/0x230:
task_work_run at kernel/task_work.c:233
visit_groups_merge.constprop.0.isra.0+0x105f/0x1190:
visit_groups_merge at kernel/events/core.c:3946
visit_groups_merge.constprop.0.isra.0+0x10b4/0x1190:
visit_groups_merge at kernel/events/core.c:3945