Wireguard Causing Kernel Oops

Software
, , ,
1

I am using the official Libre Computer Debian 12 image installed on the Libre Computer 16GB eMMC on Sweet Potato. I setup Wireguard via the pivpn.io setup scripts. After varying periods of uptime (a few hours to a few days) Wireguard appears to cause a kernel oops. After the oops, the system will run for a few hours before freezing. I have tried with kernel 6.1.92 and 6.6.85, both from the libre.computer repositories. This installation is remote, so I only have SSH access to diagnose and fix. Any ideas on how to prevent the oops while keeping wireguard active?

May 17 08:36:16 sweetie kernel: cfg80211: loaded regulatory.db is malformed or signature is missing/invalid
May 17 08:36:16 sweetie kernel: meson8b-dwmac c9410000.ethernet end0: Register MEM_TYPE_PAGE_POOL RxQ-0
May 17 08:36:16 sweetie kernel: meson8b-dwmac c9410000.ethernet end0: PHY [mdio_mux-0.1:08] driver [Meson GXL Internal PHY] (irq=27)
May 17 08:36:16 sweetie kernel: meson8b-dwmac c9410000.ethernet end0: No Safety Features support found
May 17 08:36:16 sweetie kernel: meson8b-dwmac c9410000.ethernet end0: PTP not supported by HW
May 17 08:36:16 sweetie kernel: meson8b-dwmac c9410000.ethernet end0: configuring for phy/rmii link mode
May 17 08:36:17 sweetie kernel: wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
May 17 08:36:17 sweetie kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
May 17 08:36:18 sweetie kernel: IPv6: ADDRCONF(NETDEV_CHANGE): end0: link becomes ready
May 17 08:36:18 sweetie kernel: meson8b-dwmac c9410000.ethernet end0: Link is Up - 100Mbps/Full - flow control off
May 17 22:48:06 sweetie kernel: hrtimer: interrupt took 256004 ns
May 17 23:46:45 sweetie kernel: Unable to handle kernel paging request at virtual address ffdfb00b03bc50ac
May 17 23:46:45 sweetie kernel: Mem abort info:
May 17 23:46:45 sweetie kernel:   ESR = 0x0000000086000004
May 17 23:46:45 sweetie kernel:   EC = 0x21: IABT (current EL), IL = 32 bits
May 17 23:46:45 sweetie kernel:   SET = 0, FnV = 0
May 17 23:46:45 sweetie kernel:   EA = 0, S1PTW = 0
May 17 23:46:45 sweetie kernel:   FSC = 0x04: level 0 translation fault
May 17 23:46:45 sweetie kernel: [ffdfb00b03bc50ac] address between user and kernel address ranges
May 17 23:46:45 sweetie kernel: Internal error: Oops: 0000000086000004 [#1] PREEMPT SMP
May 17 23:46:45 sweetie kernel: Modules linked in: tls wireguard libchacha20poly1305 chacha_neon poly1305_neon libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nvme_fabrics nvme_core cfg80211 rfkill binfmt_misc meson_vdec(C) nft_chain_nat videobuf2_dma_contig v4l2>
May 17 23:46:45 sweetie kernel: CPU: 1 PID: 65688 Comm: feed-airplanes Tainted: G         C         6.1.92-15907-gf36fd2695db3 #1
May 17 23:46:45 sweetie kernel: Hardware name: libre-computer aml-s905x-cc-v2/aml-s905x-cc-v2, BIOS 2023.07+ 07/01/2023
May 17 23:46:45 sweetie kernel: pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
May 17 23:46:45 sweetie kernel: pc : 0xffdfb00b03bc50ac
May 17 23:46:45 sweetie kernel: lr : 0xffffb00b03bc50ac
May 17 23:46:45 sweetie kernel: sp : ffff80000a8fbb20
May 17 23:46:45 sweetie kernel: x29: ffff80000a8fbb20 x28: ffff0799cedec880 x27: 0000000000000000
May 17 23:46:45 sweetie kernel: x26: 0000000000000014 x25: 0000000000000000 x24: ffff079988f91600
May 17 23:46:45 sweetie kernel: x23: 0000000000000001 x22: ffff80000a8fbde8 x21: ffff0799979bdb40
May 17 23:46:45 sweetie kernel: x20: 0000000000000002 x19: ffff80000a8fbc60 x18: 0000000000000000
May 17 23:46:45 sweetie kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
May 17 23:46:45 sweetie kernel: x14: 0000000000000000 x13: ffff079981944034 x12: ffff80000a8fbca4
May 17 23:46:45 sweetie kernel: x11: 0000000c32bb9e9e x10: 0000000000000002 x9 : ffffb01b03bc519c
May 17 23:46:45 sweetie kernel: x8 : ffff0799979bdb40 x7 : d0d0d0d0b5a4b1a4 x6 : 0000000000200008
May 17 23:46:45 sweetie kernel: x5 : 0000000000000135 x4 : 0000000000000064 x3 : 0000000000000000
May 17 23:46:45 sweetie kernel: x2 : 0000000b00000000 x1 : 0000000a00000000 x0 : 0000000000000001
May 17 23:46:45 sweetie kernel: Call trace:
May 17 23:46:45 sweetie kernel:  0xffdfb00b03bc50ac
May 17 23:46:45 sweetie kernel:  try_to_unlazy+0x48/0x100
May 17 23:46:45 sweetie kernel:  lookup_fast+0xcc/0x150
May 17 23:46:45 sweetie kernel:  walk_component+0x30/0x1a0
May 17 23:46:45 sweetie kernel:  path_lookupat+0x80/0x1ac
May 17 23:46:45 sweetie kernel:  filename_lookup+0xb4/0x1b0
May 17 23:46:45 sweetie kernel:  user_path_at_empty+0x54/0x80
May 17 23:46:45 sweetie kernel:  do_faccessat+0x98/0x2d0
May 17 23:46:45 sweetie kernel:  __arm64_sys_faccessat+0x2c/0x40
May 17 23:46:45 sweetie kernel:  invoke_syscall+0x50/0x120
May 17 23:46:45 sweetie kernel:  el0_svc_common.constprop.0+0x4c/0xf4
May 17 23:46:45 sweetie kernel:  do_el0_svc+0x34/0xd0
May 17 23:46:45 sweetie kernel:  el0_svc+0x2c/0x84
May 17 23:46:45 sweetie kernel:  el0t_64_sync_handler+0xf4/0x120
May 17 23:46:45 sweetie kernel:  el0t_64_sync+0x18c/0x190
May 17 23:46:45 sweetie kernel: Code: bad PC value
May 17 23:46:45 sweetie kernel: ---[ end trace 0000000000000000 ]---
2

Double check your power supply. This is usually caused by power transients if it occurs on multiple kernel versions.

3

Ok, thank you for the feedback. I forgot to mention that it is currently being powered by the Libre Computer POE hat via a Ubiquiti US-16-150W (Gen 1) POE+ switch. Specs on the switch claim 15.4W max power per port. Next time I’m in the same room as the Sweet Potato, I’ll convert it to traditional AC power (Official Raspberry Pi 4 power supply 5.1V 3A) and see if I can get stability. In the meantime, I’ve disabled wireguard.

4 
May 17 23:46:45 sweetie kernel: Unable to handle kernel paging request at virtual address ffdfb00b03bc50ac
May 17 23:46:45 sweetie kernel: Mem abort info:
May 17 23:46:45 sweetie kernel:   ESR = 0x0000000086000004
May 17 23:46:45 sweetie kernel:   EC = 0x21: IABT (current EL), IL = 32 bits
May 17 23:46:45 sweetie kernel:   SET = 0, FnV = 0
May 17 23:46:45 sweetie kernel:   EA = 0, S1PTW = 0
May 17 23:46:45 sweetie kernel:   FSC = 0x04: level 0 translation fault
May 17 23:46:45 sweetie kernel: [ffdfb00b03bc50ac] address between user and kernel address ranges
May 17 23:46:45 sweetie kernel: Internal error: Oops: 0000000086000004 [#1] PREEMPT SMP
May 17 23:46:45 sweetie kernel: Modules linked in: tls wireguard libchacha20poly1305 chacha_neon poly1305_neon libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nvme_fabrics nvme_core cfg80211 rfkill binfmt_misc meson_vdec(C) nft_chain_nat videobuf2_dma_contig v4l2>
May 17 23:46:45 sweetie kernel: CPU: 1 PID: 65688 Comm: feed-airplanes Tainted: G         C         6.1.92-15907-gf36fd2695db3 #1

feed-airplanes could also possibly have a bug in the code. But check temperature and power first.

1 Like
5

I think Wireguard is a red herring, but I’m still having trouble with kernel oops and panics after swapping out the Libre Computer POE hat for an official Raspberry Pi 4 power supply (5.1v 3A). This is the end of the log for the most recent crash:

$ sudo journalctl -b-1 --no-pager
...
Jul 28 16:27:39 sweetie piaware[683]: 35462 msgs recv'd from readsb (3117 in last 5m); 35401 msgs sent to FlightAware
Jul 28 16:27:43 sweetie kernel: Unable to handle kernel paging request at virtual address 4386dc39ff5cd480
Jul 28 16:27:43 sweetie kernel: Mem abort info:
Jul 28 16:27:43 sweetie kernel:   ESR = 0x0000000096000004
Jul 28 16:27:43 sweetie kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 28 16:27:43 sweetie kernel:   SET = 0, FnV = 0

The above was on kernel 6.12.39

$ uname -a
Linux sweetie 6.12.39-g018ca8542b87 #1 SMP PREEMPT_DYNAMIC Sun Jul 20 15:52:09 EDT 2025 aarch64 GNU/Linux

This is my grub config:

$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=50
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="noquiet"
#GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_LINUX_DEFAULT="noquiet console=tty0 console=ttyAML0,115200n8"
GRUB_CMDLINE_LINUX="panic=30 oops=panic"

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Temperature is mid to high 40’s C. (Currently 44C).
I’d love to get to the bottom of this. Is there anything else I can do (remotely) to help figure out the cause?

6

Here’s another example from this morning (kernel 6.12.39). Could this be a hardware issue? Is there a way to run memtest on this board?

Jul 29 08:49:20 sweetie piaware[697]: 64483 msgs recv'd from readsb (2506 in last 5m); 64461 msgs sent to FlightAware
Jul 29 08:49:44 sweetie kernel: Unable to handle kernel paging request at virtual address ffff3d023f9be1d0
Jul 29 08:49:44 sweetie kernel: Mem abort info:
Jul 29 08:49:44 sweetie kernel:   ESR = 0x0000000096000005
Jul 29 08:49:44 sweetie kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 29 08:49:44 sweetie kernel:   SET = 0, FnV = 0
Jul 29 08:49:44 sweetie kernel:   EA = 0, S1PTW = 0
Jul 29 08:49:44 sweetie kernel:   FSC = 0x05: level 1 translation fault
Jul 29 08:49:44 sweetie kernel: Data abort info:
Jul 29 08:49:44 sweetie kernel:   ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
Jul 29 08:49:44 sweetie kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Jul 29 08:49:44 sweetie kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Jul 29 08:49:44 sweetie kernel: swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000031ef000
Jul 29 08:49:44 sweetie kernel: [ffff3d023f9be1d0] pgd=180000007fe5a003, p4d=180000007fe5a003, pud=0000000000000000
Jul 29 08:49:44 sweetie kernel: Internal error: Oops: 0000000096000005 [#1] SMP
7 
sudo apt install memtester
sudo memtester 1G
8

Well, memtest ran for about 24 hours and didn’t show anything. :frowning:

mihanson@sweetie:~$ sudo memtester 1G
memtester version 4.6.0 (64-bit)
Copyright (C) 2001-2020 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 1024MB (1073741824 bytes)
got  1024MB (1073741824 bytes), trying mlock ...locked.
Loop 1:
  Stuck Address       : ok         
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok         
  Block Sequential    : ok         
  Checkerboard        : ok         
  Bit Spread          : ok         
  Bit Flip            : ok         
  Walking Ones        : ok         
  Walking Zeroes      : ok         
  8-bit Writes        : ok
  16-bit Writes       : ok
  ...
  Loop 61:
  Stuck Address       : ok         
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok         
  Block Sequential    : ok         
  Checkerboard        : ok         
  Bit Spread          : ok         
  Bit Flip            : ok         
  Walking Ones        : ok         
  Walking Zeroes      : ok         
  8-bit Writes        : ok
  16-bit Writes       : ok

I updated to kernel 6.12.41-gfe345274f0aa. I’ll continue to watch and report any issues.

9 
Jul 29 08:49:20 sweetie piaware[697]: 64483 msgs recv'd from readsb (2506 in last 5m); 64461 msgs sent to FlightAware
Jul 29 08:49:44 sweetie kernel: Unable to handle kernel paging request at virtual address ffff3d023f9be1d0

Is the piaware application running as root? If it is, is it writting to memory directly? If it’s not a power supply issue or a memory issue, it’s most likely the application doing something funny in memory as root. There’s a lot of Raspberry Pi software out there that does hacky things directly in memory.

10

No, piaware runs as it’s own user, as does the other programs that I use my Sweet Potato for.

mihanson@sweetie:~$ ps -u piaware
    PID TTY          TIME CMD
    688 ?        00:23:04 piaware
    986 ?        00:08:50 faup1090
   5354 ?        00:27:09 fa-mlat-client

mihanson@sweetie:~$ ps -u fr24
    PID TTY          TIME CMD
    714 ?        00:17:09 fr24feed

mihanson@sweetie:~$ ps -u airplanes
    PID TTY          TIME CMD
    405 ?        00:25:26 feed-airplanes
    406 ?        00:27:05 mlat-client

mihanson@sweetie:~$ ps -u readsb
    PID TTY          TIME CMD
    404 ?        04:29:19 readsb

mihanson@sweetie:~$ ps -u unifi
    PID TTY          TIME CMD
    732 ?        00:31:46 java
   2329 ?        00:36:12 mongod

Here’s everything running as root:

mihanson@sweetie:~$ ps -u root
    PID TTY          TIME CMD
      1 ?        00:00:40 systemd
      2 ?        00:00:00 kthreadd
      3 ?        00:00:00 pool_workqueue_release
      4 ?        00:00:00 kworker/R-kvfree_rcu_reclaim
      5 ?        00:00:00 kworker/R-rcu_gp
      6 ?        00:00:00 kworker/R-sync_wq
      7 ?        00:00:00 kworker/R-slub_flushwq
      8 ?        00:00:00 kworker/R-netns
     13 ?        00:00:00 kworker/R-mm_percpu_wq
     14 ?        00:00:00 rcu_tasks_kthread
     15 ?        00:00:00 rcu_tasks_rude_kthread
     16 ?        00:00:00 rcu_tasks_trace_kthread
     17 ?        00:00:08 ksoftirqd/0
     18 ?        00:01:07 rcu_preempt
     19 ?        00:00:00 rcu_exp_par_gp_kthread_worker/0
     20 ?        00:00:00 rcu_exp_gp_kthread_worker
     21 ?        00:00:00 migration/0
     22 ?        00:00:00 cpuhp/0
     23 ?        00:00:00 cpuhp/1
     24 ?        00:00:00 migration/1
     25 ?        00:00:01 ksoftirqd/1
     28 ?        00:00:00 cpuhp/2
     29 ?        00:00:00 migration/2
     30 ?        00:00:03 ksoftirqd/2
     33 ?        00:00:00 cpuhp/3
     34 ?        00:00:00 migration/3
     35 ?        00:00:03 ksoftirqd/3
     38 ?        00:00:00 kdevtmpfs
     39 ?        00:00:00 kworker/R-inet_frag_wq
     40 ?        00:00:00 kauditd
     41 ?        00:00:00 oom_reaper
     43 ?        00:00:00 kworker/R-writeback
     44 ?        00:00:26 kcompactd0
     45 ?        00:00:00 ksmd
     46 ?        00:00:00 khugepaged
     47 ?        00:00:00 kworker/R-kintegrityd
     48 ?        00:00:00 kworker/R-kblockd
     49 ?        00:00:00 kworker/R-blkcg_punt_bio
     51 ?        00:00:00 kworker/R-tpm_dev_wq
     53 ?        00:00:00 kworker/R-ata_sff
     54 ?        00:00:00 kworker/R-edac-poller
     55 ?        00:00:00 kworker/R-devfreq_wq
     56 ?        00:00:00 watchdogd
     58 ?        00:00:00 kworker/R-rpciod
     59 ?        00:00:00 kworker/R-xprtiod
     63 ?        00:02:58 kswapd0
     64 ?        00:00:00 kworker/R-nfsiod
     65 ?        00:00:00 kworker/R-xfsalloc
     66 ?        00:00:00 kworker/R-xfs_mru_cache
     67 ?        00:00:00 kworker/R-kthrotld
     73 ?        00:00:00 kworker/R-iscsi_conn_cleanup
     74 ?        00:00:00 kworker/R-target_completion
     75 ?        00:00:00 kworker/R-target_submission
     76 ?        00:00:00 kworker/R-xcopy_wq
     77 ?        00:00:00 kworker/R-bnx2x
     78 ?        00:00:00 kworker/R-bnx2x_iov
     79 ?        00:00:00 kworker/R-vfio-irqfd-cleanup
     80 ?        00:00:00 kworker/R-uas
     82 ?        00:00:00 kworker/R-kstrp
     84 ?        00:00:00 kworker/R-zswap-shrink
     85 ?        00:00:00 kworker/u17:0
     86 ?        00:00:00 spi0
     88 ?        00:00:00 kworker/R-stmmac_wq
     89 ?        00:00:45 irq/23-d0074000.mmc
     90 ?        00:01:38 sugov:0
     91 ?        00:00:00 card0-crtc0
     92 ?        00:00:00 irq/26-dw_hdmi_top_irq
     93 ?        00:00:00 irq/26-c883a000.hdmi-tx
     97 ?        00:00:00 kworker/R-mmc_complete
    101 ?        00:00:00 irq/30-d0072000.mmc
    102 ?        00:00:00 irq/31-d0072000.mmc cd
    166 ?        00:00:00 kworker/R-btrfs-worker
    167 ?        00:00:00 kworker/R-btrfs-delalloc
    168 ?        00:00:00 kworker/R-btrfs-flush_delalloc
    169 ?        00:00:00 kworker/R-btrfs-cache
    170 ?        00:00:00 kworker/R-btrfs-fixup
    171 ?        00:00:00 kworker/R-btrfs-endio
    172 ?        00:00:00 kworker/R-btrfs-endio-meta
    173 ?        00:00:00 kworker/R-btrfs-rmw
    174 ?        00:00:00 kworker/R-btrfs-endio-write
    175 ?        00:00:00 kworker/R-btrfs-compressed-write
    176 ?        00:00:00 kworker/R-btrfs-freespace-write
    177 ?        00:00:00 kworker/R-btrfs-delayed-meta
    178 ?        00:00:00 kworker/R-btrfs-qgroup-rescan
    179 ?        00:00:01 btrfs-cleaner
    180 ?        00:01:44 btrfs-transaction
    192 ?        00:00:00 kworker/R-mld
    193 ?        00:00:00 kworker/R-ipv6_addrconf
    248 ?        00:00:08 systemd-journal
    269 ?        00:00:00 systemd-udevd
    322 ?        00:00:00 cec-meson_ao_cec
    324 ?        00:00:00 irq/34-c8100100.cec
    325 ?        00:00:00 hwrng
    334 ?        00:00:00 c883e000.crypto-engine
    335 ?        00:00:00 c883e000.crypto-engine
    337 ?        00:00:00 kworker/R-gp
    338 ?        00:00:00 kworker/R-pp
    343 ?        00:00:00 irq/47-vdec
    402 ?        00:00:00 kworker/R-cfg80211
    403 ?        00:00:00 irq/27-mdio_mux-0.1:08
    409 ?        00:00:00 cron
    416 ?        00:02:50 fail2ban-server
    430 ?        00:00:01 rsyslogd
    435 ?        00:00:01 systemd-logind
    461 ?        00:00:00 sshd
    518 ?        00:00:00 rngd
    538 ?        00:00:00 kworker/R-wg-crypt-wg0
    561 tty1     00:00:00 agetty
    568 ?        00:00:00 napi/wg0-0
    572 ?        00:00:00 napi/wg0-0
    574 ?        00:00:00 napi/wg0-0
    584 ttyAML0  00:00:00 agetty
    604 ?        00:00:00 unattended-upgr
    724 ?        00:00:56 ddclient - slee
    813 ?        00:00:00 kworker/R-tls-strp
    869 ?        00:00:00 service-graphs1
    907 ?        00:00:01 master
   7531 ?        00:00:09 service-graphs1
   8191 ?        00:00:16 watchdog
 190457 ?        00:00:03 kworker/0:2-events
 466765 ?        00:00:02 kworker/2:1-mm_percpu_wq
 828853 ?        00:00:00 kworker/1:1-mm_percpu_wq
 844862 ?        00:00:00 kworker/0:0-events
 873609 ?        00:00:04 kworker/u16:2-btrfs-worker
 882191 ?        00:00:00 kworker/2:0-cgroup_destroy
 882208 ?        00:00:09 collectd
 916676 ?        00:00:03 kworker/u16:5-btrfs-endio-write
 921242 ?        00:00:03 kworker/u16:6-btrfs-worker
 923513 ?        00:00:00 kworker/1:1H-kblockd
 928749 ?        00:00:01 kworker/u16:8-events_power_efficient
 933274 ?        00:00:00 kworker/3:3-cgroup_destroy
 935713 ?        00:00:01 kworker/u16:7-flush-btrfs-1
 936300 ?        00:00:00 kworker/1:2-cgroup_destroy
 936890 ?        00:00:00 kworker/u16:9-btrfs-worker
 939729 ?        00:00:00 kworker/0:0H-mmc_complete
 939738 ?        00:00:00 kworker/3:1H-kblockd
 940917 ?        00:00:00 kworker/2:2H
 943782 ?        00:00:00 kworker/u16:0-events_power_efficient
 944273 ?        00:00:00 kworker/2:1H-kblockd
 945904 ?        00:00:00 sshd
 945979 ?        00:00:00 kworker/3:2-mm_percpu_wq
 946591 ?        00:00:00 kworker/3:2H
 946650 ?        00:00:00 kworker/u16:1-btrfs-worker
 946651 ?        00:00:00 kworker/1:2H
 947756 ?        00:00:00 kworker/0:2H-mmc_complete
 948407 ?        00:00:00 kworker/2:0H
 949551 ?        00:00:00 kworker/1:0H
 950146 ?        00:00:00 service-graphs1
11

Do you know if flight aware installs any drivers or interfaces into the kernel via dkms?

12

No, it does not install any drivers or interfaces into the kernel (that I’m aware of)

13

Have you monitored memory usage and system load when this occurs?

14 
Aug 07 13:05:02 sweetie piaware[688]: 716553 msgs recv'd from readsb (2728 in last 5m); 716497 msgs sent to FlightAware

top - 13:05:02 up 1 day,  5:25,  3 users,  load average: 0.67, 0.90, 1.02
Tasks: 188 total,   2 running, 186 sleeping,   0 stopped,   0 zombie
%Cpu0  :  6.7 us,  3.0 sy,  0.3 ni, 84.3 id,  2.3 wa,  2.0 hi,  1.3 si,  0.0 st 
%Cpu1  : 14.5 us,  1.6 sy,  0.3 ni, 80.9 id,  1.3 wa,  1.0 hi,  0.3 si,  0.0 st 
%Cpu2  : 25.1 us,  3.3 sy,  0.3 ni, 65.3 id,  4.0 wa,  1.0 hi,  1.0 si,  0.0 st 
%Cpu3  :  9.3 us,  3.0 sy,  0.0 ni, 84.7 id,  1.0 wa,  1.0 hi,  1.0 si,  0.0 st 
MiB Mem :   1947.5 total,    157.5 free,   1015.7 used,    957.7 buff/cache     
MiB Swap:   1997.5 total,   1486.8 free,    510.7 used.    931.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                
    404 readsb    15  -5  475596   9140   2180 S  38.0   0.5 312:49.28 readsb                 
    688 piaware   20   0   28624   6224   2896 S   4.3   0.3  27:29.24 piaware                
   2329 unifi     20   0 1092272  87324  24268 S   4.3   4.4  40:03.89 mongod                 
    406 airplan+  19  -1   28836   7128   3460 S   3.6   0.4  30:25.76 mlat-client            
    732 unifi     20   0 4385236 331620  14256 S   3.6  16.6  33:18.61 java                   
   5354 piaware   20   0   21020   5368   3200 S   3.6   0.3  30:33.69 fa-mlat-client         
    405 airplan+  19  -1  475512   5400   2196 S   2.6   0.3  28:50.26 feed-airplanes         
    714 fr24      20   0  623456  11096   4392 S   2.0   0.6  19:19.56 fr24feed               
     90 root      rt   0       0      0      0 D   1.0   0.0   2:21.60 sugov:0                
    986 piaware   20   0    3472   2396   1724 S   1.0   0.1  10:27.36 faup1090               
1017921 root      20   0       0      0      0 D   1.0   0.0   0:00.38 kworker/u16:3+btrfs_di+
1021461 mihanson  20   0   11532   5020   2868 R   0.7   0.3   0:02.23 top

Note that swap is in zram and on disk.

mihanson@sweetie:~$ sudo cat /etc/systemd/system/zram-generator.conf 
[zram0]
zram-size = min(ram / 2, 1024)
compression-algorithm = zstd
swap-priority = 200

mihanson@sweetie:~$ sudo cat /etc/sysctl.d/99-vm-zram-parameters.conf 
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0

cat /etc/fstab
...
/swap/swapfile		none		swap	defaults,pri=100	0	0
15

Disable any ram compression (zswap and zram) and enable disk based swap to test. Memory pressure can causes crashes with these tools.

16

Ok, will give that a try.

17

I disabled zram by removing systemd-zram-generator; Turned zsawp off on the kernel command line

$ cat /etc/default/grub | grep zswap
GRUB_CMDLINE_LINUX="panic=45 oops=panic zswap.enabled=0"

$ sudo dmesg | grep zswap
[    0.000000] Kernel command line: BOOT_IMAGE=/@/boot/vmlinuz-6.12.41-gfe345274f0aa root=UUID=ae802f12-183e-4db5-b8a1-d70eba710416 ro rootflags=subvol=@ panic=45 oops=panic zswap.enabled=0 noquiet console=tty0 console=ttyAML0,115200n8

I’m left with only a 1G swapfile in it’s own subvolume:

$ sudo cat /proc/swaps
Filename				Type		Size		Used		Priority
/swap/swapfile                          file		1048572		0		100

$ sudo journalctl -t kernel | grep swapfile
Aug 13 06:17:03 sweetie kernel: Adding 1048572k swap on /swap/swapfile.  Priority:100 extents:1 across:1048572k SS

$ ls -lah /toplevel/@swap/
total 1.1G
drwxr-xr-x 1 root root   16 Dec 19  2024 .
drwxr-xr-x 1 root root   54 May 29 18:44 ..
-rw------- 1 root root 1.0G Dec 19  2024 swapfile

$ mount
...
/dev/mmcblk0p2 on /swap type btrfs (rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/@swap)
...

But I’m still getting regular kernel oops’s

Aug 13 07:03:05 sweetie piaware[674]: 72324 msgs recv'd from readsb (1440 in last 5m); 72240 msgs sent to FlightAware
Aug 13 07:08:05 sweetie piaware[674]: 73807 msgs recv'd from readsb (1483 in last 5m); 73723 msgs sent to FlightAware
Aug 13 07:13:05 sweetie piaware[674]: 75402 msgs recv'd from readsb (1595 in last 5m); 75318 msgs sent to FlightAware
Aug 13 07:13:16 sweetie kernel: Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
Aug 13 07:13:16 sweetie kernel: Mem abort info:
Aug 13 07:13:16 sweetie kernel:   ESR = 0x0000000086000004
Aug 13 07:13:16 sweetie kernel:   EC = 0x21: IABT (current EL), IL = 32 bits
Aug 13 07:13:16 sweetie kernel:   SET = 0, FnV = 0
Aug 13 07:13:16 sweetie kernel:   EA = 0, S1PTW = 0
Aug 13 07:13:16 sweetie kernel:   FSC = 0x04: level 0 translation fault
Aug 13 07:13:16 sweetie kernel: user pgtable: 4k pages, 48-bit VAs, pgdp=0000000003494000
Aug 13 07:13:16 sweetie kernel: [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
Aug 13 07:13:16 sweetie kernel: Internal error: Oops: 0000000086000004 [#1] SMP

$ uname -a
Linux sweetie 6.12.41-gfe345274f0aa #1 SMP PREEMPT_DYNAMIC Tue Aug  5 23:52:40 EDT 2025 aarch64 GNU/Linux

Any other ideas?

18

If you stop piaware, does the board still crash? You can try to put a stress-ng load on the board.

Otherwise, it’s something to do with how piaware is interacting with a kernel module. Are you confident piaware’s SDR does not use DKMS for a custom module?

Userspace should not be able to cause those errors directly because it is within the kernel. It only occurs if the kernel module has a code bug or the input to a module is malformed (although quality modules should not crash either way).

You can also try a different board to see if it still happens. If it still does, likely software. If it does not, it’s hardware.

19

Unfortunately, yes, it still crashes. I also disabled airplanes-* and that didn’t stop the crashing either.

Aug 14 19:56:46 sweetie airplanes-mlat[413]: Receiver: connected   576.8 msg/s received      215.2 msg/s processed (37%)
Aug 14 19:56:46 sweetie airplanes-mlat[413]: Server:   ready         0.1 kB/s from server      0.6 kB/s to server
Aug 14 19:56:46 sweetie airplanes-mlat[413]: Results:  53.9 positions/minute
Aug 14 19:56:46 sweetie airplanes-mlat[413]: Aircraft: 2 of 12 Mode S, 12 of 62 ADS-B used
Aug 14 19:56:56 sweetie airplanes-mlat[413]: peer_count:  50 outlier_percent: 1.8 bad_sync_timeout:   0
Aug 14 20:10:36 sweetie rngd[547]: stats: bits received from HRNG source: 160064
Aug 14 20:10:36 sweetie rngd[547]: stats: bits sent to kernel pool: 120000
Aug 14 20:10:36 sweetie rngd[547]: stats: entropy added to kernel pool: 120000
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2 successes: 8
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2 failures: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2(2001-10-10) Poker: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2(2001-10-10) Runs: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2(2001-10-10) Long run: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: HRNG source speed: (min=2.847; avg=14.911; max=68.857)Mibits/s
Aug 14 20:10:36 sweetie rngd[547]: stats: FIPS tests speed: (min=17.387; avg=29.937; max=61.927)Mibits/s
Aug 14 20:10:36 sweetie rngd[547]: stats: Lowest ready-buffers level: 2
Aug 14 20:10:36 sweetie rngd[547]: stats: Entropy starvations: 0
Aug 14 20:10:36 sweetie rngd[547]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us
Aug 14 20:11:47 sweetie airplanes-mlat[413]: Receiver: connected   573.8 msg/s received      215.5 msg/s processed (38%)
Aug 14 20:11:47 sweetie airplanes-mlat[413]: Server:   ready         0.1 kB/s from server      0.6 kB/s to server
Aug 14 20:11:47 sweetie airplanes-mlat[413]: Results:  43.6 positions/minute
Aug 14 20:11:47 sweetie airplanes-mlat[413]: Aircraft: 4 of 10 Mode S, 12 of 68 ADS-B used
Aug 14 20:12:00 sweetie airplanes-mlat[413]: peer_count:  55 outlier_percent: 1.0 bad_sync_timeout:   0
Aug 14 20:15:51 sweetie kernel: Unable to handle kernel paging request at virtual address ffdf4a77408161bc
Aug 14 20:15:51 sweetie kernel: Unable to handle kernel paging request at virtual address efdf4a47bf0d6c18
Aug 14 20:15:51 sweetie kernel: Mem abort info:
Aug 14 20:15:51 sweetie kernel:   ESR = 0x0000000096000004
Aug 14 20:15:51 sweetie kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Aug 14 20:15:51 sweetie kernel: Mem abort info:
Aug 14 20:15:52 sweetie kernel:   SET = 0, FnV = 0

$ sudo systemctl status piaware.service
○ piaware.service - FlightAware ADS-B uploader
     Loaded: loaded (/lib/systemd/system/piaware.service; disabled; preset: enabled)
    Drop-In: /etc/systemd/system/piaware.service.d
             └─override.conf
     Active: inactive (dead)
       Docs: https://flightaware.com/adsb/piaware/

Yes. The SDR drivers are part of the mainline kernel. They are not added via DKMS, add-on package or built locally by me.

Here’s the most recent crash. The journal just ends abruptly. Again, no piaware or airplanes-* were running:

Aug 21 10:59:05 sweetie CRON[380361]: pam_unix(cron:session): session closed for user root
Aug 21 11:17:01 sweetie CRON[389415]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 21 11:17:01 sweetie CRON[389416]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Aug 21 11:17:01 sweetie CRON[389415]: pam_unix(cron:session): session closed for user root
Aug 21 11:37:28 sweetie rngd[523]: stats: bits received from HRNG source: 440064
Aug 21 11:37:28 sweetie rngd[523]: stats: bits sent to kernel pool: 390240
Aug 21 11:37:28 sweetie rngd[523]: stats: entropy added to kernel pool: 390240
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2 successes: 22
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2 failures: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2(2001-10-10) Poker: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2(2001-10-10) Runs: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2(2001-10-10) Long run: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: HRNG source speed: (min=16.615; avg=35.262; max=80.479)Mibits/s
Aug 21 11:37:28 sweetie rngd[523]: stats: FIPS tests speed: (min=10.260; avg=26.724; max=61.726)Mibits/s
Aug 21 11:37:28 sweetie rngd[523]: stats: Lowest ready-buffers level: 2
Aug 21 11:37:28 sweetie rngd[523]: stats: Entropy starvations: 0
Aug 21 11:37:28 sweetie rngd[523]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us
Aug 21 12:17:01 sweetie CRON[419482]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 21 12:17:01 sweetie CRON[419483]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Aug 21 12:17:01 sweetie CRON[419482]: pam_unix(cron:session): session closed for user root
20

Try installing munin and lm-sensors to check if there’s any commonalities in the crashes from sources such as temperature, memory, or application usage. The crashes should show up as blank areas in the data since data logging stops.

Otherwise, just run a clean OS or EtherealOS on the board and run stress-ng to check for memory and CPU errors. Other than that, not sure what to tell you.