Random kernel panics and crashes

[JackTrades]

Hello everyone,

I have purchased the sweet potato last year for a project. Sadly that software is still in development (hopefully I will finish it next year). Anyhow I did get around last year to install debian on it. It seemed fine but it would panic from time to time. The thing is I only started using Linux last year and did not know what I was doing, so I figured it was something I was doing wrong (I did not even know what a panic was).

Fast forward to last week and I thought that setting up a pi-hole would be a fun weekend project. Setup was easy but then it crashed. With more knowledge and experience I formatted the SD card, put a new image on it. Flashed the board with the newest firmware. And still no cigar…

So I started checking hardware:
Power
Seems fine, no flickering red light.
SD card
Basic test revealed nothing. (Its a Raspberry Pi A2 64GB)
Memory
This is the only place where I rarely find some error. When running memtester it sometimes shows FAILURE in the block sequential test. Always around 0x000000000******. Might be a clue, but I have no idea how to block this part off. It also has to be said that all other tests always run successful and the test I did with memtest86 (not the + version) also came back clean.

The Debian image I use is: “debian-12-base-arm64+aml-s905x-cc.img.xz“. I assumed this one was also for the sweet potato because I did not see a v2 in the download links.

The problems
All panics either came from BTRFS or __memcpy.
Most of the panics where like these:
Dec 06 17:30:08 debian-12-aml-s905x-cc kernel: Unable to handle kernel paging request at virtual address fffeacc7296ed55c
Dec 06 17:30:08 debian-12-aml-s905x-cc kernel: Mem abort info:
Dec 06 17:30:08 debian-12-aml-s905x-cc kernel: ESR = 0x0000000096000004
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: EC = 0x25: DABT (current EL), IL = 32 bits
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: SET = 0, FnV = 0
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: EA = 0, S1PTW = 0
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: FSC = 0x04: level 0 translation fault
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Data abort info:
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: [fffeacc7296ed55c] address between user and kernel address ranges
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Internal error: Oops: 0000000096000004 [#1] SMP
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Modules linked in: nvme_fabrics nvme_core lima gpu_sched drm_shmem_helper snd_soc_meson_gx_sound_card snd_soc_meson_card_utils crct10dif_ce meson_vdec(C) videobuf2_dma_contig v4l2_mem2mem meson_ir videobuf2_memops videobuf2_v4l2 videobuf2_common amlogic_gxl_crypto meson_rng rc_core dw_hdmi_i2s_audio crypto_engine ao_cec videodev scpi_hwmon mc snd_soc_meson_aiu snd_soc_meson_codec_glue pwm_fan binfmt_misc cfg80211 rfkill pkcs8_key_parser fuse efi_pstore pstore dm_mod ip_tables x_tables ipv6 crc_ccitt autofs4 rtc_meson_vrtc
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: CPU: 2 UID: 0 PID: 2782 Comm: kworker/u16:6 Tainted: G C 6.12.58-g44043a69aaa1 #1
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Tainted: [C]=CRAP
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Hardware name: libre-computer aml-s905x-cc-v2/aml-s905x-cc-v2, BIOS 2025.07+ 07/01/2025
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Workqueue: btrfs-worker btrfs_work_helper
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=–)
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: pc : btrfs_get_32+0xb0/0x180
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: lr : __btrfs_check_leaf+0x960/0x1ff8
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: sp : ffff8000837e3b10
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x29: ffff8000837e3b20 x28: 0000000000001000 x27: 0000000000000004
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x26: 0000000000000035 x25: 0000000000000019 x24: ffff2cc706927000
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x23: 0000000000000032 x22: 0000000000001000 x21: ffff2cc725de6780
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x20: 000000000000055c x19: fffeacc7296ed000 x18: 0000000000000000
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x17: 0000000000000000 x16: 0000000000000ffd x15: 0000000000000011
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x14: 0000000000000002 x13: 0000000121a79000 x12: 80ffffffffffffff
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x11: 00000000000000c0 x10: 0000000000000a20 x9 : 0000000040000200
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x8 : ffff000000000000 x7 : 0000000000000560 x6 : 0000000000004000
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x5 : 0000000000000fff x4 : 00000000f3958000 x3 : 100000000000402a
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: x2 : 000000000000055c x1 : fffeacc7296ed55c x0 : 0000000000000560
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Call trace:
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: btrfs_get_32+0xb0/0x180
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: __btrfs_check_leaf+0x960/0x1ff8
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: btrfs_check_leaf+0x18/0x40
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: btree_csum_one_bio+0x194/0x270
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: run_one_async_start+0x50/0x60
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: btrfs_work_helper+0xe0/0x4e8
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: process_one_work+0x164/0x400
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: worker_thread+0x264/0x388
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: kthread+0x118/0x128
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: ret_from_fork+0x10/0x20
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: Code: 91001280 8b140261 eb36c01f 540001c8 (b8746a60)
Dec 06 17:30:09 debian-12-aml-s905x-cc kernel: —[ end trace 0000000000000000 ]—
Or
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Unable to handle kernel paging request at virtual address fffffea0ab453540
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Mem abort info:
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: ESR = 0x0000000096000005
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: EC = 0x25: DABT (current EL), IL = 32 bits
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: SET = 0, FnV = 0
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: EA = 0, S1PTW = 0
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: FSC = 0x05: level 1 translation fault
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Data abort info:
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000031b5a000
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: [fffffea0ab453540] pgd=100000007f3d3003, p4d=100000007f3d3003, pud=0000000000000000
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Internal error: Oops: 0000000096000005 [#1] SMP
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Modules linked in: nvme_fabrics nvme_core meson_vdec(C) videobuf2_dma_contig v4l2_mem2mem videobuf2_memops videobuf2_v4l2 videobuf2_common meson_rng crct10dif_ce lima videodev snd_soc_meson_gx_sound_card snd_soc_meson_card_utils gpu_sched amlogic_gxl_crypto drm_shmem_helper dw_hdmi_i2s_audio crypto_engine mc scpi_hwmon snd_soc_meson_aiu meson_ir snd_soc_meson_codec_glue rc_core ao_cec pwm_fan binfmt_misc cfg80211 rfkill pkcs8_key_parser fuse efi_pstore pstore dm_mod ip_tables x_tables ipv6 crc_ccitt autofs4 rtc_meson_vrtc
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: CPU: 1 UID: 0 PID: 1124 Comm: kworker/u16:5 Tainted: G C 6.12.58-g44043a69aaa1 #1
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Tainted: [C]=CRAP
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Hardware name: libre-computer aml-s905x-cc-v2/aml-s905x-cc-v2, BIOS 2025.07+ 07/01/2025
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Workqueue: btrfs-worker btrfs_work_helper
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=–)
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: pc : btrfs_get_32+0x40/0x180
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: lr : __btrfs_check_leaf+0x230/0x1ff8
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: sp : ffff800081ff3b10
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x29: ffff800081ff3b20 x28: 00000000000023db x27: 0000000000000c67
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x26: 0000000000000092 x25: 0000000000000019 x24: ffff2c3ac6a4e000
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x23: 000000000000007e x22: ffff800081ff3bef x21: 0000000000000ccc
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x20: 000000000000007f x19: ffff2c3b3b491a50 x18: 0000000000000000
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffff8172d960
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x14: 0000000000000000 x13: 000000000002483c x12: 0c0000000000024a
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x11: 00000000000000c0 x10: 0000000000000000 x9 : ffffcf2da728bee0
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x8 : ffff000000000000 x7 : fffffea0ab453540 x6 : 0000000000000cc4
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x5 : 0000000000000fff x4 : 00000000f1ca8000 x3 : ffff2c3ac3ec10c0
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: x2 : 0000000000000011 x1 : 0000000000000cb3 x0 : ffff2c3b3b491a50
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Call trace:
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: btrfs_get_32+0x40/0x180
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: __btrfs_check_leaf+0x230/0x1ff8
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: btrfs_check_leaf+0x18/0x40
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: btree_csum_one_bio+0x194/0x270
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: run_one_async_start+0x50/0x60
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: btrfs_work_helper+0xe0/0x4e8
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: process_one_work+0x164/0x400
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: worker_thread+0x264/0x388
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: kthread+0x118/0x128
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: ret_from_fork+0x10/0x20
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: Code: f9424864 f90007e4 d2800004 f9400004 (f94000e3)
Dec 07 14:40:07 debian-12-aml-s905x-cc kernel: —[ end trace 0000000000000000 ]—

I also had a fun one where the btrfs has trouble reading or decompressing something from the SD card (either a file or block) and goes in to Read-Only mode. But I assume that is in correlation with the above mentioned panics.

Summary
Basically what I want to know is, do I keep forgetting a step (or doing something wrong)?
If not, is it a hardware problem? Yes, can it be by-passed or do I need to purchase a new board?

Thank for taking your time!

[librecomputer]

@JackTrades

the btrfs has trouble reading or decompressing something from the SD card (either a file or block) and goes in to Read-Only mode. But I assume that is in correlation with the above mentioned panics.

What kernel version? What’s the exact error from memtester? Photo of your board?

[JackTrades]

Kernel version:
6.12.58-g44043a69aaa1 (output from uname -r I assume this is the correct way)
Memtest error:
I have run these tests multiple times across multiple days and clean installs with similar results.
Block Sequential : testing 144FAILURE: 0x9090909090909090 != 0x0000ffff8a687020 at offset 0x000000003116e8010.

Photo Board:
I am only allowed to use one picture per post so this is the one from the top:

The board has been in this case since I got it:

LoveRPi Active Cooling Media Center PC Case for Raspberry Pi 3 and Lib

Set the fan up with the help from:

How to Read and Control PWM Fan Speed on AML-S905X-CC Le Potato

[20230523_212823] PWM fans come with 4 wires: Ground Power (3.3V, 5V, 12V) Tachometer PWM speed control Per the GPIO map, there is a convenient header 9J1: Ground GPIO (for tach) 5V On the 40-pin header, there are 5 hardware PWMs we can use per our PWM guide. We do not want to waste CPU cycles to generate inaccurate PWM via software GPIO bitbanging. We can hook these up to the fan via female to male jumper wires or a case with a PWM fan. pwm-a - Pin 33 pwm-ao-a - Pin 11 pwm-ao-b-6 - P…

If you need more pictures please let me know!

[librecomputer]

Is it always the same address that generates the memtest error or different ones? What power supply are you using? Is there anything generating EMI nearby? What speed is the memory clock (eg are you using our latest firmware)?

[JackTrades]

Memtest error:

Its never exactly the same offset, but they do all look close (at least to my untrained eye). These are the ones I have written down:

• 0x00000000000c06360
• 0x000000002bf167a8
• 0x000000000484a960
• 0x00000000316e8010

Powersupply:

AC/DC ADAPTER
MODEL:XBS-0530
INPUT:AC100-240V~
50/60Hz
OUTPUT:5V⎓3A

EMI

As far as I know there isn´t anything nearby generating EMI.

Firmware

BIOS 2025.07+ 07/01/2025 (I assume you meant this)
I updated it last week with these instructions:

How to Flash Firmware or Recover Firmware on Libre Computer AML Boards

Please note, these instructions are for boards starting with AML- ONLY! Plug a USB cable into the board’s OTG port with the onboard button held to enter USB ROM mode. Run the following: sudo apt install dfu-util python3-usb git clone https://github.com/libre-computer-project/pyamlboot.git pyamlboot/run.sh BOARD firmware-update #replace BOARD with AML board model For example: pyamlboot$ ./run.sh aml-s905x-cc-v2 firmware-update /usr/bin/dfu-util --2025-07-09 10:37:38-- https://boot.libre.com…