We will create a guide eventually on how to bootstrap your own OS. This is fairly easy for us as we do it day in and day out.
Apple keyboards, what can we say?
That’s not - that’s a Keychron K8 in Windows mode, and the only one that gives any explanation as to why it’s not working. (Expected to an extent.) GMMK Pro doesn’t work. Costar doesn’t work. Custom VIA doesn’t work. Generic crashcart mushboard doesn’t work.
No keyboard works at all. So something is broken. Question is, what is it? Because it’s not 5+ different keyboards.
edit: should note, this is happening on two different s905x-v1.0’s, and the same ‘no keyboard’ behavior with Raspbian and Ubuntu.
@librecomputer OK, at this point, I’ve confirmed that the problem is not me - it is, in fact, the latest u-boot in the CI. aml-s905x-cc-2022-07 is the last working one. Here’s what I went through:
- Latest Debian and Ubuntu from CI
- 2 different LePotatos (both v1; one early, one later color-coded header)
- 8 different C10U3 SD cards from 4 different vendors (64GB PNY, 64GB SanDisk, 16GB SanDisk, 32GB SanDisk)
- 7 different keyboards (toss-away crashcart style, generic USB 104, Costar CST104, GH60, Keychron K8 as control, EVGA Z20 as control)
- 3 different HDMI attachments; USB-HDMI adapter, Dell U2415, Samsung Odyssey G9
- 6 different power supply setups from 2.5A to 7.5A capability
And every single configuration has presented precisely identical results.
Debian finds keyboard and GRUB but keyboard input does not work at all, and after entering boot, the green LED turns solid, the HDMI output turns into a solid green or purple, and the processor quickly gets very hot.
Ubuntu finds keyboard and GRUB but keyboard input does not work at all, and after entering boot, the green LED turns solid, the HDMI output turns into a solid green or purple, and the processor quickly gets very hot.
Naked SD with aml-s905x-cc from July 16, 2023 detects keyboard and has expected loader error, but keyboard input does not work at all. Processor does not get hot.
Alpine Imager with aml-s905x-cc from July 16, 2023 detects keyboard, has unexpected loader error (probably uboot.env or ini), and keyboard input does not work at all. Processor does not get hot.
Alpine Imager with aml-s905x-cc-2022-07? This is the only u-boot that works correctly at all. Keyboard works, manually loading works, boots and runs. Yup. Works flawlessly.
Probing around didn’t reveal any clue as to what the board is doing other than that I/O isn’t working beyond the SD slot, and the CPU locks up when transitioning to kernel.
Thanks for bringing this to our attention. That was a pre-release CI version. Please test the current version.
Thanks for bringing this to our attention. That was a pre-release CI version. Please test the current version.
Thanks! Looking much better with latest; keyboard works again, no lockup!
Only weird thing I see now is that spi uclass doesn’t seem to be present, and pxe and dhcp are also missing. (But IIRC the s905x’s MAC needs a DTB from tree for that, and I know I have the wrong path there.)
So I think honestly at this point, really all I need is to sort out the uboot env and boot.ini for Alpine (which needs to be /boot looks like.) Is there somewhere all the addresses for the LePotato are listed? I’ve looked and not had much luck finding it. And the uboot env has fdt_addr_r and kernel_addr_r both at 0x080008000, which doesn’t seem correct?
Network is not enabled on AML-S905X-CC for security and other reasons.
There’s no SPI on AML-S905X-CC. Only AML-S905X-CC-V2 (different product than V1, not to be confused with revision).
The MAC is set in the DT and it’s passed to Linux even without the network.
S805X shares the same memory layout as S905X.
Okay, at this point, I seem to be hitting a brick wall. I should, in theory, have a bootable image. I have a boot.cmd, I have a fat32 with 0xef at /boot, I have a known good u-boot, a known good kernel, a known good DTB, and my HDMI interface isn’t a broken one (it’s just a MacroSilicon USB.)
Here’s the boot.cmd
But it stubbornly refuses to work. u-boot fails to find any bootflow at mmc@74000.bootdev or mmc@72000.bootdev. ext4ls works on mmc 1:2, fatinfo on 1:1. Manual booting is also not working, but for a very different reason.
... various load commands ...
=> load mmc 1:1 $ramdisk_addr_r /initramfs-lts
23870963 bytes read in 1039 ms (21.9MiB/s)
=> booti $kernel_addr_r $ramdisk_addr_r $fdt_addr_r
Uncompressing Kernel Image
Moving Image from 0x8080000 to 0x8200000, end=a600000
Wrong Ramdisk Image Format
Ramdisk image is corrupt or invalid
Except this is a known good initramfs-lts; it’s the same I use on rpi without issue. Just a standard gzip image with everything in it. When I bypass initramfs with - in booti and use the dtb from Libre’s CI with 2023.07+ (Jul 22 2023) I get this instead:
=> booti $kernel_addr_r - $fdt_addr_r
Uncompressing Kernel Image
Moving Image from 0x8080000 to 0x8200000, end=a600000
## Flattened Device Tree blob at 08008000
Booting using the fdt blob at 0x8008000
Working FDT set to 8008000
Loading Device Tree to 00000007ae8e000, end 00000007ae9b177 ... OK
Working FDT set to 7ae8e000
Starting kernel ...
… and then just a hard lock. Even the reset button doesn’t work. When I try the 2022-07 u-boot with the same booti command, I get:
Moving Image from 0x8080000 to 0x8200000, end=a600000
## Flattened Device Tree blob at 08008000
Booting using the fdt blob at 0x08008000
Loading Device Tree to 00000007be5800, end 00000007be621a7 ... OK
Starting kernel ...
"Synchronous Abort" handler, esr 0x020000000
elr: ffffffff8c99f770 lr : 0000000000108d5a0 (reloc)
Comparing a (non-working because they built a flat ext4 image) Armbian 6.1.30 and Alpine 6.1.42, the only thing “missing” is:
lib/modules/6.1.30-meson64/kernel/drivers/phy/amlogic:
-rw-r--r-- 1 root root 11992 Jul 29 13:48 phy-meson-axg-mipi-dphy.ko
-rw-r--r-- 1 root root 9192 Jul 29 13:48 phy-meson-g12a-mipi-dphy-analog.ko
So realistically, it’s missing nothing. Going through the bootm flow instead results in a crash at bootm fdt. But both DTBs do pass bootefi selftest $fdt_addr_r (aml-s905x-cc.dtb and dtbs-lts/amlogic/meson-gxl-s905x-libretech-cc.dtb) with 1 expected failure.
At this point, I’m completely stumped.
-
Don’t use Armbian’s boot.cmd as a starting point. It’s a giant non-portable hack and very prone to problems.
-
Just manually boot via a small boot.scr script like such:
load mmc 1 $kernel_addr_r KERNEL_FILE
load mmc 1 $ramdisk_addr_r INITRAMFS_FILE
bootm $kernel_addr_r $ramdisk_addr_r $fdtcontroladdr
@librecomputer Armbian’s closer to what the end-state needs to be (since this will cover many BSPs.)
But again, even doing that manually has the exact same results.
Latest u-boot results in a hard lock with no output and dead clock.
The previously tested u-boot results in an immediate synchronous abort crash.
Both insist a known and confirmed good gzip initramfs is corrupt or invalid.
=> load mmc 1:1 $kernel_addr_r vmlinuz-lts
10611147 bytes read in 460 ms (22 MiB/s)
=> load mmc 1:1 $ramdisk_addr_r initramfs-lts
23911485 bytes read in 1037 ms (22 MiB/s)
=> bootm $kernel_addr_r $ramdisk_addr_r $fdtcontroladdr
Wrong Image Format for bootm command
ERROR: can't get kernel image!
=> booti $kernel_addr_r $ramdisk_addr_r $fdtcontroladdr
Uncompressing Kernel Image
Moving Image from 0x80800000 to 0x8200000, end=a660000
Wrong Ramdisk Image Format
Ramdisk image is corrupt or invalid
3d6d8793f710 [/chroot/boot]$ file vmlinuz-lts
vmlinuz-lts: gzip compressed data, max compression, from Unix, original size modulo 2^32 32324096
3d6d8793f710 [/chroot/boot]$ file initramfs-lts
initramfs-lts: gzip compressed data, max compression, from Unix, original size modulo 2^32 30432216
Armbian makes a lot of assumptions and does a lot of things incorrectly. For example, it loads the image to 0x80800000 which is beyond usable memory. A lot of Armbian’s use of out-of-tree changes can be giant hacks and will cause side-band problems if you don’t know what those patches are doing and the side effect of them.
You are using booti incorrectly: booti command — Das U-Boot unknown version documentation
Armbian makes a lot of assumptions and does a lot of things incorrectly. For example, it loads the image to 0x80800000 which is beyond usable memory. A lot of Armbian’s changes are giant hacks and will cause side-band problems.
Yeah, I noticed their setup was… not good. If you look, my setup explicitly sets all of the addresses correctly (based on the s805x/s905x) and strips out a lot of their extraneous stuff. This is admittedly pretty temporary; it will actually use a board specific env file in final.
You are using booti incorrectly: booti command — Das U-Boot unknown version documentation
Ah, good catch. But alas, no change at all with that.
With latest u-boot:
=> size mmc 1:1 initramfs-lts
=> load mmc 1:1 $ramdisk_addr_r initramfs-lts
23911485 bytes read in 1037 ms (22 MiB/s)
=> load mmc 1:1 $kernel_addr_r vmlinuz-lts
10611147 bytes read in 460 ms (22 MiB/s)
=> booti $kernel_addr_r $ramdisk_addr_r:$filesize $fdtcontroladdr
Uncompressing Kernel Image
Moving Image from 0x8080000 to 0x8200000, end=a660000
## Flattened Device Tree blob at 75ea5c20
Booting using the fdt blob at 0x75ea5c20
Working FDT set to 75ea5c20
Loading Ramdisk to 737cd000, end 74e9ac3d ... OK
Loading Device Tree to 00000000737bf000, end 00000000737cc177 ... OK
Working FDT set to 737bf000
Starting kernel ...
<board completely dead>
By completely dead I mean pretty much anything that isn’t voltage is just straight dead. I can’t probe the CPU clock pins, but everything I can probe is just flat. Even the MMC_CLK line has nothing.
With the 2022-07 u-boot:
=> size mmc 1:1 initramfs-lts
=> load mmc 1:1 $ramdisk_addr_r initramfs-lts
23911485 bytes read in 1037 ms (22 MiB/s)
=> load mmc 1:1 $kernel_addr_r vmlinuz-lts
10611147 bytes read in 460 ms (22 MiB/s)
=> booti $kernel_addr_r $ramdisk_addr_r:$filesize $fdtcontroladdr
Uncompressing Kernel Image
Moving Image from 0x8080000 to 0x8200000, end=a660000
## Flattened Device Tree blob at 7be66dd0
Booting using the fdt blob at 0x7be66dd0
Loading Ramdisk to 7b443000, end 7be619cb ... OK
Loading Device Tree to 000000007b436000, end 000000007b442fc7 ... OK
Starting kernel ...
<CPU lockup?>
Here there are differences; the green LED remains lit, and MMC_CLK has good signal, but nothing on the data lines. There is no signal on GCLK0 but all of the SPI and I2C pins are held at 3.3V steady. No sign at all of clock.
Copy the working kernel and initramfs from our images and try it with those. Make sure bootargs are set (check grub.cfg in /boot).
Well, that made it extra weird. (Also note that for the moment, I’m testing using syslinux/extlinux.conf to eliminate grub2 from the variables.)
Using the Armbian 6.1.30 kernel and booti, immediate Synchronous Abort. Honestly can’t say that I’m surprised there. I haven’t had that one working at all.
Using the 2023-05-03 Raspbian Bullseye lite from CI and the 2022-07 u-boot results in similar but different symptoms. Instead of being completely dead, it goes to Starting kernel ... At that point, all three LEDs light for about 1 second, then the MMC data access LED (green) blinks at about 1Hz continuously. However, checking the data lines shows no activity, the reset button doesn’t work, and the GPIO lines are held again.
Unfortunately it’s not a flaky or cranky SD. This reproduced with all of them. No matter how slow or fast I wrote. And it’s completely unresponsive to all inputs. I’d say “oh, the board is bad” but it does the exact same on two s905x’s.
So I decided to try a wildly different tact; an SD card with nothing but u-boot and everything on a dead reliable SanDisk USB drive. That’s the only thing that changed behavior at all. With the OS on the USB drive, using load usb 0:1..., instead of even attempting anything it just sets all three LEDs to on and locks up solid with every kernel.
If you need help with Armbian, pleaase use the Armbian forum. They do too many things differently for us to be able to support.
Why would you do that? Not sure what you’re trying to do by doing this.
This is not the MMC access LED. It’s a heartbeat LED to indicate Linux has started and is operating correctly.
Again, not sure what your trying to accomplish with this. It’s obvious that your kernel and initramfs has an issue booting. It’s best to grab the working kernel and initramfs from our CI images and then try to boot it manually using the script. If it doesn’t boot, the script is bad. If the script does boot, then your custom kernel/initramfs is bad.
Why would you do that? Not sure what you’re trying to do by doing this.
Because we’ve already confirmed this specific u-boot is known working and stable, versus the latest in CI. I’m trying to control variables as much as possible, and since the board has never booted with the fixed CI, I can’t say for certain it’s 100% fixed.
This is not the MMC access LED. It’s a heartbeat LED to indicate Linux has started and is operating correctly.
Which raises a question: why is it showing heartbeat when it most definitely is not operating? If it was actually executing, there should be output on the HDMI and I should also see activity on the MMC data lines. It should be attempting root mount; it does not. It just falls flat on it’s face.
And just to be extra certain, I have a stack of other boards for testing the generic kernel builds. RPi3, RPi4, Pine A64-LTS, and a Radxa ROCK 3C. (So bcm, Allwinner A64, and RK3566.) Every one of them is able to run the exact same kernel just fine from the same build process when fed the correct DTBs. I also double checked, and their configuration for 6.1.42 checks out.
It’s best to grab the working kernel and initramfs from our CI images and then try to boot it manually using the script. If it doesn’t boot, the script is bad. If the script does boot, then your custom kernel/initramfs is bad.
Except that is exactly what I am saying: the kernel and initramfs from Libre’s CI images are not working.
I specifically grabbed 2023-05-03-raspbian-bullseye-arm64-lite, loop mounted it, copied the complete kernel including System.map to a good u-boot SD, and that kernel produces the exact same behavior.
When I write that exact image, as-is with no changes at all, it behaves exactly the same. The only difference is it finds grub, executes kernel load, and then I get a solid cyan or purple screen and everything is locked solid with no heartbeat. On both boards. Sometimes - but not consistently - it will get through the EFI stubs post-kernel and go to a solid yellow screen, still locked solid.
And obviously, with the official image doing largely the same, it’s not simply a kernel issue.
So, I had to set aside another project to swipe the FT232 off it (which I’ve been putting off because that project is a real pain to get back to a manageable state.) Which was illuminating in that I’ve been correct. It’s not reaching operating state.
GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:0;RCY:0;USB:0;SPI:0;CHK:A7;EMMC:400;N;
no sdio debug board detected
TE: 1745178
BL2 Built : 15:21:18, Aug 28 2019. gxl g1bf2b53 - luan.yuan@droid15-sz
set vcck to 1120 mv
set vddee to 1000 mv
Board ID = 3
CPU clk: 1200MHz
DQS-corr enabled
DDR scramble enabled
DDR3 chl: Rank0+1 @ 912MHz
bist_test rank: 0 1a 03 31 27 13 3b 17 00 2f 2b 14 43 18 01 2f 29 13 40 19 02 3S
Rank0: 1024MB(auto)-2T-13
Rank1: 1024MB(auto)-2T-13
AddrBus test pass!
Load fip header from SD, src: 0x0000c200, des: 0x01400000, size: 0x00004000, pa0
New fip structure!
Load bl30 from SD, src: 0x00010200, des: 0x013c0000, size: 0x0000d600, part: 0
Load bl31 from SD, src: 0x00020200, des: 0x05100000, size: 0x0001b800, part: 0
Load bl33 from SD, src: 0x0003c200, des: 0x01000000, size: 0x00086a00, part: 0
NOTICE: BL31: v1.3(release):c3714b49be
NOTICE: BL31: Built : 09:23:36, Jun 20 2023. gxl bl-3.5.0 gc3714b49be - jenkinh
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
[BL31]: tee size: 0
...
usual dmesg stuff
...
[ 4.095407] mmc1: new ultra high speed SDR104 SDXC card at address 5048
[ 4.097131] mmcblk1: mmc1:5048 SD64G 58.0 GiB
[ 4.103959] mmcblk1: p1 p2
Starting version 247.3-7+deb11u2
[ 4.183012] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[ 4.332592] usb 1-1: New USB device found, idVendor=05e3, idProduct=0610, bc8
[ 4.335226] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 4.342497] usb 1-1: Product: USB2.0 Hub
[ 4.403144] hub 1-1:1.0: USB hub found
[ 4.403558] hub 1-1:1.0: 4 ports detected
[ 4.415441] mmc0: Card stuck being busy! __mmc_poll_for_busy
[ 4.516508] meson-vrtc c81000a8.rtc: registered as rtc0
[ 4.516584] meson-vrtc c81000a8.rtc: setting system clock to 1970-01-01T00:0)
<halts here>
So post EFI, the HDMI is falling down. Every time, even though it’s a well known MacroSilicon. And as soon as it hits the RTC, dead lock.
And then, Alpine? Alpine’s been booting just fine with booti this whole time aside from a minor DTB issue! I actually had successful bring-up with both 2022-07 and latest u-boot several days ago, except the HDMI output is breaking as soon as the kernel starts booting. So I had a working config that was crashing for a wholly unrelated reason (busybox config mistake on my part.)
The adapter I’m using can also be ruled out; it’s a MacroSilicon MS2109. Is it great? Hell no. But it’s functional and cheap and I’m not admitting how many I own or have misplaced. 
So at this point, we can accurately say the following:
- Alpine Linux 3.18.2 with
linux-ltsis able to boot the kernel successfully, but HDMI fails immediately, and the boot fails early due to a distro issue I caused - Raspbian (Libre) 2023-05 does NOT boot successfully; HDMI fails either before or immediately after EFI stub, and then dies fully when the RTC is intialized. Occurs on both boards.
- Raspbian (Libre) 2023-05 also intermittently locks up attempting to execute EFI stubs which reproduced with multiple SD cards
- Kernel 6.1.40 (Raspbian) does not have functioning HDMI output, and never makes it past RTC
- Kernel 6.1.42 (Alpine) does not have functioning HDMI output, but is able to make it past RTC
- Both show identical results with u-boot 2022-07 or latest CI
So the question really is: why is HDMI breaking here, and why is the Raspbian image dying at RTC or before?
The CI images are tested on a dozen boards continuously. There’s no way it doesn’t work unless there’s a problem with power or MicroSD.
Use the latest in CI. u-boot controls the device tree which get updated for our kernel. These two go hand in hand and should not be used separately.
The heartbeat is the kernel heartbeat. If it’s running, the kernel is running code and operating correctly. HDMI output is controlled by the kernel driver which is a different subsystem. These are confirmed 100% working on all the images released on the distro server. Otherwise they will not be on the distro server.
The distro server images are confirmed working. There’s another problem with your setup if they don’t boot. Either you’re flashing incorrectly or the power is insufficient.
Just because they work on the other boards does not mean it has the necessary configs enabled for a different platform. This assumption is incorrect.
This is not possible.
Provide the full log, we can point out the issue for you.
The distro server images are confirmed working. There’s another problem with your setup if they don’t boot. Either you’re flashing incorrectly or the power is insufficient.
That’s what log I had. It is not working consistently, and the HDMI output fails every time. And the Raspbian image does not consistently boot. Given it’s multiple SD cards, up to 7.5A available, and the same writing tools I use with success daily, you’d have to tell me why. It either dies in EFI, or if EFI goes relatively slow, it has about a 50/50 shot of dying at RTC. Could be contacts need cleaned from all the swapping of SD cards, could be I’m hitting a weird firmware fault? I don’t know.
Anyways, regardless of all that, I have successful bring-up as of this afternoon. But I’m getting a crash and I’m presuming the information I need’s behind NDA.
GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:0;RCY:0;USB:0;SPI:0;CHK:A7;EMMC:400;NAND:81;SD:0;READ:0;0.0;CHK:0;
no sdio debug board detected
TE: 1806370
BL2 Built : 15:21:18, Aug 28 2019. gxl g1bf2b53 - luan.yuan@droid15-sz
set vcck to 1120 mv
set vddee to 1000 mv
Board ID = 3
CPU clk: 1200MHz
DQS-corr enabled
DDR scramble enabled
DDR3 chl: Rank0+1 @ 912MHz
bist_test rank: 0 1a 04 31 27 13 3c 17 00 2f 2b 14 43 17 00 2f 28 12 3f 18 01 2f 2b 14 42 693 rank: 1 18 06 2b 26 12 3b 16 00 2d 2b 14 43 18S
Rank0: 1024MB(auto)-2T-13
Rank1: 1024MB(auto)-2T-13
AddrBus test pass!
Load fip header from SD, src: 0x0000c200, des: 0x01400000, size: 0x00004000, part: 0
New fip structure!
Load bl30 from SD, src: 0x00010200, des: 0x013c0000, size: 0x0000d600, part: 0
Load bl31 from SD, src: 0x00020200, des: 0x05100000, size: 0x0001b800, part: 0
Load bl33 from SD, src: 0x0003c200, des: 0x01000000, size: 0x00086a00, part: 0
NOTICE: BL31: v1.3(release):c3714b49be
NOTICE: BL31: Built : 09:23:36, Jun 20 2023. gxl bl-3.5.0 gc3714b49be - jenkins@walle02-sh
NOTICE: BL3-1: GXL normal boot!
NOTICE: BL31: BL33 decompress pass
mpu_config_enable:system pre init ok
OPS=0x84
dmc sec lock
[Image: gxl_v1.1.3509-d977ed20a4 2023-06-20 09:43:46 jenkins@walle02-sh]
21 0d 84 00 92 31 1d 4a 8a 1d 30 96 80 89 47 92
[2.373746 Inits done]
secure task start!
high task start!
low task start!
ERROR: Error initializing runtime service opteed_fast
U-Boot 2023.07+ (Jul 25 2023 - 14:59:49 -0400) Libre Computer AML-S905X-CC
Model: Libre Computer AML-S905X-CC
SoC: Amlogic Meson GXL (S905X) Revision 21:d (84:2)
DRAM: 2 GiB
Core: 175 devices, 30 uclasses, devicetree: separate
WDT: Not starting watchdog@98d0
MMC: mmc@72000: 1, mmc@74000: 0
Loading Environment from FAT... Unable to read "uboot.env" from mmc1:1...
Error (-2): cannot determine file size
[BL31]: tee size: 0
[BL31]: tee size: 0
starting USB...
Bus usb@c9000000: dwc3_meson_gxl_get_phys: usb2 ports: 2
Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.00
scanning bus usb@c9000000 for devices... 2 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
Scanning for bootflows in all bootdevs
Seq Method State Uclass Part Name Filename
--- ----------- ------ -------- ---- ------------------------ ----------------
Scanning global bootmeth 'efi_mgr':
Scanning bootdev 'mmc@74000.bootdev':
Scanning bootdev 'mmc@72000.bootdev':
0 extlinux ready mmc 1 mmc@72000.bootdev.part_1 /extlinux/extlinux.conf
** Booting bootflow 'mmc@72000.bootdev.part_1' with extlinux
1: Linux Alpine 3.18
Retrieving file: /extlinux/../vmlinuz-lts
Retrieving file: /extlinux/../initramfs-lts
... all normal, nothing out of sorts or unusual ...
[ 1.108142] irq_meson_gpio: 110 to 8 gpio interrupt mux initialized
[ 1.154439] soc soc0: Amlogic Meson GXL (S905X) Revision 21:d (84:2) Detected
1.167628] c81004c0.serial: ttyAML0 at MMIO 0xc81004c0 (irq = 18, base_baud = 1500000) is a meson_uart
^^^ obviously things are looking good here
... more normal ...
[ 1.346428] Run /init as init process
[ 1.398678] Alpine Init 3.8.1-r0
[ 1.399945] Loading boot drivers...
[ 1.428891] loop: module loaded
[ 1.457164] SCSI subsystem initialized
[ 1.534959] usbcore: registered new interface driver usbfs
[ 1.535106] usbcore: registered new interface driver hub
[ 1.540222] usbcore: registered new device driver usb
[ 1.557531] usbcore: registered new interface driver usb-storage
[ 1.693580] [drm] Initialized simpledrm 1.0.0 20200625 for 7fe5b000.framebuffer on minor 0
[ 1.697272] fbcon: Deferring console take-over
[ 1.700644] simple-framebuffer 7fe5b000.framebuffer: [drm] fb0: simpledrmdrmfb frame buffer device
[ 1.724094] Loading boot drivers: ok.
[ 1.731453] Mounting boot media...
[ 2.036945] meson-gx-mmc d0072000.mmc: Got CD GPIO
[ 2.107238] mmc0: new high speed SDHC card at address 5048
!!! Reproducible crash occurs here !!!
[ 2.136138] lima d00c0000.gpu: gp - mali450 version major 0 minor 0
[ 2.136957] lima d00c0000.gpu: pp0 - mali450 version major 0 minor 0
[ 2.143216] lima d00c0000.gpu: pp1 - mali450 version major 0 minor 0
[ 2.149463] lima d00c0000.gpu: pp2 - mali450 version major 0 minor 0
[ 2.155712] lima d00c0000.gpu: l2 cache 8K, 4-way, 64byte cache line, 128bit external bus
[ 2.163788] lima d00c0000.gpu: l2 cache 64K, 4-way, 64byte cache line, 128bit external bus
[ 2.172599] lima d00c0000.gpu: bus rate = 166666667
[ 2.176862] lima d00c0000.gpu: mod rate = 24000000
[ 2.181735] lima d00c0000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
[ 2.198989] lima d00c0000.gpu: devfreq_add_device: Unable to find governor for the device
[ 2.201699] lima d00c0000.gpu: Couldn't initialize GPU devfreq
[ 2.207334] lima d00c0000.gpu: Fatal error during devfreq init
[ 2.213424] lima: probe of d00c0000.gpu failed with error -22
[ 2.219685] Internal error: synchronous external abort: 0000000096000210 [#1] SMP <-----
[ 2.226198] Modules linked in: lima(+) gpu_sched meson_gx_mmc mmc_core gpio_regulator display_connector simpledrm drm_shmem_helper drm_kmsp
[ 2.256642] CPU: 2 PID: 506 Comm: nlplug-findfs Not tainted 6.1.42-0-lts #1-Alpine
[ 2.264144] Hardware name: Libre Computer AML-S905X-CC (DT)
[ 2.269665] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 2.276564] pc : lima_pp_irq_handler+0x2c/0xf0 [lima]
[ 2.281566] lr : free_irq+0x330/0x4c4
[ 2.285189] sp : ffff80000a7fb7a0
[ 2.288466] x29: ffff80000a7fb7a0 x28: ffff80000a7fbc00 x27: 0000000000000000
[ 2.295538] x26: 0000000000000000 x25: ffff000073f8d268 x24: ffff0000741fac60
[ 2.302611] x23: ffff0000741facdc x22: ffff0000741fad90 x21: 000000000000001c
[ 2.309683] x20: ffff000073f8d080 x19: ffff000073f8d268 x18: 0000000000000000
[ 2.316756] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 2.323828] x14: 0000000000000000 x13: 32322d20726f7272 x12: 6520687469772064
[ 2.330901] x11: 656c696166207570 x10: 0000000000000000 x9 : ffff800008153ad0
[ 2.337974] x8 : ffff80000a7fb6a0 x7 : 0000000000000000 x6 : 0000000000000000
[ 2.345046] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 2.352118] x2 : ffff80000ac4d02c x1 : ffff000073f8d268 x0 : 000000000000102c
[ 2.359192] Call trace:
[ 2.361608] lima_pp_irq_handler+0x2c/0xf0 [lima]
[ 2.366265] free_irq+0x330/0x4c4
[ 2.369541] devm_irq_release+0x28/0x50
[ 2.373336] devres_release_all+0xe4/0x1ec
[ 2.377390] device_unbind_cleanup+0x24/0x84
[ 2.381616] really_probe+0x210/0x404
[ 2.385239] __driver_probe_device+0x84/0x17c
[ 2.389551] driver_probe_device+0x4c/0x134
[ 2.393691] __driver_attach+0x140/0x270
[ 2.397573] bus_for_each_dev+0x84/0xf4
[ 2.401367] driver_attach+0x34/0x60
[ 2.404904] bus_add_driver+0x1bc/0x274
[ 2.408699] driver_register+0x7c/0x170
[ 2.412494] __platform_driver_register+0x38/0x60
[ 2.417151] lima_platform_driver_init+0x34/0x1000 [lima]
[ 2.422499] do_one_initcall+0x60/0x29c
[ 2.426294] do_init_module+0x50/0x210
[ 2.430003] load_module+0x1c3c/0x21f0
[ 2.433711] __do_sys_init_module+0x1f8/0x230
[ 2.438024] __arm64_sys_init_module+0x28/0x50
[ 2.442423] invoke_syscall+0x8c/0x134
[ 2.446131] el0_svc_common.constprop.0+0x60/0x13c
[ 2.450875] do_el0_svc+0x40/0xe0
[ 2.454153] el0_svc+0x34/0xf4
[ 2.457171] el0t_64_sync_handler+0x114/0x140
[ 2.461484] el0t_64_sync+0x18c/0x190
[ 2.465112] Code: d2820580 f9400a62 f9400274 8b000042 (b9400041)
[ 2.471147] ---[ end trace 0000000000000000 ]---
[ 2.475716] note: nlplug-findfs[506] exited with irqs disabled
[ 260.516242] random: crng init done <-- extremely long crng?
[ 1716.672770] mmc0: card 5048 removed <-- showing system still responsive after crash
[ 1848.576396] mmc0: new high speed SDHC card at address 5048
I have a feeling the solution to this is behind NDA, because it’s crashing due to lima_pp_irq_handler() being called after the mali init failure. The only cited workaround from the 5.x train was the ondemand governor, which is enabled in Alpine. (MESON is also a y at line 54.)
The behavior is definitely very odd to my eyes. I haven’t looked at the mali driver in depth before, but version major 0 version minor 0 looks like a failure populating the lima_device struct. Nominally I’d chalk it up to board failure but both boards do it. And so does Raspbian when it boots.
[ 9.468171] [drm] Initialized meson 1.0.0 20161109 for d0100000.vpu on minor 0
[ 9.480867] lima d00c0000.gpu: Looking up mali-supply from device tree
[ 9.480888] lima d00c0000.gpu: Looking up mali-supply property in node /soc/apb@d0000000/gpu@c0000 failed
[ 9.486709] lima d00c0000.gpu: gp - mali450 version major 0 minor 0
[ 9.486807] lima d00c0000.gpu: pp0 - mali450 version major 0 minor 0
[ 9.486900] lima d00c0000.gpu: pp1 - mali450 version major 0 minor 0
[ 9.486981] lima d00c0000.gpu: pp2 - mali450 version major 0 minor 0
[ 9.487027] lima d00c0000.gpu: l2 cache 8K, 4-way, 64byte cache line, 128bit external bus
[ 9.487039] lima d00c0000.gpu: l2 cache 64K, 4-way, 64byte cache line, 128bit external bus
[ 9.516525] lima d00c0000.gpu: bus rate = 166666667
[ 9.516547] lima d00c0000.gpu: mod rate = 24000000
[ 9.516708] lima d00c0000.gpu: Looking up mali-supply from device tree
[ 9.516723] lima d00c0000.gpu: Looking up mali-supply property in node /soc/apb@d0000000/gpu@c0000 failed
[ 9.516754] lima d00c0000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
[ 9.518525] [drm] Initialized lima 1.1.0 20191231 for d00c0000.gpu on minor 1
The only real difference is Raspbian finding the governor. However, the dtb files between Alpine and Raspbian are identical. Both come from the kernel under the amlogic tree. So why is 6.1.42 unable to find the governor leading to the async abort?
Assuming this is using our kernel/initramfs, You don’t have the kernel modules install in your Alpine Linux path since you took only the kernel and initramfs but not the modules. When the GPU driver probe, it can’t find devfreq governor module from the disk and crashes as expected. This doesn’t mean our HDMI doesn’t work. The kernel cannot find its modules to init everything.
You don’t have the kernel modules install in your Alpine Linux path since you took only the kernel and initramfs but not the modules. When the GPU driver probe, it can’t find devfreq governor module from the disk and crashes as expected. This doesn’t mean our HDMI doesn’t work. The kernel cannot find its modules to init everything.
The HDMI problem is still present with both images, so that has to be a UEFI/u-boot issue especially since it occurs as soon as UEFI is completed. It reproduces for me on Raspbian, CoreELEC, and Alpine. Maybe a regression with EDID stuff and this particular adapter?
Anyway, I now have full kernel bringup on Alpine with 6.1.42 all the way through root mount after adding ARM_MEDIATEK_CPUFREQ=y and MEDIATEK_CCI_DEVFREQ=m. Would not have thought the Mali wants MediaTek but hey. It (mostly) works.
MMC seems to be the only remaining problem. Root by UUID and PARTUUID both fail to find the root volume unexpectedly, and by device path /dev/mmcblk0p2 mounts but the board immediately dies with the last message being [BL31]: tee size: 0
[ 1.319210] Run /init as init process
[ 1.372190] Alpine Init 3.8.1-r0
[ 1.373366] Loading boot drivers...
[ 1.413472] SCSI subsystem initialized
[ 1.490440] usbcore: registered new interface driver usbfs
[ 1.490542] usbcore: registered new interface driver hub
[ 1.495690] usbcore: registered new device driver usb
[ 1.513058] usbcore: registered new interface driver usb-storage
[ 1.642748] loop: module loaded
[ 1.780217] [drm] Initialized simpledrm 1.0.0 20200625 for 7fe5b000.framebuffer on minor 0
[ 1.783850] fbcon: Deferring console take-over
[ 1.787289] simple-framebuffer 7fe5b000.framebuffer: [drm] fb0: simpledrmdrmfb frame buffer device
[ 1.813828] Loading boot drivers: ok.
[ 1.820979] Mounting root...
[ 2.091615] meson-gx-mmc d0072000.mmc: Got CD GPIO
[ 2.159863] mmc0: new high speed SDHC card at address 5048
[ 2.176966] lima d00c0000.gpu: gp - mali450 version major 0 minor 0
[ 2.177694] lima d00c0000.gpu: pp0 - mali450 version major 0 minor 0
[ 2.184001] lima d00c0000.gpu: pp1 - mali450 version major 0 minor 0
[ 2.190301] lima d00c0000.gpu: pp2 - mali450 version major 0 minor 0
[ 2.196549] lima d00c0000.gpu: l2 cache 8K, 4-way, 64byte cache line, 128bit external bus
[ 2.204610] lima d00c0000.gpu: l2 cache 64K, 4-way, 64byte cache line, 128bit external bus
[ 2.213360] lima d00c0000.gpu: bus rate = 166666667
[ 2.217676] lima d00c0000.gpu: mod rate = 24000000
[ 2.222560] lima d00c0000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
[ 2.243884] lima d00c0000.gpu: Failed to register cooling device
[ 2.244809] [drm] Initialized lima 1.1.0 20191231 for d00c0000.gpu on minor 1
[ 2.318950] meson-drm d0100000.vpu: Queued 2 outputs on vpu
[ 2.422540] meson-dw-hdmi c883a000.hdmi-tx: Detected HDMI TX controller v2.01a with HDCP (meson_dw_hdmi_phy)
[ 2.427245] meson-dw-hdmi c883a000.hdmi-tx: registered DesignWare HDMI I2C bus driver
[ 2.434964] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
[ 2.444504] [drm] Initialized meson 1.0.0 20161109 for d0100000.vpu on minor 2
[ 2.456182] fbcon: Deferring console take-over
[ 2.456225] meson-drm d0100000.vpu: [drm] fb0: mesondrmfb frame buffer device
[ 2.629323] mmcblk0: mmc0:5048 SD16G 14.4 GiB
[ 2.632863] mmcblk0: p1 p2
[ 3.188733] EXT4-fs (mmcblk0p2): orphan cleanup on readonly fs
[ 3.189225] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none.
[ 3.198393] Mounting root: ok.
[BL31]: tee size: 0
No idea what’s going on there, as I can’t find any reference to BL31 and I presume any documentation is in the NDA’d materials. And I don’t see any other modules present in Raspbian that are not present in Alpine.
CoreELEC uses a completely different kernel than we do. They’re using the Amlogic kernel where as we’re using the upstream kernel. Not sure what exactly you’re doing here or what exactly you’re reproducing. Given that the kernels are in our images and tested across every board, if it doesn’t work for you, it means some setup is not right.
Our kernel and defconfig are all that’s needed to boot with full support. This is the same kernel in our images that has proven to work across multiple distros. There’s no need to add or remove configs.
This is an issue with whatever customizations you have in your kernel. The official images all boot via UUID. The below is taken from Raspbian and works: linux /boot/vmlinuz-6.1.42-08242-g1ca3e35555cd root=UUID=7940c4d1-4894-4442-9df8-da2c0311ee5e ro noquiet