How to debug grub2-efi -> kernel transition issues?

[Richard W.M. Jones]

I have a weird problem on an old Asus Zenbook UX305C where new kernels
cannot be installed by grub.  Specifically what happens is they appear
in the boot menu fine, but if you try to boot them then the machine
hangs hard with a completely black screen.

Oddly the kernel installed by Anaconda can boot, but obviously this
makes upgrading the kernel RPM impossible.

My real question is how on earth do I start debugging this?

I edited the kernel command line, removed “rhgb quiet”, added
“loglevel=9 nomodeset” but still no output at all is visible before
the hang.  The machine doesn't have a serial port.

Any ideas?  Does grub have debugging that can be enabled somehow?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Chris Murphy]

On Mon, Mar 29, 2021 at 1:55 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  I have a weird problem on an old Asus Zenbook UX305C where new kernels
>  cannot be installed by grub.  Specifically what happens is they appear
>  in the boot menu fine, but if you try to boot them then the machine
>  hangs hard with a completely black screen.
>
>  Oddly the kernel installed by Anaconda can boot, but obviously this
>  makes upgrading the kernel RPM impossible.
>
>  My real question is how on earth do I start debugging this?
>
>  I edited the kernel command line, removed “rhgb quiet”, added
>  “loglevel=9 nomodeset” but still no output at all is visible before
>  the hang.  The machine doesn't have a serial port.
>
>  Any ideas?  Does grub have debugging that can be enabled somehow? 
With rhgb and quiet removed, you should immediately get a boot scroll
from the kernel and a panic if that's what's happening. If not then
this is somewhere in between the hand off. When GRUB has a problem
loading kernel or initramfs, it complains and goes back to GRUB. So
this sounds like the kernel and initramfs did load, and boot command
is executed and something fails at that point or right in the kernel
at the earliest stage.

I'd ask in #fedora-kernel to be sure of a better idea. But my idea is
edit the failing grub menu entry (in GRUB) and after the initrd line
add two more lines

set pager=1
set debug=all

Then F10 or Ctrl-X to boot that edited entry. You should get maybe
just 4 lines of cryptic grub debug code (in my working case it doesn't
mean anything to me) and then it goes black then i get kernel boot
scroll.

If you get a ton of errors, probably failure in grub. If you don't,
probably already handed off to the kernel and it faceplanted during
its very early bootstrapping sequence before we even usually see
anything.



-- 
Chris Murphy

[Richard W.M. Jones]

On Mon, Mar 29, 2021 at 02:02:58PM -0600, Chris Murphy wrote:
>  On Mon, Mar 29, 2021 at 1:55 AM Richard W.M. Jones
> <rjones(a)redhat.com&gt; wrote:
>  >
>  >
>  > I have a weird problem on an old Asus Zenbook UX305C where new kernels
>  > cannot be installed by grub.  Specifically what happens is they appear
>  > in the boot menu fine, but if you try to boot them then the machine
>  > hangs hard with a completely black screen.
>  >
>  > Oddly the kernel installed by Anaconda can boot, but obviously this
>  > makes upgrading the kernel RPM impossible.
>  >
>  > My real question is how on earth do I start debugging this?
>  >
>  > I edited the kernel command line, removed “rhgb quiet”, added
>  > “loglevel=9 nomodeset” but still no output at all is visible before
>  > the hang.  The machine doesn't have a serial port.
>  >
>  > Any ideas?  Does grub have debugging that can be enabled somehow?
>  
>  With rhgb and quiet removed, you should immediately get a boot scroll
>  from the kernel and a panic if that's what's happening. If not then
>  this is somewhere in between the hand off. When GRUB has a problem
>  loading kernel or initramfs, it complains and goes back to GRUB. So
>  this sounds like the kernel and initramfs did load, and boot command
>  is executed and something fails at that point or right in the kernel
>  at the earliest stage.
>  
>  I'd ask in #fedora-kernel to be sure of a better idea. But my idea is
>  edit the failing grub menu entry (in GRUB) and after the initrd line
>  add two more lines
>  
>  set pager=1
>  set debug=all
>  
>  Then F10 or Ctrl-X to boot that edited entry. You should get maybe
>  just 4 lines of cryptic grub debug code (in my working case it doesn't
>  mean anything to me) and then it goes black then i get kernel boot
>  scroll. 
Thanks!  Yes, cryptic indeed.  This is transcribed from a photograph
so hopefully I didn't make a mistake:

  script/lexer.c:336: token 259 text [
  ]
  script/lexer.c:336: token 0 text []
  script/lexer.c:336: token 259 text [
  ]
  script/lexer.c:336: token 0 text []
  loader/efi/linux.c:95: kernel_addr: 0x1000000 handover_offset: 0x190 params:
  0x7f050000
  loader/efi/linux.c:98: handover_func() = 0x1000390
  _

With the final "_" meaning the cursor (not blinking).  At this point
the machine appears to be hung -- capslock key no longer toggles the
capslock indicator light, keys do nothing.

It seems to be a kind of well-known bug caused apparently by either
faulty firmware or microcode.  See eg:

https://bugzilla.redhat.com/show_bug.cgi?id=1724262
https://bugzilla.redhat.com/show_bug.cgi?id=1790115
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/1

I guess I'll need to find a BIOS update for this laptop somewhere on
Asus's terrible website :-(

Rich.

>  If you get a ton of errors, probably failure in grub. If you
> don't,
>  probably already handed off to the kernel and it faceplanted during
>  its very early bootstrapping sequence before we even usually see
>  anything.
>  
>  
>  
>  -- 
>  Chris Murphy
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/