Anyone know how to get fedora + xen running in a KVM VM? I'd like to be able to do basic xen/libxl/libvirt testing without needed to reboot a physical machine.
Trying with F23 AMD host, F22+xen L1, trying to boot into xen in the L1 VM just reboots very soon after trying to boot the kernel. Tried disabling all virtio for L1 VM, disabling virt extensions for the L1 VM, but it didn't seem to change anything. Unfortunately the messages scroll by so quickly I can't tell what's happening right before it reboots, and all efforts to convince it to print more debugging haven't worked.
FWIW this does work with rhel5+xen L1, but that's a world away at this point.
So anyone know if it's even possible? Is there a trick to it? If not, suggestions on getting more debug output from xen + grub2?
Thanks, Cole
On 09/15/2015 03:03 PM, Cole Robinson wrote:
Anyone know how to get fedora + xen running in a KVM VM? I'd like to be able to do basic xen/libxl/libvirt testing without needed to reboot a physical machine.
Trying with F23 AMD host, F22+xen L1, trying to boot into xen in the L1 VM just reboots very soon after trying to boot the kernel. Tried disabling all virtio for L1 VM, disabling virt extensions for the L1 VM, but it didn't seem to change anything. Unfortunately the messages scroll by so quickly I can't tell what's happening right before it reboots, and all efforts to convince it to print more debugging haven't worked.
FWIW this does work with rhel5+xen L1, but that's a world away at this point.
So anyone know if it's even possible? Is there a trick to it? If not, suggestions on getting more debug output from xen + grub2?
I've had a tubload of trouble with Xen 4.5 on Fedora 22. However, you *should* be able to boot a Xen hypervisor as a KVM guest. It requires nested virt on the host itself. I'm not sure if AMD CPUs are capable of doing that, but I've had luck with this on Intel Xeons (Nehalem and later).
If F22+Xen as a KVM guest isn't working out, you may need to try F21+Xen as a guest and see if that works.
Examples of the trouble I've had with Xen so far:
https://bugzilla.redhat.com/show_bug.cgi?id=1219197
I usually end up following this page to get some serial output:
http://wiki.xen.org/wiki/Xen_Serial_Console
Once you get some serial output and prevent Xen from rapidly power cycling, you should get something moderately useful for troubleshooting on the graphical or serial console.
The noreboot=true and sync_console options might help you troubleshoot, too:
http://xenbits.xen.org/docs/4.2-testing/misc/xen-command-line.html
-- Major Hayden
On Tue, Sep 15, 2015 at 03:19:25PM -0500, Major Hayden wrote:
On 09/15/2015 03:03 PM, Cole Robinson wrote:
Anyone know how to get fedora + xen running in a KVM VM? I'd like to be able to do basic xen/libxl/libvirt testing without needed to reboot a physical machine.
Trying with F23 AMD host, F22+xen L1, trying to boot into xen in the L1 VM just reboots very soon after trying to boot the kernel. Tried disabling all virtio for L1 VM, disabling virt extensions for the L1 VM, but it didn't seem to change anything. Unfortunately the messages scroll by so quickly I can't tell what's happening right before it reboots, and all efforts to convince it to print more debugging haven't worked.
FWIW this does work with rhel5+xen L1, but that's a world away at this point.
So anyone know if it's even possible? Is there a trick to it? If not, suggestions on getting more debug output from xen + grub2?
I've had a tubload of trouble with Xen 4.5 on Fedora 22. However, you *should* be able to boot a Xen hypervisor as a KVM guest. It requires nested virt on the host itself. I'm not sure if AMD CPUs are capable of doing that, but I've had luck with this on Intel Xeons (Nehalem and later).
If F22+Xen as a KVM guest isn't working out, you may need to try F21+Xen as a guest and see if that works. a
It works great!
Examples of the trouble I've had with Xen so far:
https://bugzilla.redhat.com/show_bug.cgi?id=1219197
I usually end up following this page to get some serial output:
http://wiki.xen.org/wiki/Xen_Serial_Console
Once you get some serial output and prevent Xen from rapidly power cycling, you should get something moderately useful for troubleshooting on the graphical or serial console.
The noreboot=true and sync_console options might help you troubleshoot, too:
http://xenbits.xen.org/docs/4.2-testing/misc/xen-command-line.html
-- Major Hayden -- xen mailing list xen@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/xen
On Tue, 15 Sep 2015, Cole Robinson wrote:
Anyone know how to get fedora + xen running in a KVM VM? I'd like to be able to do basic xen/libxl/libvirt testing without needed to reboot a physical machine.
Trying with F23 AMD host, F22+xen L1, trying to boot into xen in the L1 VM just reboots very soon after trying to boot the kernel. Tried disabling all virtio for L1 VM, disabling virt extensions for the L1 VM, but it didn't seem to change anything. Unfortunately the messages scroll by so quickly I can't tell what's happening right before it reboots, and all efforts to convince it to print more debugging haven't worked.
FWIW this does work with rhel5+xen L1, but that's a world away at this point.
So anyone know if it's even possible? Is there a trick to it? If not, suggestions on getting more debug output from xen + grub2?
It works for me after a basic and probably sub-optimal install, as follows
* qemu-kvm -cdrom Fedora-Server-DVD-x86_64-22.iso -m 1024 -hda kvm.img * do a minimal install * after reboot dnf install xen xen-hypervisor * select "Fedora, with Xen hypervisor" in the boot menu
If you need more debugging, boot without xen, and edit the grub.cfg file, adding console=com1 after xen.gz and replacing rhgb quiet with console=hvc0 in the 'Fedora, with Xen hypervisor' section, then boot kvm with -serial eg. -serial file:boot.log which should give you xen logging after grub has finished.
Michael Young
On 09/15/2015 05:35 PM, M A Young wrote:
On Tue, 15 Sep 2015, Cole Robinson wrote:
Anyone know how to get fedora + xen running in a KVM VM? I'd like to be able to do basic xen/libxl/libvirt testing without needed to reboot a physical machine.
Trying with F23 AMD host, F22+xen L1, trying to boot into xen in the L1 VM just reboots very soon after trying to boot the kernel. Tried disabling all virtio for L1 VM, disabling virt extensions for the L1 VM, but it didn't seem to change anything. Unfortunately the messages scroll by so quickly I can't tell what's happening right before it reboots, and all efforts to convince it to print more debugging haven't worked.
FWIW this does work with rhel5+xen L1, but that's a world away at this point.
So anyone know if it's even possible? Is there a trick to it? If not, suggestions on getting more debug output from xen + grub2?
It works for me after a basic and probably sub-optimal install, as follows
- qemu-kvm -cdrom Fedora-Server-DVD-x86_64-22.iso -m 1024 -hda kvm.img
- do a minimal install
- after reboot dnf install xen xen-hypervisor
- select "Fedora, with Xen hypervisor" in the boot menu
Thanks, the minimal command line worked for me. It allowed me to track it back and and find the culprit.
It comes down the -cpu value; the default qemu64 works, but any Opteron_* or -cpu host do not work. I had played around with this originally thinking maybe it was a nested virt issue, and tried Opteron_G1 which lacks svm, but never actually tried the default qemu64 cpu...
So this doesn't work:
$ qemu-kvm -m 1024 -hda kvm.img -cpu Opteron_G1
However this does:
$ qemu-kvm -m 1024 -hda kvm.img -cpu Opteron_G1,family=6
qemu64 uses family=6, but Opteron_G1 uses family=15. I see there's some bits of qemu that trigger off family>6 regarding setting some bits about MCE, but I tried disabling those in the code and things still errored. More data points:
- TCG works fine for every combo I tried - cpu qemu64,family=15 fails - None of this affects my intel host, all these work fine: -cpu host, -cpu Opteron_G1, -cpu qemu64, -cpu qemu64,family=15
So given all that, anyone have a suggestion of whether this is a KVM or Xen bug?
If you need more debugging, boot without xen, and edit the grub.cfg file, adding console=com1 after xen.gz and replacing rhgb quiet with console=hvc0 in the 'Fedora, with Xen hypervisor' section, then boot kvm with -serial eg. -serial file:boot.log which should give you xen logging after grub has finished.
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
sync_console gives an extra message but that's all I could see. Is console=hvc0 correct for a KVM guest, or is that only going to work for xen-on-xen ? I tried console=ttyS0 as well but it didn't make a difference
Thanks, Cole
On Wed, Sep 16, 2015 at 03:48:34PM -0400, Cole Robinson wrote:
On 09/15/2015 05:35 PM, M A Young wrote:
On Tue, 15 Sep 2015, Cole Robinson wrote:
Anyone know how to get fedora + xen running in a KVM VM? I'd like to be able to do basic xen/libxl/libvirt testing without needed to reboot a physical machine.
Trying with F23 AMD host, F22+xen L1, trying to boot into xen in the L1 VM just reboots very soon after trying to boot the kernel. Tried disabling all virtio for L1 VM, disabling virt extensions for the L1 VM, but it didn't seem to change anything. Unfortunately the messages scroll by so quickly I can't tell what's happening right before it reboots, and all efforts to convince it to print more debugging haven't worked.
FWIW this does work with rhel5+xen L1, but that's a world away at this point.
So anyone know if it's even possible? Is there a trick to it? If not, suggestions on getting more debug output from xen + grub2?
It works for me after a basic and probably sub-optimal install, as follows
- qemu-kvm -cdrom Fedora-Server-DVD-x86_64-22.iso -m 1024 -hda kvm.img
- do a minimal install
- after reboot dnf install xen xen-hypervisor
- select "Fedora, with Xen hypervisor" in the boot menu
Thanks, the minimal command line worked for me. It allowed me to track it back and and find the culprit.
It comes down the -cpu value; the default qemu64 works, but any Opteron_* or -cpu host do not work. I had played around with this originally thinking maybe it was a nested virt issue, and tried Opteron_G1 which lacks svm, but never actually tried the default qemu64 cpu...
So this doesn't work:
$ qemu-kvm -m 1024 -hda kvm.img -cpu Opteron_G1
However this does:
$ qemu-kvm -m 1024 -hda kvm.img -cpu Opteron_G1,family=6
qemu64 uses family=6, but Opteron_G1 uses family=15. I see there's some bits of qemu that trigger off family>6 regarding setting some bits about MCE, but I tried disabling those in the code and things still errored. More data points:
- TCG works fine for every combo I tried
- cpu qemu64,family=15 fails
- None of this affects my intel host, all these work fine: -cpu host, -cpu
Opteron_G1, -cpu qemu64, -cpu qemu64,family=15
So given all that, anyone have a suggestion of whether this is a KVM or Xen bug?
If you need more debugging, boot without xen, and edit the grub.cfg file, adding console=com1 after xen.gz and replacing rhgb quiet with console=hvc0 in the 'Fedora, with Xen hypervisor' section, then boot kvm with -serial eg. -serial file:boot.log which should give you xen logging after grub has finished.
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
Perhaps you need to enable some KVM setting to use the COM1 and output the value somewhere?
sync_console gives an extra message but that's all I could see. Is console=hvc0 correct for a KVM guest, or is that only going to work for xen-on-xen ? I tried console=ttyS0 as well but it didn't make a difference
Just Xen.
But since you are using QEMU you can figure out where the CPU is at. As in you should be able to get an EIP pointer and perhaps even a stack? There are some magic commands in the QMEU monitor pty.
B
Thanks, Cole -- xen mailing list xen@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/xen
On Wed, 16 Sep 2015, Cole Robinson wrote:
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
sync_console gives an extra message but that's all I could see. Is console=hvc0 correct for a KVM guest, or is that only going to work for xen-on-xen ? I tried console=ttyS0 as well but it didn't make a difference
It is hvc0 to send it to xen and thus onto the console you specified above. You can get some early debug messages of the kernel with earlyprintk=xen and "log_lvl=all guest_loglvl=all" are recommended at http://wiki.xenproject.org/wiki/Debugging_Xen for the xen hypervisor line.
Michael Young
On 09/16/2015 04:07 PM, M A Young wrote:
On Wed, 16 Sep 2015, Cole Robinson wrote:
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
Xen 4.5.1-5.fc22 (XEN) Xen version 4.5.1 (mockbuild@[unknown]) (gcc (GCC) 5.1.1 20150618 (Red Hat 5.1.1-4)) debug=n Sun Aug 2 16:23:51 UTC 2015 (XEN) Latest ChangeSet: (XEN) Bootloader: GRUB 2.02~beta2 (XEN) Command line: placeholder console=com1 (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009fc00 (usable) (XEN) 000000000009fc00 - 00000000000a0000 (reserved) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000007ffe0000 (usable) (XEN) 000000007ffe0000 - 0000000080000000 (reserved) (XEN) 00000000feffc000 - 00000000ff000000 (reserved) (XEN) 00000000fffc0000 - 0000000100000000 (reserved) (XEN) System RAM: 2047MB (2096636kB) (XEN) ACPI: RSDP 000F6300, 0014 (r0 BOCHS ) (XEN) ACPI: RSDT 7FFE16EE, 0034 (r1 BOCHS BXPCRSDT 1 BXPC 1) (XEN) ACPI: FACP 7FFE0C14, 0074 (r1 BOCHS BXPCFACP 1 BXPC 1) (XEN) ACPI: DSDT 7FFE0040, 0BD4 (r1 BOCHS BXPCDSDT 1 BXPC 1) (XEN) ACPI: FACS 7FFE0000, 0040 (XEN) ACPI: SSDT 7FFE0C88, 09B6 (r1 BOCHS BXPCSSDT 1 BXPC 1) (XEN) ACPI: APIC 7FFE163E, 0078 (r1 BOCHS BXPCAPIC 1 BXPC 1) (XEN) ACPI: HPET 7FFE16B6, 0038 (r1 BOCHS BXPCHPET 1 BXPC 1) (XEN) Domain heap initialised (XEN) Processor #0 5:1 APIC version 20 (XEN) IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) Not enabling x2APIC: depends on iommu_supports_eim. (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: Initializing. (XEN) AVC INITIALIZED (XEN) Flask: Starting in permissive mode. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 3624.462 MHz processor. (XEN) Initing memory sharing. (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7 (XEN) AMD-Vi: IOMMU not found! (XEN) I/O virtualisation disabled (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) Platform timer is 100.000MHz HPET (XEN) Allocated console ring of 16 KiB. (XEN) HVM: ASIDs enabled. (XEN) SVM: Supported advanced features: (XEN) - Nested Page Tables (NPT) (XEN) - Next-RIP Saved on #VMEXIT (XEN) HVM: SVM enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) Brought up 1 CPUs (XEN) Dom0 has maximum 208 PIRQs (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2044000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000074000000->0000000078000000 (465487 pages to be allocated) (XEN) Init. ramdisk: 000000007ec73000->000000007f9ffb4e (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff82044000 (XEN) Init. ramdisk: 0000000000000000->0000000000000000 (XEN) Phys-Mach map: ffffffff82044000->ffffffff823f7ee0 (XEN) Start info: ffffffff823f8000->ffffffff823f84b4 (XEN) Page tables: ffffffff823f9000->ffffffff82410000 (XEN) Boot stack: ffffffff82410000->ffffffff82411000 (XEN) TOTAL: ffffffff80000000->ffffffff82800000 (XEN) ENTRY ADDRESS: ffffffff81d381f0 (XEN) Dom0 has maximum 1 VCPUs (XEN) Scrubbing Free RAM on 1 nodes using 1 CPUs (XEN) ................done. (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Std. Loglevel: Errors and warnings (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 308kB init memory. mapping kernel into physical memory about to get started... (XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff810032b0>] (XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest (XEN) rax: 0000000000000015 rbx: ffffffff81c03e1c rcx: 00000000c0010112 (XEN) rdx: 0000000000000001 rsi: ffffffff81c03e1c rdi: 00000000c0010112 (XEN) rbp: ffffffff81c03df8 rsp: ffffffff81c03da0 r8: ffffffff81c03e28 (XEN) r9: ffffffff81c03e2c r10: 0000000000000000 r11: 00000000ffffffff (XEN) r12: ffffffff81d0ec20 r13: 0000000004000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000000406f0 (XEN) cr3: 0000000075c0b000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff81c03da0: (XEN) 00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810032b0 (XEN) 000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b (XEN) 0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48 (XEN) ffffffff81029a14 ffffffff81c03e48 ffffffff81029c9b ffffffff81c03e48 (XEN) 17c9660328d0b468 0000000001000000 ffffffff81c03ef8 0000000004000000 (XEN) 0000000000000000 ffffffff81c03e58 ffffffff81d44196 ffffffff81c03ee8 (XEN) ffffffff81d3fb6d 0000000000000000 0000000000000000 ffffffff81c03e88 (XEN) ffffffff810f2a99 ffffffff81c03ee8 ffffffff81781688 ffffffff00000010 (XEN) ffffffff81c03ef8 ffffffff81c03eb8 17c9660328d0b468 ffffffff81f02d5c (XEN) ffffffffffffffff 0000000000000000 ffffffff81ddd920 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f28 ffffffff81d38c8b 17c9660328d0b468 (XEN) 0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f38 ffffffff81d38339 ffffffff81c03ff8 (XEN) ffffffff81d3b96c 0000000000000000 00600f1200000000 0000000100000800 (XEN) 0300000100000032 0000000000000005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
On Wed, Sep 16, 2015 at 05:04:31PM -0400, Cole Robinson wrote:
On 09/16/2015 04:07 PM, M A Young wrote:
On Wed, 16 Sep 2015, Cole Robinson wrote:
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
Xen 4.5.1-5.fc22 (XEN) Xen version 4.5.1 (mockbuild@[unknown]) (gcc (GCC) 5.1.1 20150618 (Red Hat 5.1.1-4)) debug=n Sun Aug 2 16:23:51 UTC 2015 (XEN) Latest ChangeSet: (XEN) Bootloader: GRUB 2.02~beta2 (XEN) Command line: placeholder console=com1 (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009fc00 (usable) (XEN) 000000000009fc00 - 00000000000a0000 (reserved) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000007ffe0000 (usable) (XEN) 000000007ffe0000 - 0000000080000000 (reserved) (XEN) 00000000feffc000 - 00000000ff000000 (reserved) (XEN) 00000000fffc0000 - 0000000100000000 (reserved) (XEN) System RAM: 2047MB (2096636kB) (XEN) ACPI: RSDP 000F6300, 0014 (r0 BOCHS ) (XEN) ACPI: RSDT 7FFE16EE, 0034 (r1 BOCHS BXPCRSDT 1 BXPC 1) (XEN) ACPI: FACP 7FFE0C14, 0074 (r1 BOCHS BXPCFACP 1 BXPC 1) (XEN) ACPI: DSDT 7FFE0040, 0BD4 (r1 BOCHS BXPCDSDT 1 BXPC 1) (XEN) ACPI: FACS 7FFE0000, 0040 (XEN) ACPI: SSDT 7FFE0C88, 09B6 (r1 BOCHS BXPCSSDT 1 BXPC 1) (XEN) ACPI: APIC 7FFE163E, 0078 (r1 BOCHS BXPCAPIC 1 BXPC 1) (XEN) ACPI: HPET 7FFE16B6, 0038 (r1 BOCHS BXPCHPET 1 BXPC 1) (XEN) Domain heap initialised (XEN) Processor #0 5:1 APIC version 20 (XEN) IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) Not enabling x2APIC: depends on iommu_supports_eim. (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: Initializing. (XEN) AVC INITIALIZED (XEN) Flask: Starting in permissive mode. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 3624.462 MHz processor. (XEN) Initing memory sharing. (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7 (XEN) AMD-Vi: IOMMU not found! (XEN) I/O virtualisation disabled (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) Platform timer is 100.000MHz HPET (XEN) Allocated console ring of 16 KiB. (XEN) HVM: ASIDs enabled. (XEN) SVM: Supported advanced features: (XEN) - Nested Page Tables (NPT) (XEN) - Next-RIP Saved on #VMEXIT (XEN) HVM: SVM enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) Brought up 1 CPUs (XEN) Dom0 has maximum 208 PIRQs (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2044000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000074000000->0000000078000000 (465487 pages to be allocated) (XEN) Init. ramdisk: 000000007ec73000->000000007f9ffb4e (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff82044000 (XEN) Init. ramdisk: 0000000000000000->0000000000000000 (XEN) Phys-Mach map: ffffffff82044000->ffffffff823f7ee0 (XEN) Start info: ffffffff823f8000->ffffffff823f84b4 (XEN) Page tables: ffffffff823f9000->ffffffff82410000 (XEN) Boot stack: ffffffff82410000->ffffffff82411000 (XEN) TOTAL: ffffffff80000000->ffffffff82800000 (XEN) ENTRY ADDRESS: ffffffff81d381f0 (XEN) Dom0 has maximum 1 VCPUs (XEN) Scrubbing Free RAM on 1 nodes using 1 CPUs (XEN) ................done. (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Std. Loglevel: Errors and warnings (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 308kB init memory. mapping kernel into physical memory about to get started... (XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff810032b0>]
That is the Linux kernel EIP. Can you figure out what is at ffffffff810032b0 ?
gdb vmlinux and then x/20i 0xffffffff810032b0
can help with that.
(XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest (XEN) rax: 0000000000000015 rbx: ffffffff81c03e1c rcx: 00000000c0010112 (XEN) rdx: 0000000000000001 rsi: ffffffff81c03e1c rdi: 00000000c0010112 (XEN) rbp: ffffffff81c03df8 rsp: ffffffff81c03da0 r8: ffffffff81c03e28 (XEN) r9: ffffffff81c03e2c r10: 0000000000000000 r11: 00000000ffffffff (XEN) r12: ffffffff81d0ec20 r13: 0000000004000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000000406f0 (XEN) cr3: 0000000075c0b000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff81c03da0: (XEN) 00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810032b0 (XEN) 000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b (XEN) 0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48 (XEN) ffffffff81029a14 ffffffff81c03e48 ffffffff81029c9b ffffffff81c03e48 (XEN) 17c9660328d0b468 0000000001000000 ffffffff81c03ef8 0000000004000000 (XEN) 0000000000000000 ffffffff81c03e58 ffffffff81d44196 ffffffff81c03ee8 (XEN) ffffffff81d3fb6d 0000000000000000 0000000000000000 ffffffff81c03e88 (XEN) ffffffff810f2a99 ffffffff81c03ee8 ffffffff81781688 ffffffff00000010 (XEN) ffffffff81c03ef8 ffffffff81c03eb8 17c9660328d0b468 ffffffff81f02d5c (XEN) ffffffffffffffff 0000000000000000 ffffffff81ddd920 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f28 ffffffff81d38c8b 17c9660328d0b468 (XEN) 0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f38 ffffffff81d38339 ffffffff81c03ff8 (XEN) ffffffff81d3b96c 0000000000000000 00600f1200000000 0000000100000800 (XEN) 0300000100000032 0000000000000005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc (XEN) Domain 0 crashed: rebooting machine in 5 seconds. -- xen mailing list xen@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/xen
On 09/16/2015 05:08 PM, Konrad Rzeszutek Wilk wrote:
On Wed, Sep 16, 2015 at 05:04:31PM -0400, Cole Robinson wrote:
On 09/16/2015 04:07 PM, M A Young wrote:
On Wed, 16 Sep 2015, Cole Robinson wrote:
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
<snip>
about to get started... (XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff810032b0>]
That is the Linux kernel EIP. Can you figure out what is at ffffffff810032b0 ?
gdb vmlinux and then x/20i 0xffffffff810032b0
can help with that.
Updated to the latest kernel 4.1.6-201.fc22.x86_64. Trace is now:
about to get started... (XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff810031f0>] (XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest (XEN) rax: 0000000000000015 rbx: ffffffff81c03e1c rcx: 00000000c0010112 (XEN) rdx: 0000000000000001 rsi: ffffffff81c03e1c rdi: 00000000c0010112 (XEN) rbp: ffffffff81c03df8 rsp: ffffffff81c03da0 r8: ffffffff81c03e28 (XEN) r9: ffffffff81c03e2c r10: 0000000000000000 r11: 00000000ffffffff (XEN) r12: ffffffff81d25a60 r13: 0000000004000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000000406f0 (XEN) cr3: 0000000075c0b000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff81c03da0: (XEN) 00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810031f0 (XEN) 000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b (XEN) 0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48 (XEN) ffffffff8102a7a4 ffffffff81c03e48 ffffffff8102aa3b ffffffff81c03e48 (XEN) cf1fa5f5e026f464 0000000001000000 ffffffff81c03ef8 0000000004000000 (XEN) 0000000000000000 ffffffff81c03e58 ffffffff81d5d142 ffffffff81c03ee8 (XEN) ffffffff81d58b56 0000000000000000 0000000000000000 ffffffff81c03e88 (XEN) ffffffff810f8a39 ffffffff81c03ee8 ffffffff81798b13 ffffffff00000010 (XEN) ffffffff81c03ef8 ffffffff81c03eb8 cf1fa5f5e026f464 ffffffff81f1de9c (XEN) ffffffffffffffff 0000000000000000 ffffffff81df7920 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f28 ffffffff81d51c74 cf1fa5f5e026f464 (XEN) 0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f38 ffffffff81d51339 ffffffff81c03ff8 (XEN) ffffffff81d548b1 0000000000000000 00600f1200000000 0000000100000800 (XEN) 0300000100000032 0000000000000005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
gdb output:
(gdb) x/20i 0xffffffff810031f0 0xffffffff810031f0 <xen_read_msr_safe+16>: rdmsr 0xffffffff810031f2 <xen_read_msr_safe+18>: xor %ecx,%ecx 0xffffffff810031f4 <xen_read_msr_safe+20>: shl $0x20,%rdx 0xffffffff810031f8 <xen_read_msr_safe+24>: mov %eax,%eax 0xffffffff810031fa <xen_read_msr_safe+26>: mov %ecx,(%rsi) 0xffffffff810031fc <xen_read_msr_safe+28>: mov %rdx,%rbx 0xffffffff810031ff <xen_read_msr_safe+31>: or %rax,%rbx 0xffffffff81003202 <xen_read_msr_safe+34>: cmp $0x1b,%edi 0xffffffff81003205 <xen_read_msr_safe+37>: jne 0xffffffff8100323a <xen_read_msr_safe+90> 0xffffffff81003207 <xen_read_msr_safe+39>: movl $0x1,-0x18(%rbp) 0xffffffff8100320e <xen_read_msr_safe+46>: movl $0x0,-0x10(%rbp) 0xffffffff81003215 <xen_read_msr_safe+53>: lea -0x18(%rbp),%rdi 0xffffffff81003219 <xen_read_msr_safe+57>: lea -0x14(%rbp),%rsi 0xffffffff8100321d <xen_read_msr_safe+61>: lea -0x10(%rbp),%rdx 0xffffffff81003221 <xen_read_msr_safe+65>: lea -0xc(%rbp),%rcx 0xffffffff81003225 <xen_read_msr_safe+69>: callq *0xffffffff81c28618 0xffffffff8100322c <xen_read_msr_safe+76>: mov %rbx,%rax 0xffffffff8100322f <xen_read_msr_safe+79>: and $0xfb,%ah 0xffffffff81003232 <xen_read_msr_safe+82>: testb $0x20,-0xe(%rbp) 0xffffffff81003236 <xen_read_msr_safe+86>: cmove %rax,%rbx
On Wed, Sep 16, 2015 at 06:39:03PM -0400, Cole Robinson wrote:
On 09/16/2015 05:08 PM, Konrad Rzeszutek Wilk wrote:
On Wed, Sep 16, 2015 at 05:04:31PM -0400, Cole Robinson wrote:
On 09/16/2015 04:07 PM, M A Young wrote:
On Wed, 16 Sep 2015, Cole Robinson wrote:
Unfortunately I couldn't get anything else extra out of xen using any of these options or the ones Major recommended... in fact I couldn't get anything to the serial console at all. console=con1 would seem to redirect messages since they wouldn't show up on the graphical display, but nothing went to the serial log. Maybe I'm missing something...
That should be console=com1 so you have a typo either in this message or in your tests.
Yeah that was it :/ So here's the crash output use -cpu host:
- Cole
<snip>
about to get started... (XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff810032b0>]
That is the Linux kernel EIP. Can you figure out what is at ffffffff810032b0 ?
gdb vmlinux and then x/20i 0xffffffff810032b0
can help with that.
Updated to the latest kernel 4.1.6-201.fc22.x86_64. Trace is now:
about to get started... (XEN) traps.c:459:d0v0 Unhandled general protection fault fault/trap [#13] on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d08023a5d3 create_bounce_frame+0x12b/0x13a (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.5.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff810031f0>] (XEN) RFLAGS: 0000000000000282 EM: 1 CONTEXT: pv guest (XEN) rax: 0000000000000015 rbx: ffffffff81c03e1c rcx: 00000000c0010112 (XEN) rdx: 0000000000000001 rsi: ffffffff81c03e1c rdi: 00000000c0010112 (XEN) rbp: ffffffff81c03df8 rsp: ffffffff81c03da0 r8: ffffffff81c03e28 (XEN) r9: ffffffff81c03e2c r10: 0000000000000000 r11: 00000000ffffffff (XEN) r12: ffffffff81d25a60 r13: 0000000004000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000000406f0 (XEN) cr3: 0000000075c0b000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff81c03da0: (XEN) 00000000c0010112 00000000ffffffff 0000000000000000 ffffffff810031f0 (XEN) 000000010000e030 0000000000010082 ffffffff81c03de0 000000000000e02b (XEN) 0000000000000000 000000000000000c ffffffff81c03e1c ffffffff81c03e48 (XEN) ffffffff8102a7a4 ffffffff81c03e48 ffffffff8102aa3b ffffffff81c03e48 (XEN) cf1fa5f5e026f464 0000000001000000 ffffffff81c03ef8 0000000004000000 (XEN) 0000000000000000 ffffffff81c03e58 ffffffff81d5d142 ffffffff81c03ee8 (XEN) ffffffff81d58b56 0000000000000000 0000000000000000 ffffffff81c03e88 (XEN) ffffffff810f8a39 ffffffff81c03ee8 ffffffff81798b13 ffffffff00000010 (XEN) ffffffff81c03ef8 ffffffff81c03eb8 cf1fa5f5e026f464 ffffffff81f1de9c (XEN) ffffffffffffffff 0000000000000000 ffffffff81df7920 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f28 ffffffff81d51c74 cf1fa5f5e026f464 (XEN) 0000000000000000 ffffffff81c03f60 ffffffff81c03f5c 0000000000000000 (XEN) 0000000000000000 ffffffff81c03f38 ffffffff81d51339 ffffffff81c03ff8 (XEN) ffffffff81d548b1 0000000000000000 00600f1200000000 0000000100000800 (XEN) 0300000100000032 0000000000000005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
gdb output:
(gdb) x/20i 0xffffffff810031f0 0xffffffff810031f0 <xen_read_msr_safe+16>: rdmsr
Fantastic! So we have some rdmsr that makes KVM inject an GP.
Looking at the stack you have some other values: ffffffff81c03de0, ffffffff81c03e1c .. they should correspond to other functions calling this one. If you do 'nm --defined vmlinux | grep ffffffff81c03e1' that should give an idea where they are. Or use 'gdb'.
That will give us an stack - and we can find what type of MSR this is. Oh wait, it is on the registers: 00000000c0010112
Ok, so where in the code is that MSR ah, that looks to be: #define MSR_K8_TSEG_ADDR 0xc0010112
which is called at bsp_init_amd.
I think the problem here is that we are calling the 'safe' variant of MSR but we still get an injected #GP and don't expect that.
I am not really sure what the expected outcome should be here.
CC-ing xen-devel, KVM folks, and Andy who has been looking in mucking around in the _safe* pvops.
0xffffffff810031f2 <xen_read_msr_safe+18>: xor %ecx,%ecx 0xffffffff810031f4 <xen_read_msr_safe+20>: shl $0x20,%rdx 0xffffffff810031f8 <xen_read_msr_safe+24>: mov %eax,%eax 0xffffffff810031fa <xen_read_msr_safe+26>: mov %ecx,(%rsi) 0xffffffff810031fc <xen_read_msr_safe+28>: mov %rdx,%rbx 0xffffffff810031ff <xen_read_msr_safe+31>: or %rax,%rbx 0xffffffff81003202 <xen_read_msr_safe+34>: cmp $0x1b,%edi 0xffffffff81003205 <xen_read_msr_safe+37>: jne 0xffffffff8100323a <xen_read_msr_safe+90> 0xffffffff81003207 <xen_read_msr_safe+39>: movl $0x1,-0x18(%rbp) 0xffffffff8100320e <xen_read_msr_safe+46>: movl $0x0,-0x10(%rbp) 0xffffffff81003215 <xen_read_msr_safe+53>: lea -0x18(%rbp),%rdi 0xffffffff81003219 <xen_read_msr_safe+57>: lea -0x14(%rbp),%rsi 0xffffffff8100321d <xen_read_msr_safe+61>: lea -0x10(%rbp),%rdx 0xffffffff81003221 <xen_read_msr_safe+65>: lea -0xc(%rbp),%rcx 0xffffffff81003225 <xen_read_msr_safe+69>: callq *0xffffffff81c28618 0xffffffff8100322c <xen_read_msr_safe+76>: mov %rbx,%rax 0xffffffff8100322f <xen_read_msr_safe+79>: and $0xfb,%ah 0xffffffff81003232 <xen_read_msr_safe+82>: testb $0x20,-0xe(%rbp) 0xffffffff81003236 <xen_read_msr_safe+86>: cmove %rax,%rbx
-
Cole Robinson -
Konrad Rzeszutek Wilk -
M A Young -
Major Hayden