Hello,
Using Boxes, I installed a Windows 10 guest. I installed the spice drivers from spice-space.org/download (https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-t...) and things look good.
However, when the guest is idle, cpu is 0-4% the host process is significantly higher (28.7% for example below). I'm looking for advice on how to figure out what the problem is. The only thing I did so far is switch the disk bus from ide to virtio, and that didn't help.
I have an XPS 13 with plenty of ram and SSD disk space - i5 processor, running Fedora 28, 4.17.7-200.fc28.x86_64
Here's some output from top:
top - 07:13:02 up 12:49, 1 user, load average: 1.10, 1.84, 1.76 Tasks: 272 total, 1 running, 204 sleeping, 0 stopped, 0 zombie %Cpu0 : 4.7 us, 2.7 sy, 0.0 ni, 91.6 id, 0.0 wa, 0.7 hi, 0.3 si, 0.0 st %Cpu1 : 3.0 us, 4.4 sy, 0.0 ni, 92.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 4.3 us, 4.6 sy, 0.0 ni, 90.4 id, 0.0 wa, 0.3 hi, 0.3 si, 0.0 st %Cpu3 : 3.3 us, 5.0 sy, 0.0 ni, 91.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16144924 total, 272940 free, 5217992 used, 10653992 buff/cache KiB Swap: 8224764 total, 8087004 free, 137760 used. 10409752 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27942 ericp 20 0 3886324 2.130g 26340 S 28.7 13.8 27:26.31 qemu-system-x86
Reading other items online I used perf to see what is going on:
$ sudo perf kvm --host stat live
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max Time Avg time
IO_INSTRUCTION 493 56.93% 67.91% 2.91us 99913.26us 4255.31us ( +- 17.70% ) APIC_ACCESS 237 27.37% 7.71% 0.56us 99775.52us 1004.83us ( +- 51.23% ) HLT 106 12.24% 23.89% 0.59us 99841.70us 6961.72us ( +- 22.82% ) EXTERNAL_INTERRUPT 19 2.19% 0.00% 1.52us 5.56us 2.62us ( +- 9.42% ) EPT_MISCONFIG 6 0.69% 0.49% 28.66us 14837.17us 2506.17us ( +- 98.41% ) PENDING_INTERRUPT 3 0.35% 0.00% 0.74us 4.78us 2.91us ( +- 40.38% ) TPR_BELOW_THRESHOLD 2 0.23% 0.00% 4.21us 4.28us 4.24us ( +- 0.84% )
Total Samples:866, Total events handled time:3089057.95us.
Lots of IO and APIC. I was hoping the virtio bus change would take care of the high IO_INSTRUCTIONs, but it didn't. I changed the windows powercfg to be "high performance," but that didn't reduce the APIC calls.
Here's another perf:
$ sudo perf stat -e 'kvm:*' -a -- sleep 1 Performance counter stats for 'system wide': 8,750 kvm:kvm_fpu 6,823 kvm:kvm_exit 6,823 kvm:kvm_entry 4,375 kvm:kvm_userspace_exit 4,340 kvm:kvm_pio 1,576 kvm:kvm_apic 1,386 kvm:kvm_set_irq 1,386 kvm:kvm_pic_set_irq 1,386 kvm:kvm_ioapic_set_irq 920 kvm:kvm_ple_window 866 kvm:kvm_mmio 847 kvm:kvm_emulate_insn 780 kvm:kvm_inj_virq 780 kvm:kvm_eoi 780 kvm:kvm_apic_accept_irq 715 kvm:kvm_vcpu_wakeup 715 kvm:kvm_hv_timer_state 317 kvm:kvm_apic_ipi 177 kvm:kvm_halt_poll_ns 148 kvm:kvm_ack_irq 35 kvm:vcpu_match_mmio
Lots of fpu ?!?!? Lots of userspace exit. Lots of userspace, and looking at the top, the processors are doing an equal amount of usr and sys. I figure something is constantly doing system calls, causing lots of context switching.
I did "ps -Alf" to get the command line it creates. Hopefully this helps. I can provide the boxes config file if that's easier.
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/home/ericp/.config/libvirt/qemu/lib/domain-1-win10/master-key.aes -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell-noTSX-IBRS -m 2048 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -uuid c168275d-7cb1-4e98-8c66-6f51c3d0beb6 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/ericp/.config/libvirt/qemu/lib/domain-1-win10/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -device usb-ccid,id=ccid0,bus=usb.0,port=2 -device usb-hub,id=hub0,bus=usb.0,port=1 -drive file=/home/ericp/.local/share/gnome-boxes/images/win10,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7f:55:17,bus=pci.0,addr=0x3 -chardev spicevmc,id=charsmartcard0,name=smartcard -device ccid-card-passthru,chardev=charsmartcard0,id=smartcard0,bus=ccid0.0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev spiceport,id=charchannel1,name=org.spice-space.webdav.0 -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.spice-space.webdav.0 -device usb-tablet,id=input0,bus=usb.0,port=3 -device usb-mouse,id=input1,bus=usb.0,port=4 -device usb-kbd,id=input2,bus=usb.0,port=5 -spice port=0,disable-ticketing,image-compression=off,seamless-migration=on -device virtio-vga,id=video0,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=6 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=1.1 -chardev spicevmc,id=charredir2,name=usbredir -device usb-redir,chardev=charredir2,id=redir2,bus=usb.0,port=1.2 -chardev spicevmc,id=charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3,bus=usb.0,port=1.3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Any suggestions on what to do?
I've learned two more things: 1. I installed an ubuntu guest OS, and it has the same high host cpu when the guest is idle. 2. using strace, I see this is 97% of the system calls: ppoll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=13, events=POLLIN}, {fd=25, events=POLLIN}, {fd=26, events=POLLIN}, {fd=27, events=POLLIN}, {fd=30, events=POLLIN}, {fd=31, events=POLLIN}, {fd=32, events=POLLIN}, {fd=33, events=POLLIN}, {fd=34, events=POLLIN}, {fd=35, events=POLLIN}, {fd=36, events=POLLIN}, {fd=37, events=POLLIN}, {fd=38, events=POLLIN}, {fd=39, events=POLLIN}, {fd=40, events=POLLIN}, {fd=41, events=POLLIN}, {fd=42, events=POLLIN}, {fd=43, events=POLLIN}, {fd=44, events=POLLIN}, {fd=45, events=POLLIN}, {fd=46, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=49, events=POLLIN}, {fd=50, events=POLLIN}, {fd=51, events=POLLIN}, {fd=52, events=POLLIN}, {fd=53, events=POLLIN}, ...], 84, {tv_sec=0, tv_nsec=0}, NULL, 8) = 0 (Timeout)
making progress. If I start the VM using Virtual Machine Manager (virt-manager) the cpu is noticeably lower. ppoll seems to be lower, too. Comparing the outputs from ps -lf I see some differences I can test...
Solved
I removed <smartcard mode='passthrough' type='spicevmc'> <address type='ccid' controller='0' slot='0'/> </smartcard>
and the cpu usage dropped to match the guest.
FWIW, to track this down, I created a Fedora server using virt-manager and another with Boxes and reproduced the issue. I compared their ps -lf outputs and tried removing the extra components, and removing smartcard made the difference.
On 08/01/2018 08:56 AM, Eric Phetteplace wrote:
Solved
I removed
<smartcard mode='passthrough' type='spicevmc'> <address type='ccid' controller='0' slot='0'/> </smartcard>
and the cpu usage dropped to match the guest.
FWIW, to track this down, I created a Fedora server using virt-manager and another with Boxes and reproduced the issue. I compared their ps -lf outputs and tried removing the extra components, and removing smartcard made the difference.
Thanks for tracking this down Eric. Can you file a bug against Product=Fedora component=qemu in bugzilla.redhat.com so we don't lose these details?
Thanks, Cole
Will do, tonight.
Eric ________________________________ From: Cole Robinson crobinso@redhat.com Sent: Tuesday, August 7, 2018 2:11:05 PM To: Community support for Fedora users; Eric Phetteplace Subject: Re: gnome-boxes, high cpu when guest is idle
On 08/01/2018 08:56 AM, Eric Phetteplace wrote:
Solved
I removed
<smartcard mode='passthrough' type='spicevmc'> <address type='ccid' controller='0' slot='0'/> </smartcard>
and the cpu usage dropped to match the guest.
FWIW, to track this down, I created a Fedora server using virt-manager and another with Boxes and reproduced the issue. I compared their ps -lf outputs and tried removing the extra components, and removing smartcard made the difference.
Thanks for tracking this down Eric. Can you file a bug against Product=Fedora component=qemu in bugzilla.redhat.com so we don't lose these details?
Thanks, Cole
https://bugzilla.redhat.com/show_bug.cgi?id=1613990
Eric ________________________________ From: Cole Robinson crobinso@redhat.com Sent: Tuesday, August 7, 2018 2:11:05 PM To: Community support for Fedora users; Eric Phetteplace Subject: Re: gnome-boxes, high cpu when guest is idle
On 08/01/2018 08:56 AM, Eric Phetteplace wrote:
Solved
I removed
<smartcard mode='passthrough' type='spicevmc'> <address type='ccid' controller='0' slot='0'/> </smartcard>
and the cpu usage dropped to match the guest.
FWIW, to track this down, I created a Fedora server using virt-manager and another with Boxes and reproduced the issue. I compared their ps -lf outputs and tried removing the extra components, and removing smartcard made the difference.
Thanks for tracking this down Eric. Can you file a bug against Product=Fedora component=qemu in bugzilla.redhat.com so we don't lose these details?
Thanks, Cole
-
Cole Robinson -
Eric Phetteplace