Solved
I removed <smartcard mode='passthrough' type='spicevmc'> <address type='ccid' controller='0' slot='0'/> </smartcard>
and the cpu usage dropped to match the guest.
FWIW, to track this down, I created a Fedora server using virt-manager and another with Boxes and reproduced the issue. I compared their ps -lf outputs and tried removing the extra components, and removing smartcard made the difference.