I'm having a decent amount of trouble getting pciback to behave the way I'd like it to.
Originally I found xen-pciback was compiled as a module and not directly into the kernel, ruling out use of the boot-time argument xen-pciback.hide=(blah) ... so I recompiled my kernel, incluging pciback directly into the kernel.
I added my pciback.hide arguments to the GRUB_CMDLINE_XEN in /etc/default/grub and ran a grub-mkconfig updating /boot/grub2/grub.cfg
The devices were not excluded from Dom0.
So I tried adding the same arguments to the regular GRUB_CMDLINE and the devices are still not hidden.
Can someone point me in the right direction? Did I mess up something? System is FC17 on an x86_64
here's a snippet of my kernel config ... I just noticed that PCIDEV_FRONTEND=m ... maybe that needs to be set to Y along with PCIDEV_BACKEND ?
[root@xenhost linux-3.6.1-1.glmxen1.fc17.x86_64]# grep XEN .config CONFIG_XEN=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=500 CONFIG_XEN_SAVE_RESTORE=y CONFIG_XEN_DEBUG_FS=y CONFIG_PCI_XEN=y CONFIG_XEN_PCIDEV_FRONTEND=m CONFIG_XEN_BLKDEV_FRONTEND=m CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_NETXEN_NIC=m CONFIG_XEN_NETDEV_FRONTEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y CONFIG_HVC_XEN=y CONFIG_HVC_XEN_FRONTEND=y CONFIG_XEN_WDT=m CONFIG_XEN_FBDEV_FRONTEND=y CONFIG_XEN_BALLOON=y CONFIG_XEN_SELFBALLOONING=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=m CONFIG_XEN_BACKEND=y CONFIG_XENFS=m CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=m CONFIG_XEN_GRANT_DEV_ALLOC=m CONFIG_SWIOTLB_XEN=y CONFIG_XEN_TMEM=y CONFIG_XEN_PCIDEV_BACKEND=y CONFIG_XEN_PRIVCMD=m CONFIG_XEN_ACPI_PROCESSOR=m
here's the grub default [root@xenhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Fedora" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8 pciback.hide=(06:00.0)(02:00.0)(02:00.1)(07:00.0)(08:00.0)(00:1a.0)" GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M loglvl=all guest_loglvl=all pciback.hide=(06:00.0)(02:00.0)(02:00.1)(07:00.0)(08:00.0)(00:1a.0)" #GRUB_THEME="/boot/grub2/themes/system/theme.txt"
On Sat, 13 Oct 2012, Gordon McLellan wrote:
I'm having a decent amount of trouble getting pciback to behave the way I'd like it to. Originally I found xen-pciback was compiled as a module and not directly into the kernel, ruling out use of the boot-time argument xen-pciback.hide=(blah) ... so I recompiled my kernel, incluging pciback directly into the kernel.
Could you pass the option to the module by creating a file in /etc/modprobe.d/ to supply the options to the module (the modprobe.conf explains the format)? I think those files get copied over to the initramfs when you install a kernel so they should still be used even if the module is loaded early in the boot process.
Incidentally, you seem to be using xen-pciback in one place and pciback in another. With a recent Fedora based kernel I would expect you would need to use xen-pciback throughout.
Michael Young
On Sat, Oct 13, 2012 at 6:12 PM, M A Young m.a.young@durham.ac.uk wrote:
On Sat, 13 Oct 2012, Gordon McLellan wrote:
I'm having a decent amount of trouble getting pciback to behave the way
I'd like it to. Originally I found xen-pciback was compiled as a module and not directly into the kernel, ruling out use of the boot-time argument xen-pciback.hide=(blah) ... so I recompiled my kernel, incluging pciback directly into the kernel.
Could you pass the option to the module by creating a file in /etc/modprobe.d/ to supply the options to the module (the modprobe.conf explains the format)? I think those files get copied over to the initramfs when you install a kernel so they should still be used even if the module is loaded early in the boot process.
Incidentally, you seem to be using xen-pciback in one place and pciback in another. With a recent Fedora based kernel I would expect you would need to use xen-pciback throughout.
Michael Young
Michael,
Thank you for the tips. I made sure I specified xen-pciback instead of the older pciback ... not really sure which line in the configuration it goes on in /etc/default/grub - leaving it on the XEN line for now. I created /etc/modprobe.d/xen-pciback.conf to also reflect the same:
options xen-pciback hide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1)(0000:00:0b.0)'
after making these changes, I updated the grub config file and rebooted. After reboot, none of the devices are hidden from dom0. I can forcibly unbind them from dom0 with a script, but that seems messy and some of the pci devices don't seem to like it (sata controller for example)
Any suggestions?
Gordon
On Sun, 14 Oct 2012, Gordon McLellan wrote:
Thank you for the tips. I made sure I specified xen-pciback instead of the older pciback ... not really sure which line in the configuration it goes on in /etc/default/grub - leaving it on the XEN line for now. I created /etc/modprobe.d/xen-pciback.conf to also reflect the same:
options xen-pcibackhide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1 )(0000:00:0b.0)'
after making these changes, I updated the grub config file and rebooted. After reboot, none of the devices are hidden from dom0. I can forcibly unbind them from dom0 with a script, but that seems messy and some of the pci devices don't seem to like it (sata controller for example)
Any suggestions?
Could you try without the leading 0000: in one or both places. I have been looking at http://wiki.xen.org/wiki/Xen_PCI_Passthrough and the examples there don't have the leading 0000: .
Michael Young
On Sun, Oct 14, 2012 at 4:08 PM, M A Young m.a.young@durham.ac.uk wrote:
On Sun, 14 Oct 2012, Gordon McLellan wrote:
Thank you for the tips. I made sure I specified xen-pciback instead of the older pciback ... not really sure which line in the configuration it goes on in /etc/default/grub - leaving it on the XEN line for now. I created /etc/modprobe.d/xen-pciback.conf to also reflect the same:
options xen-pcibackhide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1
)(0000:00:0b.0)'
after making these changes, I updated the grub config file and rebooted. After reboot, none of the devices are hidden from dom0. I can forcibly unbind them from dom0 with a script, but that seems messy and some of the pci devices don't seem to like it (sata controller for example)
Any suggestions?
Could you try without the leading 0000: in one or both places. I have been looking at http://wiki.xen.org/wiki/Xen_PCI_Passthrough and the examples there don't have the leading 0000: .
Michael Young
Looks like I have some problem with grub passing the arguments correctly. I see them properly when I view the "dmesg" output, but when modprobe tries to load the xen-pciback module, it claims invalid argument 0) ... like the rest of the line has been lost or something. I tried single and doubel quotes, and with / without the extra 0's. I've gone back to dynamic unbinding via script and I'll wait for 4.2 to come along :)
Thanks a bunch for the help.
Gordon
On Sun, Oct 14, 2012 at 10:22:43PM -0400, Gordon McLellan wrote:
On Sun, Oct 14, 2012 at 4:08 PM, M A Young m.a.young@durham.ac.uk wrote:
On Sun, 14 Oct 2012, Gordon McLellan wrote:
Thank you for the tips. I made sure I specified xen-pciback instead of the older pciback ... not really sure which line in the configuration it goes on in /etc/default/grub - leaving it on the XEN line for now. I created /etc/modprobe.d/xen-pciback.conf to also reflect the same:
options xen-pcibackhide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1
)(0000:00:0b.0)'
after making these changes, I updated the grub config file and rebooted. After reboot, none of the devices are hidden from dom0. I can forcibly unbind them from dom0 with a script, but that seems messy and some of the pci devices don't seem to like it (sata controller for example)
Any suggestions?
Could you try without the leading 0000: in one or both places. I have been looking at http://wiki.xen.org/wiki/Xen_PCI_Passthrough and the examples there don't have the leading 0000: .
Michael YoungLooks like I have some problem with grub passing the arguments correctly. I see them properly when I view the "dmesg" output, but when modprobe tries to load the xen-pciback module, it claims invalid argument 0) ... like the rest of the line has been lost or something. I tried single and doubel quotes, and with / without the extra 0's. I've gone back to dynamic unbinding via script and I'll wait for 4.2 to come along :)
Huh? Why? This is what I've in my /etc/defalt/grub
[root@phenom konrad]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Xen 4.1" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8 SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0 xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)" GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
And it works quite well for me and I am using the stock Fedora kernel.
On Mon, Oct 15, 2012 at 9:03 AM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Huh? Why? This is what I've in my /etc/defalt/grub
[root@phenom konrad]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Xen 4.1" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8 SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0 xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)" GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
And it works quite well for me and I am using the stock Fedora kernel.
Konrad,
I'm not sure... it is probably something I have done wrong, and probably something simple so I am overlooking it. I have gone back to stock FC17 kernel and slimmed down my hide list to see if that helps: [root@xenhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Fedora" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8 xen-pciback.hide=(0b:00.0)(09:00.0)" GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=2 dom0_vcpus_pin loglvl=all guest_loglvl=all"
however the xen-pciback module is not loading at boot, and when I try to load it manually: [root@xenhost ~]# modprobe -vv xen-pciback insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko 0) libkmod: kmod_module_insert_module: Failed to insert module '/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko': No such file or directory ERROR: could not insert 'xen_pciback': No such file or directory
It is claiming file not found, but if I run the command on the first line (insmod) it claims a different error, it does not like the "0)" [root@xenhost ~]# insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko 0) -bash: syntax error near unexpected token `)'
On Mon, Oct 15, 2012 at 09:50:31AM -0400, Gordon McLellan wrote:
On Mon, Oct 15, 2012 at 9:03 AM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Huh? Why? This is what I've in my /etc/defalt/grub
[root@phenom konrad]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Xen 4.1" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8 SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0 xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)" GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
And it works quite well for me and I am using the stock Fedora kernel.
Konrad,
I'm not sure... it is probably something I have done wrong, and probably something simple so I am overlooking it. I have gone back to stock FC17 kernel and slimmed down my hide list to see if that helps:
Hm, just realized that the machine is running F16, not F17!
[root@xenhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Fedora" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8 xen-pciback.hide=(0b:00.0)(09:00.0)" GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=2 dom0_vcpus_pin loglvl=all guest_loglvl=all"
however the xen-pciback module is not loading at boot, and when I try to load it manually: [root@xenhost ~]# modprobe -vv xen-pciback insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko 0) libkmod: kmod_module_insert_module: Failed to insert module '/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko': No such file or directory ERROR: could not insert 'xen_pciback': No such file or directory
That would imply it is simply gone. And does
cat /boot/config-3.6.1-1* | grep BACKEND
tell you whether the Xen PCI backend is compiled in (=y) or is a module (=m)?
It is claiming file not found, but if I run the command on the first line (insmod) it claims a different error, it does not like the "0)" [root@xenhost ~]# insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko 0) -bash: syntax error near unexpected token `)'
What is the "0)' from?
On Mon, Oct 15, 2012 at 9:51 AM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
On Mon, Oct 15, 2012 at 09:50:31AM -0400, Gordon McLellan wrote:
On Mon, Oct 15, 2012 at 9:03 AM, Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Huh? Why? This is what I've in my /etc/defalt/grub
[root@phenom konrad]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Xen 4.1" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8 SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0 xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)" GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
And it works quite well for me and I am using the stock Fedora kernel.
Konrad,
I'm not sure... it is probably something I have done wrong, and probably something simple so I am overlooking it. I have gone back to stock FC17 kernel and slimmed down my hide list to see if that helps:
Hm, just realized that the machine is running F16, not F17!
[root@xenhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Fedora" GRUB_DEFAULT=saved GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8 xen-pciback.hide=(0b:00.0)(09:00.0)" GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=2 dom0_vcpus_pin loglvl=all guest_loglvl=all"
however the xen-pciback module is not loading at boot, and when I try to load it manually: [root@xenhost ~]# modprobe -vv xen-pciback insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko 0) libkmod: kmod_module_insert_module: Failed to insert module '/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko': No such file or directory ERROR: could not insert 'xen_pciback': No such file or directory
That would imply it is simply gone. And does
cat /boot/config-3.6.1-1* | grep BACKEND
tell you whether the Xen PCI backend is compiled in (=y) or is a module (=m)?
It is claiming file not found, but if I run the command on the first line (insmod) it claims a different error, it does not like the "0)" [root@xenhost ~]# insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko 0) -bash: syntax error near unexpected token `)'
What is the "0)' from?
pciback is compiled as a module: [root@xenhost ~]# grep BACKEND /boot/config-3.6.1-1.fc17.x86_64 CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_XEN_BACKEND=y CONFIG_XEN_PCIDEV_BACKEND=m
and the file does exist: [root@xenhost ~]# ls -lah /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/ total 108K drwxr-xr-x 2 root root 4.0K Oct 12 10:19 . drwxr-xr-x 4 root root 4.0K Oct 12 10:19 .. -rwxr--r-- 1 root root 100K Oct 10 08:41 xen-pciback.ko
Near as I can tell the 0) is coming from the kernel command line but it is getting mangled somewhere along the way. If I remove the xen-pciback.hide directive from the grub config file, the kernel is able to load the pciback module without issue, and without any configuration. I tried moving the xen-pciback.hide directive to the front of the kernel command line, it did not help.
From dmesg:
[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.6.1-1.fc17.x86_64 (mockbuild@) (gcc version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) #1 SMP Wed Oct 10 12:13:05 UTC 2012 [ 0.000000] Command line: placeholder root=/dev/mapper/vg0-root ro xen-pciback.hide=(0b:00.0)(09:00.0) rd.md=0 rd.dm=0 rd.lvm.lv=vg0/swap KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
On Mon, 15 Oct 2012, Gordon McLellan wrote:
pciback is compiled as a module: [root@xenhost ~]# grep BACKEND /boot/config-3.6.1-1.fc17.x86_64 CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_XEN_BACKEND=y CONFIG_XEN_PCIDEV_BACKEND=m
and the file does exist: [root@xenhost ~]# ls -lah /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/ total 108K drwxr-xr-x 2 root root 4.0K Oct 12 10:19 . drwxr-xr-x 4 root root 4.0K Oct 12 10:19 .. -rwxr--r-- 1 root root 100K Oct 10 08:41 xen-pciback.ko
Near as I can tell the 0) is coming from the kernel command line but it is getting mangled somewhere along the way. If I remove the xen-pciback.hide directive from the grub config file, the kernel is able to load the pciback module without issue, and without any configuration. I tried moving the xen-pciback.hide directive to the front of the kernel command line, it did not help.
From dmesg: [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.6.1-1.fc17.x86_64 (mockbuild@) (gcc version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) #1 SMP Wed Oct 10 12:13:05 UTC 2012 [ 0.000000] Command line: placeholder root=/dev/mapper/vg0-root ro xen-pciback.hide=(0b:00.0)(09:00.0) rd.md=0 rd.dm=0 rd.lvm.lv=vg0/swap KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
If you haven't already done so I suggest you get rid of anything in /etc/modprobe.d/ you added, just in case that is getting in the way.
Michael Young
On Mon, Oct 15, 2012 at 10:20 AM, M A Young m.a.young@durham.ac.uk wrote:
If you haven't already done so I suggest you get rid of anything in /etc/modprobe.d/ you added, just in case that is getting in the way.
Michael Young
I double checked, nothing about xen-pciback in /etc/modprobe.d and /etc/modprobe.conf is empty
These files appear to be stock: [root@xenhost ~]# ls /etc/modprobe.d blacklist.conf openfwwf.conf udlfb.conf
I also tried booting the 3.5.6-1 fedora kernel, same difference. modprobe xen-pciback fails, this is logged to /var/log/messages Oct 15 13:05:59 xenhost kernel: [ 178.996253] xen_pciback: Unknown parameter `0)'
-
Gordon McLellan -
Konrad Rzeszutek Wilk -
M A Young