Hi everyone!
First of all sorry for such wide distribution, but apparently that's the best way to make sure we cooperate nicely. So please be considerate as this is a cross-post between huge amount of mailing lists.
After some discussions with developers from different projects that work with libvirt one cannot but notice some common patterns and workarounds. So I set off to see how can we make all our lives better and our coding more effective (and maybe more fun as well). If all goes well we will create a project that will accommodate most of the defaulting, policies, workarounds and other common algorithms around libvirt domain definitions. And since early design gets you half way, I would like to know your feedback on several key points as well as on the general idea. Also correct me brutally in case I'm wrong.
In order to not get confused in the following descriptions, I will refer to this project idea using the name `virtuned`, but there is really no name for it yet (although an abbreviation for "Virtualization Abstraction Definition and Hypervisor Delegation" would suit well, IMHO).
Here are some common problems and use cases that virtuned could solve (or help with). Don't take it as something that's impossible to solve on your own, but rather something that could be de-duplicated from multiple projects or "done right" instead of various hack-ish solutions.
1) Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
One more thing could be automatically figuring out best values based on libosinfo-provided data.
2) Policies
Lot of the time there are parts of the domain definition that need to be added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example with the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type, but you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some DSL for example), and used by virtuned to merge them in a unified and predictable way?
3) Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a device and that's it. We could do that as well. I can't really tell how much of a demand there is for it, though.
4) Identifying devices properly
In contrast to the previous point, stateful apps might have a problem identifying devices after hotplug. For example, let's say you don't care about the addresses and leave that up to libvirt. You hotplug a device into the domain and dump the new XML of it. Depending on what type of device it was, you might need to identify it based on different values. It could be <target dev=''/> for disks, <mac address=''/> for interfaces etc. For some devices it might not even be possible and you need to remember the addresses of all the previous devices and then parse them just to identify that one device and then throw them away.
With new enough libvirt you could use the user aliases for that, but turns out it's not that easy to use them properly anyway. Also the aliases won't help users identify that device inside the guest.
<rant> We really should've gone with new attribute for the user alias instead of using an existing one, given how many problems that is causing. </rant>
5) Generating the right XML snippet for device hot-(un)plug
This is kind of related to some previous points.
When hot-plugging a device and creating an XML snippet for it, you want to keep the defaults from point 1) and policies from 2) in mind. Or something related to the already existing domain which you can describe systematically. And adding something for identification (see previous point).
Doing the hot-unplug is easy depending on how much information about that device is saved by your application. The less you save about the device (or show to the user in a GUI, if applicable) the harder it might be to generate an XML that libvirt will accept. Again, some problems with this should be fixed in libvirt, some of them are easy to workaround. But having a common ground that takes care of this should help some projects.
Hot-unplug could be implemented just based on the alias. This is something that would fit into libvirt as well.
========================================================================
To mention some pre-existing solutions:
- I understand OpenStack has some really sensible and wisely chosen and/or tested default values.
- I know KubeVirt has VirtualMachinePresets. That is something closely related to points 1) and 2). Also their abstraction of the XML might be usable for point 3).
- There was an effort on creating policy based configuration of libvirt objects called libvirt-designer. This is closely related to points 2) and 3). Unfortunately there was no much going on lately and part of virt-manager repository has currently more features implemented with the same ideas in mind, just not exported for public use.
We could utilize some of the above to various extents.
Let me know what you think and have a nice day. Martin
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. So you need a matrix of libvirt + QEMU + config option settings.
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
One more thing could be automatically figuring out best values based on libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need to be added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example with the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type, but you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some DSL for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a device and that's it. We could do that as well. I can't really tell how much of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
The libvirt-gconfig project aims to provide a C API for manipulating XML documents, with language bindings available via GObject introspection so you can use it from Vala, Perl, Python, JavaScript, etc. Go is notable missing, but for that we have libvirt-go-xml which provides a set of native Go structs to represent the XML.
The problem we've faced with libvirt-gconfig is that it is a really hard sell to get applications to convert existing code to use it. We've only had success where an applicaiton has been written to use libvirt-gconfig from day one - eg GNOME Boxes and libvirt-sandbox.
Virt-manager is the poster-child for using libvirt-gconfig, but I don't see it adopting it any time soon, as it is a massive effort to change all existing code - even if libvirt-gconfig had full XML schema coverge, which it doesn't :-(
I also wanted to use libvirt-gconfig in OpenStack, but there was resistance for adding a dependancy on another native library, so there we've basically copied what virt-manager does and defined a set of pure python objects to represent config which is serialized to / from XML :-( If libvirt-gconfig were to be used by OpenStack it would need to deal with fact that not every distro has that available, so the existing pure python config objects would need to be maintained in parallel for an indefinite amount of time. So in fact using libvirt-gconfig would increase maint burden for OpenStack, rather than reduce it.
libvirt-gconfig would be a hard sell for Go apps when you compare it to libvirt-go-xml, because the latter is following the common Go paradigm for XML manipulation.
If there was something higher level that gets more interesting, but the hard bit is that you still need a way to get at all the low level bits becuase a higher level abstracted API will never cover every niche use case.
- Identifying devices properly
In contrast to the previous point, stateful apps might have a problem identifying devices after hotplug. For example, let's say you don't care about the addresses and leave that up to libvirt. You hotplug a device into the domain and dump the new XML of it. Depending on what type of device it was, you might need to identify it based on different values. It could be <target dev=''/> for disks, <mac address=''/> for interfaces etc. For some devices it might not even be possible and you need to remember the addresses of all the previous devices and then parse them just to identify that one device and then throw them away.
With new enough libvirt you could use the user aliases for that, but turns out it's not that easy to use them properly anyway. Also the aliases won't help users identify that device inside the guest.
NB, relating between host device config and guest visible device config is a massive problem space in its own right, and not very easy to address. In OpenStack we ended up defining a concept of "device tagging" via cloud-init metadata, where openstack allows users to set opaque string tags against devices their VM has. OpenStack that generates a metadata file that records various pieces of identifying hardware attributes (PCI address, MAC addr, disk serial, etc) alongside the user tag. This metadata file is exposed to the guest with the hope that there's enough info to allow the user to decide which device is to be used for which purpose
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-... https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/...
<rant> We really should've gone with new attribute for the user alias instead of using an existing one, given how many problems that is causing. </rant>
- Generating the right XML snippet for device hot-(un)plug
This is kind of related to some previous points.
When hot-plugging a device and creating an XML snippet for it, you want to keep the defaults from point 1) and policies from 2) in mind. Or something related to the already existing domain which you can describe systematically. And adding something for identification (see previous point).
Doing the hot-unplug is easy depending on how much information about that device is saved by your application. The less you save about the device (or show to the user in a GUI, if applicable) the harder it might be to generate an XML that libvirt will accept. Again, some problems with this should be fixed in libvirt, some of them are easy to workaround. But having a common ground that takes care of this should help some projects.
Hot-unplug could be implemented just based on the alias. This is something that would fit into libvirt as well.
========================================================================
To mention some pre-existing solutions:
- I understand OpenStack has some really sensible and wisely chosen
and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's decisions have been largely inspired by previous work in oVirt and / or virt-manager. So there's obviously overlap in the conceptual area, but there's also plenty that is very specific to OpenStack - untangling the two extract the common bits from the app specific bits is hard.
- I know KubeVirt has VirtualMachinePresets. That is something closely
related to points 1) and 2). Also their abstraction of the XML might be usable for point 3).
- There was an effort on creating policy based configuration of libvirt
objects called libvirt-designer. This is closely related to points 2) and 3). Unfortunately there was no much going on lately and part of virt-manager repository has currently more features implemented with the same ideas in mind, just not exported for public use.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the use of libosinfo native library via GObject Introspection, by promising to do work to turn the osinfo database into an approved stable format which OpenStack could then consume directly, dropping the native API usage :-( Incidentally, the former was done (formal spec for the DB format), but the latter was not yet (direct DB usage by OpenStack)
BTW, I don't like that I'm being so negative to your proposal :-( I used to hope that we would be able to build higher level APIs on top of libvirt to reduce the overlap between different applications reinventing the wheel. Even the simplest bits we tried like the gconfig/gobject API are barely used. libvirt-designer is basically a failure. Though admittedly it didn't have enough development resource applied to make it compelling, in retrospect adoption was always going to be a hard sell except in greenfield developments.
Libosinfo is probably the bit we've had most success with, and has most promise for the future, particularly now that we formally allow apps to read the osinfo database directly and bypass the API. It is quite easy to fit into existing application codebases which helps alot. Even there I'm still disappointed that we only have GNOME Boxes using the kickstart generator part of osinfo - oVirt and Oz both still have their own kickstart generator code for automating OS installs.
In general though, I fear anything API based is going to be a really hard sell to get wide adoption for based on what we've seen before.
I think the biggest bang-for-buck is identifying more areas where we can turn code into data. There's definitely scope for recording more types of information in the osinfo database. There might also be scope for defining entirely new databases to complement the osinfo data, if something looks out of scope for libosinfo.
Regards, Daniel
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. [...]
This is true.
[...] So you need a matrix of libvirt +QEMU + config option settings.
But this is not. If config options need support on the lower levels of the stack (libvirt and/or QEMU and/or KVM and/or host hardware), it already has to be represented by libvirt host capabilities somehow, so management layers know it's available.
This means any new config generation system can (and must) use host(s) capabilities as input before generating the configuration.
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
Why is that a problem? If you want live migration or ABI guarantees, you simply don't use this system to generate a new configuration. The same way you don't use the "pc" machine-type if you want to ensure compatibility with existing VMs.
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
We have a huge set of features where this is simply not a problem. For most virtual hardware features, enabling them is not even a policy decision: it's just about telling the guest that the feature is now available. QEMU have been enabling new features in the "pc" machine-type for years.
Now, why can't higher layers in the stack do something similar?
The proposal is equivalent to what already happens when people use the "pc" machine-type in their configurations, but: 1) the new defaults/features wouldn't be hidden behind a opaque machine-type name, and would appear in the domain XML explicitly; 2) the higher layers won't depend on QEMU introducing a new machine-type just to have new features enabled by default; 3) features that depend on host capabilities but are available on all hosts in a cluster can now be enabled automatically if desired (which is something QEMU can't do because it doesn't have enough information about the other hosts).
Choosing reasonable defaults might not be a trivial problem, but the current approach of pushing the responsibility to management layers doesn't improve the situation.
[...]
- Policies
[...]
- Abstracting the XML
[...]
- Identifying devices properly
[...]
- Generating the right XML snippet for device hot-(un)plug
[...]
These parts are trickier and I need to read the discussion more carefully before replying.
On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote:
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. [...]
This is true.
[...] So you need a matrix of libvirt +QEMU + config option settings.
But this is not. If config options need support on the lower levels of the stack (libvirt and/or QEMU and/or KVM and/or host hardware), it already has to be represented by libvirt host capabilities somehow, so management layers know it's available.
This means any new config generation system can (and must) use host(s) capabilities as input before generating the configuration.
I don't think it is that simple. The capabilities reflect what the current host is capable of only, not whether it is desirable to actually use them. Just because a host reports that it has q35-2.11.0 machine type doesn't mean that it should be used. The mgmt app may only wish to use that if it is available on all hosts in a particular grouping. The config generation library can't query every host directly to determine this. The mgmt app may have a way to collate capabilities info from hosts, but it is probably then stored in a app specific format and data source, or it may just ends up being a global config parameter to the mgmt app per host.
There have been a number of times where a feature is available in libvirt and/or QEMU, and the mgmt app still doesn't yet may still not wish to use it because it is known broken / incompatible with certain usage patterns. So the mgmt app would require an arbitrarily newer libvirt/qemu before considering using it, regardless of whether host capabilities report it is available.
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
Why is that a problem? If you want live migration or ABI guarantees, you simply don't use this system to generate a new configuration. The same way you don't use the "pc" machine-type if you want to ensure compatibility with existing VMs.
In many mgmt apps, every VM potentially needs live migration, so unless I'm misunderstanding, you're effectively saying don't ever use this config generator in these apps.
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
We have a huge set of features where this is simply not a problem. For most virtual hardware features, enabling them is not even a policy decision: it's just about telling the guest that the feature is now available. QEMU have been enabling new features in the "pc" machine-type for years.
Now, why can't higher layers in the stack do something similar?
The proposal is equivalent to what already happens when people use the "pc" machine-type in their configurations, but:
- the new defaults/features wouldn't be hidden behind a opaque machine-type name, and would appear in the domain XML explicitly;
- the higher layers won't depend on QEMU introducing a new machine-type just to have new features enabled by default;
- features that depend on host capabilities but are available on all hosts in a cluster can now be enabled automatically if desired (which is something QEMU can't do because it doesn't have enough information about the other hosts).
Choosing reasonable defaults might not be a trivial problem, but the current approach of pushing the responsibility to management layers doesn't improve the situation.
The simple cases have been added to the "pc" machine type, but more complex cases have not been dealt with as they often require contextual knowledge of either the host setup or the guest OS choice.
We had a long debate over the best aio=threads,native setting for OpenStack. Understanding the right defaults required knowledge about the various different ways that Nova would setup its storage stack. We certainly know enough now to be able to provide good recommendations for the choice, with perf data to back it up, but interpreting those recommendations still requires the app specific knowledge about its storage mgmt approach, so ends up being code dev work.
Another case is the pvpanic device - while in theory that could have been enabled by default for all guests, by QEMU or a config generator library, doing so is not useful on its own. The hard bit of the work is adding code to the mgmt app to choose the action for when pvpanic triggers, and code to handle the results of that action.
Regards, Daniel
On Wed, Mar 21, 2018 at 06:39:52PM +0000, Daniel P. Berrangé wrote:
On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote:
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. [...]
This is true.
[...] So you need a matrix of libvirt +QEMU + config option settings.
But this is not. If config options need support on the lower levels of the stack (libvirt and/or QEMU and/or KVM and/or host hardware), it already has to be represented by libvirt host capabilities somehow, so management layers know it's available.
This means any new config generation system can (and must) use host(s) capabilities as input before generating the configuration.
I don't think it is that simple. The capabilities reflect what the current host is capable of only, not whether it is desirable to actually use them. Just because a host reports that it has q35-2.11.0 machine type doesn't mean that it should be used. The mgmt app may only wish to use that if it is available on all hosts in a particular grouping. The config generation library can't query every host directly to determine this. The mgmt app may have a way to collate capabilities info from hosts, but it is probably then stored in a app specific format and data source, or it may just ends up being a global config parameter to the mgmt app per host.
In other words, you need host capabilities from all hosts as input when generating a new config XML. We already have a format to represent host capabilities defined by libvirt, users of the new system would just need to reproduce the data they got from libvirt and give it to the config generator.
Not completely trivial, but maybe worth the effort if you want to benefit from work done by other people to find good defaults?
There have been a number of times where a feature is available in libvirt and/or QEMU, and the mgmt app still doesn't yet may still not wish to use it because it is known broken / incompatible with certain usage patterns. So the mgmt app would require an arbitrarily newer libvirt/qemu before considering using it, regardless of whether host capabilities report it is available.
If this happens sometimes, why is it better for the teams maintaining management layers to duplicate the work of finding what works, instead of solving the problem only once?
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
Why is that a problem? If you want live migration or ABI guarantees, you simply don't use this system to generate a new configuration. The same way you don't use the "pc" machine-type if you want to ensure compatibility with existing VMs.
In many mgmt apps, every VM potentially needs live migration, so unless I'm misunderstanding, you're effectively saying don't ever use this config generator in these apps.
If you only need live migration, you can choose between: a) not using it; b) using an empty host capability list as input when generating the XML (maybe this would be completely useless, but it's still an option); c) use only host _software_ capabilities as input, if you control the software that runs on all hosts. d) use an intersection of the software+host capabilities of all hosts as input.
If you care about 100% static guest ABI (not just live migration), you either generate the XML once and save it for later, or you don't use the config generation system. (IOW, the same limitations as the "pc" machine-type alias).
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
We have a huge set of features where this is simply not a problem. For most virtual hardware features, enabling them is not even a policy decision: it's just about telling the guest that the feature is now available. QEMU have been enabling new features in the "pc" machine-type for years.
Now, why can't higher layers in the stack do something similar?
The proposal is equivalent to what already happens when people use the "pc" machine-type in their configurations, but:
- the new defaults/features wouldn't be hidden behind a opaque machine-type name, and would appear in the domain XML explicitly;
- the higher layers won't depend on QEMU introducing a new machine-type just to have new features enabled by default;
- features that depend on host capabilities but are available on all hosts in a cluster can now be enabled automatically if desired (which is something QEMU can't do because it doesn't have enough information about the other hosts).
Choosing reasonable defaults might not be a trivial problem, but the current approach of pushing the responsibility to management layers doesn't improve the situation.
The simple cases have been added to the "pc" machine type, but more complex cases have not been dealt with as they often require contextual knowledge of either the host setup or the guest OS choice.
Exactly. But on how many of those cases the decision requires knowledge that is specific to the management stack being used (like the ones you listed below), and how many are decisions that could be made by simply looking at the host software/hardware and guest OS? I am under the impression that we have a reasonable number of case of the latter.
The ones I remember are all relate to CPU configuration: * Automatically enabling useful CPU features when they are available on all hosts; * Always enabling check='full' by default.
Do we have other examples?
We had a long debate over the best aio=threads,native setting for OpenStack. Understanding the right defaults required knowledge about the various different ways that Nova would setup its storage stack. We certainly know enough now to be able to provide good recommendations for the choice, with perf data to back it up, but interpreting those recommendations still requires the app specific knowledge about its storage mgmt approach, so ends up being code dev work.
Another case is the pvpanic device - while in theory that could have been enabled by default for all guests, by QEMU or a config generator library, doing so is not useful on its own. The hard bit of the work is adding code to the mgmt app to choose the action for when pvpanic triggers, and code to handle the results of that action.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Mar 21, 2018 at 04:34:23PM -0300, Eduardo Habkost wrote:
On Wed, Mar 21, 2018 at 06:39:52PM +0000, Daniel P. Berrangé wrote:
On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote:
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. [...]
This is true.
[...] So you need a matrix of libvirt +QEMU + config option settings.
But this is not. If config options need support on the lower levels of the stack (libvirt and/or QEMU and/or KVM and/or host hardware), it already has to be represented by libvirt host capabilities somehow, so management layers know it's available.
This means any new config generation system can (and must) use host(s) capabilities as input before generating the configuration.
I don't think it is that simple. The capabilities reflect what the current host is capable of only, not whether it is desirable to actually use them. Just because a host reports that it has q35-2.11.0 machine type doesn't mean that it should be used. The mgmt app may only wish to use that if it is available on all hosts in a particular grouping. The config generation library can't query every host directly to determine this. The mgmt app may have a way to collate capabilities info from hosts, but it is probably then stored in a app specific format and data source, or it may just ends up being a global config parameter to the mgmt app per host.
In other words, you need host capabilities from all hosts as input when generating a new config XML. We already have a format to represent host capabilities defined by libvirt, users of the new system would just need to reproduce the data they got from libvirt and give it to the config generator.
Things aren't that simple - when openstack reports info from each host it doesn't do it in any libvirt format - it uses an arbitrary format it defines itself. Going from libvirt host capabilities to the app specific format and back to libvirt host capabilities will loose information. Then you also have matter of hosts coming & going over time, so fragile to assume that the set of host capabilities you currently see are representative of the steady state you desire.
Not completely trivial, but maybe worth the effort if you want to benefit from work done by other people to find good defaults?
Perhaps, but there's many ways to share the work of figuring out good defaults. Beyond what's represented in libosinfo database, no one has even tried to document what current desirable defaults are. Jumping straight from no documented best practice, to lets build a API is a big ask, particularly when the suggestion involves major architectural changes to any app that wants to use it.
For most immediate benefit actually documenting some best practice would be the most tangible win for application developers, as they can much more easily adapt existing code to follow it. ALso expanding range of info we record in libosinfo would be beneficial, since there is still plenty of OS specific data not captured. Not to mention that most applications aren't even leveraging much of the stuff already available.
There have been a number of times where a feature is available in libvirt and/or QEMU, and the mgmt app still doesn't yet may still not wish to use it because it is known broken / incompatible with certain usage patterns. So the mgmt app would require an arbitrarily newer libvirt/qemu before considering using it, regardless of whether host capabilities report it is available.
If this happens sometimes, why is it better for the teams maintaining management layers to duplicate the work of finding what works, instead of solving the problem only once?
This point was in relation to my earlier thread where I said that it would be neccessary to maintain a matrix of policy vs QEMU and libvirt versions, not merely relying on host capabilities.
Now, why can't higher layers in the stack do something similar?
The proposal is equivalent to what already happens when people use the "pc" machine-type in their configurations, but:
- the new defaults/features wouldn't be hidden behind a opaque machine-type name, and would appear in the domain XML explicitly;
- the higher layers won't depend on QEMU introducing a new machine-type just to have new features enabled by default;
- features that depend on host capabilities but are available on all hosts in a cluster can now be enabled automatically if desired (which is something QEMU can't do because it doesn't have enough information about the other hosts).
Choosing reasonable defaults might not be a trivial problem, but the current approach of pushing the responsibility to management layers doesn't improve the situation.
The simple cases have been added to the "pc" machine type, but more complex cases have not been dealt with as they often require contextual knowledge of either the host setup or the guest OS choice.
Exactly. But on how many of those cases the decision requires knowledge that is specific to the management stack being used (like the ones you listed below), and how many are decisions that could be made by simply looking at the host software/hardware and guest OS? I am under the impression that we have a reasonable number of case of the latter.
Anything todo with virtual hardware that is guest OS dependant should be in scope of libosinfo project / database.
For other things, I think it would be useful if we at least started to document some recommended best practices, so we have a better idea of what we're trying to address. It would also give apps an idea of what they're missing right now letting them fix gaps, if desired.
The ones I remember are all relate to CPU configuration:
- Automatically enabling useful CPU features when they are available on all hosts;
This is really hard todo in an automated fashion because it relies on having an accessible global view of all hosts, that is accurate. I can easily see a situation where you have 20 hosts, 5 old CPUs, 15 new CPUs, and the old ones are coincidentally offline for maintenance or software upgrade. Meanwhile you spawn a guest, and check available host capabilities and never see the info from older CPUs, so automatically enable a bunch of features that we really did not want. It is more reliable if you just declare this in the application config file, and have a mgmt tool that can do distributed updates of the config file when needed.
- Always enabling check='full' by default.
Do we have other examples?
I'm sure we can find plenty, but its a matter of someone doing the work to investigate & pull together docs.
We had a long debate over the best aio=threads,native setting for OpenStack. Understanding the right defaults required knowledge about the various different ways that Nova would setup its storage stack. We certainly know enough now to be able to provide good recommendations for the choice, with perf data to back it up, but interpreting those recommendations still requires the app specific knowledge about its storage mgmt approach, so ends up being code dev work.
Another case is the pvpanic device - while in theory that could have been enabled by default for all guests, by QEMU or a config generator library, doing so is not useful on its own. The hard bit of the work is adding code to the mgmt app to choose the action for when pvpanic triggers, and code to handle the results of that action.
Regards, Daniel
On Thu, Mar 22, 2018 at 09:56:12AM +0000, Daniel P. Berrangé wrote:
On Wed, Mar 21, 2018 at 04:34:23PM -0300, Eduardo Habkost wrote:
On Wed, Mar 21, 2018 at 06:39:52PM +0000, Daniel P. Berrangé wrote:
On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote:
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. [...]
This is true.
[...] So you need a matrix of libvirt +QEMU + config option settings.
But this is not. If config options need support on the lower levels of the stack (libvirt and/or QEMU and/or KVM and/or host hardware), it already has to be represented by libvirt host capabilities somehow, so management layers know it's available.
This means any new config generation system can (and must) use host(s) capabilities as input before generating the configuration.
I don't think it is that simple. The capabilities reflect what the current host is capable of only, not whether it is desirable to actually use them. Just because a host reports that it has q35-2.11.0 machine type doesn't mean that it should be used. The mgmt app may only wish to use that if it is available on all hosts in a particular grouping. The config generation library can't query every host directly to determine this. The mgmt app may have a way to collate capabilities info from hosts, but it is probably then stored in a app specific format and data source, or it may just ends up being a global config parameter to the mgmt app per host.
In other words, you need host capabilities from all hosts as input when generating a new config XML. We already have a format to represent host capabilities defined by libvirt, users of the new system would just need to reproduce the data they got from libvirt and give it to the config generator.
Things aren't that simple - when openstack reports info from each host it doesn't do it in any libvirt format - it uses an arbitrary format it defines itself. Going from libvirt host capabilities to the app specific format and back to libvirt host capabilities will loose information. Then you also have matter of hosts coming & going over time, so fragile to assume that the set of host capabilities you currently see are representative of the steady state you desire.
Well, then the management layer should stop losing useful data. ;)
(But I understand that this is not that simple)
Not completely trivial, but maybe worth the effort if you want to benefit from work done by other people to find good defaults?
Perhaps, but there's many ways to share the work of figuring out good defaults. Beyond what's represented in libosinfo database, no one has even tried to document what current desirable defaults are. Jumping straight from no documented best practice, to lets build a API is a big ask, particularly when the suggestion involves major architectural changes to any app that wants to use it.
For most immediate benefit actually documenting some best practice would be the most tangible win for application developers, as they can much more easily adapt existing code to follow it. ALso expanding range of info we record in libosinfo would be beneficial, since there is still plenty of OS specific data not captured. Not to mention that most applications aren't even leveraging much of the stuff already available.
This is a good point.
There have been a number of times where a feature is available in libvirt and/or QEMU, and the mgmt app still doesn't yet may still not wish to use it because it is known broken / incompatible with certain usage patterns. So the mgmt app would require an arbitrarily newer libvirt/qemu before considering using it, regardless of whether host capabilities report it is available.
If this happens sometimes, why is it better for the teams maintaining management layers to duplicate the work of finding what works, instead of solving the problem only once?
This point was in relation to my earlier thread where I said that it would be neccessary to maintain a matrix of policy vs QEMU and libvirt versions, not merely relying on host capabilities.
I see what you mean. But any component in the system needs to keep a matrix of QEMU and libvirt versions, I'd argue that the APIs provided by QEMU & libvirt are broken and need to be fixed.
If this happens with QEMU, I ask for everybody involved to please ask QEMU developers for help, so we can at least document the issue, and find a better way to detect if a given feature is working.
(This request applies even if our effort is focused towards documenting best practices and not an API.)
Now, why can't higher layers in the stack do something similar?
The proposal is equivalent to what already happens when people use the "pc" machine-type in their configurations, but:
- the new defaults/features wouldn't be hidden behind a opaque machine-type name, and would appear in the domain XML explicitly;
- the higher layers won't depend on QEMU introducing a new machine-type just to have new features enabled by default;
- features that depend on host capabilities but are available on all hosts in a cluster can now be enabled automatically if desired (which is something QEMU can't do because it doesn't have enough information about the other hosts).
Choosing reasonable defaults might not be a trivial problem, but the current approach of pushing the responsibility to management layers doesn't improve the situation.
The simple cases have been added to the "pc" machine type, but more complex cases have not been dealt with as they often require contextual knowledge of either the host setup or the guest OS choice.
Exactly. But on how many of those cases the decision requires knowledge that is specific to the management stack being used (like the ones you listed below), and how many are decisions that could be made by simply looking at the host software/hardware and guest OS? I am under the impression that we have a reasonable number of case of the latter.
Anything todo with virtual hardware that is guest OS dependant should be in scope of libosinfo project / database.
For other things, I think it would be useful if we at least started to document some recommended best practices, so we have a better idea of what we're trying to address. It would also give apps an idea of what they're missing right now letting them fix gaps, if desired.
Agreed, though I expect any documented best practices will also end up including guest-specific recommendations (which may or may not be already in the libosinfo database).
The ones I remember are all relate to CPU configuration:
- Automatically enabling useful CPU features when they are available on all hosts;
This is really hard todo in an automated fashion because it relies on having an accessible global view of all hosts, that is accurate. I can easily see a situation where you have 20 hosts, 5 old CPUs, 15 new CPUs, and the old ones are coincidentally offline for maintenance or software upgrade. Meanwhile you spawn a guest, and check available host capabilities and never see the info from older CPUs, so automatically enable a bunch of features that we really did not want. It is more reliable if you just declare this in the application config file, and have a mgmt tool that can do distributed updates of the config file when needed.
This surprises me a bit. I really expect any management layer to have an accurate and updated global view of all hosts, even if they are temporarily offline. And if new hosts are expected to be added in the future, the system should have a very clearly defined expectation[1] of what are the minimal capabilities required for new hosts.
Otherwise we will be pushing complex decisions to human operators (which will probably end up making worse mistakes because they may not understand how everything works internally).
- Always enabling check='full' by default.
Do we have other examples?
I'm sure we can find plenty, but its a matter of someone doing the work to investigate & pull together docs.
Agreed that documenting this stuff is the most important step right now.
We had a long debate over the best aio=threads,native setting for OpenStack. Understanding the right defaults required knowledge about the various different ways that Nova would setup its storage stack. We certainly know enough now to be able to provide good recommendations for the choice, with perf data to back it up, but interpreting those recommendations still requires the app specific knowledge about its storage mgmt approach, so ends up being code dev work.
Another case is the pvpanic device - while in theory that could have been enabled by default for all guests, by QEMU or a config generator library, doing so is not useful on its own. The hard bit of the work is adding code to the mgmt app to choose the action for when pvpanic triggers, and code to handle the results of that action.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. So you need a matrix of libvirt + QEMU + config option settings.
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
This is a good point and would be definitely an issue. One way how to solve it could be that the virtuned would be able to return you the default policy, the management application would save it somewhere if it cares about reproducing the same XML with the same default policies with newer version of virtuned/libvirt/qemu/kvm/kernel.
The API to create a new domain or hotplugable device would take another optional parameter where you could pass the default policy in addition to application specific policy.
That way you would get the same XML even with newer components with the same benefit that you wound not have to reimplement the logic.
One more thing could be automatically figuring out best values based on libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need to be added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example with the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type, but you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some DSL for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a device and that's it. We could do that as well. I can't really tell how much of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
The libvirt-gconfig project aims to provide a C API for manipulating XML documents, with language bindings available via GObject introspection so you can use it from Vala, Perl, Python, JavaScript, etc. Go is notable missing, but for that we have libvirt-go-xml which provides a set of native Go structs to represent the XML.
The problem we've faced with libvirt-gconfig is that it is a really hard sell to get applications to convert existing code to use it. We've only had success where an applicaiton has been written to use libvirt-gconfig from day one - eg GNOME Boxes and libvirt-sandbox.
Virt-manager is the poster-child for using libvirt-gconfig, but I don't see it adopting it any time soon, as it is a massive effort to change all existing code - even if libvirt-gconfig had full XML schema coverge, which it doesn't :-(
I also wanted to use libvirt-gconfig in OpenStack, but there was resistance for adding a dependancy on another native library, so there we've basically copied what virt-manager does and defined a set of pure python objects to represent config which is serialized to / from XML :-( If libvirt-gconfig were to be used by OpenStack it would need to deal with fact that not every distro has that available, so the existing pure python config objects would need to be maintained in parallel for an indefinite amount of time. So in fact using libvirt-gconfig would increase maint burden for OpenStack, rather than reduce it.
libvirt-gconfig would be a hard sell for Go apps when you compare it to libvirt-go-xml, because the latter is following the common Go paradigm for XML manipulation.
If there was something higher level that gets more interesting, but the hard bit is that you still need a way to get at all the low level bits becuase a higher level abstracted API will never cover every niche use case.
- Identifying devices properly
In contrast to the previous point, stateful apps might have a problem identifying devices after hotplug. For example, let's say you don't care about the addresses and leave that up to libvirt. You hotplug a device into the domain and dump the new XML of it. Depending on what type of device it was, you might need to identify it based on different values. It could be <target dev=''/> for disks, <mac address=''/> for interfaces etc. For some devices it might not even be possible and you need to remember the addresses of all the previous devices and then parse them just to identify that one device and then throw them away.
With new enough libvirt you could use the user aliases for that, but turns out it's not that easy to use them properly anyway. Also the aliases won't help users identify that device inside the guest.
NB, relating between host device config and guest visible device config is a massive problem space in its own right, and not very easy to address. In OpenStack we ended up defining a concept of "device tagging" via cloud-init metadata, where openstack allows users to set opaque string tags against devices their VM has. OpenStack that generates a metadata file that records various pieces of identifying hardware attributes (PCI address, MAC addr, disk serial, etc) alongside the user tag. This metadata file is exposed to the guest with the hope that there's enough info to allow the user to decide which device is to be used for which purpose
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-... https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/...
<rant> We really should've gone with new attribute for the user alias instead of using an existing one, given how many problems that is causing. </rant>
- Generating the right XML snippet for device hot-(un)plug
This is kind of related to some previous points.
When hot-plugging a device and creating an XML snippet for it, you want to keep the defaults from point 1) and policies from 2) in mind. Or something related to the already existing domain which you can describe systematically. And adding something for identification (see previous point).
Doing the hot-unplug is easy depending on how much information about that device is saved by your application. The less you save about the device (or show to the user in a GUI, if applicable) the harder it might be to generate an XML that libvirt will accept. Again, some problems with this should be fixed in libvirt, some of them are easy to workaround. But having a common ground that takes care of this should help some projects.
Hot-unplug could be implemented just based on the alias. This is something that would fit into libvirt as well.
========================================================================
To mention some pre-existing solutions:
- I understand OpenStack has some really sensible and wisely chosen
and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's decisions have been largely inspired by previous work in oVirt and / or virt-manager. So there's obviously overlap in the conceptual area, but there's also plenty that is very specific to OpenStack - untangling the two extract the common bits from the app specific bits is hard.
This would be handled by the application specific policies. The virtuned will have some reasonable defaults that are known to work in most cases and suits the majority of users, but it's clear that sometimes you need some specific defaults and you would provide them via the application policy.
For example, to create a XML for windows guest the virtuned would not probably select virtio devices because there are no drivers for them in the standard windows installation, however, some management application may have customized preinstalled disk images or customized ISO images or it may be able to provide the drivers any other way, so they would specify in the application policy that for windows guest virtuned should use virtio as a default device model.
- I know KubeVirt has VirtualMachinePresets. That is something closely
related to points 1) and 2). Also their abstraction of the XML might be usable for point 3).
- There was an effort on creating policy based configuration of libvirt
objects called libvirt-designer. This is closely related to points 2) and 3). Unfortunately there was no much going on lately and part of virt-manager repository has currently more features implemented with the same ideas in mind, just not exported for public use.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the use of libosinfo native library via GObject Introspection, by promising to do work to turn the osinfo database into an approved stable format which OpenStack could then consume directly, dropping the native API usage :-( Incidentally, the former was done (formal spec for the DB format), but the latter was not yet (direct DB usage by OpenStack)
BTW, I don't like that I'm being so negative to your proposal :-( I used to hope that we would be able to build higher level APIs on top of libvirt to reduce the overlap between different applications reinventing the wheel. Even the simplest bits we tried like the gconfig/gobject API are barely used. libvirt-designer is basically a failure. Though admittedly it didn't have enough development resource applied to make it compelling, in retrospect adoption was always going to be a hard sell except in greenfield developments.
Libosinfo is probably the bit we've had most success with, and has most promise for the future, particularly now that we formally allow apps to read the osinfo database directly and bypass the API. It is quite easy to fit into existing application codebases which helps alot. Even there I'm still disappointed that we only have GNOME Boxes using the kickstart generator part of osinfo - oVirt and Oz both still have their own kickstart generator code for automating OS installs.
In general though, I fear anything API based is going to be a really hard sell to get wide adoption for based on what we've seen before.
I think the biggest bang-for-buck is identifying more areas where we can turn code into data. There's definitely scope for recording more types of information in the osinfo database. There might also be scope for defining entirely new databases to complement the osinfo data, if something looks out of scope for libosinfo.
This is probably the hardest part of creating higher level API on top of libvirt, not every project may be willing to rewrite their existing code. On the other hand, I know that for example Cockpit would benefit from the virtuned providing this functionality via REST API.
It's a chicken and egg problem, but if we can gather input from all the existing projects that have their own implementation and figure out how to make virtuned usable for all of them they might consider to start using it.
Pavel
On Thu, Mar 22, 2018 at 09:40:40AM +0100, Pavel Hrdina wrote:
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
- I understand OpenStack has some really sensible and wisely chosen
and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's decisions have been largely inspired by previous work in oVirt and / or virt-manager. So there's obviously overlap in the conceptual area, but there's also plenty that is very specific to OpenStack - untangling the two extract the common bits from the app specific bits is hard.
This would be handled by the application specific policies. The virtuned will have some reasonable defaults that are known to work in most cases and suits the majority of users, but it's clear that sometimes you need some specific defaults and you would provide them via the application policy.
For example, to create a XML for windows guest the virtuned would not probably select virtio devices because there are no drivers for them in the standard windows installation, however, some management application may have customized preinstalled disk images or customized ISO images or it may be able to provide the drivers any other way, so they would specify in the application policy that for windows guest virtuned should use virtio as a default device model.
As soon as we talk about configuring hardware specific to a guest OS, then that is in scope of existing libosinfo project, not something we should create a new project for.
This is probably the hardest part of creating higher level API on top of libvirt, not every project may be willing to rewrite their existing code. On the other hand, I know that for example Cockpit would benefit from the virtuned providing this functionality via REST API.
It's a chicken and egg problem, but if we can gather input from all the existing projects that have their own implementation and figure out how to make virtuned usable for all of them they might consider to start using it.
I don't doubt that Cockpit would like like, but based on previous efforts we've made I'm sceptical that anything beyond Cockpit would use any new API.
Regards, Daniel
On 22 Mar 2018, at 10:37, Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Mar 22, 2018 at 09:40:40AM +0100, Pavel Hrdina wrote:
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
- I understand OpenStack has some really sensible and wisely chosen
and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's decisions have been largely inspired by previous work in oVirt and / or virt-manager. So there's obviously overlap in the conceptual area, but there's also plenty that is very specific to OpenStack - untangling the two extract the common bits from the app specific bits is hard.
This would be handled by the application specific policies. The virtuned will have some reasonable defaults that are known to work in most cases and suits the majority of users, but it's clear that sometimes you need some specific defaults and you would provide them via the application policy.
For example, to create a XML for windows guest the virtuned would not probably select virtio devices because there are no drivers for them in the standard windows installation, however, some management application may have customized preinstalled disk images or customized ISO images or it may be able to provide the drivers any other way, so they would specify in the application policy that for windows guest virtuned should use virtio as a default device model.
As soon as we talk about configuring hardware specific to a guest OS, then that is in scope of existing libosinfo project, not something we should create a new project for.
This is probably the hardest part of creating higher level API on top of libvirt, not every project may be willing to rewrite their existing code. On the other hand, I know that for example Cockpit would benefit from the virtuned providing this functionality via REST API.
It's a chicken and egg problem, but if we can gather input from all the existing projects that have their own implementation and figure out how to make virtuned usable for all of them they might consider to start using it.
I don't doubt that Cockpit would like like, but based on previous efforts we've made I'm sceptical that anything beyond Cockpit would use any new API.
We have an opportunity to make kubevirt right, too. Sure, the project is halfway through with own presets and custom code (just repeating oVirt and Openstack), but it’s not too late to change things there, and a remote API fits well
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- You received this message because you are subscribed to the Google Groups "kubevirt-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com. To post to this group, send email to kubevirt-dev@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/20180322093722.GB3583%40redha.... For more options, visit https://groups.google.com/d/optout.
[ I fixed up ovirt-devel@redhat.com to be devel@ovirt.org since the former is deprecated. I'm also not trimming down much of the reply so that they can get the whole picture. Sorry for the confusion ]
On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. So you need a matrix of libvirt + QEMU + config option settings.
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
The idea was for updating XML based on policy, which is something you want for new machines. You should then keep the XML per domain and only do changes to if requested by the user or when libvirt fills in new values in a guest ABI compatible fashion.
One more thing could be automatically figuring out best values based on libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need to be added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example with the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type, but you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some DSL for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a device and that's it. We could do that as well. I can't really tell how much of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
Sure, this was just meant to be a question as to whether it's worth pursuing or not. You make a good point on why it is not (at least for existing apps).
However, since this was optional, the way this would look without the XML abstraction is that both input and output would be valid domain definitions, ultimately resulting in something similar to virt-xml with the added benefit of applying a policy from a file/string either supplied by the application itself. Whether that policy was taken from a common repository of such knowledge is orthogonal to this idea. Since you would work with the same data, the upgrade could be incremental as you'd only let virtuned fill in values for new options and could slowly move on to using it for some pre-existing ones. None of the previous approaches did this, if I'm not mistaken. Of course it gets more difficult when you need to expose all the bits libvirt does and keep them in sync (as you write below).
[...]
If there was something higher level that gets more interesting, but the hard bit is that you still need a way to get at all the low level bits becuase a higher level abstracted API will never cover every niche use case.
Oh, definitely not every, but I see two groups of projects that have a lot in common between themselves and between the groups as well. On the other hand just templating and defaults is something that's easy enough to do that it's not worth outsourcing that into another one's codebase.
- Identifying devices properly
In contrast to the previous point, stateful apps might have a problem identifying devices after hotplug. For example, let's say you don't care about the addresses and leave that up to libvirt. You hotplug a device into the domain and dump the new XML of it. Depending on what type of device it was, you might need to identify it based on different values. It could be <target dev=''/> for disks, <mac address=''/> for interfaces etc. For some devices it might not even be possible and you need to remember the addresses of all the previous devices and then parse them just to identify that one device and then throw them away.
With new enough libvirt you could use the user aliases for that, but turns out it's not that easy to use them properly anyway. Also the aliases won't help users identify that device inside the guest.
NB, relating between host device config and guest visible device config is a massive problem space in its own right, and not very easy to address. In OpenStack we ended up defining a concept of "device tagging" via cloud-init metadata, where openstack allows users to set opaque string tags against devices their VM has. OpenStack that generates a metadata file that records various pieces of identifying hardware attributes (PCI address, MAC addr, disk serial, etc) alongside the user tag. This metadata file is exposed to the guest with the hope that there's enough info to allow the user to decide which device is to be used for which purpose
This is good point, but I was mostly thinking about identifying devices from the host POV between two different XMLs (pre- and post- some XML-modifying action, like hotplug).
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-... https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/...
<rant> We really should've gone with new attribute for the user alias instead of using an existing one, given how many problems that is causing. </rant>
- Generating the right XML snippet for device hot-(un)plug
This is kind of related to some previous points.
When hot-plugging a device and creating an XML snippet for it, you want to keep the defaults from point 1) and policies from 2) in mind. Or something related to the already existing domain which you can describe systematically. And adding something for identification (see previous point).
Doing the hot-unplug is easy depending on how much information about that device is saved by your application. The less you save about the device (or show to the user in a GUI, if applicable) the harder it might be to generate an XML that libvirt will accept. Again, some problems with this should be fixed in libvirt, some of them are easy to workaround. But having a common ground that takes care of this should help some projects.
Hot-unplug could be implemented just based on the alias. This is something that would fit into libvirt as well.
========================================================================
To mention some pre-existing solutions:
- I understand OpenStack has some really sensible and wisely chosen
and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's decisions have been largely inspired by previous work in oVirt and / or virt-manager. So there's obviously overlap in the conceptual area, but there's also plenty that is very specific to OpenStack - untangling the two extract the common bits from the app specific bits is hard.
It definitely is, but do you think it's so difficult it's worthless to pursuit? I did a tiny PoC based on the code from virt-manager, which was trivial mainly thanks to the XMLBuilder for the domain objects. Maybe exposing an easy way to work with the XML would be enough for some projects.
Little birdie from oVirt told me that they would like some of sort of thing that does what you can achieve with virt-xml if we, for example, made it work on pure XML definitions without connecting to libvirt.
- I know KubeVirt has VirtualMachinePresets. That is something closely
related to points 1) and 2). Also their abstraction of the XML might be usable for point 3).
- There was an effort on creating policy based configuration of libvirt
objects called libvirt-designer. This is closely related to points 2) and 3). Unfortunately there was no much going on lately and part of virt-manager repository has currently more features implemented with the same ideas in mind, just not exported for public use.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
The initial idea (which I forgot to mention) was that all the decisions libvirt currently does (so that it keeps the guest ABI stable) would be moved into data (let's say some DSL) and it could then be switched or adjusted if that's not what the mgmt app wants (on a per-definition basis, of course). I didn't feel very optimistic about the upstream acceptance for that idea, so I figured that there could be something that lives beside libvirt, helps with some policies if requested and then the resulting XML could be fed into libvirt for determining the rest.
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the
By native you mean actual binary libraries or native to the OpenStack code as in python module? Because what I had in mind for this project was a python module with optional wrapper for REST API.
use of libosinfo native library via GObject Introspection, by promising to do work to turn the osinfo database into an approved stable format which OpenStack could then consume directly, dropping the native API usage :-( Incidentally, the former was done (formal spec for the DB format), but the latter was not yet (direct DB usage by OpenStack)
BTW, I don't like that I'm being so negative to your proposal :-( I used to hope that we would be able to build higher level APIs on top of libvirt to reduce the overlap between different applications reinventing the wheel. Even the simplest bits we tried like the gconfig/gobject API are barely used. libvirt-designer is basically a failure. Though admittedly it didn't have enough development resource applied to make it compelling, in retrospect adoption was always going to be a hard sell except in greenfield developments.
I'm glad for the knowledge you provided. So maybe instead of focusing on de-duplication of existing codebases we could _at least_ aim at future mgmt apps. OTOH improving documentation on how to properly build higher level concepts on top of libvirt would benefit them as well.
Libosinfo is probably the bit we've had most success with, and has most promise for the future, particularly now that we formally allow apps to read the osinfo database directly and bypass the API. It is quite easy to fit into existing application codebases which helps alot. Even there I'm still disappointed that we only have GNOME Boxes using the kickstart generator part of osinfo - oVirt and Oz both still have their own kickstart generator code for automating OS installs.
In general though, I fear anything API based is going to be a really hard sell to get wide adoption for based on what we've seen before.
I think the biggest bang-for-buck is identifying more areas where we can turn code into data. There's definitely scope for recording more types of information in the osinfo database. There might also be scope for defining entirely new databases to complement the osinfo data, if something looks out of scope for libosinfo.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Mar 22, 2018 at 03:54:01PM +0100, Martin Kletzander wrote:
One more thing could be automatically figuring out best values based on libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need to be added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example with the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type, but you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some DSL for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a device and that's it. We could do that as well. I can't really tell how much of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
Sure, this was just meant to be a question as to whether it's worth pursuing or not. You make a good point on why it is not (at least for existing apps).
However, since this was optional, the way this would look without the XML abstraction is that both input and output would be valid domain definitions, ultimately resulting in something similar to virt-xml with the added benefit of applying a policy from a file/string either supplied by the application itself. Whether that policy was taken from a common repository of such knowledge is orthogonal to this idea. Since you would work with the same data, the upgrade could be incremental as you'd only let virtuned fill in values for new options and could slowly move on to using it for some pre-existing ones. None of the previous approaches did this, if I'm not mistaken. Of course it gets more difficult when you need to expose all the bits libvirt does and keep them in sync (as you write below).
That has implications for how mgmt app deals with XML. Nova has object models for representing XML in memory, but it doesn't aim to have loss-less roundtrip from parse -> object -> format. So if Nova gets basic XML from virttuned, parses it into its object to let it set more fields and then formats it again, chances are it will have lost a bunch of stuff from virttuned. Of course if you know about this need upfront you can design the application such that it can safely round-trip, but this is just example of problem with integrating to existing apps.
The other thing that concerns is that there are dependancies between different bits of XML for a given device. ie if feature X is set to a certain value, that prevents use of feature Y. So if virttuned sets feature X, but the downstream application uses feature Y, the final result can be incompatible. The application won't know this because it doesn't control what stuff virttuned would be setting. This can in turn cause ordering constraints.
eg the application needs to say that virtio-net is being used, then virttuned can set some defaults like enabling vhost-net, and then the application can fill in more bits that it cares about. Or if we let virttuned go first, setting virtio-net model + vhost-net, then application wants to change model to e1000e, it has to be aware that it must now delete the vhost-net bit that virtuned added. This ends up being more complicated that just ignoring virttuned and coding up use of vhost-net in application code.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
The initial idea (which I forgot to mention) was that all the decisions libvirt currently does (so that it keeps the guest ABI stable) would be moved into data (let's say some DSL) and it could then be switched or adjusted if that's not what the mgmt app wants (on a per-definition basis, of course). I didn't feel very optimistic about the upstream acceptance for that idea, so I figured that there could be something that lives beside libvirt, helps with some policies if requested and then the resulting XML could be fed into libvirt for determining the rest.
I can't even imagine how we would go about encoding the stable guest ABI logic libvirt does today in data !
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the
By native you mean actual binary libraries or native to the OpenStack code as in python module? Because what I had in mind for this project was a python module with optional wrapper for REST API.
I meant native binary libraries. ie openstack is not happy in general with adding dependancies on new OS services, because there's a big time lag for getting them into all distros. By comparison a pure python library, they can just handle automatically in their deployment tools, just pip installing on any OS distro straight from pypi. This is what made use of libosinfo a hard sell in Nova.
The same thing is seen with Go / Rust where some applications have decided they're better of actually re-implementing the libvirt RPC protocol in Go / Rust rather than use the libvirt.so client. I think this is a bad tradeoff in general, but I can see why they like it
Regards, Daniel
Hi, I'd like to go one step back and discuss why we should try to do this on the high level.
For the last 5-10 years of KVM development, we are pragmatically providing the Linux host level APIs via project specific host agents/integration code (Nova agent, oVirt host agent, virt-manager). In recent time we see new projects that have similar requirements (Cockpit, different automation tool, KubeVirt), this means that all of the Linux virt stack consumers are reinventing the wheel and using very different paths to consume the partial solutions that are provided today.
The use of the Linux virt stack is well defined by the existing projects scope and it makes a lot of sense to try to provide the common patterns via the virt stack directly as a host level API that different client or management consume. The main goal is to improve the developer experience for virtualization management applications with an API set that is useful to the entire set of tools (OSP, oVirt, KubeVirt, Cockpit and so on).
The Linux virt developer community currently is not able to provide best practices and optimizations from single node knowledge. This means that all of that smarts is locked to the specific project integration in the good case or not provided at all and the projects as a whole lose from that. When testing the Linux virt stack itself and since each project has different usage pattern, we lose the ability to test abilities on the lower level making the entire stack less stable and complete for new features.
This also limits the different projects ability to contribute back to the Linux stack based on their user and market experience for others in open source to gain.
I understand this shift is technically challenging for existing projects, but I do see value in doing this even for new implementation like Cockpit and KubeVirt. I also believe that the end result could be appealing enough to cause project like OSP, virt-manager and oVirt to consider to reduce the existing capabilities of their host side integrations/agents to shims on the host level and reuse the common/better-tested pattern as clients that was developed against the experience of the different projects.
I call us all to collaborate and try to converge on a solution that will help all in the long term in the value you get from the common base.
Thanks,
YANIV LAVI
SENIOR TECHNICAL PRODUCT MANAGER
Red Hat Israel Ltd. https://www.redhat.com/
34 Jerusalem Road, Building A, 1st floor
Ra'anana, Israel 4350109
ylavi@redhat.com T: +972-9-7692306/8272306 F: +972-9-7692223 IM: ylavi https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc
On Thu, Mar 22, 2018 at 7:18 PM, Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Mar 22, 2018 at 03:54:01PM +0100, Martin Kletzander wrote:
One more thing could be automatically figuring out best values based
on
libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need
to be
added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example
with
the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type,
but
you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some
DSL
for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a
device
and that's it. We could do that as well. I can't really tell how
much
of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
Sure, this was just meant to be a question as to whether it's worth pursuing or not. You make a good point on why it is not (at least for existing apps).
However, since this was optional, the way this would look without the XML abstraction is that both input and output would be valid domain definitions, ultimately resulting in something similar to virt-xml with the added benefit of applying a policy from a file/string either supplied by the application itself. Whether that policy was taken from a common repository of such knowledge is orthogonal to this idea. Since you would work with the same data, the upgrade could be incremental as you'd only let virtuned fill in values for new options and could slowly move on to using it for some pre-existing ones. None of the previous approaches did this, if I'm not mistaken. Of course it gets more difficult when you need to expose all the bits libvirt does and keep them in sync (as you write below).
That has implications for how mgmt app deals with XML. Nova has object models for representing XML in memory, but it doesn't aim to have loss-less roundtrip from parse -> object -> format. So if Nova gets basic XML from virttuned, parses it into its object to let it set more fields and then formats it again, chances are it will have lost a bunch of stuff from virttuned. Of course if you know about this need upfront you can design the application such that it can safely round-trip, but this is just example of problem with integrating to existing apps.
The other thing that concerns is that there are dependancies between different bits of XML for a given device. ie if feature X is set to a certain value, that prevents use of feature Y. So if virttuned sets feature X, but the downstream application uses feature Y, the final result can be incompatible. The application won't know this because it doesn't control what stuff virttuned would be setting. This can in turn cause ordering constraints.
eg the application needs to say that virtio-net is being used, then virttuned can set some defaults like enabling vhost-net, and then the application can fill in more bits that it cares about. Or if we let virttuned go first, setting virtio-net model + vhost-net, then application wants to change model to e1000e, it has to be aware that it must now delete the vhost-net bit that virtuned added. This ends up being more complicated that just ignoring virttuned and coding up use of vhost-net in application code.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
The initial idea (which I forgot to mention) was that all the decisions libvirt currently does (so that it keeps the guest ABI stable) would be moved into data (let's say some DSL) and it could then be switched or adjusted if that's not what the mgmt app wants (on a per-definition basis, of course). I didn't feel very optimistic about the upstream acceptance for that idea, so I figured that there could be something that lives beside libvirt, helps with some policies if requested and then the resulting XML could be fed into libvirt for determining the rest.
I can't even imagine how we would go about encoding the stable guest ABI logic libvirt does today in data !
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the
By native you mean actual binary libraries or native to the OpenStack code as in python module? Because what I had in mind for this project was a python module with optional wrapper for REST API.
I meant native binary libraries. ie openstack is not happy in general with adding dependancies on new OS services, because there's a big time lag for getting them into all distros. By comparison a pure python library, they can just handle automatically in their deployment tools, just pip installing on any OS distro straight from pypi. This is what made use of libosinfo a hard sell in Nova.
The same thing is seen with Go / Rust where some applications have decided they're better of actually re-implementing the libvirt RPC protocol in Go / Rust rather than use the libvirt.so client. I think this is a bad tradeoff in general, but I can see why they like it
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/ dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/ dberrange :| _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[resending to include KubeVirt devs ]
YANIV LAVI
SENIOR TECHNICAL PRODUCT MANAGER
Red Hat Israel Ltd. https://www.redhat.com/
34 Jerusalem Road, Building A, 1st floor
Ra'anana, Israel 4350109
ylavi@redhat.com T: +972-9-7692306/8272306 F: +972-9-7692223 IM: ylavi https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc
On Wed, Apr 4, 2018 at 7:07 PM, Yaniv Lavi ylavi@redhat.com wrote:
Hi, I'd like to go one step back and discuss why we should try to do this on the high level.
For the last 5-10 years of KVM development, we are pragmatically providing the Linux host level APIs via project specific host agents/integration code (Nova agent, oVirt host agent, virt-manager). In recent time we see new projects that have similar requirements (Cockpit, different automation tool, KubeVirt), this means that all of the Linux virt stack consumers are reinventing the wheel and using very different paths to consume the partial solutions that are provided today.
The use of the Linux virt stack is well defined by the existing projects scope and it makes a lot of sense to try to provide the common patterns via the virt stack directly as a host level API that different client or management consume. The main goal is to improve the developer experience for virtualization management applications with an API set that is useful to the entire set of tools (OSP, oVirt, KubeVirt, Cockpit and so on).
The Linux virt developer community currently is not able to provide best practices and optimizations from single node knowledge. This means that all of that smarts is locked to the specific project integration in the good case or not provided at all and the projects as a whole lose from that. When testing the Linux virt stack itself and since each project has different usage pattern, we lose the ability to test abilities on the lower level making the entire stack less stable and complete for new features.
This also limits the different projects ability to contribute back to the Linux stack based on their user and market experience for others in open source to gain.
I understand this shift is technically challenging for existing projects, but I do see value in doing this even for new implementation like Cockpit and KubeVirt. I also believe that the end result could be appealing enough to cause project like OSP, virt-manager and oVirt to consider to reduce the existing capabilities of their host side integrations/agents to shims on the host level and reuse the common/better-tested pattern as clients that was developed against the experience of the different projects.
I call us all to collaborate and try to converge on a solution that will help all in the long term in the value you get from the common base.
Thanks,
YANIV LAVI
SENIOR TECHNICAL PRODUCT MANAGER
Red Hat Israel Ltd. https://www.redhat.com/
34 Jerusalem Road, Building A, 1st floor
Ra'anana, Israel 4350109
ylavi@redhat.com T: +972-9-7692306/8272306 F: +972-9-7692223 IM: ylavi https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc
On Thu, Mar 22, 2018 at 7:18 PM, Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Mar 22, 2018 at 03:54:01PM +0100, Martin Kletzander wrote:
One more thing could be automatically figuring out best values
based on
libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need
to be
added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example
with
the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type,
but
you don't really want to care about the machine type. Or you want
to
use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some
DSL
for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a
device
and that's it. We could do that as well. I can't really tell how
much
of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
Sure, this was just meant to be a question as to whether it's worth pursuing or not. You make a good point on why it is not (at least for existing apps).
However, since this was optional, the way this would look without the XML abstraction is that both input and output would be valid domain definitions, ultimately resulting in something similar to virt-xml with the added benefit of applying a policy from a file/string either supplied by the application itself. Whether that policy was taken from a common repository of such knowledge is orthogonal to this idea. Since you would work with the same data, the upgrade could be incremental as you'd only let virtuned fill in values for new options and could slowly move on to using it for some pre-existing ones. None of the previous approaches did this, if I'm not mistaken. Of course it gets more difficult when you need to expose all the bits libvirt does and keep them in sync (as you write below).
That has implications for how mgmt app deals with XML. Nova has object models for representing XML in memory, but it doesn't aim to have loss-less roundtrip from parse -> object -> format. So if Nova gets basic XML from virttuned, parses it into its object to let it set more fields and then formats it again, chances are it will have lost a bunch of stuff from virttuned. Of course if you know about this need upfront you can design the application such that it can safely round-trip, but this is just example of problem with integrating to existing apps.
The other thing that concerns is that there are dependancies between different bits of XML for a given device. ie if feature X is set to a certain value, that prevents use of feature Y. So if virttuned sets feature X, but the downstream application uses feature Y, the final result can be incompatible. The application won't know this because it doesn't control what stuff virttuned would be setting. This can in turn cause ordering constraints.
eg the application needs to say that virtio-net is being used, then virttuned can set some defaults like enabling vhost-net, and then the application can fill in more bits that it cares about. Or if we let virttuned go first, setting virtio-net model + vhost-net, then application wants to change model to e1000e, it has to be aware that it must now delete the vhost-net bit that virtuned added. This ends up being more complicated that just ignoring virttuned and coding up use of vhost-net in application code.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
The initial idea (which I forgot to mention) was that all the decisions libvirt currently does (so that it keeps the guest ABI stable) would be moved into data (let's say some DSL) and it could then be switched or adjusted if that's not what the mgmt app wants (on a per-definition basis, of course). I didn't feel very optimistic about the upstream acceptance for that idea, so I figured that there could be something that lives beside libvirt, helps with some policies if requested and then the resulting XML could be fed into libvirt for determining the rest.
I can't even imagine how we would go about encoding the stable guest ABI logic libvirt does today in data !
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the
By native you mean actual binary libraries or native to the OpenStack code as in python module? Because what I had in mind for this project was a python module with optional wrapper for REST API.
I meant native binary libraries. ie openstack is not happy in general with adding dependancies on new OS services, because there's a big time lag for getting them into all distros. By comparison a pure python library, they can just handle automatically in their deployment tools, just pip installing on any OS distro straight from pypi. This is what made use of libosinfo a hard sell in Nova.
The same thing is seen with Go / Rust where some applications have decided they're better of actually re-implementing the libvirt RPC protocol in Go / Rust rather than use the libvirt.so client. I think this is a bad tradeoff in general, but I can see why they like it
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/ dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dber range :| _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[resending to include OSP devs ]
YANIV LAVI
SENIOR TECHNICAL PRODUCT MANAGER
Red Hat Israel Ltd. https://www.redhat.com/
34 Jerusalem Road, Building A, 1st floor
Ra'anana, Israel 4350109
ylavi@redhat.com T: +972-9-7692306/8272306 F: +972-9-7692223 IM: ylavi https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc
On Wed, Apr 4, 2018 at 7:23 PM, Yaniv Lavi ylavi@redhat.com wrote:
[resending to include KubeVirt devs ]
YANIV LAVI
SENIOR TECHNICAL PRODUCT MANAGER
Red Hat Israel Ltd. https://www.redhat.com/
34 Jerusalem Road, Building A, 1st floor
Ra'anana, Israel 4350109
ylavi@redhat.com T: +972-9-7692306/8272306 F: +972-9-7692223 IM: ylavi https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc
On Wed, Apr 4, 2018 at 7:07 PM, Yaniv Lavi ylavi@redhat.com wrote:
Hi, I'd like to go one step back and discuss why we should try to do this on the high level.
For the last 5-10 years of KVM development, we are pragmatically providing the Linux host level APIs via project specific host agents/integration code (Nova agent, oVirt host agent, virt-manager). In recent time we see new projects that have similar requirements (Cockpit, different automation tool, KubeVirt), this means that all of the Linux virt stack consumers are reinventing the wheel and using very different paths to consume the partial solutions that are provided today.
The use of the Linux virt stack is well defined by the existing projects scope and it makes a lot of sense to try to provide the common patterns via the virt stack directly as a host level API that different client or management consume. The main goal is to improve the developer experience for virtualization management applications with an API set that is useful to the entire set of tools (OSP, oVirt, KubeVirt, Cockpit and so on).
The Linux virt developer community currently is not able to provide best practices and optimizations from single node knowledge. This means that all of that smarts is locked to the specific project integration in the good case or not provided at all and the projects as a whole lose from that. When testing the Linux virt stack itself and since each project has different usage pattern, we lose the ability to test abilities on the lower level making the entire stack less stable and complete for new features.
This also limits the different projects ability to contribute back to the Linux stack based on their user and market experience for others in open source to gain.
I understand this shift is technically challenging for existing projects, but I do see value in doing this even for new implementation like Cockpit and KubeVirt. I also believe that the end result could be appealing enough to cause project like OSP, virt-manager and oVirt to consider to reduce the existing capabilities of their host side integrations/agents to shims on the host level and reuse the common/better-tested pattern as clients that was developed against the experience of the different projects.
I call us all to collaborate and try to converge on a solution that will help all in the long term in the value you get from the common base.
Thanks,
YANIV LAVI
SENIOR TECHNICAL PRODUCT MANAGER
Red Hat Israel Ltd. https://www.redhat.com/
34 Jerusalem Road, Building A, 1st floor
Ra'anana, Israel 4350109
ylavi@redhat.com T: +972-9-7692306/8272306 F: +972-9-7692223 IM: ylavi https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatnews https://twitter.com/redhatnews Red Hat https://www.linkedin.com/company/red-hat Red Hat https://www.facebook.com/RedHatInc
On Thu, Mar 22, 2018 at 7:18 PM, Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Mar 22, 2018 at 03:54:01PM +0100, Martin Kletzander wrote:
One more thing could be automatically figuring out best values
based on
libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need
to be
added, but nobody really cares about them. Sometimes it's enough
to
have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For
example with
the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine
type, but
you don't really want to care about the machine type. Or you want
to
use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some
DSL
for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might
happen
that some apps don't really want to care about the XML at all.
They
just want an abstract view of the domain, possibly add/remove a
device
and that's it. We could do that as well. I can't really tell how
much
of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
Sure, this was just meant to be a question as to whether it's worth pursuing or not. You make a good point on why it is not (at least for existing apps).
However, since this was optional, the way this would look without the XML abstraction is that both input and output would be valid domain definitions, ultimately resulting in something similar to virt-xml with the added benefit of applying a policy from a file/string either supplied by the application itself. Whether that policy was taken from a common repository of such knowledge is orthogonal to this idea.
Since
you would work with the same data, the upgrade could be incremental as you'd only let virtuned fill in values for new options and could slowly move on to using it for some pre-existing ones. None of the previous approaches did this, if I'm not mistaken. Of course it gets more difficult when you need to expose all the bits libvirt does and keep them in sync (as you write below).
That has implications for how mgmt app deals with XML. Nova has object models for representing XML in memory, but it doesn't aim to have loss-less roundtrip from parse -> object -> format. So if Nova gets basic XML from virttuned, parses it into its object to let it set more fields and then formats it again, chances are it will have lost a bunch of stuff from virttuned. Of course if you know about this need upfront you can design the application such that it can safely round-trip, but this is just example of problem with integrating to existing apps.
The other thing that concerns is that there are dependancies between different bits of XML for a given device. ie if feature X is set to a certain value, that prevents use of feature Y. So if virttuned sets feature X, but the downstream application uses feature Y, the final result can be incompatible. The application won't know this because it doesn't control what stuff virttuned would be setting. This can in turn cause ordering constraints.
eg the application needs to say that virtio-net is being used, then virttuned can set some defaults like enabling vhost-net, and then the application can fill in more bits that it cares about. Or if we let virttuned go first, setting virtio-net model + vhost-net, then application wants to change model to e1000e, it has to be aware that it must now delete the vhost-net bit that virtuned added. This ends up being more complicated that just ignoring virttuned and coding up use of vhost-net in application code.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on
virt-manager's
toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
The initial idea (which I forgot to mention) was that all the decisions libvirt currently does (so that it keeps the guest ABI stable) would be moved into data (let's say some DSL) and it could then be switched or adjusted if that's not what the mgmt app wants (on a per-definition basis, of course). I didn't feel very optimistic about the upstream acceptance for that idea, so I figured that there could be something that lives beside libvirt, helps with some policies if requested and then the resulting XML could be fed into libvirt for determining the rest.
I can't even imagine how we would go about encoding the stable guest ABI logic libvirt does today in data !
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the
By native you mean actual binary libraries or native to the OpenStack code as in python module? Because what I had in mind for this project was a python module with optional wrapper for REST API.
I meant native binary libraries. ie openstack is not happy in general with adding dependancies on new OS services, because there's a big time lag for getting them into all distros. By comparison a pure python library, they can just handle automatically in their deployment tools, just pip installing on any OS distro straight from pypi. This is what made use of libosinfo a hard sell in Nova.
The same thing is seen with Go / Rust where some applications have decided they're better of actually re-implementing the libvirt RPC protocol in Go / Rust rather than use the libvirt.so client. I think this is a bad tradeoff in general, but I can see why they like it
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/ dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dber range :| _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
cockpit-devel@lists.fedorahosted.org
-
Daniel P. Berrangé -
Eduardo Habkost -
Martin Kletzander -
Michal Skrivanek -
Pavel Hrdina -
Yaniv Lavi