On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
- Default devices/values
Libvirt itself must default to whatever values there were before any particular element was introduced due to the fact that it strives to keep the guest ABI stable. That means, for example, that it can't just add -vmcoreinfo option (for KASLR support) or magically add the pvpanic device to all QEMU machines, even though it would be useful, as that would change the guest ABI.
For default values this is even more obvious. Let's say someone figures out some "pretty good" default values for various HyperV enlightenment feature tunables. Libvirt can't magically change them, but each one of the projects building on top of it doesn't want to keep that list updated and take care of setting them in every new XML. Some projects don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.
Lets say that you have an initial good set of hyperv config tunables. Now sometime passes and it is decided that there is a different, better set of config tunables. If the module that is providing this policy to apps like OpenStack just updates itself to provide this new policy, this can cause problems with the existing deployed applications in a number of ways.
First the new config probably depends on specific versions of libvirt and QEMU, and you can't mandate to consuming apps which versions they must be using. So you need a matrix of libvirt + QEMU + config option settings.
Even if you have the matching libvirt & QEMU versions, it is not safe to assume the application will want to use the new policy. An application may need live migration compatibility with older versions. Or it may need to retain guaranteed ABI compatibility with the way the VM was previously launched and be using transient guests, generating the XML fresh each time.
The application will have knowledge about when it wants to use new vs old hyperv tunable policy, but exposing that to your policy module is very tricky because it is inherantly application specific logic largely determined by the way the application code is written.
This is a good point and would be definitely an issue. One way how to solve it could be that the virtuned would be able to return you the default policy, the management application would save it somewhere if it cares about reproducing the same XML with the same default policies with newer version of virtuned/libvirt/qemu/kvm/kernel.
The API to create a new domain or hotplugable device would take another optional parameter where you could pass the default policy in addition to application specific policy.
That way you would get the same XML even with newer components with the same benefit that you wound not have to reimplement the logic.
One more thing could be automatically figuring out best values based on libosinfo-provided data.
- Policies
Lot of the time there are parts of the domain definition that need to be added, but nobody really cares about them. Sometimes it's enough to have few templates, another time you might want to have a policy per-scenario and want to combine them in various ways. For example with the data provided by point 1).
For example if you want PCI-Express, you need the q35 machine type, but you don't really want to care about the machine type. Or you want to use SPICE, but you don't want to care about adding QXL.
What if some of these policies could be specified once (using some DSL for example), and used by virtuned to merge them in a unified and predictable way?
- Abstracting the XML
This is probably just usable for stateless apps, but it might happen that some apps don't really want to care about the XML at all. They just want an abstract view of the domain, possibly add/remove a device and that's it. We could do that as well. I can't really tell how much of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all. Any non-trivial application has created an abstraction around XML, so that they have an API to express what they want, rather than manipulating of strings to format/parse XML.
The libvirt-gconfig project aims to provide a C API for manipulating XML documents, with language bindings available via GObject introspection so you can use it from Vala, Perl, Python, JavaScript, etc. Go is notable missing, but for that we have libvirt-go-xml which provides a set of native Go structs to represent the XML.
The problem we've faced with libvirt-gconfig is that it is a really hard sell to get applications to convert existing code to use it. We've only had success where an applicaiton has been written to use libvirt-gconfig from day one - eg GNOME Boxes and libvirt-sandbox.
Virt-manager is the poster-child for using libvirt-gconfig, but I don't see it adopting it any time soon, as it is a massive effort to change all existing code - even if libvirt-gconfig had full XML schema coverge, which it doesn't :-(
I also wanted to use libvirt-gconfig in OpenStack, but there was resistance for adding a dependancy on another native library, so there we've basically copied what virt-manager does and defined a set of pure python objects to represent config which is serialized to / from XML :-( If libvirt-gconfig were to be used by OpenStack it would need to deal with fact that not every distro has that available, so the existing pure python config objects would need to be maintained in parallel for an indefinite amount of time. So in fact using libvirt-gconfig would increase maint burden for OpenStack, rather than reduce it.
libvirt-gconfig would be a hard sell for Go apps when you compare it to libvirt-go-xml, because the latter is following the common Go paradigm for XML manipulation.
If there was something higher level that gets more interesting, but the hard bit is that you still need a way to get at all the low level bits becuase a higher level abstracted API will never cover every niche use case.
- Identifying devices properly
In contrast to the previous point, stateful apps might have a problem identifying devices after hotplug. For example, let's say you don't care about the addresses and leave that up to libvirt. You hotplug a device into the domain and dump the new XML of it. Depending on what type of device it was, you might need to identify it based on different values. It could be <target dev=''/> for disks, <mac address=''/> for interfaces etc. For some devices it might not even be possible and you need to remember the addresses of all the previous devices and then parse them just to identify that one device and then throw them away.
With new enough libvirt you could use the user aliases for that, but turns out it's not that easy to use them properly anyway. Also the aliases won't help users identify that device inside the guest.
NB, relating between host device config and guest visible device config is a massive problem space in its own right, and not very easy to address. In OpenStack we ended up defining a concept of "device tagging" via cloud-init metadata, where openstack allows users to set opaque string tags against devices their VM has. OpenStack that generates a metadata file that records various pieces of identifying hardware attributes (PCI address, MAC addr, disk serial, etc) alongside the user tag. This metadata file is exposed to the guest with the hope that there's enough info to allow the user to decide which device is to be used for which purpose
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-... https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/...
<rant> We really should've gone with new attribute for the user alias instead of using an existing one, given how many problems that is causing. </rant>
- Generating the right XML snippet for device hot-(un)plug
This is kind of related to some previous points.
When hot-plugging a device and creating an XML snippet for it, you want to keep the defaults from point 1) and policies from 2) in mind. Or something related to the already existing domain which you can describe systematically. And adding something for identification (see previous point).
Doing the hot-unplug is easy depending on how much information about that device is saved by your application. The less you save about the device (or show to the user in a GUI, if applicable) the harder it might be to generate an XML that libvirt will accept. Again, some problems with this should be fixed in libvirt, some of them are easy to workaround. But having a common ground that takes care of this should help some projects.
Hot-unplug could be implemented just based on the alias. This is something that would fit into libvirt as well.
========================================================================
To mention some pre-existing solutions:
- I understand OpenStack has some really sensible and wisely chosen
and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's decisions have been largely inspired by previous work in oVirt and / or virt-manager. So there's obviously overlap in the conceptual area, but there's also plenty that is very specific to OpenStack - untangling the two extract the common bits from the app specific bits is hard.
This would be handled by the application specific policies. The virtuned will have some reasonable defaults that are known to work in most cases and suits the majority of users, but it's clear that sometimes you need some specific defaults and you would provide them via the application policy.
For example, to create a XML for windows guest the virtuned would not probably select virtio devices because there are no drivers for them in the standard windows installation, however, some management application may have customized preinstalled disk images or customized ISO images or it may be able to provide the drivers any other way, so they would specify in the application policy that for windows guest virtuned should use virtio as a default device model.
- I know KubeVirt has VirtualMachinePresets. That is something closely
related to points 1) and 2). Also their abstraction of the XML might be usable for point 3).
- There was an effort on creating policy based configuration of libvirt
objects called libvirt-designer. This is closely related to points 2) and 3). Unfortunately there was no much going on lately and part of virt-manager repository has currently more features implemented with the same ideas in mind, just not exported for public use.
This is the same kind of problem we faced wrt libvirt-gconfig and libvirt-gobject usage from virt-manager - it has an extensive code base that already works, and rewriting it to use something new is alot of work for no short-term benefit. libvirt-gconfig/gobject were supposed to be the "easy" bits for virt-manager to adopt, as they don't really include much logic that would step on virt-manager's toes. libvirt-designer was going to be a very opinionated library and in retrospective that makes it even harder to consider adopting it for usage in virt-manager, as it'll have signficant liklihood of making functionally significant changes in behaviour.
There's also the problem with use of native libraries that would impact many apps. We only got OpenStack to grudgingly allow the use of libosinfo native library via GObject Introspection, by promising to do work to turn the osinfo database into an approved stable format which OpenStack could then consume directly, dropping the native API usage :-( Incidentally, the former was done (formal spec for the DB format), but the latter was not yet (direct DB usage by OpenStack)
BTW, I don't like that I'm being so negative to your proposal :-( I used to hope that we would be able to build higher level APIs on top of libvirt to reduce the overlap between different applications reinventing the wheel. Even the simplest bits we tried like the gconfig/gobject API are barely used. libvirt-designer is basically a failure. Though admittedly it didn't have enough development resource applied to make it compelling, in retrospect adoption was always going to be a hard sell except in greenfield developments.
Libosinfo is probably the bit we've had most success with, and has most promise for the future, particularly now that we formally allow apps to read the osinfo database directly and bypass the API. It is quite easy to fit into existing application codebases which helps alot. Even there I'm still disappointed that we only have GNOME Boxes using the kickstart generator part of osinfo - oVirt and Oz both still have their own kickstart generator code for automating OS installs.
In general though, I fear anything API based is going to be a really hard sell to get wide adoption for based on what we've seen before.
I think the biggest bang-for-buck is identifying more areas where we can turn code into data. There's definitely scope for recording more types of information in the osinfo database. There might also be scope for defining entirely new databases to complement the osinfo data, if something looks out of scope for libosinfo.
This is probably the hardest part of creating higher level API on top of libvirt, not every project may be willing to rewrite their existing code. On the other hand, I know that for example Cockpit would benefit from the virtuned providing this functionality via REST API.
It's a chicken and egg problem, but if we can gather input from all the existing projects that have their own implementation and figure out how to make virtuned usable for all of them they might consider to start using it.
Pavel