A modest proposal: Pungi 4 compose process (what we call composes, when we do them, what information we need about them)
Hi folks!
So I've been working lately on revamping the release validation process for Pungi 4 composes. I've made quite a bit of progress, but I'm now kind of stuck, because we don't know how the full release cycle is actually going to work with Pungi 4 composes. There are some questions that haven't been answered:
* What will the compose IDs be for anything other than Rawhide snapshots? * When do we compose what kinds of composes? * What information do we need about the composes and where?
I've asked if there are any plans about this a few times, and the answer has always been "not yet". So I figured instead of sitting around waiting, I'd think the issues through and come up with a proposal!
# tl;dr
(LATER) OK, this email got really long, so here's my tl;dr summary. Proposed compose ID scheme: (RELEASE)-(DATE).(INDEX).(TYPE), e.g. 24-20160401.0.s (types are SNAPSHOT, CANDIDATE, POSTRELEASE). Alternatively: type is stored as separate bit of metadata instead of / as well as in the compose ID.
Proposed additional metadata in PDC: 'nominated' (bool, whether a SNAPSHOT was nominated for manual validation testing or a CANDIDATE nominated as an RC), 'release' (one of a set of consts, 'ALPHA', 'BETA', 'FINAL', 'POSTRELEASE', indicating that a CANDIDATE was released as that milestone).
Rules: all 'override' packages go in all composes, we never build more than one compose type for one release at a time, we switch from SNAPSHOT to CANDIDATE when all blockers are addressed and back to SNAPSHOT after the milestone release, we switch to POSTRELEASE after Final.
Location: we can either have PDC be the canonical store of location information, or just have some kind of engine (whether it's still my 'fedfind' or something else) which can work out where to find a given compose based on all the metadata mentioned in this proposal.
# Full proposal in Epic AdamW Styleee
As a pre-note: I'm really only concerning myself with the "Official Release Process Composes" here, the composes we consider part of the (still) more-or-less monolithic 'release cycle'. I didn't try to think of a design that accounts for separate release cycles per image or product or 'variant' or whatever (because jeez, this is hard enough) and I didn't include any possible side/alternative composes done for testing or whatever. At this point in time I don't care what anyone wants to call or do with those, so long as I can ask PDC for a list of the 'official' composes and just get those.
Here's my sort of sample of an imaginary Fedora 24 release cycle with Pungi 4 composes:
DATE SNAPSHOT CANDIDATE POSTRELEASE MILESTONE 2016-02-28 Rawhide-20160228.0.s
== BRANCH POINT ==
2016-03-01 24-20160301.0.s 24-20160301.1.s
2016-03-02 24-20160302.0.s
== ALPHA FREEZE ==
2016-03-03 24-20160303.0.s 2016-03-08 24-20160308.0.s
== ALL BLOCKERS ADDRESSED: SWITCH TO RCs ==
2016-03-08 24-20160308.1.c (24_Alpha_RC1) 2016-03-08 24-20160308.2.c (24_Alpha_RC2) = 24 Alpha
2016-03-09 24-20160309.0.s
== ALPHA UPDATES PUSHED STABLE ==
2016-03-10 24-20160310.0.s
== BETA FREEZE == == ALL BLOCKERS ADDRESSED ==
2016-04-09 24-20160409.0.c (24 Beta RC1) = 24 Beta
2016-04-11 24-20160411.0.s
== FINAL FREEZE == == ALL BLOCKERS ADDRESSED ==
2016-05-10 24-20160510.0.c (24 Final RC1) 2016-05-11 24-20160511.0.c (24 Final RC2) = 24 Final
== FINAL RELEASE ==
2016-05-25 24-20160525.0.p 2016-05-26 24-20160525.1.p = 24 Postrelease ("2 Week" or whatever)
Obviously, that's just extracts to highlight the interesting points. I mapped this out a few different ways, but this is the one I liked best.
The basic ideas here are pretty simple. The naming scheme for composes is:
(RELEASE)-(DATE).(INDEX).(TYPE)
The compose 'types' are SNAPSHOT, CANDIDATE, and POSTRELEASE. Their shortenings for the compose IDs are 's', 'c' and 'p'. (These don't sort "correctly" alphabetically, but that shouldn't be a problem). This is similar to the scheme currently used for snapshots, but with a type identifier after the index number (I don't know if '.n.' in the current IDs is supposed to indicate "nightly" or "number" or what, but if we want to indicate the type in the compose ID, it makes much more sense to have it after the index than before).
Importantly, the compose IDs for a given release sort into their release order. The only potential issue is if we have more than 9 of a compose type on a day. To deal with that we could just make the index two digits instead of one, or it's relatively easy to do a numeric sort instead (just filter all the non-digit characters and do a numeric sort on the rest).
Note that we don't really *need* to indicate the compose 'type' in the ID. We could instead just have it in the compose metadata. I don't care strongly either way, though I think it's maybe slightly more convenient to have it in the ID. Note it should probably be specified separately at least in PDC even if it's also indicated in the compose ID; it's going to be important to be able to say "I want to find all release 'foo' composes of type 'bar'").
Rawhide is a release (we do not assign release numbers to Rawhide composes). This is something Dennis and I agree on, and convinced the Pungi / productmd folks on.
We do not ever do two types of compose simultaneously; we're not doing SNAPSHOT composes while we do CANDIDATE composes. At first I kinda envisaged this happening, but I don't think it's *necessary*, and it makes ordering difficult if it happens. We *always* increment the index when doing another compose on the same day, even if we're switching type (note we go from 20160308.0.s to 20160308.1.c).
We switch from 'SNAPSHOT' to 'CANDIDATE' composes for each milestone when all blocker bugs are addressed, just as we do now. We then switch back to 'SNAPSHOT' composes after the release of the milestone.
We switch to 'POSTRELEASE' composes after final release, of course. This is my attempt to include the current "2 Week Atomic" system in the process, and I suspect we're only likely to have more desire for "post release" composes in future.
This is not visible in the mockup, but: the only difference between 'SNAPSHOT' and 'CANDIDATE' composes besides the identifier is that 'CANDIDATE' composes have any switches that we currently flip for RCs applied. If there aren't actually any such settings besides the ones specific to Final (where we disable all the "this is a prerelease" warnings), we could potentially even only have 'SNAPSHOT' composes up until Final (Alpha and Beta could simply be blessed 'SNAPSHOT's).
Also not visible in the mockup: "compose override" packages are *always included in all types of compose*. This is the concept Dennis and I came up with for handling blocker / freeze exception fixes; it's just a more formal version of the current process, really, whereby we mark packages that should be pulled into composes. At present these are only pulled into TCs and RCs, they never appear in the old-style "nightly composes". I believe we should *always* pull them in; it makes the system a good deal simpler.
Another important topic is what data we need to store somewhere to aid things that need to interact with the compose process. I'm assuming we are going to store all necessary metadata beyond what can be a part of the compose metadata itself in PDC. I think we can make what PDC needs to store quite minimal. It only needs a couple of extra attributes beyond the compose ID and the 'type' (which should be a searchable attribute in PDC even if it's indicated in the compose ID):
'nominated' - bool 'release' - const (from 'ALPHA', 'BETA', 'FINAL', 'POSTRELEASE')
'nominated' is definitely needed to indicate that a snapshot compose was "nominated" for manual release validation testing. This is something we already do; at present the wiki is the canonical source of information on what composes have been nominated for testing, but I think this is silly. It should be in PDC.
There's also another issue we could use 'nominated' to answer. That is: when exactly do we build 'CANDIDATES'? Do we follow the current process and build them only on manual request, meaning that effectively every 'CANDIDATE' is equivalent to a current RC? Or do we build a 'CANDIDATE', say, *every time the "compose overrides" set changes*, and then 'nominate' RCs from the larger set of CANDIDATEs? If we want to do that, then the 'nominated' attribute for CANDIDATE composes would indicate which were selected as RCs.
'release' indicates that a CANDIDATE (or POSTRELEASE) compose was promoted as a public 'release'. This is something the compose metadata cannot possibly reflect, since we do not know it when the compose is created. PDC is the logical source of such information. The set of valid values for this attribute can be made as large as we want if we start doing stuff like staggering "releases" for different variants, or doing different types of post-stable "release" than "two week atomic".
That's pretty much the entire system. I had thought about things like storing compose "identifiers" like RC2, RC3 etc. in the compose metadata or in PDC directly, and stuff like requiring PDC to construct and store "sequences" of releases. But with this design, I don't *think* any of that is necessary. I believe the constraints specified in the proposal and the information in the compose IDs and the extra PDC fields is actually sufficient to all the tooling purposes I can think of. The idea is that tools can simply query PDC for groups of composes and apply logic to construct certain ideas.
For instance: say we decided we're going to build CANDIDATEs for every change to "compose overrides", and we now want to "nominate" an RC. We can just ask PDC for all CANDIDATEs for the current release which have been "nominated" so far, and it's trivial to produce the sequence of RC names from that and determine what ours should be. (To spot the milestone changes you just look for the composes which also have the 'release' attribute). So the releng tool to stage the CANDIDATE as an RC and the QA tool to create wiki pages can easily produce a nice "RC name" for humans, if we want to do that.
You can use a similar approach for doing various "previous release" comparisons or data analysis across a series of composes; all useful series can I think be derived from the attributes suggested above. Whether we want to use "alternative" names like '24 Final RC2' at all, or just always use the real compose IDs, is a question, but not one we need to settle here.
Similarly tools like fedfind that want to let the user do stuff like "find me Fedora 24" can translate easily - just ask PDC for the Fedora 24 compose with the 'release' attribute "FINAL".
The final question is the question of location. I don't know for sure what the plans are here, but my guess is this:
1. All composes will land on one server first of all - kojipkgs or wherever - in a location that can be determined based on their compose ID
2. SNAPSHOT composes likely will never be found anywhere else.
3. CANDIDATE composes may be staged to two other places: alt (as RCs) and the public mirrors (as releases).
4. POSTRELEASE composes may be staged to one other place: the public mirrors (if blessed as 'releases', whatever we mean by that at any given time). I guess we may wind up having different public locations for different types of release.
I would *like* it if we built the necessary bits such that whenever a release is staged somewhere, that information is transmitted to PDC, so that you can always just ask PDC "what is the canonical location of this compose at present?" and it would give you an answer something like "/fedora/linux/releases/24 in the mirror tree", or an absolute location for alt or kojipkgs. But we *could*, I guess, just have tools keep doing what fedfind does for this: know "the rules" about where to go and find a compose with particular attributes. That's not the ickiest bit of fedfind, and if I have to keep maintaining it (or we have to build some extra service that knows those rules and gives answers on request), it's not the end of the world.
Whew, well that wound up longer than I expected, but I think the core of the proposal is quite simple! Thoughts?
On Sat, 2016-02-20 at 15:33 -0800, Adam Williamson wrote:
As a pre-note: I'm really only concerning myself with the "Official Release Process Composes" here, the composes we consider part of the (still) more-or-less monolithic 'release cycle'. I didn't try to think of a design that accounts for separate release cycles per image or product or 'variant' or whatever (because jeez, this is hard enough) and I didn't include any possible side/alternative composes done for testing or whatever. At this point in time I don't care what anyone wants to call or do with those, so long as I can ask PDC for a list of the 'official' composes and just get those.
You know...writing this pre-note made me think "hmm, maybe this is wrong, and I should design a system that *does* think about those things". So now I'm stewing on that. But before writing another big novel, let's see what people think of this.
Still, a quick preview:
1. maybe "all the composes that are part of the full-fat Fedora release sequence" is just *one* 'type' of compose, and we should just keep building them and name them by date and index number, and instead of "type" for those composes we need info like "were all blockers addressed when this compose was built?" and we make decisions about human-readable names and blessing releases and creating test events based on *those* properties, rather than conceptually bundling them up into an erroneous "type" notion.
2. maybe we extend this notion so we store information like "what repositories were enabled for this build" and "what images did we attempt to build"; any bundling of those things gets done at the *input* end, as a sort of 'compose profile', and we use logic based on the essential properties of the output to decide what we might want to do with the result...
so we might have the information for a given compose "all images targeted, package override repository enabled, all blocker bugs addressed", we can determine "did all blocking images successfully build?", we either include the information "Alpha milestone freeze passed" or we can go get it from the schedule API, maybe QA feeds into PDC "all required validation tests passed or waived", then our rule engine can look at that build and say, hey, this is the Alpha release!
thinking about it in terms of qualitative (rather than arbitrary) properties of composes, and rules engines for imposing actions and definitions onto composes based on those properties, gives us a whole lot more flexibility to do stuff like "all Atomic blocking images built successfully, test systems report (required test set) passed, it's been 2 weeks since last Two Week Atomic release - this gets blessed as a Two Week Atomic Release!", and pretty much anything else we want to do. We just have to store the attributes and build the rules engines.
I think the *hardest* question then becomes "what do we set as the compose ID?" - perhaps we should simply assign literally a serial number to each compose, no matter what set of repositories or whatever it's built from, and use rules to create more friendly names based on the compose's attributes at the point where the compose gets staged out from the initial pool where we stick *all* the composes?
OK, I should stop now. :)
# tl;dr
(LATER) OK, this email got really long, so here's my tl;dr summary. Proposed compose ID scheme: (RELEASE)-(DATE).(INDEX).(TYPE), e.g. 24-20160401.0.s (types are SNAPSHOT, CANDIDATE, POSTRELEASE). Alternatively: type is stored as separate bit of metadata instead of / as well as in the compose ID.
Proposed additional metadata in PDC: 'nominated' (bool, whether a SNAPSHOT was nominated for manual validation testing or a CANDIDATE nominated as an RC), 'release' (one of a set of consts, 'ALPHA', 'BETA', 'FINAL', 'POSTRELEASE', indicating that a CANDIDATE was released as that milestone).
Rules: all 'override' packages go in all composes, we never build more than one compose type for one release at a time, we switch from SNAPSHOT to CANDIDATE when all blockers are addressed and back to SNAPSHOT after the milestone release, we switch to POSTRELEASE after Final.
Location: we can either have PDC be the canonical store of location information, or just have some kind of engine (whether it's still my 'fedfind' or something else) which can work out where to find a given compose based on all the metadata mentioned in this proposal.
I like the proposal, great job, thanks. Minor questions below.
The basic ideas here are pretty simple. The naming scheme for composes is:
(RELEASE)-(DATE).(INDEX).(TYPE)
The compose 'types' are SNAPSHOT, CANDIDATE, and POSTRELEASE. Their shortenings for the compose IDs are 's', 'c' and 'p'. (These don't sort "correctly" alphabetically, but that shouldn't be a problem). This is similar to the scheme currently used for snapshots, but with a type identifier after the index number (I don't know if '.n.' in the current IDs is supposed to indicate "nightly" or "number" or what, but if we want to indicate the type in the compose ID, it makes much more sense to have it after the index than before).
Importantly, the compose IDs for a given release sort into their release order. The only potential issue is if we have more than 9 of a
More than 10, the index starts from 0.
compose type on a day. To deal with that we could just make the index two digits instead of one,
Can we easily adjust the numbering scheme if it ever happens that we need more than 10 composes per day? I wouldn't do it right away since it doesn't seem likely we would need it, but if we ever hit that point, we should make sure it's fairly simply to change it even during our release cycle.
or it's relatively easy to do a numeric sort instead (just filter all the non-digit characters and do a numeric sort on the rest).
I don't follow you here. If we have 24-20160510.2.c 24-20160510.10.c Then after filtering out non-digit characters we have: 24201605102 242016051010 which still sorts incorrectly: 242016051010 24201605102
Unless you meant to parse the compose index out and then do a numeric sort just on that. That works. If we don't intend to bump up number of digits (at least potentially), we should document in PDC docs that the index is supposed to be sorted numerically, that can save us troubles in some tools in the future.
Note that we don't really *need* to indicate the compose 'type' in the ID. We could instead just have it in the compose metadata. I don't care strongly either way, though I think it's maybe slightly more convenient to have it in the ID.
Originally I wanted to say I like it, because it's then easy to go through the list of all the composes and still see where Alpha candidates started (if you roughly know the date) and which was the final Alpha compose (the last candidate). But then I realized that doesn't have to be true, the last candidate doesn't have to be the one accepted for release. So I don't know if we're not making it easy with the TYPE tag to do the same mistake by other people. (OTOH, the same problem is with our usual Alpha RCx names, the last one doesn't have to be the release one).
Still, it seems to help humans, and we should document how to do it properly with "automata".
Also not visible in the mockup: "compose override" packages are *always included in all types of compose*. This is the concept Dennis and I came up with for handling blocker / freeze exception fixes; it's just a more formal version of the current process, really, whereby we mark packages that should be pulled into composes. At present these are only pulled into TCs and RCs, they never appear in the old-style "nightly composes". I believe we should *always* pull them in; it makes the system a good deal simpler.
I'm a bit confused here. The override packages were used for TCs and RCs, but not for Live nightlies done in Koji. Pungi4 will now do all of that as part of a single compose, daily, right? So are you just saying those override packages will end up used in all compose artifacts produced, and we no longer need to care about the TC+RC vs Live nightly difference? In that case that's great.
There's also another issue we could use 'nominated' to answer. That is: when exactly do we build 'CANDIDATES'? Do we follow the current process and build them only on manual request, meaning that effectively every 'CANDIDATE' is equivalent to a current RC? Or do we build a 'CANDIDATE', say, *every time the "compose overrides" set changes*, and then 'nominate' RCs from the larger set of CANDIDATEs? If we want to do that, then the 'nominated' attribute for CANDIDATE composes would indicate which were selected as RCs.
I like the latter. More automation, less manual work for releng.
That's pretty much the entire system. I had thought about things like storing compose "identifiers" like RC2, RC3 etc. in the compose metadata or in PDC directly, and stuff like requiring PDC to construct and store "sequences" of releases. But with this design, I don't *think* any of that is necessary. I believe the constraints specified in the proposal and the information in the compose IDs and the extra PDC fields is actually sufficient to all the tooling purposes I can think of. The idea is that tools can simply query PDC for groups of composes and apply logic to construct certain ideas.
For instance: say we decided we're going to build CANDIDATEs for every change to "compose overrides", and we now want to "nominate" an RC. We can just ask PDC for all CANDIDATEs for the current release which have been "nominated" so far, and it's trivial to produce the sequence of RC names from that and determine what ours should be. (To spot the milestone changes you just look for the composes which also have the 'release' attribute). So the releng tool to stage the CANDIDATE as an RC and the QA tool to create wiki pages can easily produce a nice "RC name" for humans, if we want to do that.
For 24-20160510.0.c, it seem quite natural to me to talk about it as "the candidate from May 10". So even if we scratch "RC2", it doesn't seem as a big deal to me. The only issue I see is with 0-based index numbering, because for 24-20160510.2.c many people will say "the second candidate from May 10", which it is not, and confusion will arise. You did a similar mistake above when counting the number of possible single-digit-index composes (and I realized it only when writing this paragraph). Human mind can be deceiving. Should we start indexing from 1, if we were to abandon "RCx" names? (Despite all that I have to say I really like the zero there, it makes the date more readable, zeroes are somehow clutterless :)).
On Mon, 2016-02-22 at 08:09 -0500, Kamil Paral wrote:
or it's relatively easy to do a numeric sort instead (just filter all the non-digit characters and do a numeric sort on the rest).
I don't follow you here. If we have 24-20160510.2.c 24-20160510.10.c Then after filtering out non-digit characters we have: 24201605102 242016051010 which still sorts incorrectly: 242016051010 24201605102
Unless you meant to parse the compose index out and then do a numeric sort just on that. That works.
Sorry, yes, that's what I meant. Didn't explain it very well.
Note that we don't really *need* to indicate the compose 'type' in the ID. We could instead just have it in the compose metadata. I don't care strongly either way, though I think it's maybe slightly more convenient to have it in the ID.
Originally I wanted to say I like it, because it's then easy to go through the list of all the composes and still see where Alpha candidates started (if you roughly know the date) and which was the final Alpha compose (the last candidate). But then I realized that doesn't have to be true, the last candidate doesn't have to be the one accepted for release. So I don't know if we're not making it easy with the TYPE tag to do the same mistake by other people. (OTOH, the same problem is with our usual Alpha RCx names, the last one doesn't have to be the release one).
Still, it seems to help humans, and we should document how to do it properly with "automata".
That's a good point, yeah. Another reason we have to explicitly track the candidate selected as the release somewhere.
Also not visible in the mockup: "compose override" packages are *always included in all types of compose*. This is the concept Dennis and I came up with for handling blocker / freeze exception fixes; it's just a more formal version of the current process, really, whereby we mark packages that should be pulled into composes. At present these are only pulled into TCs and RCs, they never appear in the old-style "nightly composes". I believe we should *always* pull them in; it makes the system a good deal simpler.
I'm a bit confused here. The override packages were used for TCs and RCs, but not for Live nightlies done in Koji. Pungi4 will now do all of that as part of a single compose, daily, right? So are you just saying those override packages will end up used in all compose artifacts produced, and we no longer need to care about the TC+RC vs Live nightly difference? In that case that's great.
Yes, that is what I'm suggesting. If we still have the difference, it makes things quite messy primarily for constructing an "order" of composes, because say we do a snapshot the morning after an RC is blessed; if we haven't done the stable push by then, the snapshot will inarguably have been built *later* than the RC, but will contain older packages than it. So what's the order? This is how things work ATM, and I kinda hate it. And it just makes sense to me that "compose override" packages should wind up in all the composes, anyhow. There's no reason to leave them out of snapshots.
There's also another issue we could use 'nominated' to answer. That is: when exactly do we build 'CANDIDATES'? Do we follow the current process and build them only on manual request, meaning that effectively every 'CANDIDATE' is equivalent to a current RC? Or do we build a 'CANDIDATE', say, *every time the "compose overrides" set changes*, and then 'nominate' RCs from the larger set of CANDIDATEs? If we want to do that, then the 'nominated' attribute for CANDIDATE composes would indicate which were selected as RCs.
I like the latter. More automation, less manual work for releng.
I tend to agree. I guess I should make it explicit here that I'm kind of expecting releng, QA and any other relevant teams to be building tooling around this which causes the relevant actions to happen either automatically based on fedmsg's, or manually triggered. And I was mostly considering the "do it automatically" case. i.e., my driving consideration here was "how can we make it so every time a compose appears, the whole chain of appropriate actions kicks off and runs entirely automatically" - releng tools decide whether to stage the compose somewhere else, openQA etc. fire up and start testing, relval (or in future something less dumb) decides whether to create a manual validation "event", etc etc.
That's pretty much the entire system. I had thought about things like storing compose "identifiers" like RC2, RC3 etc. in the compose metadata or in PDC directly, and stuff like requiring PDC to construct and store "sequences" of releases. But with this design, I don't *think* any of that is necessary. I believe the constraints specified in the proposal and the information in the compose IDs and the extra PDC fields is actually sufficient to all the tooling purposes I can think of. The idea is that tools can simply query PDC for groups of composes and apply logic to construct certain ideas.
For instance: say we decided we're going to build CANDIDATEs for every change to "compose overrides", and we now want to "nominate" an RC. We can just ask PDC for all CANDIDATEs for the current release which have been "nominated" so far, and it's trivial to produce the sequence of RC names from that and determine what ours should be. (To spot the milestone changes you just look for the composes which also have the 'release' attribute). So the releng tool to stage the CANDIDATE as an RC and the QA tool to create wiki pages can easily produce a nice "RC name" for humans, if we want to do that.
For 24-20160510.0.c, it seem quite natural to me to talk about it as "the candidate from May 10". So even if we scratch "RC2", it doesn't seem as a big deal to me. The only issue I see is with 0-based index numbering, because for 24-20160510.2.c many people will say "the second candidate from May 10", which it is not, and confusion will arise. You did a similar mistake above when counting the number of possible single-digit-index composes (and I realized it only when writing this paragraph). Human mind can be deceiving. Should we start indexing from 1, if we were to abandon "RCx" names? (Despite all that I have to say I really like the zero there, it makes the date more readable, zeroes are somehow clutterless :)).
Well, that's a classic problem: 1s work better for humans, 0s work better for computers (starting at 1 makes humans less likely to make more mistakes, but makes us more likely to make mistakes in code because computers tend to start counting from 0 so everywhere we do indexing we have to remember to add 1, more or less). For me it probably depends how much importance we as humans place on the compose IDs.
That's the big problem I've been struggling with for the entire weekend, in fact! Seriously: naming is the hardest thing. I was out snowboarding on Saturday and kept nearly running into trees because I was too busy thinking about what a compose ID should be, what we ought to do with it, and suchlike concerns.
The more I think about it the more I tend to like my general proposal from the follow-up to this email, which is that we should build the tooling to think about the *essential properties* of composes. This is in specific contrast to building the tooling to work on *constructed* properties like the 'SNAPSHOT', 'CANDIDATE' and 'POSTRELEASE' concept I came up with above.
See, look at it like this: What do 'SNAPSHOT', 'CANDIDATE' and 'POSTRELEASE' really mean? They're what I'm currently thinking of as "synthetic properties", which really stand for a bunch of essential properties reduced in a particular conceptual way. The problem with this is that the way we bundle the properties up is *innately tied to our current release process*. The more I think about it, the more I'm concerned that by baking those into the compose IDs we're essentially baking certain properties of the current release process in at too low a level.
So say we look at SNAPSHOT, CANDIDATE and POSTRELEASE composes. The essential property that differs between them is really only *when* in the (current) release cycle they're built: SNAPSHOTs before the relevant freeze point is reached or all blockers for it are addressed, CANDIDATEs after the freeze point for a milestone is reached and all blockers addressed, POSTRELEASE after the "final" release is done. But they all kind of *assume* a few essential properties that aren't stated:
i) built from the repositories and compose metadata appropriate to the current 'main sequence' release process ii) targeting all the images in current 'main sequence' release composes iii) for the current primary arches
and the whole scheme for naming them is quite innately tied to *the current monolithic release process*. So by adopting this naming scheme, we're kind of inadvertently encoding an awful lot of things about the current release process right into the compose IDs, which I suspect may be a mistake.
Even the *release number* is really a synthetic property: "this is a Fedora 24 compose" is only a valid concept based on the system where we make regular numbered releases of Fedora (and a single sequence of increasingly-numbered releases, at that). If we bake that into the compose ID, then what if we switched to a rolling release model? What 'release number' do we use for composes from that point onwards? Just hardcode one and have it sitting there uselessly in the compose ID forever more? Change the scheme?
The compose ID's job, ultimately, is to be just that: an *identifier*. To let us uniquely identify a particular compose, mainly in order to ask other tools for information about it or to do something with it. The more I think about it the less I like the idea of encoding properties of the release in it, especially these *synthetic* properties that may become incorrect or irrelevant. So I start wanting to simplify more and more. The current scheme for Rawhide composes is actually not bad, because it only encodes three properties, and two of those are 'essential':
i) Release number ii) Compose date iii) Index relevant to other composes on the same date
My suggestion to just give each compose a numerical compose ID which increments by 1 every time we do a compose - *any* compose - is not very different from just doing "YYYYMMDD.(index)" or something similar. It's just that the latter kinda arbitrarily selects "date of compose" as an 'essential property' that's indicated by the compose ID instead of / as well as metadata (or PDC).
So yeah: on second thoughts, I don't think it's a good idea to construct and denote 'synthetic properties' in the compose ID.
It is significantly *less* of a problem to do it in the compose metadata / PDC, because - bluntly - you can always ignore them. It just becomes a question of what's the best design: do we aim only to store 'essential' properties in compose metadata / PDC, and construct 'synthetic' properties like "this could be an RC" above that level, or do we build the logic for constructing such properties into the compose process or PDC and store them there? I don't know for sure, but I think it's essential that we understand that when we talk about "candidates" and "types" and even "release numbers" *that's what we're doing* and always keep it in mind when we build stuff, and build anything which constructs or uses such "synthetic properties" with the possibility that those things may change in mind.
To take a concrete example: where I started with all this was updating wikitcms to work with Pungi 4 composes. I cheerfully started in on a plan to just change the validation page names a bit to include the 'new' compose IDs I originally proposed, or perform a fairly simple translation from the compose ID to a Wikitcms "Fedora 24 Final RC3"- type name. Then I started thinking harder, and it all got a bit...squishier...
My current planned approach is to treat compose IDs as an arbitrary identifier, and conceive part of wikitcms's role as being to analyze the essential properties of a compose and decide:
i) Is there a manual validation test event for this compose? If not, should there be, and... ii) What do we call it?
To circle back to where I came in, ii) is by far the hardest part of this. =)
When you look at it in these terms, python-wikitcms and fedfind have been doing a kind of interesting job: constructing a system for naming composes - a 'compose ID' by any other name - and maintaining a mapping between compose names and release validation event names. A property of Wikitcms' design that I had not previously considered is this: it expects there to be a predictable relationship between the compose name and the release validation event name. To put it simply - we expect to be able to parse the name of a release validation testing page, and from that information alone, identify and locate the compose it is for.
This hasn't been a problem so far, because I actually built Wikitcms and fedfind kinda backwards - their conception of how composes should be identified is actually derived directly from how we name release validation events. :P In A World Where my tools don't control the compose naming concept, and for all the reasons stated above we might want to give composes very uninformative 'names', this doesn't work any more.
So now I'm thinking about how and *where* to store the information "manual validation test event X is for compose Y" - both in our *current* (stupid) manual validation test system Wikitcms, and how we ought to do it in the *sane* manual validation test system we've been thinking about building lately. And in the latter case, how it should be done in such a way that it is extensible to cases where we might want to produce very *different* "test events" for different types of compose. Fun stuff!
On Mon, 2016-02-22 at 10:04 -0800, Adam Williamson wrote:
This hasn't been a problem so far, because I actually built Wikitcms and fedfind kinda backwards - their conception of how composes should be identified is actually derived directly from how we name release validation events. :P In A World Where my tools don't control the compose naming concept, and for all the reasons stated above we might want to give composes very uninformative 'names', this doesn't work any more.
Sorry, I should expand on this. Technically speaking, of course, it still "works". We *can* still do it. The problem is that a page name like:
Test Results: Fedora 20160401.n.2 Installation
isn't great for humans; it gives you no idea of the meaning of that compose, what testing process it's actually a part of. Even worse would be:
Test Results: Fedora 864356 Installation
to which the human response is...huh.
In a *sane* manual testing system, of course, the compose ID would simply be an internal property that a certain set of validation tests could be mapped to in all sorts of exciting and interesting ways, you can put all kinds of abstraction in there to handle all the possibilities in terms of how we want to relate test instances to composes and what we want to be able to show the user in the UI. But it's a problem in the current Wikitcms design because the page name is where we maintain the mapping between the compose and the event as things stand. (There are various ways I can fix this, all of them fitting in nicely with Wikitcms' general level of stupidity. :>)
On Mon, 2016-02-22 at 10:04 -0800, Adam Williamson wrote:
Also not visible in the mockup: "compose override" packages are *always included in all types of compose*. This is the concept Dennis and I came up with for handling blocker / freeze exception fixes; it's just a more formal version of the current process, really, whereby we mark packages that should be pulled into composes. At present these are only pulled into TCs and RCs, they never appear in the old-style "nightly composes". I believe we should *always* pull them in; it makes the system a good deal simpler.
I'm a bit confused here. The override packages were used for TCs and RCs, but not for Live nightlies done in Koji. Pungi4 will now do all of that as part of a single compose, daily, right? So are you just saying those override packages will end up used in all compose artifacts produced, and we no longer need to care about the TC+RC vs Live nightly difference? In that case that's great.
Yes, that is what I'm suggesting. If we still have the difference, it makes things quite messy primarily for constructing an "order" of composes, because say we do a snapshot the morning after an RC is blessed; if we haven't done the stable push by then, the snapshot will inarguably have been built *later* than the RC, but will contain older packages than it. So what's the order? This is how things work ATM, and I kinda hate it. And it just makes sense to me that "compose override" packages should wind up in all the composes, anyhow. There's no reason to leave them out of snapshots.
So Dennis explained that, unfortunately, we can't do this at least for now.
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
In theory we could consider putting the "compose override" packages into the images but not the repositories, but we might still think that was a mistake, and in any case, Dennis says, the tools just don't work like that at present. The images are built from the packages in the repos; any "override" packages for a compose *must* be in the repositories for that compose, practically speaking.
So for the medium term I think we're stuck with including "override" packages in "candidates" but not "snapshots", and things that do stuff with composes are going to have to understand that distinction.
It also means that we're going to have keep doing "snapshots" at the same time as "candidates", as that's how we sync out the stuff that gets pushed stable during freezes to the mirrors.
So we're kinda gonna have to treat the "snapshots" and the "candidates" as two separate streams of things, and accept that we can't entirely reliably construct a sequence of them mixed together in some senses (because a "snapshot" built after a "candidate" may have older packages, just as is the case now).
There are various ways we can think about dealing with this in the long term, but for now at least we'll just have to live with it.
On Mon, 2016-02-22 at 10:04 -0800, Adam Williamson wrote:
Also not visible in the mockup: "compose override" packages are *always included in all types of compose*. This is the concept Dennis and I came up with for handling blocker / freeze exception fixes; it's just a more formal version of the current process, really, whereby we mark packages that should be pulled into composes. At present these are only pulled into TCs and RCs, they never appear in the old-style "nightly composes". I believe we should *always* pull them in; it makes the system a good deal simpler.
I'm a bit confused here. The override packages were used for TCs and RCs, but not for Live nightlies done in Koji. Pungi4 will now do all of that as part of a single compose, daily, right? So are you just saying those override packages will end up used in all compose artifacts produced, and we no longer need to care about the TC+RC vs Live nightly difference? In that case that's great.
Yes, that is what I'm suggesting. If we still have the difference, it makes things quite messy primarily for constructing an "order" of composes, because say we do a snapshot the morning after an RC is blessed; if we haven't done the stable push by then, the snapshot will inarguably have been built *later* than the RC, but will contain older packages than it. So what's the order? This is how things work ATM, and I kinda hate it. And it just makes sense to me that "compose override" packages should wind up in all the composes, anyhow. There's no reason to leave them out of snapshots.
So Dennis explained that, unfortunately, we can't do this at least for now.
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
Could we include the override repo as another directory in the compose tree, and the image building tools would automatically use it if present?
On Tue, 2016-02-23 at 11:05 -0500, Kamil Paral wrote:
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
Could we include the override repo as another directory in the compose tree, and the image building tools would automatically use it if present?
Dennis would be the person to ask that kind of technical detail.
For now I'm gonna basically assume we'll have to keep doing "TCs" and "RCs" at least for F24, though we may think a bit differently about the process of producing them.
On Mon, Feb 22, 2016 at 02:11:10PM -0800, Adam Williamson wrote:
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
On Thu, 2016-02-25 at 10:45 -0500, Matthew Miller wrote:
On Mon, Feb 22, 2016 at 02:11:10PM -0800, Adam Williamson wrote:
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
Overrides are just the new name for what we've been doing so far with the /bleed repo; it's the problem where we can't wait for an update to go through the whole review and stable push process before we include it in a TC/RC, so we shove it in a side repo and include that side repo in the compose.
In some cases we're already pretty sure the package is good and we just want to short-circuit the inherent delays in the Bodhi process; if we had a magic button which instantly made the package show up in the 'stable' repo that'd be fine. So for those cases, sure, it wouldn't be a problem if the overrides got pulled into the 'stable' repo as part of the compose.
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
On Thu, Feb 25, 2016 at 11:34:57AM -0800, Adam Williamson wrote:
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
[...]
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
Well, ugh. Now I understand but am not any happier. :)
What about having _this_ type of compose be a special outside the normal system case, and after testing those separately, go back to the other situation (the short circuit)?
Instead of including overrides in "candidates", include them only in... "override tests", which would only be used to test that the particular override package was not a terrible idea.
Then, the _next_ snapshot could be evaluated / validated in full.
On Thu, 2016-02-25 at 14:57 -0500, Matthew Miller wrote:
On Thu, Feb 25, 2016 at 11:34:57AM -0800, Adam Williamson wrote:
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
[...]
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
Well, ugh. Now I understand but am not any happier. :)
What about having _this_ type of compose be a special outside the normal system case, and after testing those separately, go back to the other situation (the short circuit)?
Sure, we could do that. We'd need a name for the "special outside the normal system" cases, of course. Something something "compose", maybe? Well, we need to "test" it, so we could call it a "test compose"?
waaaaaaitaminute...
Instead of including overrides in "candidates", include them only in... "override tests", which would only be used to test that the particular override package was not a terrible idea.
Then, the _next_ snapshot could be evaluated / validated in full.
OK, I think I see what you're driving at, but then you kinda run into the mechanics of the process. Pungi 4 composes are *faster* but they're still not *fast*, they take a couple of hours at minimum. We do not yet have sufficient automated testing for everything we might want to do an "override test" for, so there needs to be a manual step in there somewhere. And we really don't want to be sneak-pushing things into the "stable" repo behind Bodhi's back, the whole design of the system is that anything in the official /development/(release)/ repos went through Bodhi, so we'd still ultimately need to stuff the things we approved via "override tests" through the Bodhi system, with its delays.
Plus I'm not sure the mechanics of doing one "override test" per blocker/FE-fixing update quite work...I think the way we can actually implement this "override" mechanism is via Koji tags, but Koji tags aren't free, so we can't auto-create a new one for every blocker/FE- fixing update that shows up, I don't think. We'd have to have one tag that we'd be continually applying to the packages from a single update, doing a compose from, scrubbing, applying to the packages from the *next* blocker/FE-fixing update, rinse and repeat...well, I mean, I guess that could work?
So, hum, I guess it's possible. I'm not entirely sure it'd work better than the current system of building candidates with all "override" packages and pushing the overridden packages stable as fast as we can, though. I can see a few advantages and disadvantages...we don't test *interaction* between the overridden updates in this system, and it involves producing and shipping around a *lot* of bits...not sure what Dennis would think.
On Thursday, February 25, 2016 02:57:04 PM Matthew Miller wrote:
On Thu, Feb 25, 2016 at 11:34:57AM -0800, Adam Williamson wrote:
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
[...]
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
Well, ugh. Now I understand but am not any happier. :)
In a world where we only push out bits that have passed openqa we should detect the types of breakages that we wouldn't want to ship. It really boils down to if we put the compose on /pub/fedora or not.
Dennis
What about having _this_ type of compose be a special outside the normal system case, and after testing those separately, go back to the other situation (the short circuit)?
Instead of including overrides in "candidates", include them only in... "override tests", which would only be used to test that the particular override package was not a terrible idea.
Then, the _next_ snapshot could be evaluated / validated in full.
The more I think about it the more I tend to like my general proposal from the follow-up to this email, which is that we should build the tooling to think about the *essential properties* of composes. This is in specific contrast to building the tooling to work on *constructed* properties like the 'SNAPSHOT', 'CANDIDATE' and 'POSTRELEASE' concept I came up with above.
See, look at it like this: What do 'SNAPSHOT', 'CANDIDATE' and 'POSTRELEASE' really mean? They're what I'm currently thinking of as "synthetic properties", which really stand for a bunch of essential properties reduced in a particular conceptual way. The problem with this is that the way we bundle the properties up is *innately tied to our current release process*. The more I think about it, the more I'm concerned that by baking those into the compose IDs we're essentially baking certain properties of the current release process in at too low a level.
So say we look at SNAPSHOT, CANDIDATE and POSTRELEASE composes. The essential property that differs between them is really only *when* in the (current) release cycle they're built: SNAPSHOTs before the relevant freeze point is reached or all blockers for it are addressed, CANDIDATEs after the freeze point for a milestone is reached and all blockers addressed, POSTRELEASE after the "final" release is done. But they all kind of *assume* a few essential properties that aren't stated:
i) built from the repositories and compose metadata appropriate to the current 'main sequence' release process ii) targeting all the images in current 'main sequence' release composes iii) for the current primary arches
and the whole scheme for naming them is quite innately tied to *the current monolithic release process*. So by adopting this naming scheme, we're kind of inadvertently encoding an awful lot of things about the current release process right into the compose IDs, which I suspect may be a mistake.
Even the *release number* is really a synthetic property: "this is a Fedora 24 compose" is only a valid concept based on the system where we make regular numbered releases of Fedora (and a single sequence of increasingly-numbered releases, at that). If we bake that into the compose ID, then what if we switched to a rolling release model? What 'release number' do we use for composes from that point onwards? Just hardcode one and have it sitting there uselessly in the compose ID forever more? Change the scheme?
The compose ID's job, ultimately, is to be just that: an *identifier*. To let us uniquely identify a particular compose, mainly in order to ask other tools for information about it or to do something with it. The more I think about it the less I like the idea of encoding properties of the release in it, especially these *synthetic* properties that may become incorrect or irrelevant. So I start wanting to simplify more and more. The current scheme for Rawhide composes is actually not bad, because it only encodes three properties, and two of those are 'essential':
i) Release number ii) Compose date iii) Index relevant to other composes on the same date
My suggestion to just give each compose a numerical compose ID which increments by 1 every time we do a compose - *any* compose - is not very different from just doing "YYYYMMDD.(index)" or something similar. It's just that the latter kinda arbitrarily selects "date of compose" as an 'essential property' that's indicated by the compose ID instead of / as well as metadata (or PDC).
So yeah: on second thoughts, I don't think it's a good idea to construct and denote 'synthetic properties' in the compose ID.
If you can distill everything above into a 30-seconds explanation how things work, which is easy to understand and doesn't sound like a university paper on nuclear physics merged with philosophy, then maybe :) Because it can be easily done for "20160401.0.c". And often the KISS principle is much better than higher abstraction, which is more universal and "cleaner" but much harder to grasp. It's very possible that I simply did not understand your revised proposal, but "20160401.0.c" seems to strike the balance between simplicity and usefulness to me.
If you mean that we will have a sequentially growing integer number of any kind of compose, and we'll define everything somewhere else (PDC), then it's probably cleaner, but it's harder to use by humans and therefore less useful. Putting the date into the ID doesn't harm anything (it's just a particular type of sequentially growing numbers) and helps human usage a lot. The snapshot type identifier might be more questionable, but still seems more useful than not.
On Wed, 2016-02-24 at 06:43 -0500, Kamil Paral wrote:
If you can distill everything above into a 30-seconds explanation how things work, which is easy to understand and doesn't sound like a university paper on nuclear physics merged with philosophy, then maybe :) Because it can be easily done for "20160401.0.c". And often the KISS principle is much better than higher abstraction, which is more universal and "cleaner" but much harder to grasp. It's very possible that I simply did not understand your revised proposal, but "20160401.0.c" seems to strike the balance between simplicity and usefulness to me.
If you mean that we will have a sequentially growing integer number of any kind of compose, and we'll define everything somewhere else (PDC), then it's probably cleaner, but it's harder to use by humans and therefore less useful.
But the compose has dozens and dozens of properties. Why decide to specify just a couple of them, not in the metadata with all the others, but in the identifier for the compose? On what logical basis? And especially, conceptual properties that might become obsolete, independent of the process of actually producing a bunch of bits?
The problem with the "human use" notion is it's based on a fundamental misconception: that we have to use the compose ID as the compose's name for public consumption. We don't. In fact, the idea that we do is where all the trouble starts.
We can take a compose with the ID "2378" or "afeg634kga" and put it in a directory called "Fedora 24" and send out emails calling it "Fedora 24". All the "compose ID" really needs to be is a unique key for a database.
Tying some quite arbitrary notion of what the compose "is" to its compose ID, and then enforcing that in the tool that builds the compose and assuming that "ID" == "name", comes with two fundamental problems:
i) It ties the - fairly generic - action of "building a compose" unnecessarily to a particular conception of a release process
ii) It makes it unnecessarily difficult to separate out the process of deciding what to do with a compose from actually composing it. It makes it unnecessarily difficult to build a compose, see if it works, run some tests on it, *then* decide it's "Alpha 1". You can still *do* it, I guess, just by ignoring the compose ID as hard as possible - but the current tooling is kind of built on the assumption that the compose ID is deeply meaningful as to the compose's nature.
Putting the date into the ID doesn't harm anything (it's just a particular type of sequentially growing numbers) and helps human usage a lot. The snapshot type identifier might be more questionable, but still seems more useful than not.
It's useful information, it shouldn't be in the compose ID. I wouldn't really mind using the date, I just like suggesting that the ID should be an incrementing integer (or just a random unique string) simply to emphasize the point that I think it's a bad design for the ID to act as a store of (just some) metadata about the compose.
If you mean that we will have a sequentially growing integer number of any kind of compose, and we'll define everything somewhere else (PDC), then it's probably cleaner, but it's harder to use by humans and therefore less useful.
But the compose has dozens and dozens of properties. Why decide to specify just a couple of them, not in the metadata with all the others, but in the identifier for the compose? On what logical basis? And especially, conceptual properties that might become obsolete, independent of the process of actually producing a bunch of bits?
I think the logical basis is to encode as little extra information at the ID as possible, while still make it useful for humans. If you needed to *tell* (not text) your colleague to try a specific compose, would you rather tell him to use compose afeg634kga or 20160225.0? And it's not just about telling people, but often you need to keep some identifiers in mind. If you need to compare today and yesterday's compose, is it easier to run your tools on 20160225.0 and 20160224.0, or afeg634kga and eb96xve458, and remember which is which?
The date component of the ID is unlikely to get ever obsolete. The same for that day's index. I'm not arguing about the compose type identifier, that one very well can be. But hey, what prevents us from changing the ID format if needed? It's just code. It's more about balance, rather than designing something that won't be necessary to change for the next 1000 years.
Tying some quite arbitrary notion of what the compose "is" to its compose ID, and then enforcing that in the tool that builds the compose and assuming that "ID" == "name", comes with two fundamental problems:
i) It ties the - fairly generic - action of "building a compose" unnecessarily to a particular conception of a release process
ii) It makes it unnecessarily difficult to separate out the process of deciding what to do with a compose from actually composing it. It makes it unnecessarily difficult to build a compose, see if it works, run some tests on it, *then* decide it's "Alpha 1". You can still *do* it, I guess, just by ignoring the compose ID as hard as possible - but the current tooling is kind of built on the assumption that the compose ID is deeply meaningful as to the compose's nature.
I agree that it would be very helpful to be able to assign some specific names to certain composes. And also that it's much better to be able to decide about the name (i.e. the meaning) of the compose after it is finished, not before it started. (So e.g. we don't need to say "this is Alpha RC1" when starting the compose only to learn that it failed and we'll end up with Alpha RC3 as the first successful Alpha RC compose. It's much better to compose first, and then be able to say "this is Alpha RC1"). I'm sure there are technical difficulties about this, but if it can be done, great.
But that does not imply to me that we need to keep the compose ID as meaningless as possible.
Putting the date into the ID doesn't harm anything (it's just a particular type of sequentially growing numbers) and helps human usage a lot. The snapshot type identifier might be more questionable, but still seems more useful than not.
It's useful information, it shouldn't be in the compose ID. I wouldn't really mind using the date, I just like suggesting that the ID should be an incrementing integer (or just a random unique string) simply to emphasize the point that I think it's a bad design for the ID to act as a store of (just some) metadata about the compose.
Random strings do not sort in chronological order. Growing numbers and dates do, so they're easier to work with. Dates are more memorable than numbers, so they're easier to work with. But you can still consider them as "meaningless", if you want, because it's just a specific number increment format. You can basically consider it the same as any other growing sequence with a custom format.
On Thu, 2016-02-25 at 06:00 -0500, Kamil Paral wrote:
If you mean that we will have a sequentially growing integer number of any kind of compose, and we'll define everything somewhere else (PDC), then it's probably cleaner, but it's harder to use by humans and therefore less useful.
But the compose has dozens and dozens of properties. Why decide to specify just a couple of them, not in the metadata with all the others, but in the identifier for the compose? On what logical basis? And especially, conceptual properties that might become obsolete, independent of the process of actually producing a bunch of bits?
I think the logical basis is to encode as little extra information at the ID as possible, while still make it useful for humans. If you needed to *tell* (not text) your colleague to try a specific compose, would you rather tell him to use compose afeg634kga or 20160225.0?
That's reasonable, but then an integer isn't that hard to say either. But as I said, I wouldn't really mind using something date-y too much. It's the other bits I really don't like.
And it's not just about telling people, but often you need to keep some identifiers in mind. If you need to compare today and yesterday's compose, is it easier to run your tools on 20160225.0 and 20160224.0, or afeg634kga and eb96xve458, and remember which is which?
Except this doesn't work, because the compose before 20160225.0 is not necessarily 20160224.0, it might be 20160224.3. You just don't know. The proper way to do this is to go ask PDC for the previous completed compose from the same series, and if you're going to do that, it doesn't matter what the format of the identifier is. (Believe me, this is something I'm literally in the middle of writing :>)
The date component of the ID is unlikely to get ever obsolete. The same for that day's index. I'm not arguing about the compose type identifier, that one very well can be. But hey, what prevents us from changing the ID format if needed? It's just code.
Well, it's kinda the entire workflow design I don't like. This is *already* causing a bit of friction because RHEL and Fedora don't necessarily have the same notion of "compose types". We have the same problem with the label, because RHEL and Fedora don't necessarily have the same notion of "milestones" and "respins" and so on.
Sure, we'd still have the conceptual mismatch if these things were metadata properties - but at least we'd just be dealing with handling simple, discrete properties, not different ways to parse concepts out of smooshed-together strings.
It's hard to explain my thoughts about the whole thing without writing another essay, and I don't want to do that, but it's basically about the whole flow from defining what the inputs of a compose are through to deciding what 'labels' to apply to it and where to sync it to. Too many of the things about that process are built into compose tools and awkwardly expressed, when they should be properly separated out into code and configuration and consistently expressed as discrete properties.
It's more about balance, rather than designing something that won't be necessary to change for the next 1000 years.
What I'd like to happen is to make it so that when we *do* change, change can be easily and minimally accomplished. That's really what I'm thinking about here: how hard is to to say, hey, from now on we're going to build a candidate compose every day and have a tool whose job it is to decide whether to call it an "RC" and sync it to the mirrors? How hard is it to say, from now on we're not going to have a "final release" but we're gonna build these four sets of outputs on different schedules and ship each one after some testing?
Right now my belief is that making those kind of changes is unnecessarily difficult because of all the excessive bundling of concepts and processes into the code of pungi, and I don't like that.
Tying some quite arbitrary notion of what the compose "is" to its compose ID, and then enforcing that in the tool that builds the compose and assuming that "ID" == "name", comes with two fundamental problems:
i) It ties the - fairly generic - action of "building a compose" unnecessarily to a particular conception of a release process
ii) It makes it unnecessarily difficult to separate out the process of deciding what to do with a compose from actually composing it. It makes it unnecessarily difficult to build a compose, see if it works, run some tests on it, *then* decide it's "Alpha 1". You can still *do* it, I guess, just by ignoring the compose ID as hard as possible - but the current tooling is kind of built on the assumption that the compose ID is deeply meaningful as to the compose's nature.
I agree that it would be very helpful to be able to assign some specific names to certain composes. And also that it's much better to be able to decide about the name (i.e. the meaning) of the compose after it is finished, not before it started. (So e.g. we don't need to say "this is Alpha RC1" when starting the compose only to learn that it failed and we'll end up with Alpha RC3 as the first successful Alpha RC compose. It's much better to compose first, and then be able to say "this is Alpha RC1"). I'm sure there are technical difficulties about this, but if it can be done, great.
But that does not imply to me that we need to keep the compose ID as meaningless as possible.
That's not really my main *goal* here, it's not like my starting point is "make the compose ID meaningless!" and all the other stuff is just a justification for that. My hope is that the compose process can be more flexible and modular and smarter. The compose ID is really just kind of an avatar for all the features of the current design which make that hard.
Putting the date into the ID doesn't harm anything (it's just a particular type of sequentially growing numbers) and helps human usage a lot. The snapshot type identifier might be more questionable, but still seems more useful than not.
It's useful information, it shouldn't be in the compose ID. I wouldn't really mind using the date, I just like suggesting that the ID should be an incrementing integer (or just a random unique string) simply to emphasize the point that I think it's a bad design for the ID to act as a store of (just some) metadata about the compose.
Random strings do not sort in chronological order. Growing numbers and dates do, so they're easier to work with. Dates are more memorable than numbers, so they're easier to work with. But you can still consider them as "meaningless", if you want, because it's just a specific number increment format. You can basically consider it the same as any other growing sequence with a custom format.
Sure, like I said, it's more that I'm making a rhetorical point. But really, I believe that if we're going to all the trouble of designing a metadata format and providing compose metadata and an entire webapp for letting you *query* compose metadata, the way to sort composes is to get whatever property you want from the metadata and do a sort on that property. "Date of compose" is *already* a metadata property. Why does it makes sense that if I want to sort by any other means I go look in the compose metadata, but if I want to sort by date I say "oh hey I can do that by sorting by compose ID!"? The job of the compose ID is to uniquely identify the compose. It's not the compose ID to sort composes. That's the job of the compose metadata. I'm just trying to promote that way of thinking.
And it's not just about telling people, but often you need to keep some identifiers in mind. If you need to compare today and yesterday's compose, is it easier to run your tools on 20160225.0 and 20160224.0, or afeg634kga and eb96xve458, and remember which is which?
Except this doesn't work, because the compose before 20160225.0 is not necessarily 20160224.0, it might be 20160224.3. You just don't know.
I do, immediately, and that's the point. When I see all the composes in the pungi directory, I can immediately see what is the today's latest one, and I see which ones are yesterday ones. In my example above, I said "compare today's and yesterday's compose", not "today's with the previous one". So I can immediately understand that 20160225.0 is the first compose today, and 20160224.0 is the first compose yesterday. By looking into that dir, I can also immediately see that 20160224.3 is the last compose yesterday. All of that quickly and naturally, without querying PDC. I wouldn't be able to do none of this if we had just growing sequential numbers.
The proper way to do this is to go ask PDC for the previous completed compose from the same series, and if you're going to do that, it doesn't matter what the format of the identifier is.
Yes, machines should do this, definitely. But as a human tester, I'll hate this, because it wastes my time by forcing me to go to PDC to learn even basic trivial properties like "when was this composed?" or "what release is this related to?", every single time I want to work with this.
What I'd like to happen is to make it so that when we *do* change, change can be easily and minimally accomplished. That's really what I'm thinking about here: how hard is to to say, hey, from now on we're going to build a candidate compose every day and have a tool whose job it is to decide whether to call it an "RC" and sync it to the mirrors? How hard is it to say, from now on we're not going to have a "final release" but we're gonna build these four sets of outputs on different schedules and ship each one after some testing?
Right now my belief is that making those kind of changes is unnecessarily difficult because of all the excessive bundling of concepts and processes into the code of pungi, and I don't like that.
Yes, that I completely agree with.
Random strings do not sort in chronological order. Growing numbers and dates do, so they're easier to work with. Dates are more memorable than numbers, so they're easier to work with. But you can still consider them as "meaningless", if you want, because it's just a specific number increment format. You can basically consider it the same as any other growing sequence with a custom format.
Sure, like I said, it's more that I'm making a rhetorical point. But really, I believe that if we're going to all the trouble of designing a metadata format and providing compose metadata and an entire webapp for letting you *query* compose metadata, the way to sort composes is to get whatever property you want from the metadata and do a sort on that property.
Yes, definitely.
"Date of compose" is *already* a metadata property. Why does it makes sense that if I want to sort by any other means I go look in the compose metadata, but if I want to sort by date I say "oh hey I can do that by sorting by compose ID!"?
Because not all of us are machines (you can't hide it anymore!:-)) and for basic operations we don't want to go to PDC. We can get some basic semantics from the ID for free. Of course we must not overdo it.
Machines, on the other hand, should always go to PDC, and not parse the the ID, yes. But if we offer a good API, I believe that is given. And even if they do parse release and date out of the ID for some very simple use cases, that doesn't seem to be a problem, and once we have some problem with that, we can fix those systems to start using PDC instead.
The job of the compose ID is to uniquely identify the compose. It's not the compose ID to sort composes. That's the job of the compose metadata. I'm just trying to promote that way of thinking.
I understand that and I agree with you in general, but let's not go too far.
On Fri, 2016-02-26 at 04:20 -0500, Kamil Paral wrote:
And it's not just about telling people, but often you need to keep some identifiers in mind. If you need to compare today and yesterday's compose, is it easier to run your tools on 20160225.0 and 20160224.0, or afeg634kga and eb96xve458, and remember which is which?
Except this doesn't work, because the compose before 20160225.0 is not necessarily 20160224.0, it might be 20160224.3. You just don't know.
I do, immediately, and that's the point. When I see all the composes in the pungi directory, I can immediately see what is the today's latest one, and I see which ones are yesterday ones.
Oh, but you referred to running tools.
If you want to order the contents of a directory by their creation date, well, that was a problem we solved a long time ago, which is why I don't have to call all my files 20160224-filesize-name , just in case I want to sort them by date and then by file size. ;)
In my example above, I said "compare today's and yesterday's compose", not "today's with the previous one". So I can immediately understand that 20160225.0 is the first compose today, and 20160224.0 is the first compose yesterday.
But with Pungi 4 we keep failed composes around, so you don't really know which of yesterday's composes you want to compare against. The concept of "yesterday's compose" is pretty arbitrary, really - it relies on an assumption that we want there to be one "compose" per day. Even though Pungi 4 calls snapshot builds "nightlies", this is not necessarily true. We want to do distribution CI, remember? We want to be firing off composes more or less every ten minutes, in the ideal case.
By looking into that dir, I can also immediately see that 20160224.3 is the last compose yesterday. All of that quickly and naturally, without querying PDC. I wouldn't be able to do none of this if we had just growing sequential numbers.
Here you're making another assumption, though: composes will be stored in directories named after their CID. Why? It's not even true, as it happens, I don't think. It's true for nightlies, but 'production' builds are output into directories based on their *label*, IIUC, not their cid.
I do, immediately, and that's the point. When I see all the composes in the pungi directory, I can immediately see what is the today's latest one, and I see which ones are yesterday ones.
Oh, but you referred to running tools.
Yes, but I meant manually running them. Diffing two trees of files. Or pointing virt-manager to the first compose of yesterday. This kind of manual usage.
If you want to order the contents of a directory by their creation date, well, that was a problem we solved a long time ago, which is why I don't have to call all my files 20160224-filesize-name , just in case I want to sort them by date and then by file size. ;)
Yes, if you have them locally and all the file properties were kept. If you see the list in a web browser, then it depends on the webpage whether it shows you and allows you these operations.
In my example above, I said "compare today's and yesterday's compose", not "today's with the previous one". So I can immediately understand that 20160225.0 is the first compose today, and 20160224.0 is the first compose yesterday.
But with Pungi 4 we keep failed composes around, so you don't really know which of yesterday's composes you want to compare against.
That's why I would check STATUS file first, if I needed it. If there were 3 composes yesterday I need to find the first successful one, it's much easier for me to see each compose's STATUS file than write a query for PDC.
The concept of "yesterday's compose" is pretty arbitrary, really - it relies on an assumption that we want there to be one "compose" per day. Even though Pungi 4 calls snapshot builds "nightlies", this is not necessarily true. We want to do distribution CI, remember? We want to be firing off composes more or less every ten minutes, in the ideal case.
Not one per day, but a small number. If we do 144 composes per day, I admit that the benefit of having friendly compose IDs begins to vanish. The date part is still very useful, but with so many composes every day, it will get harder to perform any kind of manual inspection/comparison/testing and we'll need to rely more on tooling.
By looking into that dir, I can also immediately see that 20160224.3 is the last compose yesterday. All of that quickly and naturally, without querying PDC. I wouldn't be able to do none of this if we had just growing sequential numbers.
Here you're making another assumption, though: composes will be stored in directories named after their CID. Why? It's not even true, as it happens, I don't think. It's true for nightlies, but 'production' builds are output into directories based on their *label*, IIUC, not their cid.
You got me here. The whole time I expected the compose ID be the same as the directory where it is stored (except for cases when we decide to give it human friendly labels, like Alpha RC2, which is even better). So I was not arguing that much for friendly compose IDs, but rather for friendly directory names. That is more important to me.
On Sat, 2016-02-20 at 15:33 -0800, Adam Williamson wrote:
Hi folks!
So I've been working lately on revamping the release validation process for Pungi 4 composes. I've made quite a bit of progress, but I'm now kind of stuck, because we don't know how the full release cycle is actually going to work with Pungi 4 composes. There are some questions that haven't been answered:
- What will the compose IDs be for anything other than Rawhide snapshots?
- When do we compose what kinds of composes?
- What information do we need about the composes and where?
I've asked if there are any plans about this a few times, and the answer has always been "not yet". So I figured instead of sitting around waiting, I'd think the issues through and come up with a proposal!
So I did a bit of digging around in pungi4 / productmd code today. Unfortunately it didn't really make me that happy :/
Simply put, productmd does all the things I've been thinking are a bad idea (in terms of encapsulating all sorts of specific properties of a particular compose process) and this is broadly tied into pungi4.
I'm looking at stuff like this:
https://pagure.io/pungi/blob/master/f/bin/pungi-koji
that's the executable releng actually has to call to produce a compose. Note that it doesn't let you simply say: hey, I want a compose with ID foo. Oh, no. It requires you to specify '--nightly' to produce a "nightly" compose, or '--test' to produce a "test" compose. If you leave both of those out, you get a "production" compose, and are required to specify a "label" - but you can't just pick your label, because it calls:
productmd.composeinfo.verify_label(opts.label)
so let's go look at that...
https://github.com/release-engineering/productmd/blob/master/productmd/compo...
and, well, ew. Just in this file productmd basically enforces a whole pile of assumptions about a compose process. Really, it's assuming all composes look like RHEL composes. verify_label basically requires that the label looks like this:
(MILESTONE)-(VERSION).(RESPIN)
MILESTONE has to be one of "DevelPhaseExit", "InternalAlpha", "Alpha", "InternalSnapshot", "Beta", "Snapshot", "RC" or "Update". VERSION and RESPIN have to be integers (and I'm pretty sure both have various magical meanings throughout productmd).
There aren't any compose types other than 'production', 'test' and 'nightly', that's all you can build. Each of these is of course a complex construct that doesn't really apply entirely to the Fedora compose process.
So...yeah. We can of course slowly and painstakingly revise all this magic stuff to support Fedora's compose process as well as RHEL's...but I really just don't like the design at all. All this enforcement of metadata being bundled into compose creation makes it unnecessarily difficult to be flexible in the compose process.
It also results in stuff like: you really can't do the thing where you spin up a compose, then decide you want to bless it as, say, an Alpha RC, or the Alpha itself, later. You have to declare ahead of time that this *is* "Alpha 1" or whatever. (And then the whole 'respin' thing feels like a hacky way to deal with the problem where you say "this build is going to be Alpha 1!" and then it explodes into a pile of fiery debris; if 'blessing' was separable from compose creation this would not be a problem at all).
I don't really have a very productive ending to this mail, I guess I just wanted to record what I'd found, and note that it's rather at odds with my happy-clappy view of How The World Ought To Work.
I guess it's useful to know that pungi4/productmd's assumptions about how non-nightly builds will be done and what they fundamentally *are* don't seem terribly compatible with Fedora's current practice; we're either going to have to change them in a hurry, I guess, or start thinking about how we adapt Fedora's processes to the way pungi4 wants to work, or come up with some kinda workaround.
On Tue, 2016-02-23 at 19:09 -0800, Adam Williamson wrote:
On Sat, 2016-02-20 at 15:33 -0800, Adam Williamson wrote:
Hi folks!
So I've been working lately on revamping the release validation process for Pungi 4 composes. I've made quite a bit of progress, but I'm now kind of stuck, because we don't know how the full release cycle is actually going to work with Pungi 4 composes. There are some questions that haven't been answered:
- What will the compose IDs be for anything other than Rawhide snapshots?
- When do we compose what kinds of composes?
- What information do we need about the composes and where?
I've asked if there are any plans about this a few times, and the answer has always been "not yet". So I figured instead of sitting around waiting, I'd think the issues through and come up with a proposal!
So I did a bit of digging around in pungi4 / productmd code today. Unfortunately it didn't really make me that happy :/
Simply put, productmd does all the things I've been thinking are a bad idea (in terms of encapsulating all sorts of specific properties of a particular compose process) and this is broadly tied into pungi4.
I'm looking at stuff like this:
https://pagure.io/pungi/blob/master/f/bin/pungi-koji
that's the executable releng actually has to call to produce a compose. Note that it doesn't let you simply say: hey, I want a compose with ID foo. Oh, no. It requires you to specify '--nightly' to produce a "nightly" compose, or '--test' to produce a "test" compose. If you leave both of those out, you get a "production" compose, and are required to specify a "label" - but you can't just pick your label, because it calls:
productmd.composeinfo.verify_label(opts.label)
Hmm, on second thoughts, I need to take this back a little. I got kinda lost in the code and have only just realized that the "label" and the "compose ID" are separate concepts. At least just looking at the code, I think it works like this. All "compose IDs" follow a similar format:
Fedora-RELEASE-DATE(.TYPE).RESPIN
for nightly composes type is 'n', for test composes it's 't', for production composes there is no type - the part in braces is omitted. So we might have:
Fedora-24-20160224.t.0 (a test compose) Fedora-24-20160224.n.0 (a nightly compose) Fedora-24-20160224.0 (a "production" compose)
which...I mean, it still encodes a bit more information than I'd love, but it's not awful. And it does map quite well to the current Fedora process, where as we've established, we *do* need to treat 'nightly' and 'production' composes - or what I called "SNAPSHOT" and "CANDIDATE" composes - as separate things, at least for now.
the label is a *separate* property which only "production" composes have. Test and nightly composes do not have one. That's the thing that looks something like:
Alpha-1.1
so this actually is kinda implementing the idea that you can decide what a compose "is" in terms of the release process separately from assigning an ID to it...only you're required to do it *at the time you build the compose*. I think it'd be good to separate the "apply a label" step from the "create the compose" step, but conceptually it's not terrible. It shouldn't be incredibly difficult to make productmd support a more Fedora-y label format, if we want to do that.
-
Adam Williamson -
Dennis Gilmore -
Kamil Paral -
Matthew Miller