[F24 Criteria Change] Cockpit Release Criteria

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Recently, we discovered a bug in gnutls that caused Cockpit to be
unreachable by recent versions of Google Chrome. It was ambiguous what
the release criteria actually means, since it didn't specify which
browser applications were blocking. I'd like to propose the following
additional wording for Cockpit criteria:

* All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
 - Mozilla Firefox as shipped in the same Fedora release
 - Mozilla Firefox of the latest available version on Windows at
   compose time.
 - Mozilla Firefox of the latest available version on OSX at compose
   time.
 - Google Chrome of the latest available version on Fedora at compose
   time.
 - Google Chrome of the latest available version on Windows at compose
   time.
 - Google Chrome of the latest available version on OSX at compose time.

Alternately, we could decide that it's only *blocking* if the above
browsers work with Cockpit when the browser is running on Fedora, but
that is somewhat at odds with our reasoning for having a management
console as a web UI in the first place: that it is accessible
regardless of the client system.

Comments welcome, but please keep replies on the
[e-mail] list, as that's where criteria decisions
are made.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iEYEARECAAYFAlYpLFwACgkQeiVVYja6o6NF/wCgg6iot1JKOfmAbTZMboBcPvs5
> ZIIAnA+YxRAjPMt69lqv2nOR7qXnCYnV
> =PIuy
> -----END PGP SIGNATURE-----

[Dennis Gilmore]

On Thursday, October 22, 2015 02:35:11 PM Stephen Gallagher wrote:
> Recently, we discovered a bug in gnutls that caused Cockpit to be
> unreachable by recent versions of Google Chrome. It was ambiguous what
> the release criteria actually means, since it didn't specify which
> browser applications were blocking. I'd like to propose the following
> additional wording for Cockpit criteria:
> All Cockpit functional criteria must be satisfied when the user is
>
> running any of the following blocking browsers:
> Mozilla Firefox as shipped in the same Fedora release
> Mozilla Firefox of the latest available version on Windows at
> compose time.
> Mozilla Firefox of the latest available version on OSX at compose
> time.
> Google Chrome of the latest available version on Fedora at compose
> time.
> Google Chrome of the latest available version on Windows at compose
> time.
> Google Chrome of the latest available version on OSX at compose time.
>
> Alternately, we could decide that it's only *blocking* if the above
> browsers work with Cockpit when the browser is running on Fedora, but
> that is somewhat at odds with our reasoning for having a management
> console as a web UI in the first place: that it is accessible
> regardless of the client system.

I think that it is fine. But you need to make sure you have resources 
available to test on Windows and OS X. I wonder what can be done to do 
automated testing on the platforms to ensure things work. I would like to have 
us try and automate most if not all of the validation,  at least in a basic 
level.

Dennis

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/23/2015 08:41 AM, Dennis Gilmore wrote:
> On Thursday, October 22, 2015 02:35:11 PM Stephen Gallagher wrote:
> > Recently, we discovered a bug in gnutls that caused Cockpit to
> > be unreachable by recent versions of Google Chrome. It was
> > ambiguous what the release criteria actually means, since it
> > didn't specify which browser applications were blocking. I'd like
> > to propose the following additional wording for Cockpit
> > criteria:
> > All Cockpit functional criteria must be satisfied when the user
> >
> > is running any of the following blocking browsers: - Mozilla
> > Firefox as shipped in the same Fedora release - Mozilla Firefox
> > of the latest available version on Windows at compose time. -
> > Mozilla Firefox of the latest available version on OSX at
> > compose time. - Google Chrome of the latest available version on
> > Fedora at compose time. - Google Chrome of the latest available
> > version on Windows at compose time. - Google Chrome of the latest
> > available version on OSX at compose time.
> >
> > Alternately, we could decide that it's only *blocking* if the
> > above browsers work with Cockpit when the browser is running on
> > Fedora, but that is somewhat at odds with our reasoning for
> > having a management console as a web UI in the first place: that
> > it is accessible regardless of the client system.
>
> I think that it is fine. But you need to make sure you have
> resources available to test on Windows and OS X. I wonder what can
> be done to do automated testing on the platforms to ensure things
> work. I would like to have us try and automate most if not all of
> the validation,  at least in a basic level.

Stef Walter (Cockpit lead) proposed the same statement in another
reply (which I suspect is sitting in moderation). So yes, the plan is
to have that handled at the very least in the upstream automated
testing, which should presumably be reusable by Fedora QA.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iEYEARECAAYFAlYqLgoACgkQeiVVYja6o6M2zQCcCYA4LR+xtmvw8uti5VeORgSR
> kEsAnRPh1OzbM5Z5rWNI2XLkHT8oCPF1
> =Lucl
> -----END PGP SIGNATURE-----

[Mike Ruckman]

On Fri, Oct 23, 2015 at 07:41:56AM -0500, Dennis Gilmore wrote:
> On Thursday, October 22, 2015 02:35:11 PM Stephen Gallagher wrote:
> > Recently, we discovered a bug in gnutls that caused Cockpit to be
> > unreachable by recent versions of Google Chrome. It was ambiguous what
> > the release criteria actually means, since it didn't specify which
> > browser applications were blocking. I'd like to propose the following
> > additional wording for Cockpit criteria:
> > All Cockpit functional criteria must be satisfied when the user is
> >
> > running any of the following blocking browsers:
> > Mozilla Firefox as shipped in the same Fedora release
> > Mozilla Firefox of the latest available version on Windows at
> > compose time.
> > Mozilla Firefox of the latest available version on OSX at compose
> > time.
> > Google Chrome of the latest available version on Fedora at compose
> > time.
> > Google Chrome of the latest available version on Windows at compose
> > time.
> > Google Chrome of the latest available version on OSX at compose time.
> >
> > Alternately, we could decide that it's only *blocking* if the above
> > browsers work with Cockpit when the browser is running on Fedora, but
> > that is somewhat at odds with our reasoning for having a management
> > console as a web UI in the first place: that it is accessible
> > regardless of the client system.
>
> I think that it is fine. But you need to make sure you have resources 
> available to test on Windows and OS X. I wonder what can be done to do 
> automated testing on the platforms to ensure things work. I would like to have 
> us try and automate most if not all of the validation,  at least in a basic 
> level.
>
> Dennis

FWIW, I'm willing to help write some selenium [0] tests for validating
cockpit. Depending on if we can get licenses for the different OSs we
want to validate against, we could also set up a grid [1] for that.

It's been a while since I've worked with selenium, but I don't think
it'd take me long to get back up to speed.

[0] http://www.seleniumhq.org/
[1] https://github.com/SeleniumHQ/selenium/wiki/Grid2

[Adam Williamson]

On Fri, 2015-10-23 at 07:41 -0500, Dennis Gilmore wrote:
> On Thursday, October 22, 2015 02:35:11 PM Stephen Gallagher wrote:
> > Recently, we discovered a bug in gnutls that caused Cockpit to be
> > unreachable by recent versions of Google Chrome. It was ambiguous
> > what
> > the release criteria actually means, since it didn't specify which
> > browser applications were blocking. I'd like to propose the
> > following
> > additional wording for Cockpit criteria:
> > All Cockpit functional criteria must be satisfied when the user
> >
> > is
> > running any of the following blocking browsers:
> >  - Mozilla Firefox as shipped in the same Fedora release
> >  - Mozilla Firefox of the latest available version on Windows at
> >    compose time.
> >  - Mozilla Firefox of the latest available version on OSX at
> > compose
> >    time.
> >  - Google Chrome of the latest available version on Fedora at
> > compose
> >    time.
> >  - Google Chrome of the latest available version on Windows at
> > compose
> >    time.
> >  - Google Chrome of the latest available version on OSX at compose
> > time.
> >
> > Alternately, we could decide that it's only *blocking* if the above
> > browsers work with Cockpit when the browser is running on Fedora,
> > but
> > that is somewhat at odds with our reasoning for having a management
> > console as a web UI in the first place: that it is accessible
> > regardless of the client system.
>
> I think that it is fine. But you need to make sure you have resources
> available to test on Windows and OS X.

Not necessarily. we don't have an exact 1:1 mapping of testing to
criteria. We test some stuff beyond the criteria, and we don't test
some stuff that *is* in the criteria; things like the 'data corruption'
criterion aren't realistically testable, exactly.

There's a precedent that it's OK to have criteria which basically work
this way: if someone reports a violation and it gets nominated, it will
be approved.
> I wonder what can be done to do 
> automated testing on the platforms to ensure things work.

There are various 'see how your site looks in X' services, I don't know
much about any of them though.
> I would like to have 
> us try and automate most if not all of the validation,  at least in a
> basic 
> level.

This is exactly what we've been doing with openQA for the last twelve
months.

[Mike Ruckman]

On Thu, Oct 22, 2015 at 02:35:11PM -0400, Stephen Gallagher wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Recently, we discovered a bug in gnutls that caused Cockpit to be
> unreachable by recent versions of Google Chrome. It was ambiguous what
> the release criteria actually means, since it didn't specify which
> browser applications were blocking. I'd like to propose the following
> additional wording for Cockpit criteria:
> All Cockpit functional criteria must be satisfied when the user is
>
> running any of the following blocking browsers:
> Mozilla Firefox as shipped in the same Fedora release
> Mozilla Firefox of the latest available version on Windows at
> compose time.
> Mozilla Firefox of the latest available version on OSX at compose
> time.
> Google Chrome of the latest available version on Fedora at compose
> time.
> Google Chrome of the latest available version on Windows at compose
> time.
> Google Chrome of the latest available version on OSX at compose time.
>
> Alternately, we could decide that it's only *blocking* if the above
> browsers work with Cockpit when the browser is running on Fedora, but
> that is somewhat at odds with our reasoning for having a management
> console as a web UI in the first place: that it is accessible
> regardless of the client system.
>
> Comments welcome, but please keep replies on the
> [e-mail] list, as that's where criteria decisions
> are made.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iEYEARECAAYFAlYpLFwACgkQeiVVYja6o6NF/wCgg6iot1JKOfmAbTZMboBcPvs5
> ZIIAnA+YxRAjPMt69lqv2nOR7qXnCYnV
> =PIuy
> -----END PGP SIGNATURE-----

I'm +1 in general to these criteria. As Dennis said, we just need to
make sure we have access to Windows/OSX in order to actually do the
testing.