New subject: Blocking criteria proposal for F30+: Printing

28 Feb 2019

      I just realized I only responded to Zdenek the other day. Re-sending
my response now.
On Tue, Feb 12, 2019 at 9:13 AM Zdenek Dohnal zdohnal@redhat.com wrote:
...
Hi,
comments are in the text:
On 2/11/19 9:17 PM, Stephen Gallagher wrote:
...
On Mon, Feb 11, 2019 at 2:24 PM Chris Murphy lists@colorremedies.com wrote:
...
On Mon, Feb 11, 2019 at 9:58 AM Stephen Gallagher sgallagh@redhat.com wrote:
...
Sorry that it's taken me so long to get back to this.
I think the feedback on this has been mostly positive on the Beta
criteria, but I'd like to tweak the phrasing a bit and see if this
comes off more favorable:
I'd like to propose that we add the following criteria to Beta for Fedora 30+:

Printing must work on at least one printer available to Fedora QA.

"Work" is defined as the output from the device matching a preview
shown on the GNOME print preview display. (Note that differences in
color reproduction are not considered "non-working".)
Does the criterion  pply strictly to the printing of text and line
art, or does it also apply to gross departures in photographs? If the
latter:
^minor differences in color reproduction are not considered "non-working"; or
^only major differences in color reproduction are considered "non-working"
Major defined as any of:
obvious and grossly incorrect scaling (e.g. +/- 20%)
color inversion, torqued primaries (white becomes black, black becomes
white; red becomes blue, blue becomes green, etc)
tone reproduction that obliterates relevant identifying detail in two
or more test images
With that language I'm trying to carve out only remarkable, WTF level,
bugs as blockers.
I think we can *probably* leave this as a thing to be decided at a
blocker bug review. I really want to avoid trying to set a hard line
on a topic that is inherently subjective. In general, I think we can
just rely on the "last blocker at Go/No-Go" test for this.
I agree with Stephen - such topics can be really subjective and even the
fault does not have to be on Fedora side (f.e. when you catch the file
which goes to the printer, you look into it and it looks fine, but
output paper has 'slightly' different colors, scale etc... - so there
can be issues in the printer itself).
...
...
Next question is what applications to use for printing, since the
initiating application matters. What if there's a bug in just one
application? That shouldn't be a printing blocker (it might be a basic
functionality blocker for that application if it's included in default
installations). So I'd say pick two. Firefox and LibreOffice? Firefox
and evince?
How about "Desktop environment's 'test page' functionality" and
whichever basic text editor comes with it.
IMHO it is not correct blocker criteria for printing as itself, but it
is more like blocker for these applications. AFAIK blocker is the issue,
which can not be worked around - if the file is printable by CUPS CLI
commands 'lp'/'lpr', but not from a app, IMHO it is not blocker for
printing.
IMO issues like 'not being able to print from X application' should be
blocking/release criteria for some common/widely used apps like
Firefox/evince/libreoffice, not for printing itself. (If the issue would
be actually connected to CUPS, I'll cooperate with them to fix the issue).
Well, we don't have to be that specific in the release criteria,
honestly. We're talking about blocker criteria specifically for
blocking desktops, so in my opinion it's okay to have "test page" and
"basic text editor" as the stand-ins for this. (This is similar to how
we have "package manager must be able to download and apply updates"
as a stand-in for "the network must not be totally broken".)
I'd be fine if we wanted to add a corollary that either of these are
not blockers if it can be shown that other applications can print
successfully. I just wanted to suggest those as the basic litmus test.
...
...
...
Next question, test document(s). European Color Initiative has several
test PDFs already prepared, perhaps the most applicable for our
purposes is the visual test (and a subset of it).And for font scaling
and reproduction, Ghent Working Group has test GWG 9.1 which tests
various encodings of TrueType, PostScript, and OpenType rendering.
Also, there's a suite of LibreOffice test files, and while I haven't
gone through it, I'm willing to bet there's one or two that'd serve as
a decent sanity tester (in any case I'm not proposing printing out
entire test suites):
https://github.com/freedesktop/libreoffice-test-files
Chris, would you mind elaborating more on the topic of these test files
and tests from these sources? Martin (mosvald in CC) currently does only
comparing sample file and output file in ghostscript and I'm on my way
to do it the similar way in CUPS and printer driver packages.
Do they have special tests available to look into them? I saw mostly
only pdf file in ECI downloads, I did not see anything in GWG and only
docx or xlsx files in libreoffice tests.
<snip>
I'm going to suggest that going into this level of detail on how to
write the tests is mostly going to cloud the issue. I think we first
want to make sure we agree that the basics of the proposal are sound.
I'm perfectly happy to delegate the specifics of how to verify the
functionality to the subject matter experts.
I'll update the proposal again with some of the feedback:
* Printing must work on at least one printer available to Fedora QA.
"Work" is defined as the output from the device matching a preview
shown on the GNOME print preview display. (Note that non-ridiculous
differences in color reproduction are not considered "non-working". In
general, we'll apply the "last blocker at Go/No-Go" principle here
when deciding whether a print glitch is truly blocking.)
and this to Final for Fedora 30+:
* Printing must work (as defined above) on at least one printer using
each of the following drivers:
    - The built-in print-to-PDF driver
    - The generic IPP driver
* For each blocking desktop, it must be possible to print:
    - A test page from the desktop environment's built-in "test page"
feature, if such a feature exists.
    - A simple text document of at least 100 words (lorem ipsum) from
the standard basic text editor accompanying that desktop.
This does not mean that all printers need to function properly that
use the IPP driver, just that at least one does (so we
know that printing as a whole is unbroken). We won't specify any
particular hardware makes or models that must work.

Re: Blocking criteria proposal for F30+: Printing