To test or not to test testsuites?

[Jens-Ulrik Petersen]

Hi,

I wanted to put out this question about our Fedora Haskell packages.

Currently about 31 of our packages have their testsuites turned on.
(I don't have figures for how many (of our) packages actually have
testsuites: the rest are basically turned off due to missing test library
dependencies.)

I think due to a patch I applied to our ghc-8.2.2 to workaround issues with
abi-depends, it happened at the end of Feb building unsubpackaged
ghc-type-process, which enabled the testsuite causing its package hash to
change (Fedora subpackaged libs don't run their testsuites). :-(
(I think deleting the abi-depends field from every package .conf would fix
this but that would require rebuilding everything and patching Cabal. Well
a hack could be to add an RPM file trigger to do this, but I am wary of
doing that.)

So first of all please be careful if rebuilding package-versions for F28
not to toggle testsuites on or off. If we ship ghc-8.4 in F29 then I think
this issue will be gone, since it no longer generates and ignores
abi-depends metadata.

But the bigger question I wanted to ask is: how worthwhile is it to run the
testsuites of our Haskell libraries in Fedora? I often feel the testsuites
are an extra maintenance burden which we could probably not worry about in
general - it would also simplify our spec files if we didn't need to worry
about them. They could be enabled optionally by opt in - I am happy to add
a "--test" option to cabal-rpm for packages that want to run their tests.

Thoughts?

Jens

[Elliott Sales de Andrade]

Hi Jens,

On 2 April 2018 at 03:58, Jens-Ulrik Petersen <petersen(a)redhat.com&gt; wrote:

>  Hi,
>
>  I wanted to put out this question about our Fedora Haskell packages.
>
>  Currently about 31 of our packages have their testsuites turned on.
>  (I don't have figures for how many (of our) packages actually have
>  testsuites: the rest are basically turned off due to missing test library
>  dependencies.)
>
>  I think due to a patch I applied to our ghc-8.2.2 to workaround issues
>  with abi-depends, it happened at the end of Feb building unsubpackaged
>  ghc-type-process, which enabled the testsuite causing its package hash to
>  change (Fedora subpackaged libs don't run their testsuites). :-(
>  (I think deleting the abi-depends field from every package .conf would fix
>  this but that would require rebuilding everything and patching Cabal. Well
>  a hack could be to add an RPM file trigger to do this, but I am wary of
>  doing that.)
>
>  So first of all please be careful if rebuilding package-versions for F28
>  not to toggle testsuites on or off. If we ship ghc-8.4 in F29 then I think
>  this issue will be gone, since it no longer generates and ignores
>  abi-depends metadata.
>
>  
This should be fine if you're rebuilding all dependents (possibly for other
reasons) though, right?


>  But the bigger question I wanted to ask is: how worthwhile is it to
> run
>  the testsuites of our Haskell libraries in Fedora? I often feel the
>  testsuites are an extra maintenance burden which we could probably not
>  worry about in general - it would also simplify our spec files if we didn't
>  need to worry about them. They could be enabled optionally by opt in - I am
>  happy to add a "--test" option to cabal-rpm for packages that want to run
>  their tests.
>
>  
Generally, I like to enable test suites in my packages. Mostly, I haven't
been doing this for Haskell because we don't have all the testing deps
available. Also, cabal-rpm already adds testing deps behind a --with-tests
flag and since they're disabled by default, I usually never get around to
enabling them. I think many more of the dependencies should be available
now that I added some for the git-annex test suite, but there are likely
quite a few more.

Thoughts?
>  Jens
>
>  
-- 
Elliott

[Jens-Ulrik Petersen]

>  On 2 April 2018 at 03:58, Jens-Ulrik Petersen <petersen(a)redhat.com&gt; wrote:
>
> > (I think deleting the abi-depends field from every package .conf would
> > fix this but that would require rebuilding everything and patching Cabal.
> > Well a hack could be to add an RPM file trigger to do this, but I am wary
> > of doing that.)
> >
>  
BTW I am patching ghc-pkg for F28 so that it doesn't complain about
abi-depends when installing packages.


>  So first of all please be careful if rebuilding package-versions for
> F28
> > not to toggle testsuites on or off. If we ship ghc-8.4 in F29 then I think
> > this issue will be gone, since it no longer generates and ignores
> > abi-depends metadata.
> >
>
>  This should be fine if you're rebuilding all dependents (possibly for
>  other reasons) though, right?
>  
Right

This probably mostly affects un-subpackaging of deps, which is how it hit
me.


>  Generally, I like to enable test suites in my packages. Mostly, I
> haven't
>  been doing this for Haskell because we don't have all the testing deps
>  available. Also, cabal-rpm already adds testing deps behind a --with-tests
>  flag and since they're disabled by default, I usually never get around to
>  enabling them. I think many more of the dependencies should be available
>  now that I added some for the git-annex test suite, but there are likely
>  quite a few more.
>  
Okay, sure

cabal-rpm disables tests when deps are missing otherwise it enables them.

So anyway be careful with Fedora ghc-8.2.2 if you enable or disable
testsuites for a package version.

Jens

[Elliott Sales de Andrade]

On 12 April 2018 at 04:25, Jens-Ulrik Petersen <petersen(a)redhat.com&gt; wrote:

>  On 2 April 2018 at 03:58, Jens-Ulrik Petersen
> <petersen(a)redhat.com&gt; wrote:
> >
> >> (I think deleting the abi-depends field from every package .conf would
> >> fix this but that would require rebuilding everything and patching Cabal.
> >> Well a hack could be to add an RPM file trigger to do this, but I am wary
> >> of doing that.)
> >>
> >
>  BTW I am patching ghc-pkg for F28 so that it doesn't complain about
>  abi-depends when installing packages.
>
>  
Are you saying it's our bug that the hash changes? I find it a bit weird
that the hash changes. I tried building with and without, and nothing
changed in the required shared libraries. So I wonder why the hash would
change in the first place.

[Jens-Ulrik Petersen]

On Wed, Apr 18, 2018 at 6:14 PM, Elliott Sales de Andrade <
quantum.analyst(a)gmail.com&gt; wrote:
>  Are you saying it's our bug that the hash changes? I find it a bit weird
>  that the hash changes. I tried building with and without, and nothing
>  changed in the required shared libraries. So I wonder why the hash would
>  change in the first place.
>  
I *think* it may be due to a patch I backported from 8.4 to workaround an
issue with subpackaged libs.
You are saying the hash doesn't change for you with our 8.2.2?

Thanks, Jens