Haskell executables dynamic vs static linking

[Jens-Ulrik Petersen]

For a long time I was quite "proud" that Fedora Haskell
is using dynamic Haskell linking on primary intel architectures
(ghc-7.8 should support shared libs also on ARM,
but it and cabal 1.18 still default to static linking.)
But of course there are pros and cons for dynamic linking
against Haskell libraries.

pros:
- smaller lighter executables
- enforces rebuilding of reverse deps
  (though this can also be a cons)
- makes use of shared libs to share Haskell object code in memory
- faster linking

cons:
- harder to deploy
- less portable
- dynamically linked tools makes testing/building (for) newer ghc harder

Today I decided to make hscolour statically linked against
Haskell libs in Rawhide.  This will make bootstrapping ghc-7.8.1
or snapshots easier for example, or building ghc rpms for any other version. [1]

I think this is less problematic anyway for BinLib packages
like hscolour, since ghc-hscolour will pick up  any potential
dependency breakage anyway.

I would like to have a static executable for cabal-install too.
Maybe a cleaner alternative is to provide both
a dynamic and a static subpackage, but I am not sure yet
on the best way to do that.  Using alternatives might
be the only realistic way currently perhaps?

Should we do that for more executables packages?

Currently in Fedora because cabal-install is dynamically linked
it is some work to build cabal-install for a new ghc version
which makes testing newer ghc harder.  If we had a statically
linked cabal-install package we could just use it with
any (recent) version of ghc... ie it would be more portable.

Any thoughts or opinions on this?

AFAIK Fedora is the only big distro that packages executables
dynamically linked to Haskell libraries.

Jens

[1] Actually I wish hscolour was just included in ghc.
    It would make life easier.  And likewise cabal-install actually.

[Garrett Mitchener]

My first reaction is that if static linking is necessary or very helpful
for the compiler-documentation tool chain, that's probably okay.  But for
other programs, I think it might cause problems.  I've had trouble with
static linking in a few of my own programs.  For instance, I have one that
uses the BerkeleyDB package, and I had to edit the SPECs and rebuild the
RPMs for libdb and libffi to produce static libraries so I could statically
link my final program.  There's something in the Fedora packaging
guidelines about not packaging static libraries.  I don't know how well it
would work to link with Haskell libraries statically and others dynamically
-- I'm not even sure how to tell ghc to do that.

-- Garrett Mitchener



On Thu, Feb 27, 2014 at 4:18 AM, Jens Petersen <petersen(a)redhat.com&gt; wrote:

>  For a long time I was quite "proud" that Fedora Haskell
>  is using dynamic Haskell linking on primary intel architectures
>  (ghc-7.8 should support shared libs also on ARM,
>  but it and cabal 1.18 still default to static linking.)
>  But of course there are pros and cons for dynamic linking
>  against Haskell libraries.
>
>  pros:
>  - smaller lighter executables
>  - enforces rebuilding of reverse deps
>    (though this can also be a cons)
>  - makes use of shared libs to share Haskell object code in memory
>  - faster linking
>
>  cons:
>  - harder to deploy
>  - less portable
>  - dynamically linked tools makes testing/building (for) newer ghc harder
>
>  Today I decided to make hscolour statically linked against
>  Haskell libs in Rawhide.  This will make bootstrapping ghc-7.8.1
>  or snapshots easier for example, or building ghc rpms for any other
>  version. [1]
>
>  I think this is less problematic anyway for BinLib packages
>  like hscolour, since ghc-hscolour will pick up  any potential
>  dependency breakage anyway.
>
>  I would like to have a static executable for cabal-install too.
>  Maybe a cleaner alternative is to provide both
>  a dynamic and a static subpackage, but I am not sure yet
>  on the best way to do that.  Using alternatives might
>  be the only realistic way currently perhaps?
>
>  Should we do that for more executables packages?
>
>  Currently in Fedora because cabal-install is dynamically linked
>  it is some work to build cabal-install for a new ghc version
>  which makes testing newer ghc harder.  If we had a statically
>  linked cabal-install package we could just use it with
>  any (recent) version of ghc... ie it would be more portable.
>
>  Any thoughts or opinions on this?
>
>  AFAIK Fedora is the only big distro that packages executables
>  dynamically linked to Haskell libraries.
>
>  Jens
>
>  [1] Actually I wish hscolour was just included in ghc.
>      It would make life easier.  And likewise cabal-install actually.
>  _______________________________________________
>  haskell mailing list
>  haskell(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/haskell
>

[Jens-Ulrik Petersen]

Thanks for your reply. 

>  There's something in the Fedora packaging guidelines about not
> packaging
>  static libraries. 
There is... 
There is also something about it being ok for Haskell packages 
in the Haskell Packaging Guidelines we made. :) 

Current ghc stable releases only support Linux shared libs on ia32 arch's. 
ie all Haskell executables on other archs are statically linked to Haskell libs anyway. 
I believe ocaml also links its executables statically to ocaml libs. 

I don't think we can drop Haskell static libraries any time soon, 
but for ghc-7.8 (Fedora 21) I am thinking to subpackage them 
since ghc-pkg no longer screams about missing static libs. 
(ghc-7.8 uses dynmically linking internally by default though 
which somewhat complicates its use of non-exported libs.) 

>  I don't know how well it would work to link with Haskell
> libraries statically
>  and others dynamically -- I'm not even sure how to tell ghc to do that. 
It works just fine - it is what ghc does by default and has been doing forever. :) 
To link dynamically you have to give the -dynamic option to ghc. 

As I wrote, afaik other distros do not distribute dynamically linked Hasklell executables.


Anyway my question was more about providing both for some packages in an optimal way. 

Jens 

$ cat > Main.hs 
main = putStrLn "hi" 
$ ghc Main 
: 
$ ldd Main | grep libHS 
$ ghc Main -dynamic 
: 
$ ldd Main | grep libHS | wc -l 
4 
$

[Garrett Mitchener]

Okay, thanks for the info.

I'm a little leery of the alternatives framework.  I've had to deal with
that with java, and it's complicated because there are several
implementations of java that don't get packaged consistently with each
other.  As in some alternatives provide X, Y, and Z, and some provide W, X,
and Y, and figuring all that out has been frustrating.  What your proposing
probably wouldn't be as bad since the alternative Haskell packages would be
coming from the same source and would presumably be exactly consistent with
each other.

But, if the dynamic executable packages are working, are parallel static
packages necessary?

Another issue is the experimental nature of a lot of Haskell packages: I
recently had to make a lot of minor updates to my simulation because of a
change in the aeson package, which is itself under fairly heavy
development.  We can probably link dynamically to well-established Haskell
libraries with no problem, but the more experimental ones could result in
the usual trap that executable foo requires the new features in libbar-5
but executable baz hasn't been updated yet and requires libbar-4 ...  From
what I can tell, GHC and Cabal are better at dealing with multiple
coexisting versions of packages than RPM and Yum are.

Just thinking out loud here...



On Thu, Feb 27, 2014 at 8:36 PM, Jens Petersen <petersen(a)redhat.com&gt; wrote:

>  Thanks for your reply.
>
>  There's something in the Fedora packaging guidelines about not packaging
>  static libraries.
>
>  There is...
>  There is also something about it being ok for Haskell packages
>  in the Haskell Packaging Guidelines we made. :)
>
>  Current ghc stable releases only support Linux shared libs on ia32 arch's.
>  ie all Haskell executables on other archs are statically linked to Haskell
>  libs anyway.
>  I believe ocaml also links its executables statically to ocaml libs.
>
>  I don't think we can drop Haskell static libraries any time soon,
>  but for ghc-7.8 (Fedora 21) I am thinking to subpackage them
>  since ghc-pkg no longer screams about missing static libs.
>  (ghc-7.8 uses dynmically linking internally by default though
>  which somewhat complicates its use of non-exported libs.)
>
>   I don't know how well it would work to link with Haskell libraries
>  statically and others dynamically -- I'm not even sure how to tell ghc to
>  do that.
>
>  It works just fine - it is what ghc does by default and has been doing
>  forever. :)
>  To link dynamically you have to give the -dynamic option to ghc.
>
>  As I wrote, afaik other distros do not distribute dynamically linked
>  Hasklell executables.
>
>  Anyway my question was more about providing both for some packages in an
>  optimal way.
>
>  Jens
>
>
>  $ cat > Main.hs
>  main = putStrLn "hi"
>  $ ghc Main
>  :
>  $ ldd Main | grep libHS
>  $ ghc Main -dynamic
>  :
>  $ ldd Main | grep libHS | wc -l
>  4
>  $
>
>
>  _______________________________________________
>  haskell mailing list
>  haskell(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/haskell
>
>

[Sébastien Willmann]

Le jeudi 27 févr. 2014 à 04:18:56 (-0500), Jens Petersen a écrit : 
>  For a long time I was quite "proud" that Fedora Haskell
>  is using dynamic Haskell linking on primary intel architectures
>  (ghc-7.8 should support shared libs also on ARM,
>  but it and cabal 1.18 still default to static linking.)
>  But of course there are pros and cons for dynamic linking
>  against Haskell libraries.
>  
>  pros:
>  - smaller lighter executables
>  - enforces rebuilding of reverse deps
>    (though this can also be a cons)
>  - makes use of shared libs to share Haskell object code in memory
>  - faster linking
>  
>  cons:
>  - harder to deploy
>  - less portable
>  - dynamically linked tools makes testing/building (for) newer ghc harder
>  
>  Today I decided to make hscolour statically linked against
>  Haskell libs in Rawhide.  This will make bootstrapping ghc-7.8.1
>  or snapshots easier for example, or building ghc rpms for any other version. [1]
>  
>  I think this is less problematic anyway for BinLib packages
>  like hscolour, since ghc-hscolour will pick up  any potential
>  dependency breakage anyway.
>  
>  I would like to have a static executable for cabal-install too.
>  Maybe a cleaner alternative is to provide both
>  a dynamic and a static subpackage, but I am not sure yet
>  on the best way to do that.  Using alternatives might
>  be the only realistic way currently perhaps?
>  
>  Should we do that for more executables packages?
>  
>  Currently in Fedora because cabal-install is dynamically linked
>  it is some work to build cabal-install for a new ghc version
>  which makes testing newer ghc harder.  If we had a statically
>  linked cabal-install package we could just use it with
>  any (recent) version of ghc... ie it would be more portable.
>  
>  Any thoughts or opinions on this?
>  
>  AFAIK Fedora is the only big distro that packages executables
>  dynamically linked to Haskell libraries.
>  
>  Jens
>  
>  [1] Actually I wish hscolour was just included in ghc.
>      It would make life easier.  And likewise cabal-install actually.
>  _______________________________________________
>  haskell mailing list
>  haskell(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/haskell 
The general packaging guidelines forbid static linking. Now that ghc
supports dynamic linking, shouldn't we just follow those guidelines? Is
there an issue specific to Haskell that makes it harder than with other
languages?

Sébastien Willmann

[Jens-Ulrik Petersen]

Sébastien Willmann wrote:
>  Is there an issue specific to Haskell that makes it harder than with
> other languages? 
Well yes, the fact that I can't take a dynamic executable from F19 and run it in F20
or vice-versa, etc.

I just "bumped" into this again when trying to get cabal-rpm under ghc-7.8.

I guess one could argue that tools like cabal-install and cblrpm need/should
not be tied to a specific ghc version: at least in this sense statically
linking them for more portability makes some sense.  Arguably cabal-install
would be enough but in practice some executables have a lot of deps
making that impractical.

Overall I agree we should continue to do dynamic linking generally,
but also consider linking static Haskell libs for greater portability
across releases/ghc versions perhaps where it makes particular sense.

Though this would really be a workaround for the lack of parallel
installation of Fedora Haskell shared libraries per ghc version
(Software Collections could be used for this)
or actually ghc's hyperstrict library versioning.
Another approach which seems good for dynamic
might be "bundle" cabal-install into the ghc package
though upstream might well be relunctant to do it.

Jens

[Jens-Ulrik Petersen]

[sorry Sébastien, this was my original reply to your mail
 which I thought I had already posted but seems not!]

>  The general packaging guidelines forbid static linking. Now that ghc
>  supports dynamic linking, shouldn't we just follow those guidelines? 
We do already, as far as possible.

I am not sure yet if Fedora secondary archs will also get
shared libs or not with ghc-7.8.

>  Is there an issue specific to Haskell that makes it harder than with
> other
>  languages? 
Well I tried to explain the issues in my original posting.
I guess I could go into more detail.

But perhaps you're right in that there are other solutions/workarounds:

- a copr repo with statically linked cabal-install for example
  (I already did one for pandoc for EPEL
http://petersen.fedorapeople.org/pandoc-standalone/)
  I planning to do one or more big Haskell copr repos anyway,
  since koji + package reviews are hardwork/slow anyway.

- adding static subpackages: I think ghc could pull in hscolour-static
  and use /usr/bin/hscolour.static when bootstrapping say
  - then people could install cabal-install-static instead of cabal-install
    when testing a newer ghc snapshot etc

So perhaps we could propose updating the Haskell Guidelines to allow
both dynamic and static executables wrt to Haskell libs.

So maybe the problems are not so insurmountable. :)
I would rather avoid using alternatives (like emacs does) if possible...

Jens