Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>  [ Cc: qemu-block ]
>  
>  Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
>  > On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley wrote:
>  > > > The background to this is I've spent far too long trying to optimize
>  > > > the conversion of qcow2 files to raw files.  Most existing qcow2 files
>  > > > that you can find online are zlib compressed, including the qcow2
>  > > > images provided by Fedora.  Each cluster in the file is separately
>  > > > compressed as a zlib stream, and qemu uses zlib library functions to
>  > > > decompress them.  When downloading and decompressing these files, I
>  > > > measured 40%+ of the total CPU time is doing zlib decompression.
>  > > > 
>  > > > [You don't need to tell me how great Zstd is, qcow2 supports this for
>  > > > compression also, but it is not widely used by existing content.]
>  
>  You make it sound like compressing each cluster individually has a big
>  impact. If so, does increasing the cluster size make a difference, too?
>  That could be an change with less compatibility concerns. 
The issue we're discussing in the original thread is speed of
decompression.  We noted that using zlib-ng (a not-quite drop-in
replacement for zlib) improves decompression speed by 40% or more.

Original thread:
https://lists.fedoraproject.org/archives/list/[e-mail].o...
zlib-ng proposed change:
https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3

Size of the compressed file is also a concern, but wasn't discussed.

>  > > Independent from the decision to use zlib-ng as a
> distro-wide zlib
>  > > replacement, which is a good idea regardless.
>  > >
>  > > Are there reasons why Fedora's qcow2 images cannot switch to Zstd
>  > > compression?  Zstd support appears to have been added by QEMU 5.1 in
>  > > August 2020, and both EL8 and EL9 appear to have newer versions QEMU
>  > > available (therefore, they ought to be able to support those
>  > > images).
>  > 
>  > TBH I think the most probable reason is that people don't know about
>  > it and it is not obvious that you have to enable it.  To generate a
>  > zlib-compressed qcow2 file, you simply add the -c option, easy.  To
>  > use zstd compression you have to use this mouthful:
>  > 
>  >   qemu-img convert -f raw disk.img -O qcow2 disk.qcow2 -c -o compression_type=zstd
>  > 
>  > The qemu-img man page doesn't even mention it.
>  
>  Good point, that needs to be fixed.
>  
>  (Though I don't think that '-o compression_type=zstd' in an unreasonable
>  mouthful for enabling a non-standard compression algorithm.)
>  
>  > I think all recent qcow2-based tools should be fine with zstd, but I
>  > didn't check them all (RHEL 7 is still quite popular so that platform
>  > would no longer be compatible).
>  
>  So my first thought was that maybe we can just change the default now
>  that the option has been there for quite a while. But then it occurred
>  to me that it's not a hard dependency. So at least, the default would
>  still have to be zlib if zstd isn't even compiled in, which makes it a
>  bit less nice in theory. In practice, it depends on what build options
>  distros actually use.
>  
>  Unfortunately, we seem to build the RHEL packages with --disable-zstd
>  (I suppose just because we tend to disable everything nobody explicitly
>  asked for). Maybe we should check other distros. If zstd is commonly
>  enabled, we could still make it the default upstream (because honestly,
>  it probably does makes sense from an upstream perspective). Of course,
>  for RHEL this would mean that images out there are likely to use zstd
>  soon, so it might need to enable zstd in newer versions, too. 
Oh that is bad actually.  We really should enable zstd support in RHEL :-/

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

zlib-ng as a compat replacement for zlib

[Florian Weimer]

* Richard W. M. Jones:

>  On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> > Unfortunately, we seem to build the RHEL packages with --disable-zstd
> > (I suppose just because we tend to disable everything nobody explicitly
> > asked for). Maybe we should check other distros. If zstd is commonly
> > enabled, we could still make it the default upstream (because honestly,
> > it probably does makes sense from an upstream perspective). Of course,
> > for RHEL this would mean that images out there are likely to use zstd
> > soon, so it might need to enable zstd in newer versions, too.
>
>  Oh that is bad actually.  We really should enable zstd support in RHEL :-/ 
The zstd package was added to RHEL 8 only after its release, so it's not
that you had much choice initially.

Thanks,
Florian

zlib-ng as a compat replacement for zlib

[Kevin Wolf]

Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
>  On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>  > [ Cc: qemu-block ]
>  > 
>  > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
>  > > On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley wrote:
>  > > > > The background to this is I've spent far too long trying to
> optimize
>  > > > > the conversion of qcow2 files to raw files.  Most existing qcow2
> files
>  > > > > that you can find online are zlib compressed, including the qcow2
>  > > > > images provided by Fedora.  Each cluster in the file is separately
>  > > > > compressed as a zlib stream, and qemu uses zlib library functions to
>  > > > > decompress them.  When downloading and decompressing these files, I
>  > > > > measured 40%+ of the total CPU time is doing zlib decompression.
>  > > > > 
>  > > > > [You don't need to tell me how great Zstd is, qcow2 supports this
> for
>  > > > > compression also, but it is not widely used by existing content.]
>  > 
>  > You make it sound like compressing each cluster individually has a big
>  > impact. If so, does increasing the cluster size make a difference, too?
>  > That could be an change with less compatibility concerns.
>  
>  The issue we're discussing in the original thread is speed of
>  decompression.  We noted that using zlib-ng (a not-quite drop-in
>  replacement for zlib) improves decompression speed by 40% or more.
>  
>  Original thread:
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  zlib-ng proposed change:
>  https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
>  
>  Size of the compressed file is also a concern, but wasn't discussed. 
I understand the context and didn't really think about file size at all.

My question was essentially if decompressing many small blocks (as we do
today) performs significantly different from decompressing fewer larger
blocks (as we would do with a larger cluster size).

Kevin

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
>  Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
>  > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>  > > [ Cc: qemu-block ]
>  > > 
>  > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
>  > > > On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley wrote:
>  > > > > > The background to this is I've spent far too long trying to
> optimize
>  > > > > > the conversion of qcow2 files to raw files.  Most existing qcow2
> files
>  > > > > > that you can find online are zlib compressed, including the
> qcow2
>  > > > > > images provided by Fedora.  Each cluster in the file is
> separately
>  > > > > > compressed as a zlib stream, and qemu uses zlib library
> functions to
>  > > > > > decompress them.  When downloading and decompressing these
> files, I
>  > > > > > measured 40%+ of the total CPU time is doing zlib
> decompression.
>  > > > > > 
>  > > > > > [You don't need to tell me how great Zstd is, qcow2 supports
> this for
>  > > > > > compression also, but it is not widely used by existing
> content.]
>  > > 
>  > > You make it sound like compressing each cluster individually has a big
>  > > impact. If so, does increasing the cluster size make a difference, too?
>  > > That could be an change with less compatibility concerns.
>  > 
>  > The issue we're discussing in the original thread is speed of
>  > decompression.  We noted that using zlib-ng (a not-quite drop-in
>  > replacement for zlib) improves decompression speed by 40% or more.
>  > 
>  > Original thread:
>  >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > zlib-ng proposed change:
>  > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
>  > 
>  > Size of the compressed file is also a concern, but wasn't discussed.
>  
>  I understand the context and didn't really think about file size at all.
>  
>  My question was essentially if decompressing many small blocks (as we do
>  today) performs significantly different from decompressing fewer larger
>  blocks (as we would do with a larger cluster size). 
I did a quick test just by adjusting the cluster size of a qcow2 file:

  $ virt-builder fedora-36
  $ ls -lsh fedora-36.img 
  1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
  $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
  $ ls -lsh test.raw 
  4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
  $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o
compression_type=zlib,cluster_size=4096

(for cluster sizes 4k, 64k, 512k, 2048k, and
compression types zlib & zstd)

I tested the speed of decompression using:

  $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw test.out'
  (qemu 8.0.0-4.fc39.x86_64)

  $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
'\''nbdcopy --request-size "$uri" test.out'\'' '
  (nbdkit-1.35.11-2.fc40.x86_64)

Results:

  Cluster  Compression  Compressed size  Prog   Decompression speed

  4k       zlib         3228811264       qemu   5.921 s ±  0.074 s
  4k       zstd         3258097664       qemu   5.189 s ±  0.158 s

  4k       zlib/zstd                     nbdkit failed, bug!!

  64k      zlib         3164667904       qemu   3.579 s ±  0.094 s
  64k      zstd         3132686336       qemu   1.770 s ±  0.060 s

  64k      zlib         3164667904       nbdkit 1.254 s ±  0.065 s
  64k      zstd         3132686336       nbdkit 1.315 s ±  0.037 s

  512k     zlib         3158744576       qemu   4.008 s ±  0.058 s
  512k     zstd         3032697344       qemu   1.503 s ±  0.072 s

  512k     zlib         3158744576       nbdkit 1.702 s ±  0.026 s
  512k     zstd         3032697344       nbdkit 1.593 s ±  0.039 s

  2048k    zlib         3197569024       qemu   4.327 s ±  0.051 s
  2048k    zstd         2995143168       qemu   1.465 s ±  0.085 s

  2048k    zlib         3197569024       nbdkit 3.660 s ±  0.011 s
  2048k    zstd         2995143168       nbdkit 3.368 s ±  0.057 s

No great surprises - very small cluster size is inefficient, but
scaling up the cluster size gain performance, and zstd performs better
than zlib once the cluster size is sufficiently large.

Something strange happens with 2048k clusters + zlib - the file gets
larger and decompression gets slower again.

nbdkit has several issues, fails completely at 4k, and scaling issues
at the largest cluster size.  The scaling issue turns out to be caused
by the default request size used by nbdcopy (256k, increasing it cures
the problem).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>    $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX
> --run '\''nbdcopy --request-size "$uri" test.out'\''
> ' 
Sorry, copy and paste error, the command is:

hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
'\''nbdcopy "$uri" test.out'\'' '

unless you want to adjust the --request-size parameter to fix
the scaling problem at the largest cluster size.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

zlib-ng as a compat replacement for zlib

[Daniel P. Berrangé]

On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>  On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
>  > Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
>  > > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>  > > > [ Cc: qemu-block ]
>  > > > 
>  > > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
>  > > > > On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley wrote:
>  > > > > > > The background to this is I've spent far too long
> trying to optimize
>  > > > > > > the conversion of qcow2 files to raw files.  Most existing
> qcow2 files
>  > > > > > > that you can find online are zlib compressed, including the
> qcow2
>  > > > > > > images provided by Fedora.  Each cluster in the file is
> separately
>  > > > > > > compressed as a zlib stream, and qemu uses zlib library
> functions to
>  > > > > > > decompress them.  When downloading and decompressing these
> files, I
>  > > > > > > measured 40%+ of the total CPU time is doing zlib
> decompression.
>  > > > > > > 
>  > > > > > > [You don't need to tell me how great Zstd is, qcow2
> supports this for
>  > > > > > > compression also, but it is not widely used by existing
> content.]
>  > > > 
>  > > > You make it sound like compressing each cluster individually has a big
>  > > > impact. If so, does increasing the cluster size make a difference, too?
>  > > > That could be an change with less compatibility concerns.
>  > > 
>  > > The issue we're discussing in the original thread is speed of
>  > > decompression.  We noted that using zlib-ng (a not-quite drop-in
>  > > replacement for zlib) improves decompression speed by 40% or more.
>  > > 
>  > > Original thread:
>  > >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > > zlib-ng proposed change:
>  > > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
>  > > 
>  > > Size of the compressed file is also a concern, but wasn't discussed.
>  > 
>  > I understand the context and didn't really think about file size at all.
>  > 
>  > My question was essentially if decompressing many small blocks (as we do
>  > today) performs significantly different from decompressing fewer larger
>  > blocks (as we would do with a larger cluster size).
>  
>  I did a quick test just by adjusting the cluster size of a qcow2 file:
>  
>    $ virt-builder fedora-36
>    $ ls -lsh fedora-36.img 
>    1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
>    $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
>    $ ls -lsh test.raw 
>    4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
>    $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o
> compression_type=zlib,cluster_size=4096
>  
>  (for cluster sizes 4k, 64k, 512k, 2048k, and
>  compression types zlib & zstd)
>  
>  I tested the speed of decompression using:
>  
>    $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw
> test.out'
>    (qemu 8.0.0-4.fc39.x86_64)
>  
>    $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>    (nbdkit-1.35.11-2.fc40.x86_64)
>  
>  Results:
>  
>    Cluster  Compression  Compressed size  Prog   Decompression speed
>  
>    4k       zlib         3228811264       qemu   5.921 s ±  0.074 s
>    4k       zstd         3258097664       qemu   5.189 s ±  0.158 s
>  
>    4k       zlib/zstd                     nbdkit failed, bug!!
>  
>    64k      zlib         3164667904       qemu   3.579 s ±  0.094 s
>    64k      zstd         3132686336       qemu   1.770 s ±  0.060 s
>  
>    64k      zlib         3164667904       nbdkit 1.254 s ±  0.065 s
>    64k      zstd         3132686336       nbdkit 1.315 s ±  0.037 s
>  
>    512k     zlib         3158744576       qemu   4.008 s ±  0.058 s
>    512k     zstd         3032697344       qemu   1.503 s ±  0.072 s
>  
>    512k     zlib         3158744576       nbdkit 1.702 s ±  0.026 s
>    512k     zstd         3032697344       nbdkit 1.593 s ±  0.039 s
>  
>    2048k    zlib         3197569024       qemu   4.327 s ±  0.051 s
>    2048k    zstd         2995143168       qemu   1.465 s ±  0.085 s
>  
>    2048k    zlib         3197569024       nbdkit 3.660 s ±  0.011 s
>    2048k    zstd         2995143168       nbdkit 3.368 s ±  0.057 s
>  
>  No great surprises - very small cluster size is inefficient, but
>  scaling up the cluster size gain performance, and zstd performs better
>  than zlib once the cluster size is sufficiently large. 
The default qcow2 cluster size is 64k, which means we've already
got the vast majority of the perfornmance and file size win. Going
beyond 64k defaults doesn't seem massively compelling.

zstd does have a small space win over zlib as expected, but again
nothing so drastic that it seems compelling to change - that win
will be line noise in the overall bigger picture of image storage
and download times.

The major difference here is that zstd is much faster than zlib
at decompress. I'd be curious if zlib-ng closes that gap ?

If it does, then for the sake of image portability it'd be better
to stick with zlib compression in qcow2 and leverage zlib-ng for
speed, and ignore zstd.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:55:50AM +0100, Daniel P. Berrangé wrote:
>  On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>  > On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
>  > > Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
>  > > > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>  > > > > [ Cc: qemu-block ]
>  > > > > 
>  > > > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
>  > > > > > On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley wrote:
>  > > > > > > > The background to this is I've spent far too long
> trying to optimize
>  > > > > > > > the conversion of qcow2 files to raw files.  Most
> existing qcow2 files
>  > > > > > > > that you can find online are zlib compressed,
> including the qcow2
>  > > > > > > > images provided by Fedora.  Each cluster in the file
> is separately
>  > > > > > > > compressed as a zlib stream, and qemu uses zlib
> library functions to
>  > > > > > > > decompress them.  When downloading and decompressing
> these files, I
>  > > > > > > > measured 40%+ of the total CPU time is doing zlib
> decompression.
>  > > > > > > > 
>  > > > > > > > [You don't need to tell me how great Zstd is,
> qcow2 supports this for
>  > > > > > > > compression also, but it is not widely used by
> existing content.]
>  > > > > 
>  > > > > You make it sound like compressing each cluster individually has a
> big
>  > > > > impact. If so, does increasing the cluster size make a difference,
> too?
>  > > > > That could be an change with less compatibility concerns.
>  > > > 
>  > > > The issue we're discussing in the original thread is speed of
>  > > > decompression.  We noted that using zlib-ng (a not-quite drop-in
>  > > > replacement for zlib) improves decompression speed by 40% or more.
>  > > > 
>  > > > Original thread:
>  > > >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > > > zlib-ng proposed change:
>  > > > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
>  > > > 
>  > > > Size of the compressed file is also a concern, but wasn't discussed.
>  > > 
>  > > I understand the context and didn't really think about file size at all.
>  > > 
>  > > My question was essentially if decompressing many small blocks (as we do
>  > > today) performs significantly different from decompressing fewer larger
>  > > blocks (as we would do with a larger cluster size).
>  > 
>  > I did a quick test just by adjusting the cluster size of a qcow2 file:
>  > 
>  >   $ virt-builder fedora-36
>  >   $ ls -lsh fedora-36.img 
>  >   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
>  >   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
>  >   $ ls -lsh test.raw 
>  >   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
>  >   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o
> compression_type=zlib,cluster_size=4096
>  > 
>  > (for cluster sizes 4k, 64k, 512k, 2048k, and
>  > compression types zlib & zstd)
>  > 
>  > I tested the speed of decompression using:
>  > 
>  >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw
> test.out'
>  >   (qemu 8.0.0-4.fc39.x86_64)
>  > 
>  >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>  >   (nbdkit-1.35.11-2.fc40.x86_64)
>  > 
>  > Results:
>  > 
>  >   Cluster  Compression  Compressed size  Prog   Decompression speed
>  > 
>  >   4k       zlib         3228811264       qemu   5.921 s ±  0.074 s
>  >   4k       zstd         3258097664       qemu   5.189 s ±  0.158 s
>  > 
>  >   4k       zlib/zstd                     nbdkit failed, bug!!
>  > 
>  >   64k      zlib         3164667904       qemu   3.579 s ±  0.094 s
>  >   64k      zstd         3132686336       qemu   1.770 s ±  0.060 s
>  > 
>  >   64k      zlib         3164667904       nbdkit 1.254 s ±  0.065 s
>  >   64k      zstd         3132686336       nbdkit 1.315 s ±  0.037 s
>  > 
>  >   512k     zlib         3158744576       qemu   4.008 s ±  0.058 s
>  >   512k     zstd         3032697344       qemu   1.503 s ±  0.072 s
>  > 
>  >   512k     zlib         3158744576       nbdkit 1.702 s ±  0.026 s
>  >   512k     zstd         3032697344       nbdkit 1.593 s ±  0.039 s
>  > 
>  >   2048k    zlib         3197569024       qemu   4.327 s ±  0.051 s
>  >   2048k    zstd         2995143168       qemu   1.465 s ±  0.085 s
>  > 
>  >   2048k    zlib         3197569024       nbdkit 3.660 s ±  0.011 s
>  >   2048k    zstd         2995143168       nbdkit 3.368 s ±  0.057 s
>  > 
>  > No great surprises - very small cluster size is inefficient, but
>  > scaling up the cluster size gain performance, and zstd performs better
>  > than zlib once the cluster size is sufficiently large.
>  
>  The default qcow2 cluster size is 64k, which means we've already
>  got the vast majority of the perfornmance and file size win. Going
>  beyond 64k defaults doesn't seem massively compelling.
>  
>  zstd does have a small space win over zlib as expected, but again
>  nothing so drastic that it seems compelling to change - that win
>  will be line noise in the overall bigger picture of image storage
>  and download times. 
Yeah, I was a bit surprised by this.  I expected zstd files to be
significantly smaller than zlib even though that's not what zstd is
optimized for.  Not that they'd be about the same.

>  The major difference here is that zstd is much faster than zlib
>  at decompress. I'd be curious if zlib-ng closes that gap ? 
It's quite hard to use zlib-ng in Fedora (currently) since it requires
changes to the source code.  That is what the pull request being
discussed would change, as you could simply install zlib-ng-compat
which would replace libz.so.  But anyway I can't easily get results
for qemu + zlib-ng, but we expect it would be ~ 40% faster at
decompression, and decompression is what is taking most of the time in
the qemu numbers above.

I forgot to say that nbdkit is using zlib-ng, since I made the source
level changes a few weeks back (but most of the nbdkit performance
improvement comes from being able to use lots of threads).

>  If it does, then for the sake of image portability it'd be
> better
>  to stick with zlib compression in qcow2 and leverage zlib-ng for
>  speed, and ignore zstd. 
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

zlib-ng as a compat replacement for zlib

[Daniel P. Berrangé]

On Fri, Sep 01, 2023 at 11:03:54AM +0100, Richard W.M. Jones wrote:
>  On Fri, Sep 01, 2023 at 10:55:50AM +0100, Daniel P. Berrangé wrote:
>  > On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>  > > On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
>  > > > Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
>  > > > > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>  > > > > > [ Cc: qemu-block ]
>  > > > > > 
>  > > > > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
>  > > > > > > On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley
> wrote:
>  > > > > > > > > The background to this is I've spent far too
> long trying to optimize
>  > > > > > > > > the conversion of qcow2 files to raw files.  Most
> existing qcow2 files
>  > > > > > > > > that you can find online are zlib compressed,
> including the qcow2
>  > > > > > > > > images provided by Fedora.  Each cluster in the
> file is separately
>  > > > > > > > > compressed as a zlib stream, and qemu uses zlib
> library functions to
>  > > > > > > > > decompress them.  When downloading and
> decompressing these files, I
>  > > > > > > > > measured 40%+ of the total CPU time is doing zlib
> decompression.
>  > > > > > > > > 
>  > > > > > > > > [You don't need to tell me how great Zstd is,
> qcow2 supports this for
>  > > > > > > > > compression also, but it is not widely used by
> existing content.]
>  > > > > > 
>  > > > > > You make it sound like compressing each cluster individually has
> a big
>  > > > > > impact. If so, does increasing the cluster size make a
> difference, too?
>  > > > > > That could be an change with less compatibility concerns.
>  > > > > 
>  > > > > The issue we're discussing in the original thread is speed of
>  > > > > decompression.  We noted that using zlib-ng (a not-quite drop-in
>  > > > > replacement for zlib) improves decompression speed by 40% or more.
>  > > > > 
>  > > > > Original thread:
>  > > > >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > > > > zlib-ng proposed change:
>  > > > > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
>  > > > > 
>  > > > > Size of the compressed file is also a concern, but wasn't
> discussed.
>  > > > 
>  > > > I understand the context and didn't really think about file size at
> all.
>  > > > 
>  > > > My question was essentially if decompressing many small blocks (as we do
>  > > > today) performs significantly different from decompressing fewer larger
>  > > > blocks (as we would do with a larger cluster size).
>  > > 
>  > > I did a quick test just by adjusting the cluster size of a qcow2 file:
>  > > 
>  > >   $ virt-builder fedora-36
>  > >   $ ls -lsh fedora-36.img 
>  > >   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
>  > >   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
>  > >   $ ls -lsh test.raw 
>  > >   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
>  > >   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o
> compression_type=zlib,cluster_size=4096
>  > > 
>  > > (for cluster sizes 4k, 64k, 512k, 2048k, and
>  > > compression types zlib & zstd)
>  > > 
>  > > I tested the speed of decompression using:
>  > > 
>  > >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw
> test.out'
>  > >   (qemu 8.0.0-4.fc39.x86_64)
>  > > 
>  > >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>  > >   (nbdkit-1.35.11-2.fc40.x86_64)
>  > > 
>  > > Results:
>  > > 
>  > >   Cluster  Compression  Compressed size  Prog   Decompression speed
>  > > 
>  > >   4k       zlib         3228811264       qemu   5.921 s ±  0.074 s
>  > >   4k       zstd         3258097664       qemu   5.189 s ±  0.158 s
>  > > 
>  > >   4k       zlib/zstd                     nbdkit failed, bug!!
>  > > 
>  > >   64k      zlib         3164667904       qemu   3.579 s ±  0.094 s
>  > >   64k      zstd         3132686336       qemu   1.770 s ±  0.060 s
>  > > 
>  > >   64k      zlib         3164667904       nbdkit 1.254 s ±  0.065 s
>  > >   64k      zstd         3132686336       nbdkit 1.315 s ±  0.037 s
>  > > 
>  > >   512k     zlib         3158744576       qemu   4.008 s ±  0.058 s
>  > >   512k     zstd         3032697344       qemu   1.503 s ±  0.072 s
>  > > 
>  > >   512k     zlib         3158744576       nbdkit 1.702 s ±  0.026 s
>  > >   512k     zstd         3032697344       nbdkit 1.593 s ±  0.039 s
>  > > 
>  > >   2048k    zlib         3197569024       qemu   4.327 s ±  0.051 s
>  > >   2048k    zstd         2995143168       qemu   1.465 s ±  0.085 s
>  > > 
>  > >   2048k    zlib         3197569024       nbdkit 3.660 s ±  0.011 s
>  > >   2048k    zstd         2995143168       nbdkit 3.368 s ±  0.057 s
>  > > 
>  > > No great surprises - very small cluster size is inefficient, but
>  > > scaling up the cluster size gain performance, and zstd performs better
>  > > than zlib once the cluster size is sufficiently large.
>  > 
>  > The default qcow2 cluster size is 64k, which means we've already
>  > got the vast majority of the perfornmance and file size win. Going
>  > beyond 64k defaults doesn't seem massively compelling.
>  > 
>  > zstd does have a small space win over zlib as expected, but again
>  > nothing so drastic that it seems compelling to change - that win
>  > will be line noise in the overall bigger picture of image storage
>  > and download times.
>  
>  Yeah, I was a bit surprised by this.  I expected zstd files to be
>  significantly smaller than zlib even though that's not what zstd is
>  optimized for.  Not that they'd be about the same.
>  
>  > The major difference here is that zstd is much faster than zlib
>  > at decompress. I'd be curious if zlib-ng closes that gap ?
>  
>  It's quite hard to use zlib-ng in Fedora (currently) since it requires
>  changes to the source code.  That is what the pull request being
>  discussed would change, as you could simply install zlib-ng-compat
>  which would replace libz.so.  But anyway I can't easily get results
>  for qemu + zlib-ng, but we expect it would be ~ 40% faster at
>  decompression, and decompression is what is taking most of the time in
>  the qemu numbers above.
>  
>  I forgot to say that nbdkit is using zlib-ng, since I made the source
>  level changes a few weeks back (but most of the nbdkit performance
>  improvement comes from being able to use lots of threads). 
Ah that last point is interesting. If we look at nbdkit results we can
see that while zstd is clearly faster, the margin of the win is massively
lower. So I presume we can infer similar margins if qemu-img were
switched too.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 11:06:20AM +0100, Daniel P. Berrangé wrote:
>  On Fri, Sep 01, 2023 at 11:03:54AM +0100, Richard W.M. Jones wrote:
>  > I forgot to say that nbdkit is using zlib-ng, since I made the source
>  > level changes a few weeks back (but most of the nbdkit performance
>  > improvement comes from being able to use lots of threads).
>  
>  Ah that last point is interesting. If we look at nbdkit results we can
>  see that while zstd is clearly faster, the margin of the win is massively
>  lower. So I presume we can infer similar margins if qemu-img were
>  switched too. 
FWIW here's the nbdkit commit which added zlib-ng support (assuming
you don't want to wait for a compat package).  It's not a massive
change so something similar might be done for qemu:

https://gitlab.com/nbdkit/nbdkit/-/commit/1b67e323e998a5d719f1afe43d5be84...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

zlib-ng as a compat replacement for zlib

[Kevin Wolf]

Am 01.09.2023 um 12:03 hat Richard W.M. Jones geschrieben:
>  On Fri, Sep 01, 2023 at 10:55:50AM +0100, Daniel P. Berrangé wrote:
>  > On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>  > > On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
>  > > > I understand the context and didn't really think about file size at
> all.
>  > > > 
>  > > > My question was essentially if decompressing many small blocks (as we do
>  > > > today) performs significantly different from decompressing fewer larger
>  > > > blocks (as we would do with a larger cluster size).
>  > > 
>  > > I did a quick test just by adjusting the cluster size of a qcow2 file:
>  > > 
>  > >   $ virt-builder fedora-36
>  > >   $ ls -lsh fedora-36.img 
>  > >   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
>  > >   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
>  > >   $ ls -lsh test.raw 
>  > >   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
>  > >   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o
> compression_type=zlib,cluster_size=4096
>  > > 
>  > > (for cluster sizes 4k, 64k, 512k, 2048k, and
>  > > compression types zlib & zstd)
>  > > 
>  > > I tested the speed of decompression using:
>  > > 
>  > >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw
> test.out'
>  > >   (qemu 8.0.0-4.fc39.x86_64)
>  > > 
>  > >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>  > >   (nbdkit-1.35.11-2.fc40.x86_64)
>  > > 
>  > > Results:
>  > > 
>  > >   Cluster  Compression  Compressed size  Prog   Decompression speed
>  > > 
>  > >   4k       zlib         3228811264       qemu   5.921 s ±  0.074 s
>  > >   4k       zstd         3258097664       qemu   5.189 s ±  0.158 s
>  > > 
>  > >   4k       zlib/zstd                     nbdkit failed, bug!!
>  > > 
>  > >   64k      zlib         3164667904       qemu   3.579 s ±  0.094 s
>  > >   64k      zstd         3132686336       qemu   1.770 s ±  0.060 s
>  > > 
>  > >   64k      zlib         3164667904       nbdkit 1.254 s ±  0.065 s
>  > >   64k      zstd         3132686336       nbdkit 1.315 s ±  0.037 s
>  > > 
>  > >   512k     zlib         3158744576       qemu   4.008 s ±  0.058 s
>  > >   512k     zstd         3032697344       qemu   1.503 s ±  0.072 s
>  > > 
>  > >   512k     zlib         3158744576       nbdkit 1.702 s ±  0.026 s
>  > >   512k     zstd         3032697344       nbdkit 1.593 s ±  0.039 s
>  > > 
>  > >   2048k    zlib         3197569024       qemu   4.327 s ±  0.051 s
>  > >   2048k    zstd         2995143168       qemu   1.465 s ±  0.085 s
>  > > 
>  > >   2048k    zlib         3197569024       nbdkit 3.660 s ±  0.011 s
>  > >   2048k    zstd         2995143168       nbdkit 3.368 s ±  0.057 s
>  > > 
>  > > No great surprises - very small cluster size is inefficient, but
>  > > scaling up the cluster size gain performance, and zstd performs better
>  > > than zlib once the cluster size is sufficiently large. 
It's interesting that for zstd, qemu-img keeps getting better with
increasing cluster size, while zlib numbers get worse again above 64k.
And nbdkit seems to get worse instead of better with larger cluster
size, no matter whether zlib or zstd is used.

>  > The default qcow2 cluster size is 64k, which means we've
> already
>  > got the vast majority of the perfornmance and file size win. Going
>  > beyond 64k defaults doesn't seem massively compelling.
>  > 
>  > zstd does have a small space win over zlib as expected, but again
>  > nothing so drastic that it seems compelling to change - that win
>  > will be line noise in the overall bigger picture of image storage
>  > and download times.
>  
>  Yeah, I was a bit surprised by this.  I expected zstd files to be
>  significantly smaller than zlib even though that's not what zstd is
>  optimized for.  Not that they'd be about the same.
>  
>  > The major difference here is that zstd is much faster than zlib
>  > at decompress. I'd be curious if zlib-ng closes that gap ?
>  
>  It's quite hard to use zlib-ng in Fedora (currently) since it requires
>  changes to the source code.  That is what the pull request being
>  discussed would change, as you could simply install zlib-ng-compat
>  which would replace libz.so.  But anyway I can't easily get results
>  for qemu + zlib-ng, but we expect it would be ~ 40% faster at
>  decompression, and decompression is what is taking most of the time in
>  the qemu numbers above.
>  
>  I forgot to say that nbdkit is using zlib-ng, since I made the source
>  level changes a few weeks back (but most of the nbdkit performance
>  improvement comes from being able to use lots of threads). 
Ah, that's actually a very important detail. I was wondering why zlib
was performing so much better in nbdkit when zstd is more or less
comparable in both.

If you think using more threads is the key for the remaining difference
at 64k, would increasing QCOW2_MAX_THREADS (currently only 4) help on
the qemu-img side?

>  > If it does, then for the sake of image portability it'd be
> better
>  > to stick with zlib compression in qcow2 and leverage zlib-ng for
>  > speed, and ignore zstd. 
At the first sight, it looks like this would already help a lot. I guess
we'd have to create a PoC patch and measure it.

Kevin

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 01:10:16PM +0200, Kevin Wolf wrote:
>  And nbdkit seems to get worse instead of better with larger cluster
>  size, no matter whether zlib or zstd is used. 
It's caused by nbdcopy's default request size being 256k.  Increasing
it to 2M cures the scaling problem - see updated results below.  (Note
nbdkit & nbdcopy are being used together so we're copying between two
programs.  The request size is the size of NBD requests between the two.)

>  If you think using more threads is the key for the remaining
> difference
>  at 64k, would increasing QCOW2_MAX_THREADS (currently only 4) help on
>  the qemu-img side? 
Results:

  qemu800 = qemu-img-8.0.0-4.fc39.x86_64 [previous results]
  qemugit = qemu @ 17780edd81d
  qemuthr = qemu @ 17780edd81d with QCOW2_MAX_THREADS changed from 4 to 16
  nbdkit = nbdkit-1.35.11-2.fc40.x86_64 [previous results]
  nbdkit2M = nbdkit with nbdcopy --request-size=$((2*1024*1024))


Cluster  Compression  Compressed size  Prog      Decompression speed

4k       zlib         3228811264       qemu800   5.921 s ±  0.074 s
4k       zstd         3258097664       qemu800   5.189 s ±  0.158 s

4k       zlib         3228811264       qemugit   7.021 s ±  0.234 s
4k       zstd         3258097664       qemugit   6.594 s ±  0.170 s

4k       zlib         3228811264       qemuthr   6.744 s ±  0.111 s
4k       zstd         3258097664       qemuthr   6.428 s ±  0.206 s

4k       zlib         3228811264       nbdkit    1.390 s ±  0.094 s
4k       zstd         3258097664       nbdkit    1.328 s ±  0.055 s


64k      zlib         3164667904       qemu800   3.579 s ±  0.094 s
64k      zstd         3132686336       qemu800   1.770 s ±  0.060 s

64k      zlib         3164667904       qemugit   3.644 s ±  0.018 s
64k      zstd         3132686336       qemugit   1.814 s ±  0.098 s

64k      zlib         3164667904       qemuthr   1.356 s ±  0.058 s
64k      zstd         3132686336       qemuthr   1.266 s ±  0.064 s

64k      zlib         3164667904       nbdkit    1.254 s ±  0.065 s
64k      zstd         3132686336       nbdkit    1.315 s ±  0.037 s


512k     zlib         3158744576       qemu800   4.008 s ±  0.058 s
512k     zstd         3032697344       qemu800   1.503 s ±  0.072 s

512k     zlib         3158744576       qemugit   4.015 s ±  0.040 s
512k     zstd         3032697344       qemugit   1.557 s ±  0.025 s

512k     zlib         3158744576       qemuthr   1.233 s ±  0.050 s
512k     zstd         3032697344       qemuthr   1.149 s ±  0.032 s

512k     zlib         3158744576       nbdkit    1.702 s ±  0.026 s
512k     zstd         3032697344       nbdkit    1.593 s ±  0.039 s


2048k    zlib         3197569024       qemu800   4.327 s ±  0.051 s
2048k    zstd         2995143168       qemu800   1.465 s ±  0.085 s

2048k    zlib         3197569024       qemugit   4.323 s ±  0.031 s
2048k    zstd         2995143168       qemugit   1.484 s ±  0.067 s

2048k    zlib         3197569024       qemuthr   1.299 s ±  0.055 s
2048k    zstd         2995143168       qemuthr   1.229 s ±  0.046 s

2048k    zlib         3197569024       nbdkit2M  1.636 s ±  0.071 s
2048k    zstd         2995143168       nbdkit2M  1.644 s ±  0.040 s


Increasing the number of threads makes a big difference, so I think
changing the default (or making it run-time adjustable somehow) is a
good idea, also an easy win.

Increased qcow2 threads + zlib-ng would be _very_ interesting.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

zlib-ng as a compat replacement for zlib

[Richard Shaw]

Drive by comment because I found this thread interesting...

What compression level was used for zstd? My understanding is that higher
levels take longer to compress but decompression time remains virtually the
same.

Thanks,
Richard

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 07:10:59AM -0500, Richard Shaw wrote:
>  Drive by comment because I found this thread interesting...
>  
>  What compression level was used for zstd? My understanding is that higher
>  levels take longer to compress but decompression time remains virtually the
>  same. 
Seems to be the default level?  Or at least it doesn't seem like it is
being set.  Can you tell from this code:

https://gitlab.com/qemu-project/qemu/-/blob/17780edd81d27fcfdb7a802efc870...

Rich.

>  Thanks,
>  Richard 
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

zlib-ng as a compat replacement for zlib

[Richard Shaw]

On Fri, Sep 1, 2023 at 9:02 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:

>  On Fri, Sep 01, 2023 at 07:10:59AM -0500, Richard Shaw wrote:
>  > Drive by comment because I found this thread interesting...
>  >
>  > What compression level was used for zstd? My understanding is that higher
>  > levels take longer to compress but decompression time remains virtually
>  the
>  > same.
>
>  Seems to be the default level?  Or at least it doesn't seem like it is
>  being set.  Can you tell from this code:
>
>
>
> https://gitlab.com/qemu-project/qemu/-/blob/17780edd81d27fcfdb7a802efc870...

Yeah didn't see anything referencing the `compression_level` property, and
the default is 3. If compression performance doesn't matter as much I would
try increasing it to maybe 9? The levels are 1 to 22 but I'm sure there's a
diminishing return.

Thanks,
Richard

zlib-ng as a compat replacement for zlib

[Daniel Alley]

Yeah, the Zstd default of 3 is tuned to be roughly on par with alternatives but much
faster.  To get better compression than gzip you generally need to go to higher levels.

zlib-ng as a compat replacement for zlib

[Leslie Satenstein]

Thank you. 


Leslie Satenstein    
 

    On Saturday, September 2, 2023 at 09:29:36 a.m. EDT, Daniel Alley
<dalley(a)redhat.com&gt; wrote:  
 
 Yeah, the Zstd default of 3 is tuned to be roughly on par with alternatives but much
faster.  To get better compression than gzip you generally need to go to higher levels.  
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>  Results:
>  
>    Cluster  Compression  Compressed size  Prog   Decompression speed
>  
>    4k       zlib         3228811264       qemu   5.921 s ±  0.074 s
>    4k       zstd         3258097664       qemu   5.189 s ±  0.158 s
>  
>    4k       zlib/zstd                     nbdkit failed, bug!! 
Stupid bounds checking bug, fixed now [1]:

    4k       zlib         3228811264       nbdkit 1.390 s ±  0.094 s
    4k       zstd         3258097664       nbdkit 1.328 s ±  0.055 s

>    64k      zlib         3164667904       qemu   3.579 s ±  0.094 s
>    64k      zstd         3132686336       qemu   1.770 s ±  0.060 s
>  
>    64k      zlib         3164667904       nbdkit 1.254 s ±  0.065 s
>    64k      zstd         3132686336       nbdkit 1.315 s ±  0.037 s
>  
>    512k     zlib         3158744576       qemu   4.008 s ±  0.058 s
>    512k     zstd         3032697344       qemu   1.503 s ±  0.072 s
>  
>    512k     zlib         3158744576       nbdkit 1.702 s ±  0.026 s
>    512k     zstd         3032697344       nbdkit 1.593 s ±  0.039 s
>  
>    2048k    zlib         3197569024       qemu   4.327 s ±  0.051 s
>    2048k    zstd         2995143168       qemu   1.465 s ±  0.085 s
>  
>    2048k    zlib         3197569024       nbdkit 3.660 s ±  0.011 s
>    2048k    zstd         2995143168       nbdkit 3.368 s ±  0.057 s 
[1] https://gitlab.com/nbdkit/nbdkit/-/commit/73b58dc13dd7a3bbc7e7b3b718a535e...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

zlib-ng as a compat replacement for zlib

[Florian Weimer]

* Richard W. M. Jones:

>  I tested the speed of decompression using:
>
>    $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw
> test.out'
>    (qemu 8.0.0-4.fc39.x86_64)
>
>    $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>    (nbdkit-1.35.11-2.fc40.x86_64) 
How realistic is that?  Larger cluster sizes will make random access
perform noticeably worse is some cases.  Think about reading a few bytes
towards the end of the cluster.  It makes a difference whether you have
to decompress 64 KiB bytes for that, or 2 MiB.  As far as I understand
it, the above commands use all data decompressed, so they don't suffer
from this issue (particularly with read-ahead to deal with unfortunate
cluster boundaries).

Time to first HTTP request served after boot or something like that
might be a better comparison.

Thanks,
Florian

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 06:45:22PM +0200, Florian Weimer wrote:
>  * Richard W. M. Jones:
>  
>  > I tested the speed of decompression using:
>  >
>  >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw
> test.out'
>  >   (qemu 8.0.0-4.fc39.x86_64)
>  >
>  >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>  >   (nbdkit-1.35.11-2.fc40.x86_64)
>  
>  How realistic is that?  Larger cluster sizes will make random access
>  perform noticeably worse is some cases.  Think about reading a few bytes
>  towards the end of the cluster.  It makes a difference whether you have
>  to decompress 64 KiB bytes for that, or 2 MiB.  As far as I understand
>  it, the above commands use all data decompressed, so they don't suffer
>  from this issue (particularly with read-ahead to deal with unfortunate
>  cluster boundaries).
>  
>  Time to first HTTP request served after boot or something like that
>  might be a better comparison. 
Yes, this is a good point.

Current Fedora images use 64k cluster size which I think is also the
default:

$ qemu-img info
https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86...

image:
https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86...
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: unavailable
cluster_size: 65536                    <----
Format specific information:
    compat: 0.10
    compression type: zlib
    refcount bits: 16

And the tests showed there's barely any performance gain above the
default cluster size anyway.  Only very small clusters should be
avoided, but there are good reasons to avoid those already such as
metadata overhead and small maximum image size.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top