3:32 p.m.
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
== Summary ==
Fedora users / developers who need to debug/trace distro binaries can
make use of the recently activated elfutils-debuginfod servers to
automatically fetch debugging data and source code, instead of having
to use `# sudo dnf` commands.
== Owner ==
* Name: [[User:fche| Frank Ch. Eigler]]
* Email: fche(a)redhat.com
== Detailed Description ==
Numerous fedora debugging-type tools have built-in capabilities to use
the [https://sourceware.org/elfutils/Debuginfod.html debuginfod]
protocol to fetch debuginfo/source code automatically. We would like
to activate a setting so that Fedora's debuginfod servers are
automatically used, rather than requiring hand-editing individual
users' dot files.
== Feedback ==
There has existed
[https://www.spinics.net/lists/fedora-devel/msg279002.html broad
interest] in a Fedora debuginfod server since the project was proposed
/ announced in 2020, and several distros already operate public
servers of this nature. Some of the distros configure their
installations by default to talk to those servers, some do not.
Turning on this by default has some limited privacy implications.
Some Debian users have
[https://lists.debian.org/debian-devel/2021/02/msg00262.html expressed
concerns] that this facility "calls home" during debugging, so it may
expose a limited amount of information about what a user is debugging.
The information is limited to the build-id and source code file names
of programs being debugged, and is only sent to the servers if their
machine lacks locally installed debuginfo. Whether this should be
opt-in or opt-out and how has been resolved there via an install-time
query to the sysadmin. In contrast, on OpenSUSE Tumbleweed, it is
[https://build.opensuse.org/request/show/879926 simply defaulted-on],
and we have heard of no controversy.
We anticipate discussing this topic on the mailing list, and noting it
in the release notes either way.
== Benefit to Fedora ==
This will improve developers' experience.
It may reduce download server burden, as only individual
ELF/DWARF/source files are downloaded rather than entire `-debuginfo`
and `-debugsource` RPMs.
It would help Fedora catch up to other distros who put up `debuginfod`
servers already. :-)
== Scope ==
* Proposal owners:
The work could consist one extra parameter in the `elfutils.spec`
`%configure`. Its effect is to arrange for the
`elfutils-debuginfod-client` RPM
to install an `/etc/profile.d` file that sets the `DEBUGINFOD_URLS`
environment variable automatically to
`https://debuginfod.fedoraproject.org/`. (At the time of this
writing, the _staging_ server is getting ready for testing:
`https://debuginfod.stg.fedoraproject.org/`.)
* Other developers: None - relevant code has been previously upstreamed!
* Release engineering: None - our team is operating the
`debuginfod[.stg].fedoraproject.org` VMs.
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
None.
Note that these servers index all active Fedora releases (32-), all
architectures, so users of those versions can already set
`DEBUGINFOD_URLS` manually to take advantage.
== How To Test ==
* Install `elfutils-debuginfod-client`
* Open arbitrary fedora binary via gdb.
** Admire the immediate downloading of debuginfo and source code.
* Run `eu-stack -v -p $pid` for an arbitrary process.
** Admire the immediate downloading of debuginfo to give precise file:line data.
== User Experience ==
Primarily: users running debuggers, profilers, tracing tools on
internet-capable machines can work immediately, without switching to
privileged users and fragile manual `dnf` commands to install this
data.
== Dependencies ==
The `debuginfod` servers at `fedora-infra` need to be up.
== Contingency Plan ==
* Contingency mechanism: change the elfutils-debuginfod-client subrpm
to not set the default `DEBUGINFOD_URLS` environment variable for all
users. In the case of a server outage, the debugger tools revert to
debuginfo-less operation, prior to this feature.
* Contingency deadline: shortly before freeze
* Blocks release? No
== Documentation ==
There is upstream documentation in the debugging tools as well as
associated with the client code / cli tooling. What our Release Notes
would focus on however is the _automatic activation_ of this facility
via the environment variable.
== Release Notes ==
TBD.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3:54 p.m.
On Wed, Apr 7, 2021 at 4:33 PM Ben Cotton <bcotton(a)redhat.com> wrote:
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
== Summary ==
Fedora users / developers who need to debug/trace distro binaries can
make use of the recently activated elfutils-debuginfod servers to
automatically fetch debugging data and source code, instead of having
to use `# sudo dnf` commands.
== Owner ==
* Name: [[User:fche| Frank Ch. Eigler]]
* Email: fche(a)redhat.com
== Detailed Description ==
Numerous fedora debugging-type tools have built-in capabilities to use
the [https://sourceware.org/elfutils/Debuginfod.html debuginfod]
protocol to fetch debuginfo/source code automatically. We would like
to activate a setting so that Fedora's debuginfod servers are
automatically used, rather than requiring hand-editing individual
users' dot files.
== Feedback ==
There has existed
[https://www.spinics.net/lists/fedora-devel/msg279002.html broad
interest] in a Fedora debuginfod server since the project was proposed
/ announced in 2020, and several distros already operate public
servers of this nature. Some of the distros configure their
installations by default to talk to those servers, some do not.
Turning on this by default has some limited privacy implications.
Some Debian users have
[https://lists.debian.org/debian-devel/2021/02/msg00262.html expressed
concerns] that this facility "calls home" during debugging, so it may
expose a limited amount of information about what a user is debugging.
The information is limited to the build-id and source code file names
of programs being debugged, and is only sent to the servers if their
machine lacks locally installed debuginfo. Whether this should be
opt-in or opt-out and how has been resolved there via an install-time
query to the sysadmin. In contrast, on OpenSUSE Tumbleweed, it is
[https://build.opensuse.org/request/show/879926 simply defaulted-on],
and we have heard of no controversy.
We anticipate discussing this topic on the mailing list, and noting it
in the release notes either way.
== Benefit to Fedora ==
This will improve developers' experience.
It may reduce download server burden, as only individual
ELF/DWARF/source files are downloaded rather than entire `-debuginfo`
and `-debugsource` RPMs.
It would help Fedora catch up to other distros who put up `debuginfod`
servers already. :-)
== Scope ==
* Proposal owners:
The work could consist one extra parameter in the `elfutils.spec`
`%configure`. Its effect is to arrange for the
`elfutils-debuginfod-client` RPM
to install an `/etc/profile.d` file that sets the `DEBUGINFOD_URLS`
environment variable automatically to
`https://debuginfod.fedoraproject.org/`. (At the time of this
writing, the _staging_ server is getting ready for testing:
`https://debuginfod.stg.fedoraproject.org/`.)
* Other developers: None - relevant code has been previously upstreamed!
* Release engineering: None - our team is operating the
`debuginfod[.stg].fedoraproject.org` VMs.
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
None.
Note that these servers index all active Fedora releases (32-), all
architectures, so users of those versions can already set
`DEBUGINFOD_URLS` manually to take advantage.
== How To Test ==
* Install `elfutils-debuginfod-client`
* Open arbitrary fedora binary via gdb.
** Admire the immediate downloading of debuginfo and source code.
* Run `eu-stack -v -p $pid` for an arbitrary process.
** Admire the immediate downloading of debuginfo to give precise file:line data.
== User Experience ==
Primarily: users running debuggers, profilers, tracing tools on
internet-capable machines can work immediately, without switching to
privileged users and fragile manual `dnf` commands to install this
data.
== Dependencies ==
The `debuginfod` servers at `fedora-infra` need to be up.
== Contingency Plan ==
* Contingency mechanism: change the elfutils-debuginfod-client subrpm
to not set the default `DEBUGINFOD_URLS` environment variable for all
users. In the case of a server outage, the debugger tools revert to
debuginfo-less operation, prior to this feature.
* Contingency deadline: shortly before freeze
* Blocks release? No
== Documentation ==
There is upstream documentation in the debugging tools as well as
associated with the client code / cli tooling. What our Release Notes
would focus on however is the _automatic activation_ of this facility
via the environment variable.
== Release Notes ==
TBD.
Woohoo! I'm excited for this!
I got a chance to use debuginfod to do some debugging of DNF on
openSUSE last Saturday and the experience was fantastic.
I'm looking forward to this being wired up into GDB, ABRT, and everything else!
--
真実はいつも一つ！/ Always, there's only one truth!
10:19 a.m.
ngompa13 wrote:
Woohoo! I'm excited for this!
Thanks!
I got a chance to use debuginfod to do some debugging of DNF on
openSUSE last Saturday and the experience was fantastic.
I'm looking forward to this being wired up into GDB, ABRT, and
everything else
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable
that way. (The staging indexer is processing the l* packages, after a
few days of churning from 0* through k*. It has a one or two more days
to go, but may already be used.)
The purpose of this Change is to configure this environment variable by
default, pointing to the forthcoming production version of the server
at https://debuginfod.fedoraproject.org/.
- FChE
11:06 a.m.
On Thu, Apr 8, 2021 at 11:20 AM Frank Ch. Eigler <fche(a)redhat.com> wrote:
ngompa13 wrote:
> Woohoo! I'm excited for this!
Thanks!
> I got a chance to use debuginfod to do some debugging of DNF on
> openSUSE last Saturday and the experience was fantastic.
>
> I'm looking forward to this being wired up into GDB, ABRT, and
> everything else
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable
that way. (The staging indexer is processing the l* packages, after a
few days of churning from 0* through k*. It has a one or two more days
to go, but may already be used.)
The purpose of this Change is to configure this environment variable by
default, pointing to the forthcoming production version of the server
at https://debuginfod.fedoraproject.org/.
We should probably have a halfway decent splash page for
debuginfod.fedoraproject.org, similar to what the openSUSE one has:
https://debuginfod.opensuse.org/
--
真実はいつも一つ！/ Always, there's only one truth!
9:19 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Thu, Apr 08, 2021 at 11:19:59AM -0400, Frank Ch. Eigler wrote:
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable
that way.
Really cool. Thanks for making it happen.
9:54 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Fri, Apr 9 2021 at 04:19:31 PM +0200, Lars Seipel <ls(a)slrz.net>
wrote:
Really cool. Thanks for making it happen.
I started testing the staging server yesterday. It seems a little slow
-- I worry for anyone debugging anything that links to WebKit, which on
the desktop is a lot -- but otherwise it works *very* well. I'm
impressed. Very nice.
2:50 p.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change proposal)
mcatanzaro wrote:
I started testing the staging server yesterday. It seems a little
slow
-- I worry for anyone debugging anything that links to WebKit, which
on the desktop is a lot -- but otherwise it works *very* well. I'm
impressed. Very nice.
It'd be good to get a sense of what you found slow. The .stg. server is
just finishing up ingesting all the koji rpms, so has been fully
occupied with that. In the case where a new file is sought from inside
some large rpm, yeah there will be some inherent latency in
decompressing it (CPU bound), saving the selected file to tmp disk
(RAM/storage bound) and then sending you the file (network bound).
At the server, there are some prefetching/caching options available to
take advantage of the spacious servers kindly provided by fedora-infra.
There is also aggressive caching on the client side, with an adjustable
week-long retention for files. See the CACHE section in [man
debuginfod_find_debuginfo]. Plus web caching proxies or federated
intermediate debuginfod servers can add site-level caching near you.
- FChE
7:03 a.m.
On Fri, Apr 9, 2021 at 11:01 AM Michael Catanzaro <mcatanzaro(a)gnome.org> wrote:
On Fri, Apr 9 2021 at 04:19:31 PM +0200, Lars Seipel <ls(a)slrz.net>
wrote:
> Really cool. Thanks for making it happen.
I started testing the staging server yesterday. It seems a little slow
-- I worry for anyone debugging anything that links to WebKit, which on
the desktop is a lot -- but otherwise it works *very* well. I'm
impressed. Very nice.
Did you notice that it also works for the Fedora Flatpaks (thanks,
Frank!) - basic proof of concept:
$ flatpak run --command=sh --filesystem=home --share=network --devel
org.gnome.Aisleriot
[📦 org.gnome.Aisleriot ~]$
DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
(Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache
ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
- Owen
8:49 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Sat, Apr 10 2021 at 08:03:09 AM -0400, Owen Taylor
<otaylor(a)redhat.com> wrote:
Did you notice that it also works for the Fedora Flatpaks (thanks,
Frank!) - basic proof of concept:
$ flatpak run --command=sh --filesystem=home --share=network --devel
org.gnome.Aisleriot
[📦 org.gnome.Aisleriot ~]$
DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
(Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache
ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
I think that's OK for a manual debugging workflow, since it's pretty
rare to want to do that under flatpak in my experience. Normally what's
most important to me is being able to easily generate a backtrace for a
previous crash using 'flatpak-coredumpctl'. Ideally
'flatpak-coredumpctl' would handle setting the right environment
variables and executing flatpak with the right permissions to make it
work. (In the future, ABRT could do something similar.)
12:52 p.m.
On Sat, Apr 10, 2021 at 9:55 AM Michael Catanzaro <mcatanzaro(a)gnome.org>
wrote:
On Sat, Apr 10 2021 at 08:03:09 AM -0400, Owen Taylor
<otaylor(a)redhat.com> wrote:
> Did you notice that it also works for the Fedora Flatpaks (thanks,
> Frank!) - basic proof of concept:
>
> $ flatpak run --command=sh --filesystem=home --share=network --devel
> org.gnome.Aisleriot
> [📦 org.gnome.Aisleriot ~]$
> DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
> DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
>
> (Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache
> ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
I think that's OK for a manual debugging workflow, since it's pretty
rare to want to do that under flatpak in my experience. Normally what's
most important to me is being able to easily generate a backtrace for a
previous crash using 'flatpak-coredumpctl'. Ideally
'flatpak-coredumpctl' would handle setting the right environment
variables and executing flatpak with the right permissions to make it
work. (In the future, ABRT could do something similar.)
I think we could store the debuginfo urls in repository metadata (ostree
summary / oci json index) and have flatpak automatically set things up for
'flatpak run --devel'. This isn't Fedora specific - e.g. there's an
eventual goal to have a debuginfo server for Flathub as well.
For coredump+debuginfod you can actually do the backtrace with a system (or
toolbox) gdb instead of downloading the SDK ... if we want to deal with
that complexity, we could add a mode like that into flatpak-coredumpctl.
One gap here is that it's a little hard to figure out what exact revision
of the Flatpak and runtime coredumped - you can reverse-engineer that out
of COREDUMP_PROC_MOUNTINFO information in the journal.
As you say, the eventual goal would be ABRT support.
Regards,
Owen
8:41 a.m.
On Sun, Apr 11, 2021 at 1:52 PM Owen Taylor <otaylor(a)redhat.com> wrote:
On Sat, Apr 10, 2021 at 9:55 AM Michael Catanzaro <mcatanzaro(a)gnome.org>
wrote:
> On Sat, Apr 10 2021 at 08:03:09 AM -0400, Owen Taylor
> <otaylor(a)redhat.com> wrote:
> > Did you notice that it also works for the Fedora Flatpaks (thanks,
> > Frank!) - basic proof of concept:
> >
> > $ flatpak run --command=sh --filesystem=home --share=network --devel
> > org.gnome.Aisleriot
> > [📦 org.gnome.Aisleriot ~]$
> > DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
> > DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
> >
> > (Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache
> > ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
>
> I think that's OK for a manual debugging workflow, since it's pretty
> rare to want to do that under flatpak in my experience. Normally what's
> most important to me is being able to easily generate a backtrace for a
> previous crash using 'flatpak-coredumpctl'. Ideally
> 'flatpak-coredumpctl' would handle setting the right environment
> variables and executing flatpak with the right permissions to make it
> work. (In the future, ABRT could do something similar.)
>
I think we could store the debuginfo urls in repository metadata (ostree
summary / oci json index) and have flatpak automatically set things up for
'flatpak run --devel'. This isn't Fedora specific - e.g. there's an
eventual goal to have a debuginfo server for Flathub as well.
Filed an upstream pull-request for this (still needs some work):
https://github.com/flatpak/flatpak/pull/4222
- Owen
11:25 p.m.
While trying to collect a backtrace for org.gnome.Tetravex, I got this in gdb:
===
Downloading separate debug info for /lib64/liblzma.so.5...
Download failed: Timer expired. Continuing without debug info for
/lib64/libbrotlicommon.so.1.
Missing separate debuginfo for /lib64/libbrotlicommon.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/0e/bb3270fdbf40dbe56ea79d6630ac594b897ffe.debug
Download failed: Timer expired. Continuing without debug info for /lib64/libzstd.so.1.
Missing separate debuginfo for /lib64/libzstd.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/33/70d80a1bf749b3c2baaad0188c864ee9e4bbc4.debug
Downloading separate debug info for /lib64/liblz4.so.1...
Download failed: Timer expired. Continuing without debug info for
/home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo.
Error while reading shared library symbols for /lib64/liblz4.so.1:
could not find '.gnu_debugaltlink' file for
/home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo
Downloading separate debug info for /lib64/libcap.so.2...
===
After running all the "Try" suggestions, the second run of gdb do not complain.
I wanted to know how I can collect the required backtrace so that I can attach it to the
bugzilla entry to help developers to looking at the bug.
Thanks a lot!
4:29 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
FUNG Chi Chuen Sampson wrote:
While trying to collect a backtrace for org.gnome.Tetravex, I got
this in gdb:
===
Downloading separate debug info for /lib64/liblzma.so.5...
Download failed: Timer expired. Continuing without debug info for
/lib64/libbrotlicommon.so.1.
Missing separate debuginfo for /lib64/libbrotlicommon.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/0e/bb3270fdbf40dbe56ea79d6630ac594b897ffe.debug
Download failed: Timer expired. Continuing without debug info for /lib64/libzstd.so.1.
Missing separate debuginfo for /lib64/libzstd.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/33/70d80a1bf749b3c2baaad0188c864ee9e4bbc4.debug
Downloading separate debug info for /lib64/liblz4.so.1...
Download failed: Timer expired. Continuing without debug info for
/home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo.
Error while reading shared library symbols for /lib64/liblz4.so.1:
could not find '.gnu_debugaltlink' file for
/home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo
Downloading separate debug info for /lib64/libcap.so.2...
I was wondering what the user experience would be like in such a
situation. Could you estimate how long you had to wait in total? Was
there a long delay before each "Timer expired" message, or only one
delay?
Björn Persson
1:37 p.m.
Björn Persson <Bjorn(a)xn--rombobjrn-67a.se> writes:
I was wondering what the user experience would be like in such a
situation. Could you estimate how long you had to wait in total? Was
there a long delay before each "Timer expired" message, or only one
delay?
Each outright-hung request could entail a $DEBUGINFOD_TIMEOUT (default
90s) wait. The common code does not "hold a grudge" against a server
that formerly timed out.
- FChE
2:11 p.m.
It is just a short delay then the "Try" suggestion is given.
My first run using my Notebook takes quite some time.
A few hours later, I run again using my Desktop, this time is very fast and no
"Try" given.
2:24 p.m.
I have run the same debug session using two different machines.
The first run, giving my "Try"s, takes much longer than the second run, which
gives no "Trys".
Just from impression:
1st run: From run to download finish, I will say it takes about 5+ minutes.
2nd run: ~1 minute
For each "Try" given, the delay is not obvious to me.
3:16 p.m.
"Sampson Fung" <sampsonfung(a)gmail.com> writes:
The first run, giving my "Try"s, takes much longer than the
second run, which gives no "Trys".
Just from impression:
1st run: From run to download finish, I will say it takes about 5+ minutes.
2nd run: ~1 minute
For each "Try" given, the delay is not obvious to me.
I'm pretty sure Sampson was affected by a proxy.stg.fedoraproject.org
misconfiguration problem that was fixed about an hour ago, so those
timeouts should not be happening any more. That means we'd be down to
normal service latencies ameliorated by caching effects.
A cute demonstration of the cost/benefit of this capability now in the
distro, I recently ran
% gdb /usr/bin/gnome-control-center
On a normal machine, you'll get no debuginfo and a suggestion to install
a wall-of-text list of RPMs as root.
On a debuginfod configured machine, you'll get gdb downloading ~400MB of
debuginfo (HTTP compressed -- decompresses to ~6GB) as rapidly as your
network connection allows. This could take some minutes, for the first
time. After that time, you get instant visibility into the entire
enormous gnome software stack, including LLVM, mesa, x11, samba, gst,
opengl, glib, etc. etc. etc. right down to the glibc assembly wrappers
for syscalls.
- FChE
9:26 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
Frank Ch. Eigler wrote:
"Sampson Fung" <sampsonfung(a)gmail.com> writes:
> The first run, giving my "Try"s, takes much longer than the second run,
which gives no "Trys".
> Just from impression:
> 1st run: From run to download finish, I will say it takes about 5+ minutes.
> 2nd run: ~1 minute
> For each "Try" given, the delay is not obvious to me.
I'm pretty sure Sampson was affected by a proxy.stg.fedoraproject.org
misconfiguration problem that was fixed about an hour ago, so those
timeouts should not be happening any more.
It is however a good illustration of how a network problem can destroy
the user experience. Five minutes is a long wait. I'm glad that we now
have this information.
Björn Persson
12:11 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
Björn Persson wrote on Thu, Apr 22, 2021 at 04:26:22PM +0200:
It is however a good illustration of how a network problem can
destroy
the user experience. Five minutes is a long wait. I'm glad that we now
have this information.
This is an old post but I just used debuginfod for the first time (on
debian actually (debuginfod 0.183-1, gdb 10.1-1.7) in case things have
changed - forgive me and please correct me if anything I've said is
significantly better on current version on rawhide), so figured I'd
share...
- My usecase admitedly wasn't a very nice one (graphical application,
info dll says I have 265 linked libraries...), but it felt very slow.
Looking at iftop the server was reliably sending me things at 1mbps
which isn't great but with the amount of data involved being slow can't
be helped, I guess.
What makes it really bad is that such applications link extra libraries
at runtime, so when I thought it was finally done and it got into the
breakpoints I had set, continuing a bit just ran into another wall of
downloads, but this also isn't debuginfod's wrongdoings.
Most of these libraries are nothing I care about. I understand
gdb tries to find debug infos for libraries it encounters when these are
loaded into the program's address space, but could it be possible to not
download anything at this point and only download debuginfos when they
are really used (the first time an address in such libraries appear in a
backtrace to get line number / local variables / etc) ?
I realize this isn't obvious, but it'd make my usage much more
appreciable -- downloading 4 or maybe 5 libraries for the area I'm
debugging in that monster instead of the full 265 of them.
- None of ^C, ^\, ^Z work, when I grew tired of waiting I had to switch
to another terminal and kill the gdb process.
It'd be great if e.g. ^C could skip the current download (or future
downloads too?) and continue debugging, or enter the (gdb) prompt like
it normally does -- iirc it went into that prompt after the download? or
batch of downloads I'm not sure, I just remember it took too long.
- unsetting DEBUGINFOD_URLS completely disables the debuginfod client,
it would have been nice to use the data available in the client cache
anyway.
Ultimately I set it to localhost so debuginfod would give up immediately
on new requests but still use previously downloaded data.
Perhaps there could be an "offline mode" of sorts?
I also agree with the security concerns given, debuginfos are usually
seen as trusted data and I wouldn't be surprised new bugs get found in
their parsing which could compromise developers since the server can
send whatever it wants to send.
I'm not going to be very helpful on that respect though as I don't have
any bright idea (distributing just checksums of all debug files in the
main package, so when debuginfod downloads something it can compare with
the checksum that was installed and verified locally by signed rpm?
Didn't reread the whole thread but I think something similar was
suggested), but that topic didn't get much discussion and it sounds
important to me.
All in all I think it's a very valuable tool, but it would be really
great if we could minimize the amount of data actually downloaded, and
establish some chain of trust.
Thanks,
--
Dominique
1:57 p.m.
Hi -
asmadeus wrote:
- My usecase admitedly wasn't a very nice one (graphical
application,
info dll says I have 265 linked libraries...), but it felt very slow.
Yes, a first-time download series for such a program is going to take
time. (Afterwards, it's cached.)
Looking at iftop the server was reliably sending me things at 1mbps
which isn't great but with the amount of data involved being slow can't
be helped, I guess.
Right. The fedora debuginfod server seems well situated on the net, so
can generally send you compressed DWARF files fast. (I'm getting 5ish
megabytes/s across https.)
[...] Most of these libraries are nothing I care about. [...] but
could it be possible to not download anything at this point and only
download debuginfos when they are really used [...]
This sounds like a worthwhile suggestion for upstream gdb. We will
follow up with them.
[...] - None of ^C, ^\, ^Z work, when I grew tired of waiting I had
to switch to another terminal and kill the gdb process. [...]
OK, in my experiments, ^C did work, so something must have broken here.
We will follow up with gdb folks.
- unsetting DEBUGINFOD_URLS completely disables the debuginfod
client,
it would have been nice to use the data available in the client cache
anyway. Ultimately I set it to localhost so debuginfod would give up
immediately on new requests but still use previously downloaded data.
Perhaps there could be an "offline mode" of sorts?
I'll document this trick on the wiki.
By the way, consider operating your own debuginfod (rawhide or 0.185)
server, federating to upstream. The gdb-gui-download-series process you
experienced could be maybe twice as fast, due to http connection
keepalive improvements. Maybe even faster, once gdb itself takes direct
advantage (patches there are pending). Plus you get a shared cache for
your network.
[... re. security ...] (distributing just checksums of all debug
files
in the main package, so when debuginfod downloads something it can
compare with the checksum that was installed and verified locally by
signed rpm?
That sounds like something that could fall out from this effort [1], when/if
comes to fruition. We in debuginfod land could naturally follow up with
interfacing & checking glue code.
[1] https://fedoraproject.org/wiki/Changes/Signed_RPM_Contents
All in all I think it's a very valuable tool, but it would be
really
great if we could minimize the amount of data actually downloaded, and
establish some chain of trust.
I'm hoping we can make some improvements between now and F35 release,
though practically all of these rely on changes to other projects.
- FChE
5:57 p.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
Frank Ch. Eigler wrote on Tue, May 25, 2021 at 02:57:54PM -0400:
> [...] Most of these libraries are nothing I care about. [...]
but
> could it be possible to not download anything at this point and only
> download debuginfos when they are really used [...]
This sounds like a worthwhile suggestion for upstream gdb. We will
follow up with them.
Thanks!
> [...] - None of ^C, ^\, ^Z work, when I grew tired of waiting I
had
> to switch to another terminal and kill the gdb process. [...]
OK, in my experiments, ^C did work, so something must have broken here.
We will follow up with gdb folks.
I just tried on fedora34, this works here so that must be related to the
version used on debian, thanks for pointing out it works.
(For others: ^c cancels the download of a single object file, so might
need quite a few of these to eventually get to a gdb prompt if there are
many objects to download, but it works)
By the way, consider operating your own debuginfod (rawhide or
0.185)
server, federating to upstream. The gdb-gui-download-series process you
experienced could be maybe twice as fast, due to http connection
keepalive improvements. Maybe even faster, once gdb itself takes direct
advantage (patches there are pending). Plus you get a shared cache for
your network.
Interesting, I wasn'te aware of such a debuginfod proxifying mode.
Are there noticeable advantages over a simple caching http proxy
(e.g. squid) besides serving off in-house binaries debuginfo along the
way?
In my usecase this was a one-off thing for now, but I can see the point
if using it a lot.
> [... re. security ...] (distributing just checksums of all debug
files
> in the main package, so when debuginfod downloads something it can
> compare with the checksum that was installed and verified locally by
> signed rpm?
That sounds like something that could fall out from this effort [1], when/if
comes to fruition. We in debuginfod land could naturally follow up with
interfacing & checking glue code.
[1] https://fedoraproject.org/wiki/Changes/Signed_RPM_Contents
Hm, I guess debuginfod could distribute the signature files together
with the dwarf files straight from the rpm, but I'm a bit pessimistic
whether everyone will have IMA enabled even at relatively long term.
The change proposal does state that they won't actually enforce anything:
Note explicitly that we do not intend to install a default policy as
part of this change, and users will need to deploy their own policy
before anything is measured or appraised. This means that after this is
done, users will have the option to enable a policy and have that be
enforced, but there will be nothing automatic. We will, however,
document various example policies people can adapt to their needs.
Even when it does become standard many machines with developers will
want to continue letting users compile their own code, and these are
users likely to rely on debuggers/debuginfod to debug what they just
wrote, so would they be protected? (I'm not familiar with how that
works, now I'm reading up it looks possible to integrate with selinux to
require appraisal by context, but there are still many many machines
without selinux or equivalent in enforcing mode nowadays)
I guess I'd be fine with that if the debuginfod client implements the
signature verification in userspace regardless of the policy if
configured for it (I almost wrote if the server sends it, but that'd let
malicious server just pretend there is no signature available... heh),
other distros will probably take a bit more time to deploy but I guess
it will come together eventually.
Anyway, thanks a bunch for following up, it's appreciated.
--
Dominique
7:51 p.m.
Hi -
> > [...] Most of these libraries are nothing I care about.
[...] but
> > could it be possible to not download anything at this point and only
> > download debuginfos when they are really used [...]
>
> This sounds like a worthwhile suggestion for upstream gdb. We will
> follow up with them.
https://sourceware.org/bugzilla/show_bug.cgi?id=17547
> OK, in my experiments, ^C did work, so something must have broken
here.
> We will follow up with gdb folks.
[...]
(For others: ^c cancels the download of a single object file, so might
need quite a few of these to eventually get to a gdb prompt if there are
many objects to download, but it works)
https://sourceware.org/bugzilla/show_bug.cgi?id=27911
[...] Interesting, I wasn't aware of such a debuginfod
proxifying
mode.
Yes, we refer to it as "federation", and designed into debuginfod from
the beginning, so that one can establish a tree of servers, e.g. along
personal/group/corporate/public sorts of lines.
Are there noticeable advantages over a simple caching http proxy
(e.g. squid) besides serving off in-house binaries debuginfo along the
way? [...]
It should be about the same, except cache management policy is
different, and that debuginfod will pass requests to its upstream peers
*concurrently*, so this can create more redundancy and possibly
performance. By design, squid proxies or CDNs should interoperate just
fine with debuginfod and within federation trees.
[...]
> That sounds like something that could fall out from this effort [1], when/if
> comes to fruition. We in debuginfod land could naturally follow up with
> interfacing & checking glue code.
>
> [1] https://fedoraproject.org/wiki/Changes/Signed_RPM_Contents
Hm, I guess debuginfod could distribute the signature files together
with the dwarf files straight from the rpm, but I'm a bit pessimistic
whether everyone will have IMA enabled even at relatively long term.
The change proposal does state that they won't actually enforce
anything:
Right, I'm referring only to that if the per-file signatures were built
& packaged, we could trick the debuginfod client/server to use that for
manual signature checking, vs. relying on selinux/kernel enforcement.
I guess I'd be fine with that if the debuginfod client implements
the
signature verification in userspace regardless of the policy if
configured for it (I almost wrote if the server sends it, but that'd let
malicious server just pretend there is no signature available... heh),
(Good point!)
- FChE
1:10 p.m.
sampsonfung wrote:
While trying to collect a backtrace for org.gnome.Tetravex, I got
this in gdb:
[...]
Download failed: Timer expired. Continuing without debug info for /lib64/libzstd.so.1.
Missing separate debuginfo for /lib64/libzstd.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/33/70d80a1bf749b3c2baaad0188c864ee9e4bbc4.debug
[...]
We have had reports that from some corners of the internet (e.g. behind
the "great firewall"), that maintaining a waiting HTTPS connection for
10s of seconds was fragile. Extending the $DEBUGINFOD_TIMEOUT on the
other hand was sometimes reported to help.
https://sourceware.org/bugzilla/show_bug.cgi?id=27531
Looking over the system logs for that particular request (3370d80a...),
I'm seeing debuginfod.fedoraproject.org answer that particular buildid
in about tens of milliseconds (!) a bunch of times, and one failure to
transmit (usually a broken HTTP connection). If you could send me
(directly) the IP address from where you requested it, and the
$DEBUGINFOD_URLS you were using, I could look further in the logs. But
so far I see no sign of something triggering timeouts on this side.
- FChE
1:29 p.m.
"FUNG Chi Chuen Sampson" <sampsonfung(a)gmail.com> writes:
Downloading separate debug info for /lib64/liblzma.so.5...
Download failed: Timer expired. Continuing without debug info for
/lib64/libbrotlicommon.so.1.
Missing separate debuginfo for /lib64/libbrotlicommon.so.1
[...]
By the way, if you were using the staging (.stg.) debuginfod server,
please switch to the main one, which is in full service.
DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/
We're currently tracking down some infrastructure/proxy problem that
intermittently affects only the .stg. ones.
- FChE
3:20 p.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Thu, 2021-04-08 at 11:19 -0400, Frank Ch. Eigler wrote:
ngompa13 wrote:
> Woohoo! I'm excited for this!
Thanks!
> I got a chance to use debuginfod to do some debugging of DNF on
> openSUSE last Saturday and the experience was fantastic.
>
> I'm looking forward to this being wired up into GDB, ABRT, and
> everything else
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be
debuggable
that way. (The staging indexer is processing the l* packages, after
a
few days of churning from 0* through k*. It has a one or two more
days
to go, but may already be used.)
The purpose of this Change is to configure this environment variable
by
default, pointing to the forthcoming production version of the server
at https://debuginfod.fedoraproject.org/.
Something that occurred to me about this change is that as well as
sources and DWARF data, some of our debuginfo files contain python
scripts.
Back in Fedora 13 with:
https://fedoraproject.org/wiki/Features/EasierPythonDebugging
I added python scripts to python-debuginfo and python3-debuginfo so
that if you install them, gdb becomes gains type-specific pretty-
printers and extra commands relevant to what you're debugging.
That made sense to me back then, given that if you're debugging a
process you had to download the debuginfo package, so it made sense to
put the .py scripts in there so they would get autoadded (I don't see
any reference in my old wiki page to "add-auto-load-safe-path", so
maybe those directories would be regarded as safe automatically).
Having the sources and DWARF on-demand via debuginfod sounds great, and
hopefully means never having to manually install debuginfo rpms again -
but what about those .py files? I'm assuming debuginfod isn't
supplying those (is it?). I'm much more nervous about arbitrary python
scripts being supplied over this service, as the barrier to entry for
bad guys to do Bad Things would be so much lower as compared to
malformed DWARF, so perhaps if people want the .py files, they have to
install the debuginfo package in the traditional way? (It's still .py
files from rpm payloads, but having them autodownloaded with little
user involvement seems troublesome as compared to manually installing
debuginfo rpms).
Thoughts?
Dave
5:44 p.m.
dmalcolm wrote:
[...]
Something that occurred to me about this change is that as well as
sources and DWARF data, some of our debuginfo files contain python
scripts.
Because these files are stand-alone .py files rather than elf/dwarf
files, debuginfod has no way of serving those. If they were embedded
inside the DWARF file as a special section, then it could travel
implicitly.
[...]
Having the sources and DWARF on-demand via debuginfod sounds great, and
hopefully means never having to manually install debuginfo rpms again -
but what about those .py files?
If we were interested in serving them, we could do it by extending the
webapi protocol with a way to refer to a buildid-indexed
debuginfo-related file. Like
/buildid/BUILDID/debuginfo-gdb.py
[...] I'm much more nervous about arbitrary python scripts being
supplied over this service, as the barrier to entry for bad guys to do
Bad Things would be so much lower as compared to malformed DWARF [...]
Perhaps ... I'm not sure bad DWARF is inherently safer than bad Python.
Nevertheless, all this is based on the proposition that the files are
coming from a generally trusted build system, serving trusted artifacts
maintained by trusted individuals. If malevolent enough files get in
there, the service can be degraded and/or clients can be affected.
- FChE
9:39 p.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Fri, Apr 9 2021 at 04:20:16 PM -0400, David Malcolm
<dmalcolm(a)redhat.com> wrote:
Back in Fedora 13 with:
https://fedoraproject.org/wiki/Features/EasierPythonDebugging
I added python scripts to python-debuginfo and python3-debuginfo so
that if you install them, gdb becomes gains type-specific pretty-
printers and extra commands relevant to what you're debugging.
For glib, we put these pretty printers under the -devel package. That's
probably a better place for them.
9:39 p.m.
fche(a)redhat.com (Frank Ch. Eigler) writes:
The purpose of this Change is to configure this environment variable
by
default, pointing to the forthcoming production version of the server
at https://debuginfod.fedoraproject.org/.
With elfutils-0.184-4.fc35, this is now active for rawhide. If you're
interested in testing, please "# yum update", and try debugging things.
- FChE
5:34 p.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
The change page lacks a discussion of security implications. An informed
decision requires answers to questions such as:
· What kinds of attacks might be possible with malicious debuginfo files?
(For example debugging tools might have undiscovered bugs that could be
exploited by malformed DWARF data.)
· How is it verified that files received from debuginfo servers have not
been tampered with?
· Is there any end-to-end authentication from the Fedora build system to
my workstation – which there is with signed debuginfo packages – or do
the tools blindly trust a whole network of federated debuginfo servers?
Some Debian users have
[https://lists.debian.org/debian-devel/2021/02/msg00262.html expressed
concerns] that this facility "calls home" during debugging, so it may
expose a limited amount of information about what a user is debugging.
To fully understand the privacy implications, one needs to know:
· Does that happen every time, or are downloaded files cached locally?
· If there is a cache, when are old files purged from the cache?
The change page should also mention how a network problem can impact the
usability of debugging tools. Could it for example make GDB hang for a
minute every time it encounters a new source filename?
Finally, if somebody doesn't like the answers to the above questions,
then they'll want to know how to disable the feature.
Björn Persson
2:09 p.m.
Bjorn wrote:
> https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
The change page lacks a discussion of security implications. An informed
decision requires answers to questions such as: [...]
Thank you for your questions. Added a section and placed initial
answers there:
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault#Security_FAQ
- FChE
10:41 a.m.
Hi -
Björn Persson writes:
· How is it verified that files received from debuginfo servers have
not
been tampered with?
Following up further to this, we're planning to add optional client-side
hash-verification of cached content, to provide some protection against
tampering:
https://sourceware.org/bugzilla/show_bug.cgi?id=27758
- FChE
4:26 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
Frank Ch. Eigler wrote:
Björn Persson writes:
> · How is it verified that files received from debuginfo servers have not
> been tampered with?
Following up further to this, we're planning to add optional client-side
hash-verification of cached content, to provide some protection against
tampering:
https://sourceware.org/bugzilla/show_bug.cgi?id=27758
The design you propose there won't improve anything for anyone. If the
hash is computed on the debuginfo server, then an attacker who can make
the server serve malicious debuginfo can also make it serve hashes that
match the malicious files. And as you noted yourself, an attacker who
can manipulate cached files client-side has already taken over the user
account anyway.
Quote from Sourceware Bugzilla:
As transport over HTTPS protects the content, we can safely assume
that immediately during/after the download, the content will be fine.
However, what of cached files?
Of course – from your point of view. From my point of view, I can safely
assume that nobody has tampered with my cache. However, what of files on
the debuginfo server?
I see that debuginfod.fedoraproject.org is currently another name for
koji.fedoraproject.org. Given that it serves debuginfo only for Fedora
packages, and does not forward requests to any other debuginfo servers,
using this server seems equivalent security-wise to downloading unsigned
packages from Koji.
As far as I understand, packages are built and signed on separate
servers with a smaller attack surface than the web front-end to minimize
the risk that an attacker could tamper with them. To make the debuginfo
protocol as secure as signed debuginfo packages, the client should
verify the files against a hash computed and signed on the signing
server.
Perhaps a hash of the debuginfo could be stored in a signed RPM package,
either the main package or a separate debughash package?
For those who are concerned about privacy, the proposal would make that
problem worse as it would cause the "phoning home" to happen every time
they debug something.
By the way, the change page still doesn't say enough about how network
problems will affect the user experience. Making a previously offline
activity network-dependent also makes it sensitive to network problems.
For example, if great packet loss causes TCP timeouts or long delays,
will that make GDB hang for minutes at a time, or is it handled
asynchronously? Will GDB hang once per process, once per login session,
or every time it encounters a new source filename?
Björn Persson
10:37 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Wed, Apr 21, 2021 at 11:26:10AM +0200, Björn Persson wrote:
Frank Ch. Eigler wrote:
> Björn Persson writes:
>
> > · How is it verified that files received from debuginfo servers have not
> > been tampered with?
>
> Following up further to this, we're planning to add optional client-side
> hash-verification of cached content, to provide some protection against
> tampering:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=27758
The design you propose there won't improve anything for anyone. If the
hash is computed on the debuginfo server, then an attacker who can make
the server serve malicious debuginfo can also make it serve hashes that
match the malicious files. And as you noted yourself, an attacker who
can manipulate cached files client-side has already taken over the user
account anyway.
Quote from Sourceware Bugzilla:
> As transport over HTTPS protects the content, we can safely assume
> that immediately during/after the download, the content will be fine.
> However, what of cached files?
Of course – from your point of view. From my point of view, I can safely
assume that nobody has tampered with my cache. However, what of files on
the debuginfo server?
I see that debuginfod.fedoraproject.org is currently another name for
koji.fedoraproject.org. Given that it serves debuginfo only for Fedora
packages, and does not forward requests to any other debuginfo servers,
using this server seems equivalent security-wise to downloading unsigned
packages from Koji.
As far as I understand, packages are built and signed on separate
servers with a smaller attack surface than the web front-end to minimize
the risk that an attacker could tamper with them. To make the debuginfo
protocol as secure as signed debuginfo packages, the client should
verify the files against a hash computed and signed on the signing
server.
Yes, this is the scenario which I think is worrisome. This was also raised
during the FESCo meeting, and I want to clarify a bit.
A hypothetical attack through -debuginfo files would require gdb (or
other consumers of the debug data) to incorrectly handle corrupt debug
data. Even if we don't know any such cases right now, gdb and the
underlying libraries are written in C. We have a long history of
buffer overflows and other exploitable memory handling errors, and we
should assume that sooner or later some will be discovered in those
code paths too.
Currently, the -debuginfo packages are distributed as any other
package, i.e. they are built and signed on dedicated machines. A
modification anywhere at a later point would cause a signature
mismatch. The trust level for -debuginfo data is the same as any other
package. A hypothetical attacker who gained access to the package contents
*before signing* would probably be better off modifying executable code in
those packages, instead of a roundabout attack through debug data.
OTOH, the debuginfo files distributed through the debuginfod server
are not signed and there is no direct way to verify that they match
the (signed) contents of the debuginfo package. Thus, the debuginfod
server becomes a juicy target. There are a few things which make it
particularly attractive to an attacker: the debugger is very likely to
be ran directly from a developer account. The debugger is executed
without any sandboxing, and possibly even with elevated privileges
(when debugging system services). The debugger code was not written
with security it mind, so it's more likely to be exploitable than say
a web browser.
As to the debuginfod code itself, it is in C++, and has SQL and
threads, a http server, and also does bunch of low-level string
parsing. It is also fairly new code. I don't have any particular
knowledge about the code, but some exploit being found is not outside
of the realm of possibility.
Thus, to summarize: debuginfo files served over the web provide a new
fairly big attack surface, with attacks most likely leading to a
compromise of a developer or privileged account.
Zbyszek
Perhaps a hash of the debuginfo could be stored in a signed RPM package,
either the main package or a separate debughash package?
For those who are concerned about privacy, the proposal would make that
problem worse as it would cause the "phoning home" to happen every time
they debug something.
By the way, the change page still doesn't say enough about how network
problems will affect the user experience. Making a previously offline
activity network-dependent also makes it sensitive to network problems.
For example, if great packet loss causes TCP timeouts or long delays,
will that make GDB hang for minutes at a time, or is it handled
asynchronously? Will GDB hang once per process, once per login session,
or every time it encounters a new source filename?
Björn Persson
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
3:03 p.m.
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> writes:
OTOH, the debuginfo files distributed through the debuginfod server
are not signed and there is no direct way to verify that they match
the (signed) contents of the debuginfo package.
A direct way would be for someone to koji-download the given rpm, and
hand-extract/compare the files. (It's obviously not economical.)
Thus, the debuginfod server becomes a juicy target.
Yes. The Changes FAQ section discusses this topic.
Unfortunately, in the absence of per-file signatures generated by the
build system, and securely distributed out-of-band, I can't think of any
way to provide client-side verifiability of a debuginfod type service.
That's independent of any particular level of server code robustness.
Interestingly, if debuginfod were considered *trusted*, then it could be
used as the *basis* for such a capability, because the client-side hash
verification feature being prototyped. It would serve trusted hashes to
RPM-based artifacts on demand.
- FChE
3:28 p.m.
On Wed, Apr 21, 2021 at 4:09 PM Frank Ch. Eigler <fche(a)redhat.com> wrote:
A direct way would be for someone to koji-download the given rpm,
and
hand-extract/compare the files. (It's obviously not economical.)
> Thus, the debuginfod server becomes a juicy target.
Yes. The Changes FAQ section discusses this topic.
Unfortunately, in the absence of per-file signatures generated by the
build system, and securely distributed out-of-band, I can't think of any
way to provide client-side verifiability of a debuginfod type service.
That's independent of any particular level of server code robustness.
I think there *are* solutions that reduce the attack surface so that
the public facing server no longer needs to be trusted.
Service 1: indexes signed debuginfo files in Fedora, verifying RPM
signatures, puts the object IDs and hashes into a Merkle tree
[Root node of Merkle tree is signed]
Service 2: serves out those debuginfo files to clients, along with
root signature and the nodes from the root to the file of interest
But I don't want to see this proposal blocked on implementing
something that technically complex - I think it offers big benefits to
Fedora users as is. Certainly there are other programs that are
typically run without sandboxing by developers and connect to network
services - even entirely untrusted network services - and we
typically consider that acceptable.
Owen
3:30 p.m.
* Frank Ch. Eigler:
Unfortunately, in the absence of per-file signatures generated by
the
build system, and securely distributed out-of-band, I can't think of any
way to provide client-side verifiability of a debuginfod type service.
That's independent of any particular level of server code robustness.
I hat to bring them up, but IMA signatures could *almost* be used for
this. They cover individual files.
The problem is that any Fedora developer can get an IMA signature of any
file contents. There is nothing in the signature that says that it's
been produced by debuginfo generation. So I'm not sure if IMA
signatures actually reduce the attack surface in any significant way.
Thanks,
Florian
2:15 p.m.
Björn Persson <Bjorn(a)xn--rombobjrn-67a.se> writes:
> https://sourceware.org/bugzilla/show_bug.cgi?id=27758
The design you propose there won't improve anything for anyone. If the
hash is computed on the debuginfo server, then an attacker who can make
the server serve malicious debuginfo can also make it serve hashes that
match the malicious files.
Yes, this does not provide protection against a penetrated server. It
does not claim to.
And as you noted yourself, an attacker who can manipulate cached
files
client-side has already taken over the user account anyway.
Yes and no, and so I must disagree with your "won't improve ... for
anyone". The proposed client-side verification is roughly analogous to
running "rpm -V" on a machine. Yes, if an attacker has control at that
moment, it's not reliable. Nevertheless, to detect residue of a
-previous attack- or accidental data corruption, it can be worthwhile.
[...] I see that debuginfod.fedoraproject.org is currently another
name for koji.fedoraproject.org.
They are separate VMs, if that's what you mean. (You may be confused by
use of a number of shared HTTP front-end proxies.)
Given that it serves debuginfo only for Fedora packages, and does
not
forward requests to any other debuginfo servers, using this server
seems equivalent security-wise to downloading unsigned packages from
Koji.
Not exactly. All the data is -from- signed packages.
To make the debuginfo protocol as secure as signed debuginfo
packages,
the client should verify the files against a hash computed and signed
on the signing server.
If the threat model includes a -local active attacker-, then this would
not help either. An attacker could interfere with the local keystore
and/or trust chains and/or signature verification software.
For those who are concerned about privacy, the proposal would make
that problem worse as it would cause the "phoning home" to happen
every time they debug something.
That's if they wish to rely on live verification.
Note that the privacy being leaked consists of the hex buildid of the
program being debugged, and an elfutils#/fedora#/arch, and of course IP
address. It's not nothing, but it's nothing more. It's roughly
equivalent to the dnf-automatic.timer call-home in this respect.
By the way, the change page still doesn't say enough about how
network
problems will affect the user experience. [...]
I'm not sure why you say "still" when this question was not posed here
before. I will add some text on this.
- FChE
2:48 p.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On Wed, Apr 21, 2021 at 03:15:23PM -0400, Frank Ch. Eigler wrote:
Björn Persson <Bjorn(a)xn--rombobjrn-67a.se> writes:
>> https://sourceware.org/bugzilla/show_bug.cgi?id=27758
>
> The design you propose there won't improve anything for anyone. If the
> hash is computed on the debuginfo server, then an attacker who can make
> the server serve malicious debuginfo can also make it serve hashes that
> match the malicious files.
Yes, this does not provide protection against a penetrated server. It
does not claim to.
> And as you noted yourself, an attacker who can manipulate cached files
> client-side has already taken over the user account anyway.
Yes and no, and so I must disagree with your "won't improve ... for
anyone". The proposed client-side verification is roughly analogous to
running "rpm -V" on a machine. Yes, if an attacker has control at that
moment, it's not reliable. Nevertheless, to detect residue of a
-previous attack- or accidental data corruption, it can be worthwhile.
We have btrfs now… It's not exactly the same, but it provides protection
against the most likely corruption scenario — disk errors.
> [...] I see that debuginfod.fedoraproject.org is currently
another
> name for koji.fedoraproject.org.
They are separate VMs, if that's what you mean. (You may be confused by
use of a number of shared HTTP front-end proxies.)
> Given that it serves debuginfo only for Fedora packages, and does not
> forward requests to any other debuginfo servers, using this server
> seems equivalent security-wise to downloading unsigned packages from
> Koji.
Not exactly. All the data is -from- signed packages.
It is equivalent in the following sense: if the server is compromised,
it can serve any data it wants, and the client has no chance of
knowing.
Zbyszek
9:27 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
Frank Ch. Eigler wrote:
Björn Persson <Bjorn(a)xn--rombobjrn-67a.se> writes:
> And as you noted yourself, an attacker who can manipulate cached files
> client-side has already taken over the user account anyway.
Yes and no, and so I must disagree with your "won't improve ... for
anyone". The proposed client-side verification is roughly analogous to
running "rpm -V" on a machine. Yes, if an attacker has control at that
moment, it's not reliable. Nevertheless, to detect residue of a
-previous attack- or accidental data corruption, it can be worthwhile.
I fail to imagine how you believe attackers operate to make the
distinction between an attacker who has control and a previous attack
relevant.
It can detect accidental data corruption, yes. If you want to checksum
the cache to detect accidental data corruption, that's fine by me, but
that's better done locally, so that the checksums can be verified
without contacting any server.
> Given that it serves debuginfo only for Fedora packages, and
does not
> forward requests to any other debuginfo servers, using this server
> seems equivalent security-wise to downloading unsigned packages from
> Koji.
Not exactly. All the data is -from- signed packages.
Okay, so it's equivalent to downloading packages that were once signed,
but had the signatures removed before the packages were offered for
downloading – which is in turn equivalent to downloading unsigned
packages.
> To make the debuginfo protocol as secure as signed debuginfo
packages,
> the client should verify the files against a hash computed and signed
> on the signing server.
If the threat model includes a -local active attacker-, then this would
not help either. An attacker could interfere with the local keystore
and/or trust chains and/or signature verification software.
A local active attacker who can already read, write and execute
whatever they want has nothing more to gain from tampering with cached
debuginfo.
> By the way, the change page still doesn't say enough about
how network
> problems will affect the user experience. [...]
I'm not sure why you say "still" when this question was not posed here
before.
Because I posed the question in my first message in this thread, on the
11th.
Björn Persson
9:41 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On 4/7/21 22:32, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
== Summary ==
Fedora users / developers who need to debug/trace distro binaries can
make use of the recently activated elfutils-debuginfod servers to
automatically fetch debugging data and source code, instead of having
to use `# sudo dnf` commands.
Now readelf, annobin and hell knows what else started to talk to
REMOTE SERVERS, deep out of internals of complicated build infrastructure
running on presumably secure build machines of various IT corporations
and whatnot!
This is devastatingly insecure, just ONE remote exploit bug, and many IT
corporations can be exposed.
Do you understand how many fetches of debuginfo will be attempted by e.g.
a kernel build tooling when it runs readelf on 8000 freshly built modules
_for every kernel build_? How slow it is?
Now various tools need to forcibly unset the variable to stop this madness.
commit b927c044b8809c4dd892f75737240a20c32c2b90
Author: Panu Matilainen <pmatilai(a)redhat.com>
Date: Thu Feb 16 12:25:24 2023 +0200
Disable debuginfod server lookups in build and dependency generator scripts
With recent elfutils (0.182 or so) various seemingly innocuous tools
such as `readelf` like to do network lookups for ELF symbol information.
There's no circumstance where we'd want that to happen during rpmbuild,
so disable these lookups during all spec build scriptlets and also
dependency generator children.
+ unsetenv("DEBUGINFOD_URLS");
1:43 p.m.
Hi, Denys -
dvlasenk wrote:
[...]
Now readelf, annobin and hell knows what else started to talk to
REMOTE SERVERS, deep out of internals of complicated build infrastructure
running on presumably secure build machines of various IT corporations
and whatnot!
(It may not be appropriate for secure build machines to have general
internet access, but that's for the operators to decide.)
[...] Do you understand how many fetches of debuginfo will be
attempted by e.g. a kernel build tooling when it runs readelf on 8000
freshly built modules _for every kernel build_? How slow it is?
If remote debuginfo is not needed for these particular readelf
invocations, then the tools should not be making debuginfod calls.
Can you help identify examples?
Now various tools need to forcibly unset the variable to stop this
madness.
The defaults are set with normal developers in mind.
- FChE
1:26 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
On 6/4/24 21:43, Frank Ch. Eigler wrote:
Hi, Denys -
dvlasenk wrote:
> [...]
> Now readelf, annobin and hell knows what else started to talk to
> REMOTE SERVERS, deep out of internals of complicated build infrastructure
> running on presumably secure build machines of various IT corporations
> and whatnot!
(It may not be appropriate for secure build machines to have general
internet access, but that's for the operators to decide.)
> [...] Do you understand how many fetches of debuginfo will be
> attempted by e.g. a kernel build tooling when it runs readelf on 8000
> freshly built modules _for every kernel build_? How slow it is?
If remote debuginfo is not needed for these particular readelf
invocations, then the tools should not be making debuginfod calls.
Can you help identify examples?
> Now various tools need to forcibly unset the variable to stop this madness.
The defaults are set with normal developers in mind.
The problem is that nobody expects a tool this low-level to do internet
lookups, and silently at that. It's like 'ls' suddenly started looking
up stuff from the net *by default*.
And, not just interactive use but scripts too, scripts that existed
years or even decades before this thing and cannot possibly expect such
activity. Like the case of
https://bugzilla.redhat.com/show_bug.cgi?id=2079600
And, it's made much worse by the packaging which means you cannot just
remove the package and be done with it, because so much important
tooling links to it. If the library and the configuration to actually
enable it were split between different packages, it'd be less offensive
at least.
- Panu -
5:25 a.m.
New subject: F35 Change: Debuginfod By Default (Self-Contained Change
proposal)
Hi Panu,
On Wed, Jun 05, 2024 at 09:26:14AM +0300, Panu Matilainen wrote:
On 6/4/24 21:43, Frank Ch. Eigler wrote:
>dvlasenk wrote:
>
>>[...] Do you understand how many fetches of debuginfo will be
>>attempted by e.g. a kernel build tooling when it runs readelf on 8000
>>freshly built modules _for every kernel build_? How slow it is?
>
>If remote debuginfo is not needed for these particular readelf
>invocations, then the tools should not be making debuginfod calls.
>Can you help identify examples?
>
>>Now various tools need to forcibly unset the variable to stop this
>>madness.
>
>The defaults are set with normal developers in mind.
The problem is that nobody expects a tool this low-level to do
internet lookups, and silently at that. It's like 'ls' suddenly
started looking up stuff from the net *by default*.
And, not just interactive use but scripts too, scripts that existed
years or even decades before this thing and cannot possibly expect
such activity. Like the case of
https://bugzilla.redhat.com/show_bug.cgi?id=2079600
This surprises me. You are right that nobody would expect readelf -d
(to get the dynamic table entries) would use debuginfod. And it indeed
doesn't. Might this just have been an old bug in binutils since fixed?
And, it's made much worse by the packaging which means you
cannot
just remove the package and be done with it, because so much
important tooling links to it. If the library and the configuration
to actually enable it were split between different packages, it'd be
less offensive at least.
I am sorry you are so offended by the packaging, but nobody has
suggested this before. Please do file a bug with this suggestion if
you think it makes sense to split the default /etc files and the
actual debuginfod-client library.
Cheers,
Mark
12
days inactive
1167
days old
43 comments
16 participants
participants (16)
-
Ben Cotton -
Björn Persson -
David Malcolm -
Denys Vlasenko -
Dominique Martinet -
Florian Weimer -
Frank Ch. Eigler -
FUNG Chi Chuen Sampson -
Lars Seipel -
Mark Wielaard -
Michael Catanzaro -
Neal Gompa -
Owen Taylor -
Panu Matilainen -
Sampson Fung
-
Zbigniew Jędrzejewski-Szmek