mysterious quick-fedora-mirror errors

[Jing Luo]

Hi all,

I am the admin of repo.jing.rocks. I have encountered a problem and I am looking for your help.

I use the quick-fedora-mirror script to pull from one of the upstream tier 1 mirrors 
maintained by the fedora project. I set up a systemd.timer to run the script every 
10 min and a systemd.unit to email me in case of a failure. Now, every once in a 
while (about every two weeks?), I get a super large email in text/plain (~3MB) with 
the systemd log showing the failed job. It looks like this:

--------------------------------------------
● syncrepo-fedora.service - Sync Fedora repo
     Loaded: loaded (/etc/systemd/system/syncrepo-fedora.service; static)
     Active: activating (start) since Wed 2023-11-01 10:52:50 JST; 8ms ago
TriggeredBy: ● syncrepo-fedora.timer
   Main PID: 236492 (zsh)
      Tasks: 1 (limit: 38448)
     Memory: 328.0K
        CPU: 1ms
     CGroup: /system.slice/syncrepo-fedora.service
             └─236492 /usr/bin/zsh /srv/utils/quick-fedora-mirror/quick-fedora-mirror -c /srv/utils/quick-fedora-mirror/quick-fedora-mirror.conf

Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/38/Everything/x86_64/debug/Packages/l/lighttpd-mod_maxminddb-debuginfo-1.4.73-1.fc38.x86_64.rpm" (in fedora-buffet0) failed: No such file or directory (2)
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/38/Everything/x86_64/debug/Packages/l/lighttpd-mod_mbedtls-debuginfo-1.4.73-1.fc38.x86_64.rpm" (in fedora-buffet0) failed: No such file or directory (2)
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/38/Everything/x86_64/debug/Packages/l/lighttpd-mod_nss-debuginfo-1.4.73-1.fc38.x86_64.rpm" (in fedora-buffet0) failed: No such file or directory (2)
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/38/Everything/x86_64/debug/Packages/l/lighttpd-mod_openssl-debuginfo-1.4.73-1.fc38.x86_64.rpm" (in fedora-buffet0) failed: No such file or directory (2)
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/38/Everything/x86_64/debug/Packages/l/lighttpd-mod_vhostdb_dbi-debuginfo-1.4.73-1.fc38.x86_64.rpm" (in fedora-buffet0) failed: No such file or directory (2)
[...]
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/39/Everything/x86_64/repodata/d2573a5621171e75997dba400c260c4afcbd4a684f058f213aef490f52d95ace-updateinfo.xml.xz" (in fedora-buffet0) failed: No such file or directory (2)
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender] link_stat "/fedora/linux/updates/testing/39/Everything/x86_64/repodata/f60ba5e553236f8edc71b3171d29dc6c6a63ba039060f60f9ba854270c0c3a9c-updateinfo.xml.zck" (in fedora-buffet0) failed: No such file or directory (2)
Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1861) [generator=3.2.7]
Nov 01 10:52:51 repo1.jing.rocks zsh[236485]: rsync failed; aborting run.
Nov 01 10:52:51 repo1.jing.rocks zsh[236485]: Will not check in or delete anything.
Nov 01 10:52:51 repo1.jing.rocks zsh[236487]: Mirror starting: Wed Nov  1 10:32:16 JST 2023
Nov 01 10:52:51 repo1.jing.rocks zsh[236487]: Changes in fedora-enchilada: 2925 files/dirs
Nov 01 10:52:51 repo1.jing.rocks zsh[236487]: =========================
Nov 01 10:52:51 repo1.jing.rocks zsh[236487]: Mirror finished: Wed Nov  1 10:52:50 JST 2023 (1)
Nov 01 10:52:51 repo1.jing.rocks zsh[236488]: rm: cannot remove '/srv/tmp/quick-mirror.BmADlXw1xA': Directory not empty
--------------------------------------------------

Should I be concerned about this? Sometimes I manually run quick-fedora-mirror 
with the option to always check all files, and it looks OK.
It looks like many files are deleted but the file list is not yet updated. 
Has anyone else seen this kind of errors?

[Jason Tibbitts]

> > > > > Jing Luo  writes:
> Nov 01 10:52:51 repo1.jing.rocks zsh[236484]: rsync: [sender]
> link_stat
> "/fedora/linux/updates/testing/38/Everything/x86_64/debug/Packages/l/lighttpd-mod_maxminddb-debuginfo-1.4.73-1.fc38.x86_64.rpm"
> (in fedora-buffet0) failed: No such file or directory

That just means that the file wasn't on the server when rsync went to
fetch it.  There is always a delay between when the file list is
generated and when your local rsync attempts to transfer the file.  If
the mirror contents changes in this time, then you will see errors like
this.  There is essentially no way around it; even if the file list was
instantly generated on the server with every change, the contents of the
repo might change between the start of your transfer and when rsync
actually gets through the list to transfer any specific file.

I would generally recommend against just emailing the entire systemd log
with every transfer unless you really care about seeing that much
detail.  The script can generate email itself if you want transfer
status.

- J<

[Jing Luo]

On 2023-11-01 22:47, Jason Tibbitts wrote:
> I would generally recommend against just emailing the entire systemd 
> log
> with every transfer unless you really care about seeing that much
> detail.  The script can generate email itself if you want transfer
> status.

Thanks for the explanation. Since the sync frequency is every 10 min, 
unless the internet gateway is down (I have other ways to monitor the 
internet gateway tho), I agree that the systemd email "onFailure" is not 
necessary.