On Tue, Nov 10, 2020 at 7:00 PM Frantisek Zatloukal <fzatlouk(a)redhat.com>
wrote:
On Tue, Nov 10, 2020 at 6:11 PM Brendan Early <mymindstorm1(a)gmail.com>
wrote:
> There seems to be an issue bug caching, it's showing bugs I closed a few
> weeks ago.
>
>
https://packager.fedorainfracloud.org/mymindstorm
>
Mhm, I've fixed that. For some reason, the sync of these two packages
wasn't happening. I'll analyze the problem more deeply tomorrow. Thanks for
the report!
Nevermind, I wouldn't have been able to sleep before diagnosing it :) .
So, without running into our backend details too much: We're using custom
counters in Redis to make sure we never queue sync of something twice (or
more times, if users share their packages or keep refreshing the page
before the data are loaded). Once we queue sync for something, we set a
counter for that item to 1 and it can't be queued again until it's synced
and counter resets to zero.
However, since we're running on a pretty exhausted and overloaded temporary
server, OOM sometimes kills some of our workers or even the redis/database.
In these cases, when we lose our tasks queue, but counters stay
incremented, we would end up not syncing these items ever (with incremented
counters but not in the queue).
To workaround this and further increase reliability, we have implemented a
watchdog. It simply goes throughout the cache for any items which weren't
synced when they should've been and resets the counter for them. It runs
once a day.
However, due to insufficient filtering of non-characters for user inputted
FAS names, we have ended up with an escaped quote in our database. That
wouldn't be a problem since sqlalchemy is doing it's job to fend off
SQLInjections but regex in our watchdog code was crashing on escaped quotes.
Thank you once again for reporting this, we have temporarily hot patched
the code on the server and will come up with a proper solution momentarily.