Request for testing: fail2ban 1.0.1 update

[Richard Shaw]

I have disabled auto-stable based on time for the updates: I have submitted
updates for f36 and up and EPEL 9.

https://bodhi.fedoraproject.org/updates/?search=fail2ban

I would like some sort of confirmation on the update itself or by direct
communication that everything appears to be working properly.

Fail2ban is somewhat fragile by nature and this is a big update.

Updates for EPEL 7 & 8 will take a bit longer.

Thanks,
Richard

[Charlie Dennett]

On Mon, Oct 3, 2022 at 2:57 PM Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:

>  I have disabled auto-stable based on time for the updates: I have
>  submitted updates for f36 and up and EPEL 9.
>
>  https://bodhi.fedoraproject.org/updates/?search=fail2ban
>
>  I would like some sort of confirmation on the update itself or by direct
>  communication that everything appears to be working properly.
>
>  Fail2ban is somewhat fragile by nature and this is a big update.
>
>  
Updated yesterday (10/13/22) to the latest 1.0.1 release on my F36 system.
Upon startup, fail2ban-server was using 100% of a cpu.  Tried removing the
sqlite database and restarting (also restarted iptables after stopping
fail2ban to remove all the blocks)  but that did not help.  Downgraded back
to 0.11.2 and the problem went away.

I will attempt to reinstall 1.0.1 and increase its verbosity to see if I
can discover anything.  It's just my home system.  Use it as my desktop and
I also have an apache web server running on it for my genealogy research.

Charlie

[Charlie Dennett]

On Fri, Oct 14, 2022 at 9:55 AM Charlie Dennett <cdennett(a)gmail.com&gt; wrote:

>  On Mon, Oct 3, 2022 at 2:57 PM Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:
>
> > I have disabled auto-stable based on time for the updates: I have
> > submitted updates for f36 and up and EPEL 9.
> >
> > https://bodhi.fedoraproject.org/updates/?search=fail2ban
> >
> > I would like some sort of confirmation on the update itself or by direct
> > communication that everything appears to be working properly.
> >
> > Fail2ban is somewhat fragile by nature and this is a big update.
> >
> >
>  Updated yesterday (10/13/22) to the latest 1.0.1 release on my F36
>  system.  Upon startup, fail2ban-server was using 100% of a cpu.  Tried
>  removing the sqlite database and restarting (also restarted iptables after
>  stopping fail2ban to remove all the blocks)  but that did not help.
>  Downgraded back to 0.11.2 and the problem went away.
>
>  I will attempt to reinstall 1.0.1 and increase its verbosity to see if I
>  can discover anything.  It's just my home system.  Use it as my desktop and
>  I also have an apache web server running on it for my genealogy research.
>
>  Sigh.  I stopped fail2ban (the 0.11 version), restarted iptables to flush 
it out,
upgraded fail2ban to 1.0.1 and started it.  No issues.  It started
up, rebanned what was in the database and settled right down.  Seems to be
working fine now.
-- 
Charlie

I saw a similar problem in my f36 system.  100% CPU usage for the fail2ban-server process
after upgrading to 1.0.1-1.  I also tried removing the sqlite database with no impact. 
fail2ban-server was so busy it would not receive connections from fail2ban-client.  I
enabled DEBUG logging, but that just showed that the process was so busy that everything
was moving slower (e.g. jails that normally start immediately when the server starts did
not start for minutes or longer).

I downgraded to 0.11.2-11 and all is good again.

Given the success of moving forward again reported here, I'll try the upgrade again
this afternoon.

I'm all ears for advice on how to determine what the server is doing.  I didn't
see any child processes and didn't immediately see any hints from open files that the
process had.

David

[Richard Shaw]

Interestingly, I did not have any issues, but I'm only running a sshd jail,
which is one of the reasons I wanted wider testing.

I verified that fail2ban was restarted (systemctl status fail2ban) and then
checked the status (fail2ban-client status sshd) and everything was fine.

I also did a tail -f on both the ssh log and the fail2ban log for a couple
of days and everything seemed fine.


Thanks,
Richard

[Charles Dennett]

On 10/14/22 13:07, Richard Shaw wrote:
>  Interestingly, I did not have any issues, but I'm only running a
> sshd 
>  jail, which is one of the reasons I wanted wider testing.
>  
>  I verified that fail2ban was restarted (systemctl status fail2ban) and 
>  then checked the status (fail2ban-client status sshd) and everything was 
>  fine.
>  
>  I also did a tail -f on both the ssh log and the fail2ban log for a 
>  couple of days and everything seemed fine. 
I just checked fail2ban and it was once again pegged at 100% cpu and had 
missed several things I look for in the apache logs.  systemctl stop 
does not work and I have to manually kill the process.  FWIW, I ran an 
strace on the fail2ban-server process and it produces the following 
output that goes on until I stop the strace:

futex(0x7f819854cac8, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=105267, 
tv_nsec=807769996}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection 
timed out)

futex(0x7f819854cad0, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f819854cac8, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=105267, 
tv_nsec=812963592}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection 
timed out)

futex(0x7f819854cad0, FUTEX_WAKE_PRIVATE, 1) = 0

After this I disabled all jails and restarted fail2ban.  Then one by one 
I enabled them in jail.local.  After enabling a jail I did a 
fail2ban-client add, then a fail2ban-client start and a fail2ban-cleint 
restart.  The latter is what allowed it reban past events.  Everything 
went fine as I reenabled my sshd and all the various apache jails. 
Finally, I did my dovecot jail.  Bang!  As soon as I did the fail2ban 
restart on it the fail2ban server process went to 100%.  Right now I 
assume something is wrong with the dovecot jail.

I then stopped fail2ban.  I disabled just the dovecot jail and restarted 
fail2ban.  It seems to be working as expected.  I'll let it run 
overnight and check in in the morning.  Then I'll see if I can figure it 
out.  At first glance the latest fail2ban  dovecot.conf has a new macro 
called _auth_worker_info.  There may be more but I'll check that more 
thoroughly tomorrow.  Maybe just restore the older doveconf.conf from a 
backup and see if it works.

Charlie

[Charles Dennett]

Version 1.0.1 now seems to be running.  As I noted before, my problem 
had something to do with the dovecot jail.  I did notice a few config 
files had .rpmnew versions after upgrading.  I went through just to make 
sure there were no local mods. I do have .local files so I should not 
have made any changes.  I put the .rpmnew files in place of their 
corresponding .conf files and that may have been what fixed it.  I was 
able to break it before doing that and it worked fine after that.

I did notice one warning when f2b is started:

fail2ban.configreader   [113329]: WARNING 'allowipv6' not defined in 
'Definition'. Using default one: 'auto'

I'll keep an eye on it and report back if it breaks again.

Charlie

[Charles Dennett]

I've been testing f2b over the past few days and I can now make it fail 
on demand.  By that I mean the fail2ban-server process uses 100% of a 
cpu and becomes unresponsive and will not respond to fail2ban-client or 
process any other log entries (apache, sshd, etc.) whenever it attempts 
to process a dovecot log entry.

This will happen if I use the latest 1.0.1 package.  However, if I use 
that but replace the doveconf.conf filter with the one from an earlier 
version (I used the one from my last fedora 35 backup) everything works 
fine.  So, I would hazard a guess that something in the regex's in the 
newest dovecot.conf file is the issue.  I won't pretend to even begin to 
understand them.  They are quite complex to me.

So for now I will use the older dovecot.conf file.  If someone wants me 
to test an updated version of dovecont.conf, I'd be glad to do that.

Charlie

[Richard Shaw]

On Mon, Oct 17, 2022 at 2:12 PM Charles Dennett <cdennett(a)gmail.com&gt; wrote:

>  I've been testing f2b over the past few days and I can now make it fail
>  on demand.  By that I mean the fail2ban-server process uses 100% of a
>  cpu and becomes unresponsive and will not respond to fail2ban-client or
>  process any other log entries (apache, sshd, etc.) whenever it attempts
>  to process a dovecot log entry.
>
>  This will happen if I use the latest 1.0.1 package.  However, if I use
>  that but replace the doveconf.conf filter with the one from an earlier
>  version (I used the one from my last fedora 35 backup) everything works
>  fine.  So, I would hazard a guess that something in the regex's in the
>  newest dovecot.conf file is the issue.  I won't pretend to even begin to
>  understand them.  They are quite complex to me.
>
>  So for now I will use the older dovecot.conf file.  If someone wants me
>  to test an updated version of dovecont.conf, I'd be glad to do that.
>  
If you can, please file an issue upstream and let me know what the URL is.
Include as much info as you're able, especially any relevant log file
entries.

https://github.com/fail2ban/fail2ban/issues

Thanks,
Richard

[Charles Dennett]

On 10/17/22 15:14, Richard Shaw wrote:

>  
>  If you can, please file an issue upstream and let me know what the URL 
>  is. Include as much info as you're able, especially any relevant log 
>  file entries.
>  
>  https://github.com/fail2ban/fail2ban/issues 
>  <https://github.com/fail2ban/fail2ban/issues>
>  
>  Thanks,
>  Richard 
I think they already know about it.  I was just poking around on the 
github pages for f2b and found this:

https://github.com/fail2ban/fail2ban/issues/3370
dovecot jail causes 100% CPU usage #3370

if I understand, it has something to do with python's regex engine not 
detecting nested unlimited loops.

Update coming if not already released.

Charlie