On Fri, 2018-05-25 at 20:31 +0000, Alexander Bokovoy wrote:
To be totally honest, I did not get this message either, Alex - my understanding was that once we finally got all the intended packages landing and the automated tests worked, you actually thought FreeIPA was in acceptable shape fore real use. If it was known that it was not, we absolutely ought to have communicated this *far* more widely than on a niche mailing list: FreeIPA is supposed to be a key feature of Fedora Server which is itself a key edition of Fedora. This should have been up-front in the release notes and the release announcement, or frankly, should've caused us to rethink the release plans.
I did tell that several times but the only real answer I've got: "these issues are not blocking criteria for Fedora Server". At some point you choose your own fights: fixing software or fixing release criteria. For Fedora 29 I'd like us to extend Fedora Server blocking criteria, now that majority of porting has been completed.
For us a push with Python3 migration (we have to migrate all Python base, not a selected module here or there), NSS to OpenSSL migration, mod_nss to mod_ssl migration, NSS default database format migration, Apache ignorance of its ecosystem (changes in ABI in mod_proxy in minor versions), modularity inconsistence through the course of year 2017, have killed a lot of the productive time.
Obviously there was some sort of significant communication fail if enough people missed the message that this got totally whiffed on, so we should absolutely figure out what we can do better there.
Perhaps this also suggests our existing release criteria and test cases for FreeIPA are insufficient: if it can pass our existing tests and thus appear to meet our existing criteria, yet be in your judgment "not ready for production", that seems fundamentally wrong. How do you we think we could address that? Can you give some kind of summary of the issues here, which we can use to think about how to extend the test cases and criteria?
The issues were listed in the email referenced by Jonathan already.
- Replication failures should have been a blocker alone (they are for FreeIPA team) but Fedora Server criteria does not include them.
- Broken NSS sqldb defaults caused us several months working on fixes. The latest one, https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cf042000b, was only pushed after Fedora 28 release. https://bugzilla.redhat.com/show_bug.cgi?id=1568271 was found in late April, after we did fight all the previous issues. We started with NSS sqldb adaptation in October 2017.
- Only on Thursday this week we've finally tracked down a nasty python-ldap bug that crashed FreeIPA framework on every time --all option was used on a host or service entry with additional access controls defined. This is not part of Fedora Server criteria but kills FreeIPA use with delegated permissions to retrieve Kerberos credentials.
- We had to do a lot of Python 3 porting work for other projects. Time is not unlimited, especially when it comes to releases and blocking criteria.
- Dogtag had to work on Tomcat 8.5 adaptation where existing API it dependent on was removed.
So on May 15th we released https://www.freeipa.org/page/Releases/4.6.90.pre2 which is now in Fedora 28 stable updates. We consider it as one of closer candidates to being stable. Between 4.6.90.pre1 and pre2 are two months of hard work across several sizeable projects (freeipa, sssd, 389-ds, MIT Kerberos, dogtag, nss, authselect, gssproxy, to name a few).
We have a testing setup at FreeIPA upstream that allows us to test complex topologies. Only recently we were able to move to Fedora 28 testing there as we had issues with our components. There we test also what OpenQA is unable to test so far. I think 4.6.90.pre2 is in much better shape than what Fedora 28 had released. However, if we were to get it as a blocking release, Fedora 28 would have been delayed by at least a month.
As I said, we had no choice: a push of NSS sqldb defaults change forced us to work on both nss-related code and openssl migration at the same time. It made impossible to keep FreeIPA from Fedora 27 and do our work in a separate module. This was known since autumn 2017 and was a well voiced situation.