Cristian Le via FreeIPA-users wrote:
Ok, let me walk through some of the specific errors, and I will also censor out some of the output since this is going to the public mail-list as well.
Starting from the beginning.
- I have set the date to `1 month` before certificate expired with `sudo
date`
- I ran `ipactl restart --force` and `pki-tomcatd` failed
If tomcat fails to start then you need to stop and figure out why. Any further poking at certificates won't yield anything useful without a working CA.
- I try to run `ipa-cacert-manage renew` with `--external-ca` and
`--external-cert-file`, with the CSR signed with `openssl x509 -req -copy_extensions=copyall`, but this one failed because of `ERROR: CA certificate chain in is incomplete: missing certificate with subject`, even though I passed both the root and new CA
What failed?
Then I have tried to manually add the new CA, but then, that did not work. So moving on to the next layer, when I run to `sudo getcert list`, the certificates I get:
Manually add how?
Request ID '20220810070559': status: CA_UNREACHABLE ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for requested realm.
The KDC isn't running.
stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=ipa1.my.lab,O=My Lab issued: 2023-09-20 14:21:40 UTC expires: 2025-09-20 14:21:40 UTC Request ID '20230920203122': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=CA Audit,O=My Lab issued: 2023-09-20 14:21:35 UTC expires: 2025-09-09 14:21:35 UTC Request ID '20230920203123': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=OCSP Subsystem,O=My Lab issued: 2023-09-20 14:21:33 UTC expires: 2025-09-09 14:21:33 UTC Request ID '20230920203125': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=CA Subsystem,O=My Lab issued: 2023-09-20 14:21:31 UTC expires: 2025-09-09 14:21:31 UTC Request ID '20230920203128': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=ipa1.my.lab,O=My Lab issued: 2023-09-20 11:29:21 UTC expires: 2023-12-20 11:29:21 UTC Request ID '20230920203130': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=IPA RA,O=My Lab issued: 2023-09-20 14:21:36 UTC expires: 2025-09-09 14:21:36 UTC Request ID '20230920203131': status: CA_UNREACHABLE ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for requested realm. stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=FreeIPA Root Certificate Authority,O=My Lab subject: CN=ipa1.my.lab,O=My Lab issued: 2023-09-20 14:21:39 UTC expires: 2025-09-20 14:21:39 UTC Request ID '20230920203134': status: CA_UNREACHABLE stuck: no key pair storage: type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA' certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt' CA: IPA issuer: CN=My Lab Root CA,O=My Lab subject: CN=ipa1.my.lab,O=My Lab issued: 2023-07-10 04:31:16 UTC expires: 2023-08-09 04:31:16 UTC Request ID '20230510042436': status: MONITORING ca-error: Updated certificate not available stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=My Lab Root CA,O=My Lab subject: CN=FreeIPA Root Certificate Authority,O=My Lab issued: 2022-08-10 04:29:31 UTC expires: 2023-08-10 04:29:31 UTC
Note that even the `MONITORING` ones are unusable due to the non-overlapping date. So when I do something like `ipa-certupdate` I get: `The ipa-certupdate command failed, exception: DatabaseError: Connect error: error:0A000086:SSL routines::certificate verify failed (unable to get issuer certificate)` when trying to use `ipaldap.py`. It is a mess, that's why I think it is better to have a way to simply re-bootstrap all certificates.
There isn't really a good way to start over.
rob
On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
Hi,
On Fri, Sep 22, 2023 at 12:36 PM Cristian Le fedora@lecris.me wrote:
Hi Florence, Thanks for the feedback, let me clarify the situation on the certificates: - External CA is still valid and it is a self-signed certificate that we use for other services. So we can manually sign any service certificates to get them back up and running - IPA CA is expired, let's say Aug/10 - I have managed to import a renewed IPA CA and ran `ipa-cert-fix` (and also seemed to have run `ipa-certupdate`) on a current date, let's say Sep/20. But not all services were recovered and now there is no overlap between earliest date in service certificates and the original IPA CA
Which are the not-recovered services? Can you provide the output of "getcert list" at the current date? flo
- I have run a backup, but also did some system upgrades to get the `ipa-cacert-manage prune` command, but when I've tried to recover it, I've found that the backup was not there. > you can still find the original certificates in the LDAP database (below ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of searching. You would need to restore the expired certificates, go back in time and force the renewal. I suspect we cannot do this all within LDAP right? If we get back the expired certificates, how do we restore them in each service? `httpd` is straightforward, and I guess `nssdb` should be doable, assuming the same key is used, but is there another database type where the certificates are located? Are all the certificates tracked by `getcert list`? Is it safe to assume that after running something like `ipa-cert-fix`, they are using the same private key? Some symptoms in the current setup: - When we are forward in time, `pki-tomcatd` is able to run, but then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`. From what I've read, all of these commands (or at least `ipa-cacert-manage renew`) must be done backwards in time. - When we are backwards in time `pki-tomcatd` is unable to run, failing to access `:8080/ca/admin/ca/getStatus`. This then blocks various other services to be run. But about `ipactl restart`, only `pki-tomcatd` service is actually failing (and ipa service itself of course). I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and indeed there are still a bunch of certificates there in linear order. What are the services I should look for in there? I am using Apache Directory Studio and I can download the `userCertificate`. Should I just run `certutil -A` with those values with corresponding `subjectName`? BTW, I want to document this process on the website, should I make a PR on the github repo or is there somewhere else? Kind regards, Cristian On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
Hi, On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote: I have tried my luck around with all the helpers: `pki-server cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but each one is failing on me for multiple reasons. - `ipa-cacert-manage` Cannot update the CA with `--external-cert-file` because the root ca is not detected to be in the trust list This command is useful if you need to trust a new external CA or renew IPA CA. Is your IPA CA expired? - `ipa-cert-fix` Was run without overlapping validity time, and the certificate were re-created, so now it is not recoverable, neither back in time, nor in current time It is recommended to do a backup before running ipa-cert-fix. If you didn't, and want to try the back-in-time method, you can still find the original certificates in the LDAP database (below ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of searching. You would need to restore the expired certificates, go back in time and force the renewal. - `pki-tomcat` is failing What is the current situation? Which certs are expired (getcert list)? If you start the services with "ipactl start --ignore-service-failures", is pki-tomcat the only service failing? flo It is quite a mess and I would like to ask for some guidance on how one could recover manually from such dependency issues: - Is it possible to do a `ipa-server-install` and keep the user data? - If I sign all of the service's certificates manually, what are all of the manual steps needed to get the services back up so that the helpers can be run. - I've tried to install the CA certificate in the nssdb database, ldap, and /etc/ipa/ca.crt. Are there other locations? - I've recreated an httpd certificate signed by the root, but I can't figure how to do the same with the ones located in the nssdb database, i.e. to recreate a csr with the same data as one of the certificates there - What is the order of services that should be updated. My understanding is CA -> `certutil`'s CA -> httpd + slapd + pki-tomcat (not sure where the last one is or how to edit it) -> `ipa-certupdate` _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue