Ok, let me walk through some of the specific errors, and I will also censor out some of the output since this is going to the public mail-list as well.

Starting from the beginning.
- I have set the date to `1 month` before certificate expired with `sudo date`
- I ran `ipactl restart --force` and `pki-tomcatd` failed
- I try to run `ipa-cacert-manage renew` with `--external-ca` and `--external-cert-file`, with the CSR signed with `openssl x509 -req -copy_extensions=copyall`, but this one failed because of `ERROR: CA certificate chain in is incomplete: missing certificate with subject`, even though I passed both the root and new CA

Then I have tried to manually add the new CA, but then, that did not work. So moving on to the next layer, when I run to `sudo getcert list`, the certificates I get:
```
Request ID '20220810070559':
        status: CA_UNREACHABLE
        ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for requested realm.
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=ipa1.my.lab,O=My Lab
        issued: 2023-09-20 14:21:40 UTC
        expires: 2025-09-20 14:21:40 UTC
Request ID '20230920203122':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=CA Audit,O=My Lab
        issued: 2023-09-20 14:21:35 UTC
        expires: 2025-09-09 14:21:35 UTC
Request ID '20230920203123':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=OCSP Subsystem,O=My Lab
        issued: 2023-09-20 14:21:33 UTC
        expires: 2025-09-09 14:21:33 UTC
Request ID '20230920203125':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=CA Subsystem,O=My Lab
        issued: 2023-09-20 14:21:31 UTC
        expires: 2025-09-09 14:21:31 UTC
Request ID '20230920203128':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=ipa1.my.lab,O=My Lab
        issued: 2023-09-20 11:29:21 UTC
        expires: 2023-12-20 11:29:21 UTC
Request ID '20230920203130':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=IPA RA,O=My Lab
        issued: 2023-09-20 14:21:36 UTC
        expires: 2025-09-09 14:21:36 UTC
Request ID '20230920203131':
        status: CA_UNREACHABLE
        ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for requested realm.
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
        subject: CN=ipa1.my.lab,O=My Lab
        issued: 2023-09-20 14:21:39 UTC
        expires: 2025-09-20 14:21:39 UTC
Request ID '20230920203134':
        status: CA_UNREACHABLE
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA'
        certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
        CA: IPA
        issuer: CN=My Lab Root CA,O=My Lab
        subject: CN=ipa1.my.lab,O=My Lab
        issued: 2023-07-10 04:31:16 UTC
        expires: 2023-08-09 04:31:16 UTC
Request ID '20230510042436':
        status: MONITORING
        ca-error: Updated certificate not available
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=My Lab Root CA,O=My Lab
        subject: CN=FreeIPA Root Certificate Authority,O=My Lab
        issued: 2022-08-10 04:29:31 UTC
        expires: 2023-08-10 04:29:31 UTC
```
Note that even the `MONITORING` ones are unusable due to the non-overlapping date. So when I do something like `ipa-certupdate` I get:
`The ipa-certupdate command failed, exception: DatabaseError: Connect error: error:0A000086:SSL routines::certificate verify failed (unable to get issuer certificate)` when trying to use `ipaldap.py`.

It is a mess, that's why I think it is better to have a way to simply re-bootstrap all certificates.

On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
Hi,

On Fri, Sep 22, 2023 at 12:36 PM Cristian Le <fedora@lecris.me> wrote:

Hi Florence,

Thanks for the feedback, let me clarify the situation on the certificates:
- External CA is still valid and it is a self-signed certificate that we use for other services. So we can manually sign any service certificates to get them back up and running
- IPA CA is expired, let's say Aug/10
- I have managed to import a renewed IPA CA and ran `ipa-cert-fix` (and also seemed to have run `ipa-certupdate`) on a current date, let's say Sep/20. But not all services were recovered and now there is no overlap between earliest date in service certificates and the original IPA CA

Which are the not-recovered services? Can you provide the output of "getcert list" at the current date?
flo

- I have run a backup, but also did some system upgrades to get the `ipa-cacert-manage prune` command, but when I've tried to recover it, I've found that the backup was not there.

 > you can still find the original certificates in the LDAP database (below ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of searching. You would need to restore the expired certificates, go back in time and force the renewal.

I suspect we cannot do this all within LDAP right? If we get back the expired certificates, how do we restore them in each service? `httpd` is straightforward, and I guess `nssdb` should be doable, assuming the same key is used, but is there another database type where the certificates are located? Are all the certificates tracked by `getcert list`? Is it safe to assume that after running something like `ipa-cert-fix`, they are using the same private key?

Some symptoms in the current setup:
- When we are forward in time, `pki-tomcatd` is able to run, but then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`. From what I've read, all of these commands (or at least `ipa-cacert-manage renew`) must be done backwards in time.
- When we are backwards in time `pki-tomcatd` is unable to run, failing to access `:8080/ca/admin/ca/getStatus`. This then blocks various other services to be run. But about `ipactl restart`, only `pki-tomcatd` service is actually failing (and ipa service itself of course).

I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and indeed there are still a bunch of certificates there in linear order. What are the services I should look for in there? I am using Apache Directory Studio and I can download the `userCertificate`. Should I just run `certutil -A` with those values with corresponding `subjectName`?

BTW, I want to document this process on the website, should I make a PR on the github repo or is there somewhere else?

Kind regards,
Cristian

On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
Hi,

On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
I have tried my luck around with all the helpers: `pki-server cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but each one is failing on me for multiple reasons.
- `ipa-cacert-manage` Cannot update the CA with `--external-cert-file` because the root ca is not detected to be in the trust list
This command is useful if you need to trust a new external CA or renew IPA CA. Is your IPA CA expired?
- `ipa-cert-fix` Was run without overlapping validity time, and the certificate were re-created, so now it is not recoverable, neither back in time, nor in current time
It is recommended to do a backup before running ipa-cert-fix. If you didn't, and want to try the back-in-time method, you can still find the original certificates in the LDAP database (below ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of searching. You would need to restore the expired certificates, go back in time and force the renewal.

- `pki-tomcat` is failing

What is the current situation? Which certs are expired (getcert list)? If you start the services with "ipactl start --ignore-service-failures", is pki-tomcat the only service failing?
flo

It is quite a mess and I would like to ask for some guidance on how one could recover manually from  such dependency issues:
- Is it possible to do a `ipa-server-install` and keep the user data?
- If I sign all of the service's certificates manually, what are all of the manual steps needed to get the services back up so that the helpers can be run.
  - I've tried to install the CA certificate in the nssdb database, ldap, and /etc/ipa/ca.crt. Are there other locations?
  - I've recreated an httpd certificate signed by the root, but I can't figure how to do the same with the ones located in the nssdb database, i.e. to recreate a csr with the same data as one of the certificates there
- What is the order of services that should be updated. My understanding is CA -> `certutil`'s CA -> httpd + slapd + pki-tomcat (not sure where the last one is or how to edit it) -> `ipa-certupdate`
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue