4:43 a.m.
If tomcat fails to start then you need to stop and figure out why.
Any further poking at certificates won't yield anything useful without a
working CA.
So at all times I should make sure `pki-tomcatd` is working. Well it is
often failing with:
`ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:
for url: http://ipa1.my.lab:8080/ca/admin/ca/getStatus`
Presumably this is meant to be fixed by `ipa-cacert-manage renew`, but
as indicated it fails with `ERROR: CA
certificate chain in is incomplete`, specifically the traceback:
```
ipapython.admintool: DEBUG: File
"/usr/lib/python3.11/site-packages/ipapython/admintool.py", line 180, in
execute
return_value = self.run()
^^^^^^^^^^
File
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py",
line 138, in run
return self.renew()
^^^^^^^^^^^^
File
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py",
line 201, in renew
return self.renew_external_step_2(ca, cert)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py",
line 277, in renew_external_step_2
cert_file, ca_file = installutils.load_external_cert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/lib/python3.11/site-packages/ipaserver/install/installutils.py",
line 1194, in load_external_cert
raise ScriptError(
```
(disregard the slightly offset line numbers, I've added some debugging
to confirm that in the loop at [1] the trust chain has indeed only 1
item and it is not self-signed).
One thing suspicious is that the root CA is not set to `CT,C,C`:
```
ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d',
'sql:/tmp/tmprpz4cur1', '-M', '-n', 'CN=FreeIPA Root
Certificate
Authority,O=My Lab', '-t', 'C,,', '-f',
'/tmp/tmprpz4cur1/pwdfile.txt']
ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d',
'sql:/tmp/tmprpz4cur1', '-M', '-n', 'CN=My Lab Root CA,O=My
Lab', '-t',
'C,,', '-f', '/tmp/tmprpz4cur1/pwdfile.txt']
```
Do you think it is worth delving more into this issue and finding a way
to recover `ipa-cacert-manage renew`?
Manually add how?
Using `certutil -A -d` on the `nssdb` and individual files like in
`/etc/httpd`. I get the locations as reported in `getcert list`.
The KDC isn't running.
The KDC is running, but the `kdc.crt` certificates are busted.
[1]
https://pagure.io/freeipa/blob/master/f/ipaserver/install/installutils.py...
On 2023/09/25 21:06, Rob Crittenden wrote:
> Cristian Le via FreeIPA-users wrote:
>> Ok, let me walk through some of the specific errors, and I will also
>> censor out some of the output since this is going to the public
>> mail-list as well.
>>
>> Starting from the beginning.
>> - I have set the date to `1 month` before certificate expired with `sudo
>> date`
>> - I ran `ipactl restart --force` and `pki-tomcatd` failed
> If tomcat fails to start then you need to stop and figure out why. Any
> further poking at certificates won't yield anything useful without a
> working CA.
>
>> - I try to run `ipa-cacert-manage renew` with `--external-ca` and
>> `--external-cert-file`, with the CSR signed with `openssl x509 -req
>> -copy_extensions=copyall`, but this one failed because of `ERROR: CA
>> certificate chain in is incomplete: missing certificate with subject`,
>> even though I passed both the root and new CA
> What failed?
>
>> Then I have tried to manually add the new CA, but then, that did not
>> work. So moving on to the next layer, when I run to `sudo getcert list`,
>> the certificates I get:
> Manually add how?
>
>> ```
>> Request ID '20220810070559':
>> status: CA_UNREACHABLE
>> ca-error: Error setting up ccache for "host" service on
client
>> using default keytab: Cannot contact any KDC for requested realm.
> The KDC isn't running.
>
>> stuck: no
>> key pair storage:
type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>> certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>> CA: IPA
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=ipa1.my.lab,O=My Lab
>> issued: 2023-09-20 14:21:40 UTC
>> expires: 2025-09-20 14:21:40 UTC
>> Request ID '20230920203122':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>> cert-pki-ca',token='NSS Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=CA Audit,O=My Lab
>> issued: 2023-09-20 14:21:35 UTC
>> expires: 2025-09-09 14:21:35 UTC
>> Request ID '20230920203123':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS
>> Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS
>> Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=OCSP Subsystem,O=My Lab
>> issued: 2023-09-20 14:21:33 UTC
>> expires: 2025-09-09 14:21:33 UTC
>> Request ID '20230920203125':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>> cert-pki-ca',token='NSS Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=CA Subsystem,O=My Lab
>> issued: 2023-09-20 14:21:31 UTC
>> expires: 2025-09-09 14:21:31 UTC
>> Request ID '20230920203128':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>> cert-pki-ca',token='NSS Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=ipa1.my.lab,O=My Lab
>> issued: 2023-09-20 11:29:21 UTC
>> expires: 2023-12-20 11:29:21 UTC
>> Request ID '20230920203130':
>> status: MONITORING
>> stuck: no
>> key pair storage:
type=FILE,location='/var/lib/ipa/ra-agent.key'
>> certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=IPA RA,O=My Lab
>> issued: 2023-09-20 14:21:36 UTC
>> expires: 2025-09-09 14:21:36 UTC
>> Request ID '20230920203131':
>> status: CA_UNREACHABLE
>> ca-error: Error setting up ccache for "host" service on
client
>> using default keytab: Cannot contact any KDC for requested realm.
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
>> Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt'
>> certificate:
>>
type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
>> Certificate DB'
>> CA: IPA
>> issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>> subject: CN=ipa1.my.lab,O=My Lab
>> issued: 2023-09-20 14:21:39 UTC
>> expires: 2025-09-20 14:21:39 UTC
>> Request ID '20230920203134':
>> status: CA_UNREACHABLE
>> stuck: no
>> key pair storage:
>>
type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA'
>> certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
>> CA: IPA
>> issuer: CN=My Lab Root CA,O=My Lab
>> subject: CN=ipa1.my.lab,O=My Lab
>> issued: 2023-07-10 04:31:16 UTC
>> expires: 2023-08-09 04:31:16 UTC
>> Request ID '20230510042436':
>> status: MONITORING
>> ca-error: Updated certificate not available
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>> cert-pki-ca',token='NSS Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=My Lab Root CA,O=My Lab
>> subject: CN=FreeIPA Root Certificate Authority,O=My Lab
>> issued: 2022-08-10 04:29:31 UTC
>> expires: 2023-08-10 04:29:31 UTC
>> ```
>> Note that even the `MONITORING` ones are unusable due to the
>> non-overlapping date. So when I do something like `ipa-certupdate` I get:
>> `The ipa-certupdate command failed, exception: DatabaseError: Connect
>> error: error:0A000086:SSL routines::certificate verify failed (unable to
>> get issuer certificate)` when trying to use `ipaldap.py`.
>>
>> It is a mess, that's why I think it is better to have a way to simply
>> re-bootstrap all certificates.
> There isn't really a good way to start over.
>
> rob
>
>> On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
>>> Hi,
>>>
>>> On Fri, Sep 22, 2023 at 12:36 PM Cristian Le <fedora(a)lecris.me> wrote:
>>>
>>> Hi Florence,
>>>
>>> Thanks for the feedback, let me clarify the situation on the
>>> certificates:
>>> - External CA is still valid and it is a self-signed certificate
>>> that we use for other services. So we can manually sign any
>>> service certificates to get them back up and running
>>> - IPA CA is expired, let's say Aug/10
>>> - I have managed to import a renewed IPA CA and ran `ipa-cert-fix`
>>> (and also seemed to have run `ipa-certupdate`) on a current date,
>>> let's say Sep/20. But not all services were recovered and now
>>> there is no overlap between earliest date in service certificates
>>> and the original IPA CA
>>>
>>> Which are the not-recovered services? Can you provide the output of
>>> "getcert list" at the current date?
>>> flo
>>>
>>> - I have run a backup, but also did some system upgrades to get
>>> the `ipa-cacert-manage prune` command, but when I've tried to
>>> recover it, I've found that the backup was not there.
>>>
>>> > you can still find the original certificates in the LDAP
>>> database (below ou=certificateRepository,ou=ca,o=ipaca) but it
>>> requires a bit of searching. You would need to restore the expired
>>> certificates, go back in time and force the renewal.
>>>
>>> I suspect we cannot do this all within LDAP right? If we get back
>>> the expired certificates, how do we restore them in each service?
>>> `httpd` is straightforward, and I guess `nssdb` should be doable,
>>> assuming the same key is used, but is there another database type
>>> where the certificates are located? Are all the certificates
>>> tracked by `getcert list`? Is it safe to assume that after running
>>> something like `ipa-cert-fix`, they are using the same private key?
>>>
>>> Some symptoms in the current setup:
>>> - When we are forward in time, `pki-tomcatd` is able to run, but
>>> then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`.
>>> From what I've read, all of these commands (or at least
>>> `ipa-cacert-manage renew`) must be done backwards in time.
>>> - When we are backwards in time `pki-tomcatd` is unable to run,
>>> failing to access `:8080/ca/admin/ca/getStatus`. This then blocks
>>> various other services to be run. But about `ipactl restart`, only
>>> `pki-tomcatd` service is actually failing (and ipa service itself
>>> of course).
>>>
>>> I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and
>>> indeed there are still a bunch of certificates there in linear
>>> order. What are the services I should look for in there? I am
>>> using Apache Directory Studio and I can download the
>>> `userCertificate`. Should I just run `certutil -A` with those
>>> values with corresponding `subjectName`?
>>>
>>> BTW, I want to document this process on the website, should I make
>>> a PR on the github repo or is there somewhere else?
>>>
>>> Kind regards,
>>> Cristian
>>>
>>> On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
>>>> Hi,
>>>>
>>>> On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users
>>>> <freeipa-users(a)lists.fedorahosted.org> wrote:
>>>>
>>>> I have tried my luck around with all the helpers: `pki-server
>>>> cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but
>>>> each one is failing on me for multiple reasons.
>>>> - `ipa-cacert-manage` Cannot update the CA with
>>>> `--external-cert-file` because the root ca is not detected to
>>>> be in the trust list
>>>>
>>>> This command is useful if you need to trust a new external CA or
>>>> renew IPA CA. Is your IPA CA expired?
>>>>
>>>> - `ipa-cert-fix` Was run without overlapping validity time,
>>>> and the certificate were re-created, so now it is not
>>>> recoverable, neither back in time, nor in current time
>>>>
>>>> It is recommended to do a backup before running ipa-cert-fix. If
>>>> you didn't, and want to try the back-in-time method, you can
>>>> still find the original certificates in the LDAP database (below
>>>> ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of
>>>> searching. You would need to restore the expired certificates, go
>>>> back in time and force the renewal.
>>>>
>>>> - `pki-tomcat` is failing
>>>>
>>>> What is the current situation? Which certs are expired (getcert
>>>> list)? If you start the services with "ipactl start
>>>> --ignore-service-failures", is pki-tomcat the only service
failing?
>>>> flo
>>>>
>>>> It is quite a mess and I would like to ask for some guidance
>>>> on how one could recover manually from such dependency issues:
>>>> - Is it possible to do a `ipa-server-install` and keep the
>>>> user data?
>>>>
>>>> - If I sign all of the service's certificates manually,
what
>>>> are all of the manual steps needed to get the services back
>>>> up so that the helpers can be run.
>>>> - I've tried to install the CA certificate in the nssdb
>>>> database, ldap, and /etc/ipa/ca.crt. Are there other locations?
>>>> - I've recreated an httpd certificate signed by the root,
>>>> but I can't figure how to do the same with the ones located
>>>> in the nssdb database, i.e. to recreate a csr with the same
>>>> data as one of the certificates there
>>>> - What is the order of services that should be updated. My
>>>> understanding is CA -> `certutil`'s CA -> httpd +
slapd +
>>>> pki-tomcat (not sure where the last one is or how to edit it)
>>>> -> `ipa-certupdate`
>>>> _______________________________________________
>>>> FreeIPA-users mailing list --
>>>> freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to
>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>> Fedora Code of Conduct:
>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines:
>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives:
>>>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>> Do not reply to spam, report it:
>>>> https://pagure.io/fedora-infrastructure/new_issue
>>>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>> Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
>>