6:38 a.m.
Hi
Our freeipa certificates need to be renewed due to passing their expiry
dates.
While some certificates have renewed ok, the ipaCert and
auditSigningCert are renewing but the new certificates have the wrong
Subject.
Environment is:
serverA (CRL, first, master) RHEL 7.3, ipa 4.4
serverB (replica) RHEL 7.3, ipa 4.4
serverC (replica) RHEL 7.4, ipa 4.5
Once there are renewed certificates with the wrong Subject present,
there are various problems with renewing the remaining certificates,
which I think might be related to the bad Subject:
1) When just ipaCert has the wrong subject no further renewals happen
2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
service will not start and no further renewals happen.
I've been round the following loop many times on ServerA, our first master:
1) Restore good certificates from backup
2) Put the clock back to a time when certificates are all valid
3) Resubmit certificates for renewal
Each time the ipaCert renews it has the same wrong Subject. The wrong
Subject includes the host name of one of our ipa client systems.
Each time the auditSigningCert renews it has the same wrong Subject but
a different subject to the ipaCert. The wrong Subject in this case
includes the host name of a system which has never been an ipa client,
but might have been added and removed with ipa host-add and ipa host-del
for testing something, a while ago.
As far as I can see, the "cert_subject" is set correctly in the file
/var/lib/certmonger/<request id> until the point at which the
certificate is actually renewed.
I'd be very grateful for some pointers as to which configuration options
and logs to check through to resolve this problem on our production system.
If its of any relevance we did change which server is the first master
some time ago.
Thanks
Roderick Johnstone
10:06 a.m.
Roderick Johnstone via FreeIPA-users wrote:
Hi
Our freeipa certificates need to be renewed due to passing their expiry
dates.
While some certificates have renewed ok, the ipaCert and
auditSigningCert are renewing but the new certificates have the wrong
Subject.
Environment is:
serverA (CRL, first, master) RHEL 7.3, ipa 4.4
serverB (replica) RHEL 7.3, ipa 4.4
serverC (replica) RHEL 7.4, ipa 4.5
Once there are renewed certificates with the wrong Subject present,
there are various problems with renewing the remaining certificates,
which I think might be related to the bad Subject:
1) When just ipaCert has the wrong subject no further renewals happen
2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
service will not start and no further renewals happen.
I've been round the following loop many times on ServerA, our first master:
1) Restore good certificates from backup
2) Put the clock back to a time when certificates are all valid
3) Resubmit certificates for renewal
Each time the ipaCert renews it has the same wrong Subject. The wrong
Subject includes the host name of one of our ipa client systems.
Each time the auditSigningCert renews it has the same wrong Subject but
a different subject to the ipaCert. The wrong Subject in this case
includes the host name of a system which has never been an ipa client,
but might have been added and removed with ipa host-add and ipa host-del
for testing something, a while ago.
As far as I can see, the "cert_subject" is set correctly in the file
/var/lib/certmonger/<request id> until the point at which the
certificate is actually renewed.
I'd be very grateful for some pointers as to which configuration options
and logs to check through to resolve this problem on our production system.
If its of any relevance we did change which server is the first master
some time ago.
I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
the subject is.
rob
12:14 p.m.
On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> Hi
>
> Our freeipa certificates need to be renewed due to passing their expiry
> dates.
>
> While some certificates have renewed ok, the ipaCert and
> auditSigningCert are renewing but the new certificates have the wrong
> Subject.
>
> Environment is:
> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
> serverB (replica) RHEL 7.3, ipa 4.4
> serverC (replica) RHEL 7.4, ipa 4.5
>
> Once there are renewed certificates with the wrong Subject present,
> there are various problems with renewing the remaining certificates,
> which I think might be related to the bad Subject:
>
> 1) When just ipaCert has the wrong subject no further renewals happen
>
> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
> service will not start and no further renewals happen.
>
> I've been round the following loop many times on ServerA, our first master:
>
> 1) Restore good certificates from backup
> 2) Put the clock back to a time when certificates are all valid
> 3) Resubmit certificates for renewal
>
> Each time the ipaCert renews it has the same wrong Subject. The wrong
> Subject includes the host name of one of our ipa client systems.
>
> Each time the auditSigningCert renews it has the same wrong Subject but
> a different subject to the ipaCert. The wrong Subject in this case
> includes the host name of a system which has never been an ipa client,
> but might have been added and removed with ipa host-add and ipa host-del
> for testing something, a while ago.
>
> As far as I can see, the "cert_subject" is set correctly in the file
> /var/lib/certmonger/<request id> until the point at which the
> certificate is actually renewed.
>
> I'd be very grateful for some pointers as to which configuration options
> and logs to check through to resolve this problem on our production system.
>
> If its of any relevance we did change which server is the first master
> some time ago.
I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
the subject is.
I'm not seeing any obvious CSR fields in the
/etc/pki/pki-tomcat/ca/CS.cfg file.
The CSR in the certmonger requests file for the auditSigningCert seems
to be showing with the correct Subject. This is different from the bad
subject showing in the requests file field:
cert_subject=
and the Subject which is showing in the 'getcert list' output (which is
the same as that in the cert_subject= field.
I'm not quite sure what this all means.
Roderick
>
> rob
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>
2:07 p.m.
Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> Hi
>>
>> Our freeipa certificates need to be renewed due to passing their expiry
>> dates.
>>
>> While some certificates have renewed ok, the ipaCert and
>> auditSigningCert are renewing but the new certificates have the wrong
>> Subject.
>>
>> Environment is:
>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>> serverB (replica) RHEL 7.3, ipa 4.4
>> serverC (replica) RHEL 7.4, ipa 4.5
>>
>> Once there are renewed certificates with the wrong Subject present,
>> there are various problems with renewing the remaining certificates,
>> which I think might be related to the bad Subject:
>>
>> 1) When just ipaCert has the wrong subject no further renewals happen
>>
>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>> service will not start and no further renewals happen.
>>
>> I've been round the following loop many times on ServerA, our first
>> master:
>>
>> 1) Restore good certificates from backup
>> 2) Put the clock back to a time when certificates are all valid
>> 3) Resubmit certificates for renewal
>>
>> Each time the ipaCert renews it has the same wrong Subject. The wrong
>> Subject includes the host name of one of our ipa client systems.
>>
>> Each time the auditSigningCert renews it has the same wrong Subject but
>> a different subject to the ipaCert. The wrong Subject in this case
>> includes the host name of a system which has never been an ipa client,
>> but might have been added and removed with ipa host-add and ipa host-del
>> for testing something, a while ago.
>>
>> As far as I can see, the "cert_subject" is set correctly in the file
>> /var/lib/certmonger/<request id> until the point at which the
>> certificate is actually renewed.
>>
>> I'd be very grateful for some pointers as to which configuration options
>> and logs to check through to resolve this problem on our production
>> system.
>>
>> If its of any relevance we did change which server is the first master
>> some time ago.
>
> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
> the subject is.
I'm not seeing any obvious CSR fields in the
/etc/pki/pki-tomcat/ca/CS.cfg file.
foo.bar.certreq=
The CSR in the certmonger requests file for the auditSigningCert
seems
to be showing with the correct Subject. This is different from the bad
subject showing in the requests file field:
cert_subject=
The value of cert_subject comes from the issued certificate.
and the Subject which is showing in the 'getcert list' output
(which is
the same as that in the cert_subject= field.>
I'm not quite sure what this all means.
It is displayed from the data within the tracked certmonger request.
certmonger logs to syslog so you can check there or you can stop the
process and run it manually with: certmonger -n -d 9 2>&1 | tee
certmonger.log
That will provide a lot of debugging output that may show what is going on.
rob
6:14 a.m.
On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> Hi
>>>
>>> Our freeipa certificates need to be renewed due to passing their expiry
>>> dates.
>>>
>>> While some certificates have renewed ok, the ipaCert and
>>> auditSigningCert are renewing but the new certificates have the wrong
>>> Subject.
>>>
>>> Environment is:
>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>> serverB (replica) RHEL 7.3, ipa 4.4
>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>
>>> Once there are renewed certificates with the wrong Subject present,
>>> there are various problems with renewing the remaining certificates,
>>> which I think might be related to the bad Subject:
>>>
>>> 1) When just ipaCert has the wrong subject no further renewals happen
>>>
>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>> service will not start and no further renewals happen.
>>>
>>> I've been round the following loop many times on ServerA, our first
>>> master:
>>>
>>> 1) Restore good certificates from backup
>>> 2) Put the clock back to a time when certificates are all valid
>>> 3) Resubmit certificates for renewal
>>>
>>> Each time the ipaCert renews it has the same wrong Subject. The wrong
>>> Subject includes the host name of one of our ipa client systems.
>>>
>>> Each time the auditSigningCert renews it has the same wrong Subject but
>>> a different subject to the ipaCert. The wrong Subject in this case
>>> includes the host name of a system which has never been an ipa client,
>>> but might have been added and removed with ipa host-add and ipa host-del
>>> for testing something, a while ago.
>>>
>>> As far as I can see, the "cert_subject" is set correctly in the
file
>>> /var/lib/certmonger/<request id> until the point at which the
>>> certificate is actually renewed.
>>>
>>> I'd be very grateful for some pointers as to which configuration options
>>> and logs to check through to resolve this problem on our production
>>> system.
>>>
>>> If its of any relevance we did change which server is the first master
>>> some time ago.
>>
>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
>> the subject is.
>
> I'm not seeing any obvious CSR fields in the
> /etc/pki/pki-tomcat/ca/CS.cfg file.
foo.bar.certreq=
Thanks for the hint (and for your input in general).
I'm seeing only the following four certreq entries in CS.cfg, nothing
obvious for ca.audit_signing:
$ pwd
/etc/pki/pki-tomcat/ca
$ ls -l CS.cfg
-rw-rw---- 1 pkiuser pkiuser 82417 Oct 24 12:00 CS.cfg
$ grep certreq CS.cfg | awk -F= '{print $1}'
ca.ocsp_signing.certreq
ca.signing.certreq
ca.sslserver.certreq
ca.subsystem.certreq
Interestingly the CS.cfg file was written at just the time I have been
putting the clock back to for certificate renewal purposes.
If I look at a backup of the CS.cfg file that I have I see all certreq
as expected:
$ pwd
/var/tmp/rmj/etc/pki/pki-tomcat/ca
$ ls -l CS.cfg
-rw-rw---- 1 pkiuser pkiuser 83015 Aug 18 22:43 CS.cfg
$ grep certreq CS.cfg | awk -F= '{print $1}'
ca.audit_signing.certreq
ca.ocsp_signing.certreq
ca.signing.certreq
ca.sslserver.certreq
ca.subsystem.certreq
Here are the results of checking the CSRs in both the cetmonger requests
and CS.cfg locations in the live system:
The CSRs in the CS.cfg file show:
ca.ocsp_signing.certreq
Subject: O=<my domain in caps>, CN=OCSP Subsystem
ca.signing.certreq
Subject: O=<my domain in caps>, CN=Certificate Authority
ca.sslserver.certreq
Subject: O=<my domain in caps>, CN=<hostname of first master>
ca.subsystem.certreq
Subject: O=<my domain in caps>, CN=CA Subsystem
The CSRs in certmonger requests show:
auditSigningCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=CA Audit
ipaCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=IPA RA
ocspSigningCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=OCSP Subsystem
Server-Cert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=<hostname of first master>
subsystemCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=CA Subsystem
So those look ok, except that there is no entry for
ca.audit_signing.certreq in the live CS.cfg file.
The auditSigningCert in the backup of the CS.cfg looks to be correct:
Subject: O=<my domain in caps>, CN=CA Audit
The getcert list output that shows the bad subject for current
auditSigningCert:
$ getcert list | egrep "certificate:|subject|expires"
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=<hostname of system never a freeipa client>,O=<my domain in
caps>
expires: 2019-10-25 11:19:50 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=OCSP Subsystem,O=<my domain in caps>
expires: 2017-10-25 12:24:01 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=CA Subsystem,O=<my domain in caps>
expires: 2017-10-25 12:24:02 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=Certificate Authority,O=<my domain in caps>
expires: 2035-11-05 12:24:00 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2017-10-26 12:22:11 UTC
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-AST-CAM-AC-UK',nickname='Server-Cert',token='NSS
Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2019-10-23 23:04:22 UTC
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2019-10-25 08:42:33 UTC
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
subject: CN=IPA RA,O=<my domain in caps>
expires: 2017-10-25 12:24:23 UTC
Originally I was working on the theory that the bad Subject in the
auditSigningCert was what was blocking the server from issuing any more
certificates, but I have now tried a few things and done quite a bit of
digging and found some other issues, so I'd like to recap what I have
done and what the situation is.
Everything I'm looking at is on the first master server. I also have
certificates which have expired on the replicas but I understand that I
need to fix the first master first.
I restored certificate databases in /etc/httpd/aliases and
/etc/pki/pki-tomcat/alias from a good backup, showing correct subjects
for all certificates, but with various certificates expiring between 23
Oct 2017 and 5 Nov 2017.
I managed to renew the /etc/httpd/alias Server-Cert ok.
The /etc/httpd/alias ipaCert renewed with the bad subject:
CN=<hostname of a freeipa client>,O=<my domain in caps>
I replaced the ipaCert with the good backup, expiring 25 Oct 2017.
I put the clock back to 24 Oct 2017 12:00:00 when all the certificates
were valid.
I restarted certmonger and the auditSigningCert was renewed but with
subject: CN=<hostname of system that was never a freeipa client>,O=<my
domain in caps>
At this point the freeipa pki-tomcat service will not start.
In fact it seems that the pki-tomcat will not start at all or produce
any logs until I do:
pki-server subsystem-enable ca
(it had become disabled).
Now when I check the /var/log/pki/pki-tomcatd/ca/selftest.log file I see:
0.localhost-startStop-1 - [24/Oct/2017:12:00:37 BST] [20] [1]
SystemCertsVerification: system certs verification failure: Certificate
auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101)
Certificate type not approved for application.
0.localhost-startStop-1 - [24/Oct/2017:12:00:37 BST] [20] [1]
SelfTestSubsystem: The CRITICAL self test plugin called
selftests.container.instance.SystemCertsVerification running at startup
FAILED!
Can you elaborate a bit on what "Certificate type not approved for
application" might mean in this context please?
When I check the certificate keys in the database
/etc/pki/pki-tomcat/alias I see:
$ certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa hex-numbers (orphan)
< 1> rsa hex-numbers caSigningCert cert-pki-ca
< 2> rsa hex-numbers ocspSigningCert cert-pki-ca
< 3> rsa hex-numbers subsystemCert cert-pki-ca
< 4> rsa hex-numbers Server-Cert cert-pki-ca
So something has gone wrong here too as there is no entry for
auditSigningCert and there is an orphan key.
I've checked the backup I have of the /etc/pki/pki-tomcat/alias database
files and it shows more correctly that there is a key for auditSigningCert:
$ pwd
/var/tmp/rmj/etc/pki/pki-tomcat/alias
$ certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa hex-numbers (orphan)
< 1> rsa hex-numbers caSigningCert cert-pki-ca
< 2> rsa hex-numbers auditSigningCert cert-pki-ca
< 3> rsa hex-numbers ocspSigningCert cert-pki-ca
< 4> rsa hex-numbers subsystemCert cert-pki-ca
< 5> rsa hex-numbers Server-Cert cert-pki-ca
So it looks like something in the renewal process has removed the key
for auditSigningCert.
So, I'm not sure whether the bad subject in the renewed auditSigningCert
certificate is causing all the breakage or whether thats a symptom of
something else thats wrong.
Any idea which of these is more likely?
> The CSR in the certmonger requests file for the auditSigningCert seems
> to be showing with the correct Subject. This is different from the bad
> subject showing in the requests file field:
> cert_subject=
The value of cert_subject comes from the issued certificate.
> and the Subject which is showing in the 'getcert list' output (which is
> the same as that in the cert_subject= field.>
> I'm not quite sure what this all means.
It is displayed from the data within the tracked certmonger request.
Thanks for the info.
certmonger logs to syslog so you can check there or you can stop the
process and run it manually with: certmonger -n -d 9 2>&1 | tee
certmonger.log
That will provide a lot of debugging output that may show what is going on.
So,
what I'm planning now is:
1) restore the /etc/pki/pki-tomcatd/alias database files from backup
2) restore the CS.cfg file from backup
3) make sure the clock is at a time when all certificates are valid
4) Stop certmonger systemd service
5) Start cermonger with the command you gave above
6) run ipactl start and check pki-tomcatd starts ok
7) resubmit the auditSigningCert with:
getcert resubmit id <request>
8) Check the certmonger.log
Does that sound like a reasonable way forward at this point?
Anything else worth checking at this time?
Roderick
>
> rob
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>
7:04 a.m.
On 16/01/2018 12:14, Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> Hi
>>>>
>>>> Our freeipa certificates need to be renewed due to passing their
>>>> expiry
>>>> dates.
>>>>
>>>> While some certificates have renewed ok, the ipaCert and
>>>> auditSigningCert are renewing but the new certificates have the wrong
>>>> Subject.
>>>>
>>>> Environment is:
>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>
>>>> Once there are renewed certificates with the wrong Subject present,
>>>> there are various problems with renewing the remaining certificates,
>>>> which I think might be related to the bad Subject:
>>>>
>>>> 1) When just ipaCert has the wrong subject no further renewals happen
>>>>
>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>>> service will not start and no further renewals happen.
>>>>
>>>> I've been round the following loop many times on ServerA, our first
>>>> master:
>>>>
>>>> 1) Restore good certificates from backup
>>>> 2) Put the clock back to a time when certificates are all valid
>>>> 3) Resubmit certificates for renewal
>>>>
>>>> Each time the ipaCert renews it has the same wrong Subject. The wrong
>>>> Subject includes the host name of one of our ipa client systems.
>>>>
>>>> Each time the auditSigningCert renews it has the same wrong Subject
>>>> but
>>>> a different subject to the ipaCert. The wrong Subject in this case
>>>> includes the host name of a system which has never been an ipa client,
>>>> but might have been added and removed with ipa host-add and ipa
>>>> host-del
>>>> for testing something, a while ago.
>>>>
>>>> As far as I can see, the "cert_subject" is set correctly in the
file
>>>> /var/lib/certmonger/<request id> until the point at which the
>>>> certificate is actually renewed.
>>>>
>>>> I'd be very grateful for some pointers as to which configuration
>>>> options
>>>> and logs to check through to resolve this problem on our production
>>>> system.
>>>>
>>>> If its of any relevance we did change which server is the first master
>>>> some time ago.
>>>
>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
>>> the subject is.
>>
>> I'm not seeing any obvious CSR fields in the
>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>
> foo.bar.certreq=
Thanks for the hint (and for your input in general).
I'm seeing only the following four certreq entries in CS.cfg, nothing
obvious for ca.audit_signing:
$ pwd
/etc/pki/pki-tomcat/ca
$ ls -l CS.cfg
-rw-rw---- 1 pkiuser pkiuser 82417 Oct 24 12:00 CS.cfg
$ grep certreq CS.cfg | awk -F= '{print $1}'
ca.ocsp_signing.certreq
ca.signing.certreq
ca.sslserver.certreq
ca.subsystem.certreq
Interestingly the CS.cfg file was written at just the time I have been
putting the clock back to for certificate renewal purposes.
If I look at a backup of the CS.cfg file that I have I see all certreq
as expected:
$ pwd
/var/tmp/rmj/etc/pki/pki-tomcat/ca
$ ls -l CS.cfg
-rw-rw---- 1 pkiuser pkiuser 83015 Aug 18 22:43 CS.cfg
$ grep certreq CS.cfg | awk -F= '{print $1}'
ca.audit_signing.certreq
ca.ocsp_signing.certreq
ca.signing.certreq
ca.sslserver.certreq
ca.subsystem.certreq
Here are the results of checking the CSRs in both the cetmonger requests
and CS.cfg locations in the live system:
The CSRs in the CS.cfg file show:
ca.ocsp_signing.certreq
Subject: O=<my domain in caps>, CN=OCSP Subsystem
ca.signing.certreq
Subject: O=<my domain in caps>, CN=Certificate Authority
ca.sslserver.certreq
Subject: O=<my domain in caps>, CN=<hostname of first master>
ca.subsystem.certreq
Subject: O=<my domain in caps>, CN=CA Subsystem
The CSRs in certmonger requests show:
auditSigningCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=CA Audit
ipaCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=IPA RA
ocspSigningCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=OCSP Subsystem
Server-Cert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=<hostname of first master>
subsystemCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=CA Subsystem
So those look ok, except that there is no entry for
ca.audit_signing.certreq in the live CS.cfg file.
The auditSigningCert in the backup of the CS.cfg looks to be correct:
Subject: O=<my domain in caps>, CN=CA Audit
The getcert list output that shows the bad subject for current
auditSigningCert:
$ getcert list | egrep "certificate:|subject|expires"
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=<hostname of system never a freeipa client>,O=<my
domain in caps>
expires: 2019-10-25 11:19:50 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS
Certificate DB'
subject: CN=OCSP Subsystem,O=<my domain in caps>
expires: 2017-10-25 12:24:01 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=CA Subsystem,O=<my domain in caps>
expires: 2017-10-25 12:24:02 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=Certificate Authority,O=<my domain in caps>
expires: 2035-11-05 12:24:00 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2017-10-26 12:22:11 UTC
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-AST-CAM-AC-UK',nickname='Server-Cert',token='NSS
Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2019-10-23 23:04:22 UTC
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2019-10-25 08:42:33 UTC
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
subject: CN=IPA RA,O=<my domain in caps>
expires: 2017-10-25 12:24:23 UTC
Originally I was working on the theory that the bad Subject in the
auditSigningCert was what was blocking the server from issuing any more
certificates, but I have now tried a few things and done quite a bit of
digging and found some other issues, so I'd like to recap what I have
done and what the situation is.
Everything I'm looking at is on the first master server. I also have
certificates which have expired on the replicas but I understand that I
need to fix the first master first.
I restored certificate databases in /etc/httpd/aliases and
/etc/pki/pki-tomcat/alias from a good backup, showing correct subjects
for all certificates, but with various certificates expiring between 23
Oct 2017 and 5 Nov 2017.
I managed to renew the /etc/httpd/alias Server-Cert ok.
The /etc/httpd/alias ipaCert renewed with the bad subject:
CN=<hostname of a freeipa client>,O=<my domain in caps>
I replaced the ipaCert with the good backup, expiring 25 Oct 2017.
I put the clock back to 24 Oct 2017 12:00:00 when all the certificates
were valid.
I restarted certmonger and the auditSigningCert was renewed but with
subject: CN=<hostname of system that was never a freeipa client>,O=<my
domain in caps>
At this point the freeipa pki-tomcat service will not start.
In fact it seems that the pki-tomcat will not start at all or produce
any logs until I do:
pki-server subsystem-enable ca
(it had become disabled).
Now when I check the /var/log/pki/pki-tomcatd/ca/selftest.log file I see:
0.localhost-startStop-1 - [24/Oct/2017:12:00:37 BST] [20] [1]
SystemCertsVerification: system certs verification failure: Certificate
auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101)
Certificate type not approved for application.
0.localhost-startStop-1 - [24/Oct/2017:12:00:37 BST] [20] [1]
SelfTestSubsystem: The CRITICAL self test plugin called
selftests.container.instance.SystemCertsVerification running at startup
FAILED!
Can you elaborate a bit on what "Certificate type not approved for
application" might mean in this context please?
When I check the certificate keys in the database
/etc/pki/pki-tomcat/alias I see:
$ certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa hex-numbers (orphan)
< 1> rsa hex-numbers caSigningCert cert-pki-ca
< 2> rsa hex-numbers ocspSigningCert cert-pki-ca
< 3> rsa hex-numbers subsystemCert cert-pki-ca
< 4> rsa hex-numbers Server-Cert cert-pki-ca
So something has gone wrong here too as there is no entry for
auditSigningCert and there is an orphan key.
I've checked the backup I have of the /etc/pki/pki-tomcat/alias database
files and it shows more correctly that there is a key for auditSigningCert:
$ pwd
/var/tmp/rmj/etc/pki/pki-tomcat/alias
$ certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa hex-numbers (orphan)
< 1> rsa hex-numbers caSigningCert cert-pki-ca
< 2> rsa hex-numbers auditSigningCert cert-pki-ca
< 3> rsa hex-numbers ocspSigningCert cert-pki-ca
< 4> rsa hex-numbers subsystemCert cert-pki-ca
< 5> rsa hex-numbers Server-Cert cert-pki-ca
So it looks like something in the renewal process has removed the key
for auditSigningCert.
So, I'm not sure whether the bad subject in the renewed auditSigningCert
certificate is causing all the breakage or whether thats a symptom of
something else thats wrong.
Any idea which of these is more likely?
>
>> The CSR in the certmonger requests file for the auditSigningCert seems
>> to be showing with the correct Subject. This is different from the bad
>> subject showing in the requests file field:
>> cert_subject=
>
> The value of cert_subject comes from the issued certificate.
>
>> and the Subject which is showing in the 'getcert list' output (which is
>> the same as that in the cert_subject= field.>
>> I'm not quite sure what this all means.
>
> It is displayed from the data within the tracked certmonger request.
Thanks for the info.
>
> certmonger logs to syslog so you can check there or you can stop the
> process and run it manually with: certmonger -n -d 9 2>&1 | tee
> certmonger.log
>
> That will provide a lot of debugging output that may show what is
> going on.
So, what I'm planning now is:
1) restore the /etc/pki/pki-tomcatd/alias database files from backup
2) restore the CS.cfg file from backup
3) make sure the clock is at a time when all certificates are valid
4) Stop certmonger systemd service
5) Start cermonger with the command you gave above
6) run ipactl start and check pki-tomcatd starts ok
7) resubmit the auditSigningCert with:
getcert resubmit id <request>
8) Check the certmonger.log
Does that sound like a reasonable way forward at this point?
Hi Rob
This is all on my first master server.
I put the clock back to when the certificates that O restore form backup
are all valid.
I restored the databases in /etc/httpd/alias and
/etc/pki/pki-tomcat/alias from the last good backup I had.
I also restored the CS.cfg file from backup.
I updated the trusts in /etc/pki/pki-tomcat/alias for
caSigningCert cert-pki-ca
to match what is in section 5 of:
https://access.redhat.com/solutions/643753 This was previously:
caSigningCert cert-pki-ca CTu,u,u
for some reason.
I stopped the certmonger service and run the certmonger command you gave
to start verbose logging.
I was able to start all the ipa services after running:
pki-server subsystem-enable ca
(this seems to become disabled when the tomcatd service cannot start.
I ran getcert resubmit -i <requestid> for the expiring certificates.
The first one I tried (ocspSigningCert) renewed but gets an odd Subject.
It includes the hostname of one of my replica servers.
The other certificates have not renewed.
As you said, there is a large amount of info in the verbose certmonger
debug logs, but it is not immediately obvious to me what has gone wrong,
except that there are some instances of:
Internal error
Would you be prepared to have a look at the log file off-list (3.3MB
file, uncompressed) to see if it means more to you.
Thanks.
Roderick
>
> Anything else worth checking at this time?
>
> Roderick
>>
>> rob
10:07 a.m.
Roderick Johnstone via FreeIPA-users wrote:
On 16/01/2018 12:14, Roderick Johnstone via FreeIPA-users wrote:
Hi Rob
This is all on my first master server.
I put the clock back to when the certificates that O restore form backup
are all valid.
I restored the databases in /etc/httpd/alias and
/etc/pki/pki-tomcat/alias from the last good backup I had.
I also restored the CS.cfg file from backup.
I updated the trusts in /etc/pki/pki-tomcat/alias for
caSigningCert cert-pki-ca
to match what is in section 5 of:
https://access.redhat.com/solutions/643753 This was previously:
caSigningCert cert-pki-ca CTu,u,u
for some reason.
I stopped the certmonger service and run the certmonger command you gave
to start verbose logging.
I was able to start all the ipa services after running:
pki-server subsystem-enable ca
(this seems to become disabled when the tomcatd service cannot start.
I ran getcert resubmit -i <requestid> for the expiring certificates.
The first one I tried (ocspSigningCert) renewed but gets an odd Subject.
It includes the hostname of one of my replica servers.
The other certificates have not renewed.
As you said, there is a large amount of info in the verbose certmonger
debug logs, but it is not immediately obvious to me what has gone wrong,
except that there are some instances of:
Internal error
Would you be prepared to have a look at the log file off-list (3.3MB
file, uncompressed) to see if it means more to you.
Sure, feel free to send it to me directly.
rob
4:13 a.m.
On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> Hi
>>>
>>> Our freeipa certificates need to be renewed due to passing their expiry
>>> dates.
>>>
>>> While some certificates have renewed ok, the ipaCert and
>>> auditSigningCert are renewing but the new certificates have the wrong
>>> Subject.
>>>
>>> Environment is:
>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>> serverB (replica) RHEL 7.3, ipa 4.4
>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>
>>> Once there are renewed certificates with the wrong Subject present,
>>> there are various problems with renewing the remaining certificates,
>>> which I think might be related to the bad Subject:
>>>
>>> 1) When just ipaCert has the wrong subject no further renewals happen
>>>
>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>> service will not start and no further renewals happen.
>>>
>>> I've been round the following loop many times on ServerA, our first
>>> master:
>>>
>>> 1) Restore good certificates from backup
>>> 2) Put the clock back to a time when certificates are all valid
>>> 3) Resubmit certificates for renewal
>>>
>>> Each time the ipaCert renews it has the same wrong Subject. The wrong
>>> Subject includes the host name of one of our ipa client systems.
>>>
>>> Each time the auditSigningCert renews it has the same wrong Subject but
>>> a different subject to the ipaCert. The wrong Subject in this case
>>> includes the host name of a system which has never been an ipa client,
>>> but might have been added and removed with ipa host-add and ipa host-del
>>> for testing something, a while ago.
>>>
>>> As far as I can see, the "cert_subject" is set correctly in the
file
>>> /var/lib/certmonger/<request id> until the point at which the
>>> certificate is actually renewed.
>>>
>>> I'd be very grateful for some pointers as to which configuration options
>>> and logs to check through to resolve this problem on our production
>>> system.
>>>
>>> If its of any relevance we did change which server is the first master
>>> some time ago.
>>
>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
>> the subject is.
>
> I'm not seeing any obvious CSR fields in the
> /etc/pki/pki-tomcat/ca/CS.cfg file.
foo.bar.certreq=
> The CSR in the certmonger requests file for the auditSigningCert seems
> to be showing with the correct Subject. This is different from the bad
> subject showing in the requests file field:
> cert_subject=
The value of cert_subject comes from the issued certificate.
> and the Subject which is showing in the 'getcert list' output (which is
> the same as that in the cert_subject= field.>
> I'm not quite sure what this all means.
It is displayed from the data within the tracked certmonger request.
certmonger logs to syslog so you can check there or you can stop the
process and run it manually with: certmonger -n -d 9 2>&1 | tee
certmonger.log
That will provide a lot of debugging output that may show what is going on.
I've restored certificate databases from backup and put the clock back
to a time when certificates are valid and renewed the ocspSigining
certificate with:
getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302
(I've previously tried without the -N with similar results)
What I am seeing in the certmonger logs is:
2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
different subject, removing certificate "ocspSigningCert cert-pki-ca"
with subject "CN=OCSP Subsystem,O=<REALM>".
2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:39 [48576] Adding hook
"/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"" (0).
2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" issued by CA and saved.
I now have a date valid ocspSigningCertificate, but with the wrong
subject, and a broken certificate system which will no longer start.
ipactl status
...
pki-tomcatd Service: STOPPED
I can't renew other expired certificates.
I also note that there is now no key for ocspSigningCertificate as shown by:
certutil -K -d /etc/pki/pki-tomcat/alias
I wonder if this is because the certificate subject changed? There was a
key before the certificate renewed.
The ca debug logs are showing:
[23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
'ocspSigningCert cert-pki-ca' with serial number: 268370108
[23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
[23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
object not found
Certificate object not found
at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
Any help in repairing my broken ipa system will be much appreciated.
Thanks
Roderick Johnstone
8:34 a.m.
Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> Hi
>>>>
>>>> Our freeipa certificates need to be renewed due to passing their
>>>> expiry
>>>> dates.
>>>>
>>>> While some certificates have renewed ok, the ipaCert and
>>>> auditSigningCert are renewing but the new certificates have the wrong
>>>> Subject.
>>>>
>>>> Environment is:
>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>
>>>> Once there are renewed certificates with the wrong Subject present,
>>>> there are various problems with renewing the remaining certificates,
>>>> which I think might be related to the bad Subject:
>>>>
>>>> 1) When just ipaCert has the wrong subject no further renewals happen
>>>>
>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>>> service will not start and no further renewals happen.
>>>>
>>>> I've been round the following loop many times on ServerA, our first
>>>> master:
>>>>
>>>> 1) Restore good certificates from backup
>>>> 2) Put the clock back to a time when certificates are all valid
>>>> 3) Resubmit certificates for renewal
>>>>
>>>> Each time the ipaCert renews it has the same wrong Subject. The wrong
>>>> Subject includes the host name of one of our ipa client systems.
>>>>
>>>> Each time the auditSigningCert renews it has the same wrong Subject
>>>> but
>>>> a different subject to the ipaCert. The wrong Subject in this case
>>>> includes the host name of a system which has never been an ipa client,
>>>> but might have been added and removed with ipa host-add and ipa
>>>> host-del
>>>> for testing something, a while ago.
>>>>
>>>> As far as I can see, the "cert_subject" is set correctly in the
file
>>>> /var/lib/certmonger/<request id> until the point at which the
>>>> certificate is actually renewed.
>>>>
>>>> I'd be very grateful for some pointers as to which configuration
>>>> options
>>>> and logs to check through to resolve this problem on our production
>>>> system.
>>>>
>>>> If its of any relevance we did change which server is the first master
>>>> some time ago.
>>>
>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
>>> the subject is.
>>
>> I'm not seeing any obvious CSR fields in the
>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>
> foo.bar.certreq=
>
>> The CSR in the certmonger requests file for the auditSigningCert seems
>> to be showing with the correct Subject. This is different from the bad
>> subject showing in the requests file field:
>> cert_subject=
>
> The value of cert_subject comes from the issued certificate.
>
>> and the Subject which is showing in the 'getcert list' output (which is
>> the same as that in the cert_subject= field.>
>> I'm not quite sure what this all means.
>
> It is displayed from the data within the tracked certmonger request.
>
> certmonger logs to syslog so you can check there or you can stop the
> process and run it manually with: certmonger -n -d 9 2>&1 | tee
> certmonger.log
>
> That will provide a lot of debugging output that may show what is
> going on.
I've restored certificate databases from backup and put the clock back
to a time when certificates are valid and renewed the ocspSigining
certificate with:
getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302
(I've previously tried without the -N with similar results)
What I am seeing in the certmonger logs is:
2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
different subject, removing certificate "ocspSigningCert cert-pki-ca"
with subject "CN=OCSP Subsystem,O=<REALM>".
2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:39 [48576] Adding hook
"/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"" (0).
2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" issued by CA and saved.
I now have a date valid ocspSigningCertificate, but with the wrong
subject, and a broken certificate system which will no longer start.
ipactl status
...
pki-tomcatd Service: STOPPED
I can't renew other expired certificates.
I also note that there is now no key for ocspSigningCertificate as shown
by:
certutil -K -d /etc/pki/pki-tomcat/alias
I wonder if this is because the certificate subject changed? There was a
key before the certificate renewed.
The ca debug logs are showing:
[23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
'ocspSigningCert cert-pki-ca' with serial number: 268370108
[23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
[23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
object not found
Certificate object not found
at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
Any help in repairing my broken ipa system will be much appreciated.
Sorry for the delay. I think my previous reading of the certmonger
csrgen code was wrong.
IIRC in your certmonger entry the cert_subject has the hostname value
right? Do you also have cert_subject_der?
You can decode that by:
1. Running a hex-to-bin, something hacky like this in python:
import binascii
hex_string = "<hex value>"
binary_string = binascii.unhexlify(hex_string)
fd = open("out", "wb")
fd.write(binary_string)
fd.close()
2. Run: openssl asn1parse -in out -inform der
I'm going to assume this also has the hostname encoded in it.
Can you try this:
1. Make a backup of the request file (just in case)
2. Remove cert_subject_der
3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
3. Try the renewal again
rob
4:11 a.m.
On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> Hi
>>>>>
>>>>> Our freeipa certificates need to be renewed due to passing their
>>>>> expiry
>>>>> dates.
>>>>>
>>>>> While some certificates have renewed ok, the ipaCert and
>>>>> auditSigningCert are renewing but the new certificates have the
wrong
>>>>> Subject.
>>>>>
>>>>> Environment is:
>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>
>>>>> Once there are renewed certificates with the wrong Subject present,
>>>>> there are various problems with renewing the remaining certificates,
>>>>> which I think might be related to the bad Subject:
>>>>>
>>>>> 1) When just ipaCert has the wrong subject no further renewals
happen
>>>>>
>>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>>>> service will not start and no further renewals happen.
>>>>>
>>>>> I've been round the following loop many times on ServerA, our
first
>>>>> master:
>>>>>
>>>>> 1) Restore good certificates from backup
>>>>> 2) Put the clock back to a time when certificates are all valid
>>>>> 3) Resubmit certificates for renewal
>>>>>
>>>>> Each time the ipaCert renews it has the same wrong Subject. The
wrong
>>>>> Subject includes the host name of one of our ipa client systems.
>>>>>
>>>>> Each time the auditSigningCert renews it has the same wrong Subject
>>>>> but
>>>>> a different subject to the ipaCert. The wrong Subject in this case
>>>>> includes the host name of a system which has never been an ipa
client,
>>>>> but might have been added and removed with ipa host-add and ipa
>>>>> host-del
>>>>> for testing something, a while ago.
>>>>>
>>>>> As far as I can see, the "cert_subject" is set correctly in
the file
>>>>> /var/lib/certmonger/<request id> until the point at which the
>>>>> certificate is actually renewed.
>>>>>
>>>>> I'd be very grateful for some pointers as to which configuration
>>>>> options
>>>>> and logs to check through to resolve this problem on our production
>>>>> system.
>>>>>
>>>>> If its of any relevance we did change which server is the first
master
>>>>> some time ago.
>>>>
>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see
what
>>>> the subject is.
>>>
>>> I'm not seeing any obvious CSR fields in the
>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>
>> foo.bar.certreq=
>>
>>> The CSR in the certmonger requests file for the auditSigningCert seems
>>> to be showing with the correct Subject. This is different from the bad
>>> subject showing in the requests file field:
>>> cert_subject=
>>
>> The value of cert_subject comes from the issued certificate.
>>
>>> and the Subject which is showing in the 'getcert list' output (which
is
>>> the same as that in the cert_subject= field.>
>>> I'm not quite sure what this all means.
>>
>> It is displayed from the data within the tracked certmonger request.
>>
>> certmonger logs to syslog so you can check there or you can stop the
>> process and run it manually with: certmonger -n -d 9 2>&1 | tee
>> certmonger.log
>>
>> That will provide a lot of debugging output that may show what is
>> going on.
>
> I've restored certificate databases from backup and put the clock back
> to a time when certificates are valid and renewed the ocspSigining
> certificate with:
> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302
>
> (I've previously tried without the -N with similar results)
>
> What I am seeing in the certmonger logs is:
>
>
> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'.
> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
> cert-pki-ca' to public key.
> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
> cert-pki-ca".
> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
> cert-pki-ca" in token "NSS Certificate DB" in database
> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'.
> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
> cert-pki-ca' to public key.
> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
> cert-pki-ca".
> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'.
> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
> cert-pki-ca' to public key.
> 2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
> different subject, removing certificate "ocspSigningCert cert-pki-ca"
> with subject "CN=OCSP Subsystem,O=<REALM>".
> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
> cert-pki-ca".
> 2017-10-23 00:05:39 [48576] Adding hook
> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
> cert-pki-ca"" (0).
> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
> cert-pki-ca" in token "NSS Certificate DB" in database
> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>
> I now have a date valid ocspSigningCertificate, but with the wrong
> subject, and a broken certificate system which will no longer start.
>
> ipactl status
> ...
> pki-tomcatd Service: STOPPED
>
> I can't renew other expired certificates.
>
> I also note that there is now no key for ocspSigningCertificate as shown
> by:
> certutil -K -d /etc/pki/pki-tomcat/alias
>
> I wonder if this is because the certificate subject changed? There was a
> key before the certificate renewed.
>
> The ca debug logs are showing:
>
> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
> object not found
> Certificate object not found
> at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>
> Any help in repairing my broken ipa system will be much appreciated.
Sorry for the delay. I think my previous reading of the certmonger
csrgen code was wrong.
IIRC in your certmonger entry the cert_subject has the hostname value
right? Do you also have cert_subject_der?
You can decode that by:
1. Running a hex-to-bin, something hacky like this in python:
import binascii
hex_string = "<hex value>"
binary_string = binascii.unhexlify(hex_string)
fd = open("out", "wb")
fd.write(binary_string)
fd.close()
2. Run: openssl asn1parse -in out -inform der
I'm going to assume this also has the hostname encoded in it.
Hi Again
Yes, correct. The cert_subject_der does have the bad CN with the
hostname encoded in it.
Can you try this:
1. Make a backup of the request file (just in case) > 2. Remove cert_subject_der
3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
3. Try the renewal again
I ran though all of this, checking that the request file was still what
we wanted after restarting certmonger with verbose logging, restoring
all the databases, fixing permissions, checking keys etc., and ran the
getcert resubmit.
It still generates a certificate with the bad CN including a hostname.
Then the pki-tomcat server fails to start again.
Other things I have checked are:
1) The csr= field in the (modified) request file has the correct subject.
2) The dogtag ca debug log file is showing:
processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
...
RenewalSubmitter: renewal original profileId=caIPAserviceCert
RenewalSubmitter: for renewal, original authenticator raCertAuth found
CertProcessor: No authenticator credentials required
processRenewal: set original Inputs into profile Context
RenewalSubmitter: setInputsIntoContext() getting input name=
cert_request_type
RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
RenewalSubmitter: setInputsIntoContext() getting input name= cert_request
RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
encoded csr>
...
Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
The hex encoded csr in the last line decodes to have the bad subject
where the hostname is one of our other ipa servers.
The certificate always getthe same hostname in the bad subject cn.
Do you have any other ideas of things to check to find out what's
generating the bad subject?
Thanks again.
Roderick
9:22 a.m.
Roderick Johnstone via FreeIPA-users wrote:
On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>> Hi
>>>>>>
>>>>>> Our freeipa certificates need to be renewed due to passing their
>>>>>> expiry
>>>>>> dates.
>>>>>>
>>>>>> While some certificates have renewed ok, the ipaCert and
>>>>>> auditSigningCert are renewing but the new certificates have the
>>>>>> wrong
>>>>>> Subject.
>>>>>>
>>>>>> Environment is:
>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>
>>>>>> Once there are renewed certificates with the wrong Subject
present,
>>>>>> there are various problems with renewing the remaining
certificates,
>>>>>> which I think might be related to the bad Subject:
>>>>>>
>>>>>> 1) When just ipaCert has the wrong subject no further renewals
>>>>>> happen
>>>>>>
>>>>>> 2) When auditSigningCert has the wrong subject the ipa
pki-tomcatd
>>>>>> service will not start and no further renewals happen.
>>>>>>
>>>>>> I've been round the following loop many times on ServerA, our
first
>>>>>> master:
>>>>>>
>>>>>> 1) Restore good certificates from backup
>>>>>> 2) Put the clock back to a time when certificates are all valid
>>>>>> 3) Resubmit certificates for renewal
>>>>>>
>>>>>> Each time the ipaCert renews it has the same wrong Subject. The
>>>>>> wrong
>>>>>> Subject includes the host name of one of our ipa client systems.
>>>>>>
>>>>>> Each time the auditSigningCert renews it has the same wrong
Subject
>>>>>> but
>>>>>> a different subject to the ipaCert. The wrong Subject in this
case
>>>>>> includes the host name of a system which has never been an ipa
>>>>>> client,
>>>>>> but might have been added and removed with ipa host-add and ipa
>>>>>> host-del
>>>>>> for testing something, a while ago.
>>>>>>
>>>>>> As far as I can see, the "cert_subject" is set
correctly in the file
>>>>>> /var/lib/certmonger/<request id> until the point at which
the
>>>>>> certificate is actually renewed.
>>>>>>
>>>>>> I'd be very grateful for some pointers as to which
configuration
>>>>>> options
>>>>>> and logs to check through to resolve this problem on our
production
>>>>>> system.
>>>>>>
>>>>>> If its of any relevance we did change which server is the first
>>>>>> master
>>>>>> some time ago.
>>>>>
>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see
what
>>>>> the subject is.
>>>>
>>>> I'm not seeing any obvious CSR fields in the
>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>
>>> foo.bar.certreq=
>>>
>>>> The CSR in the certmonger requests file for the auditSigningCert seems
>>>> to be showing with the correct Subject. This is different from the bad
>>>> subject showing in the requests file field:
>>>> cert_subject=
>>>
>>> The value of cert_subject comes from the issued certificate.
>>>
>>>> and the Subject which is showing in the 'getcert list' output
>>>> (which is
>>>> the same as that in the cert_subject= field.>
>>>> I'm not quite sure what this all means.
>>>
>>> It is displayed from the data within the tracked certmonger request.
>>>
>>> certmonger logs to syslog so you can check there or you can stop the
>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee
>>> certmonger.log
>>>
>>> That will provide a lot of debugging output that may show what is
>>> going on.
>>
>> I've restored certificate databases from backup and put the clock back
>> to a time when certificates are valid and renewed the ocspSigining
>> certificate with:
>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i
20161124081302
>>
>> (I've previously tried without the -N with similar results)
>>
>> What I am seeing in the certmonger logs is:
>>
>>
>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'.
>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
>> cert-pki-ca' to public key.
>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
>> cert-pki-ca".
>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
>> cert-pki-ca" in token "NSS Certificate DB" in database
>> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'.
>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
>> cert-pki-ca' to public key.
>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
>> cert-pki-ca".
>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'.
>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
>> cert-pki-ca' to public key.
>> 2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
>> different subject, removing certificate "ocspSigningCert cert-pki-ca"
>> with subject "CN=OCSP Subsystem,O=<REALM>".
>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
>> cert-pki-ca".
>> 2017-10-23 00:05:39 [48576] Adding hook
>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>> cert-pki-ca"" (0).
>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
>> cert-pki-ca" in token "NSS Certificate DB" in database
>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>
>> I now have a date valid ocspSigningCertificate, but with the wrong
>> subject, and a broken certificate system which will no longer start.
>>
>> ipactl status
>> ...
>> pki-tomcatd Service: STOPPED
>>
>> I can't renew other expired certificates.
>>
>> I also note that there is now no key for ocspSigningCertificate as shown
>> by:
>> certutil -K -d /etc/pki/pki-tomcat/alias
>>
>> I wonder if this is because the certificate subject changed? There was a
>> key before the certificate renewed.
>>
>> The ca debug logs are showing:
>>
>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
>> object not found
>> Certificate object not found
>> at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>
>> Any help in repairing my broken ipa system will be much appreciated.
>
> Sorry for the delay. I think my previous reading of the certmonger
> csrgen code was wrong.
>
> IIRC in your certmonger entry the cert_subject has the hostname value
> right? Do you also have cert_subject_der?
>
> You can decode that by:
>
> 1. Running a hex-to-bin, something hacky like this in python:
>
> import binascii
>
> hex_string = "<hex value>"
>
> binary_string = binascii.unhexlify(hex_string)
>
> fd = open("out", "wb")
> fd.write(binary_string)
> fd.close()
>
> 2. Run: openssl asn1parse -in out -inform der
>
> I'm going to assume this also has the hostname encoded in it.
Hi Again
Yes, correct. The cert_subject_der does have the bad CN with the
hostname encoded in it.
>
> Can you try this:
>
> 1. Make a backup of the request file (just in case) > 2. Remove
> cert_subject_der
> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
> 3. Try the renewal again
I ran though all of this, checking that the request file was still what
we wanted after restarting certmonger with verbose logging, restoring
all the databases, fixing permissions, checking keys etc., and ran the
getcert resubmit.
It still generates a certificate with the bad CN including a hostname.
Then the pki-tomcat server fails to start again.
Other things I have checked are:
1) The csr= field in the (modified) request file has the correct subject.
2) The dogtag ca debug log file is showing:
processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
...
RenewalSubmitter: renewal original profileId=caIPAserviceCert
RenewalSubmitter: for renewal, original authenticator raCertAuth found
CertProcessor: No authenticator credentials required
processRenewal: set original Inputs into profile Context
RenewalSubmitter: setInputsIntoContext() getting input name=
cert_request_type
RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
RenewalSubmitter: setInputsIntoContext() getting input name= cert_request
RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
encoded csr>
...
Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
The hex encoded csr in the last line decodes to have the bad subject
where the hostname is one of our other ipa servers.
The certificate always getthe same hostname in the bad subject cn.
Do you have any other ideas of things to check to find out what's
generating the bad subject?
The order certmonger uses for the subject is:
1. cert_subject_der
2. If there is no DER, try to encode cert_subject
3. If that fails try again a different way
4. If it fails yet again use CN=localhost
It is baffling that it would pick a hostname much less one that
certmonger shouldn't even know about.
I assume the CSR found in the dogtag logs matches the csr value in the
certmonger request?
Can you share the certmonger request file privately?
rob
11:18 a.m.
On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Our freeipa certificates need to be renewed due to passing
their
>>>>>>> expiry
>>>>>>> dates.
>>>>>>>
>>>>>>> While some certificates have renewed ok, the ipaCert and
>>>>>>> auditSigningCert are renewing but the new certificates have
the
>>>>>>> wrong
>>>>>>> Subject.
>>>>>>>
>>>>>>> Environment is:
>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>
>>>>>>> Once there are renewed certificates with the wrong Subject
present,
>>>>>>> there are various problems with renewing the remaining
certificates,
>>>>>>> which I think might be related to the bad Subject:
>>>>>>>
>>>>>>> 1) When just ipaCert has the wrong subject no further
renewals
>>>>>>> happen
>>>>>>>
>>>>>>> 2) When auditSigningCert has the wrong subject the ipa
pki-tomcatd
>>>>>>> service will not start and no further renewals happen.
>>>>>>>
>>>>>>> I've been round the following loop many times on ServerA,
our first
>>>>>>> master:
>>>>>>>
>>>>>>> 1) Restore good certificates from backup
>>>>>>> 2) Put the clock back to a time when certificates are all
valid
>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>
>>>>>>> Each time the ipaCert renews it has the same wrong Subject.
The
>>>>>>> wrong
>>>>>>> Subject includes the host name of one of our ipa client
systems.
>>>>>>>
>>>>>>> Each time the auditSigningCert renews it has the same wrong
Subject
>>>>>>> but
>>>>>>> a different subject to the ipaCert. The wrong Subject in this
case
>>>>>>> includes the host name of a system which has never been an
ipa
>>>>>>> client,
>>>>>>> but might have been added and removed with ipa host-add and
ipa
>>>>>>> host-del
>>>>>>> for testing something, a while ago.
>>>>>>>
>>>>>>> As far as I can see, the "cert_subject" is set
correctly in the file
>>>>>>> /var/lib/certmonger/<request id> until the point at
which the
>>>>>>> certificate is actually renewed.
>>>>>>>
>>>>>>> I'd be very grateful for some pointers as to which
configuration
>>>>>>> options
>>>>>>> and logs to check through to resolve this problem on our
production
>>>>>>> system.
>>>>>>>
>>>>>>> If its of any relevance we did change which server is the
first
>>>>>>> master
>>>>>>> some time ago.
>>>>>>
>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to
see what
>>>>>> the subject is.
>>>>>
>>>>> I'm not seeing any obvious CSR fields in the
>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>
>>>> foo.bar.certreq=
>>>>
>>>>> The CSR in the certmonger requests file for the auditSigningCert
seems
>>>>> to be showing with the correct Subject. This is different from the
bad
>>>>> subject showing in the requests file field:
>>>>> cert_subject=
>>>>
>>>> The value of cert_subject comes from the issued certificate.
>>>>
>>>>> and the Subject which is showing in the 'getcert list'
output
>>>>> (which is
>>>>> the same as that in the cert_subject= field.>
>>>>> I'm not quite sure what this all means.
>>>>
>>>> It is displayed from the data within the tracked certmonger request.
>>>>
>>>> certmonger logs to syslog so you can check there or you can stop the
>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee
>>>> certmonger.log
>>>>
>>>> That will provide a lot of debugging output that may show what is
>>>> going on.
>>>
>>> I've restored certificate databases from backup and put the clock back
>>> to a time when certificates are valid and renewed the ocspSigining
>>> certificate with:
>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i
20161124081302
>>>
>>> (I've previously tried without the -N with similar results)
>>>
>>> What I am seeing in the certmonger logs is:
>>>
>>>
>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert
cert-pki-ca'.
>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
>>> cert-pki-ca' to public key.
>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
>>> cert-pki-ca".
>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>> "/etc/pki/pki-tomcat/alias" will not be valid after
20171025122401.
>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert
cert-pki-ca'.
>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
>>> cert-pki-ca' to public key.
>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
>>> cert-pki-ca".
>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert
cert-pki-ca'.
>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
>>> cert-pki-ca' to public key.
>>> 2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
>>> different subject, removing certificate "ocspSigningCert
cert-pki-ca"
>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
>>> cert-pki-ca".
>>> 2017-10-23 00:05:39 [48576] Adding hook
>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>> cert-pki-ca"" (0).
>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>>
>>> I now have a date valid ocspSigningCertificate, but with the wrong
>>> subject, and a broken certificate system which will no longer start.
>>>
>>> ipactl status
>>> ...
>>> pki-tomcatd Service: STOPPED
>>>
>>> I can't renew other expired certificates.
>>>
>>> I also note that there is now no key for ocspSigningCertificate as shown
>>> by:
>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>
>>> I wonder if this is because the certificate subject changed? There was a
>>> key before the certificate renewed.
>>>
>>> The ca debug logs are showing:
>>>
>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
>>> object not found
>>> Certificate object not found
>>> at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>
>>> Any help in repairing my broken ipa system will be much appreciated.
>>
>> Sorry for the delay. I think my previous reading of the certmonger
>> csrgen code was wrong.
>>
>> IIRC in your certmonger entry the cert_subject has the hostname value
>> right? Do you also have cert_subject_der?
>>
>> You can decode that by:
>>
>> 1. Running a hex-to-bin, something hacky like this in python:
>>
>> import binascii
>>
>> hex_string = "<hex value>"
>>
>> binary_string = binascii.unhexlify(hex_string)
>>
>> fd = open("out", "wb")
>> fd.write(binary_string)
>> fd.close()
>>
>> 2. Run: openssl asn1parse -in out -inform der
>>
>> I'm going to assume this also has the hostname encoded in it.
>
> Hi Again
>
> Yes, correct. The cert_subject_der does have the bad CN with the
> hostname encoded in it.
>
>>
>> Can you try this:
>>
>> 1. Make a backup of the request file (just in case) > 2. Remove
>> cert_subject_der
>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
>> 3. Try the renewal again
>
> I ran though all of this, checking that the request file was still what
> we wanted after restarting certmonger with verbose logging, restoring
> all the databases, fixing permissions, checking keys etc., and ran the
> getcert resubmit.
>
> It still generates a certificate with the bad CN including a hostname.
>
> Then the pki-tomcat server fails to start again.
>
> Other things I have checked are:
> 1) The csr= field in the (modified) request file has the correct subject.
>
> 2) The dogtag ca debug log file is showing:
> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
> ...
> RenewalSubmitter: renewal original profileId=caIPAserviceCert
> RenewalSubmitter: for renewal, original authenticator raCertAuth found
> CertProcessor: No authenticator credentials required
> processRenewal: set original Inputs into profile Context
> RenewalSubmitter: setInputsIntoContext() getting input name=
> cert_request_type
> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
> RenewalSubmitter: setInputsIntoContext() getting input name= cert_request
> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
> encoded csr>
> ...
> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>
> The hex encoded csr in the last line decodes to have the bad subject
> where the hostname is one of our other ipa servers.
>
> The certificate always getthe same hostname in the bad subject cn.
>
> Do you have any other ideas of things to check to find out what's
> generating the bad subject?
The order certmonger uses for the subject is:
1. cert_subject_der
2. If there is no DER, try to encode cert_subject
3. If that fails try again a different way
4. If it fails yet again use CN=localhost
It is baffling that it would pick a hostname much less one that
certmonger shouldn't even know about.
Indeed, and if I try to renew other certificates for some of them it
chooses other host names that should not be known about. Each
certificate seems to get the same bad hostname each time I try to renew.
I assume the CSR found in the dogtag logs matches the csr value in the
certmonger request?
No, its different. The issued certificate matches the csr seen in dogtag
which makes sense. But the csr in the dogtag logs has the bad subject.
The csr in the request directory file has the good subject.
Can you share the certmonger request file privately?
Yes, certainly.
Thanks for your continued help on this.
Roderick
3:09 p.m.
Roderick Johnstone via FreeIPA-users wrote:
On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Our freeipa certificates need to be renewed due to
passing their
>>>>>>>> expiry
>>>>>>>> dates.
>>>>>>>>
>>>>>>>> While some certificates have renewed ok, the ipaCert and
>>>>>>>> auditSigningCert are renewing but the new certificates
have the
>>>>>>>> wrong
>>>>>>>> Subject.
>>>>>>>>
>>>>>>>> Environment is:
>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>
>>>>>>>> Once there are renewed certificates with the wrong
Subject
>>>>>>>> present,
>>>>>>>> there are various problems with renewing the remaining
>>>>>>>> certificates,
>>>>>>>> which I think might be related to the bad Subject:
>>>>>>>>
>>>>>>>> 1) When just ipaCert has the wrong subject no further
renewals
>>>>>>>> happen
>>>>>>>>
>>>>>>>> 2) When auditSigningCert has the wrong subject the ipa
pki-tomcatd
>>>>>>>> service will not start and no further renewals happen.
>>>>>>>>
>>>>>>>> I've been round the following loop many times on
ServerA, our
>>>>>>>> first
>>>>>>>> master:
>>>>>>>>
>>>>>>>> 1) Restore good certificates from backup
>>>>>>>> 2) Put the clock back to a time when certificates are all
valid
>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>
>>>>>>>> Each time the ipaCert renews it has the same wrong
Subject. The
>>>>>>>> wrong
>>>>>>>> Subject includes the host name of one of our ipa client
systems.
>>>>>>>>
>>>>>>>> Each time the auditSigningCert renews it has the same
wrong
>>>>>>>> Subject
>>>>>>>> but
>>>>>>>> a different subject to the ipaCert. The wrong Subject in
this case
>>>>>>>> includes the host name of a system which has never been
an ipa
>>>>>>>> client,
>>>>>>>> but might have been added and removed with ipa host-add
and ipa
>>>>>>>> host-del
>>>>>>>> for testing something, a while ago.
>>>>>>>>
>>>>>>>> As far as I can see, the "cert_subject" is set
correctly in the
>>>>>>>> file
>>>>>>>> /var/lib/certmonger/<request id> until the point at
which the
>>>>>>>> certificate is actually renewed.
>>>>>>>>
>>>>>>>> I'd be very grateful for some pointers as to which
configuration
>>>>>>>> options
>>>>>>>> and logs to check through to resolve this problem on our
>>>>>>>> production
>>>>>>>> system.
>>>>>>>>
>>>>>>>> If its of any relevance we did change which server is the
first
>>>>>>>> master
>>>>>>>> some time ago.
>>>>>>>
>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger
to see
>>>>>>> what
>>>>>>> the subject is.
>>>>>>
>>>>>> I'm not seeing any obvious CSR fields in the
>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>
>>>>> foo.bar.certreq=
>>>>>
>>>>>> The CSR in the certmonger requests file for the auditSigningCert
>>>>>> seems
>>>>>> to be showing with the correct Subject. This is different from
>>>>>> the bad
>>>>>> subject showing in the requests file field:
>>>>>> cert_subject=
>>>>>
>>>>> The value of cert_subject comes from the issued certificate.
>>>>>
>>>>>> and the Subject which is showing in the 'getcert list'
output
>>>>>> (which is
>>>>>> the same as that in the cert_subject= field.>
>>>>>> I'm not quite sure what this all means.
>>>>>
>>>>> It is displayed from the data within the tracked certmonger request.
>>>>>
>>>>> certmonger logs to syslog so you can check there or you can stop the
>>>>> process and run it manually with: certmonger -n -d 9 2>&1 |
tee
>>>>> certmonger.log
>>>>>
>>>>> That will provide a lot of debugging output that may show what is
>>>>> going on.
>>>>
>>>> I've restored certificate databases from backup and put the clock
back
>>>> to a time when certificates are valid and renewed the ocspSigining
>>>> certificate with:
>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i
20161124081302
>>>>
>>>> (I've previously tried without the -N with similar results)
>>>>
>>>> What I am seeing in the certmonger logs is:
>>>>
>>>>
>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert
>>>> cert-pki-ca'.
>>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
>>>> cert-pki-ca' to public key.
>>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
>>>> cert-pki-ca".
>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
>>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>>> "/etc/pki/pki-tomcat/alias" will not be valid after
20171025122401.
>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert
>>>> cert-pki-ca'.
>>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
>>>> cert-pki-ca' to public key.
>>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
>>>> cert-pki-ca".
>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert
>>>> cert-pki-ca'.
>>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
>>>> cert-pki-ca' to public key.
>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same
>>>> nickname but
>>>> different subject, removing certificate "ocspSigningCert
cert-pki-ca"
>>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
>>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
>>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
>>>> cert-pki-ca".
>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>>> cert-pki-ca"" (0).
>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
>>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>>>
>>>> I now have a date valid ocspSigningCertificate, but with the wrong
>>>> subject, and a broken certificate system which will no longer start.
>>>>
>>>> ipactl status
>>>> ...
>>>> pki-tomcatd Service: STOPPED
>>>>
>>>> I can't renew other expired certificates.
>>>>
>>>> I also note that there is now no key for ocspSigningCertificate as
>>>> shown
>>>> by:
>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>
>>>> I wonder if this is because the certificate subject changed? There
>>>> was a
>>>> key before the certificate renewed.
>>>>
>>>> The ca debug logs are showing:
>>>>
>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>> x509CertImpl
>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>> Certificate
>>>> object not found
>>>> Certificate object not found
>>>> at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>
>>>> Any help in repairing my broken ipa system will be much appreciated.
>>>
>>> Sorry for the delay. I think my previous reading of the certmonger
>>> csrgen code was wrong.
>>>
>>> IIRC in your certmonger entry the cert_subject has the hostname value
>>> right? Do you also have cert_subject_der?
>>>
>>> You can decode that by:
>>>
>>> 1. Running a hex-to-bin, something hacky like this in python:
>>>
>>> import binascii
>>>
>>> hex_string = "<hex value>"
>>>
>>> binary_string = binascii.unhexlify(hex_string)
>>>
>>> fd = open("out", "wb")
>>> fd.write(binary_string)
>>> fd.close()
>>>
>>> 2. Run: openssl asn1parse -in out -inform der
>>>
>>> I'm going to assume this also has the hostname encoded in it.
>>
>> Hi Again
>>
>> Yes, correct. The cert_subject_der does have the bad CN with the
>> hostname encoded in it.
>>
>>>
>>> Can you try this:
>>>
>>> 1. Make a backup of the request file (just in case) > 2. Remove
>>> cert_subject_der
>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
>>> 3. Try the renewal again
>>
>> I ran though all of this, checking that the request file was still what
>> we wanted after restarting certmonger with verbose logging, restoring
>> all the databases, fixing permissions, checking keys etc., and ran the
>> getcert resubmit.
>>
>> It still generates a certificate with the bad CN including a hostname.
>>
>> Then the pki-tomcat server fails to start again.
>>
>> Other things I have checked are:
>> 1) The csr= field in the (modified) request file has the correct
>> subject.
>>
>> 2) The dogtag ca debug log file is showing:
>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>> ...
>> RenewalSubmitter: renewal original profileId=caIPAserviceCert
>> RenewalSubmitter: for renewal, original authenticator raCertAuth found
>> CertProcessor: No authenticator credentials required
>> processRenewal: set original Inputs into profile Context
>> RenewalSubmitter: setInputsIntoContext() getting input name=
>> cert_request_type
>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
>> RenewalSubmitter: setInputsIntoContext() getting input name=
>> cert_request
>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
>> encoded csr>
>> ...
>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>
>> The hex encoded csr in the last line decodes to have the bad subject
>> where the hostname is one of our other ipa servers.
>>
>> The certificate always getthe same hostname in the bad subject cn.
>>
>> Do you have any other ideas of things to check to find out what's
>> generating the bad subject?
>
> The order certmonger uses for the subject is:
>
> 1. cert_subject_der
> 2. If there is no DER, try to encode cert_subject
> 3. If that fails try again a different way
> 4. If it fails yet again use CN=localhost
>
> It is baffling that it would pick a hostname much less one that
> certmonger shouldn't even know about.
Indeed, and if I try to renew other certificates for some of them it
chooses other host names that should not be known about. Each
certificate seems to get the same bad hostname each time I try to renew.
>
> I assume the CSR found in the dogtag logs matches the csr value in the
> certmonger request?
No, its different. The issued certificate matches the csr seen in dogtag
which makes sense. But the csr in the dogtag logs has the bad subject.
The csr in the request directory file has the good subject.
>
> Can you share the certmonger request file privately?
Yes, certainly.
Thanks for your continued help on this.
Let's try this. We'll drop the current request and create a new one.
Stop tomcat, restore the old cert database, start tomcat, then:
# getcert stop-tracking -i <request id>
# getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
"ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin> -B
/usr/libexec/ipa/certmonger/stop_pkicad -C
'/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"'
Then try resubmitting the request.
rob
6:11 a.m.
On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Our freeipa certificates need to be renewed due to
passing their
>>>>>>>>> expiry
>>>>>>>>> dates.
>>>>>>>>>
>>>>>>>>> While some certificates have renewed ok, the ipaCert
and
>>>>>>>>> auditSigningCert are renewing but the new
certificates have the
>>>>>>>>> wrong
>>>>>>>>> Subject.
>>>>>>>>>
>>>>>>>>> Environment is:
>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>>
>>>>>>>>> Once there are renewed certificates with the wrong
Subject
>>>>>>>>> present,
>>>>>>>>> there are various problems with renewing the
remaining
>>>>>>>>> certificates,
>>>>>>>>> which I think might be related to the bad Subject:
>>>>>>>>>
>>>>>>>>> 1) When just ipaCert has the wrong subject no further
renewals
>>>>>>>>> happen
>>>>>>>>>
>>>>>>>>> 2) When auditSigningCert has the wrong subject the
ipa pki-tomcatd
>>>>>>>>> service will not start and no further renewals
happen.
>>>>>>>>>
>>>>>>>>> I've been round the following loop many times on
ServerA, our
>>>>>>>>> first
>>>>>>>>> master:
>>>>>>>>>
>>>>>>>>> 1) Restore good certificates from backup
>>>>>>>>> 2) Put the clock back to a time when certificates are
all valid
>>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>>
>>>>>>>>> Each time the ipaCert renews it has the same wrong
Subject. The
>>>>>>>>> wrong
>>>>>>>>> Subject includes the host name of one of our ipa
client systems.
>>>>>>>>>
>>>>>>>>> Each time the auditSigningCert renews it has the same
wrong
>>>>>>>>> Subject
>>>>>>>>> but
>>>>>>>>> a different subject to the ipaCert. The wrong Subject
in this case
>>>>>>>>> includes the host name of a system which has never
been an ipa
>>>>>>>>> client,
>>>>>>>>> but might have been added and removed with ipa
host-add and ipa
>>>>>>>>> host-del
>>>>>>>>> for testing something, a while ago.
>>>>>>>>>
>>>>>>>>> As far as I can see, the "cert_subject" is
set correctly in the
>>>>>>>>> file
>>>>>>>>> /var/lib/certmonger/<request id> until the
point at which the
>>>>>>>>> certificate is actually renewed.
>>>>>>>>>
>>>>>>>>> I'd be very grateful for some pointers as to
which configuration
>>>>>>>>> options
>>>>>>>>> and logs to check through to resolve this problem on
our
>>>>>>>>> production
>>>>>>>>> system.
>>>>>>>>>
>>>>>>>>> If its of any relevance we did change which server is
the first
>>>>>>>>> master
>>>>>>>>> some time ago.
>>>>>>>>
>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or
certmonger to see
>>>>>>>> what
>>>>>>>> the subject is.
>>>>>>>
>>>>>>> I'm not seeing any obvious CSR fields in the
>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>
>>>>>> foo.bar.certreq=
>>>>>>
>>>>>>> The CSR in the certmonger requests file for the
auditSigningCert
>>>>>>> seems
>>>>>>> to be showing with the correct Subject. This is different
from
>>>>>>> the bad
>>>>>>> subject showing in the requests file field:
>>>>>>> cert_subject=
>>>>>>
>>>>>> The value of cert_subject comes from the issued certificate.
>>>>>>
>>>>>>> and the Subject which is showing in the 'getcert
list' output
>>>>>>> (which is
>>>>>>> the same as that in the cert_subject= field.>
>>>>>>> I'm not quite sure what this all means.
>>>>>>
>>>>>> It is displayed from the data within the tracked certmonger
request.
>>>>>>
>>>>>> certmonger logs to syslog so you can check there or you can stop
the
>>>>>> process and run it manually with: certmonger -n -d 9 2>&1
| tee
>>>>>> certmonger.log
>>>>>>
>>>>>> That will provide a lot of debugging output that may show what
is
>>>>>> going on.
>>>>>
>>>>> I've restored certificate databases from backup and put the clock
back
>>>>> to a time when certificates are valid and renewed the ocspSigining
>>>>> certificate with:
>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i
20161124081302
>>>>>
>>>>> (I've previously tried without the -N with similar results)
>>>>>
>>>>> What I am seeing in the certmonger logs is:
>>>>>
>>>>>
>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert
>>>>> cert-pki-ca'.
>>>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
>>>>> cert-pki-ca' to public key.
>>>>> 2017-10-23 00:05:28 [439] Located the certificate
"ocspSigningCert
>>>>> cert-pki-ca".
>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
>>>>> cert-pki-ca" in token "NSS Certificate DB" in
database
>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after
20171025122401.
>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert
>>>>> cert-pki-ca'.
>>>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
>>>>> cert-pki-ca' to public key.
>>>>> 2017-10-23 00:05:28 [443] Located the certificate
"ocspSigningCert
>>>>> cert-pki-ca".
>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert
>>>>> cert-pki-ca'.
>>>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
>>>>> cert-pki-ca' to public key.
>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same
>>>>> nickname but
>>>>> different subject, removing certificate "ocspSigningCert
cert-pki-ca"
>>>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
>>>>> cert-pki-ca", got nickname "ocspSigningCert
cert-pki-ca".
>>>>> 2017-10-23 00:05:39 [583] Located the certificate
"ocspSigningCert
>>>>> cert-pki-ca".
>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>> cert-pki-ca"" (0).
>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
>>>>> cert-pki-ca" in token "NSS Certificate DB" in
database
>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>>>>
>>>>> I now have a date valid ocspSigningCertificate, but with the wrong
>>>>> subject, and a broken certificate system which will no longer start.
>>>>>
>>>>> ipactl status
>>>>> ...
>>>>> pki-tomcatd Service: STOPPED
>>>>>
>>>>> I can't renew other expired certificates.
>>>>>
>>>>> I also note that there is now no key for ocspSigningCertificate as
>>>>> shown
>>>>> by:
>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>
>>>>> I wonder if this is because the certificate subject changed? There
>>>>> was a
>>>>> key before the certificate renewed.
>>>>>
>>>>> The ca debug logs are showing:
>>>>>
>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by
nickname:
>>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>>> x509CertImpl
>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>>> Certificate
>>>>> object not found
>>>>> Certificate object not found
>>>>> at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>
>>>>> Any help in repairing my broken ipa system will be much appreciated.
>>>>
>>>> Sorry for the delay. I think my previous reading of the certmonger
>>>> csrgen code was wrong.
>>>>
>>>> IIRC in your certmonger entry the cert_subject has the hostname value
>>>> right? Do you also have cert_subject_der?
>>>>
>>>> You can decode that by:
>>>>
>>>> 1. Running a hex-to-bin, something hacky like this in python:
>>>>
>>>> import binascii
>>>>
>>>> hex_string = "<hex value>"
>>>>
>>>> binary_string = binascii.unhexlify(hex_string)
>>>>
>>>> fd = open("out", "wb")
>>>> fd.write(binary_string)
>>>> fd.close()
>>>>
>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>
>>>> I'm going to assume this also has the hostname encoded in it.
>>>
>>> Hi Again
>>>
>>> Yes, correct. The cert_subject_der does have the bad CN with the
>>> hostname encoded in it.
>>>
>>>>
>>>> Can you try this:
>>>>
>>>> 1. Make a backup of the request file (just in case) > 2. Remove
>>>> cert_subject_der
>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
>>>> 3. Try the renewal again
>>>
>>> I ran though all of this, checking that the request file was still what
>>> we wanted after restarting certmonger with verbose logging, restoring
>>> all the databases, fixing permissions, checking keys etc., and ran the
>>> getcert resubmit.
>>>
>>> It still generates a certificate with the bad CN including a hostname.
>>>
>>> Then the pki-tomcat server fails to start again.
>>>
>>> Other things I have checked are:
>>> 1) The csr= field in the (modified) request file has the correct
>>> subject.
>>>
>>> 2) The dogtag ca debug log file is showing:
>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>>> ...
>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert
>>> RenewalSubmitter: for renewal, original authenticator raCertAuth found
>>> CertProcessor: No authenticator credentials required
>>> processRenewal: set original Inputs into profile Context
>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>> cert_request_type
>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>> cert_request
>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
>>> encoded csr>
>>> ...
>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>
>>> The hex encoded csr in the last line decodes to have the bad subject
>>> where the hostname is one of our other ipa servers.
>>>
>>> The certificate always getthe same hostname in the bad subject cn.
>>>
>>> Do you have any other ideas of things to check to find out what's
>>> generating the bad subject?
>>
>> The order certmonger uses for the subject is:
>>
>> 1. cert_subject_der
>> 2. If there is no DER, try to encode cert_subject
>> 3. If that fails try again a different way
>> 4. If it fails yet again use CN=localhost
>>
>> It is baffling that it would pick a hostname much less one that
>> certmonger shouldn't even know about.
>
> Indeed, and if I try to renew other certificates for some of them it
> chooses other host names that should not be known about. Each
> certificate seems to get the same bad hostname each time I try to renew.
>
>>
>> I assume the CSR found in the dogtag logs matches the csr value in the
>> certmonger request?
>
> No, its different. The issued certificate matches the csr seen in dogtag
> which makes sense. But the csr in the dogtag logs has the bad subject.
> The csr in the request directory file has the good subject.
>
>>
>> Can you share the certmonger request file privately?
>
> Yes, certainly.
>
> Thanks for your continued help on this.
Let's try this. We'll drop the current request and create a new one.
Stop tomcat, restore the old cert database, start tomcat, then:
# getcert stop-tracking -i <request id>
# getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
"ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin> -B
/usr/libexec/ipa/certmonger/stop_pkicad -C
'/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"'
Then try resubmitting the request.
Hi Rob
When you say 'stop tomcat', I'm just doing an ipactl stop. That is
stopiing the ipa pki-tomcat service. Is that good enough or are there
other services that need stopping too?
I tried the stop-tracking/start-tracking. Exactly the same result as
before. Same hostname appears in the subject CN field of the new
certificate and then the pki-tomcat service won't start.
I've also been back and redone the manual submits changing the point at
which I copied in the edited request as I wondered if there was some
caching of csr's going on in certmonger and it was remembering a
previous request it had read on startup before I replaced the request file.
But nothing seems to stop dogtag issuing a certificate with the Subject
CN being that host name.
Is it possible that dogtag is somehow overriding the Subject its being
explicitly given?
FWIW the CSR in the dogtag debug log is always identical, but I guess
thats expected if the CSR information is always the same.
I'm not sure if its possible to increase the verbosity on the dogtag
side. Can you advise please?
I've been assuming that its the bad subject that is preventing the
pki-tomcat from starting after the new certificate is issued. Does that
make sense to you?
I wonder where we can go from here? There must be a good reason for
whats happening!
Thanks again.
Roderick
7:43 a.m.
Roderick Johnstone via FreeIPA-users wrote:
On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Our freeipa certificates need to be renewed due
to passing their
>>>>>>>>>> expiry
>>>>>>>>>> dates.
>>>>>>>>>>
>>>>>>>>>> While some certificates have renewed ok, the
ipaCert and
>>>>>>>>>> auditSigningCert are renewing but the new
certificates have the
>>>>>>>>>> wrong
>>>>>>>>>> Subject.
>>>>>>>>>>
>>>>>>>>>> Environment is:
>>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>>>
>>>>>>>>>> Once there are renewed certificates with the
wrong Subject
>>>>>>>>>> present,
>>>>>>>>>> there are various problems with renewing the
remaining
>>>>>>>>>> certificates,
>>>>>>>>>> which I think might be related to the bad
Subject:
>>>>>>>>>>
>>>>>>>>>> 1) When just ipaCert has the wrong subject no
further renewals
>>>>>>>>>> happen
>>>>>>>>>>
>>>>>>>>>> 2) When auditSigningCert has the wrong subject
the ipa
>>>>>>>>>> pki-tomcatd
>>>>>>>>>> service will not start and no further renewals
happen.
>>>>>>>>>>
>>>>>>>>>> I've been round the following loop many times
on ServerA, our
>>>>>>>>>> first
>>>>>>>>>> master:
>>>>>>>>>>
>>>>>>>>>> 1) Restore good certificates from backup
>>>>>>>>>> 2) Put the clock back to a time when certificates
are all valid
>>>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>>>
>>>>>>>>>> Each time the ipaCert renews it has the same
wrong Subject. The
>>>>>>>>>> wrong
>>>>>>>>>> Subject includes the host name of one of our ipa
client systems.
>>>>>>>>>>
>>>>>>>>>> Each time the auditSigningCert renews it has the
same wrong
>>>>>>>>>> Subject
>>>>>>>>>> but
>>>>>>>>>> a different subject to the ipaCert. The wrong
Subject in this
>>>>>>>>>> case
>>>>>>>>>> includes the host name of a system which has
never been an ipa
>>>>>>>>>> client,
>>>>>>>>>> but might have been added and removed with ipa
host-add and ipa
>>>>>>>>>> host-del
>>>>>>>>>> for testing something, a while ago.
>>>>>>>>>>
>>>>>>>>>> As far as I can see, the "cert_subject"
is set correctly in the
>>>>>>>>>> file
>>>>>>>>>> /var/lib/certmonger/<request id> until the
point at which the
>>>>>>>>>> certificate is actually renewed.
>>>>>>>>>>
>>>>>>>>>> I'd be very grateful for some pointers as to
which configuration
>>>>>>>>>> options
>>>>>>>>>> and logs to check through to resolve this problem
on our
>>>>>>>>>> production
>>>>>>>>>> system.
>>>>>>>>>>
>>>>>>>>>> If its of any relevance we did change which
server is the first
>>>>>>>>>> master
>>>>>>>>>> some time ago.
>>>>>>>>>
>>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or
certmonger to see
>>>>>>>>> what
>>>>>>>>> the subject is.
>>>>>>>>
>>>>>>>> I'm not seeing any obvious CSR fields in the
>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>>
>>>>>>> foo.bar.certreq=
>>>>>>>
>>>>>>>> The CSR in the certmonger requests file for the
auditSigningCert
>>>>>>>> seems
>>>>>>>> to be showing with the correct Subject. This is different
from
>>>>>>>> the bad
>>>>>>>> subject showing in the requests file field:
>>>>>>>> cert_subject=
>>>>>>>
>>>>>>> The value of cert_subject comes from the issued certificate.
>>>>>>>
>>>>>>>> and the Subject which is showing in the 'getcert
list' output
>>>>>>>> (which is
>>>>>>>> the same as that in the cert_subject= field.>
>>>>>>>> I'm not quite sure what this all means.
>>>>>>>
>>>>>>> It is displayed from the data within the tracked certmonger
>>>>>>> request.
>>>>>>>
>>>>>>> certmonger logs to syslog so you can check there or you can
stop
>>>>>>> the
>>>>>>> process and run it manually with: certmonger -n -d 9
2>&1 | tee
>>>>>>> certmonger.log
>>>>>>>
>>>>>>> That will provide a lot of debugging output that may show
what is
>>>>>>> going on.
>>>>>>
>>>>>> I've restored certificate databases from backup and put the
clock
>>>>>> back
>>>>>> to a time when certificates are valid and renewed the
ocspSigining
>>>>>> certificate with:
>>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>"
-i 20161124081302
>>>>>>
>>>>>> (I've previously tried without the -N with similar results)
>>>>>>
>>>>>> What I am seeing in the certmonger logs is:
>>>>>>
>>>>>>
>>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert
>>>>>> cert-pki-ca'.
>>>>>> 2017-10-23 00:05:28 [438] Converted private key
'ocspSigningCert
>>>>>> cert-pki-ca' to public key.
>>>>>> 2017-10-23 00:05:28 [439] Located the certificate
"ocspSigningCert
>>>>>> cert-pki-ca".
>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
>>>>>> cert-pki-ca" in token "NSS Certificate DB" in
database
>>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after
20171025122401.
>>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert
>>>>>> cert-pki-ca'.
>>>>>> 2017-10-23 00:05:28 [442] Converted private key
'ocspSigningCert
>>>>>> cert-pki-ca' to public key.
>>>>>> 2017-10-23 00:05:28 [443] Located the certificate
"ocspSigningCert
>>>>>> cert-pki-ca".
>>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert
>>>>>> cert-pki-ca'.
>>>>>> 2017-10-23 00:05:28 [444] Converted private key
'ocspSigningCert
>>>>>> cert-pki-ca' to public key.
>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same
>>>>>> nickname but
>>>>>> different subject, removing certificate "ocspSigningCert
>>>>>> cert-pki-ca"
>>>>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>>>>> 2017-10-23 00:05:39 [581] Imported certificate
"ocspSigningCert
>>>>>> cert-pki-ca", got nickname "ocspSigningCert
cert-pki-ca".
>>>>>> 2017-10-23 00:05:39 [583] Located the certificate
"ocspSigningCert
>>>>>> cert-pki-ca".
>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>> cert-pki-ca"" (0).
>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
>>>>>> cert-pki-ca" in token "NSS Certificate DB" in
database
>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>>>>>
>>>>>> I now have a date valid ocspSigningCertificate, but with the
wrong
>>>>>> subject, and a broken certificate system which will no longer
start.
>>>>>>
>>>>>> ipactl status
>>>>>> ...
>>>>>> pki-tomcatd Service: STOPPED
>>>>>>
>>>>>> I can't renew other expired certificates.
>>>>>>
>>>>>> I also note that there is now no key for ocspSigningCertificate
as
>>>>>> shown
>>>>>> by:
>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>
>>>>>> I wonder if this is because the certificate subject changed?
There
>>>>>> was a
>>>>>> key before the certificate renewed.
>>>>>>
>>>>>> The ca debug logs are showing:
>>>>>>
>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by
>>>>>> nickname:
>>>>>> 'ocspSigningCert cert-pki-ca' with serial number:
268370108
>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>>>> x509CertImpl
>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>>>> Certificate
>>>>>> object not found
>>>>>> Certificate object not found
>>>>>> at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>
>>>>>> Any help in repairing my broken ipa system will be much
appreciated.
>>>>>
>>>>> Sorry for the delay. I think my previous reading of the certmonger
>>>>> csrgen code was wrong.
>>>>>
>>>>> IIRC in your certmonger entry the cert_subject has the hostname
value
>>>>> right? Do you also have cert_subject_der?
>>>>>
>>>>> You can decode that by:
>>>>>
>>>>> 1. Running a hex-to-bin, something hacky like this in python:
>>>>>
>>>>> import binascii
>>>>>
>>>>> hex_string = "<hex value>"
>>>>>
>>>>> binary_string = binascii.unhexlify(hex_string)
>>>>>
>>>>> fd = open("out", "wb")
>>>>> fd.write(binary_string)
>>>>> fd.close()
>>>>>
>>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>>
>>>>> I'm going to assume this also has the hostname encoded in it.
>>>>
>>>> Hi Again
>>>>
>>>> Yes, correct. The cert_subject_der does have the bad CN with the
>>>> hostname encoded in it.
>>>>
>>>>>
>>>>> Can you try this:
>>>>>
>>>>> 1. Make a backup of the request file (just in case) > 2. Remove
>>>>> cert_subject_der
>>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
>>>>> 3. Try the renewal again
>>>>
>>>> I ran though all of this, checking that the request file was still
>>>> what
>>>> we wanted after restarting certmonger with verbose logging, restoring
>>>> all the databases, fixing permissions, checking keys etc., and ran the
>>>> getcert resubmit.
>>>>
>>>> It still generates a certificate with the bad CN including a hostname.
>>>>
>>>> Then the pki-tomcat server fails to start again.
>>>>
>>>> Other things I have checked are:
>>>> 1) The csr= field in the (modified) request file has the correct
>>>> subject.
>>>>
>>>> 2) The dogtag ca debug log file is showing:
>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>>>> ...
>>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert
>>>> RenewalSubmitter: for renewal, original authenticator raCertAuth found
>>>> CertProcessor: No authenticator credentials required
>>>> processRenewal: set original Inputs into profile Context
>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>> cert_request_type
>>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>> cert_request
>>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
>>>> encoded csr>
>>>> ...
>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>>
>>>> The hex encoded csr in the last line decodes to have the bad subject
>>>> where the hostname is one of our other ipa servers.
>>>>
>>>> The certificate always getthe same hostname in the bad subject cn.
>>>>
>>>> Do you have any other ideas of things to check to find out what's
>>>> generating the bad subject?
>>>
>>> The order certmonger uses for the subject is:
>>>
>>> 1. cert_subject_der
>>> 2. If there is no DER, try to encode cert_subject
>>> 3. If that fails try again a different way
>>> 4. If it fails yet again use CN=localhost
>>>
>>> It is baffling that it would pick a hostname much less one that
>>> certmonger shouldn't even know about.
>>
>> Indeed, and if I try to renew other certificates for some of them it
>> chooses other host names that should not be known about. Each
>> certificate seems to get the same bad hostname each time I try to renew.
>>
>>>
>>> I assume the CSR found in the dogtag logs matches the csr value in the
>>> certmonger request?
>>
>> No, its different. The issued certificate matches the csr seen in dogtag
>> which makes sense. But the csr in the dogtag logs has the bad subject.
>> The csr in the request directory file has the good subject.
>>
>>>
>>> Can you share the certmonger request file privately?
>>
>> Yes, certainly.
>>
>> Thanks for your continued help on this.
>
> Let's try this. We'll drop the current request and create a new one.
>
> Stop tomcat, restore the old cert database, start tomcat, then:
>
> # getcert stop-tracking -i <request id>
> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin>
-B
> /usr/libexec/ipa/certmonger/stop_pkicad -C
> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"'
>
> Then try resubmitting the request.
>
Hi Rob
When you say 'stop tomcat', I'm just doing an ipactl stop. That is
stopiing the ipa pki-tomcat service. Is that good enough or are there
other services that need stopping too?
I tried the stop-tracking/start-tracking. Exactly the same result as
before. Same hostname appears in the subject CN field of the new
certificate and then the pki-tomcat service won't start.
I've also been back and redone the manual submits changing the point at
which I copied in the edited request as I wondered if there was some
caching of csr's going on in certmonger and it was remembering a
previous request it had read on startup before I replaced the request file.
But nothing seems to stop dogtag issuing a certificate with the Subject
CN being that host name.
Is it possible that dogtag is somehow overriding the Subject its being
explicitly given?
FWIW the CSR in the dogtag debug log is always identical, but I guess
thats expected if the CSR information is always the same.
I'm not sure if its possible to increase the verbosity on the dogtag
side. Can you advise please?
I've been assuming that its the bad subject that is preventing the
pki-tomcat from starting after the new certificate is issued. Does that
make sense to you?
I wonder where we can go from here? There must be a good reason for
whats happening!
Let's step back and gather some more information.
Can you restore the files again and send me the output of:
# getcert list
# certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
cert-pki-ca"
And the exact commands you are using to do all this, including
re-issuing the certs?
Using ipactl is fine. You just don't want to touch the NSS databases
while the service is running. Similarly you probably don't want to mess
with certmonger requests while it is running as it won't see the changes
until it restarts.
rob
10:56 a.m.
On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> Our freeipa certificates need to be renewed
due to passing their
>>>>>>>>>>> expiry
>>>>>>>>>>> dates.
>>>>>>>>>>>
>>>>>>>>>>> While some certificates have renewed ok, the
ipaCert and
>>>>>>>>>>> auditSigningCert are renewing but the new
certificates have the
>>>>>>>>>>> wrong
>>>>>>>>>>> Subject.
>>>>>>>>>>>
>>>>>>>>>>> Environment is:
>>>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa
4.4
>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>>>>
>>>>>>>>>>> Once there are renewed certificates with the
wrong Subject
>>>>>>>>>>> present,
>>>>>>>>>>> there are various problems with renewing the
remaining
>>>>>>>>>>> certificates,
>>>>>>>>>>> which I think might be related to the bad
Subject:
>>>>>>>>>>>
>>>>>>>>>>> 1) When just ipaCert has the wrong subject no
further renewals
>>>>>>>>>>> happen
>>>>>>>>>>>
>>>>>>>>>>> 2) When auditSigningCert has the wrong
subject the ipa
>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>> service will not start and no further
renewals happen.
>>>>>>>>>>>
>>>>>>>>>>> I've been round the following loop many
times on ServerA, our
>>>>>>>>>>> first
>>>>>>>>>>> master:
>>>>>>>>>>>
>>>>>>>>>>> 1) Restore good certificates from backup
>>>>>>>>>>> 2) Put the clock back to a time when
certificates are all valid
>>>>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>>>>
>>>>>>>>>>> Each time the ipaCert renews it has the same
wrong Subject. The
>>>>>>>>>>> wrong
>>>>>>>>>>> Subject includes the host name of one of our
ipa client systems.
>>>>>>>>>>>
>>>>>>>>>>> Each time the auditSigningCert renews it has
the same wrong
>>>>>>>>>>> Subject
>>>>>>>>>>> but
>>>>>>>>>>> a different subject to the ipaCert. The wrong
Subject in this
>>>>>>>>>>> case
>>>>>>>>>>> includes the host name of a system which has
never been an ipa
>>>>>>>>>>> client,
>>>>>>>>>>> but might have been added and removed with
ipa host-add and ipa
>>>>>>>>>>> host-del
>>>>>>>>>>> for testing something, a while ago.
>>>>>>>>>>>
>>>>>>>>>>> As far as I can see, the
"cert_subject" is set correctly in the
>>>>>>>>>>> file
>>>>>>>>>>> /var/lib/certmonger/<request id> until
the point at which the
>>>>>>>>>>> certificate is actually renewed.
>>>>>>>>>>>
>>>>>>>>>>> I'd be very grateful for some pointers as
to which configuration
>>>>>>>>>>> options
>>>>>>>>>>> and logs to check through to resolve this
problem on our
>>>>>>>>>>> production
>>>>>>>>>>> system.
>>>>>>>>>>>
>>>>>>>>>>> If its of any relevance we did change which
server is the first
>>>>>>>>>>> master
>>>>>>>>>>> some time ago.
>>>>>>>>>>
>>>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg)
and/or certmonger to see
>>>>>>>>>> what
>>>>>>>>>> the subject is.
>>>>>>>>>
>>>>>>>>> I'm not seeing any obvious CSR fields in the
>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>>>
>>>>>>>> foo.bar.certreq=
>>>>>>>>
>>>>>>>>> The CSR in the certmonger requests file for the
auditSigningCert
>>>>>>>>> seems
>>>>>>>>> to be showing with the correct Subject. This is
different from
>>>>>>>>> the bad
>>>>>>>>> subject showing in the requests file field:
>>>>>>>>> cert_subject=
>>>>>>>>
>>>>>>>> The value of cert_subject comes from the issued
certificate.
>>>>>>>>
>>>>>>>>> and the Subject which is showing in the 'getcert
list' output
>>>>>>>>> (which is
>>>>>>>>> the same as that in the cert_subject= field.>
>>>>>>>>> I'm not quite sure what this all means.
>>>>>>>>
>>>>>>>> It is displayed from the data within the tracked
certmonger
>>>>>>>> request.
>>>>>>>>
>>>>>>>> certmonger logs to syslog so you can check there or you
can stop
>>>>>>>> the
>>>>>>>> process and run it manually with: certmonger -n -d 9
2>&1 | tee
>>>>>>>> certmonger.log
>>>>>>>>
>>>>>>>> That will provide a lot of debugging output that may show
what is
>>>>>>>> going on.
>>>>>>>
>>>>>>> I've restored certificate databases from backup and put
the clock
>>>>>>> back
>>>>>>> to a time when certificates are valid and renewed the
ocspSigining
>>>>>>> certificate with:
>>>>>>> getcert resubmit -N "CN=OCSP
Subsystem,O=<REALM>" -i 20161124081302
>>>>>>>
>>>>>>> (I've previously tried without the -N with similar
results)
>>>>>>>
>>>>>>> What I am seeing in the certmonger logs is:
>>>>>>>
>>>>>>>
>>>>>>> 2017-10-23 00:05:28 [438] Located the key
'ocspSigningCert
>>>>>>> cert-pki-ca'.
>>>>>>> 2017-10-23 00:05:28 [438] Converted private key
'ocspSigningCert
>>>>>>> cert-pki-ca' to public key.
>>>>>>> 2017-10-23 00:05:28 [439] Located the certificate
"ocspSigningCert
>>>>>>> cert-pki-ca".
>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in
database
>>>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after
20171025122401.
>>>>>>> 2017-10-23 00:05:28 [442] Located the key
'ocspSigningCert
>>>>>>> cert-pki-ca'.
>>>>>>> 2017-10-23 00:05:28 [442] Converted private key
'ocspSigningCert
>>>>>>> cert-pki-ca' to public key.
>>>>>>> 2017-10-23 00:05:28 [443] Located the certificate
"ocspSigningCert
>>>>>>> cert-pki-ca".
>>>>>>> 2017-10-23 00:05:28 [444] Located the key
'ocspSigningCert
>>>>>>> cert-pki-ca'.
>>>>>>> 2017-10-23 00:05:28 [444] Converted private key
'ocspSigningCert
>>>>>>> cert-pki-ca' to public key.
>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same
>>>>>>> nickname but
>>>>>>> different subject, removing certificate
"ocspSigningCert
>>>>>>> cert-pki-ca"
>>>>>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate
"ocspSigningCert
>>>>>>> cert-pki-ca", got nickname "ocspSigningCert
cert-pki-ca".
>>>>>>> 2017-10-23 00:05:39 [583] Located the certificate
"ocspSigningCert
>>>>>>> cert-pki-ca".
>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>>> cert-pki-ca"" (0).
>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
>>>>>>> cert-pki-ca" in token "NSS Certificate DB" in
database
>>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and
saved.
>>>>>>>
>>>>>>> I now have a date valid ocspSigningCertificate, but with the
wrong
>>>>>>> subject, and a broken certificate system which will no longer
start.
>>>>>>>
>>>>>>> ipactl status
>>>>>>> ...
>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>
>>>>>>> I can't renew other expired certificates.
>>>>>>>
>>>>>>> I also note that there is now no key for
ocspSigningCertificate as
>>>>>>> shown
>>>>>>> by:
>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>
>>>>>>> I wonder if this is because the certificate subject changed?
There
>>>>>>> was a
>>>>>>> key before the certificate renewed.
>>>>>>>
>>>>>>> The ca debug logs are showing:
>>>>>>>
>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by
>>>>>>> nickname:
>>>>>>> 'ocspSigningCert cert-pki-ca' with serial number:
268370108
>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>>>>> x509CertImpl
>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>>>>> Certificate
>>>>>>> object not found
>>>>>>> Certificate object not found
>>>>>>> at
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>>
>>>>>>> Any help in repairing my broken ipa system will be much
appreciated.
>>>>>>
>>>>>> Sorry for the delay. I think my previous reading of the
certmonger
>>>>>> csrgen code was wrong.
>>>>>>
>>>>>> IIRC in your certmonger entry the cert_subject has the hostname
value
>>>>>> right? Do you also have cert_subject_der?
>>>>>>
>>>>>> You can decode that by:
>>>>>>
>>>>>> 1. Running a hex-to-bin, something hacky like this in python:
>>>>>>
>>>>>> import binascii
>>>>>>
>>>>>> hex_string = "<hex value>"
>>>>>>
>>>>>> binary_string = binascii.unhexlify(hex_string)
>>>>>>
>>>>>> fd = open("out", "wb")
>>>>>> fd.write(binary_string)
>>>>>> fd.close()
>>>>>>
>>>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>>>
>>>>>> I'm going to assume this also has the hostname encoded in
it.
>>>>>
>>>>> Hi Again
>>>>>
>>>>> Yes, correct. The cert_subject_der does have the bad CN with the
>>>>> hostname encoded in it.
>>>>>
>>>>>>
>>>>>> Can you try this:
>>>>>>
>>>>>> 1. Make a backup of the request file (just in case) > 2.
Remove
>>>>>> cert_subject_der
>>>>>> 3. Modify cert_subject to be CN=OCSP
Subsystem,O=<YOUR_REALM>
>>>>>> 3. Try the renewal again
>>>>>
>>>>> I ran though all of this, checking that the request file was still
>>>>> what
>>>>> we wanted after restarting certmonger with verbose logging,
restoring
>>>>> all the databases, fixing permissions, checking keys etc., and ran
the
>>>>> getcert resubmit.
>>>>>
>>>>> It still generates a certificate with the bad CN including a
hostname.
>>>>>
>>>>> Then the pki-tomcat server fails to start again.
>>>>>
>>>>> Other things I have checked are:
>>>>> 1) The csr= field in the (modified) request file has the correct
>>>>> subject.
>>>>>
>>>>> 2) The dogtag ca debug log file is showing:
>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>>>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>>>>> ...
>>>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert
>>>>> RenewalSubmitter: for renewal, original authenticator raCertAuth
found
>>>>> CertProcessor: No authenticator credentials required
>>>>> processRenewal: set original Inputs into profile Context
>>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>>> cert_request_type
>>>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
>>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>>> cert_request
>>>>> RenewalSubmitter: setInputsIntoContext() setting value in
ctx:<hex
>>>>> encoded csr>
>>>>> ...
>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>>>
>>>>> The hex encoded csr in the last line decodes to have the bad subject
>>>>> where the hostname is one of our other ipa servers.
>>>>>
>>>>> The certificate always getthe same hostname in the bad subject cn.
>>>>>
>>>>> Do you have any other ideas of things to check to find out
what's
>>>>> generating the bad subject?
>>>>
>>>> The order certmonger uses for the subject is:
>>>>
>>>> 1. cert_subject_der
>>>> 2. If there is no DER, try to encode cert_subject
>>>> 3. If that fails try again a different way
>>>> 4. If it fails yet again use CN=localhost
>>>>
>>>> It is baffling that it would pick a hostname much less one that
>>>> certmonger shouldn't even know about.
>>>
>>> Indeed, and if I try to renew other certificates for some of them it
>>> chooses other host names that should not be known about. Each
>>> certificate seems to get the same bad hostname each time I try to renew.
>>>
>>>>
>>>> I assume the CSR found in the dogtag logs matches the csr value in the
>>>> certmonger request?
>>>
>>> No, its different. The issued certificate matches the csr seen in dogtag
>>> which makes sense. But the csr in the dogtag logs has the bad subject.
>>> The csr in the request directory file has the good subject.
>>>
>>>>
>>>> Can you share the certmonger request file privately?
>>>
>>> Yes, certainly.
>>>
>>> Thanks for your continued help on this.
>>
>> Let's try this. We'll drop the current request and create a new one.
>>
>> Stop tomcat, restore the old cert database, start tomcat, then:
>>
>> # getcert stop-tracking -i <request id>
>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P
<pin> -B
>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"'
>>
>> Then try resubmitting the request.
>>
> Hi Rob
>
> When you say 'stop tomcat', I'm just doing an ipactl stop. That is
> stopiing the ipa pki-tomcat service. Is that good enough or are there
> other services that need stopping too?
>
> I tried the stop-tracking/start-tracking. Exactly the same result as
> before. Same hostname appears in the subject CN field of the new
> certificate and then the pki-tomcat service won't start.
>
> I've also been back and redone the manual submits changing the point at
> which I copied in the edited request as I wondered if there was some
> caching of csr's going on in certmonger and it was remembering a
> previous request it had read on startup before I replaced the request file.
>
> But nothing seems to stop dogtag issuing a certificate with the Subject
> CN being that host name.
>
> Is it possible that dogtag is somehow overriding the Subject its being
> explicitly given?
>
> FWIW the CSR in the dogtag debug log is always identical, but I guess
> thats expected if the CSR information is always the same.
>
> I'm not sure if its possible to increase the verbosity on the dogtag
> side. Can you advise please?
>
> I've been assuming that its the bad subject that is preventing the
> pki-tomcat from starting after the new certificate is issued. Does that
> make sense to you?
>
> I wonder where we can go from here? There must be a good reason for
> whats happening!
Let's step back and gather some more information.
Can you restore the files again and send me the output of:
# getcert list
# certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
cert-pki-ca"
Hi Rob
You should have the output you requested by private email now.
And the exact commands you are using to do all this, including
re-issuing the cert
Thats a great idea. Maybe I'm just doing something in the wrong order.
Using ipactl is fine. You just don't want to touch the NSS databases
while the service is running. Similarly you probably don't want to mess
with certmonger requests while it is running as it won't see the changes
until it restarts.
Here is the procedure I'm using to renew the certificates.
One of the
tricky things is to not have the certificate system working when
certmonger is started so that it doesn't renew a certificate on startup
that might not be one I have changed the request file for (other
certificates have similar problems with bad subject CN and stop the
certificate system.)
I've found it convenient to keep reinitializing the log files because
they get very confusing when the clock keeps getting set back.
The request number has changed since we did the stop-tracking /
start-tracking.
Roderick
ipactl stop
# Set clock back:
systemctl stop ntpd
timedatectl set-time "2017-10-23 00:00:00"
date
#Stop certificate renewals the systemd way
systemctl stop certmonger
# Sort out certmonger log
mv /var/log/certmonger.log /var/log/certmonger.log-$(date +%Y%m%d:%H:%M)
# ****** Copy in the fixed certmonger request
\cp /root/20161124081302.ocsp_backup
/var/lib/certmonger/requests/20161124081302
# In another window
certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
# In the original window
# Sort out dogtag log
mv /var/log/pki/pki-tomcat/ca/debug
/var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
touch /var/log/pki/pki-tomcat/ca/debug
chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
mv /var/log/pki/pki-tomcat/ca/selftests.log
/var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
touch /var/log/pki/pki-tomcat/ca/selftests.log
chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
\cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
\cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
/etc/pki/pki-tomcat/alias/cert8.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
/etc/pki/pki-tomcat/alias/key3.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg /etc/pki/pki-tomcat/ca/CS.cfg
# Fix up certificate permissions and check all is ok
certutil -M -n "caSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias
-t CTu,Cu,Cu
certutil -L -d /etc/pki/pki-tomcat/alias
certutil -K -d /etc/pki/pki-tomcat/alias
<pin>
# Fix dogtag so it starts
pki-server subsystem-enable ca
# Start IdM services
ipactl start
ipactl status
#Resubmit certificate ocsp
date;getcert resubmit -i 20161124081302
2:20 a.m.
On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote:
On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> Our freeipa certificates need to be
renewed due to passing
>>>>>>>>>>>> their
>>>>>>>>>>>> expiry
>>>>>>>>>>>> dates.
>>>>>>>>>>>>
>>>>>>>>>>>> While some certificates have renewed ok,
the ipaCert and
>>>>>>>>>>>> auditSigningCert are renewing but the new
certificates have
>>>>>>>>>>>> the
>>>>>>>>>>>> wrong
>>>>>>>>>>>> Subject.
>>>>>>>>>>>>
>>>>>>>>>>>> Environment is:
>>>>>>>>>>>> serverA (CRL, first, master) RHEL 7.3,
ipa 4.4
>>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>>>>>
>>>>>>>>>>>> Once there are renewed certificates with
the wrong Subject
>>>>>>>>>>>> present,
>>>>>>>>>>>> there are various problems with renewing
the remaining
>>>>>>>>>>>> certificates,
>>>>>>>>>>>> which I think might be related to the bad
Subject:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) When just ipaCert has the wrong
subject no further renewals
>>>>>>>>>>>> happen
>>>>>>>>>>>>
>>>>>>>>>>>> 2) When auditSigningCert has the wrong
subject the ipa
>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>> service will not start and no further
renewals happen.
>>>>>>>>>>>>
>>>>>>>>>>>> I've been round the following loop
many times on ServerA, our
>>>>>>>>>>>> first
>>>>>>>>>>>> master:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) Restore good certificates from backup
>>>>>>>>>>>> 2) Put the clock back to a time when
certificates are all
>>>>>>>>>>>> valid
>>>>>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>>>>>
>>>>>>>>>>>> Each time the ipaCert renews it has the
same wrong Subject.
>>>>>>>>>>>> The
>>>>>>>>>>>> wrong
>>>>>>>>>>>> Subject includes the host name of one of
our ipa client
>>>>>>>>>>>> systems.
>>>>>>>>>>>>
>>>>>>>>>>>> Each time the auditSigningCert renews it
has the same wrong
>>>>>>>>>>>> Subject
>>>>>>>>>>>> but
>>>>>>>>>>>> a different subject to the ipaCert. The
wrong Subject in this
>>>>>>>>>>>> case
>>>>>>>>>>>> includes the host name of a system which
has never been an ipa
>>>>>>>>>>>> client,
>>>>>>>>>>>> but might have been added and removed
with ipa host-add and
>>>>>>>>>>>> ipa
>>>>>>>>>>>> host-del
>>>>>>>>>>>> for testing something, a while ago.
>>>>>>>>>>>>
>>>>>>>>>>>> As far as I can see, the
"cert_subject" is set correctly in
>>>>>>>>>>>> the
>>>>>>>>>>>> file
>>>>>>>>>>>> /var/lib/certmonger/<request id>
until the point at which the
>>>>>>>>>>>> certificate is actually renewed.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd be very grateful for some
pointers as to which
>>>>>>>>>>>> configuration
>>>>>>>>>>>> options
>>>>>>>>>>>> and logs to check through to resolve this
problem on our
>>>>>>>>>>>> production
>>>>>>>>>>>> system.
>>>>>>>>>>>>
>>>>>>>>>>>> If its of any relevance we did change
which server is the
>>>>>>>>>>>> first
>>>>>>>>>>>> master
>>>>>>>>>>>> some time ago.
>>>>>>>>>>>
>>>>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg)
and/or certmonger to
>>>>>>>>>>> see
>>>>>>>>>>> what
>>>>>>>>>>> the subject is.
>>>>>>>>>>
>>>>>>>>>> I'm not seeing any obvious CSR fields in the
>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>>>>
>>>>>>>>> foo.bar.certreq=
>>>>>>>>>
>>>>>>>>>> The CSR in the certmonger requests file for the
auditSigningCert
>>>>>>>>>> seems
>>>>>>>>>> to be showing with the correct Subject. This is
different from
>>>>>>>>>> the bad
>>>>>>>>>> subject showing in the requests file field:
>>>>>>>>>> cert_subject=
>>>>>>>>>
>>>>>>>>> The value of cert_subject comes from the issued
certificate.
>>>>>>>>>
>>>>>>>>>> and the Subject which is showing in the
'getcert list' output
>>>>>>>>>> (which is
>>>>>>>>>> the same as that in the cert_subject= field.>
>>>>>>>>>> I'm not quite sure what this all means.
>>>>>>>>>
>>>>>>>>> It is displayed from the data within the tracked
certmonger
>>>>>>>>> request.
>>>>>>>>>
>>>>>>>>> certmonger logs to syslog so you can check there or
you can stop
>>>>>>>>> the
>>>>>>>>> process and run it manually with: certmonger -n -d 9
2>&1 | tee
>>>>>>>>> certmonger.log
>>>>>>>>>
>>>>>>>>> That will provide a lot of debugging output that may
show what is
>>>>>>>>> going on.
>>>>>>>>
>>>>>>>> I've restored certificate databases from backup and
put the clock
>>>>>>>> back
>>>>>>>> to a time when certificates are valid and renewed the
ocspSigining
>>>>>>>> certificate with:
>>>>>>>> getcert resubmit -N "CN=OCSP
Subsystem,O=<REALM>" -i
>>>>>>>> 20161124081302
>>>>>>>>
>>>>>>>> (I've previously tried without the -N with similar
results)
>>>>>>>>
>>>>>>>> What I am seeing in the certmonger logs is:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2017-10-23 00:05:28 [438] Located the key
'ocspSigningCert
>>>>>>>> cert-pki-ca'.
>>>>>>>> 2017-10-23 00:05:28 [438] Converted private key
'ocspSigningCert
>>>>>>>> cert-pki-ca' to public key.
>>>>>>>> 2017-10-23 00:05:28 [439] Located the certificate
"ocspSigningCert
>>>>>>>> cert-pki-ca".
>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
>>>>>>>> cert-pki-ca" in token "NSS Certificate DB"
in database
>>>>>>>> "/etc/pki/pki-tomcat/alias" will not be valid
after
>>>>>>>> 20171025122401.
>>>>>>>> 2017-10-23 00:05:28 [442] Located the key
'ocspSigningCert
>>>>>>>> cert-pki-ca'.
>>>>>>>> 2017-10-23 00:05:28 [442] Converted private key
'ocspSigningCert
>>>>>>>> cert-pki-ca' to public key.
>>>>>>>> 2017-10-23 00:05:28 [443] Located the certificate
"ocspSigningCert
>>>>>>>> cert-pki-ca".
>>>>>>>> 2017-10-23 00:05:28 [444] Located the key
'ocspSigningCert
>>>>>>>> cert-pki-ca'.
>>>>>>>> 2017-10-23 00:05:28 [444] Converted private key
'ocspSigningCert
>>>>>>>> cert-pki-ca' to public key.
>>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the
same
>>>>>>>> nickname but
>>>>>>>> different subject, removing certificate
"ocspSigningCert
>>>>>>>> cert-pki-ca"
>>>>>>>> with subject "CN=OCSP
Subsystem,O=<REALM>".
>>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate
"ocspSigningCert
>>>>>>>> cert-pki-ca", got nickname "ocspSigningCert
cert-pki-ca".
>>>>>>>> 2017-10-23 00:05:39 [583] Located the certificate
"ocspSigningCert
>>>>>>>> cert-pki-ca".
>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>>>> cert-pki-ca"" (0).
>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
>>>>>>>> cert-pki-ca" in token "NSS Certificate DB"
in database
>>>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and
saved.
>>>>>>>>
>>>>>>>> I now have a date valid ocspSigningCertificate, but with
the wrong
>>>>>>>> subject, and a broken certificate system which will no
longer
>>>>>>>> start.
>>>>>>>>
>>>>>>>> ipactl status
>>>>>>>> ...
>>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>>
>>>>>>>> I can't renew other expired certificates.
>>>>>>>>
>>>>>>>> I also note that there is now no key for
ocspSigningCertificate as
>>>>>>>> shown
>>>>>>>> by:
>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>
>>>>>>>> I wonder if this is because the certificate subject
changed? There
>>>>>>>> was a
>>>>>>>> key before the certificate renewed.
>>>>>>>>
>>>>>>>> The ca debug logs are showing:
>>>>>>>>
>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert
by
>>>>>>>> nickname:
>>>>>>>> 'ocspSigningCert cert-pki-ca' with serial number:
268370108
>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted
to
>>>>>>>> x509CertImpl
>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]:
SigningUnit:
>>>>>>>> Certificate
>>>>>>>> object not found
>>>>>>>> Certificate object not found
>>>>>>>> at
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>>>
>>>>>>>> Any help in repairing my broken ipa system will be much
>>>>>>>> appreciated.
>>>>>>>
>>>>>>> Sorry for the delay. I think my previous reading of the
certmonger
>>>>>>> csrgen code was wrong.
>>>>>>>
>>>>>>> IIRC in your certmonger entry the cert_subject has the
hostname
>>>>>>> value
>>>>>>> right? Do you also have cert_subject_der?
>>>>>>>
>>>>>>> You can decode that by:
>>>>>>>
>>>>>>> 1. Running a hex-to-bin, something hacky like this in
python:
>>>>>>>
>>>>>>> import binascii
>>>>>>>
>>>>>>> hex_string = "<hex value>"
>>>>>>>
>>>>>>> binary_string = binascii.unhexlify(hex_string)
>>>>>>>
>>>>>>> fd = open("out", "wb")
>>>>>>> fd.write(binary_string)
>>>>>>> fd.close()
>>>>>>>
>>>>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>>>>
>>>>>>> I'm going to assume this also has the hostname encoded in
it.
>>>>>>
>>>>>> Hi Again
>>>>>>
>>>>>> Yes, correct. The cert_subject_der does have the bad CN with the
>>>>>> hostname encoded in it.
>>>>>>
>>>>>>>
>>>>>>> Can you try this:
>>>>>>>
>>>>>>> 1. Make a backup of the request file (just in case) > 2.
Remove
>>>>>>> cert_subject_der
>>>>>>> 3. Modify cert_subject to be CN=OCSP
Subsystem,O=<YOUR_REALM>
>>>>>>> 3. Try the renewal again
>>>>>>
>>>>>> I ran though all of this, checking that the request file was
still
>>>>>> what
>>>>>> we wanted after restarting certmonger with verbose logging,
>>>>>> restoring
>>>>>> all the databases, fixing permissions, checking keys etc., and
>>>>>> ran the
>>>>>> getcert resubmit.
>>>>>>
>>>>>> It still generates a certificate with the bad CN including a
>>>>>> hostname.
>>>>>>
>>>>>> Then the pki-tomcat server fails to start again.
>>>>>>
>>>>>> Other things I have checked are:
>>>>>> 1) The csr= field in the (modified) request file has the correct
>>>>>> subject.
>>>>>>
>>>>>> 2) The dogtag ca debug log file is showing:
>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>>>>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>>>>>> ...
>>>>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert
>>>>>> RenewalSubmitter: for renewal, original authenticator raCertAuth
>>>>>> found
>>>>>> CertProcessor: No authenticator credentials required
>>>>>> processRenewal: set original Inputs into profile Context
>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>>>> cert_request_type
>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in
ctx:pkcs10
>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>>>> cert_request
>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in
ctx:<hex
>>>>>> encoded csr>
>>>>>> ...
>>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>>>>
>>>>>> The hex encoded csr in the last line decodes to have the bad
subject
>>>>>> where the hostname is one of our other ipa servers.
>>>>>>
>>>>>> The certificate always getthe same hostname in the bad subject
cn.
>>>>>>
>>>>>> Do you have any other ideas of things to check to find out
what's
>>>>>> generating the bad subject?
>>>>>
>>>>> The order certmonger uses for the subject is:
>>>>>
>>>>> 1. cert_subject_der
>>>>> 2. If there is no DER, try to encode cert_subject
>>>>> 3. If that fails try again a different way
>>>>> 4. If it fails yet again use CN=localhost
>>>>>
>>>>> It is baffling that it would pick a hostname much less one that
>>>>> certmonger shouldn't even know about.
>>>>
>>>> Indeed, and if I try to renew other certificates for some of them it
>>>> chooses other host names that should not be known about. Each
>>>> certificate seems to get the same bad hostname each time I try to
>>>> renew.
>>>>
>>>>>
>>>>> I assume the CSR found in the dogtag logs matches the csr value in
>>>>> the
>>>>> certmonger request?
>>>>
>>>> No, its different. The issued certificate matches the csr seen in
>>>> dogtag
>>>> which makes sense. But the csr in the dogtag logs has the bad subject.
>>>> The csr in the request directory file has the good subject.
>>>>
>>>>>
>>>>> Can you share the certmonger request file privately?
>>>>
>>>> Yes, certainly.
>>>>
>>>> Thanks for your continued help on this.
>>>
>>> Let's try this. We'll drop the current request and create a new one.
>>>
>>> Stop tomcat, restore the old cert database, start tomcat, then:
>>>
>>> # getcert stop-tracking -i <request id>
>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
>>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P
<pin> -B
>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>> cert-pki-ca"'
>>>
>>> Then try resubmitting the request.
>>>
>> Hi Rob
>>
>> When you say 'stop tomcat', I'm just doing an ipactl stop. That is
>> stopiing the ipa pki-tomcat service. Is that good enough or are there
>> other services that need stopping too?
>>
>> I tried the stop-tracking/start-tracking. Exactly the same result as
>> before. Same hostname appears in the subject CN field of the new
>> certificate and then the pki-tomcat service won't start.
>>
>> I've also been back and redone the manual submits changing the point at
>> which I copied in the edited request as I wondered if there was some
>> caching of csr's going on in certmonger and it was remembering a
>> previous request it had read on startup before I replaced the request
>> file.
>>
>> But nothing seems to stop dogtag issuing a certificate with the Subject
>> CN being that host name.
>>
>> Is it possible that dogtag is somehow overriding the Subject its being
>> explicitly given?
>>
>> FWIW the CSR in the dogtag debug log is always identical, but I guess
>> thats expected if the CSR information is always the same.
>>
>> I'm not sure if its possible to increase the verbosity on the dogtag
>> side. Can you advise please?
>>
>> I've been assuming that its the bad subject that is preventing the
>> pki-tomcat from starting after the new certificate is issued. Does that
>> make sense to you?
>>
>> I wonder where we can go from here? There must be a good reason for
>> whats happening!
>
> Let's step back and gather some more information.
>
> Can you restore the files again and send me the output of:
>
> # getcert list
>
> # certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
> cert-pki-ca"
Hi Rob
You should have the output you requested by private email now.
>
> And the exact commands you are using to do all this, including
> re-issuing the cert
Thats a great idea. Maybe I'm just doing something in the wrong order.
>
> Using ipactl is fine. You just don't want to touch the NSS databases
> while the service is running. Similarly you probably don't want to mess
> with certmonger requests while it is running as it won't see the changes
> until it restarts.
Here is the procedure I'm using to renew the certificates. One of the
tricky things is to not have the certificate system working when
certmonger is started so that it doesn't renew a certificate on startup
that might not be one I have changed the request file for (other
certificates have similar problems with bad subject CN and stop the
certificate system.)
I've found it convenient to keep reinitializing the log files because
they get very confusing when the clock keeps getting set back.
The request number has changed since we did the stop-tracking /
start-tracking.
Roderick
ipactl stop
# Set clock back:
systemctl stop ntpd
timedatectl set-time "2017-10-23 00:00:00"
date
#Stop certificate renewals the systemd way
systemctl stop certmonger
# Sort out certmonger log
mv /var/log/certmonger.log /var/log/certmonger.log-$(date +%Y%m%d:%H:%M)
# ****** Copy in the fixed certmonger request
\cp /root/20161124081302.ocsp_backup
/var/lib/certmonger/requests/20161124081302
# In another window
certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
# In the original window
# Sort out dogtag log
mv /var/log/pki/pki-tomcat/ca/debug
/var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
touch /var/log/pki/pki-tomcat/ca/debug
chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
mv /var/log/pki/pki-tomcat/ca/selftests.log
/var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
touch /var/log/pki/pki-tomcat/ca/selftests.log
chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
\cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
\cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
/etc/pki/pki-tomcat/alias/cert8.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
/etc/pki/pki-tomcat/alias/key3.db
\cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg /etc/pki/pki-tomcat/ca/CS.cfg
# Fix up certificate permissions and check all is ok
certutil -M -n "caSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias
-t CTu,Cu,Cu
certutil -L -d /etc/pki/pki-tomcat/alias
certutil -K -d /etc/pki/pki-tomcat/alias
<pin>
# Fix dogtag so it starts
pki-server subsystem-enable ca >
# Start IdM services
ipactl start
ipactl status
#Resubmit certificate ocsp
date;getcert resubmit -i 20161124081302
Hi Rob
Does it look like there is anything I should be doing differently in
this command sequence to try to get the certificate renewed correctly?
I'm starting to wonder whether I should look at alternative strategies
for recovering the freeipa servers if the certificates won't renew
correctly. Do you have any ideas on this please?
Thanks
Roderick
2:36 p.m.
Roderick Johnstone via FreeIPA-users wrote:
On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote:
> On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users
wrote:
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> Our freeipa certificates need to be
renewed due to passing
>>>>>>>>>>>>> their
>>>>>>>>>>>>> expiry
>>>>>>>>>>>>> dates.
>>>>>>>>>>>>>
>>>>>>>>>>>>> While some certificates have renewed
ok, the ipaCert and
>>>>>>>>>>>>> auditSigningCert are renewing but the
new certificates
>>>>>>>>>>>>> have the
>>>>>>>>>>>>> wrong
>>>>>>>>>>>>> Subject.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Environment is:
>>>>>>>>>>>>> serverA (CRL, first, master) RHEL
7.3, ipa 4.4
>>>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>>>>>>
>>>>>>>>>>>>> Once there are renewed certificates
with the wrong Subject
>>>>>>>>>>>>> present,
>>>>>>>>>>>>> there are various problems with
renewing the remaining
>>>>>>>>>>>>> certificates,
>>>>>>>>>>>>> which I think might be related to the
bad Subject:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) When just ipaCert has the wrong
subject no further
>>>>>>>>>>>>> renewals
>>>>>>>>>>>>> happen
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) When auditSigningCert has the
wrong subject the ipa
>>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>>> service will not start and no further
renewals happen.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been round the following
loop many times on ServerA, our
>>>>>>>>>>>>> first
>>>>>>>>>>>>> master:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Restore good certificates from
backup
>>>>>>>>>>>>> 2) Put the clock back to a time when
certificates are all
>>>>>>>>>>>>> valid
>>>>>>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>>>>>>
>>>>>>>>>>>>> Each time the ipaCert renews it has
the same wrong
>>>>>>>>>>>>> Subject. The
>>>>>>>>>>>>> wrong
>>>>>>>>>>>>> Subject includes the host name of one
of our ipa client
>>>>>>>>>>>>> systems.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Each time the auditSigningCert renews
it has the same wrong
>>>>>>>>>>>>> Subject
>>>>>>>>>>>>> but
>>>>>>>>>>>>> a different subject to the ipaCert.
The wrong Subject in this
>>>>>>>>>>>>> case
>>>>>>>>>>>>> includes the host name of a system
which has never been an
>>>>>>>>>>>>> ipa
>>>>>>>>>>>>> client,
>>>>>>>>>>>>> but might have been added and removed
with ipa host-add
>>>>>>>>>>>>> and ipa
>>>>>>>>>>>>> host-del
>>>>>>>>>>>>> for testing something, a while ago.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As far as I can see, the
"cert_subject" is set correctly
>>>>>>>>>>>>> in the
>>>>>>>>>>>>> file
>>>>>>>>>>>>> /var/lib/certmonger/<request
id> until the point at which the
>>>>>>>>>>>>> certificate is actually renewed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd be very grateful for some
pointers as to which
>>>>>>>>>>>>> configuration
>>>>>>>>>>>>> options
>>>>>>>>>>>>> and logs to check through to resolve
this problem on our
>>>>>>>>>>>>> production
>>>>>>>>>>>>> system.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If its of any relevance we did change
which server is the
>>>>>>>>>>>>> first
>>>>>>>>>>>>> master
>>>>>>>>>>>>> some time ago.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd pull the CSR out of dogtag
(CS.cfg) and/or certmonger
>>>>>>>>>>>> to see
>>>>>>>>>>>> what
>>>>>>>>>>>> the subject is.
>>>>>>>>>>>
>>>>>>>>>>> I'm not seeing any obvious CSR fields in
the
>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>>>>>
>>>>>>>>>> foo.bar.certreq=
>>>>>>>>>>
>>>>>>>>>>> The CSR in the certmonger requests file for
the
>>>>>>>>>>> auditSigningCert
>>>>>>>>>>> seems
>>>>>>>>>>> to be showing with the correct Subject. This
is different from
>>>>>>>>>>> the bad
>>>>>>>>>>> subject showing in the requests file field:
>>>>>>>>>>> cert_subject=
>>>>>>>>>>
>>>>>>>>>> The value of cert_subject comes from the issued
certificate.
>>>>>>>>>>
>>>>>>>>>>> and the Subject which is showing in the
'getcert list' output
>>>>>>>>>>> (which is
>>>>>>>>>>> the same as that in the cert_subject=
field.>
>>>>>>>>>>> I'm not quite sure what this all means.
>>>>>>>>>>
>>>>>>>>>> It is displayed from the data within the tracked
certmonger
>>>>>>>>>> request.
>>>>>>>>>>
>>>>>>>>>> certmonger logs to syslog so you can check there
or you can stop
>>>>>>>>>> the
>>>>>>>>>> process and run it manually with: certmonger -n
-d 9 2>&1 | tee
>>>>>>>>>> certmonger.log
>>>>>>>>>>
>>>>>>>>>> That will provide a lot of debugging output that
may show
>>>>>>>>>> what is
>>>>>>>>>> going on.
>>>>>>>>>
>>>>>>>>> I've restored certificate databases from backup
and put the clock
>>>>>>>>> back
>>>>>>>>> to a time when certificates are valid and renewed
the
>>>>>>>>> ocspSigining
>>>>>>>>> certificate with:
>>>>>>>>> getcert resubmit -N "CN=OCSP
Subsystem,O=<REALM>" -i
>>>>>>>>> 20161124081302
>>>>>>>>>
>>>>>>>>> (I've previously tried without the -N with
similar results)
>>>>>>>>>
>>>>>>>>> What I am seeing in the certmonger logs is:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2017-10-23 00:05:28 [438] Located the key
'ocspSigningCert
>>>>>>>>> cert-pki-ca'.
>>>>>>>>> 2017-10-23 00:05:28 [438] Converted private key
'ocspSigningCert
>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>> 2017-10-23 00:05:28 [439] Located the certificate
>>>>>>>>> "ocspSigningCert
>>>>>>>>> cert-pki-ca".
>>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
>>>>>>>>> cert-pki-ca" in token "NSS Certificate
DB" in database
>>>>>>>>> "/etc/pki/pki-tomcat/alias" will not be
valid after
>>>>>>>>> 20171025122401.
>>>>>>>>> 2017-10-23 00:05:28 [442] Located the key
'ocspSigningCert
>>>>>>>>> cert-pki-ca'.
>>>>>>>>> 2017-10-23 00:05:28 [442] Converted private key
'ocspSigningCert
>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>> 2017-10-23 00:05:28 [443] Located the certificate
>>>>>>>>> "ocspSigningCert
>>>>>>>>> cert-pki-ca".
>>>>>>>>> 2017-10-23 00:05:28 [444] Located the key
'ocspSigningCert
>>>>>>>>> cert-pki-ca'.
>>>>>>>>> 2017-10-23 00:05:28 [444] Converted private key
'ocspSigningCert
>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate with
the same
>>>>>>>>> nickname but
>>>>>>>>> different subject, removing certificate
"ocspSigningCert
>>>>>>>>> cert-pki-ca"
>>>>>>>>> with subject "CN=OCSP
Subsystem,O=<REALM>".
>>>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate
"ocspSigningCert
>>>>>>>>> cert-pki-ca", got nickname "ocspSigningCert
cert-pki-ca".
>>>>>>>>> 2017-10-23 00:05:39 [583] Located the certificate
>>>>>>>>> "ocspSigningCert
>>>>>>>>> cert-pki-ca".
>>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>>>>> cert-pki-ca"" (0).
>>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
>>>>>>>>> cert-pki-ca" in token "NSS Certificate
DB" in database
>>>>>>>>> "/etc/pki/pki-tomcat/alias" issued by CA
and saved.
>>>>>>>>>
>>>>>>>>> I now have a date valid ocspSigningCertificate, but
with the
>>>>>>>>> wrong
>>>>>>>>> subject, and a broken certificate system which will
no longer
>>>>>>>>> start.
>>>>>>>>>
>>>>>>>>> ipactl status
>>>>>>>>> ...
>>>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>>>
>>>>>>>>> I can't renew other expired certificates.
>>>>>>>>>
>>>>>>>>> I also note that there is now no key for
>>>>>>>>> ocspSigningCertificate as
>>>>>>>>> shown
>>>>>>>>> by:
>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>
>>>>>>>>> I wonder if this is because the certificate subject
changed?
>>>>>>>>> There
>>>>>>>>> was a
>>>>>>>>> key before the certificate renewed.
>>>>>>>>>
>>>>>>>>> The ca debug logs are showing:
>>>>>>>>>
>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found
cert by
>>>>>>>>> nickname:
>>>>>>>>> 'ocspSigningCert cert-pki-ca' with serial
number: 268370108
>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]:
converted to
>>>>>>>>> x509CertImpl
>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]:
SigningUnit:
>>>>>>>>> Certificate
>>>>>>>>> object not found
>>>>>>>>> Certificate object not found
>>>>>>>>> at
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>>>>
>>>>>>>>> Any help in repairing my broken ipa system will be
much
>>>>>>>>> appreciated.
>>>>>>>>
>>>>>>>> Sorry for the delay. I think my previous reading of the
certmonger
>>>>>>>> csrgen code was wrong.
>>>>>>>>
>>>>>>>> IIRC in your certmonger entry the cert_subject has the
hostname
>>>>>>>> value
>>>>>>>> right? Do you also have cert_subject_der?
>>>>>>>>
>>>>>>>> You can decode that by:
>>>>>>>>
>>>>>>>> 1. Running a hex-to-bin, something hacky like this in
python:
>>>>>>>>
>>>>>>>> import binascii
>>>>>>>>
>>>>>>>> hex_string = "<hex value>"
>>>>>>>>
>>>>>>>> binary_string = binascii.unhexlify(hex_string)
>>>>>>>>
>>>>>>>> fd = open("out", "wb")
>>>>>>>> fd.write(binary_string)
>>>>>>>> fd.close()
>>>>>>>>
>>>>>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>>>>>
>>>>>>>> I'm going to assume this also has the hostname
encoded in it.
>>>>>>>
>>>>>>> Hi Again
>>>>>>>
>>>>>>> Yes, correct. The cert_subject_der does have the bad CN with
the
>>>>>>> hostname encoded in it.
>>>>>>>
>>>>>>>>
>>>>>>>> Can you try this:
>>>>>>>>
>>>>>>>> 1. Make a backup of the request file (just in case) >
2. Remove
>>>>>>>> cert_subject_der
>>>>>>>> 3. Modify cert_subject to be CN=OCSP
Subsystem,O=<YOUR_REALM>
>>>>>>>> 3. Try the renewal again
>>>>>>>
>>>>>>> I ran though all of this, checking that the request file was
still
>>>>>>> what
>>>>>>> we wanted after restarting certmonger with verbose logging,
>>>>>>> restoring
>>>>>>> all the databases, fixing permissions, checking keys etc.,
and
>>>>>>> ran the
>>>>>>> getcert resubmit.
>>>>>>>
>>>>>>> It still generates a certificate with the bad CN including a
>>>>>>> hostname.
>>>>>>>
>>>>>>> Then the pki-tomcat server fails to start again.
>>>>>>>
>>>>>>> Other things I have checked are:
>>>>>>> 1) The csr= field in the (modified) request file has the
correct
>>>>>>> subject.
>>>>>>>
>>>>>>> 2) The dogtag ca debug log file is showing:
>>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>>>>>>> processRenewal: orig subj dn =CN=OCSP
Subsystem,O=AST.CAM.AC.UK
>>>>>>> ...
>>>>>>> RenewalSubmitter: renewal original
profileId=caIPAserviceCert
>>>>>>> RenewalSubmitter: for renewal, original authenticator
raCertAuth
>>>>>>> found
>>>>>>> CertProcessor: No authenticator credentials required
>>>>>>> processRenewal: set original Inputs into profile Context
>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>>>>> cert_request_type
>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in
>>>>>>> ctx:pkcs10
>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>>>>>> cert_request
>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in
ctx:<hex
>>>>>>> encoded csr>
>>>>>>> ...
>>>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>>>>>
>>>>>>> The hex encoded csr in the last line decodes to have the bad
>>>>>>> subject
>>>>>>> where the hostname is one of our other ipa servers.
>>>>>>>
>>>>>>> The certificate always getthe same hostname in the bad
subject cn.
>>>>>>>
>>>>>>> Do you have any other ideas of things to check to find out
what's
>>>>>>> generating the bad subject?
>>>>>>
>>>>>> The order certmonger uses for the subject is:
>>>>>>
>>>>>> 1. cert_subject_der
>>>>>> 2. If there is no DER, try to encode cert_subject
>>>>>> 3. If that fails try again a different way
>>>>>> 4. If it fails yet again use CN=localhost
>>>>>>
>>>>>> It is baffling that it would pick a hostname much less one that
>>>>>> certmonger shouldn't even know about.
>>>>>
>>>>> Indeed, and if I try to renew other certificates for some of them it
>>>>> chooses other host names that should not be known about. Each
>>>>> certificate seems to get the same bad hostname each time I try to
>>>>> renew.
>>>>>
>>>>>>
>>>>>> I assume the CSR found in the dogtag logs matches the csr value
>>>>>> in the
>>>>>> certmonger request?
>>>>>
>>>>> No, its different. The issued certificate matches the csr seen in
>>>>> dogtag
>>>>> which makes sense. But the csr in the dogtag logs has the bad
>>>>> subject.
>>>>> The csr in the request directory file has the good subject.
>>>>>
>>>>>>
>>>>>> Can you share the certmonger request file privately?
>>>>>
>>>>> Yes, certainly.
>>>>>
>>>>> Thanks for your continued help on this.
>>>>
>>>> Let's try this. We'll drop the current request and create a new
one.
>>>>
>>>> Stop tomcat, restore the old cert database, start tomcat, then:
>>>>
>>>> # getcert stop-tracking -i <request id>
>>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
>>>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P
<pin> -B
>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>>> cert-pki-ca"'
>>>>
>>>> Then try resubmitting the request.
>>>>
>>> Hi Rob
>>>
>>> When you say 'stop tomcat', I'm just doing an ipactl stop. That
is
>>> stopiing the ipa pki-tomcat service. Is that good enough or are there
>>> other services that need stopping too?
>>>
>>> I tried the stop-tracking/start-tracking. Exactly the same result as
>>> before. Same hostname appears in the subject CN field of the new
>>> certificate and then the pki-tomcat service won't start.
>>>
>>> I've also been back and redone the manual submits changing the point at
>>> which I copied in the edited request as I wondered if there was some
>>> caching of csr's going on in certmonger and it was remembering a
>>> previous request it had read on startup before I replaced the
>>> request file.
>>>
>>> But nothing seems to stop dogtag issuing a certificate with the Subject
>>> CN being that host name.
>>>
>>> Is it possible that dogtag is somehow overriding the Subject its being
>>> explicitly given?
>>>
>>> FWIW the CSR in the dogtag debug log is always identical, but I guess
>>> thats expected if the CSR information is always the same.
>>>
>>> I'm not sure if its possible to increase the verbosity on the dogtag
>>> side. Can you advise please?
>>>
>>> I've been assuming that its the bad subject that is preventing the
>>> pki-tomcat from starting after the new certificate is issued. Does that
>>> make sense to you?
>>>
>>> I wonder where we can go from here? There must be a good reason for
>>> whats happening!
>>
>> Let's step back and gather some more information.
>>
>> Can you restore the files again and send me the output of:
>>
>> # getcert list
>>
>> # certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
>> cert-pki-ca"
>
> Hi Rob
>
> You should have the output you requested by private email now.
>
>>
>> And the exact commands you are using to do all this, including
>> re-issuing the cert
>
> Thats a great idea. Maybe I'm just doing something in the wrong order.
>
>>
>> Using ipactl is fine. You just don't want to touch the NSS databases
>> while the service is running. Similarly you probably don't want to mess
>> with certmonger requests while it is running as it won't see the changes
>> until it restarts.
> Here is the procedure I'm using to renew the certificates. One of the
> tricky things is to not have the certificate system working when
> certmonger is started so that it doesn't renew a certificate on
> startup that might not be one I have changed the request file for
> (other certificates have similar problems with bad subject CN and stop
> the certificate system.)
>
> I've found it convenient to keep reinitializing the log files because
> they get very confusing when the clock keeps getting set back.
>
> The request number has changed since we did the stop-tracking /
> start-tracking.
>
> Roderick
>
> ipactl stop
>
> # Set clock back:
> systemctl stop ntpd
> timedatectl set-time "2017-10-23 00:00:00"
> date
>
> #Stop certificate renewals the systemd way
> systemctl stop certmonger
>
> # Sort out certmonger log
> mv /var/log/certmonger.log /var/log/certmonger.log-$(date +%Y%m%d:%H:%M)
>
> # ****** Copy in the fixed certmonger request
> \cp /root/20161124081302.ocsp_backup
> /var/lib/certmonger/requests/20161124081302
>
>
> # In another window
> certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
>
> # In the original window
> # Sort out dogtag log
> mv /var/log/pki/pki-tomcat/ca/debug
> /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
> touch /var/log/pki/pki-tomcat/ca/debug
> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
>
> mv /var/log/pki/pki-tomcat/ca/selftests.log
> /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
> touch /var/log/pki/pki-tomcat/ca/selftests.log
> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
>
> \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
> \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
> /etc/pki/pki-tomcat/alias/cert8.db
> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
> /etc/pki/pki-tomcat/alias/key3.db
>
> \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg
> /etc/pki/pki-tomcat/ca/CS.cfg
>
> # Fix up certificate permissions and check all is ok
> certutil -M -n "caSigningCert cert-pki-ca" -d
> /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu
> certutil -L -d /etc/pki/pki-tomcat/alias
> certutil -K -d /etc/pki/pki-tomcat/alias
> <pin>
>
> # Fix dogtag so it starts
> pki-server subsystem-enable ca >
> # Start IdM services
>
> ipactl start
> ipactl status
>
> #Resubmit certificate ocsp
> date;getcert resubmit -i 20161124081302
Hi Rob
Does it look like there is anything I should be doing differently in
this command sequence to try to get the certificate renewed correctly?
I'm starting to wonder whether I should look at alternative strategies
for recovering the freeipa servers if the certificates won't renew
correctly. Do you have any ideas on this please?
This looks reasonable. Have you tried stop-tracking and start-tracking I
suggested?
I'm still baffled how the name(s) are getting mixed up.
Something else you might want to try would be to move all the other
requests out of the way in case there is some sort of memory corruption
causing the hostname to get in there somehow.
rob
12:36 p.m.
On 31/01/2018 20:36, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
> On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote:
>> On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users
wrote:
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Our freeipa certificates need to
be renewed due to passing
>>>>>>>>>>>>>> their
>>>>>>>>>>>>>> expiry
>>>>>>>>>>>>>> dates.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> While some certificates have
renewed ok, the ipaCert and
>>>>>>>>>>>>>> auditSigningCert are renewing but
the new certificates
>>>>>>>>>>>>>> have the
>>>>>>>>>>>>>> wrong
>>>>>>>>>>>>>> Subject.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Environment is:
>>>>>>>>>>>>>> serverA (CRL, first, master) RHEL
7.3, ipa 4.4
>>>>>>>>>>>>>> serverB (replica) RHEL 7.3, ipa
4.4
>>>>>>>>>>>>>> serverC (replica) RHEL 7.4, ipa
4.5
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Once there are renewed
certificates with the wrong Subject
>>>>>>>>>>>>>> present,
>>>>>>>>>>>>>> there are various problems with
renewing the remaining
>>>>>>>>>>>>>> certificates,
>>>>>>>>>>>>>> which I think might be related to
the bad Subject:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) When just ipaCert has the
wrong subject no further
>>>>>>>>>>>>>> renewals
>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) When auditSigningCert has the
wrong subject the ipa
>>>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>>>> service will not start and no
further renewals happen.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been round the following
loop many times on ServerA, our
>>>>>>>>>>>>>> first
>>>>>>>>>>>>>> master:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Restore good certificates from
backup
>>>>>>>>>>>>>> 2) Put the clock back to a time
when certificates are all
>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>> 3) Resubmit certificates for
renewal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Each time the ipaCert renews it
has the same wrong
>>>>>>>>>>>>>> Subject. The
>>>>>>>>>>>>>> wrong
>>>>>>>>>>>>>> Subject includes the host name of
one of our ipa client
>>>>>>>>>>>>>> systems.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Each time the auditSigningCert
renews it has the same wrong
>>>>>>>>>>>>>> Subject
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> a different subject to the
ipaCert. The wrong Subject in this
>>>>>>>>>>>>>> case
>>>>>>>>>>>>>> includes the host name of a
system which has never been an
>>>>>>>>>>>>>> ipa
>>>>>>>>>>>>>> client,
>>>>>>>>>>>>>> but might have been added and
removed with ipa host-add
>>>>>>>>>>>>>> and ipa
>>>>>>>>>>>>>> host-del
>>>>>>>>>>>>>> for testing something, a while
ago.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As far as I can see, the
"cert_subject" is set correctly
>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> /var/lib/certmonger/<request
id> until the point at which the
>>>>>>>>>>>>>> certificate is actually renewed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd be very grateful for some
pointers as to which
>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>> options
>>>>>>>>>>>>>> and logs to check through to
resolve this problem on our
>>>>>>>>>>>>>> production
>>>>>>>>>>>>>> system.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If its of any relevance we did
change which server is the
>>>>>>>>>>>>>> first
>>>>>>>>>>>>>> master
>>>>>>>>>>>>>> some time ago.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd pull the CSR out of dogtag
(CS.cfg) and/or certmonger
>>>>>>>>>>>>> to see
>>>>>>>>>>>>> what
>>>>>>>>>>>>> the subject is.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not seeing any obvious CSR fields
in the
>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>>>>>>
>>>>>>>>>>> foo.bar.certreq=
>>>>>>>>>>>
>>>>>>>>>>>> The CSR in the certmonger requests file
for the
>>>>>>>>>>>> auditSigningCert
>>>>>>>>>>>> seems
>>>>>>>>>>>> to be showing with the correct Subject.
This is different from
>>>>>>>>>>>> the bad
>>>>>>>>>>>> subject showing in the requests file
field:
>>>>>>>>>>>> cert_subject=
>>>>>>>>>>>
>>>>>>>>>>> The value of cert_subject comes from the
issued certificate.
>>>>>>>>>>>
>>>>>>>>>>>> and the Subject which is showing in the
'getcert list' output
>>>>>>>>>>>> (which is
>>>>>>>>>>>> the same as that in the cert_subject=
field.>
>>>>>>>>>>>> I'm not quite sure what this all
means.
>>>>>>>>>>>
>>>>>>>>>>> It is displayed from the data within the
tracked certmonger
>>>>>>>>>>> request.
>>>>>>>>>>>
>>>>>>>>>>> certmonger logs to syslog so you can check
there or you can stop
>>>>>>>>>>> the
>>>>>>>>>>> process and run it manually with: certmonger
-n -d 9 2>&1 | tee
>>>>>>>>>>> certmonger.log
>>>>>>>>>>>
>>>>>>>>>>> That will provide a lot of debugging output
that may show
>>>>>>>>>>> what is
>>>>>>>>>>> going on.
>>>>>>>>>>
>>>>>>>>>> I've restored certificate databases from
backup and put the clock
>>>>>>>>>> back
>>>>>>>>>> to a time when certificates are valid and renewed
the
>>>>>>>>>> ocspSigining
>>>>>>>>>> certificate with:
>>>>>>>>>> getcert resubmit -N "CN=OCSP
Subsystem,O=<REALM>" -i
>>>>>>>>>> 20161124081302
>>>>>>>>>>
>>>>>>>>>> (I've previously tried without the -N with
similar results)
>>>>>>>>>>
>>>>>>>>>> What I am seeing in the certmonger logs is:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2017-10-23 00:05:28 [438] Located the key
'ocspSigningCert
>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>> 2017-10-23 00:05:28 [438] Converted private key
'ocspSigningCert
>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>> 2017-10-23 00:05:28 [439] Located the
certificate
>>>>>>>>>> "ocspSigningCert
>>>>>>>>>> cert-pki-ca".
>>>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named
"ocspSigningCert
>>>>>>>>>> cert-pki-ca" in token "NSS Certificate
DB" in database
>>>>>>>>>> "/etc/pki/pki-tomcat/alias" will not be
valid after
>>>>>>>>>> 20171025122401.
>>>>>>>>>> 2017-10-23 00:05:28 [442] Located the key
'ocspSigningCert
>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>> 2017-10-23 00:05:28 [442] Converted private key
'ocspSigningCert
>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>> 2017-10-23 00:05:28 [443] Located the
certificate
>>>>>>>>>> "ocspSigningCert
>>>>>>>>>> cert-pki-ca".
>>>>>>>>>> 2017-10-23 00:05:28 [444] Located the key
'ocspSigningCert
>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>> 2017-10-23 00:05:28 [444] Converted private key
'ocspSigningCert
>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate
with the same
>>>>>>>>>> nickname but
>>>>>>>>>> different subject, removing certificate
"ocspSigningCert
>>>>>>>>>> cert-pki-ca"
>>>>>>>>>> with subject "CN=OCSP
Subsystem,O=<REALM>".
>>>>>>>>>> 2017-10-23 00:05:39 [581] Imported certificate
"ocspSigningCert
>>>>>>>>>> cert-pki-ca", got nickname
"ocspSigningCert cert-pki-ca".
>>>>>>>>>> 2017-10-23 00:05:39 [583] Located the
certificate
>>>>>>>>>> "ocspSigningCert
>>>>>>>>>> cert-pki-ca".
>>>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>>>>>> cert-pki-ca"" (0).
>>>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named
"ocspSigningCert
>>>>>>>>>> cert-pki-ca" in token "NSS Certificate
DB" in database
>>>>>>>>>> "/etc/pki/pki-tomcat/alias" issued by
CA and saved.
>>>>>>>>>>
>>>>>>>>>> I now have a date valid ocspSigningCertificate,
but with the
>>>>>>>>>> wrong
>>>>>>>>>> subject, and a broken certificate system which
will no longer
>>>>>>>>>> start.
>>>>>>>>>>
>>>>>>>>>> ipactl status
>>>>>>>>>> ...
>>>>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>>>>
>>>>>>>>>> I can't renew other expired certificates.
>>>>>>>>>>
>>>>>>>>>> I also note that there is now no key for
>>>>>>>>>> ocspSigningCertificate as
>>>>>>>>>> shown
>>>>>>>>>> by:
>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>>
>>>>>>>>>> I wonder if this is because the certificate
subject changed?
>>>>>>>>>> There
>>>>>>>>>> was a
>>>>>>>>>> key before the certificate renewed.
>>>>>>>>>>
>>>>>>>>>> The ca debug logs are showing:
>>>>>>>>>>
>>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]:
Found cert by
>>>>>>>>>> nickname:
>>>>>>>>>> 'ocspSigningCert cert-pki-ca' with serial
number: 268370108
>>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]:
converted to
>>>>>>>>>> x509CertImpl
>>>>>>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]:
SigningUnit:
>>>>>>>>>> Certificate
>>>>>>>>>> object not found
>>>>>>>>>> Certificate object not found
>>>>>>>>>> at
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>>>>>
>>>>>>>>>> Any help in repairing my broken ipa system will
be much
>>>>>>>>>> appreciated.
>>>>>>>>>
>>>>>>>>> Sorry for the delay. I think my previous reading of
the certmonger
>>>>>>>>> csrgen code was wrong.
>>>>>>>>>
>>>>>>>>> IIRC in your certmonger entry the cert_subject has
the hostname
>>>>>>>>> value
>>>>>>>>> right? Do you also have cert_subject_der?
>>>>>>>>>
>>>>>>>>> You can decode that by:
>>>>>>>>>
>>>>>>>>> 1. Running a hex-to-bin, something hacky like this in
python:
>>>>>>>>>
>>>>>>>>> import binascii
>>>>>>>>>
>>>>>>>>> hex_string = "<hex value>"
>>>>>>>>>
>>>>>>>>> binary_string = binascii.unhexlify(hex_string)
>>>>>>>>>
>>>>>>>>> fd = open("out", "wb")
>>>>>>>>> fd.write(binary_string)
>>>>>>>>> fd.close()
>>>>>>>>>
>>>>>>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>>>>>>
>>>>>>>>> I'm going to assume this also has the hostname
encoded in it.
>>>>>>>>
>>>>>>>> Hi Again
>>>>>>>>
>>>>>>>> Yes, correct. The cert_subject_der does have the bad CN
with the
>>>>>>>> hostname encoded in it.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you try this:
>>>>>>>>>
>>>>>>>>> 1. Make a backup of the request file (just in case)
> 2. Remove
>>>>>>>>> cert_subject_der
>>>>>>>>> 3. Modify cert_subject to be CN=OCSP
Subsystem,O=<YOUR_REALM>
>>>>>>>>> 3. Try the renewal again
>>>>>>>>
>>>>>>>> I ran though all of this, checking that the request file
was still
>>>>>>>> what
>>>>>>>> we wanted after restarting certmonger with verbose
logging,
>>>>>>>> restoring
>>>>>>>> all the databases, fixing permissions, checking keys
etc., and
>>>>>>>> ran the
>>>>>>>> getcert resubmit.
>>>>>>>>
>>>>>>>> It still generates a certificate with the bad CN
including a
>>>>>>>> hostname.
>>>>>>>>
>>>>>>>> Then the pki-tomcat server fails to start again.
>>>>>>>>
>>>>>>>> Other things I have checked are:
>>>>>>>> 1) The csr= field in the (modified) request file has the
correct
>>>>>>>> subject.
>>>>>>>>
>>>>>>>> 2) The dogtag ca debug log file is showing:
>>>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST
2017
>>>>>>>> processRenewal: orig subj dn =CN=OCSP
Subsystem,O=AST.CAM.AC.UK
>>>>>>>> ...
>>>>>>>> RenewalSubmitter: renewal original
profileId=caIPAserviceCert
>>>>>>>> RenewalSubmitter: for renewal, original authenticator
raCertAuth
>>>>>>>> found
>>>>>>>> CertProcessor: No authenticator credentials required
>>>>>>>> processRenewal: set original Inputs into profile Context
>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input
name=
>>>>>>>> cert_request_type
>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value
in
>>>>>>>> ctx:pkcs10
>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting input
name=
>>>>>>>> cert_request
>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting value in
ctx:<hex
>>>>>>>> encoded csr>
>>>>>>>> ...
>>>>>>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>>>>>>
>>>>>>>> The hex encoded csr in the last line decodes to have the
bad
>>>>>>>> subject
>>>>>>>> where the hostname is one of our other ipa servers.
>>>>>>>>
>>>>>>>> The certificate always getthe same hostname in the bad
subject cn.
>>>>>>>>
>>>>>>>> Do you have any other ideas of things to check to find
out what's
>>>>>>>> generating the bad subject?
>>>>>>>
>>>>>>> The order certmonger uses for the subject is:
>>>>>>>
>>>>>>> 1. cert_subject_der
>>>>>>> 2. If there is no DER, try to encode cert_subject
>>>>>>> 3. If that fails try again a different way
>>>>>>> 4. If it fails yet again use CN=localhost
>>>>>>>
>>>>>>> It is baffling that it would pick a hostname much less one
that
>>>>>>> certmonger shouldn't even know about.
>>>>>>
>>>>>> Indeed, and if I try to renew other certificates for some of them
it
>>>>>> chooses other host names that should not be known about. Each
>>>>>> certificate seems to get the same bad hostname each time I try
to
>>>>>> renew.
>>>>>>
>>>>>>>
>>>>>>> I assume the CSR found in the dogtag logs matches the csr
value
>>>>>>> in the
>>>>>>> certmonger request?
>>>>>>
>>>>>> No, its different. The issued certificate matches the csr seen
in
>>>>>> dogtag
>>>>>> which makes sense. But the csr in the dogtag logs has the bad
>>>>>> subject.
>>>>>> The csr in the request directory file has the good subject.
>>>>>>
>>>>>>>
>>>>>>> Can you share the certmonger request file privately?
>>>>>>
>>>>>> Yes, certainly.
>>>>>>
>>>>>> Thanks for your continued help on this.
>>>>>
>>>>> Let's try this. We'll drop the current request and create a
new one.
>>>>>
>>>>> Stop tomcat, restore the old cert database, start tomcat, then:
>>>>>
>>>>> # getcert stop-tracking -i <request id>
>>>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
>>>>> "ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias
-P <pin> -B
>>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>>>> cert-pki-ca"'
>>>>>
>>>>> Then try resubmitting the request.
>>>>>
>>>> Hi Rob
>>>>
>>>> When you say 'stop tomcat', I'm just doing an ipactl stop.
That is
>>>> stopiing the ipa pki-tomcat service. Is that good enough or are there
>>>> other services that need stopping too?
>>>>
>>>> I tried the stop-tracking/start-tracking. Exactly the same result as
>>>> before. Same hostname appears in the subject CN field of the new
>>>> certificate and then the pki-tomcat service won't start.
>>>>
>>>> I've also been back and redone the manual submits changing the point
at
>>>> which I copied in the edited request as I wondered if there was some
>>>> caching of csr's going on in certmonger and it was remembering a
>>>> previous request it had read on startup before I replaced the
>>>> request file.
>>>>
>>>> But nothing seems to stop dogtag issuing a certificate with the Subject
>>>> CN being that host name.
>>>>
>>>> Is it possible that dogtag is somehow overriding the Subject its being
>>>> explicitly given?
>>>>
>>>> FWIW the CSR in the dogtag debug log is always identical, but I guess
>>>> thats expected if the CSR information is always the same.
>>>>
>>>> I'm not sure if its possible to increase the verbosity on the dogtag
>>>> side. Can you advise please?
>>>>
>>>> I've been assuming that its the bad subject that is preventing the
>>>> pki-tomcat from starting after the new certificate is issued. Does that
>>>> make sense to you?
>>>>
>>>> I wonder where we can go from here? There must be a good reason for
>>>> whats happening!
>>>
>>> Let's step back and gather some more information.
>>>
>>> Can you restore the files again and send me the output of:
>>>
>>> # getcert list
>>>
>>> # certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
>>> cert-pki-ca"
>>
>> Hi Rob
>>
>> You should have the output you requested by private email now.
>>
>>>
>>> And the exact commands you are using to do all this, including
>>> re-issuing the cert
>>
>> Thats a great idea. Maybe I'm just doing something in the wrong order.
>>
>>>
>>> Using ipactl is fine. You just don't want to touch the NSS databases
>>> while the service is running. Similarly you probably don't want to mess
>>> with certmonger requests while it is running as it won't see the changes
>>> until it restarts.
>> Here is the procedure I'm using to renew the certificates. One of the
>> tricky things is to not have the certificate system working when
>> certmonger is started so that it doesn't renew a certificate on
>> startup that might not be one I have changed the request file for
>> (other certificates have similar problems with bad subject CN and stop
>> the certificate system.)
>>
>> I've found it convenient to keep reinitializing the log files because
>> they get very confusing when the clock keeps getting set back.
>>
>> The request number has changed since we did the stop-tracking /
>> start-tracking.
>>
>> Roderick
>>
>> ipactl stop
>>
>> # Set clock back:
>> systemctl stop ntpd
>> timedatectl set-time "2017-10-23 00:00:00"
>> date
>>
>> #Stop certificate renewals the systemd way
>> systemctl stop certmonger
>>
>> # Sort out certmonger log
>> mv /var/log/certmonger.log /var/log/certmonger.log-$(date +%Y%m%d:%H:%M)
>>
>> # ****** Copy in the fixed certmonger request
>> \cp /root/20161124081302.ocsp_backup
>> /var/lib/certmonger/requests/20161124081302
>>
>>
>> # In another window
>> certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
>>
>> # In the original window
>> # Sort out dogtag log
>> mv /var/log/pki/pki-tomcat/ca/debug
>> /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
>> touch /var/log/pki/pki-tomcat/ca/debug
>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
>>
>> mv /var/log/pki/pki-tomcat/ca/selftests.log
>> /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
>> touch /var/log/pki/pki-tomcat/ca/selftests.log
>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
>>
>> \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
>> \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
>> /etc/pki/pki-tomcat/alias/cert8.db
>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
>> /etc/pki/pki-tomcat/alias/key3.db
>>
>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg
>> /etc/pki/pki-tomcat/ca/CS.cfg
>>
>> # Fix up certificate permissions and check all is ok
>> certutil -M -n "caSigningCert cert-pki-ca" -d
>> /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu
>> certutil -L -d /etc/pki/pki-tomcat/alias
>> certutil -K -d /etc/pki/pki-tomcat/alias
>> <pin>
>>
>> # Fix dogtag so it starts
>> pki-server subsystem-enable ca >
>> # Start IdM services
>>
>> ipactl start
>> ipactl status
>>
>> #Resubmit certificate ocsp
>> date;getcert resubmit -i 20161124081302
>
> Hi Rob
>
> Does it look like there is anything I should be doing differently in
> this command sequence to try to get the certificate renewed correctly?
>
> I'm starting to wonder whether I should look at alternative strategies
> for recovering the freeipa servers if the certificates won't renew
> correctly. Do you have any ideas on this please?
>
This looks reasonable. Have you tried stop-tracking and start-tracking I
suggested?
Yes,I tried the stop-tracking / start-tracking. Same result, bad Subject CN.
I'm still baffled how the name(s) are getting mixed up.
Indeed, something is going round changing the subject. In the dogtag
debug log is:
[23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal:
origNotAfter =Wed Oct 25 13:24:01 BST 2017
[23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal: orig subj
dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
and then very shortly afterwards there is a csr listed that decodes to
have the bad Subject CN.
The listed csr doesn't match any I have been able to find on the system,
so something must be going round and changing it.
Something else you might want to try would be to move all the other
requests out of the way in case there is some sort of memory corruption
causing the hostname to get in there somehow.
Thanks for the suggestion which I have now tried. Same result I'm afraid.
I've now been battling this issue since the beginning of November.
I'm wondering whether, just to get our IPA system fully operational
again, whether we might be able to add the correct CN into a Subject
Alternative name?
Does this sound like a possible way to at least get all the IPA services
started properly?
Do you think it would cause further problems down the line?
Would you be able to advise how to get the certificates to have the
correct CN in the Subject Alternative Name.
Maybe there are some other tricks we could use just to get going again.
Can you advise please?
Thanks again.
Roderick
1:44 p.m.
Roderick Johnstone wrote:
On 31/01/2018 20:36, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote:
>>> On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users
wrote:
>>>>>>>>>>>>> On 15/01/2018 16:06, Rob Crittenden
via FreeIPA-users wrote:
>>>>>>>>>>>>>> Roderick Johnstone via
FreeIPA-users wrote:
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Our freeipa certificates need
to be renewed due to passing
>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>> expiry
>>>>>>>>>>>>>>> dates.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> While some certificates have
renewed ok, the ipaCert and
>>>>>>>>>>>>>>> auditSigningCert are renewing
but the new certificates
>>>>>>>>>>>>>>> have the
>>>>>>>>>>>>>>> wrong
>>>>>>>>>>>>>>> Subject.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Environment is:
>>>>>>>>>>>>>>> serverA (CRL, first, master)
RHEL 7.3, ipa 4.4
>>>>>>>>>>>>>>> serverB (replica) RHEL 7.3,
ipa 4.4
>>>>>>>>>>>>>>> serverC (replica) RHEL 7.4,
ipa 4.5
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Once there are renewed
certificates with the wrong Subject
>>>>>>>>>>>>>>> present,
>>>>>>>>>>>>>>> there are various problems
with renewing the remaining
>>>>>>>>>>>>>>> certificates,
>>>>>>>>>>>>>>> which I think might be
related to the bad Subject:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) When just ipaCert has the
wrong subject no further
>>>>>>>>>>>>>>> renewals
>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) When auditSigningCert has
the wrong subject the ipa
>>>>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>>>>> service will not start and no
further renewals happen.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've been round the
following loop many times on
>>>>>>>>>>>>>>> ServerA, our
>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>> master:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) Restore good certificates
from backup
>>>>>>>>>>>>>>> 2) Put the clock back to a
time when certificates are all
>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>> 3) Resubmit certificates for
renewal
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Each time the ipaCert renews
it has the same wrong
>>>>>>>>>>>>>>> Subject. The
>>>>>>>>>>>>>>> wrong
>>>>>>>>>>>>>>> Subject includes the host
name of one of our ipa client
>>>>>>>>>>>>>>> systems.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Each time the
auditSigningCert renews it has the same wrong
>>>>>>>>>>>>>>> Subject
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> a different subject to the
ipaCert. The wrong Subject in
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>> includes the host name of a
system which has never been an
>>>>>>>>>>>>>>> ipa
>>>>>>>>>>>>>>> client,
>>>>>>>>>>>>>>> but might have been added and
removed with ipa host-add
>>>>>>>>>>>>>>> and ipa
>>>>>>>>>>>>>>> host-del
>>>>>>>>>>>>>>> for testing something, a
while ago.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As far as I can see, the
"cert_subject" is set correctly
>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>
/var/lib/certmonger/<request id> until the point at
>>>>>>>>>>>>>>> which the
>>>>>>>>>>>>>>> certificate is actually
renewed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd be very grateful for
some pointers as to which
>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>> and logs to check through to
resolve this problem on our
>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>> system.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If its of any relevance we
did change which server is the
>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>> master
>>>>>>>>>>>>>>> some time ago.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd pull the CSR out of
dogtag (CS.cfg) and/or certmonger
>>>>>>>>>>>>>> to see
>>>>>>>>>>>>>> what
>>>>>>>>>>>>>> the subject is.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not seeing any obvious CSR
fields in the
>>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>>>>>>>
>>>>>>>>>>>> foo.bar.certreq=
>>>>>>>>>>>>
>>>>>>>>>>>>> The CSR in the certmonger requests
file for the
>>>>>>>>>>>>> auditSigningCert
>>>>>>>>>>>>> seems
>>>>>>>>>>>>> to be showing with the correct
Subject. This is different
>>>>>>>>>>>>> from
>>>>>>>>>>>>> the bad
>>>>>>>>>>>>> subject showing in the requests file
field:
>>>>>>>>>>>>> cert_subject=
>>>>>>>>>>>>
>>>>>>>>>>>> The value of cert_subject comes from the
issued certificate.
>>>>>>>>>>>>
>>>>>>>>>>>>> and the Subject which is showing in
the 'getcert list' output
>>>>>>>>>>>>> (which is
>>>>>>>>>>>>> the same as that in the cert_subject=
field.>
>>>>>>>>>>>>> I'm not quite sure what this all
means.
>>>>>>>>>>>>
>>>>>>>>>>>> It is displayed from the data within the
tracked certmonger
>>>>>>>>>>>> request.
>>>>>>>>>>>>
>>>>>>>>>>>> certmonger logs to syslog so you can
check there or you can
>>>>>>>>>>>> stop
>>>>>>>>>>>> the
>>>>>>>>>>>> process and run it manually with:
certmonger -n -d 9 2>&1 |
>>>>>>>>>>>> tee
>>>>>>>>>>>> certmonger.log
>>>>>>>>>>>>
>>>>>>>>>>>> That will provide a lot of debugging
output that may show
>>>>>>>>>>>> what is
>>>>>>>>>>>> going on.
>>>>>>>>>>>
>>>>>>>>>>> I've restored certificate databases from
backup and put the
>>>>>>>>>>> clock
>>>>>>>>>>> back
>>>>>>>>>>> to a time when certificates are valid and
renewed the
>>>>>>>>>>> ocspSigining
>>>>>>>>>>> certificate with:
>>>>>>>>>>> getcert resubmit -N "CN=OCSP
Subsystem,O=<REALM>" -i
>>>>>>>>>>> 20161124081302
>>>>>>>>>>>
>>>>>>>>>>> (I've previously tried without the -N
with similar results)
>>>>>>>>>>>
>>>>>>>>>>> What I am seeing in the certmonger logs is:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2017-10-23 00:05:28 [438] Located the key
'ocspSigningCert
>>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>>> 2017-10-23 00:05:28 [438] Converted private
key
>>>>>>>>>>> 'ocspSigningCert
>>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>>> 2017-10-23 00:05:28 [439] Located the
certificate
>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>> cert-pki-ca".
>>>>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate
named
>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>> cert-pki-ca" in token "NSS
Certificate DB" in database
>>>>>>>>>>> "/etc/pki/pki-tomcat/alias" will
not be valid after
>>>>>>>>>>> 20171025122401.
>>>>>>>>>>> 2017-10-23 00:05:28 [442] Located the key
'ocspSigningCert
>>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>>> 2017-10-23 00:05:28 [442] Converted private
key
>>>>>>>>>>> 'ocspSigningCert
>>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>>> 2017-10-23 00:05:28 [443] Located the
certificate
>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>> cert-pki-ca".
>>>>>>>>>>> 2017-10-23 00:05:28 [444] Located the key
'ocspSigningCert
>>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>>> 2017-10-23 00:05:28 [444] Converted private
key
>>>>>>>>>>> 'ocspSigningCert
>>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>>> 2017-10-23 00:05:39 [581] Found a certificate
with the same
>>>>>>>>>>> nickname but
>>>>>>>>>>> different subject, removing certificate
"ocspSigningCert
>>>>>>>>>>> cert-pki-ca"
>>>>>>>>>>> with subject "CN=OCSP
Subsystem,O=<REALM>".
>>>>>>>>>>> 2017-10-23 00:05:39 [581] Imported
certificate "ocspSigningCert
>>>>>>>>>>> cert-pki-ca", got nickname
"ocspSigningCert cert-pki-ca".
>>>>>>>>>>> 2017-10-23 00:05:39 [583] Located the
certificate
>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>> cert-pki-ca".
>>>>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>>>>>>>
"/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>>>>>>>>>> cert-pki-ca"" (0).
>>>>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate
named
>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>> cert-pki-ca" in token "NSS
Certificate DB" in database
>>>>>>>>>>> "/etc/pki/pki-tomcat/alias" issued
by CA and saved.
>>>>>>>>>>>
>>>>>>>>>>> I now have a date valid
ocspSigningCertificate, but with the
>>>>>>>>>>> wrong
>>>>>>>>>>> subject, and a broken certificate system
which will no longer
>>>>>>>>>>> start.
>>>>>>>>>>>
>>>>>>>>>>> ipactl status
>>>>>>>>>>> ...
>>>>>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>>>>>
>>>>>>>>>>> I can't renew other expired
certificates.
>>>>>>>>>>>
>>>>>>>>>>> I also note that there is now no key for
>>>>>>>>>>> ocspSigningCertificate as
>>>>>>>>>>> shown
>>>>>>>>>>> by:
>>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>>>
>>>>>>>>>>> I wonder if this is because the certificate
subject changed?
>>>>>>>>>>> There
>>>>>>>>>>> was a
>>>>>>>>>>> key before the certificate renewed.
>>>>>>>>>>>
>>>>>>>>>>> The ca debug logs are showing:
>>>>>>>>>>>
>>>>>>>>>>>
[23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by
>>>>>>>>>>> nickname:
>>>>>>>>>>> 'ocspSigningCert cert-pki-ca' with
serial number: 268370108
>>>>>>>>>>>
[23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>>>>>>>>> x509CertImpl
>>>>>>>>>>>
[23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>>>>>>>>> Certificate
>>>>>>>>>>> object not found
>>>>>>>>>>> Certificate object not found
>>>>>>>>>>> at
>>>>>>>>>>>
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>>>>>>
>>>>>>>>>>> Any help in repairing my broken ipa system
will be much
>>>>>>>>>>> appreciated.
>>>>>>>>>>
>>>>>>>>>> Sorry for the delay. I think my previous reading
of the
>>>>>>>>>> certmonger
>>>>>>>>>> csrgen code was wrong.
>>>>>>>>>>
>>>>>>>>>> IIRC in your certmonger entry the cert_subject
has the hostname
>>>>>>>>>> value
>>>>>>>>>> right? Do you also have cert_subject_der?
>>>>>>>>>>
>>>>>>>>>> You can decode that by:
>>>>>>>>>>
>>>>>>>>>> 1. Running a hex-to-bin, something hacky like
this in python:
>>>>>>>>>>
>>>>>>>>>> import binascii
>>>>>>>>>>
>>>>>>>>>> hex_string = "<hex value>"
>>>>>>>>>>
>>>>>>>>>> binary_string = binascii.unhexlify(hex_string)
>>>>>>>>>>
>>>>>>>>>> fd = open("out", "wb")
>>>>>>>>>> fd.write(binary_string)
>>>>>>>>>> fd.close()
>>>>>>>>>>
>>>>>>>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>>>>>>>
>>>>>>>>>> I'm going to assume this also has the
hostname encoded in it.
>>>>>>>>>
>>>>>>>>> Hi Again
>>>>>>>>>
>>>>>>>>> Yes, correct. The cert_subject_der does have the bad
CN with the
>>>>>>>>> hostname encoded in it.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you try this:
>>>>>>>>>>
>>>>>>>>>> 1. Make a backup of the request file (just in
case) > 2. Remove
>>>>>>>>>> cert_subject_der
>>>>>>>>>> 3. Modify cert_subject to be CN=OCSP
Subsystem,O=<YOUR_REALM>
>>>>>>>>>> 3. Try the renewal again
>>>>>>>>>
>>>>>>>>> I ran though all of this, checking that the request
file was
>>>>>>>>> still
>>>>>>>>> what
>>>>>>>>> we wanted after restarting certmonger with verbose
logging,
>>>>>>>>> restoring
>>>>>>>>> all the databases, fixing permissions, checking keys
etc., and
>>>>>>>>> ran the
>>>>>>>>> getcert resubmit.
>>>>>>>>>
>>>>>>>>> It still generates a certificate with the bad CN
including a
>>>>>>>>> hostname.
>>>>>>>>>
>>>>>>>>> Then the pki-tomcat server fails to start again.
>>>>>>>>>
>>>>>>>>> Other things I have checked are:
>>>>>>>>> 1) The csr= field in the (modified) request file has
the correct
>>>>>>>>> subject.
>>>>>>>>>
>>>>>>>>> 2) The dogtag ca debug log file is showing:
>>>>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST
2017
>>>>>>>>> processRenewal: orig subj dn =CN=OCSP
Subsystem,O=AST.CAM.AC.UK
>>>>>>>>> ...
>>>>>>>>> RenewalSubmitter: renewal original
profileId=caIPAserviceCert
>>>>>>>>> RenewalSubmitter: for renewal, original authenticator
raCertAuth
>>>>>>>>> found
>>>>>>>>> CertProcessor: No authenticator credentials required
>>>>>>>>> processRenewal: set original Inputs into profile
Context
>>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting
input name=
>>>>>>>>> cert_request_type
>>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting
value in
>>>>>>>>> ctx:pkcs10
>>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting
input name=
>>>>>>>>> cert_request
>>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting
value in
>>>>>>>>> ctx:<hex
>>>>>>>>> encoded csr>
>>>>>>>>> ...
>>>>>>>>> Finish parsePKCS10 -
CN=<HOSTNAME>,O=<REALM>
>>>>>>>>>
>>>>>>>>> The hex encoded csr in the last line decodes to have
the bad
>>>>>>>>> subject
>>>>>>>>> where the hostname is one of our other ipa servers.
>>>>>>>>>
>>>>>>>>> The certificate always getthe same hostname in the
bad subject
>>>>>>>>> cn.
>>>>>>>>>
>>>>>>>>> Do you have any other ideas of things to check to
find out what's
>>>>>>>>> generating the bad subject?
>>>>>>>>
>>>>>>>> The order certmonger uses for the subject is:
>>>>>>>>
>>>>>>>> 1. cert_subject_der
>>>>>>>> 2. If there is no DER, try to encode cert_subject
>>>>>>>> 3. If that fails try again a different way
>>>>>>>> 4. If it fails yet again use CN=localhost
>>>>>>>>
>>>>>>>> It is baffling that it would pick a hostname much less
one that
>>>>>>>> certmonger shouldn't even know about.
>>>>>>>
>>>>>>> Indeed, and if I try to renew other certificates for some of
>>>>>>> them it
>>>>>>> chooses other host names that should not be known about.
Each
>>>>>>> certificate seems to get the same bad hostname each time I
try to
>>>>>>> renew.
>>>>>>>
>>>>>>>>
>>>>>>>> I assume the CSR found in the dogtag logs matches the csr
value
>>>>>>>> in the
>>>>>>>> certmonger request?
>>>>>>>
>>>>>>> No, its different. The issued certificate matches the csr
seen in
>>>>>>> dogtag
>>>>>>> which makes sense. But the csr in the dogtag logs has the
bad
>>>>>>> subject.
>>>>>>> The csr in the request directory file has the good subject.
>>>>>>>
>>>>>>>>
>>>>>>>> Can you share the certmonger request file privately?
>>>>>>>
>>>>>>> Yes, certainly.
>>>>>>>
>>>>>>> Thanks for your continued help on this.
>>>>>>
>>>>>> Let's try this. We'll drop the current request and create
a new one.
>>>>>>
>>>>>> Stop tomcat, restore the old cert database, start tomcat, then:
>>>>>>
>>>>>> # getcert stop-tracking -i <request id>
>>>>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
>>>>>> "ocspSigningCert cert-pki-ca" -d
/etc/pki/pki-tomcat/alias -P
>>>>>> <pin> -B
>>>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>> cert-pki-ca"'
>>>>>>
>>>>>> Then try resubmitting the request.
>>>>>>
>>>>> Hi Rob
>>>>>
>>>>> When you say 'stop tomcat', I'm just doing an ipactl
stop. That is
>>>>> stopiing the ipa pki-tomcat service. Is that good enough or are
there
>>>>> other services that need stopping too?
>>>>>
>>>>> I tried the stop-tracking/start-tracking. Exactly the same result as
>>>>> before. Same hostname appears in the subject CN field of the new
>>>>> certificate and then the pki-tomcat service won't start.
>>>>>
>>>>> I've also been back and redone the manual submits changing the
>>>>> point at
>>>>> which I copied in the edited request as I wondered if there was some
>>>>> caching of csr's going on in certmonger and it was remembering a
>>>>> previous request it had read on startup before I replaced the
>>>>> request file.
>>>>>
>>>>> But nothing seems to stop dogtag issuing a certificate with the
>>>>> Subject
>>>>> CN being that host name.
>>>>>
>>>>> Is it possible that dogtag is somehow overriding the Subject its
>>>>> being
>>>>> explicitly given?
>>>>>
>>>>> FWIW the CSR in the dogtag debug log is always identical, but I
guess
>>>>> thats expected if the CSR information is always the same.
>>>>>
>>>>> I'm not sure if its possible to increase the verbosity on the
dogtag
>>>>> side. Can you advise please?
>>>>>
>>>>> I've been assuming that its the bad subject that is preventing
the
>>>>> pki-tomcat from starting after the new certificate is issued. Does
>>>>> that
>>>>> make sense to you?
>>>>>
>>>>> I wonder where we can go from here? There must be a good reason for
>>>>> whats happening!
>>>>
>>>> Let's step back and gather some more information.
>>>>
>>>> Can you restore the files again and send me the output of:
>>>>
>>>> # getcert list
>>>>
>>>> # certutil -L -d /etc/pki/pki-tomcat/alias/ -n "ocspSigningCert
>>>> cert-pki-ca"
>>>
>>> Hi Rob
>>>
>>> You should have the output you requested by private email now.
>>>
>>>>
>>>> And the exact commands you are using to do all this, including
>>>> re-issuing the cert
>>>
>>> Thats a great idea. Maybe I'm just doing something in the wrong order.
>>>
>>>>
>>>> Using ipactl is fine. You just don't want to touch the NSS databases
>>>> while the service is running. Similarly you probably don't want to
>>>> mess
>>>> with certmonger requests while it is running as it won't see the
>>>> changes
>>>> until it restarts.
>>> Here is the procedure I'm using to renew the certificates. One of the
>>> tricky things is to not have the certificate system working when
>>> certmonger is started so that it doesn't renew a certificate on
>>> startup that might not be one I have changed the request file for
>>> (other certificates have similar problems with bad subject CN and stop
>>> the certificate system.)
>>>
>>> I've found it convenient to keep reinitializing the log files because
>>> they get very confusing when the clock keeps getting set back.
>>>
>>> The request number has changed since we did the stop-tracking /
>>> start-tracking.
>>>
>>> Roderick
>>>
>>> ipactl stop
>>>
>>> # Set clock back:
>>> systemctl stop ntpd
>>> timedatectl set-time "2017-10-23 00:00:00"
>>> date
>>>
>>> #Stop certificate renewals the systemd way
>>> systemctl stop certmonger
>>>
>>> # Sort out certmonger log
>>> mv /var/log/certmonger.log /var/log/certmonger.log-$(date
>>> +%Y%m%d:%H:%M)
>>>
>>> # ****** Copy in the fixed certmonger request
>>> \cp /root/20161124081302.ocsp_backup
>>> /var/lib/certmonger/requests/20161124081302
>>>
>>>
>>> # In another window
>>> certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
>>>
>>> # In the original window
>>> # Sort out dogtag log
>>> mv /var/log/pki/pki-tomcat/ca/debug
>>> /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
>>> touch /var/log/pki/pki-tomcat/ca/debug
>>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
>>>
>>> mv /var/log/pki/pki-tomcat/ca/selftests.log
>>> /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
>>> touch /var/log/pki/pki-tomcat/ca/selftests.log
>>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
>>>
>>> \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
>>> \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
>>> /etc/pki/pki-tomcat/alias/cert8.db
>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
>>> /etc/pki/pki-tomcat/alias/key3.db
>>>
>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg
>>> /etc/pki/pki-tomcat/ca/CS.cfg
>>>
>>> # Fix up certificate permissions and check all is ok
>>> certutil -M -n "caSigningCert cert-pki-ca" -d
>>> /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu
>>> certutil -L -d /etc/pki/pki-tomcat/alias
>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>> <pin>
>>>
>>> # Fix dogtag so it starts
>>> pki-server subsystem-enable ca >
>>> # Start IdM services
>>>
>>> ipactl start
>>> ipactl status
>>>
>>> #Resubmit certificate ocsp
>>> date;getcert resubmit -i 20161124081302
>>
>> Hi Rob
>>
>> Does it look like there is anything I should be doing differently in
>> this command sequence to try to get the certificate renewed correctly?
>>
>> I'm starting to wonder whether I should look at alternative strategies
>> for recovering the freeipa servers if the certificates won't renew
>> correctly. Do you have any ideas on this please?
>>
>
> This looks reasonable. Have you tried stop-tracking and start-tracking I
> suggested?
Yes,I tried the stop-tracking / start-tracking. Same result, bad Subject
CN.
>
> I'm still baffled how the name(s) are getting mixed up.
Indeed, something is going round changing the subject. In the dogtag
debug log is:
[23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal:
origNotAfter =Wed Oct 25 13:24:01 BST 2017
[23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal: orig subj
dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
and then very shortly afterwards there is a csr listed that decodes to
have the bad Subject CN.
The listed csr doesn't match any I have been able to find on the system,
so something must be going round and changing it.
>
> Something else you might want to try would be to move all the other
> requests out of the way in case there is some sort of memory corruption
> causing the hostname to get in there somehow.
Thanks for the suggestion which I have now tried. Same result I'm afraid.
I've now been battling this issue since the beginning of November.
I'm wondering whether, just to get our IPA system fully operational
again, whether we might be able to add the correct CN into a Subject
Alternative name?
Does this sound like a possible way to at least get all the IPA services
started properly?
Do you think it would cause further problems down the line?
Would you be able to advise how to get the certificates to have the
correct CN in the Subject Alternative Name.
Maybe there are some other tricks we could use just to get going again.
Can you advise please?
I'm really stumped here.
Do you have the CA installed on any other masters? You might try setting
it as the renewal master and trying the renewals there instead.
rob
4:33 a.m.
On 05/02/2018 19:44, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone wrote:
> On 31/01/2018 20:36, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 25/01/2018 16:56, Roderick Johnstone via FreeIPA-users wrote:
>>>> On 25/01/2018 13:43, Rob Crittenden via FreeIPA-users wrote:
>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>> On 24/01/2018 21:09, Rob Crittenden via FreeIPA-users wrote:
>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users
wrote:
>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>> On 23/01/2018 14:34, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>>>>> On 15/01/2018 20:07, Rob Crittenden via
FreeIPA-users wrote:
>>>>>>>>>>>>> Roderick Johnstone via FreeIPA-users
wrote:
>>>>>>>>>>>>>> On 15/01/2018 16:06, Rob
Crittenden via FreeIPA-users wrote:
>>>>>>>>>>>>>>> Roderick Johnstone via
FreeIPA-users wrote:
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Our freeipa certificates
need to be renewed due to passing
>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>> expiry
>>>>>>>>>>>>>>>> dates.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> While some certificates
have renewed ok, the ipaCert and
>>>>>>>>>>>>>>>> auditSigningCert are
renewing but the new certificates
>>>>>>>>>>>>>>>> have the
>>>>>>>>>>>>>>>> wrong
>>>>>>>>>>>>>>>> Subject.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Environment is:
>>>>>>>>>>>>>>>> serverA (CRL, first,
master) RHEL 7.3, ipa 4.4
>>>>>>>>>>>>>>>> serverB (replica) RHEL
7.3, ipa 4.4
>>>>>>>>>>>>>>>> serverC (replica) RHEL
7.4, ipa 4.5
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Once there are renewed
certificates with the wrong Subject
>>>>>>>>>>>>>>>> present,
>>>>>>>>>>>>>>>> there are various
problems with renewing the remaining
>>>>>>>>>>>>>>>> certificates,
>>>>>>>>>>>>>>>> which I think might be
related to the bad Subject:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1) When just ipaCert has
the wrong subject no further
>>>>>>>>>>>>>>>> renewals
>>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2) When auditSigningCert
has the wrong subject the ipa
>>>>>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>>>>>> service will not start
and no further renewals happen.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've been round the
following loop many times on
>>>>>>>>>>>>>>>> ServerA, our
>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>> master:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1) Restore good
certificates from backup
>>>>>>>>>>>>>>>> 2) Put the clock back to
a time when certificates are all
>>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>> 3) Resubmit certificates
for renewal
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Each time the ipaCert
renews it has the same wrong
>>>>>>>>>>>>>>>> Subject. The
>>>>>>>>>>>>>>>> wrong
>>>>>>>>>>>>>>>> Subject includes the host
name of one of our ipa client
>>>>>>>>>>>>>>>> systems.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Each time the
auditSigningCert renews it has the same wrong
>>>>>>>>>>>>>>>> Subject
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> a different subject to
the ipaCert. The wrong Subject in
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>> includes the host name of
a system which has never been an
>>>>>>>>>>>>>>>> ipa
>>>>>>>>>>>>>>>> client,
>>>>>>>>>>>>>>>> but might have been added
and removed with ipa host-add
>>>>>>>>>>>>>>>> and ipa
>>>>>>>>>>>>>>>> host-del
>>>>>>>>>>>>>>>> for testing something, a
while ago.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As far as I can see, the
"cert_subject" is set correctly
>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>
/var/lib/certmonger/<request id> until the point at
>>>>>>>>>>>>>>>> which the
>>>>>>>>>>>>>>>> certificate is actually
renewed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd be very grateful
for some pointers as to which
>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>> and logs to check through
to resolve this problem on our
>>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>> system.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If its of any relevance
we did change which server is the
>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>> master
>>>>>>>>>>>>>>>> some time ago.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd pull the CSR out of
dogtag (CS.cfg) and/or certmonger
>>>>>>>>>>>>>>> to see
>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>> the subject is.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not seeing any obvious
CSR fields in the
>>>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg
file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> foo.bar.certreq=
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The CSR in the certmonger
requests file for the
>>>>>>>>>>>>>> auditSigningCert
>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>> to be showing with the correct
Subject. This is different
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>> the bad
>>>>>>>>>>>>>> subject showing in the requests
file field:
>>>>>>>>>>>>>> cert_subject=
>>>>>>>>>>>>>
>>>>>>>>>>>>> The value of cert_subject comes from
the issued certificate.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> and the Subject which is showing
in the 'getcert list' output
>>>>>>>>>>>>>> (which is
>>>>>>>>>>>>>> the same as that in the
cert_subject= field.>
>>>>>>>>>>>>>> I'm not quite sure what this
all means.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is displayed from the data within
the tracked certmonger
>>>>>>>>>>>>> request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> certmonger logs to syslog so you can
check there or you can
>>>>>>>>>>>>> stop
>>>>>>>>>>>>> the
>>>>>>>>>>>>> process and run it manually with:
certmonger -n -d 9 2>&1 |
>>>>>>>>>>>>> tee
>>>>>>>>>>>>> certmonger.log
>>>>>>>>>>>>>
>>>>>>>>>>>>> That will provide a lot of debugging
output that may show
>>>>>>>>>>>>> what is
>>>>>>>>>>>>> going on.
>>>>>>>>>>>>
>>>>>>>>>>>> I've restored certificate databases
from backup and put the
>>>>>>>>>>>> clock
>>>>>>>>>>>> back
>>>>>>>>>>>> to a time when certificates are valid and
renewed the
>>>>>>>>>>>> ocspSigining
>>>>>>>>>>>> certificate with:
>>>>>>>>>>>> getcert resubmit -N "CN=OCSP
Subsystem,O=<REALM>" -i
>>>>>>>>>>>> 20161124081302
>>>>>>>>>>>>
>>>>>>>>>>>> (I've previously tried without the -N
with similar results)
>>>>>>>>>>>>
>>>>>>>>>>>> What I am seeing in the certmonger logs
is:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2017-10-23 00:05:28 [438] Located the key
'ocspSigningCert
>>>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>>>> 2017-10-23 00:05:28 [438] Converted
private key
>>>>>>>>>>>> 'ocspSigningCert
>>>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>>>> 2017-10-23 00:05:28 [439] Located the
certificate
>>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca".
>>>>>>>>>>>> 2017-10-23 00:05:28 [440] 0x1d
Certificate named
>>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca" in token "NSS
Certificate DB" in database
>>>>>>>>>>>> "/etc/pki/pki-tomcat/alias"
will not be valid after
>>>>>>>>>>>> 20171025122401.
>>>>>>>>>>>> 2017-10-23 00:05:28 [442] Located the key
'ocspSigningCert
>>>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>>>> 2017-10-23 00:05:28 [442] Converted
private key
>>>>>>>>>>>> 'ocspSigningCert
>>>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>>>> 2017-10-23 00:05:28 [443] Located the
certificate
>>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca".
>>>>>>>>>>>> 2017-10-23 00:05:28 [444] Located the key
'ocspSigningCert
>>>>>>>>>>>> cert-pki-ca'.
>>>>>>>>>>>> 2017-10-23 00:05:28 [444] Converted
private key
>>>>>>>>>>>> 'ocspSigningCert
>>>>>>>>>>>> cert-pki-ca' to public key.
>>>>>>>>>>>> 2017-10-23 00:05:39 [581] Found a
certificate with the same
>>>>>>>>>>>> nickname but
>>>>>>>>>>>> different subject, removing certificate
"ocspSigningCert
>>>>>>>>>>>> cert-pki-ca"
>>>>>>>>>>>> with subject "CN=OCSP
Subsystem,O=<REALM>".
>>>>>>>>>>>> 2017-10-23 00:05:39 [581] Imported
certificate "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca", got nickname
"ocspSigningCert cert-pki-ca".
>>>>>>>>>>>> 2017-10-23 00:05:39 [583] Located the
certificate
>>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca".
>>>>>>>>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>>>>>>>>>
"/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca"" (0).
>>>>>>>>>>>> 2017-10-23 00:10:43 [942] 0x1d
Certificate named
>>>>>>>>>>>> "ocspSigningCert
>>>>>>>>>>>> cert-pki-ca" in token "NSS
Certificate DB" in database
>>>>>>>>>>>> "/etc/pki/pki-tomcat/alias"
issued by CA and saved.
>>>>>>>>>>>>
>>>>>>>>>>>> I now have a date valid
ocspSigningCertificate, but with the
>>>>>>>>>>>> wrong
>>>>>>>>>>>> subject, and a broken certificate system
which will no longer
>>>>>>>>>>>> start.
>>>>>>>>>>>>
>>>>>>>>>>>> ipactl status
>>>>>>>>>>>> ...
>>>>>>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>>>>>>
>>>>>>>>>>>> I can't renew other expired
certificates.
>>>>>>>>>>>>
>>>>>>>>>>>> I also note that there is now no key for
>>>>>>>>>>>> ocspSigningCertificate as
>>>>>>>>>>>> shown
>>>>>>>>>>>> by:
>>>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>>>>
>>>>>>>>>>>> I wonder if this is because the
certificate subject changed?
>>>>>>>>>>>> There
>>>>>>>>>>>> was a
>>>>>>>>>>>> key before the certificate renewed.
>>>>>>>>>>>>
>>>>>>>>>>>> The ca debug logs are showing:
>>>>>>>>>>>>
>>>>>>>>>>>>
[23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by
>>>>>>>>>>>> nickname:
>>>>>>>>>>>> 'ocspSigningCert cert-pki-ca'
with serial number: 268370108
>>>>>>>>>>>>
[23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>>>>>>>>>> x509CertImpl
>>>>>>>>>>>>
[23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>>>>>>>>>> Certificate
>>>>>>>>>>>> object not found
>>>>>>>>>>>> Certificate object not found
>>>>>>>>>>>> at
>>>>>>>>>>>>
com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>>>>>>>>
>>>>>>>>>>>> Any help in repairing my broken ipa
system will be much
>>>>>>>>>>>> appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Sorry for the delay. I think my previous
reading of the
>>>>>>>>>>> certmonger
>>>>>>>>>>> csrgen code was wrong.
>>>>>>>>>>>
>>>>>>>>>>> IIRC in your certmonger entry the
cert_subject has the hostname
>>>>>>>>>>> value
>>>>>>>>>>> right? Do you also have cert_subject_der?
>>>>>>>>>>>
>>>>>>>>>>> You can decode that by:
>>>>>>>>>>>
>>>>>>>>>>> 1. Running a hex-to-bin, something hacky like
this in python:
>>>>>>>>>>>
>>>>>>>>>>> import binascii
>>>>>>>>>>>
>>>>>>>>>>> hex_string = "<hex value>"
>>>>>>>>>>>
>>>>>>>>>>> binary_string =
binascii.unhexlify(hex_string)
>>>>>>>>>>>
>>>>>>>>>>> fd = open("out", "wb")
>>>>>>>>>>> fd.write(binary_string)
>>>>>>>>>>> fd.close()
>>>>>>>>>>>
>>>>>>>>>>> 2. Run: openssl asn1parse -in out -inform
der
>>>>>>>>>>>
>>>>>>>>>>> I'm going to assume this also has the
hostname encoded in it.
>>>>>>>>>>
>>>>>>>>>> Hi Again
>>>>>>>>>>
>>>>>>>>>> Yes, correct. The cert_subject_der does have the
bad CN with the
>>>>>>>>>> hostname encoded in it.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Can you try this:
>>>>>>>>>>>
>>>>>>>>>>> 1. Make a backup of the request file (just in
case) > 2. Remove
>>>>>>>>>>> cert_subject_der
>>>>>>>>>>> 3. Modify cert_subject to be CN=OCSP
Subsystem,O=<YOUR_REALM>
>>>>>>>>>>> 3. Try the renewal again
>>>>>>>>>>
>>>>>>>>>> I ran though all of this, checking that the
request file was
>>>>>>>>>> still
>>>>>>>>>> what
>>>>>>>>>> we wanted after restarting certmonger with
verbose logging,
>>>>>>>>>> restoring
>>>>>>>>>> all the databases, fixing permissions, checking
keys etc., and
>>>>>>>>>> ran the
>>>>>>>>>> getcert resubmit.
>>>>>>>>>>
>>>>>>>>>> It still generates a certificate with the bad CN
including a
>>>>>>>>>> hostname.
>>>>>>>>>>
>>>>>>>>>> Then the pki-tomcat server fails to start again.
>>>>>>>>>>
>>>>>>>>>> Other things I have checked are:
>>>>>>>>>> 1) The csr= field in the (modified) request file
has the correct
>>>>>>>>>> subject.
>>>>>>>>>>
>>>>>>>>>> 2) The dogtag ca debug log file is showing:
>>>>>>>>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01
BST 2017
>>>>>>>>>> processRenewal: orig subj dn =CN=OCSP
Subsystem,O=AST.CAM.AC.UK
>>>>>>>>>> ...
>>>>>>>>>> RenewalSubmitter: renewal original
profileId=caIPAserviceCert
>>>>>>>>>> RenewalSubmitter: for renewal, original
authenticator raCertAuth
>>>>>>>>>> found
>>>>>>>>>> CertProcessor: No authenticator credentials
required
>>>>>>>>>> processRenewal: set original Inputs into profile
Context
>>>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting
input name=
>>>>>>>>>> cert_request_type
>>>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting
value in
>>>>>>>>>> ctx:pkcs10
>>>>>>>>>> RenewalSubmitter: setInputsIntoContext() getting
input name=
>>>>>>>>>> cert_request
>>>>>>>>>> RenewalSubmitter: setInputsIntoContext() setting
value in
>>>>>>>>>> ctx:<hex
>>>>>>>>>> encoded csr>
>>>>>>>>>> ...
>>>>>>>>>> Finish parsePKCS10 -
CN=<HOSTNAME>,O=<REALM>
>>>>>>>>>>
>>>>>>>>>> The hex encoded csr in the last line decodes to
have the bad
>>>>>>>>>> subject
>>>>>>>>>> where the hostname is one of our other ipa
servers.
>>>>>>>>>>
>>>>>>>>>> The certificate always getthe same hostname in
the bad subject
>>>>>>>>>> cn.
>>>>>>>>>>
>>>>>>>>>> Do you have any other ideas of things to check to
find out what's
>>>>>>>>>> generating the bad subject?
>>>>>>>>>
>>>>>>>>> The order certmonger uses for the subject is:
>>>>>>>>>
>>>>>>>>> 1. cert_subject_der
>>>>>>>>> 2. If there is no DER, try to encode cert_subject
>>>>>>>>> 3. If that fails try again a different way
>>>>>>>>> 4. If it fails yet again use CN=localhost
>>>>>>>>>
>>>>>>>>> It is baffling that it would pick a hostname much
less one that
>>>>>>>>> certmonger shouldn't even know about.
>>>>>>>>
>>>>>>>> Indeed, and if I try to renew other certificates for some
of
>>>>>>>> them it
>>>>>>>> chooses other host names that should not be known about.
Each
>>>>>>>> certificate seems to get the same bad hostname each time
I try to
>>>>>>>> renew.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I assume the CSR found in the dogtag logs matches the
csr value
>>>>>>>>> in the
>>>>>>>>> certmonger request?
>>>>>>>>
>>>>>>>> No, its different. The issued certificate matches the csr
seen in
>>>>>>>> dogtag
>>>>>>>> which makes sense. But the csr in the dogtag logs has the
bad
>>>>>>>> subject.
>>>>>>>> The csr in the request directory file has the good
subject.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you share the certmonger request file privately?
>>>>>>>>
>>>>>>>> Yes, certainly.
>>>>>>>>
>>>>>>>> Thanks for your continued help on this.
>>>>>>>
>>>>>>> Let's try this. We'll drop the current request and
create a new one.
>>>>>>>
>>>>>>> Stop tomcat, restore the old cert database, start tomcat,
then:
>>>>>>>
>>>>>>> # getcert stop-tracking -i <request id>
>>>>>>> # getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
>>>>>>> "ocspSigningCert cert-pki-ca" -d
/etc/pki/pki-tomcat/alias -P
>>>>>>> <pin> -B
>>>>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert
>>>>>>> cert-pki-ca"'
>>>>>>>
>>>>>>> Then try resubmitting the request.
>>>>>>>
>>>>>> Hi Rob
>>>>>>
>>>>>> When you say 'stop tomcat', I'm just doing an ipactl
stop. That is
>>>>>> stopiing the ipa pki-tomcat service. Is that good enough or are
there
>>>>>> other services that need stopping too?
>>>>>>
>>>>>> I tried the stop-tracking/start-tracking. Exactly the same result
as
>>>>>> before. Same hostname appears in the subject CN field of the new
>>>>>> certificate and then the pki-tomcat service won't start.
>>>>>>
>>>>>> I've also been back and redone the manual submits changing
the
>>>>>> point at
>>>>>> which I copied in the edited request as I wondered if there was
some
>>>>>> caching of csr's going on in certmonger and it was
remembering a
>>>>>> previous request it had read on startup before I replaced the
>>>>>> request file.
>>>>>>
>>>>>> But nothing seems to stop dogtag issuing a certificate with the
>>>>>> Subject
>>>>>> CN being that host name.
>>>>>>
>>>>>> Is it possible that dogtag is somehow overriding the Subject its
>>>>>> being
>>>>>> explicitly given?
>>>>>>
>>>>>> FWIW the CSR in the dogtag debug log is always identical, but I
guess
>>>>>> thats expected if the CSR information is always the same.
>>>>>>
>>>>>> I'm not sure if its possible to increase the verbosity on the
dogtag
>>>>>> side. Can you advise please?
>>>>>>
>>>>>> I've been assuming that its the bad subject that is
preventing the
>>>>>> pki-tomcat from starting after the new certificate is issued.
Does
>>>>>> that
>>>>>> make sense to you?
>>>>>>
>>>>>> I wonder where we can go from here? There must be a good reason
for
>>>>>> whats happening!
>>>>>
>>>>> Let's step back and gather some more information.
>>>>>
>>>>> Can you restore the files again and send me the output of:
>>>>>
>>>>> # getcert list
>>>>>
>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias/ -n
"ocspSigningCert
>>>>> cert-pki-ca"
>>>>
>>>> Hi Rob
>>>>
>>>> You should have the output you requested by private email now.
>>>>
>>>>>
>>>>> And the exact commands you are using to do all this, including
>>>>> re-issuing the cert
>>>>
>>>> Thats a great idea. Maybe I'm just doing something in the wrong
order.
>>>>
>>>>>
>>>>> Using ipactl is fine. You just don't want to touch the NSS
databases
>>>>> while the service is running. Similarly you probably don't want
to
>>>>> mess
>>>>> with certmonger requests while it is running as it won't see the
>>>>> changes
>>>>> until it restarts.
>>>> Here is the procedure I'm using to renew the certificates. One of
the
>>>> tricky things is to not have the certificate system working when
>>>> certmonger is started so that it doesn't renew a certificate on
>>>> startup that might not be one I have changed the request file for
>>>> (other certificates have similar problems with bad subject CN and stop
>>>> the certificate system.)
>>>>
>>>> I've found it convenient to keep reinitializing the log files
because
>>>> they get very confusing when the clock keeps getting set back.
>>>>
>>>> The request number has changed since we did the stop-tracking /
>>>> start-tracking.
>>>>
>>>> Roderick
>>>>
>>>> ipactl stop
>>>>
>>>> # Set clock back:
>>>> systemctl stop ntpd
>>>> timedatectl set-time "2017-10-23 00:00:00"
>>>> date
>>>>
>>>> #Stop certificate renewals the systemd way
>>>> systemctl stop certmonger
>>>>
>>>> # Sort out certmonger log
>>>> mv /var/log/certmonger.log /var/log/certmonger.log-$(date
>>>> +%Y%m%d:%H:%M)
>>>>
>>>> # ****** Copy in the fixed certmonger request
>>>> \cp /root/20161124081302.ocsp_backup
>>>> /var/lib/certmonger/requests/20161124081302
>>>>
>>>>
>>>> # In another window
>>>> certmonger -n -d 9 2>&1 | tee /var/log/certmonger.log
>>>>
>>>> # In the original window
>>>> # Sort out dogtag log
>>>> mv /var/log/pki/pki-tomcat/ca/debug
>>>> /var/log/pki/pki-tomcat/ca/debug-$(date +%Y%m%d:%H:%M).log
>>>> touch /var/log/pki/pki-tomcat/ca/debug
>>>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/debug
>>>>
>>>> mv /var/log/pki/pki-tomcat/ca/selftests.log
>>>> /var/log/pki/pki-tomcat/ca/selftests.log-$(date +%Y%m%d:%H:%M)
>>>> touch /var/log/pki/pki-tomcat/ca/selftests.log
>>>> chown pkiuser:pkiuser /var/log/pki/pki-tomcat/ca/selftests.log
>>>>
>>>> \cp /var/tmp/rmj/etc/httpd/alias/cert8.db /etc/httpd/alias/cert8.db
>>>> \cp /var/tmp/rmj/etc/httpd/alias/key3.db /etc/httpd/alias/key3.db
>>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/cert8.db
>>>> /etc/pki/pki-tomcat/alias/cert8.db
>>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/alias/key3.db
>>>> /etc/pki/pki-tomcat/alias/key3.db
>>>>
>>>> \cp /var/tmp/rmj/etc/pki/pki-tomcat/ca/CS.cfg
>>>> /etc/pki/pki-tomcat/ca/CS.cfg
>>>>
>>>> # Fix up certificate permissions and check all is ok
>>>> certutil -M -n "caSigningCert cert-pki-ca" -d
>>>> /etc/pki/pki-tomcat/alias -t CTu,Cu,Cu
>>>> certutil -L -d /etc/pki/pki-tomcat/alias
>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>> <pin>
>>>>
>>>> # Fix dogtag so it starts
>>>> pki-server subsystem-enable ca >
>>>> # Start IdM services
>>>>
>>>> ipactl start
>>>> ipactl status
>>>>
>>>> #Resubmit certificate ocsp
>>>> date;getcert resubmit -i 20161124081302
>>>
>>> Hi Rob
>>>
>>> Does it look like there is anything I should be doing differently in
>>> this command sequence to try to get the certificate renewed correctly?
>>>
>>> I'm starting to wonder whether I should look at alternative strategies
>>> for recovering the freeipa servers if the certificates won't renew
>>> correctly. Do you have any ideas on this please?
>>>
>>
>> This looks reasonable. Have you tried stop-tracking and start-tracking I
>> suggested?
>
> Yes,I tried the stop-tracking / start-tracking. Same result, bad Subject
> CN.
>
>>
>> I'm still baffled how the name(s) are getting mixed up.
>
> Indeed, something is going round changing the subject. In the dogtag
> debug log is:
> [23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal:
> origNotAfter =Wed Oct 25 13:24:01 BST 2017
> [23/Oct/2017:00:13:52][http-bio-8080-exec-9]: processRenewal: orig subj
> dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>
> and then very shortly afterwards there is a csr listed that decodes to
> have the bad Subject CN.
>
> The listed csr doesn't match any I have been able to find on the system,
> so something must be going round and changing it.
>
>>
>> Something else you might want to try would be to move all the other
>> requests out of the way in case there is some sort of memory corruption
>> causing the hostname to get in there somehow.
>
> Thanks for the suggestion which I have now tried. Same result I'm afraid.
>
> I've now been battling this issue since the beginning of November.
>
> I'm wondering whether, just to get our IPA system fully operational
> again, whether we might be able to add the correct CN into a Subject
> Alternative name?
>
> Does this sound like a possible way to at least get all the IPA services
> started properly?
>
> Do you think it would cause further problems down the line?
>
> Would you be able to advise how to get the certificates to have the
> correct CN in the Subject Alternative Name.
>
> Maybe there are some other tricks we could use just to get going again.
>
> Can you advise please?
>
I'm really stumped here.
Do you have the CA installed on any other masters? You might try setting
it as the renewal master and trying the renewals there instead.
Hi Rob
Thanks for all your thoughts on this.
Some of the certificates have renewed automatically (but incorrectly) on
the other masters. I have two other replicas, but the same sort of thing
happens.
At the moment I have an RHEL 7.3 replica with:
auditSigningCert renewed but with a bad subject CN
ocspSigningCert expired
subsystemCert expired
ipaCert renewed but with bad subject CN
and an RHEL 7.4 replica with:
auditSigningCert renewed but with a same bad subject CN as RHEL 7.3 replica
ocspSigningCert expired
subsystemCert expired
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem', which I
think is the equivalent of ipaCert, renewed but with the same bad
subject CN as RHEL 7.3 replica
All the other certificates seem to have plausible Subjects and are valid.
I'm not sure that this this helps me much though.
I wondered whether the CSR is getting the bad subject because its
accessing a wrong certificate profile. Its almost like its trying to get
an IPA service certificate, but I can't see how a certificate is tied to
a profile.
For the oscpSigningCert I can see in the dogtag access logs on the host
we have been using all along:
"GET
/ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=2&renewal=true&xml=true
HTTP/1.1" 200 139
"GET /ca/agent/ca/profileReview?requestId=9997550&xml=true HTTP/1.1" 200
14421
"GET
/ca/agent/ca/profileProcess?requestId=9997550&xml=true&op=approve&name=CN%3D<fqdn
of host3>%2CO%3D<REALM>
Anyway, I clearly need to trace the certificate renewal process in more
detail. In the certmonger logs it just says:
Request2('20161124081302') moved to state 'GENERATING_CSR
and doesn't give any clue as to how its doing this.
Do you think there is anyone else who might be able to help with this?
Thanks.
Roderick
4:05 p.m.
I know this is an old thread but I'm just posting this for someone who comes along the
same issue like me...
In order to fix my problem I had to do the following to fix for example the
'ocspSigningCert cert-pki-ca' certificate renewing with wrong subjects:
Find the Serial number for that certificate:
#certutil -L -d /etc/pki/pki-tomcat/alias -n "ocspSigningCert cert-pki-ca" |
grep Serial
Get the reqeustID:
#ldapsearch -D "cn=Directory Manager" -W -s sub -b
cn={SERIALNUMBER},ou=certificateRepository,ou=ca,o=ipaca "metaInfo"
Get the request data:
#ldapsearch -D "cn=Directory Manager" -W -s sub -b
cn={REQUESTID},ou=ca,ou=requests,o=ipaca
If the request data does not match the current certificate, we need to find one which
should be used instead.
#certutil -L -d /etc/pki/pki-tomcat/alias -n "ocspSigningCert cert-pki-ca" |
grep Subject
#ldapsearch -D "cn=Directory Manager" -W -s sub -b ou=ca,ou=requests,o=ipaca
"extdata-req--005fsubject--005fname--002ecn={SUBJECT}"
If we have multiple results check the one which has the right attributes set comparing to
a different system. Once you know which request to use change the requestid in the
certificateRepository to the one selected. I used ldapadmin to connect to change but the
ldapmodify should also work.
Hope this helps someone in the future...
2112
days inactive
2330
days old
freeipa-users@lists.fedorahosted.org
21 comments
3 participants
participants (3)
-
Jakob Ackermann -
Rob Crittenden -
Roderick Johnstone