Another Cert Expiration Problem

[Russ Long]

I have a single-server IPA environment in my homelab.  I noticed today that I was unable to delete a host from IPA, and found that pki-tomcatd was down and unable to start.

I found that several certificates had expired for some reason.  I tried `ipa-cert-fix`, but that failed as pki-tomcat will not start.

I attempted to set the server date/time to a date 24 hours before the certificates expired, and was able to get tomcat to start, however the `ipa-cert-fix` now fails with this error:

CalledProcessError(Command ['pki-server', 'cert-fix', '--ldapi-socket', '/run/slapd-IPA-DOMAIN-CO.socket', '--agent-uid', 'ipara', '--cert', 'sslserver', '--cert', 'subsystem', '--cert', 'ca_ocsp_signing', '--cert', 'ca_audit_signing', '--extra-cert', '16'] returned non-zero exit status 1: "INFO: Loading instance type: pki-tomcatd\nINFO: Loading instance: pki-tomcat\nINFO: Loading global Tomcat config: /etc/tomcat/tomcat.conf\nINFO: Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf\nINFO: Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf\nINFO: Loading password config: /etc/pki/pki-tomcat/password.conf\nINFO: Loading subsystem config: /etc/pki/pki-tomcat/ca/CS.cfg\nINFO: Loading subsystem registry: /etc/pki/pki-tomcat/ca/registry.cfg\nINFO: Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat\nINFO: Fixing the following system certs: ['sslserver', 'subsystem', 'ca_ocsp_signing', 'ca_audit_signing']\nINFO: Renewing the following additional c
 erts: ['16']\nINFO: Stopping the instance to proceed with system cert renewal\nINFO: Configuring LDAP connection for CA\nINFO: Setting pkidbuser password via ldappasswd\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\n")

I reviewed the blog at https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... (Thanks Flo!) but was still unable to get anything working.  The Certificate password test fails with these errors:

[root@master ca]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.
[root@master ca]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'NSS Certificate DB: subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.

Any ideas what I can try?

[Florence Blanc-Renaud]

Hi,

On Fri, Sep 15, 2023 at 7:43 PM Russ Long via FreeIPA-users <
[e-mail]> wrote:
> I have a single-server IPA environment in my homelab.  I noticed today
> that I was unable to delete a host from IPA, and found that pki-tomcatd was
> down and unable to start.
>
> I found that several certificates had expired for some reason.  I tried
> `ipa-cert-fix`, but that failed as pki-tomcat will not start.
>
> I attempted to set the server date/time to a date 24 hours before the
> certificates expired, and was able to get tomcat to start, however the
> `ipa-cert-fix` now fails with this error:
>
> CalledProcessError(Command ['pki-server', 'cert-fix', '--ldapi-socket',
> '/run/slapd-IPA-DOMAIN-CO.socket', '--agent-uid', 'ipara', '--cert',
> 'sslserver', '--cert', 'subsystem', '--cert', 'ca_ocsp_signing', '--cert',
> 'ca_audit_signing', '--extra-cert', '16'] returned non-zero exit status 1:
> "INFO: Loading instance type: pki-tomcatd\nINFO: Loading instance:
> pki-tomcat\nINFO: Loading global Tomcat config:
> /etc/tomcat/tomcat.conf\nINFO: Loading PKI Tomcat config:
> /usr/share/pki/etc/tomcat.conf\nINFO: Loading instance Tomcat config:
> /etc/pki/pki-tomcat/tomcat.conf\nINFO: Loading password config:
> /etc/pki/pki-tomcat/password.conf\nINFO: Loading subsystem config:
> /etc/pki/pki-tomcat/ca/CS.cfg\nINFO: Loading subsystem registry:
> /etc/pki/pki-tomcat/ca/registry.cfg\nINFO: Loading instance registry:
> /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat\nINFO: Fixing the following
> system certs: ['sslserver', 'subsystem', 'ca_ocsp_signing',
> 'ca_audit_signing']\nINFO: Renewing the following additional c
>  erts: ['16']\nINFO: Stopping the instance to proceed with system cert
> renewal\nINFO: Configuring LDAP connection for CA\nINFO: Setting pkidbuser
> password via ldappasswd\nSASL/EXTERNAL authentication started\nSASL
> username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL
> SSF: 0\n")

mixing ipa-cert-fix method with the date manipulation often leads to more
issues if ipa-cert-fix was able to fix some of the certs but not all of
them (the first execution creates a cert valid from present date only, and
as soon as you go in the past this cert is not considered valid yet).

To provide any advice we would need to have an exact description of the
current situation. Can you provide the output of "getcert list" executed as
root? This will show the "valid from" and "valid to" dates for each
certificate. Is your system still in the past or did you move back to
current date?

I reviewed the blog at
> https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> (Thanks Flo!) but was still unable to get anything working.  The
> Certificate password test fails with these errors:
>
> [root@master ca]# certutil -K -d /etc/pki/pki-tomcat/alias -f
> /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
> certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
> Key and Certificate Services"
> certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library:
> invalid arguments.
> [root@master ca]# certutil -K -d /etc/pki/pki-tomcat/alias -f
> /tmp/pwdfile.txt -n 'NSS Certificate DB: subsystemCert cert-pki-ca'
> certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
> Key and Certificate Services"
> certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library:
> invalid arguments.
>
> If you run the same command without -n <alias>, you should be able to see

all the keys stored in the NSS database:
# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
Is there an entry for something like 'subsystemCert cert-pki-ca'?
flo

Any ideas what I can try?
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Russell Long]

Re-sending this as I forgot to send to the list itself, sorry.

On Mon, Sep 18, 2023 at 6:55 AM Florence Blanc-Renaud 
wrote:
> Hi,
>
> On Fri, Sep 15, 2023 at 7:43 PM Russ Long via FreeIPA-users <
> [e-mail]> wrote:
> > I have a single-server IPA environment in my homelab.  I noticed today
> > that I was unable to delete a host from IPA, and found that pki-tomcatd was
> > down and unable to start.
> >
> > I found that several certificates had expired for some reason.  I tried
> > `ipa-cert-fix`, but that failed as pki-tomcat will not start.
> >
> > I attempted to set the server date/time to a date 24 hours before the
> > certificates expired, and was able to get tomcat to start, however the
> > `ipa-cert-fix` now fails with this error:
> >
> > CalledProcessError(Command ['pki-server', 'cert-fix', '--ldapi-socket',
> > '/run/slapd-IPA-DOMAIN-CO.socket', '--agent-uid', 'ipara', '--cert',
> > 'sslserver', '--cert', 'subsystem', '--cert', 'ca_ocsp_signing', '--cert',
> > 'ca_audit_signing', '--extra-cert', '16'] returned non-zero exit status 1:
> > "INFO: Loading instance type: pki-tomcatd\nINFO: Loading instance:
> > pki-tomcat\nINFO: Loading global Tomcat config:
> > /etc/tomcat/tomcat.conf\nINFO: Loading PKI Tomcat config:
> > /usr/share/pki/etc/tomcat.conf\nINFO: Loading instance Tomcat config:
> > /etc/pki/pki-tomcat/tomcat.conf\nINFO: Loading password config:
> > /etc/pki/pki-tomcat/password.conf\nINFO: Loading subsystem config:
> > /etc/pki/pki-tomcat/ca/CS.cfg\nINFO: Loading subsystem registry:
> > /etc/pki/pki-tomcat/ca/registry.cfg\nINFO: Loading instance registry:
> > /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat\nINFO: Fixing the following
> > system certs: ['sslserver', 'subsystem', 'ca_ocsp_signing',
> > 'ca_audit_signing']\nINFO: Renewing the following additional c
> >  erts: ['16']\nINFO: Stopping the instance to proceed with system cert
> > renewal\nINFO: Configuring LDAP connection for CA\nINFO: Setting pkidbuser
> > password via ldappasswd\nSASL/EXTERNAL authentication started\nSASL
> > username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL
> > SSF: 0\n")
>
> mixing ipa-cert-fix method with the date manipulation often leads to more
> issues if ipa-cert-fix was able to fix some of the certs but not all of
> them (the first execution creates a cert valid from present date only, and
> as soon as you go in the past this cert is not considered valid yet).
>
> To provide any advice we would need to have an exact description of the
> current situation. Can you provide the output of "getcert list" executed as
> root? This will show the "valid from" and "valid to" dates for each
> certificate. Is your system still in the past or did you move back to
> current date?

Getcert list:
Number of certificates and requests being tracked: 7.
Request ID '20220906145805':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit: Couldn't connect
to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=CA Audit,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2021-09-06 12:07:45 EDT
expires: 2023-08-27 12:07:45 EDT
key usage: digitalSignature,nonRepudiation
profile: caSignedLogCert
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20220906145806':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit: Couldn't connect
to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=OCSP Subsystem,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2021-09-06 12:07:52 EDT
expires: 2023-08-27 12:07:52 EDT
eku: id-kp-OCSPSigning
profile: caOCSPCert
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20220906145807':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit: Couldn't connect
to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=CA Subsystem,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2021-09-06 12:07:43 EDT
expires: 2023-08-27 12:07:43 EDT
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-clientAuth
profile: caSubsystemCert
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20220906145808':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2019-10-15 12:07:28 EDT
expires: 2039-10-15 12:07:28 EDT
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
profile: caCACert
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20220906145809':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit: Couldn't connect
to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=master.ipa.example.co,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2021-09-06 12:07:42 EDT
expires: 2023-08-27 12:07:42 EDT
dns: master.ipa.example.co
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
profile: caServerCert
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20220906145810':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit: Couldn't connect
to server.
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=IPA RA,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2021-09-06 12:08:40 EDT
expires: 2023-08-27 12:08:40 EDT
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-clientAuth
profile: caSubsystemCert
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20220906145820':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=master.ipa.example.co,O=IPA.EXAMPLE.CO http://ipa.example.co/
subject: CN=master.ipa.example.co,O=IPA.EXAMPLE.CO http://ipa.example.co/
issued: 2023-08-31 10:10:23 EDT
expires: 2024-08-31 10:10:23 EDT
dns: master.ipa.example.co
principal name: krbtgt/[e-mail]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
certificate template/profile: KDCs_PKINIT_Certs
profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes

System is back to present day, but pki-tomcat will not start in present
day, so I can move back to the past.  I moved it back to present day as
most things still work.
> I reviewed the blog at
> > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> > (Thanks Flo!) but was still unable to get anything working.  The
> > Certificate password test fails with these errors:
> >
> > [root@master ca]# certutil -K -d /etc/pki/pki-tomcat/alias -f
> > /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
> > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
> > Key and Certificate Services"
> > certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library:
> > invalid arguments.
> > [root@master ca]# certutil -K -d /etc/pki/pki-tomcat/alias -f
> > /tmp/pwdfile.txt -n 'NSS Certificate DB: subsystemCert cert-pki-ca'
> > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
> > Key and Certificate Services"
> > certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library:
> > invalid arguments.
> >
> > If you run the same command without -n <alias>, you should be able to see
>
> all the keys stored in the NSS database:
> # certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
> Is there an entry for something like 'subsystemCert cert-pki-ca'?
> flo

Here's the certutil:
[root@master ~]# certutil -K -d /etc/pki/pki-tomcat/alias -f
/tmp/pwdfile.txt
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key
and Certificate Services"
< 0> rsa      redacted   NSS Certificate DB:Server-Cert cert-pki-ca
< 1> rsa      redacted   NSS Certificate DB:caSigningCert cert-pki-ca
< 2> rsa      redacted   NSS Certificate DB:ocspSigningCert cert-pki-ca
< 3> rsa      redacted   NSS Certificate DB:subsystemCert cert-pki-ca
< 4> rsa      redacted   NSS Certificate DB:auditSigningCert cert-pki-ca
 (Redactions are mine)
> Any ideas what I can try?
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to
> > [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue

[Russ Long]

Any other advice here? I have also tried setting system back to when certificates were valid, restarting certmonger and pki-tomcatd, and running getcert resubmit on the affected certs, this moves them to a "Monitoring" status, but they still never renew when in present day or when the system is back in time.

When the system is back in time to when certs are valid, if I startup certmonger in debug mode and submit the getcert resubmit, I get this:
2023-08-25 00:29:24 [106919] Certificate submission attempt complete.
2023-08-25 00:29:24 [106919] Child status = 2.
2023-08-25 00:29:24 [106919] Child output:
"Server at "http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit" replied: Request 1 - Server Internal Error
"
2023-08-25 00:29:24 [106919] Server at "http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit" replied: Request 1 - Server Internal Error
2023-08-25 00:29:24 [106919] Certificate not (yet?) issued.
2023-08-25 00:29:24 [106919] Request2('20230825040038') already had a certificate, going back to monitoring it
2023-08-25 00:29:24 [106919] Request2('20230825040038') moved to state 'MONITORING'
2023-08-25 00:29:24 [106919] Wrote to /var/lib/certmonger/requests/20230825040039
2023-08-25 00:29:24 [106919] Will revisit Request2('20230825040038') soonish.
2023-08-25 00:29:54 [106919] Will revisit Request2('20230825040038') in 41876 seconds.

Digging further on this, pki-tomcat logs show an LDAP error: 
2023-08-25 00:29:23 [http-nio-8080-exec-3] WARNING: Unable to update certificate request: Unable to modify LDAP record: Object class violation
Unable to modify LDAP record: Object class violation
    at com.netscape.cmscore.dbs.LDAPSession.modify(LDAPSession.java:276)
    at com.netscape.cmscore.request.RequestRepository.modifyRequest(RequestRepository.java:322)
    at com.netscape.cmscore.request.RequestRepository.updateRequest(RequestRepository.java:290)
    at com.netscape.cms.servlet.cert.CertProcessor.submitRequests(CertProcessor.java:323)
    at com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:207)
    at com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:97)
    at com.netscape.cms.servlet.profile.ProfileSubmitServlet.processEnrollment(ProfileSubmitServlet.java:278)
    at com.netscape.cms.servlet.profile.ProfileSubmitServlet.process(ProfileSubmitServlet.java:131)
    at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:487)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
    at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584)
    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:207)
    at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
    at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584)
    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
    at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
    at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:83)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: netscape.ldap.LDAPException: Object class violation (65); unknown object class "request"

at netscape.ldap.LDAPConnection.checkMsg(Unknown Source)
    at netscape.ldap.LDAPConnection.modify(Unknown Source)
    at netscape.ldap.LDAPConnection.modify(Unknown Source)
    at netscape.ldap.LDAPConnection.modify(Unknown Source)
    at netscape.ldap.LDAPConnection.modify(Unknown Source)
    at com.netscape.cmscore.dbs.LDAPSession.modify(LDAPSession.java:264)
    ... 54 more

I really have no idea where to go from here with this.

Thanks in advance, 
Russ

[Rob Crittenden]

Russ Long via FreeIPA-users wrote:
> Any other advice here? I have also tried setting system back to when certificates were valid, restarting certmonger and pki-tomcatd, and running getcert resubmit on the affected certs, this moves them to a "Monitoring" status, but they still never renew when in present day or when the system is back in time.
>
> When the system is back in time to when certs are valid, if I startup certmonger in debug mode and submit the getcert resubmit, I get this:
> 2023-08-25 00:29:24 [106919] Certificate submission attempt complete.
> 2023-08-25 00:29:24 [106919] Child status = 2.
> 2023-08-25 00:29:24 [106919] Child output:
> "Server at "http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit" replied: Request 1 - Server Internal Error
> "
> 2023-08-25 00:29:24 [106919] Server at "http://master.ipa.example.co:8080/ca/ee/ca/profileSubmit" replied: Request 1 - Server Internal Error
> 2023-08-25 00:29:24 [106919] Certificate not (yet?) issued.
> 2023-08-25 00:29:24 [106919] Request2('20230825040038') already had a certificate, going back to monitoring it
> 2023-08-25 00:29:24 [106919] Request2('20230825040038') moved to state 'MONITORING'
> 2023-08-25 00:29:24 [106919] Wrote to /var/lib/certmonger/requests/20230825040039
> 2023-08-25 00:29:24 [106919] Will revisit Request2('20230825040038') soonish.
> 2023-08-25 00:29:54 [106919] Will revisit Request2('20230825040038') in 41876 seconds.
>
> Digging further on this, pki-tomcat logs show an LDAP error: 
> 2023-08-25 00:29:23 [http-nio-8080-exec-3] WARNING: Unable to update certificate request: Unable to modify LDAP record: Object class violation
> Unable to modify LDAP record: Object class violation
>    at com.netscape.cmscore.dbs.LDAPSession.modify(LDAPSession.java:276)
>    at com.netscape.cmscore.request.RequestRepository.modifyRequest(RequestRepository.java:322)
>    at com.netscape.cmscore.request.RequestRepository.updateRequest(RequestRepository.java:290)
>    at com.netscape.cms.servlet.cert.CertProcessor.submitRequests(CertProcessor.java:323)
>    at com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:207)
>    at com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:97)
>    at com.netscape.cms.servlet.profile.ProfileSubmitServlet.processEnrollment(ProfileSubmitServlet.java:278)
>    at com.netscape.cms.servlet.profile.ProfileSubmitServlet.process(ProfileSubmitServlet.java:131)
>    at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:487)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
>    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>    at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
>    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
>    at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584)
>    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
>    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
>    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:207)
>    at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
>    at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
>    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
>    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
>    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>    at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
>    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
>    at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584)
>    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
>    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
>    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
>    at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
>    at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
>    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
>    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
>    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
>    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
>    at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:83)
>    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
>    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
>    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
>    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
>    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
>    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
>    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
>    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
>    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
>    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
>    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>    at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: netscape.ldap.LDAPException: Object class violation (65); unknown object class "request"
>
> at netscape.ldap.LDAPConnection.checkMsg(Unknown Source)
>    at netscape.ldap.LDAPConnection.modify(Unknown Source)
>    at netscape.ldap.LDAPConnection.modify(Unknown Source)
>    at netscape.ldap.LDAPConnection.modify(Unknown Source)
>    at netscape.ldap.LDAPConnection.modify(Unknown Source)
>    at com.netscape.cmscore.dbs.LDAPSession.modify(LDAPSession.java:264)
>    ... 54 more
>
> I really have no idea where to go from here with this.

It means you are missing at least one objectclass definition in schema
that the CA adds. How this can happen I have no idea.

You can add missing schema with:

ldapadd -c -D 'cn=directory manager' -W -f
/usr/share/pki/server/database/ds/schema.ldif

The -c means it will continue loading the ldif on errors (like the
schema already exists).

rob

[Russ Long]

Rob,

Thanks so much, running that command, and then the `ipa-cert-fix` with the server in current time appears to have fixed the issue. I did manually run a `getcert resubmit -i ID_HERE` for a couple certs that were still showing CA_UNREACHABLE in `getcert list`, but not sure if that was necessary or if I was just impatient.

--Russ