502 Server Error: Proxy Error when creating CA replica on RockyLinux 8.9

[Satish Patel]

Folks,

I have Master freeIPA running on CentOS 7 and now trying to migrate it to
RockyLinux 8.9 (because centos7 is EOL).

When I am running  # ipa-replica-install --setup-ca I encounter following
error

Custodia uses 'ldap-vx-010101-4.site5.example.com' as master peer.
Configuring ipa-custodia
  [1/4]: Generating ipa-custodia config file
  [2/4]: Generating ipa-custodia keys
  [3/4]: starting ipa-custodia
  [4/4]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

502 Server Error: Proxy Error for url:
https://ldap-vx-010101-4.site5.example.com/ipa/keys/ca/caSigningCert%20ce...

I did google and found a similar issue but no solutions. Any idea what
could be wrong here? I have checked and all certs are updated and not
expired.

Above error isn't great to understand what is going on. I am able to use
curls etc. That means cert is updated and valid.

[Florence Blanc-Renaud]

Hi,


On Thu, May 16, 2024 at 4:42 AM Satish Patel via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Folks,
>
>  I have Master freeIPA running on CentOS 7 and now trying to migrate it to
>  RockyLinux 8.9 (because centos7 is EOL).
>
>  When I am running  # ipa-replica-install --setup-ca I encounter following
>  error
>
>  Custodia uses 'ldap-vx-010101-4.site5.example.com' as master peer.
>  
Is the above node running the CA instance? You can check with
# ipa config-show | grep CA
  IPA CA servers: server.ipa.test
  IPA CA renewal master: server.ipa.test

Then on this "master peer" machine, check that the custodia service is able
to find all the keys:
# /usr/libexec/ipa/ipa-custodia-check  `hostname`

I would also check the redirection for ipa/keys that should be defined in
/etc/httpd/conf.d/ipa.conf. You should see lines similar to the following
on the "master peer":
# Custodia stuff is redirected to the custodia daemon
# after authentication
<Location "/ipa/keys/">
    ProxyPass "unix:/run/httpd/ipa-custodia.sock|http://localhost/keys/"
    RequestHeader set GSS_NAME %{GSS_NAME}s
    RequestHeader set REMOTE_USER %{REMOTE_USER}s
</Location>

And check that the custodia service is running on this "master peer":
# systemctl status ipa-custodia

flo

Configuring ipa-custodia
>    [1/4]: Generating ipa-custodia config file
>    [2/4]: Generating ipa-custodia keys
>    [3/4]: starting ipa-custodia
>    [4/4]: configuring ipa-custodia to start on boot
>  Done configuring ipa-custodia.
>  Your system may be partly configured.
>  Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
>  502 Server Error: Proxy Error for url:
>
> https://ldap-vx-010101-4.site5.example.com/ipa/keys/ca/caSigningCert%20ce...
>
>  I did google and found a similar issue but no solutions. Any idea what
>  could be wrong here? I have checked and all certs are updated and not
>  expired.
>
>  Above error isn't great to understand what is going on. I am able to use
>  curls etc. That means cert is updated and valid.
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Satish Patel]

Hi Florence,

Thanks for the reply, the following is my output of all the commands you
provided. look like something is wrong on custodia self-check command.

[root@ldap-vx-010101-4 ~]# ipa config-show | grep CA
  IPA CA servers: ldap-vx-010101-1.site5.example.com,
ldap-vx-010101-4.site5.example.com
  IPA CA renewal master: ldap-vx-010101-1.site5.example.com




[root@ldap-vx-010101-4 ~]# /usr/libexec/ipa/ipa-custodia-check  `hostname`
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Platform:
Linux-3.10.0-514.el7.x86_64-x86_64-with-centos-7.3.1611-Core
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: IPA version: 4.6.5
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: IPA vendor version:
4.6.5-11.el7.centos
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Realm: example.COM
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Host:
ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Remote server:
ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:31 ipa-custodia-tester] <WARNING>: Performing self-test
only.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: File
'/etc/ipa/default.conf' exists.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: File '/etc/krb5.keytab'
exists.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: File
'/etc/ipa/custodia/custodia.conf' exists.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: File
'/etc/ipa/custodia/server.keys' exists.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Custodia client created.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Loaded key for usage
'sig' from '/etc/ipa/custodia/server.keys'.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: JWK KID matches host's
service principal name &#39;host/ldap-vx-010101-4.site5.example.com(a)example.COM
'.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Checked host LDAP keys
&#39;host/ldap-vx-010101-4.site5.example.com(a)example.COM&#39; for usage sig.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Local key for usage 'sig'
matches key in LDAP.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Checked server LDAP keys
&#39;host/ldap-vx-010101-4.site5.example.com(a)example.COM&#39; for usage sig.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Loaded key for usage
'enc' from '/etc/ipa/custodia/server.keys'.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: JWK KID matches host's
service principal name &#39;host/ldap-vx-010101-4.site5.example.com(a)example.COM
'.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Checked host LDAP keys
&#39;host/ldap-vx-010101-4.site5.example.com(a)example.COM&#39; for usage enc.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Local key for usage 'enc'
matches key in LDAP.
[2024-05-16T11:42:31 ipa-custodia-tester] <INFO>: Checked server LDAP keys
&#39;host/ldap-vx-010101-4.site5.example.com(a)example.COM&#39; for usage enc.
[2024-05-16T11:42:31 requests.packages.urllib3.connectionpool] <INFO>:
Starting new HTTPS connection (1): ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:31 ipa-custodia-tester] <ERROR>: Failed to retrieve key
'dm/DMHash': 502 Server Error: Proxy Error.
[2024-05-16T11:42:31 requests.packages.urllib3.connectionpool] <INFO>:
Starting new HTTPS connection (1): ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:31 ipa-custodia-tester] <ERROR>: Failed to retrieve key
'ra/ipaCert': 502 Server Error: Proxy Error.
[2024-05-16T11:42:31 requests.packages.urllib3.connectionpool] <INFO>:
Starting new HTTPS connection (1): ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:33 ipa-custodia-tester] <ERROR>: Failed to retrieve key
'ca/auditSigningCert cert-pki-ca': 502 Server Error: Proxy Error.
[2024-05-16T11:42:33 requests.packages.urllib3.connectionpool] <INFO>:
Starting new HTTPS connection (1): ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:34 ipa-custodia-tester] <ERROR>: Failed to retrieve key
'ca/caSigningCert cert-pki-ca': 502 Server Error: Proxy Error.
[2024-05-16T11:42:34 requests.packages.urllib3.connectionpool] <INFO>:
Starting new HTTPS connection (1): ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:36 ipa-custodia-tester] <ERROR>: Failed to retrieve key
'ca/ocspSigningCert cert-pki-ca': 502 Server Error: Proxy Error.
[2024-05-16T11:42:36 requests.packages.urllib3.connectionpool] <INFO>:
Starting new HTTPS connection (1): ldap-vx-010101-4.site5.example.com
[2024-05-16T11:42:37 ipa-custodia-tester] <ERROR>: Failed to retrieve key
'ca/subsystemCert cert-pki-ca': 502 Server Error: Proxy Error.
[ERROR] One or more tests have failed.





# Custodia stuff is redirected to the custodia daemon
# after authentication
<Location "/ipa/keys/">
    ProxyPass "unix:/run/httpd/ipa-custodia.sock|http://localhost/keys/"
    RequestHeader set GSS_NAME %{GSS_NAME}s
    RequestHeader set REMOTE_USER %{REMOTE_USER}s
</Location>




[root@ldap-vx-010101-4 ~]# systemctl status ipa-custodia
● ipa-custodia.service - IPA Custodia Service
   Loaded: loaded (/usr/lib/systemd/system/ipa-custodia.service; disabled;
vendor preset: disabled)
   Active: active (running) since Fri 2024-05-10 20:13:53 UTC; 5 days ago
 Main PID: 16656 (ipa-custodia)
   CGroup: /system.slice/ipa-custodia.service
           └─16656 /usr/bin/python2 /usr/libexec/ipa/ipa-custodia
/etc/ipa/custodia/custodia.conf

Warning: Journal has been rotated since unit was started. Log output is
incomplete or unavailable.

On Thu, May 16, 2024 at 2:05 AM Florence Blanc-Renaud <flo(a)redhat.com&gt;
wrote:

>  Hi,
>
>
>  On Thu, May 16, 2024 at 4:42 AM Satish Patel via FreeIPA-users <
>  freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
> > Folks,
> >
> > I have Master freeIPA running on CentOS 7 and now trying to migrate it to
> > RockyLinux 8.9 (because centos7 is EOL).
> >
> > When I am running  # ipa-replica-install --setup-ca I encounter following
> > error
> >
> > Custodia uses 'ldap-vx-010101-4.site5.example.com' as master peer.
> >
>  Is the above node running the CA instance? You can check with
>  # ipa config-show | grep CA
>    IPA CA servers: server.ipa.test
>    IPA CA renewal master: server.ipa.test
>
>  Then on this "master peer" machine, check that the custodia service is
>  able to find all the keys:
>  # /usr/libexec/ipa/ipa-custodia-check  `hostname`
>
>  I would also check the redirection for ipa/keys that should be defined in
>  /etc/httpd/conf.d/ipa.conf. You should see lines similar to the following
>  on the "master peer":
>  # Custodia stuff is redirected to the custodia daemon
>  # after authentication
>  <Location "/ipa/keys/">
>      ProxyPass "unix:/run/httpd/ipa-custodia.sock|http://localhost/keys/"
>      RequestHeader set GSS_NAME %{GSS_NAME}s
>      RequestHeader set REMOTE_USER %{REMOTE_USER}s
>  </Location>
>
>  And check that the custodia service is running on this "master peer":
>  # systemctl status ipa-custodia
>
>  flo
>
>  Configuring ipa-custodia
> >   [1/4]: Generating ipa-custodia config file
> >   [2/4]: Generating ipa-custodia keys
> >   [3/4]: starting ipa-custodia
> >   [4/4]: configuring ipa-custodia to start on boot
> > Done configuring ipa-custodia.
> > Your system may be partly configured.
> > Run /usr/sbin/ipa-server-install --uninstall to clean up.
> >
> > 502 Server Error: Proxy Error for url:
> >
> https://ldap-vx-010101-4.site5.example.com/ipa/keys/ca/caSigningCert%20ce...
> >
> > I did google and found a similar issue but no solutions. Any idea what
> > could be wrong here? I have checked and all certs are updated and not
> > expired.
> >
> > Above error isn't great to understand what is going on. I am able to use
> > curls etc. That means cert is updated and valid.
> > --
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>