389-ds-base name log pipe problems

[Nyquist]

Hello

 

We are using 18 389-ds-base-1.3.10.2,

Recently, one server (A) was set up to log to a pipe.

 

Afterwards, a phenomenon occurs where all server connections are occupied by an external
system.

The external system has stopped.

 

After this incident, the servers returned to normal condition, but

Only server A had intermittent port 389 inaccessibility and delays.

After this, we determined that the problem on server A was due to logging in the pipe.

After changing the pipe to a file, I restarted the server.

After that the problem went away.

 

We want to know the exact cause of the delay and inaccessibility that occurred on server
A.

How can logging to pipe affect directory server port 389 access?

Could it be a file descriptor related issue? Why were the other 17 servers able to recover
to normal?

 

Thanks

[Thierry Bordaz]

Hi,

It would be helpful to have some details how you configured the log pipe 
and did the tests. I wonder if it could be related to 
https://github.com/389ds/389-ds-base/issues/198.

regards
thierry

On 12/7/23 09:06, Nyquist wrote:
>  Hello
>
>    
>
>  We are using 18 389-ds-base-1.3.10.2,
>
>  Recently, one server (A) was set up to log to a pipe.
>
>    
>
>  Afterwards, a phenomenon occurs where all server connections are occupied by an external
> system.
>
>  The external system has stopped.
>
>    
>
>  After this incident, the servers returned to normal condition, but
>
>  Only server A had intermittent port 389 inaccessibility and delays.
>
>  After this, we determined that the problem on server A was due to logging in the pipe.
>
>  After changing the pipe to a file, I restarted the server.
>
>  After that the problem went away.
>
>    
>
>  We want to know the exact cause of the delay and inaccessibility that occurred on server
> A.
>
>  How can logging to pipe affect directory server port 389 access?
>
>  Could it be a file descriptor related issue? Why were the other 17 servers able to
> recover to normal?
>
>    
>
>  Thanks
>  --
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Nyquist]

1. Stop directory server and remove access log file.
2. Run logpipe.py. Python create access.pipe (pipe)
3. Change directory server settings
nsslapd-accesslog-maxlogsperdir: 1
nsslapd-accesslog-logexpirationtime: -1
nsslapd-accesslog-logrotationtime: -1
nsslapd-accesslog: /var/log/dirsrv/slapd-localhost/access.pipe
nsslapd-accesslog-logbuffering: off
4. Run directory server
=> Check normal execution, logpipe.py and custom.py are operating normally.

The problem arises after this. This server has consumed all connections. A forced shutdown
of the external system returned the connection, and we expected it to work normally.
However, port 389 health check failed intermittently. Joining and removing from the L4
switch is repeated. Could this be caused by my logging settings to pipe?
Or should I think it's a problem with the Linux file system?

[Pierre Rogier]

Hi,

I see two possible reasons why 389ds could intermittently stop listening on
389 port:
  [1] nearly all connections are exhausted.
  [2] the server is blocked (probably trying to write in the pipe)

[1] is possible if most of the connections did not get released when the
external system was stopped. (It can happen if the ip connection gets
silently broken (typically by rebooting a switch) )
  In that case it is the fact of restarting 389ds that solved the issue.
[2] Would mean that the application that read the pipe was not able to read
the data fast enough. (Either too much data were logged or there was an
issue on that application)

Regards,
   Pierre

On Thu, Dec 7, 2023 at 11:25 AM Nyquist <dongminic(a)gmail.com&gt; wrote:

>  1. Stop directory server and remove access log file.
>  2. Run logpipe.py. Python create access.pipe (pipe)
>  3. Change directory server settings
>  nsslapd-accesslog-maxlogsperdir: 1
>  nsslapd-accesslog-logexpirationtime: -1
>  nsslapd-accesslog-logrotationtime: -1
>  nsslapd-accesslog: /var/log/dirsrv/slapd-localhost/access.pipe
>  nsslapd-accesslog-logbuffering: off
>  4. Run directory server
>  => Check normal execution, logpipe.py and custom.py are operating normally.
>
>  The problem arises after this. This server has consumed all connections. A
>  forced shutdown of the external system returned the connection, and we
>  expected it to work normally. However, port 389 health check failed
>  intermittently. Joining and removing from the L4 switch is repeated. Could
>  this be caused by my logging settings to pipe?
>  Or should I think it's a problem with the Linux file system?
>  --
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
--

389 Directory Server Development Team

[Nyquist]

Hello.

If a log cannot be written (lack of space in the case of a file, lack of reader in the
case of a pipe, etc.), does the directory server block the port? Will I be in a state
where I cannot perform operations such as BIND, SEARCH?

Thank you

[Nyquist]

As a result of the cause analysis, it was confirmed that the pipe reader process
performance was poor. However, what was more problematic than that was the log buffering
off option among the LDAP options. When buffering was not used, a block on port 389
occurred on all servers in response to large requests.
 Here's what I need to do: 
1. Increasing the performance of the pipe reader, 
2. If the log buffer is not used, measure the limitations of the LDAP server and reflect
them in my architecture.

Thanks

[Viktor Ashirov]

On Thu, Dec 7, 2023 at 11:25 AM Nyquist <dongminic(a)gmail.com&gt; wrote:

>  1. Stop directory server and remove access log file.
>  2. Run logpipe.py. Python create access.pipe (pipe)
>  3. Change directory server settings
>  nsslapd-accesslog-maxlogsperdir: 1
>  nsslapd-accesslog-logexpirationtime: -1
>  nsslapd-accesslog-logrotationtime: -1
>  nsslapd-accesslog: /var/log/dirsrv/slapd-localhost/access.pipe
>  nsslapd-accesslog-logbuffering: off
>  4. Run directory server
>  => Check normal execution, logpipe.py and custom.py are operating normally.
>
>  The problem arises after this. This server has consumed all connections. A
>  forced shutdown of the external system returned the connection, and we
>  expected it to work normally. However, port 389 health check failed
>  intermittently. Joining and removing from the L4 switch is repeated. Could
>  this be caused by my logging settings to pipe?
>  Or should I think it's a problem with the Linux file system?
>  
Judging by the version, it's a system with systemd, where logpipe was not
working properly and the server would refuse to start.
Some time ago I investigated issues with logpipe and systemd and wrote
these instructions to make it work:
https://bugzilla.redhat.com/show_bug.cgi?id=1520373#c5 (you can skip dsconf
part as you already did have this configuration in cn=config).
Could you please try this approach?

Thanks.

>  --
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Viktor