Slapi Entry duplication observation
by Mark Reynolds
As we've been discussing performance recently, I did a little checking
on how many times we duplicate a slapi_entry when doing a single
modification. I updated an entry like this (which is actually pretty
small):
dn: uid=mareynol,ou=people,dc=example,dc=com
uid: mareynol
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetOrgPerson
objectClass: ntUser
ntUserDeleteAccount: true
sn: Reynolds
givenName: Mark
initials: T
cn: Mark T. Reynolds
ntUserCodePage: 0
ntUserAcctExpires: 9223372036854775807
ntUserDomainId: Mareynol
ntUniqueId: 76ae38ecf89b884cbfe2b713082f16a3
So every time the server duplicates this particular entry it triggers
159 mallocs and 64 frees. The more attributes the entry has the more
more mallocs would be required to copy it. I found that when I just
change a single attribute in this entry that the server calls
slapi_entry_dup() 8 times! (that's over 1200 mallocs and 500 frees)
The entry is duplicated two times by the MEP plugin for just being
enabled out of the box. We also call backentry_dup() two times which is
even more expensive, but I don't think we can avoid that one as it's
needed for backend plugin transactions. Below are the stack traces from
each "duplication" in order. Surely I would hope we can reduce this
number of entry dups for a single modify operation ...
#0 slapi_entry_dup (e=0x7fc881864240) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8b062528b in internal_plugin_search_entry_callback
(e=0x7fc881864240, callback_data=0x7fc8855fcc20)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:531
#2 0x00007fc8b0624549 in internal_srch_entry_callback
(be=0x7fc8aee457e0, conn=0x0, op=0x7fc88263b400, e=0x7fc881864240)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:99
#3 0x00007fc8b063a053 in send_ldap_search_entry_ext (pb=0x7fc8825fe120,
e=0x7fc881864240, ectrls=0x0, attrs=0x0, attrsonly=0,
send_result=0, nentries=0, urls=0x0) at
../389-ds-base/ldap/servers/slapd/result.c:1507
#4 0x00007fc8b0639194 in send_ldap_search_entry (pb=0x7fc8825fe120,
e=0x7fc881864240, ectrls=0x0, attrs=0x0, attrsonly=0)
at ../389-ds-base/ldap/servers/slapd/result.c:1050
#5 0x00007fc8b060ded1 in send_entry (pb=0x7fc8825fe120,
e=0x7fc881864240, operation=0x7fc88263b400, attrs=0x0, attrsonly=0,
pnentries=0x7fc8855f855c) at
../389-ds-base/ldap/servers/slapd/opshared.c:1132
#6 0x00007fc8b060e497 in iterate (pb=0x7fc8825fe120, be=0x7fc8aee457e0,
send_result=1, pnentries=0x7fc8855f855c, pagesize=-1,
pr_statp=0x7fc8855f854c) at
../389-ds-base/ldap/servers/slapd/opshared.c:1463
#7 0x00007fc8b060e8a5 in send_results_ext (pb=0x7fc8825fe120,
send_result=1, nentries=0x7fc8855f855c, pagesize=-1,
pr_stat=0x7fc8855f854c) at
../389-ds-base/ldap/servers/slapd/opshared.c:1680
#8 0x00007fc8b060d444 in op_shared_search (pb=0x7fc8825fe120,
send_result=1) at ../389-ds-base/ldap/servers/slapd/opshared.c:875
#9 0x00007fc8b06258c8 in search_internal_callback_pb
(pb=0x7fc8825fe120, callback_data=0x7fc8855fcc20,
prc=0x7fc8b062534c <internal_plugin_result_callback>,
psec=0x7fc8b0625257 <internal_plugin_search_entry_callback>,
prec=0x7fc8b06252e2 <internal_plugin_search_referral_callback>) at
../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:730
#10 0x00007fc8b062542e in search_internal_pb (pb=0x7fc8825fe120) at
../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:594
#11 0x00007fc8b06251ca in slapi_search_internal_pb (pb=0x7fc8825fe120)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:504
#12 0x00007fc8b0625cb8 in slapi_search_internal_get_entry
(dn=0x7fc8855fccf0, attrs=0x0, ret_entry=0x7fc8855fccd8,
component_identity=0x7fc8ac713320) at
../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:864
#13 0x00007fc8b0632eea in get_entry (pb=0x7fc8825fe000,
dn=0x7fc8abe82fa0 "uid=mareynol,ou=people,dc=example,dc=com")
at ../389-ds-base/ldap/servers/slapd/pw_retry.c:210
#14 0x00007fc8b060405f in op_shared_modify (pb=0x7fc8825fe000,
pw_change=0, old_pw=0x0)
at ../389-ds-base/ldap/servers/slapd/modify.c:726
#15 0x00007fc8b06033bb in do_modify (pb=0x7fc8825fe000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#16 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acf6c0, op=0x7fc8ac85a000, pb=0x7fc8825fe000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#17 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#18 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#19 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#20 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc880c68000) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8b0625d24 in slapi_search_internal_get_entry
(dn=0x7fc8825facf0, attrs=0x0, ret_entry=0x7fc8825facd8,
component_identity=0x7fc8ac713320)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:873
#2 0x00007fc8b0632eea in get_entry (pb=0x7fc880bff000,
dn=0x7fc880c2c320 "uid=mareynol,ou=people,dc=example,dc=com")
at ../389-ds-base/ldap/servers/slapd/pw_retry.c:210
#3 0x00007fc8b060405f in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:726
#4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#5 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#6 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc882667180) at
../389-ds-base/ldap/servers/slapd/entry.c:2057
#1 0x00007fc8ac53b501 in ldbm_back_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:614
#2 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021
#3 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#4 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#5 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#6 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#7 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#8 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc882667180) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8ac4fe456 in backentry_dup (e=0x7fc88267ebc0) at
../389-ds-base/ldap/servers/slapd/back-ldbm/backentry.c:86
#2 0x00007fc8ac53b572 in ldbm_back_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:622
#3 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021
#4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#5 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#6 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc880c68180) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8ac4fe456 in backentry_dup (e=0x7fc880c7d000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/backentry.c:86
#2 0x00007fc8ac53b846 in ldbm_back_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:680
#3 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021
#4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#5 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#6 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc882667180) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8b062528b in internal_plugin_search_entry_callback
(e=0x7fc882667180, callback_data=0x7fc8825fa9b0)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:531
#2 0x00007fc8b0624549 in internal_srch_entry_callback
(be=0x7fc8aee457e0, conn=0x0, op=0x7fc880c3c400, e=0x7fc882667180)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:99
#3 0x00007fc8b063a053 in send_ldap_search_entry_ext (pb=0x7fc880bff120,
e=0x7fc882667180, ectrls=0x0, attrs=0x0, attrsonly=0, send_result=0,
nentries=0,
urls=0x0) at ../389-ds-base/ldap/servers/slapd/result.c:1507
#4 0x00007fc8b0639194 in send_ldap_search_entry (pb=0x7fc880bff120,
e=0x7fc882667180, ectrls=0x0, attrs=0x0, attrsonly=0)
at ../389-ds-base/ldap/servers/slapd/result.c:1050
#5 0x00007fc8b060ded1 in send_entry (pb=0x7fc880bff120,
e=0x7fc882667180, operation=0x7fc880c3c400, attrs=0x0, attrsonly=0,
pnentries=0x7fc8825f62ec)
at ../389-ds-base/ldap/servers/slapd/opshared.c:1132
#6 0x00007fc8b060e497 in iterate (pb=0x7fc880bff120, be=0x7fc8aee457e0,
send_result=1, pnentries=0x7fc8825f62ec, pagesize=-1,
pr_statp=0x7fc8825f62dc)
at ../389-ds-base/ldap/servers/slapd/opshared.c:1463
#7 0x00007fc8b060e8a5 in send_results_ext (pb=0x7fc880bff120,
send_result=1, nentries=0x7fc8825f62ec, pagesize=-1, pr_stat=0x7fc8825f62dc)
at ../389-ds-base/ldap/servers/slapd/opshared.c:1680
#8 0x00007fc8b060d444 in op_shared_search (pb=0x7fc880bff120,
send_result=1) at ../389-ds-base/ldap/servers/slapd/opshared.c:875
#9 0x00007fc8b06258c8 in search_internal_callback_pb
(pb=0x7fc880bff120, callback_data=0x7fc8825fa9b0, prc=0x7fc8b062534c
<internal_plugin_result_callback>,
psec=0x7fc8b0625257 <internal_plugin_search_entry_callback>,
prec=0x7fc8b06252e2 <internal_plugin_search_referral_callback>)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:730
#10 0x00007fc8b062542e in search_internal_pb (pb=0x7fc880bff120) at
../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:594
#11 0x00007fc8b06251ca in slapi_search_internal_pb (pb=0x7fc880bff120)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:504
#12 0x00007fc8b0625cb8 in slapi_search_internal_get_entry
(dn=0x7fc8abe90240, attrs=0x0, ret_entry=0x7fc8825faa68,
component_identity=0x7fc8ac6985e0)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:864
#13 0x00007fc8ac4ca221 in mep_pre_op (pb=0x7fc880bff000, modop=4) at
../389-ds-base/ldap/servers/plugins/mep/mep.c:2169
#14 0x00007fc8ac4ca64f in mep_mod_pre_op (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/plugins/mep/mep.c:2303
#15 0x00007fc8b061ed1c in plugin_call_func (list=0x7fc8ac6c3e00,
operation=461, pb=0x7fc880bff000, call_one=0) at
../389-ds-base/ldap/servers/slapd/plugin.c:2030
#16 0x00007fc8b061eb85 in plugin_call_list (list=0x7fc8aef6fe00,
operation=461, pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/plugin.c:1973
#17 0x00007fc8b061b912 in plugin_call_plugins (pb=0x7fc880bff000,
whichfunction=461) at ../389-ds-base/ldap/servers/slapd/plugin.c:442
#18 0x00007fc8ac53b87c in ldbm_back_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:687
#19 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021
#20 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#21 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#22 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#23 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#24 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#25 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc880c68300) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8b0625d24 in slapi_search_internal_get_entry
(dn=0x7fc8abe90240, attrs=0x0, ret_entry=0x7fc8825faa68,
component_identity=0x7fc8ac6985e0)
at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:873
#2 0x00007fc8ac4ca221 in mep_pre_op (pb=0x7fc880bff000, modop=4) at
../389-ds-base/ldap/servers/plugins/mep/mep.c:2169
#3 0x00007fc8ac4ca64f in mep_mod_pre_op (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/plugins/mep/mep.c:2303
#4 0x00007fc8b061ed1c in plugin_call_func (list=0x7fc8ac6c3e00,
operation=461, pb=0x7fc880bff000, call_one=0) at
../389-ds-base/ldap/servers/slapd/plugin.c:2030
#5 0x00007fc8b061eb85 in plugin_call_list (list=0x7fc8aef6fe00,
operation=461, pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/plugin.c:1973
#6 0x00007fc8b061b912 in plugin_call_plugins (pb=0x7fc880bff000,
whichfunction=461) at ../389-ds-base/ldap/servers/slapd/plugin.c:442
#7 0x00007fc8ac53b87c in ldbm_back_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:687
#8 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021
#9 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#10 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#11 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#12 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#13 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#14 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
#0 slapi_entry_dup (e=0x7fc880c68180) at
../389-ds-base/ldap/servers/slapd/entry.c:1997
#1 0x00007fc8ac53bfb6 in ldbm_back_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:844
#2 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000,
pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021
#3 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at
../389-ds-base/ldap/servers/slapd/modify.c:380
#4 0x0000000000418c63 in connection_dispatch_operation
(conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000)
at ../389-ds-base/ldap/servers/slapd/connection.c:624
#5 0x000000000041ad49 in connection_threadmain () at
../389-ds-base/ldap/servers/slapd/connection.c:1753
#6 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so
#7 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0
#8 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6
--
389 Directory Server Development Team
4 years, 1 month
Is there going to be a console for RHEL/CentOS 8?
by Jorge F. Hernandez
Hello,
I just installed 389 to a CentOS 8 using the instructions from your website
(yum module), but I cannot find an admin/management tool for it, before
there used to be a console or an admin web site, but I don't see it anymore,
any ideas if this is going to be available in the future?
I can see the 389 Directory Server in the cockpit, but there is no way to
add/edit users/groups on it.
Thanks,
============
Jorge F. Hernandez
4 years, 1 month
