Author Topic: using proftpd on LDAP enabled Solaris client causes performance issues  (Read 983 times)

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
We are using proftpd since years under Solaris 10 and it always worked fine for us. But we have lately LDAP enabled our Solaris clients. The servers are highly frequented with ftp logins. As soon as we LDAP enabled the client the performance was drastically going down and we saw CPU spikes on the LDAP server. Logins we taking up to a minute to return a prompt for user and password. We opened a ticket at Oracle to analyze what is going on and trussed the inet daemon during ftp logins. In the analysis Oracle said the following:

That software does not make use of the PAM-framework, otherwise we should have seen functions pam_sm_* in the truss output.
$ grep pam_sm truss3

That software does not call initgroups(3C) to resolve the supplementary groups for the user, otherwise we should have seen the function initgroups() in the truss output.
$ grep initgroups truss3


That software does enumerate the group database, so the truss contains 8582 calls getgrent():
$ grep 398876 truss3|grep "\-> libc:getgrent" |wc -l
8582

Enumerating the entire group database that contains 8582 entries took 3.5 seconds in this example.
That makes out the delay seen when starting FTP-sessions towards this system.

Why does that proftpd binary call getgrent() to enumerate all entries that are contained in the group database,
while it should better call the function initgroups(3C) to determine all suppelementary groups for that user?

This is how we have configured proftpd:

./configure -prefix=/usr/local/proftpd.1.3.5b --with-modules=mod_ldap --enable-openssl --with-includes=/opt/csw/include --with-libraries=/opt/csw/lib --sysconfdir=/usr/local/etc CFLAGS=-m64 LDFLAGS=-m64

Unfortunately I was not able to find the mod_ldap module in the list of documented modules, but as it exists I assume that there is also some configuration work around it apart from just builing proftpd with the LDAP module. Did anyone already run into this issue and have an idea on how to solve it?

Thanks in advance for your help!

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #1 on: August 02, 2016, 09:42:34 am »
one main thing that I forgot to mention is that the majority of the users that try to ftp to the affected servers are local users that don't even exist in LDAP so another question is if there exists a directive that allows to avoid LDAP lookups for specific users.

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5274
    • View Profile
    • http://www.castaglia.org/
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #2 on: August 24, 2016, 11:34:01 pm »
There's currently no option to avoid using LDAP for specific names, since ProFTPD doesn't trust the name, as provided by a client, until that name is authenticated.  But by that point, you will have needed to use LDAP (or not) to perform the authentication; this chicken-and-egg issue makes it harder to filter/avoid LDAP by name.  You *can* enable/disable use of mod_ldap by "network class" (IP address/range from which the client is connecting), if that helps.

ProFTPD does not use initgroups(3) because that function, on many platforms, does not return an error if more than the kernel-supported maximum number of groups per user is exceeded.  That is, we encountered cases where a user might be a member of more than 16 groups -- and the kernel in question only supported 16 groups per user.  The initgroups(3) function did NOT return an error, and thus the admin was confused as to why that user was unable to access a file (whose permissions allowed one of the silently truncated groups).  So now, to work around behavior like that, ProFTPD will manually iterate through all of the groups, and do its own setting of group IDs for the proceses.  These are the getgrent(3) function calls you are seeing.

And ProFTPD *does* support/use PAM -- but only for authenticating a user, not for obtaining additional information about that user.  See:

  http://www.proftpd.org/docs/howto/Authentication.html
  http://www.proftpd.org/docs/contrib/mod_ldap.html

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #3 on: September 23, 2016, 07:49:16 am »
Thanks a lot for your answer and explanation!

However it is not completely clear to me and I don't agree to everything. I understand that it is not ideal that the initgroups function does not return an error when a user belongs to more groups than allowed in the kernel, but this is an kernel feature so also other connection methods like ssh have to deal with that. Furthermore (at least for Solaris) NGROUPS_MAX is a system tunable so it can easily be set to a higher value to avoid this problem whereas using getgrent instead of initgroups makes proftpd simply unusable for highly frequented environments with many groups.

Wouldn't it be an option instead to leave the choice of using initgroups or getgrent to the user depending on the use case when building proftpd? So would you be able to deliver source code that offers both options?

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5274
    • View Profile
    • http://www.castaglia.org/
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #4 on: September 23, 2016, 05:48:59 pm »
Yes, it's possible to modify proftpd to use initgroups(3), and make it configurable.  I've filed an issue on GitHub for this; see:

  https://github.com/proftpd/proftpd/issues/338

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #5 on: September 27, 2016, 09:59:34 am »
Sounds great, thanks a lot in advance!

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5274
    • View Profile
    • http://www.castaglia.org/
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #6 on: September 27, 2016, 11:14:08 pm »
Are you able to test the pull request associated with that issue, to make sure that the code change works/does what you expect/want?  See:

  https://github.com/proftpd/proftpd/pull/339

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #7 on: September 30, 2016, 10:04:46 am »
What do I need to do for that?

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5274
    • View Profile
    • http://www.castaglia.org/
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #8 on: November 20, 2016, 08:30:33 pm »
That pull request has been merged to the master branch on GitHub.  To test, now all you need do is download the source code from the master branch on GitHub, compile/install as you normally would, and test the results.

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
sorry for the late reply, but finally I am able to move forward with this. I had tested your 1.3.5b (maint) version and it seemed to work fine. As some time past in the meantime I see that there exists version 1.3.5.d in the meantime. Is this change implemented in that version as well?

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5274
    • View Profile
    • http://www.castaglia.org/
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #10 on: March 17, 2017, 06:00:31 am »
Yes.

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #11 on: March 20, 2017, 10:44:26 am »
Great, thanks for your feedback!

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #12 on: March 20, 2017, 01:32:25 pm »
sorry, but I need to ask once more for verification ...

The release notes of version 1.3.6rc3 list:

+ Use initgroups(3) for group membership discovery by default, as it is
    faster/more performant on most systems.  For the previous behavior, use
    the NoInitgroups AuthUnixOption.

The release notes of version 1.3.5d however don't list the initgroup change. Was that simply forgotten in the release notes or is it really not included in that version?

And I have just checked again which version I had tested. It was 1.3.6rc3 and not 1.3.5b as I had written before.

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5274
    • View Profile
    • http://www.castaglia.org/
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #13 on: March 20, 2017, 03:54:42 pm »
The use of initgroups(3) is not in the 1.3.5 release series, being a new feature.  That's why, as you noted, it first appeared in 1.3.6rc3:

  https://github.com/proftpd/proftpd/blob/master/RELEASE_NOTES#L60

Offline smaass

  • New user
  • *
  • Posts: 9
    • View Profile
Re: using proftpd on LDAP enabled Solaris client causes performance issues
« Reply #14 on: March 22, 2017, 08:44:14 am »
ok, thanks for the clarification!