Re: [SSSD] [RFC][PATCH] Add new getgrgid2(), getgrnam2() interfaces to glibc

[Ralf Haferkamp <rhafer at suse dot de>]

Am Dienstag 19 Oktober 2010, 14:04:06 schrieb Stephen Gallagher:
> On 10/18/2010 07:42 PM, Petr Baudis wrote:
> > This RFC patch adds support for new interfaces: getgrgid2(),
> > getgrnam2() and their *_r() variants. These interfaces allow
> > the user to specify whether the group.gr_mem field shall be
> > filled in.
> > 
> > The issue we are trying to solve is that needlessly high load is
> > generated in large deployments with NIS/LDAP user groups with
> > possibly thousands of members. Even when nscd is used, the
> > information is requested and transferred frequently even when there
> > never arises any real use for the huge lists of members at the node
> > at all.
> > 
> > We have looked at some commonly used applications and e.g. for a
> > normal desktop, if some basic utilities and file managers would be
> > trivially modified to use the new interface, the full lists of
> > users in a group would never have to be downloaded. Right now, even
> > ls will trigger getting full lists of users in all groups it
> > encounters. Moreover, e.g. the sssd guys are reportedly trying to
> > solve this problem by evil hacks like randomly not filling or only
> > partially filling the gr_mem field.
> 
> Actually, your information is a little bit incorrect. We recently
> discovered that we had a few bugs in SSSD that were causing us to only
> present a subset of the gr_mem field. This was entirely incorrect, and
> as of SSSD 1.2.4 and 1.4.0 we have rectified this problem.
From the recent discussion on sssd-devel it seemed to have been done as 
an "optimization" initially. But in the end that doesn't really matter.
  
> The way we do things in SSSD is, however, designed to significantly
> reduce the load on LDAP servers. When we perform a group lookup with
> getgrnam() or getgrgid(), we request ONLY this group from the LDAP
> server. What we then do on our client-side is to create in our cache
> mechanism a series of "fake" users in our data store. These users will
> be reported back to getgrnam(), but they are not separately looked up
> and populated with their complete user information unless they are
> directly requested by a getpwnam() or getpwid() request.
This is not correct. See below.

> The net result is that we will report all of the users that belong in
> a group, but we will not make additional requests to populate those
> users unless they are directly requested.
> 
> Yes, for groups that have many thousands of users, we may make a
> single request that transfers a couple hundred kilobytes, but we do
> not make subsequent requests for each of those users in turn.
> 
> RFC2307bis servers require slightly more processing, as we need to
> make multiple requests to LDAP if there are nested groups in play.
> But we never make more requests than the configured nesting limit.
I think your are mixing things up a bit. For RFC2307bis, even if nested 
groups are not used (IIRC nested groups were never really part of 
RFC2307bis), sssd still needs to make an LDAP lookup for every single 
member to correctly populate the gr_mem attribute of the struct group, 
you can't create any fake entries in the sssd cache for that (the fake 
user approach only works with plain old RFC2307 groups).

> > Any comments on the API are welcome, including thoughts on how to
> > make it more elegant. Originally, I wanted to implement it as
> > gid_t getgidbyname(const char *name) etc., but I would have to redo
> > some parts of the NSS infrastructure to easily allow returning gid_t
> > instead of a struct.
> > 
> > If there is a consensus that this is a good idea and Ulrich gives
> > his blessing, I will of course extend the patch to be compatible
> > with older NSS modules and add NIS support and nscd support.
> 
> I'm really not sure what the value of this approach is. Realistically,
> you're talking about creating an interface whose sole purpose is to
> map GID<->groupname and nothing else?
Yes.
 
> I'm not sure there's sufficient value in this approach, given that
> it's possible as described above to limit the network requests. I
> think it makes much more sense to guide the respective name-service
> providers along the path of handling these requests more efficiently.
That's the problem, for e.g. RFC2307bis this can't be done more 
efficiently in a reliable manner (see the recent discussion around 
getgrgid/getgrnam() on the sssd-devel list). So as most callers of 
getgrnam/getgrgid don't use the member information anyways I think such 
gid<->groupname mapping function could be quite useful. 

> I'm expanding this mailing to include the sssd-devel list for broader
> discussion.

-- 
Ralf