[RFC] Avoiding dlopen in statically linked applications that use nss [#27959]

[Arjun Shankar]

Hi all,

I have been working on a fix for bug 27959 ([1], "glibc 2.33 NSS
refactoring lost static NSS support") that involves getting rid of
"--enable-static-nss" entirely and instead moves the decision of
allowing/disallowing dlopen to application build-time. The solution I
was trying to arrive at was:

Statically linked applications that are built "as usual" continue to
behave like they do currently, which is: except for the built-in
"files" and "dns" backends, any other backends configured via
nsswitch.conf *do* lead to the statically linked application
dlopen'ing the corresponding module. However, for applications that
want to avoid having nss related dlopen calls, a new statically linked
library is provided by glibc - say "libnss_nodlopen.a". The archive
would contain just enough of an alternative dlopen-free nss
implementation that compiling with something like:

$CC -static libnss_nodlopen.a application.c

...leads to the linker picking the alternative implementation of nss
functions that are involved in loading nss modules from the new
archive instead of libc.a -- and these wouldn't dlopen. Presumably,
any non-built-in modules configured in nsswitch.conf get treated as if
the corresponding service was unavailable during the call.

Florian referred to this sort of solution back when he moved the
"files" backend into libc [2]. I have been in touch with him and
asking him questions about this topic as I work through this problem.

I started by compiling an additional copy of nss/nss_module.c - the
code that is responsible for module loading. The second copy, through
some macros, avoids calling dlopen. My initial understanding was that
this would be sufficient content in the "new" libnss_nodlopen.a and if
it were provided first on the command line, the linker would prefer
the implementations from there when resolving nss related references
(e.g. "__nss_module_get_function"). This should lead to the
application having no dlopen calls in the statically linked
executable.

This isn't what actually happens. The linker presumably first notes
that the application calls, say "getpwent", then doesn't manage to
resolve that reference until it arrives at libc.a. At that point,
internal functions referred to by "getpwent" going up the nss call
stack *all* get picked from libc.a, i.e. the libc.a copy of
"__nss_module_get_function" gets picked up, and not the one in the
"new" archive I was producing.

Therefore, if my understanding of the situation is correct, this new
statically linked library will need to have more bits from nss
including all public functions available for name resolution in order
for it to work. I expect this would also mean moving code from various
places in the glibc source tree into nss/ (e.g. pwd/, resolv/, grp/
etc.) so that they can then be included in the new library.

How reasonable does that sound as a solution? What other alternatives
are there? Another way of tackling this could be changing the default
so that libc.a doesn't call dlopen for nss and making the dlopen
version optional instead -- presumably via a similar statically linked
library that *does* use dlopen.

Cheers!

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=27959
[2] https://sourceware.org/pipermail/libc-alpha/2021-June/127273.html

-- 
Arjun Shankar
he/him/his