Re: static TLS exhausted on ppc64le

[Rich Felker <dalias at libc dot org>]

On Mon, Sep 30, 2019 at 05:47:29PM +0200, Florian Weimer wrote:
> * Szabolcs Nagy:
> 
> > On 30/09/2019 15:02, Dan Horák wrote:
> >> Hi,
> >> 
> >> I would like to open a problem we have already met twice in Fedora. the
> >> symptom is
> >> 
> >> "/lib64/libgomp.so.1: cannot allocate memory in static TLS block"
> >> 
> >> usually when loading a lot of libraries/modules into a Python
> >> application. It happened on ppc64le and also on aarch64 systems.
> >> 
> >> We have 2 reports in Fedora bugzilla about with more details.
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1722181
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1738752
> >> 
> >> We have already discussed that briefly with Florian and other members of
> >> the Red Hat toolchain team, but outcome was in form of a recommendation
> >> to reduce the usage of "static TLS" objects in the individual libraries.
> >> But the open question still is - is there a fix for the TLS space
> >> exhaustion? I believe it can easily become a more serious problem soon.
> >
> > (a workaround is preloading the problematic libs at startup time)
> >
> > i think it's a bug in libgomp.so.1, gcc should not build
> > broken dsos by default (unless it can ensure they are never
> > loaded dynamically):
> >
> > https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libgomp/configure.tgt;h=b88bf72fe3de3735929635c874b8da375c841b1d;hb=HEAD#l13
> 
> I like the simplicity of initial-exec TLS.

I wouldn't really characterize it as simplicity. It's a trade of
complex (at least to the user) constraints on whether or not it works
for some simplicity of implementation.

I guess for glibc at present, there's a lot more complexity to dynamic
models because of lazy allocation and installation and generation
counters, and these interact with AS-safety and failsafety in
undesirable ways. I'd like to see that fixed but I know it's a big
change.

> I think there was a change on POWER to use the static TLS reservation
> for dynamic TLS, as an optimization.  Obviously, that's going to hurt
> those cases where a library with initial-exec TLS is loaded late, even
> if the static TLS reservation would ordinarily be large enough.

Was that because of the PLT-stub hack on powerpc done in lieu of
tlsdesc? That should really be abandoned entirely IMO, since it
*doesn't* give you any of the benefit of tlsdesc -- the whole point is
not the short code path but avoiding register spills for the standard
ABI call to __tls_get_addr, and the powerpc hack doesn't let you avoid
them. Real tlsdesc should be added to powerpc.

Rich