Re: Fixing the distribution problems with TLS and DTV_SURPLUS slots.

[Alexandre Oliva <aoliva at redhat dot com>]

On Oct  6, 2014, "Carlos O'Donell" <carlos@redhat.com> wrote:

> This code is a *heuristic*, it basically fails the load if there
> are no DTV slots left, even though we can still do the following:

> (a) Grow the DTV dynamically as many times as we want, with the
>     generation counter causing other threads to update.

or

  (a)' Stop wasting DTV entries with modules assigned to static TLS.
       There's no reason whatsoever to do so.

       This optimization is even described in the GCC Summit article in
       which I first proposed TLS Descriptors.  Unfortunately, I never
       got around to implementing it.

> and

> (b) Allocate from the static TLS image surplus until it is exhausted.


> - Remove the check above, allowing the code to grow the DTV as large
>   as it wants for as many STATIC_TLS modules as it wants.

We don't really need to grow the DTV right away.  If we have static TLS,
we could just leave the DTV alone.  No code will ever access the
corresponding DTV entry.  If any code needs to update the DTV, because
of some module assigned to dynamic TLS, then, and only then, should the
DTV grow.

> WARNING: On AArch64 or any architecture that uses the generic-ish
> code for TLS descriptors, you will have further problems. There
> the descriptors consume static TLS image greedily, which means
> you may find that there is zero static TLS image space when you
> go to dlopen an application.

That's perfectly valid behavior, that exposes the bug in libraries that
are expected to be loadable after a program starts (say, by dlopen) when
relocations indicate they had to be brought in by Initial Exec.

That they worked was not by design; it was pretty much by accident,
because glibc led by (bad) example instead of coming up with a real
solution, and others followed suit, breaking glibc's own assumption that
only a very small amount of static TLS space would ever be used after
theprogram started, and that the consumer of that space would be glibc
itself.

> We need to further subdivide the static TLS image space into "reserved
> for general use" and "reserved for DSO load uses."  With the TLS
> descriptors allocating from the general use space only.

?!?

Static TLS space grows as much as needed to fit all IE DSOs.  Some
excess is reserved (and this should be configurable), but if we don't
use it for modules that could benefit from it, what should we use it
for?

> On Fedora for AArch64 this
> caused no end of headaches attempting to load TLS IE using DSOs
> only to find it was literally impossible because so much of the
> implementation used TLS descriptors that the surplus static TLS
> image space was gone, and while descriptors can be allocated 
> dynamically, the DSOs can't.

Err...  I get a feeling I have no idea of what you refer to as DSO.
>From the description, it's not Dynamically-loaded Shared Object.  What
is it, then?

I suppose you may be speaking of modules that assume IE is usable to
access TLS of some module (itself, or any other), even though the
assumption is no warranted.

So assume you load a module A defining a TLS section, and conservatively
assign it to dynamic TLS, for whatever reason.  Then you load a module B
that expects A to be in static TLS, because it uses IE to reference its
TLS symbols.  Kaboom.  The âconservativeâ approach just broke what would
have worked if you hadn't gratuitously taken it out of TLS.

Now, of course when you load A you don't know whether module B is going
to be loaded, and whether it will require A to use static TLS or not, or
whether module C would fail to load afterwards because there's not
enough static TLS space for its own TLS section, and it uses IE even
though it's NOT being loaded as a dependency of the IE.

So not saving static TLS space for later use may expose breakage in
subsequently loaded modules, whereas saving it may equally expose
breakage in subsequently loaded modules, but waste static TLS space and
*significantly* impact performance of TLS Descriptor-using modules that
could have got IE-like performance.  That sounds like a losing strategy
to me.

Greedy allocation doesn't guarantee optimal results, but it won't break
anything that isn't already broken, and if and when such breakage is
exposed, switching the broken modules to TLS Descriptors will get them
nearly identical performance for TLS references that happen to land in
static TLS, but that will NOT cause the library to fail to load
otherwise: it will just get GD-like performance.

So, in addition to stopping wasting DTV entries with static TLS
segments, Isuggest not papering over the problem in glibc, but rather
proactively convert dlopenable libraries that use IE to access TLS
symbols that are not guaranteed to have been loaded along with the IE to
use TLS Descriptors in General Dynamic mode.

In order to ease this sort of transition, I've been thinking of
introducing an alias access model in GCC, that would map to GD if TLS
Descriptors are enable, or to the failure-prone IE with old-style TLS.
Then those who incorrectly use IE today can switch to that; it will be a
no-op on arches that don't have TLSDesc, or that don't use it by
default, but it will make the fix automatic as each platform switches to
the superior TLS design.

> In Fedora we disallow greedy consumption of TLS descriptors on any
> targets that have TLS descriptors on by default.

Oh, wow, this is such a great move that it makes TLS Descriptors's
performance the *worst* of all existing access models.  If we want to
artificially force them into their worst case, we might as well get rid
of them altogether!

Whom should I thank for making my work appear to suck?  :-(

:-P :-)

> We need to turn on TLS descriptors by default on x86_64 such
> that we can get the benefits there, and start moving DSOs away
> from TLS IE.

Hallelujah! :-)

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer