Thread Properties API

Background

Various dynamic tools may need to query glibc about various properties of the current thread, such as stack boundaries or static/dynamic TLS boundaries.

The initial discussion was done in https://sourceware.org/bugzilla/show_bug.cgi?id=16291, this wiki page will eventually describe the required Thread Properties API.

Boundaries of stack and static TLS

Proposed interface for retrieving boundaries of stack and static TLS:

// Returns the stack bounds for the current thread. 
// For the main thread both values are assigned to __libc_stack_end.
// This function does not have any visible side effect (such as calling an exported function).
void __libc_get_stack_bounds(void **stack_beg, void **stack_end);

// Get the bounds of static tls for the current thread.
// This function does not have any visible side effect (such as calling an exported function).
void __libc_get_static_tls_bounds(void **stls_beg, void **stls_end);

Dynamic TLS

The tools need to be notified about creation/destruction of dynamic TLS:

immediately after for creation and immediately before for destruction.

The most desirable interface would be:

// This function creates a chunk of dynamic TLS.
// It is AS-safe.
// A tool may interpose this function and either replace of wrap it.
void *__libc_create_dynamic_tls(size_t size, size_t alignment);

// This function destroyes a chunk of dynamic TLS created by __libc_create_dynamic_tls.
// It is AS-safe. 
// A tool may interpose this function and either replace of wrap it.
void __libc_destroy_dynamic_tls(void *dtls, size_t size);

Equally attractive alternative is to have a way to register a callback that is called after DTLS creation and before DTLS destruction.

Less attractive alternative is to have an interface to iterate DTLS chunks in the current (or in a given) thread.

typedef void (*dtls_callback_t)(void *dtls_beg, void *dtls_end, size_t dso_id, void *arg);

// Iterate over all DTLS chunks in the given thread.
// Is 't != pthread_self()', the thread 't' should be blocked before and during this call.
// NOTE: the semantics in case 't != pthread_self()' is not clear! 
// AS-safe.
__libc_iterate_dynamic_tls(pthread_t t, dtls_callback_t cb, void *arg);

This approach is problematic for multiple reasons:

The tools may need to be notified about the DTLS creation/destruction immediately, so they will need to call __libc_iterate_dynamic_tls from __tls_get_addr/dlclose interceptors and in thread shutdown code.

This means greater performance penalty due to required __tls_get_addr interceptor. Besides, the semantics of __libc_iterate_dynamic_tls called in dlclose from a different thread is unclear.

__libc_iterate_dynamic_tls should be AS-safe because it may be called from a signal handler while e.g. __tls_get_addr is being called from a regular code.

Callback for thread exit

Tools such as ASAN, MSAN and LSAN (see below) need to perform various cleanup actions just before a thread exits.

Currently these tools use an ugly hack to get notified about thread's destruction: call pthread_setspecific recursively PTHREAD_DESTRUCTOR_ITERATIONS times; when the PTHREAD_DESTRUCTOR_ITERATIONS-th call is made we consider the thread as dead. A cleaner interface would be much appreciated (not sure if it easy to do though)

// Register callback to be called right before the thread is totally destroyed.
// The callbacks are chained, they are called in the order opposite to the order they were registered.
// The callbacks must be registered only before any threads were created, at most 8 callbacks can be registered.
// No signals may arrive during the calls to these callbacks; immediately after the last of these calls the thread is dead.
void __libc_register_thread_exit_callback(void (*cb)());

Use cases

Here we describe the motivating use cases for the Thread Properties API

AddressSanitizer

AddressSanitizer (aka ASAN) is a tool that detects use-after-free, heap- stack- and global- overflows, and other bugs. Available in Clang (starting from 3.2) and GCC (starting from 4.8).

ASAN needs to know the stack and tls boundaries to properly report error messages.

LeakSanitizer

LeakSanitizer (aka LSAN) is a memory leak detector that can work as a separate library or bundled with AddressSanitizer. Available in Clang (starting from 3.4) and GCC (starting from 4.9).

LSAN needs to know all stack and tls boundaries for all living threads in order to avoid false positive leaks reports. At the beginning of the leak detection phase (typically, at the program shutdown), LSAN creates a separate process that shares the address space with the main process, then attaches to all threads with ptrace. At this point LSAN should be able to extract properties of all threads (such as stack and tls bounds) w/o relying on libc functions, which means that all thread properties should be stored somewhere in LSAN's own data structures.

Example: test case where not knowing about dynamic TLS leads to false leak report

MemorySanitizer

MemorySanitizer (aka MSAN) is a detector of uninitialized reads, available in Clang (starting from 3.4).

MSAN maintains the shadow bit for every bit of the application memory. If the shadow bit is set, it indicates that the corresponding bit in the application memory is poisoned (not initialized). When a thread (or dynamic TLS) is destroyed, the shadow for the stack (or dynamic TLS) should be unpoisoned. Otherwise this memory may be reused by a following mmap call and MSAN will report false positives.

Example: test case where not knowing about dynamic TLS leads to false uninitialized memory report