[PATCH] malloc: fix startup races

[Joern Engel <joern at purestorage dot com>]

Malloc() can get called before ptmalloc_init().  A check in arena_get()
covers that.  ptmalloc_init() should use cmpxchg or we could have
several concurrent initializers.

There still is a race where early threads can malloc_starter() and
exhaust the available memory.  Most programs don't expect malloc() to
return NULL, so I consider that yet another failure.  Easy workaround is
to do an allocation before creating threads.

JIRA: PURE-27597
---
 tpc/malloc2.13/arena.h | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/tpc/malloc2.13/arena.h b/tpc/malloc2.13/arena.h
index 8890e83ad18f..86e77ffe57f6 100644
--- a/tpc/malloc2.13/arena.h
+++ b/tpc/malloc2.13/arena.h
@@ -333,11 +333,13 @@ static void ptmalloc_init(void)
 	const char *s;
 	int i, secure = 0;
 
-	if (__malloc_initialized >= 0)
+	if (!__sync_bool_compare_and_swap(&__malloc_initialized, -1, 0)) {
+		do {
+			sched_yield();
+		} while (__malloc_initialized <= 0);
 		return;
-	__malloc_initialized = 0;
-
-		ptmalloc_init_minimal();
+	}
+	ptmalloc_init_minimal();
 
 #ifndef NO_THREADS
 #ifndef NO_STARTER
@@ -792,13 +794,19 @@ static inline int getnode(void)
    readily available, create a new one.  In this latter case, `size'
    is just a hint as to how much memory will be required immediately
    in the new arena. */
-
 static struct malloc_state *arena_get(size_t size)
 {
 	struct malloc_state *arena = NULL;
 	int node = getnode();
 
 	/*
+	 * It is possible to race with malloc_init and "win".  The
+	 * bug has existed for decades, but the race window grew
+	 * with numa_arenas.
+	 */
+	if (__malloc_initialized <= 0)
+		ptmalloc_init();
+	/*
 	 * getnode() is inherently racy.  It returns the correct node
 	 * number at the time of the syscall, but the thread may be
 	 * migrated to a different node at any moment, even before
-- 
2.7.0.rc3