Something about the mt_allocator.

[Felix Yen]

Re: false sharing.
 >  >  I eventually concluded that no action is required.
 >
 >  Why?

Because doing nothing is always a good idea :-)  More seriously, 
because I don't think there's anything that can be done without 
replacing the entire algorithm and/or violating the standard.  We have 
to return memory that was allocated by operator new(), and are 
committed to recycling on a per-object size basis as long as the 
performance test results support this effort.

 >  >  This style of recycling probably does interfere with whatever the
 >  >  underlying allocator is doing to prevent false sharing.
 >
 >  How?

In the classic producer-consumer scenario, there's one producer and one 
consumer and recycling should work fairly well.  The two threads 
communicate via some sort of queue, and via the global free list for 
some size.  Contention is reduced by amortizing lock usage, e.g. by 
adding/removing multiple blocks to/from the free list whenever its lock 
is acquired.

(If the producer always tries to remove a chunk's worth of blocks from 
the free list, the consumer should never add less than that amount, and 
there is nothing to gain by waiting once it has that many blocks to 
recycle.  In other words, in this scenario the headroom concept adds no 
value.)

In the more general case, there are P producers and C consumers.  
Initially, the producers allocate chunks directly from the heap and 
enqueue blocks from these chunks in an orderly fashion.  However, each 
consumer recycles blocks received from all of its producers, and the 
producers eventually remove motley collections of blocks from the free 
list.  Adding to the chaos, the producers and consumers may be hosted 
by different processors.  Furthermore, the underlying allocator cannot 
assist because the recycler never releases any of its memory.

(Completing my digression: Headroom becomes a factor if the consumer is 
also a producer or if it uses blocks of the same size internally.  
However, its use of the allocator depends on the relationship between 
its consumption and allocation rates, so optimizing allocator support 
for this thread is difficult.)


Felix