[RFC] Detecting malloc errors.

[Ondřej Bílka]

Hi,

As I looked through various malloc sanity checks and with bit of effort
we could use much stronger checks with similar slowdown. A main problem
here is that a data layout needs to be changed. Then these default checks
 would replace what mcheck does.

For a 64bit systems we do not need most of time 64bits to describe a
chunk size so we can get with 32bit sizes that allow us start with 64bit
sentinel. If a more is needed then we will use a different sentinel
value followed by pointer to actual chunk header.

We need to move flags to prev_size if we want to detect off-by-one
writes. These will be done by writing a rotated sentinel after user
data, which allows us to also use next chunk's sentinel (unless its
unlikely case of a custom sentinel.)

For a 32bit systems we need to adjust sizes accordingly.

A orthogonal improvement is detecting double frees by having a flag that
memory is in use and different flag that memory is freed (and chunks in
fastbins have both bits set.)

struct malloc_chunk
{
  uint64_t	sentinel;

  uint32_t      prev_size;  /* Size of previous chunk (if free). */
  uint32_t      size;       /* Size in bytes as requested. */

  struct malloc_chunk* fd;         /* double links -- used only if free. */
  struct malloc_chunk* bk;

  /* Only used for large blocks: pointer to next larger size.  */
  struct malloc_chunk* fd_nextsize; /* double links -- used only if free. */
  struct malloc_chunk* bk_nextsize
}