[PATCH] malloc/malloc.c: Mitigate null-byte overflow attacks

[DJ Delorie]

> I like those ideas, the first two seem to be straight forward to
> integrate. If that's something you guys want, I can write a patch for that?

Assuming we have a fast way to convert to big-endian, I think it would be interesting to make all the size fields big-endian, and see if that affects performance measurably.  I think it would make overflow hacks significantly more difficult.  The MSB alone isn't enough, so a simple rotate is insufficient, as the MSB tends to be zero already on 64-bit platforms.

Alternately, a simple XOR with a magic number means a set-to-zero would un-XOR to a horribly wrong new "size".  Even a fixed magic number would increase hackability significantly, although a per-process one would be better (and more expensive to do at runtime, unfortunately).

Heck, even ~size would be interesting to ponder.  The question is, which operations will break-in attempts have access to?

This will, of course, further break dumped heaps, like emacs, but hopefully we're past that by now.