[PATCH] malloc/malloc.c: Mitigate null-byte overflow attacks

[Florian Weimer]

On 11/03/2017 06:44 PM, DJ Delorie wrote:
> 
> 
>> I like those ideas, the first two seem to be straight forward to
>> integrate. If that's something you guys want, I can write a patch for that?
> 
> Assuming we have a fast way to convert to big-endian, I think it would be interesting to make all the size fields big-endian, and see if that affects performance measurably.  I think it would make overflow hacks significantly more difficult.

__builtin_bswap64 ((uintpr_t)p) should be decent.

32-bit will need __builtin_bswap, which is a bit of a wart.

> The MSB alone isn't enough, so a simple rotate is insufficient, as the MSB tends to be zero already on 64-bit platforms.

I'd suggest to rotate by more than a byte if BSWAP is not fast enough.

> Alternately, a simple XOR with a magic number means a set-to-zero would un-XOR to a horribly wrong new "size".  Even a fixed magic number would increase hackability significantly, although a per-process one would be better (and more expensive to do at runtime, unfortunately).

See my old heap protector patches.  You could probably swap in bswap in 
place of the encryption, and it will just work.

> Heck, even ~size would be interesting to ponder.  The question is, which operations will break-in attempts have access to?

Most overflows are more than just a single NUL byte, unfortunately.

> This will, of course, further break dumped heaps, like emacs, but hopefully we're past that by now.

Actually, that's not a problem.  I think my heap protector patch simply 
rewrites the dumped chunk headers into the appropriate format.

I will likely be busy with ABI-impacting work for many months to come, 
so I won't finish the heap protector patches anytime soon.

Thanks,
Florian