[PATCH v2] libio: use PTR_MANGLE/PTR_DEMANGLE for FILE vtables

[Florian Weimer]

On 10/01/2015 09:01 PM, Kees Cook wrote:
> On Thu, Oct 1, 2015 at 11:52 AM, Florian Weimer <fweimer@redhat.com> wrote:
>> On 10/01/2015 08:40 PM, Kees Cook wrote:
>>> Hi!
>>>
>>> This is a resend (from 2011!) of my PTR_MANGLE series for the FILE vtable.
>>> It was previously discussed here:
>>> https://sourceware.org/ml/libc-alpha/2011-12/msg00073.html
>>>
>>> It sounds like only very very old code was sharing FILE externally,
>>> so this appears safe now. "make xcheck" passes. If this is NOT okay for
>>> general use, I'd like to make it a configure option, since for modern
>>> glibc users, it works fine. (e.g. Chrome OS has been using this patch
>>> for 3 years now.)
>>
>> This needs a configure option for now, but we can enable it for some
>> architectures per default.
> 
> How do you think this should best be handled? (Are there examples of
> similar configure flags?)

Probably something like this in configure.ac:

AC_ARG_ENABLE([stackguard-randomization],
              AC_HELP_STRING([--enable-stackguard-randomization],
                             [initialize __stack_chk_guard canary with a
random number at program start]),
              [enable_stackguard_randomize=$enableval],
              [enable_stackguard_randomize=no])
if test "$enable_stackguard_randomize" = yes; then
  AC_DEFINE(ENABLE_STACKGUARD_RANDOMIZE)
fi

And you need to update config.h.in.

>> At one point, someone needs to do the thankless job and unearth where
>> the cut-over point to modern libstdc++ was, and which architectures
>> actually need the vtable compatibility.  (If there was no GCC version
>> that could compile the relevant libstdc++ versions, then compatibility
>> is obviously unnecessary.)
> 
> Is there a hint about how to detect this? I don't know what I'd be looking for.

libio.h is apparently an installed header, so it is still part of the
public API.  This means that this API is technically supported on all
current architectures, even those which never saw the old libstdc++
version.  (libstdc++ switched in 2003…)

This means that vtable mangling is very much a backwards-incompatible
change.  We can still salvage this in some way.

Do you care mainly about statically linked libc, dynamically linked
libc, or both?