This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: Ruby testsuite failures because of pointer mangling on 32-bit ARM?
- From: Will Newton <will dot newton at linaro dot org>
- To: "Carlos O'Donell" <carlos at redhat dot com>
- Cc: David Miller <davem at davemloft dot net>, Andrew Pinski <pinskia at gmail dot com>, libc-alpha <libc-alpha at sourceware dot org>
- Date: Thu, 12 Dec 2013 21:46:16 +0000
- Subject: Re: Ruby testsuite failures because of pointer mangling on 32-bit ARM?
- Authentication-results: sourceware.org; auth=none
- References: <20131210 dot 211901 dot 1840879367475720601 dot davem at davemloft dot net> <52A94FB5 dot 7060100 at redhat dot com> <20131212 dot 121441 dot 261870704107659875 dot davem at davemloft dot net> <20131212 dot 132053 dot 446942135510037848 dot davem at davemloft dot net> <52AA2BAB dot 50309 at redhat dot com>
On 12 December 2013 21:33, Carlos O'Donell <carlos@redhat.com> wrote:
> On 12/12/2013 01:20 PM, David Miller wrote:
>> From: David Miller <davem@davemloft.net>
>> Date: Thu, 12 Dec 2013 12:14:41 -0500 (EST)
>>
>>> From: "Carlos O'Donell" <carlos@redhat.com>
>>> Date: Thu, 12 Dec 2013 00:55:01 -0500
>>>
>>>> It treats the jmp_buf as an array of VALUE sized pointers that
>>>> it can examine to determine if there are pointers to the heap.
>>>
>>> Sounds similar to what any other garbage collector will do, scan
>>> the processes address space looking for pointers.
>>>
>>> I'm pretty sure Boehm-GC does something similar, although perhaps
>>> it scans the entire process stack from the point in which it is
>>> called instead of using jmpbuf's to delineate spans of stack
>>> areas like Ruby does.
>>
>> And, indirectly, realize that even a straight stack scan is going
>> to potentially break if you start mangling pointers in jmpbuf.
>>
>> Consider the case where if the jmpbuf is on the processes stack, and
>> normally it would get scanned by GC and the pointer followed to find
>> memory references, and now that would not work because the pointer is
>> mangled.
>
> Only if the jpmbuf pointer is the only way to access the other objects.
> It seems like ruby is optimizing by starting with the jmpbuf to look
> for object pointers.
>
>> I think all of these schemes are legitimate and erroneously broken
>> by pointer mangling.
>
> The problem I have is that I can't objectively evaluate how well
> pointer mangling is working as a security feature because I lack
> any data about it. Therefore I can't compare it to the pain it would
> cause ruby to reimplement GC using some other more costly scanning.
>
> What do we do here?
If the frame pointer register is being used as a frame pointer it
can't contain a data pointer. Likewise the stack pointer and lr. As
far as i can tell the only issue is encrypting the frame pointer in
the -fomit-frame-pointer case.
Did the patch I submitted fix the ruby testsuite issue?
--
Will Newton
Toolchain Working Group, Linaro