Bug 9832 - LD_POINTER_GUARD does not fully work
Summary: LD_POINTER_GUARD does not fully work
Status: RESOLVED WONTFIX
Alias: None
Product: glibc
Classification: Unclassified
Component: libc (show other bugs)
Version: 2.8
: P2 normal
Target Milestone: ---
Assignee: Ulrich Drepper
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-02-09 23:17 UTC by Tom Tromey
Modified: 2014-07-01 20:56 UTC (History)
3 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:
fweimer: security-

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Tromey 2009-02-09 23:17:13 UTC 
gdb has some automatic support for nexting over longjmp.
It relies on the user setting LD_POINTER_GUARD=0 to avoid
pointer mangling.

This code does not work on glibc 2.8 (Fedora 9).
I believe the problem is that on x86 (and x86-64), 
LD_POINTER_GUARD=0 still does a transformation.

From libc/sysdeps/unix/sysv/linux/i386/sysdep.h:

#  define PTR_MANGLE(reg)	xorl %gs:POINTER_GUARD, reg;		      \
				roll $9, reg

The rotate is done unconditionally.
(x86-64 is similar)
Comment 1 Ulrich Drepper 2009-03-15 20:47:28 UTC 
There is nothing that can be realistically done.  Pointer encryption is
performed in meant to be possible in performance critical situations.  There the
additional test to see whether the envvar is set is too expensive.

The effect of the envvar is that no formula is needed to decode the pointers. 
No randomness involved.  Just always shift the pointers.