RFC: GCC plugin to find encrypted function pointer calls in glibc

[Aldy Hernandez <aldyh at redhat dot com>]

Hi folks!

I'm working on a GCC plugin to pinpoint places in glibc where indirect 
function calls are made through a function pointer that has not been 
demangled.

As a reference, I am attaching a test case with various things that my 
plugin currently warns on that may serve as examples to what I am trying 
to accomplish.

The general idea is that I have defined a decryption operation as the 
output of a binary operator (xor, ior, and), so while we would warn for 
something like this:

	typedef void (*callback_t) (void);
	void foo (callback_t cb)
	{
	  cb ();
	}

...the following would be OK:

	#define PTR_DEMANGLE(var) \
	  (var) = (__typeof(var)) ((unsigned long) (var) ^ MAGIC)

	typedef void (*callback_t) (void);
	void foo (callback_t cb)
	{
	  callback_t tmp = cb;
	  PTR_DEMANGLE (tmp);
	  tmp ();
	}

However, I see glibc uses the following idiom:

#  define PTR_DEMANGLE(reg)	asm ("ror $2*" LP_SIZE "+1....)

Since I would prefer not to assume the output of all inline asm's are a 
demangling operation, I would like to get feedback from the community on 
what would be preferred.

My preferred approach is to add an attribute to an inline function that 
would wrap the asm:

	__attribute__((decrypt)) static inline funcp demangler (funcp f)
	{
		asm("blah");
	}

This is straightforward, clean, and follows language semantics (not to 
mention that I already have it implemented into my plugin :)), but 
Florian made funny faces when I showed it to him, so here I am :).

It would be neat if GCC had a way of tagging individual gimple 
statements with an attribute, so we could tag them with 
__attribute__((decrypt)), but alas we don't have such mechanism, and I'd 
prefer not to perform major surgery to GCC to make it so.

Another alternative would be to tag inline asms with commented out magic 
at the end, such that the plugin would notice and take appropriate action:

#ifdef FUNCTION_POINTER_CHECKS
#define DEMANGLE_TAG " ##_DEMANGLE_##"
#else
#define DEMANGLE_TAG ""
#endif
#  define PTR_DEMANGLE(var)  asm("ror $2*..."##DEMANGLE_TAG \
				: "=r" (var)
				: "0" (var)
				etc
				etc

This is straightforward, but I still prefer the inline function plus 
attribute idea.

If anyone is interested, I can post the code to my plugin.

What do y'all think?

Aldy

Attachment: test-funcp.c
Description: Text document