RFC: GCC plugin to find encrypted function pointer calls in glibc

[Aldy Hernandez]

On 04/29/2016 09:21 PM, Carlos O'Donell wrote:
> On 04/29/2016 06:23 AM, Aldy Hernandez wrote:
>> Since I would prefer not to assume the output of all inline asm's are
>> a demangling operation, I would like to get feedback from the
>> community on what would be preferred.
>
> Awesome looking work!
>
>> My preferred approach is to add an attribute to an inline function that would wrap the asm:
>>
>>      __attribute__((decrypt)) static inline funcp demangler (funcp f)
>>      {
>>          asm("blah");
>>      }
>>
>> This is straightforward, clean, and follows language semantics (not
>> to mention that I already have it implemented into my plugin :)), but
>> Florian made funny faces when I showed it to him, so here I am :).
>
> You can build glibc with gcc 4.7 or newer.
>
> (1) static inline wrapper with function attribute:
>
> For a representative set of architectures, say x86_64, i686, ppc64,
> s390x, aarch64, and arm, does a 4.7 or newer compiler do as good a
> job with a static inline function as it does with the macro that
> evaluates to a bare assembly?

This requires some careful analysis.  I will look into this next week 
and report back.

>
> (2) Asm attribute:
>
> We have function, variable, type, label, and enumerator attributes,
> why not support them on asm statements to mark them as decrypt or
> encrypt functions? This would lend itself to a more natural conversion
> of inline assembly that users may want to write and embed into their
> programs.

I haven't looked deeply into this, but here are my initial thoughts in 
the interest of not stalling the conversation.

All of the things you mention are types (sort of).  GCC has support for 
attributes on types.  We don't support attributes on statements, which 
have an entirely different representation (gimple).  I would be hesitant 
to add attributes to statements, just for supporting one plugin.  Plus 
the fact that unless we revamp the way attributes work in GCC 
altogether, we'll end up having to strcmp() our way through the 
attribute list in each pass that is interested in attributes (for every 
single interesting statement).

So... (a) inefficient in its current state, (b) involved to get it 
working efficiently, yet only for one plugin.  And that one plugin could 
conceivably use the present infrastructure (inline functions).

That being said, I owe you some analysis of what GCC 4.7 does with 
inline asm's.

>
> It is entirely conceivable that (1) does a good-enough job, and we
> add (2) at the same time to the latest gcc and backport. When we raise
> the supported build compiler version we can switch to (2) if needed.
>

I take it option 3 is too disgusting, provided #1 is a no go?  That is, 
adding a carefully crafted comment at the end of an inline asm string?

Errr, and by the way, is it a requirement that this plugin work with 
4.7?  Because, we only need this for static analysis.  Is building glibc 
twice a no go (once with the plugin for static analysis, and once for 
building the actual production code)?  I thought the purpose of the 
plugin was just to analyze what's already there, but I do see the 
benefit of using the same compiler for static analysis _and_ for 
building the library.

If the same compiler is a requirement, the plugin requires some 
rewriting, because unfortunately, the plugin infrastructure is a moving 
API target.  I doubt a plugin for GCC 6 will even compile the boiler 
plate nonsense we need for another GCC version (and vice versa).

Aldy