[C++/78252] libiberty demangler crash with lambda (auto)

[Nathan Sidwell]

This patch fixes a problem in libiberty's symbol demangler.  With a 
templated forwarding function such as std::forward, we can end up 
emitting mangled function names that encode lambda information.  Lambdas 
with auto argument types have a synthesized templated operator(), and 
g++ uses that when mangling the lambda.

Unfortunately g++ doesn't notice the template parameters there mean 
'auto' and emits regular template parameter references. (This is a bug, 
see below.)

But, as the forwarding function itself is a template, and the lambda is 
part of a template parameter substitution, we can end up with the 
demangler recursing unboundedly.  In other cases we can fail to demangle 
(returning null), or demangle to an unexpected type (substituting the 
current template parameter type into the place of the 'auto').

This patch fixes the demangler by noting when it's printing the argument 
types of a lambda.  In that case whenever we encounter a template 
parameter reference we emit 'auto', and also inhibit some &/&& smushing 
that needs checking.  AFAICT, once inside a lambda argument list we 
cannot encounter template parameter references that actually refer to an 
enclosing template argument list. That means we don't have the problem 
of disabling this additional check within the argument list printing.  I 
don't think we can meet a nested lambda type either, but the ++...-- 
idiom seemed safer to me.

We cannot do this substitution when parsing the mangled name, because 
g++ applies the usual squangling back references as-if there really was 
a template parameter reference.  Later squangling references to the type 
containing the lambda argument may or may not require the reference to 
be to an enclosing template argument, or be auto, depending on the 
context of the squangle reference.

I've also included a c++ testcase to check the mangling of the lambdas 
that cause this.  While this is a g++ bug, it's an ABI-affecting one, 
and we shouldn't change the behaviour unintentionally.  I've not 
investigated why the mangler's failing to check is_auto, and will look 
at that later.  I imagine a fix will be -fabi-version dependent. I have 
filed 78621 to track it.

ok?

Nick, we originally found this when GDB exploded. If you're ok with it, 
I'll commit to binutils/gdb when approved for gcc.

nathan
-- 
Nathan Sidwell
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cp-dem.patch
Type: text/x-patch
Size: 7230 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/binutils/attachments/20161130/b9e85aca/attachment.bin>