type_has_non_canonicalized_subtype() gives up too quickly.
For instance, suppose it's looking a type 'foo'. If foo has no
canonicalized type yet and has a data member which type is foo* (for
instance), then type_has_non_canonicalized_subtype() just sees that
type 'foo*' has no canonicalized type, and so it returns, saying that
he found a non-canonicalized subtype for foo.
In that case though, what type_has_non_canonicalized_subtype() should
do is detect that foo* is a pointer to foo itself, so it shouldn't
count as a non-canonicalized sub-type. It should keep going and look
for other meaningful non-canonicalized sub-types.
And this what this patch does. It changes the sub-type walker that
type_has_non_canonicalized_subtype() uses, so that
- it doesn't flag sub-types that refer to the type we are looking
at as non-canonicalized sub-types. This is for sub-types that
are combinations of pointers, references and typedefs.
- it doesn't consider sub-types of member functions of the type we
are looking at, unless that member function is virtual.
The result is that more types are canonicalized early during DWARF
reading, and so there are less types to store on the side for late
canonicalization. This can have a big impact on, e.g, C++ libraries
with tens of thousands of types.
* include/abg-fwd.h (is_typedef, is_pointer_type)
(is_reference_type): Declare new overloads.
(peel_typedef_type): Renamed get_typedef_underlying_type into
this.
(peel_pointer_type, peel_reference_type)
(peel_typedef_pointer_or_reference_type): Declare new functions.
* src/abg-ir.cc (peel_typedef_type): Renamed
get_typedef_underlying_type into this.
(is_typedef, is_pointer_type, is_reference_type): Define new
overloads.
(peel_pointer_type, peel_reference_type)
(peel_typedef_pointer_or_reference_type): Define new functions.
(non_canonicalized_subtype_detector::has_non_canonical_type_):
Make the type of this data member be a type_base*, not a bool.
This is so that we can return the first non-canonicalized subtype
of the type we are looking at.
(non_canonicalized_subtype_detector::non_canonicalized_subtype_detector):
Adjust the data member initialization.
(non_canonicalized_subtype_detector::visit_begin): Add an overload
for function_decl*, to avoid looking into non-virtual member
functions.
In the overload for type_base*, peel typedefs, pointers and
reference of each sub-type that has no canonical type, to see if
refers to the type we are actually walking. If yes, then keep
going.
(type_has_non_canonicalized_subtype): Return the non-canonicalized
sub-type found.
* src/abg-comparison.cc (type_suppression::suppresses_diff):
Adjust for the get_typedef_underlying_type -> peel_typedef_type
renaming.