Summary: | save-index makes gdb slow with dwarf4 | ||
---|---|---|---|
Product: | gdb | Reporter: | Luboš Luňák <l.lunak> |
Component: | gdb | Assignee: | dje |
Status: | RESOLVED FIXED | ||
Severity: | minor | CC: | dje, lionel, mark, ssbssa, tromey, tromey |
Priority: | P2 | ||
Version: | HEAD | ||
Target Milestone: | --- | ||
Host: | Target: | ||
Build: | Last reconfirmed: | ||
Attachments: | WIP patch |
Description
Luboš Luňák
2011-12-14 17:21:01 UTC
Ok, I think I've figured out where the problem is. When completing, location_completer() calls also make_source_files_completion_list(), which calls map_partial_symbol_filenames(). That one is where most of the time during the completion is spent - it calls the proper map_symbol_filenames function depending on the debugging format - for .so's without an index, it's map_symbol_filenames_psymtab(), for indexed files, it is dw2_map_symbol_filenames(). The problem is, with dwarf4-based index, dw2_map_symbol_filenames() iterates over a huge number of filenames, magnitudes more than with dwarf2-based index or psymtabs (in my case, it's 35M filenames with dwarf4 index, 380k with dwarf4 without index, 720k with dwarf2 index, 220k with dwarf2 without index). All the time is spent just iterating over them. And the number is so high because there are many duplicates, as high as 175k duplicates for the most common header files. I don't know DWARF or gdb, so I'm not sure what the proper fix should be. Possibly the save-index functionality should try to eliminate these duplicates. It takes so long because dw2_map_symbol_filenames() loops over both dwarf2_per_objfile::all_comp_units and dwarf2_per_objfile::type_comp_units, where e.g. LibreOffice's libswlo.so has ~700 of the first and ~16000 of the second. It appears unnecessary to iterate over type_comp_units, as all_comp_units should already include all the files encountered during compilation. But it turns out gdb performs poorly with DWARF4 in general. In this case dw2_map_symbol_filenames() could be fixed, assuming what I wrote above is correct, but there appear to be other inefficiencies (related to .debug_types too?). E.g. 'break shared<tab>' (trying to complete a filename) is very fast with DWARF2, but takes very long with DWARF4, with or without the gdb index, and it also noticeably increases memory usage. Thanks for your analysis. It was very helpful. The LibreOffice build crushed my machine, I'm sure you've heard this before :), but I was able to reproduce the problem to my satisfaction just with libooxlo.so. So far I've only built with DWARF-4 and .debug_types (if you are using a recent Fedora you must also specify -fdebug-types-section). I can definitely see a slowdown when moving from the non-indexed to the indexed library (times from "maint time 1", which irritatingly does not work with -ex options): No index (gdb) complete break int_CRYPTO_set_do_dynlock_call break int_CRYPTO_set_do_dynlock_callback Command execution time: 0.118982 (cpu), 0.120245 (wall) (gdb) complete break int_CRYPTO_set_do_dynlock_call break int_CRYPTO_set_do_dynlock_callback Command execution time: 0.123981 (cpu), 0.125739 (wall) Index (gdb) complete break int_CRYPTO_set_do_dynlock_call break int_CRYPTO_set_do_dynlock_callback Command execution time: 4.362337 (cpu), 4.382212 (wall) (gdb) complete break int_CRYPTO_set_do_dynlock_call break int_CRYPTO_set_do_dynlock_callback Command execution time: 3.187515 (cpu), 3.201839 (wall) With a reasonably simple patch I can get this down to: (gdb) complete break int_CRYPTO_set_do_dynlock_call break int_CRYPTO_set_do_dynlock_callback Command execution time: 1.255809 (cpu), 1.260110 (wall) (gdb) complete break int_CRYPTO_set_do_dynlock_call break int_CRYPTO_set_do_dynlock_callback Command execution time: 0.111983 (cpu), 0.112815 (wall) ... the first completion is still quite slow, but later ones are fast. Further investigation and fixes will have to wait until next week. I'll attach my WIP patch momentarily. Created attachment 6226 [details]
WIP patch
I'd like to point out again the second paragraph in comment #2. It's been a while since I did this analysis, so I don't remember all the details anymore and getting into it again would take some time, therefore this is a bit of guesswork and trying to recall, but as far as I remember: I think I concluded that this is not a localized problem in dw2_map_symbol_filenames(), but a design problem with the gdb internal structures. I had a patch for dw2_map_symbol_filenames() that would have worked sufficiently for me for that one specific problem, but I went back to dwarf-2 anyway, and IIRC I decided that larger changes were needed that I wasn't feeling up to. Specifically, I think I found the bottleneck to be dwarf2_per_objfile->n_type_units being a large number in general that slows down more operations than just the one specific example given. As I understand it dwarf-2 had this number quite low, so it wasn't a problem, but dwarf-4 by design causes the number to be much higher. Again, this is just trying to recall things from 2 months ago with my limited gdb knowledge, so you should verify this yourself, but I suggest first checking gdb's performance in general when dealing with large dwarf-4-built .so files. Maybe it'll be necessary to change all accesses to the CU-list (or whatever it's called exactly) to be more efficient (compact it into less items, hashtable for access, whatever). Also, for the save-index case, since gdb creates the index on its own anyway, it could possibly make sense to prepare it better for the completion case. As I understand it, gdb always first collects all items for completion and only afterwards it selects ones eligible for completion, so maybe the index could also contain a sorted list of (references to) all completion items in the matching .so, which presumably would take only relatively little space and would make completion almost instant. (In reply to comment #5) > I think I concluded that this is not a localized problem in > dw2_map_symbol_filenames(), but a design problem with the gdb internal > structures. I had a patch for dw2_map_symbol_filenames() that would have worked > sufficiently for me for that one specific problem, but I went back to dwarf-2 > anyway, and IIRC I decided that larger changes were needed that I wasn't > feeling up to. What I did today is look at all the places where we iterate over all CUs and TUs when using the index. Then I went backward from here to find the callers and thus construct test cases that may exhibit bad behavior. Aside from map_symbol_filenames, which I've already fixed locally, there were only 2 such methods that could plausibly cause problems: map_symtabs_matching_filenames and expand_symtabs_matching. The former had a very bad performance bug in CVS HEAD: http://sourceware.org/ml/gdb-patches/2012-02/msg00409.html However, this is pretty new and I don't think it would have shown up in your tests :-( The test case for this one, with libooxlo.so is just: break numberformatsbuffer.cxx:100 In my testing, the latter is slower when type units are present, but not drastically so. The index speeds things up in both cases: Normal Index +-------+-------+ DW3 |0:07.61|0:01.76| +-------+-------+ DW4 |0:08.80|0:03.45| +-------+-------+ I plan to look into this one a bit more. The test case here is: info function CRYPTO > Specifically, I think I found the bottleneck to be > dwarf2_per_objfile->n_type_units being a large number in general that slows > down more operations than just the one specific example given. As I understand > it dwarf-2 had this number quite low, so it wasn't a problem, but dwarf-4 by > design causes the number to be much higher. Yes, in DWARF < 4, there are no type units. BTW you might wish to use -gdwarf-4 -fno-debug-types-section. This gets you some DWARF 4 benefits without type units; it should be a pure improvement over DWARF[23]. I'm definitely interested if you know of other specific things that are too slow. Ok, I found one more slowdown. The test case: rbreak numberformatsbuffer.cxx:zardoz Is notably slower for DWARF 4 + index. I think this is due to the same problem as affected map_symbol_filenames. CVSROOT: /cvs/src Module name: src Changes by: tromey@sourceware.org 2012-02-20 20:56:13 Modified files: gdb : ChangeLog dwarf2read.c Log message: PR gdb/13498: * dwarf2read.c (dw2_expand_symtabs_matching): Only visit a particular set of file names once. (dw2_map_symbol_filenames): Likewise. Patches: http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog.diff?cvsroot=src&r1=1.13847&r2=1.13848 http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/dwarf2read.c.diff?cvsroot=src&r1=1.613&r2=1.614 What's there now re. TUs could stand for a bit of reorg. 'tis on my long symtab wishlist. I've tried current gdb CVS with the patch mentioned here, and it looks like the problems caused by the index have been fixed by it, so I guess this bugreport can be closed. Attaching gdb to LibreOffice Writer and doing 'break sha<tab>' is still awfully slow with dwarf-4 though, roughly 30 seconds on a rather fast machine (it appears the time is spent in dwarf2_psymtab_to_symtab() and below). That is however with or without the index. With dwarf-2 it is almost instant, but the difference is that with dwarf-2 the completion list does not include template instances, which is a majority of the completions. I find this still unacceptable for dwarf-4 to be usable with something the size LibreOffice. CVSROOT: /cvs/src Module name: src Changes by: devans@sourceware.org 2012-07-10 20:28:34 Modified files: gdb : ChangeLog dwarf2read.c Log message: PR gdb/13498 * dwarf2read.c (dwarf2_per_objfile): New members n_type_unit_groups, all_type_unit_groups, type_unit_groups, tu_stats. (dwarf2_per_cu_data): Move "imported_symtabs" into new union "s". All uses updated. Add type_unit_group to union "s". (type_unit_group): New struct. (IS_TYPE_UNIT_GROUP): New macro. (abbrev_table): Delete unused member "section". (dw2_do_instantiate_symtab): Early exit if type_unit_group. (dw2_get_cu): Assert not used with type_unit_group. (dw2_get_primary_cu): New function. (dw2_build_type_unit_groups_reader): New function. (dw2_build_type_unit_groups): New function. (dw2_get_file_names): Assert not called on type units. (dw2_map_symtabs_matching_filename): Call dw2_build_type_unit_groups. Redo loop to iterate over type unit groups instead of type units. (dw2_expand_symtabs_matching, dw2_map_symbol_filenames): Ditto. (read_abbrev_offset): New function. (init_cutu_and_read_dies): New arg "abbrev_table". All callers updated. (create_partial_symtab): New function. (process_psymtab_comp_unit_reader): Assert not used with type units. Call create_partial_symtab. (process_psymtab_type_unit): Delete. (hash_type_unit_group, eq_type_unit_group): New functions. (allocate_type_unit_groups_table): New function. (NO_STMT_LIST_TYPE_UNIT_PSYMTAB): New macro. (NO_STMT_LIST_TYPE_UNIT_PSYMTAB_SIZE): New macro. (create_type_unit_group, get_type_unit_group): New functions. (tu_abbrev_offset): New struct. (sort_tu_by_abbrev_offset): New function. (add_type_unit_group_to_table): New function. (build_type_unit_groups): New function. (build_type_psymtabs_reader): New function. (build_type_psymtab_dependencies): New function. (build_type_psymtabs): Rewrite. (scan_partial_symbols): Flag an error if a DW_TAG_imported_unit is seen in a type unit. (process_queue): Move symtab expansion debugging printfs here. Call process_full_type_unit for type units. (compute_symtab_includes): Assert not called for type units. (process_cu_includes): Don't call compute_symtab_includes for type units. (process_full_type_unit): New function. (process_imported_unit_die): Flag an error if called for type units. (handle_DW_AT_stmt_list): Delete arg "want_line_info". All callers updated. Assert not called for type units. (read_file_scope): Call dwarf2_start_symtab. (setup_type_unit_groups): New function. (read_type_unit_scope): Rewrite. (abbrev_table_read_table): Initialize abbrev_table->offset. (abbrev_table_free_cleanup): New function. (dwarf2_start_symtab): New function. (load_full_type_unit): Assert not called for type unit groups. Patches: http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog.diff?cvsroot=src&r1=1.14457&r2=1.14458 http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/dwarf2read.c.diff?cvsroot=src&r1=1.683&r2=1.684 (In reply to Sourceware Commits from comment #11) > CVSROOT: /cvs/src > Module name: src > Changes by: devans@sourceware.org 2012-07-10 20:28:34 > > Modified files: > gdb : ChangeLog dwarf2read.c > > Log message: > PR gdb/13498 > * dwarf2read.c (dwarf2_per_objfile): New members n_type_unit_groups, > all_type_unit_groups, type_unit_groups, tu_stats. > (dwarf2_per_cu_data): Move "imported_symtabs" into new union "s". > All uses updated. Add type_unit_group to union "s". > (type_unit_group): New struct. > (IS_TYPE_UNIT_GROUP): New macro. > (abbrev_table): Delete unused member "section". > (dw2_do_instantiate_symtab): Early exit if type_unit_group. > (dw2_get_cu): Assert not used with type_unit_group. > (dw2_get_primary_cu): New function. > (dw2_build_type_unit_groups_reader): New function. > (dw2_build_type_unit_groups): New function. > (dw2_get_file_names): Assert not called on type units. > (dw2_map_symtabs_matching_filename): Call dw2_build_type_unit_groups. > Redo loop to iterate over type unit groups instead of type units. > (dw2_expand_symtabs_matching, dw2_map_symbol_filenames): Ditto. > (read_abbrev_offset): New function. > (init_cutu_and_read_dies): New arg "abbrev_table". All callers > updated. > (create_partial_symtab): New function. > (process_psymtab_comp_unit_reader): Assert not used with type units. > Call create_partial_symtab. > (process_psymtab_type_unit): Delete. > (hash_type_unit_group, eq_type_unit_group): New functions. > (allocate_type_unit_groups_table): New function. > (NO_STMT_LIST_TYPE_UNIT_PSYMTAB): New macro. > (NO_STMT_LIST_TYPE_UNIT_PSYMTAB_SIZE): New macro. > (create_type_unit_group, get_type_unit_group): New functions. > (tu_abbrev_offset): New struct. > (sort_tu_by_abbrev_offset): New function. > (add_type_unit_group_to_table): New function. > (build_type_unit_groups): New function. > (build_type_psymtabs_reader): New function. > (build_type_psymtab_dependencies): New function. > (build_type_psymtabs): Rewrite. > (scan_partial_symbols): Flag an error if a DW_TAG_imported_unit > is seen in a type unit. > (process_queue): Move symtab expansion debugging printfs here. > Call process_full_type_unit for type units. > (compute_symtab_includes): Assert not called for type units. > (process_cu_includes): Don't call compute_symtab_includes for > type units. > (process_full_type_unit): New function. > (process_imported_unit_die): Flag an error if called for type units. > (handle_DW_AT_stmt_list): Delete arg "want_line_info". All callers > updated. Assert not called for type units. > (read_file_scope): Call dwarf2_start_symtab. > (setup_type_unit_groups): New function. > (read_type_unit_scope): Rewrite. > (abbrev_table_read_table): Initialize abbrev_table->offset. > (abbrev_table_free_cleanup): New function. > (dwarf2_start_symtab): New function. > (load_full_type_unit): Assert not called for type unit groups. > > Patches: > http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog. > diff?cvsroot=src&r1=1.14457&r2=1.14458 > http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/dwarf2read.c. > diff?cvsroot=src&r1=1.683&r2=1.684 Did this coommit fix the issue, and can it be closed? > Did this coommit fix the issue, and can it be closed? Based on comment #10, I'd say yes. |