The default for flag_compress_debug is none, except for gas/config/tc-i386.c which sets it to gabi. Since various tools cannot handle compressed debug sections yet it would be nice to be able to configure the default value to none (or gabi or gnu) for all arches.
(In reply to Mark Wielaard from comment #0) > The default for flag_compress_debug is none, except for gas/config/tc-i386.c > which sets it to gabi. Since various tools cannot handle compressed debug > sections yet it would be nice to be able to configure the default value to > none (or gabi or gnu) for all arches. A couple questions: 1. What are those affected tools? Will they be fixed? When? 2. What should be the default? 3. Should the new option also affect linker? If not, do we need separate options to control gas and ld/gold? Since zlib-gnu compression is deprecated, tools don't have to support it and it shouldn't be made default. My preference is to fix those tools. Otherwise, they may not ever be fixed.
in GCC itself, libbacktrace should get support for this.
(In reply to Matthias Klose from comment #2) > in GCC itself, libbacktrace should get support for this. How is libbacktrace affected by this?
Created attachment 8706 [details] Create a configure time option to control default debug section compression Hi Guys, Here is a patch that adds a configure option: --enable-default-compressed-debug-sections=[yes|no]. This option defaults to "no" for all targets except x86/Linux where it defaults to "yes". H.J. - do you have any problems with this patch ? Cheers Nick
(In reply to H.J. Lu from comment #1) > 1. What are those affected tools? Will they be fixed? When? I don't believe any non-binutils tools handle gabi compressed elf sections yet. In particular elfutils tools don't handle them. elfutils libdw does support gnu compressed debug sections for ET_EXEC and ET_DYN files, but not ET_REL files. elfutils libelf doesn't support any compressed section type currently. There is some work on supporting compressed sections for libelf, but that needs some coordination with other implementations and won't be in any release this year. > 2. What should be the default? none. > 3. Should the new option also affect linker? If not, do we need separate > options to control gas and ld/gold? I think the default should be the same for all tools, but making it possible to configure them with different defaults won't be bad.
(In reply to Nick Clifton from comment #4) > Here is a patch that adds a configure option: > --enable-default-compressed-debug-sections=[yes|no]. This option defaults > to "no" for all targets except x86/Linux where it defaults to "yes". Is there anything target specific about compressed debug sections? Why can't we make the default apply to all targets?
(In reply to Mark Wielaard from comment #5) > (In reply to H.J. Lu from comment #1) > > 1. What are those affected tools? Will they be fixed? When? > > I don't believe any non-binutils tools handle gabi compressed elf sections > yet. In particular elfutils tools don't handle them. elfutils libdw does > support gnu compressed debug sections for ET_EXEC and ET_DYN files, but not > ET_REL files. elfutils libelf doesn't support any compressed section type > currently. There is some work on supporting compressed sections for libelf, > but that needs some coordination with other implementations and won't be in > any release this year. Just to make it clear. Only tools dealing with DWARF info on ET_REL files are effected. There is no impact on tools dealing with ET_EXEC and ET_DYN files since linker compression is off by default. > > 2. What should be the default? > > none. > > > 3. Should the new option also affect linker? If not, do we need separate > > options to control gas and ld/gold? > > I think the default should be the same for all tools, but making it possible > to configure them with different defaults won't be bad. Do other tools, like valgrind and libbacktracce, support compressed ET_EXEC and ET_DYN files? If not, is anyone working on? I'd like to know if DWARF info compression in binutils will be ever be on by default. If no, why bother with a configure option? We just default it to "no". If yes, when will it happen?
Hi Mark, > Is there anything target specific about compressed debug sections? Yes and no. If the ABI for a given target does not support compressed debug sections then technically they might be considered to be breaking the ABI. Practically, if all of the tools that consume that target's binaries know about compressed debug sections, or don't care, then their presence should not matter. Of course this PR raises the issue that not all consumers of x86 linux binaries are aware of these sections. Hence one of the features of the patch: You can choose when configuring a particular target whether or not you want compressed debug sections to be the default. The patch also allows the binutils maintainers to set a default value for this configure option on a per-target basis. So as targets, and their tools, become compressed debug section aware the default for that target can be changed to "yes". > Why can't we make the default apply to all targets? I was trying to make a patch that preserved the status quo whilst at the same time allowing toolchain builders to choose whether or not compressed debug sections are the default. Currently x86 linux toolchains default to compressing debug sections. (My assumption here is that H.J. is pushing for the adoption of this technology, which is why he made this change). The patch preserves the current default behaviour of x86 linux toolchains, but it does allow individual toolchain builders, or package maintainers, to configure with --enable-default-compressed-debug-sections=no to turn this off. We could make compressing debug sections the default for all targets, but this is bound to break something. Better for now I think to default to normal debug sections for targets until the maintainers for those targets are sure that they are ready for compressed debug sections. > From H.J. > I'd like to know if DWARF info compression in binutils will be ever > be on by default. Yes, I expect this to happen. > If no, why bother with a configure option? We just > default it to "no". If yes, when will it happen? I do not know when. My guess is that the default will stay at "no" (for non x86 toolchains). Then slowly each target will change over their default to be "yes" and at some point, we will say, "this is silly - lets just make the default be yes and specify the targets that still need a default of no". Cheers Nick
Yes, I'd like to see compression is on by default for both gas and ld. It will take a long time to get other tools to understand compressed DWARF sections. But we have to start from somewhere. I chose to enable compressed DWARF sections by default in Linux/x86 assembler since it only impacts tools which work on DWARF sections in relocatable files. I am offering my help to update those tools by providing a standalone library which decompress ELFCOMPRESS_ZLIB ELF files.
I'd expect the option's default to be os-dependent, not arch-dependent.
On Mon, 12 Oct 2015, schwab@linux-m68k.org wrote: > I'd expect the option's default to be os-dependent, not arch-dependent. I agree, as I said in <https://sourceware.org/ml/binutils/2014-12/msg00154.html>.
(In reply to joseph@codesourcery.com from comment #11) > On Mon, 12 Oct 2015, schwab@linux-m68k.org wrote: > > > I'd expect the option's default to be os-dependent, not arch-dependent. > > I agree, as I said in > <https://sourceware.org/ml/binutils/2014-12/msg00154.html>. Agreed. Looking at that thread the unconditional enabling was somewhat controversial. Could we make it a target independent configure option that is off by default for the current release? Then people could first experiment with setting the default for gas generated ET_REL files to gnu or gabi and see which tools break and update them if possible. And experiment with how this impacts performance. Before enabling this by default (maybe in some next version if enough feedback is collected). BTW. How is the impact measured? It seems a positive for simple storage of the ET_REL files (assuming the compression always reduces the size). But how does it compare with just compressing the whole file, which is simpler to handle and would give larger size reduction. For exchanging data between as and ld or other tools dealing with ET_REL files it seems dependent on runtime factors. Basically memory/cache size vs cpu performance for compressing/decompressing. It prevents simply mmaping the ET_REL files and applying any relocations to the compressed sections. One always has to read and decompress before processing any such section. At which point is that a performance benefit? If the goal is to minimize the data needed to process by a linker tool then how does it compare to the GNU debug fission proposal (which might be standardized for DWARF 5) that simply makes sure there is less DWARF data seen by the linker by storing all non-relocatable debug sections in separate .dwo files.
(In reply to Mark Wielaard from comment #12) > Agreed. Looking at that thread the unconditional enabling was somewhat > controversial. Could we make it a target independent configure option that > is off by default for the current release? This is entirely possible with a tweak to the proposed patch. I would just like to get H.J. on board, or at least to agree to changing the default to on just in his binutils releases rather than the FSF mainline sources. Cheers Nick
(In reply to Nick Clifton from comment #13) > (In reply to Mark Wielaard from comment #12) > > > Agreed. Looking at that thread the unconditional enabling was somewhat > > controversial. Could we make it a target independent configure option that > > is off by default for the current release? > > This is entirely possible with a tweak to the proposed patch. I would just > like to get H.J. on board, or at least to agree to changing the default to > on just in his binutils releases rather than the FSF mainline sources. I'd like to see a plan and a timeline how we move forward with this.
Hi H.J. (In reply to H.J. Lu from comment #14) > I'd like to see a plan and a timeline how we move forward with this. OK, how about this: 1. We apply the patch, change the default to all targets back to not compressing debug sections. You add a patch to your binutils releases that re-enables compressed debug sections by default for x86/x86_64 linux targets. If there are any bug reports about problems with compressed debug sections we fix them as a matter of priority. 2. After the next major binutils release (2.26) we change the default to be compressed debug sections. We add an entry in the gas and ld NEWS files about this (noting how the default can also be changed by a configure option). Linux distro maintainers who use the development binutils sources will doubtless complain to us if they find problems with this change. If we can fix them, we do so, otherwise we revert the change on a per-target basis. 3. For the problem targets we maintain an eye on the issues, helping where we can, with the objective of having every major target supporting compressed debug sections by the time that 2.27 is released. Does this sound OK ? Cheers Nick
(In reply to Nick Clifton from comment #15) > Hi H.J. > > (In reply to H.J. Lu from comment #14) > > I'd like to see a plan and a timeline how we move forward with this. > > OK, how about this: > > 1. We apply the patch, change the default to all targets back to > not compressing debug sections. You add a patch to your binutils > releases that re-enables compressed debug sections by default for > x86/x86_64 linux targets. > > If there are any bug reports about problems with compressed debug > sections we fix them as a matter of priority. Do valgrind and libbacktrace support compressed debug sections? Do we have a plan to address those tools which read debug sections in shared library and executable? > 2. After the next major binutils release (2.26) we change the default > to be compressed debug sections. We add an entry in the gas and ld > NEWS files about this (noting how the default can also be changed > by a configure option). Linux distro maintainers who use the > development binutils sources will doubtless complain to us if they > find problems with this change. If we can fix them, we do so, > otherwise we revert the change on a per-target basis. I assume compression will only be turned on by default for Linux targets, not all ELF targets nor non-ELF targets. > 3. For the problem targets we maintain an eye on the issues, helping > where we can, with the objective of having every major target > supporting compressed debug sections by the time that 2.27 is > released. > > Does this sound OK ? My only concern is when we turn on compression by default in linker, we will run into problems if we don't fix those tools which read debug sections in shared library and executable. We need a way to make it less painful to enable compression support in those tools.
That does sound like a good plan for the assembler (lets do a separate discussion about the linker). I do like to see a way for os/target to evaluate the risks/costs/benefits before enabling the default in step 2. How and what will be measured? See also the questions asked in comment #12.
The master branch has been updated by Nick Clifton <nickc@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=e12fe5554c3b70139f68e33ded9e2a8075b9d484 commit e12fe5554c3b70139f68e33ded9e2a8075b9d484 Author: Nick Clifton <nickc@redhat.com> Date: Mon Oct 19 11:45:54 2015 +0100 Add a gas configure option to select the default behaviour for the generation of debug sections - compressed or uncompressed. PR gas/19109 * configure.ac: Add option --enable-compressed-debug-sections. This sets the default behaviour for compressing debug sections. * as.c (flag_compress_debug): Define and initialise to COMPRESS_DEBUG_GABI_ZLIB if DEFAULT_COMPRESS_DEBUG is set. (show_usage): Indicate whether --no-compress-debug-sections or --compress-debug-sections is the default. * config/tc-i386.c (flag_compress_debug): Delete definition. * doc/as.texinfo (--nocompress-debug-sectionas): Update description. * NEWS: Announce the new feature. * config.in: Regenerate. * configure: Regenerate.
(In reply to cvs-commit@gcc.gnu.org from comment #18) > The master branch has been updated by Nick Clifton <nickc@sourceware.org>: > > https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git; > h=e12fe5554c3b70139f68e33ded9e2a8075b9d484 > > commit e12fe5554c3b70139f68e33ded9e2a8075b9d484 > Author: Nick Clifton <nickc@redhat.com> > Date: Mon Oct 19 11:45:54 2015 +0100 > > Add a gas configure option to select the default behaviour for the > generation of debug sections - compressed or uncompressed. > > PR gas/19109 > * configure.ac: Add option --enable-compressed-debug-sections. > This sets the default behaviour for compressing debug sections. > * as.c (flag_compress_debug): Define and initialise to > COMPRESS_DEBUG_GABI_ZLIB if DEFAULT_COMPRESS_DEBUG is set. > (show_usage): Indicate whether --no-compress-debug-sections > or --compress-debug-sections is the default. > * config/tc-i386.c (flag_compress_debug): Delete definition. > * doc/as.texinfo (--nocompress-debug-sectionas): Update > description. > * NEWS: Announce the new feature. > * config.in: Regenerate. > * configure: Regenerate. Please rename --enable-compressed-debug-sections to make it clear that it is only for assembler, not linker.
Right - I have checked my patch in. H.J. - if you add a patch like this to your sources then x86/x86_64 linux targets will again default to compressing debug sections: diff --git a/gas/configure.ac b/gas/configure.ac index aa6bb1d..0394eda 100644 --- a/gas/configure.ac +++ b/gas/configure.ac @@ -457,6 +469,15 @@ changequote([,])dnl if test $this_target = $target ; then AC_DEFINE_UNQUOTED(DEFAULT_ARCH, "${arch}", [Default architecture.]) fi + + # For x86 Linux targets default to compressing + # debug sections unless configured otherwise. + cds=${ac_default_compressed_debug_sections}-${cpu_type}-${target_os} + case $cds in + unset-i386-linux-*) + ac_default_compressed_debug_sections=yes + ;; + esac ;; rl78) Once 2.26 is released we can add code like this for the other Linux targets as well. As far as the linker is concerned, I guess that the best thing to would be to have the same configure option work it too. (Currently this does not happen - the option only affects the assembler's default). Aside - is there any reason why someone would want the assembler and linker to behave differently ? Ie have one produce compressed debug sections, but the other not ? My plan is that once 2.26 is out, and we flip the default for Linux targets, we contact the binutils maintainers of as many distributions as we can find and let them know about the change. We can also offer help in migrating tools to support the new default, and suggestions as to how the new feature can be tested and evaluated. Cheers Nick
(In reply to H.J. Lu from comment #19) > Please rename --enable-compressed-debug-sections to make it clear that > it is only for assembler, not linker. How about --enable-assembler-compressed-debug-sections or --enable-compressed-debug-sections-in-assembler?
Hi H.J. > How about --enable-assembler-compressed-debug-sections or > --enable-compressed-debug-sections-in-assembler? Either is OK, but ... if we agree that we should also being configuring the linker to also generate compressed debug sections by default, then it would make sense to just have one configure option that applies to both tools. Cheers Nick
(In reply to Nick Clifton from comment #22) > Hi H.J. > > > How about --enable-assembler-compressed-debug-sections or > > --enable-compressed-debug-sections-in-assembler? > > Either is OK, but ... if we agree that we should also being configuring > the linker to also generate compressed debug sections by default, then > it would make sense to just have one configure option that applies to > both tools. > > Cheers > Nick --enable-compressed-debug-sections is no-op on linker at the moment. Do we want to change the --enable-compressed-debug-sections behavior on linker in the future? Should we do --enable-compressed-debug-sections=gas,ld or --with-compressed-debug-sections=gas,ld
Created attachment 8729 [details] A patch A patch to add --enable-compressed-debug-sections={all,gas,gold,ld} Enable compressed debug sections for gas, gold or ld by default
A patch is posted at https://sourceware.org/ml/binutils/2015-10/msg00165.html
Hi H.J. > A patch to add > > --enable-compressed-debug-sections={all,gas,gold,ld} > Enable compressed debug sections for gas, gold or ld > by default Before the patch goes in I would like to check a few things: * If compression is enabled for gas but not ld for example, does this mean that the linker will uncompress compressed debug sections when it writes them out ? Or rather, is that what configuring with --enable-compressed-debug-sections=gas implies ? * What happens if the linker is given some input files containing compressed debug sections and some input files not containing compressed debug sections ? Is the output always compressed ? * What advantage is gained by compressing debug sections, as opposed to splitting out the debug information and the compressing the whole debug file ? Cheers Nick
(In reply to Nick Clifton from comment #26) > Hi H.J. > > > A patch to add > > > > --enable-compressed-debug-sections={all,gas,gold,ld} > > Enable compressed debug sections for gas, gold or ld > > by default > > Before the patch goes in I would like to check a few things: > > * If compression is enabled for gas but not ld for example, > does this mean that the linker will uncompress compressed > debug sections when it writes them out ? Or rather, is Yes. > that what configuring with --enable-compressed-debug-sections=gas > implies ? Yes. > * What happens if the linker is given some input files containing > compressed debug sections and some input files not containing > compressed debug sections ? Is the output always compressed ? Linker always decompresses any compressed debug sections when reading input files and generates compressed debug sections in output when --enable-compressed-debug-sections=all or --enable-compressed-debug-sections=[gas,]ld[,gold] are used. > * What advantage is gained by compressing debug sections, as opposed > to splitting out the debug information and the compressing the whole > debug file ? How do you do that with .o files? As for executable/shared libraries, it should be decided by users. In my case, I can compress debug sections in as/ld and run them directly in gdb. I don't need to a separate step to extract/compress debug info. I have many different versions of binutils binaries laying around. Separate debug files will only lead to trouble for me.
OK - please apply your patch. Cheers Nick
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=edeefb679201858453502e21446d7a65fedeaf0e commit edeefb679201858453502e21446d7a65fedeaf0e Author: H.J. Lu <hjl.tools@gmail.com> Date: Tue Oct 20 08:58:10 2015 -0700 Add --enable-compressed-debug-sections={all,gas,gold,ld} This patch removes the gas configure option: --enable-compressed-debug-sections and adds a toplevel configure option: --enable-compressed-debug-sections={all,gas,gold,ld} to enable compressed debug sections for gas, gold or ld by default. At the moment, this configure option is ignored by gold and ld. For x86 Linux targets, default to compressing debug sections in gas. PR gas/19109 * configure.ac: Add --enable-compressed-debug-sections={all,gas,gold,ld}. * configure: Regenerated. gas/ PR gas/19109 * NEWS: Update --enable-compressed-debug-sections=. * configure.ac: Remove --enable-compressed-debug-sections. (DEFAULT_FLAG_COMPRESS_DEBUG): Check --enable-compressed-debug-sections={all,gas} instead of --enable-compressed-debug-sections. For x86 Linux targets, default to compressing debug sections. * configure: Regenerated.
Fixed.
The master branch has been updated by Nick Clifton <nickc@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=6c3bc0f82c625d425b09c01f96b64946abe4118e commit 6c3bc0f82c625d425b09c01f96b64946abe4118e Author: Nick Clifton <nickc@redhat.com> Date: Wed Oct 21 13:12:19 2015 +0100 Reset x86 Linux targets to not compressing debug sections by default. Enable compression of debug sections by default in the linker, if so configured. PR gas/19109 . * configure.ac: Note the 'none' is an acceptable argument to --enable-compressed-debug-sections. * configure: Regenerate. gas * configure.ac: Restore --enable-compressed-debug-sections. Do not enable compressed debug sections by default for x86 Linux targets. * configure: Regenerate. ld * configure.ac: Add --enable-compressed-debug-sections. * configure: Regenerate. * config.in: Regenerate. * ld.texinfo: Document how to determine the default action for debug sections. * ldmain.c (main): If DEFAULT_FLAG_COMPRESS_DEBUG is defined then set the compress_debug field of the link_info structure to zlib-gabi. * lexsup.c (elf_static_list_options): Output the default setting for the --compress-debug-sections option. * NEWS: Mention the new configure option.
Hi H.J. I have checked in a supplimentary patch. This does three things: * It default x86 Linux gas to not compressing debug sections. This was the whole point of this PR. We can enable compressed debug sections for x86 Linux after the 2.26 release has been made. Plus of course you can enable it in your own binutils releases. * It fixes gas so that the --enable-compressed-debug-sections option inherited from the top level will be recognized and acted upon. * It updates the linker so the --enable-compressed-debug-section option will also be recognized and acted upon. Cheers Nick
(In reply to Nick Clifton from comment #32) > Hi H.J. > > I have checked in a supplimentary patch. This does three things: > > * It default x86 Linux gas to not compressing debug sections. > This was the whole point of this PR. We can enable compressed > debug sections for x86 Linux after the 2.26 release has been made. > Plus of course you can enable it in your own binutils releases. > > * It fixes gas so that the --enable-compressed-debug-sections > option inherited from the top level will be recognized and acted > upon. > > * It updates the linker so the --enable-compressed-debug-section > option will also be recognized and acted upon. > > Cheers > Nick The new configure change between topevel and gas/ld is inconsistent. Toplevel configure has --enable-compressed-debug-sections={all,gas,gold,ld,none} and gas/ld has --enable-compressed-debug-sections. Toplevel configure should take --enable-compressed-debug-sections=all --enable-compressed-debug-sections=gas --enable-compressed-debug-sections=gold --enable-compressed-debug-sections=ld --enable-compressed-debug-sections=ld,gold --enable-compressed-debug-sections=ld,gas What should --enable-compressed-debug-sections without list do at toplevel?
Hi H.J. > The new configure change between topevel and gas/ld is inconsistent. > Toplevel configure has > > --enable-compressed-debug-sections={all,gas,gold,ld,none} > > and gas/ld has --enable-compressed-debug-sections. OK this was a typo. The gas and ld configure options should accept a subset of the top level configure options. I have checked in the obvious fix for this. Cheers Nick
(In reply to Nick Clifton from comment #34) > Hi H.J. > > > The new configure change between topevel and gas/ld is inconsistent. > > Toplevel configure has > > > > --enable-compressed-debug-sections={all,gas,gold,ld,none} > > > > and gas/ld has --enable-compressed-debug-sections. > > OK this was a typo. The gas and ld configure options should accept a subset > of the top level configure options. I have checked in the obvious fix for > this. > > Cheers > Nick Should --enable-compressed-debug-sections=ld,gas work? If not, should --enable-compressed-debug-sections=ld also apply to gold?
Hi H.J. > Should --enable-compressed-debug-sections=ld,gas work? As opposed to --enable-compressed-debug-sections=all ? Hmm, yes I guess that it should. Although it will make parsing this option trickier. > If not, > should --enable-compressed-debug-sections=ld also apply to gold? Well I have not even implemented the configure option for gold yet, but I would have to say no. We are explicitly stating which tools will compress debug sections by default on the configure command line and we are treating ld as separate from gold. Cheers Nick
(In reply to Nick Clifton from comment #36) > Hi H.J. > > > Should --enable-compressed-debug-sections=ld,gas work? > > As opposed to --enable-compressed-debug-sections=all ? Hmm, yes I guess > that it should. Although it will make parsing this option trickier. > We should support combination and permutation of gas, ld, gold. We can borrow fragment from --enable-languages=LIST.
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=4894d80b508ad2626eb2cfd23cfcb5635cdf7442 commit 4894d80b508ad2626eb2cfd23cfcb5635cdf7442 Author: H.J. Lu <hjl.tools@gmail.com> Date: Thu Oct 22 05:18:21 2015 -0700 Handle --enable-compressed-debug-sections=ld,gas gas/ PR gas/19109 * configure.ac: Handle --enable-compressed-debug-sections=*,gas,*. * configure: Regenerated. ld/ PR gas/19109 * configure.ac: Handle --enable-compressed-debug-sections=*,ld,*. * configure: Regenerated.
.
If binutils adds zstd (PR29397), the option --enable-compressed-debug-sections= needs to be adjusted:)
(In reply to Fangrui Song from comment #40) > If binutils adds zstd (PR29397), the option > --enable-compressed-debug-sections= needs to be adjusted:) Well, this option is telling for which tools do we want to use compression and we will likely need --default-compressed-debug-sections-algorithm that can be any of the currently supported options. Lemme prepare a patch for it.
(In reply to Martin Liska from comment #41) > (In reply to Fangrui Song from comment #40) > > If binutils adds zstd (PR29397), the option > > --enable-compressed-debug-sections= needs to be adjusted:) > > Well, this option is telling for which tools do we want to use compression > and we will likely need --default-compressed-debug-sections-algorithm that > can be any of the currently supported options. Lemme prepare a patch for it. I take that back, using --enable-zstd-compressed-debug-sections boolean configure option would be much simpler.