[PATCH] Fix problem with COFF and plugins

[Andrew Jenner <andrew at codesourcery dot com>]

Since the change 
https://www.sourceware.org/ml/binutils/2018-05/msg00166.html, LTO has 
been broken for mingw32 targets specifically (and probably other COFF 
targets as well). The error is "file not recognized: file format not 
recognized" for the object file that will be claimed by the plugin. I 
investigated, and what is going on is as follows:

The object file is recognized as target pe-i386 by the loop in 
bfd_check_format_matches(). However (since this change) we call 
_bfd_check_format with the plugin target as well. This causes the input 
BFD's plugin_format to be set to bfd_plugin_yes. However, because the 
target of the input file is not yet known, the target of the input BFD's 
plugin_dummy_bfd is set (via the fallback case in 
plugin_get_ir_dummy_bfd()) to link_info.output_bfd which in this case is 
pei-i386 (the output format) instead of pe-i386 (this is not a problem 
for ELF because the output and object formats are the same).

Then, when plugin_maybe_claim() is called, the real file's BFD is 
replaced with the dummy one (with the wrong format) and the file doesn't 
match the format.

There are a few different ways that this could be fixed, and I'm not 
sure which one is preferred. One would be to change 
bfd_check_format_matches() to not check the plugin format if we've 
already found a better format. I don't think that would affect the fix 
to PR22458.

Another possible fix (for which a patch is attached) is to reset 
plugin_format after the call to bfd_check_format, so that it is 
recomputed (this time with sufficient information) and the 
plugin_dummy_bfd's target is correct. We have done a test pass with this 
patch and it didn't cause any problems. Is this change okay to commit, 
or would a change to bfd_check_format_matches() be preferred?

Thanks,

Andrew


ld/ChangeLog

2018-06-10  Andrew Jenner  <andrew@codesourcery.com>

        * ldfile.c (ldfile_try_open_bfd): Reset plugin_format to
        bfd_plugin_unknown before calling plugin_maybe_claim.

Attachment: coff_plugins.patch
Description: Text document