[PATCH] Fix problem with COFF and plugins

[Andrew Jenner]

Since the change 
https://www.sourceware.org/ml/binutils/2018-05/msg00166.html, LTO has 
been broken for mingw32 targets specifically (and probably other COFF 
targets as well). The error is "file not recognized: file format not 
recognized" for the object file that will be claimed by the plugin. I 
investigated, and what is going on is as follows:

The object file is recognized as target pe-i386 by the loop in 
bfd_check_format_matches(). However (since this change) we call 
_bfd_check_format with the plugin target as well. This causes the input 
BFD's plugin_format to be set to bfd_plugin_yes. However, because the 
target of the input file is not yet known, the target of the input BFD's 
plugin_dummy_bfd is set (via the fallback case in 
plugin_get_ir_dummy_bfd()) to link_info.output_bfd which in this case is 
pei-i386 (the output format) instead of pe-i386 (this is not a problem 
for ELF because the output and object formats are the same).

Then, when plugin_maybe_claim() is called, the real file's BFD is 
replaced with the dummy one (with the wrong format) and the file doesn't 
match the format.

There are a few different ways that this could be fixed, and I'm not 
sure which one is preferred. One would be to change 
bfd_check_format_matches() to not check the plugin format if we've 
already found a better format. I don't think that would affect the fix 
to PR22458.

Another possible fix (for which a patch is attached) is to reset 
plugin_format after the call to bfd_check_format, so that it is 
recomputed (this time with sufficient information) and the 
plugin_dummy_bfd's target is correct. We have done a test pass with this 
patch and it didn't cause any problems. Is this change okay to commit, 
or would a change to bfd_check_format_matches() be preferred?

Thanks,

Andrew


ld/ChangeLog

2018-06-10  Andrew Jenner  <andrew@codesourcery.com>

         * ldfile.c (ldfile_try_open_bfd): Reset plugin_format to
         bfd_plugin_unknown before calling plugin_maybe_claim.

-------------- next part --------------
diff --git a/ld/ldfile.c b/ld/ldfile.c
index a72ff13..eab0740 100644
--- a/ld/ldfile.c
+++ b/ld/ldfile.c
@@ -315,7 +315,13 @@ success:
   if (link_info.lto_plugin_active
       && !no_more_claiming
       && bfd_check_format (entry->the_bfd, bfd_object))
-    plugin_maybe_claim (entry);
+    {
+      /* The target of the plugin_dummy_bfd may have been
+	 determined based on incomplete information.  Force
+	 it to be recomputed now that the target is known.  */
+      entry->the_bfd->plugin_format = bfd_plugin_unknown;
+      plugin_maybe_claim (entry);
+    }
 #endif /* ENABLE_PLUGINS */
 
   /* It opened OK, the format checked out, and the plugins have had