bfd: read/write header characteristics of Win32/PE executable

[Sean McNamara <smcnam at gmail dot com>]

Hello,

I am new to developing with bfd, but I would like to write an
application that accepts a Win32/PE executable as input, modifies
certain PE headers (particularly, the Large Address Aware bit), then
writes out the executable, _without_ re-linking the program. If the
linker gets involved, then presumably we have the source code, or at
least the object files and the link-time dependencies needed to
produce the executable, which means we can use GNU ld or Visual Studio
to do the link and set LAA, which makes this entire project moot. So,
this project is about performing this header tweak "in-situ" without
the involvement of a linker.

The change I want to make is so simple that it only changes one 32-bit
word in the file when the Visual Studio program editbin.exe makes the
change. But the location of these bits and the value to set them to is
variable, depending on the size of some structures and the existing
flags, so I was hoping to use BFD to make this change. Evidenced by
the behavior of editbin.exe, it is possible to perform this change if
one knows how the PE file is structured, and from that, determine
where to set the flag -- all without modifying the rest of the
executable.

I have been using the bfd source code and documentation to try to
understand how to do this, but most of the COFF/PE stuff is documented
in the source, not in the binutils docs. The only user of BFD that
appears to do what I want is in GNU ld, in the file pe.em:

case OPTION_LARGE_ADDRESS_AWARE:
      real_flags |= IMAGE_FILE_LARGE_ADDRESS_AWARE;

This seems to be the place where GNU ld parses the option
"--large-address-aware" and translates that into a flag set by BFD. I
started researching BFD from this, and inferred that eventually I need
to do something like this in my app:

    abfd->tdata.pe_obj_data->real_flags |= IMAGE_FILE_LARGE_ADDRESS_AWARE;

I managed to drag in the required headers from include/ and bfd/ to
get this to build, but when I call bfd_close(), no changes are made to
the resulting executable.

Currently I am opening the executable with bfd_fopen in "rw" mode,
hoping that I could just make this one change and write it back out,
without having to open the file for reading, create a new BFD for
writing, and somehow (?) copy the contents of the one into the other.

If it would be non-trivial to do this without mangling the input
executable, I suppose I will have to look elsewhere. Otherwise, please
provide some tips for how I could do this.

My second problem (which appears to be an actual bug) is that, when
reading a Win32/PE file into BFD, the large address aware flag is not
accounted for by BFD. It appears to default to being set (the flag
value turns out to be 0x7865742e no matter what executable you run it
on, which does return nonzero when you AND it with 0x0020, the value
of IMAGE_FILE_LARGE_ADDRESS_AWARE). It is set even on executables that
verifiably are not large address aware, so the header is not actually
being checked before BFD constructs its internal representation of the
file. Seems BFD can only write out this flag (as evidenced by GNU ld's
ability to produce executables that are large address aware), but
cannot determine whether executables support LAA.

However, determining whether LAA is enabled is only a secondary
concern. My ultimate goal is to be able to _set_ the LAA flag. I have
a feeling, though, that somehow my problems are related...

My current non-working code for setting the LAA bit, a program called
bfdlaa, does not crash or report any errors, but it doesn't do
anything to the executable, either -- the output is bitwise identical
to the input.

I also wrote a very similar second program, laacheck, which does the
opposite of bfdlaa: it tries to determine whether an existing
executable is LAA-enabled or not, and prints the result to the
console. Theoretically laacheck and bfdlaa will be merged into a
single program once this all works.

The code: http://tiyukquellmalz.org/bfdlaa.tar.bz2

Tarball contents:

bfdlaa.c: Source to the main application. Contains several
commented-out things I tried. Syntax: ./bfdlaa
<path-to-win32-pe-executable.exe>
laacheck.c: Source to the checker application. Syntax: ./laacheck
<path-to-win32-executable.exe>
test.c: A very trivial C program, to be compiled by MinGW to create test.exe.
test.exe: test.c compiled with MinGW32 gcc-4.5. NOT Large Address Aware.
testlaa.exe: test.c compiled with MinGW32 gcc-4.5. This one _is_ Large
Address Aware, as confirmed by the Visual Studio utility, dumpbin.
Makefile: run `make' to compile the application, `make test' to
compile the test apps (if you have MinGW32), and `make test-exec' to
run laacheck on each of the test apps. Run `make mingw32' if you want
to build bfdlaa and laacheck for Win32, otherwise they'll be built
with your system gcc (e.g. producing a Linux ELF binary).
All other files: From binutils-2.20.1.

It seems like the breakage in bfdlaa is either my fault (wrong coding
/ wrong approach), or a fact about BFD being completely unsuitable for
what I want to do. The breakage in laacheck, though, is definitely
something that I think can be addressed, since my code appears
correct.

Thanks,

Sean