Bug 30836 - ld.exe stuck while processing ctf symbols on 64bit Windows builds
Summary: ld.exe stuck while processing ctf symbols on 64bit Windows builds
Alias: None
Product: binutils
Classification: Unclassified
Component: libctf (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
Depends on:
Reported: 2023-09-08 12:54 UTC by Torbjörn SVENSSON
Modified: 2023-10-18 11:42 UTC (History)
0 users

See Also:
Last reconfirmed:

pr41893-1.o (671 bytes, application/x-object)
2023-09-08 12:54 UTC, Torbjörn SVENSSON
pr41893-2.o (529 bytes, application/x-object)
2023-09-08 12:55 UTC, Torbjörn SVENSSON

Note You need to log in before you can comment on or make changes to this bug.
Description Torbjörn SVENSSON 2023-09-08 12:54:27 UTC
Created attachment 15104 [details]

When running the GCC12 test suite for arm-none-eabi, I discovered that the processing in libctf jumps between 0 and INT_MAX when finding the "offset" in the function "ctf_dedup_rhash_type".
INT_MAX comes from that the function ctf_member_next is supposed to return negative value on error, but the function ctf_set_errno is defined to return an unsigned long and ctf_member_next is returning ssize_t that has a wider type on 64bit Windows ABI (extending 32bit unsigned to signed 64bit is not doing the expected here).

To reproduce the issue, use the attached object files (created using the pr41893-1.c and pr41893-2.c files from the GCC12 source tree), by running:

.../ld.exe -o foo.exe pr41893-1.o pr41893-2.o

Be sure that ld.exe is built using x86_64-w64-mingw32 triplet in order to trigger the issue.
Comment 1 Torbjörn SVENSSON 2023-09-08 12:55:26 UTC
Created attachment 15105 [details]
Comment 2 Torbjörn SVENSSON 2023-09-09 07:31:03 UTC
After playing a bit more with this, it's actually enough to only use the pr41893-1.o object file, no need for pr41893-1.o to trigger the fault.

Also, the object files were created using:

arm-none-eabi-gcc -c -mthumb -march=armv6s-m -mcpu=cortex-m0 -mfloat-abi=soft -gctf  pr41893-1.c
arm-none-eabi-gcc -c -mthumb -march=armv6s-m -mcpu=cortex-m0 -mfloat-abi=soft -gctf  pr41893-2.c

I've also shared a patch before that should fix this:
Comment 3 Sourceware Commits 2023-10-17 15:33:27 UTC
The master branch has been updated by Torbjorn Svensson <azoff@sourceware.org>:


commit 998a4f589d68503f79695f180fdf1742eeb0a39d
Author: Torbjörn SVENSSON <torbjorn.svensson@foss.st.com>
Date:   Wed Sep 13 11:02:36 2023 +0200

    libctf: Sanitize error types for PR 30836
    Made sure there is no implicit conversion between signed and unsigned
    return value for functions setting the ctf_errno value.
    An example of the problem is that in ctf_member_next, the "offset" value
    is either 0L or (ctf_id_t)-1L, but it should have been 0L or -1L.
    The issue was discovered while building a 64 bit ld binary to be
    executed on the Windows platform.
    Example object file that demonstrates the issue is attached in the PR.
            Affected functions adjusted.
    Signed-off-by: Torbjörn SVENSSON <torbjorn.svensson@foss.st.com>
    Co-Authored-By: Yvan ROUX <yvan.roux@foss.st.com>
Comment 4 Torbjörn SVENSSON 2023-10-17 15:37:38 UTC
Fixed by 998a4f589d68503f79695f180fdf1742eeb0a39d.
Comment 5 Sourceware Commits 2023-10-18 07:40:27 UTC
The master branch has been updated by Torbjorn Svensson <azoff@sourceware.org>:


commit 0f79aa900f3a69780dde1e934ffe21e30236934e
Author: Torbjörn SVENSSON <torbjorn.svensson@foss.st.com>
Date:   Tue Oct 17 20:36:50 2023 +0200

    libctf: Return CTF_ERR in ctf_type_resolve_unsliced PR 30836
    In commit 998a4f589d68503f79695f180fdf1742eeb0a39d, all but one return
    statement was updated to return the error proper value. This commit
    rectifies that missed return statement.
            ctf-types.c (ctf_type_resolve_unsliced): Return CTF_ERR on error.
    Signed-off-by: Torbjörn SVENSSON <torbjorn.svensson@foss.st.com>
Comment 6 Sourceware Commits 2023-10-18 11:42:22 UTC
The master branch has been updated by Nick Alcock <nix@sourceware.org>:


commit 29736fc507c7a9c6e797b7f83e8df4be73d37767
Author: Nick Alcock <nick.alcock@oracle.com>
Date:   Wed Oct 11 15:15:16 2023 +0100

    libctf: check for problems with error returns
    We do this as a writable test because the only known-affected platforms
    (with ssize_t longer than unsigned long) use PE, and we do not have support
    for CTF linkage in the PE linker yet.
            PR libctf/30836
            * libctf/testsuite/libctf-writable/libctf-errors.*: New test.