x86: Move strcmp SSE2 implementation to multiarch/strcmp-sse2.S
This commit doesn't affect libc.so.6, its just housekeeping to prepare
for adding explicit ISA level support.
Because strcmp-sse2.S implements so many functions (more from
avx2/evex/sse42) add a new file 'strcmp-naming.h' to assist in
getting the correct symbol name for all the function across
multiarch/non-multiarch builds.
Tested build on x86_64 and x86_32 with/without multiarch.
x86: Rename STRCASECMP_NONASCII macro to STRCASECMP_L_NONASCII
The previous macro name can be confusing given that both
`__strcasecmp_l_nonascii` and `__strcasecmp_nonascii` are
functions and we use the `_l` version.
x86: Remove generic strncat, strncpy, and stpncpy implementations
These functions all have optimized versions:
__strncat_sse2_unaligned, __strncpy_sse2_unaligned, and
stpncpy_sse2_unaligned which are faster than their respective generic
implementations. Since the sse2 versions can run on baseline x86_64,
we should use these as the baseline implementation and can remove the
generic implementations.
Geometric mean of N=20 runs of the entire benchmark suite on:
11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz (Tigerlake)
Tom Honermann [Thu, 30 Jun 2022 12:52:15 +0000 (08:52 -0400)]
stdlib: Tests for mbrtoc8, c8rtomb, and the char8_t typedef.
This change adds tests for the mbrtoc8 and c8rtomb functions adopted for
C++20 via WG21 P0482R6 and for C2X via WG14 N2653, and for the char8_t
typedef adopted for C2X from WG14 N2653.
The tests for mbrtoc8 and c8rtomb specifically exercise conversion to
and from Big5-HKSCS because of special cases that arise with that encoding.
Big5-HKSCS defines some double byte sequences that convert to more than
one Unicode code point. In order to test this, the locale dependencies
for running tests under wcsmbs is expanded to include zh_HK.BIG5-HKSCS.
Tom Honermann [Thu, 30 Jun 2022 12:52:14 +0000 (08:52 -0400)]
stdlib: Implement mbrtoc8, c8rtomb, and the char8_t typedef.
This change provides implementations for the mbrtoc8 and c8rtomb
functions adopted for C++20 via WG21 P0482R6 and for C2X via WG14
N2653. It also provides the char8_t typedef from WG14 N2653.
The mbrtoc8 and c8rtomb functions are declared in uchar.h in C2X
mode or when the _GNU_SOURCE macro or C++20 __cpp_char8_t feature
test macro is defined.
The char8_t typedef is declared in uchar.h in C2X mode or when the
_GNU_SOURCE macro is defined and the C++20 __cpp_char8_t feature
test macro is not defined (if __cpp_char8_t is defined, then char8_t
is a builtin type).
Tom Honermann [Thu, 30 Jun 2022 12:52:13 +0000 (08:52 -0400)]
gconv: Correct Big5-HKSCS conversion to preserve all state bits. [BZ #25744]
This patch corrects the Big5-HKSCS converter to preserve the lowest 3 bits of
the mbstate_t __count data member when the converter encounters an incomplete
multibyte character.
Danila Kutenin [Mon, 27 Jun 2022 16:12:13 +0000 (16:12 +0000)]
aarch64: Optimize string functions with shrn instruction
We found that string functions were using AND+ADDP
to find the nibble/syndrome mask but there is an easier
opportunity through `SHRN dst.8b, src.8h, 4` (shift
right every 2 bytes by 4 and narrow to 1 byte) and has
same latency on all SIMD ARMv8 targets as ADDP. There
are also possible gaps for memcmp but that's for
another patch.
We see 10-20% savings for small-mid size cases (<=128)
which are primary cases for general workloads.
Xi Ruoyao [Tue, 28 Jun 2022 10:44:03 +0000 (18:44 +0800)]
test-container: return UNSUPPORTED for ENOSPC on clone()
Since Linux 4.9, the kernel provides
/proc/sys/user/max_{mnt,pid,user}_namespace as a limitation of number of
namespaces. Some distros (for example, Slint Linux 14.2.1) set them (or
only max_user_namespace) to zero as a "security policy" for disabling
namespaces.
The clone() call will set errno to ENOSPC under such a limitation. We
didn't check ENOSPC in the code so the test will FAIL, and report:
unable to unshare user/fs: No space left on device
This message is, unfortunately, very unhelpful. It leads people to
check the memory or disk space, instead of finding the real issue.
To improve the situation, we should check for ENOSPC and return
UNSUPPORTED as the test result. Also refactor check_for_unshare_hints()
to emit a proper message telling people how to make the test work, if
they really need to run the namespaced tests.
Reported-by: Philippe Delavalade <philippe.delavalade@orange.fr>
URL: https://lists.linuxfromscratch.org/sympa/arc/lfs-support/2022-06/msg00022.html Signed-off-by: Xi Ruoyao <xry111@xry111.site> Reviewed-by: DJ Delorie <dj@redhat.com>
Noah Goldstein [Wed, 29 Jun 2022 23:11:22 +0000 (16:11 -0700)]
x86: Add support for building {w}memcmp{eq} with explicit ISA level
1. Refactor files so that all implementations are in the multiarch
directory
- Moved the implementation portion of memcmp sse2 from memcmp.S to
multiarch/memcmp-sse2.S
- The non-multiarch file now only includes one of the
implementations in the multiarch directory based on the compiled
ISA level (only used for non-multiarch builds. Otherwise we go
through the ifunc selector).
2. Add ISA level build guards to different implementations.
- I.e memcmp-avx2-movsb.S which is ISA level 3 will only build if
compiled ISA level <= 3. Otherwise there is no reason to include
it as we will always use one of the ISA level 4
implementations (memcmp-evex-movbe.S).
3. Add new multiarch/rtld-{w}memcmp{eq}.S that just include the
non-multiarch {w}memcmp{eq}.S which will in turn select the best
implementation based on the compiled ISA level.
4. Refactor the ifunc selector and ifunc implementation list to use
the ISA level aware wrapper macros that allow functions below the
compiled ISA level (with a guranteed replacement) to be skipped.
Tested with and without multiarch on x86_64 for ISA levels:
{generic, x86-64-v2, x86-64-v3, x86-64-v4}
Noah Goldstein [Wed, 29 Jun 2022 23:07:06 +0000 (16:07 -0700)]
x86: Add support for building {w}memset{_chk} with explicit ISA level
1. Refactor files so that all implementations are in the multiarch
directory
- Moved the implementation portion of memset sse2 from memset.S to
multiarch/memset-sse2.S
- The non-multiarch file now only includes one of the
implementations in the multiarch directory based on the compiled
ISA level (only used for non-multiarch builds. Otherwise we go
through the ifunc selector).
2. Add ISA level build guards to different implementations.
- I.e memset-avx2-unaligned-erms.S which is ISA level 3 will only
build if compiled ISA level <= 3. Otherwise there is no reason
to include it as we will always use one of the ISA level 4
implementations (memset-evex-unaligned-erms.S).
3. Add new multiarch/rtld-memset.S that just include the
non-multiarch memset.S which will in turn select the best
implementation based on the compiled ISA level.
4. Refactor the ifunc selector and ifunc implementation list to use
the ISA level aware wrapper macros that allow functions below the
compiled ISA level (with a guranteed replacement) to be skipped.
Tested with and without multiarch on x86_64 for ISA levels:
{generic, x86-64-v2, x86-64-v3, x86-64-v4}
x86: Add support for building {w}memmove{_chk} with explicit ISA level
1. Refactor files so that all implementations are in the multiarch
directory
- Moved the implementation portion of memmove sse2 from memmove.S
to multiarch/memmove-sse2.S
- The non-multiarch file now only includes one of the
implementations in the multiarch directory based on the compiled
ISA level (only used for non-multiarch builds. Otherwise we go
through the ifunc selector).
2. Add ISA level build guards to different implementations.
- I.e memmove-avx2-unaligned-erms.S which is ISA level 3 will only
build if compiled ISA level <= 3. Otherwise there is no reason
to include it as we will always use one of the ISA level 4
implementations (memmove-evex-unaligned-erms.S).
3. Add new multiarch/rtld-memmove.S that just include the
non-multiarch memmove.S which will in turn select the best
implementation based on the compiled ISA level.
4. Refactor the ifunc selector and ifunc implementation list to use
the ISA level aware wrapper macros that allow functions below the
compiled ISA level (with a guranteed replacement) to be skipped.
Tested with and without multiarch on x86_64 for ISA levels:
{generic, x86-64-v2, x86-64-v3, x86-64-v4}
And m32 with and without multiarch.
isa raising memmove
x86: Add support for building str{c|p}{brk|spn} with explicit ISA level
The changes for these functions are different than the others because
the best implementation (sse4_2) requires the generic
implementation as a fallback to be built as well.
Changes are:
1. Add non-multiarch functions for str{c|p}{brk|spn}.c to statically
select the best implementation based on the configured ISA build
level.
2. Add stubs for str{c|p}{brk|spn}-generic and varshift.c to in the
sysdeps/x86_64 directory so that the the sse4 implementation will
have all of its dependencies for the non-multiarch / rtld build
when ISA level >= 2.
3. Add new multiarch/rtld-strcspn.c that just include the
non-multiarch strcspn.c which will in turn select the best
implementation based on the compiled ISA level.
4. Refactor the ifunc selector and ifunc implementation list to use
the ISA level aware wrapper macros that allow functions below the
compiled ISA level (with a guranteed replacement) to be skipped.
Tested with and without multiarch on x86_64 for ISA levels:
{generic, x86-64-v2, x86-64-v3, x86-64-v4}
Replace __libc_multiple_threads with __libc_single_threaded
And also fixes the SINGLE_THREAD_P macro for SINGLE_THREAD_BY_GLOBAL,
since header inclusion single-thread.h is in the wrong order, the define
needs to come before including sysdeps/unix/sysdep.h. The macro
is now moved to a per-arch single-threade.h header.
The SINGLE_THREAD_P is used on some more places.
Checked on aarch64-linux-gnu and x86_64-linux-gnu.
It was added on Linux 5.12 (2a1867219c7b27f928e2545782b86daaf9ad50bd)
to allow change the properties of a mount or a mount tree using file
descriptors which the new mount api is based on.
It was added on Linux 5.2 (cf3cba4a429be43e5527a3f78859b1bfd9ebc5fb)
that can be used to pick an existing mountpoint into an filesystem
context which can thereafter be used to reconfigure a superblock
with fsconfig syscall.
It was added on Linux 5.2 (ecdab150fddb42fe6a739335257949220033b782)
as a way to a configure filesystem creation context and trigger
actions upon it, to be used in conjunction with fsopen, fspick and
fsmount.
The fsconfig_command commands are currently only defined as an enum,
so they can't be checked on tst-mount-consts.py with current test
support.
AArch64: Reset HWCAP2_AFP bits in FPCR for default fenv
The AFP feature (Alternate floating-point behavior) was added in armv8.7 and
introduced new FPCR bits.
Currently, HWCAP2_AFP bits (bit 0, 1, 2) in FPCR are preserved when fenv is
set to default environment. This is a deviation from standard behaviour.
Clear these bits when setting the fenv to default.
There is no libc API to modify the new FPCR bits. Restoring those bits matters
if the user changed them directly.
elf: Fix direction of NODELETE log messages during symbol lookup
NODELETE status is propagated from the referencing object to the
referenced object, not the other way round. The code is correct, only
the log message has the wrong direction.
stdlib: Simplify buffer management in canonicalize
Move the buffer management from realpath_stk to __realpath. This
allows returning directly after allocation errors.
Always make a copy of the result buffer using strdup even if it is
already heap-allocated. (Heap-allocated buffers are somewhat rare.)
This avoids GCC warnings at certain optimization levels.
locale: localdef input files are now encoded in UTF-8
Previously, they were assumed to be in ISO-8859-1, and that the output
charset overlapped with ISO-8859-1 for the characters actually used.
However, this did not work as intended on many architectures even for
an ISO-8859-1 output encoding because of the char signedness bug in
lr_getc. Therefore, this commit switches to UTF-8 without making
provisions for backwards compatibility.
The following Elisp code can be used to convert locale definition files
to UTF-8:
The array lr->buf contains characters, which can be signed. A 0xff
byte in the input could be incorrectly reported as EOF. More
importantly, get_string in linereader.c converts a signed input byte
to a Unicode code point using ADDWC ((uint32_t) ch), under the
assumption that this decodes the ISO-8859-1 input encoding. If char
is signed, this does not give the correct result. This means that
ISO-8859-1 input files for localedef are not actually supported,
contrary to the comment in get_string. This is a happy accident because
we can therefore change the file encoding to UTF-8 without impacting
backwards compatibility.
While at it, remove the \32 check for MS-DOS end-of-file character (^Z).
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
Fangrui Song [Tue, 5 Jul 2022 04:15:51 +0000 (21:15 -0700)]
libc-symbols.h: remove unused macros
Beside weak_hidden_alias/declare_symbol_alias/hidden_data_ver, many
*_hidden_* macros are removed. If there is a rare need to use one, one
may write something like `#if IS_IN (libm)\nhidden_def (...)\n#endif`
instead.
The headers mempcpy.h, strcase.h, strchrnul.h, and strndup.h are
included if not building argp for glibc. Commit c5af724c0b214a517f8558887f7a70efcfa2c813 added them in 2003 for gnulib,
but gnulib's current master patches them out:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/argp-namefrob.h;h=9c82ac79c215540f986c3e04398edba3ba1b7234;hb=HEAD
The main drive is to optimize the internal usage and required size
when sigset_t is embedded in other data structures. On Linux, the
current supported signal set requires up to 8 bytes (16 on mips),
was lower than the user defined sigset_t (128 bytes).
A new internal type internal_sigset_t is added, along with the
functions to operate on it similar to the ones for sigset_t. The
internal-signals.h is also refactored to remove unused functions
Besides small stack usage on some functions (posix_spawn, abort)
it lower the struct pthread by about 120 bytes (112 on mips).
Kito Cheng [Tue, 28 Jun 2022 13:52:19 +0000 (21:52 +0800)]
riscv: Use memcpy to handle unaligned access when fixing R_RISCV_RELATIVE
Although RISC-V Linux will enable the unaligned memory access handler by
default, that is quite expensive in general, using memcpy will be much cheaper
- just break down that into several load/store byte instructions.
Tejas Belagod [Mon, 27 Jun 2022 18:00:50 +0000 (18:00 +0000)]
AArch64: Add asymmetric faulting mode for tag violations in mem.tagging tunable
The new asymmetric mode is available when HWCAP2_MTE3 is set (support is
available), bit2 is set in the tunable (user request per application),
and the system is configured such that the asymmetric mode is preferred over
sync or async (per-cpu system-wide setting).
Reviewed-by: Szabolcs Nagy <szabolcs.nagy@arm.com>
linux: Fix mq_timereceive check for 32 bit fallback code (BZ 29304)
On success, mq_receive() and mq_timedreceive() return the number of
bytes in the received message, so it requires to check if the value
is larger than 0.
Noah Goldstein [Tue, 28 Jun 2022 15:26:26 +0000 (08:26 -0700)]
x86-64: Small improvements to dl-trampoline.S
1. Remove sse2 instructions when using the avx512 or avx version.
2. Fixup some format nits in how the address offsets where aligned.
3. Use more space efficient instructions in the conditional AVX
restoral.
- vpcmpeqq -> vpcmpeqb
- cmp imm32, r; jz -> inc r; jz
4. Use `rep movsb` instead of `rep movsq`. The former is guranteed to
be fast with the ERMS flags, the latter is not. The latter also
wastes an instruction in size setup.
Noah Goldstein [Wed, 29 Jun 2022 23:07:15 +0000 (16:07 -0700)]
x86: Move mem{p}{mov|cpy}_{chk_}erms to its own file
The primary memmove_{impl}_unaligned_erms implementations don't
interact with this function. Putting them in same file both
wastes space and unnecessarily bloats a hot code section.
Noah Goldstein [Wed, 29 Jun 2022 23:07:05 +0000 (16:07 -0700)]
x86: Move and slightly improve memset_erms
Implementation wise:
1. Remove the VZEROUPPER as memset_{impl}_unaligned_erms does not
use the L(stosb) label that was previously defined.
2. Don't give the hotpath (fallthrough) to zero size.
Code positioning wise:
Move memset_{chk}_erms to its own file. Leaving it in between the
memset_{impl}_unaligned both adds unnecessary complexity to the
file and wastes space in a relatively hot cache section.
Arjun Shankar [Wed, 29 Jun 2022 22:37:34 +0000 (00:37 +0200)]
linux: Remove unnecessary nice.c and signal.c
These files simply include the sysdeps/posix implementations which would
be used even in the absence of the files. They have been unnecessary
since 7b17aeda0c5e when nice and signal were removed from the
syscalls.list file.
Florian Weimer [Wed, 29 Jun 2022 11:50:50 +0000 (13:50 +0200)]
Linux: Forward declaration of struct iovec for process_madvise
This maintains compatibility between <sys/mman.h> and <linux/uio.h>.
Before that, the addition of process_madvise made those two header
files incompatible. This has been observed resulting in a build
failure in LLDB's Process/Linux/NativeRegisterContextLinux_s390x.cpp
source file.
H.J. Lu [Mon, 27 Jun 2022 18:36:28 +0000 (11:36 -0700)]
x86-64: Only define used SSE/AVX/AVX512 run-time resolvers
When glibc is built with x86-64 ISA level v3, SSE run-time resolvers
aren't used. For x86-64 ISA level v4 build, both SSE and AVX resolvers
are unused. Check the minimum x86-64 ISA level to exclude the unused
run-time resolvers.
Noah Goldstein [Fri, 24 Jun 2022 23:15:42 +0000 (16:15 -0700)]
x86: Fix backwards Prefer_No_VZEROUPPER check in ifunc-evex.h
Add third argument to X86_ISA_CPU_FEATURES_ARCH_P macro so the runtime
CPU_FEATURES_ARCH_P check can be inverted if the
MINIMUM_X86_ISA_LEVEL is not high enough to constantly evaluate
the check.
Use this new macro to correct the backwards check in ifunc-evex.h
Fangrui Song [Sun, 26 Jun 2022 22:31:19 +0000 (15:31 -0700)]
Makerules: Remove no-op -Wl,-d when linking libc_pic.os
In GNU ld, -d assigns space to common symbols for -r (i.e. change common
symbols to STB_GLOBAL definitions). This option was added in commit da2d1bc5adf49352232ad0514e79fbd5dcae08e8 (1998) perhaps because ld at
that time had a bug that common symbols did not override shared object
definitions. -d has been long unneeded and more so since -fno-common
was added to +cflags.
misc: Optimize internal usage of __libc_single_threaded
By adding an internal alias to avoid the GOT indirection.
On some architecture, __libc_single_thread may be accessed through
copy relocations and thus it requires to update also the copies
default copy.
This is done by adding a new internal macro,
libc_hidden_data_{proto,def}, which has an addition argument that
specifies the alias name (instead of default __GI_ one).
It was added on Linux 5.2 (2db154b3ea8e14b04fee23e3fdfd5e9d17fbc6ae)
as way t move a mount from one place to another and, in the next
commit, allow to attach an unattached mount tree.
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
It was added on 5.2 (93766fbd2696c2c4453dd8e1070977e9cd4e6b6d) to
provide a way by which a filesystem opened with fsopen and configured
by a series of fsconfig calls can have a detached mount object
created for it.
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
It was added on Linux 5.2 (24dcb3d90a1f67fe08c68a004af37df059d74005)
to start the process of preparing to create a superblock that will
then be mountable, using an fd as a context handle.
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Fri, 24 Jun 2022 17:38:14 +0000 (19:38 +0200)]
resolv/tst-resolv-noaaaa: Support building for older C standards
This avoids a compilation error:
tst-resolv-noaaaa.c: In function 'response':
tst-resolv-noaaaa.c:74:11: error: a label can only be part of a statement and a declaration is not a statement
char ipv4[4] = {192, 0, 2, i + 1};
^~~~
tst-resolv-noaaaa.c:79:11: error: a label can only be part of a statement and a declaration is not a statement
char *name = xasprintf ("ptr-%d", i);
^~~~
Noah Goldstein [Thu, 23 Jun 2022 17:49:19 +0000 (10:49 -0700)]
x86: Remove faulty sanity tests for RTLD build with no multiarch
The sanity tests where meant to ensure that the default implementation
was only being built without multiarch with the exception of the
multiarch/rtld-*.S files.
The code used IS_IN (rtld) to check if the build for was for an
multiarch/rtld-*.S file which is incorrect as IS_IN (rtld) is set for
the non-multiarch build as well.
Noah Goldstein [Wed, 22 Jun 2022 23:51:20 +0000 (16:51 -0700)]
x86: Add support for compiling {raw|w}memchr with high ISA level
1. Refactor files so that all implementations for in the multiarch
directory.
- Essentially moved sse2 {raw|w}memchr.S implementation to
multiarch/{raw|w}memchr-sse2.S
- The non-multiarch {raw|w}memchr.S file now only includes one of
the implementations in the multiarch directory based on the
compiled ISA level (only used for non-multiarch builds.
Otherwise we go through the ifunc selector).
2. Add ISA level build guards to different implementations.
- I.e memchr-avx2.S which is ISA level 3 will only build if
compiled ISA level <= 3. Otherwise there is no reason to include
it as we will always use one of the ISA level 4
implementations (memchr-evex{-rtm}.S).
3. Add new multiarch/rtld-{raw}memchr.S that just include the
non-multiarch {raw}memchr.S which will in turn select the best
implementation based on the compiled ISA level.
4. Refactor the ifunc selector and ifunc implementation list to use
the ISA level aware wrapper macros that allow functions below the
compiled ISA level (with a guranteed replacement) to be skipped.
- Guranteed replacement essentially means that for any ISA level
build there must be a function that the baseline of the ISA
supports. So for {raw|w}memchr.S since there is not ISA level 2
function, the ISA level 2 build still includes the ISA level
1 (sse2) function. Once we reach the ISA level 3 build, however,
{raw|w}memchr-avx2{-rtm}.S will always be sufficient so the ISA
level 1 implementation ({raw|w}memchr-sse2.S) will not be built.
Tested with and without multiarch on x86_64 for ISA levels:
{generic, x86-64-v2, x86-64-v3, x86-64-v4}
x86: Rename generic functions with unique postfix for clarity
Changed the names of the strspn-c, strcspn-c, and strpbrk-c files
in a general refactor. It didn't change the include paths for the
i386 files breaking the i386 build. This commit fixes that. Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
Noah Goldstein [Thu, 16 Jun 2022 22:07:12 +0000 (15:07 -0700)]
x86: Rename generic functions with unique postfix for clarity
No functions are changed. It just renames generic implementations from
'{func}_sse2' to '{func}_generic'. This is just because the postfix
"_sse2" was overloaded and was used for files that had hand-optimized
sse2 assembly implementations and files that just redirected back
to the generic implementation.
Fangrui Song [Thu, 16 Jun 2022 18:48:15 +0000 (11:48 -0700)]
x86-64: Handle fewer relocation types for RTLD_BOOTSTRAP
The RTLD_BOOTSTRAP branch is used to relocate ld.so itself. It only
needs to handle RELATIVE, GLOB_DAT, and JUMP_SLOT. RELATIVE has been
handled (by _ELF_DYNAMIC_DO_RELOC due to DT_RELACOUNT, or RELR), so the
switch statement only needs to handle GLOB_DAT and JUMP_SLOT.
We can drop these `#if[n]def RTLD_BOOTSTRAP` and add a large
`# ifndef RTLD_BOOTSTRAP` instead.
Fangrui Song [Thu, 16 Jun 2022 02:21:53 +0000 (19:21 -0700)]
aarch64: Handle fewer relocations for RTLD_BOOTSTRAP
The RTLD_BOOTSTRAP branch is used to relocate ld.so itself. It only
needs to handle RELATIVE, GLOB_DAT, and JUMP_SLOT.
TLSDESC/TLS_DTPMOD/TLS_DTPREL handling can be removed. Remove
`case AARCH64_R(RELATIVE)` as well as elf_machine_rela has checked it.
Fangrui Song [Thu, 16 Jun 2022 01:42:03 +0000 (18:42 -0700)]
riscv: Change the relocations handled for RTLD_BOOTSTRAP
The RTLD_BOOTSTRAP branch is used to relocate ld.so itself. It only
needs to handle RELATIVE, GLOB_DAT, and the symbolic relocation type
(R_RISCV_{32,64}). NONE and IRELATIVE can be removed.
The code relies on ld.so having DT_RELACOUNT so that the RTLD_BOOTSTRAP
branch does not need handle RELATIVE. Drop this minor size
optimization for clarity.
Noah Goldstein [Wed, 15 Jun 2022 17:41:28 +0000 (10:41 -0700)]
x86: Cleanup bounds checking in large memcpy case
1. Fix incorrect lower-bound threshold in L(large_memcpy_2x).
Previously was using `__x86_rep_movsb_threshold` and should
have been using `__x86_shared_non_temporal_threshold`.
2. Avoid reloading __x86_shared_non_temporal_threshold before
the L(large_memcpy_4x) bounds check.
3. Document the second bounds check for L(large_memcpy_4x)
more clearly.
Noah Goldstein [Wed, 15 Jun 2022 17:41:29 +0000 (10:41 -0700)]
x86: Add bounds `x86_non_temporal_threshold`
The lower-bound (16448) and upper-bound (SIZE_MAX / 16) are assumed
by memmove-vec-unaligned-erms.
The lower-bound is needed because memmove-vec-unaligned-erms unrolls
the loop aggressively in the L(large_memset_4x) case.
The upper-bound is needed because memmove-vec-unaligned-erms
right-shifts the value of `x86_non_temporal_threshold` by
LOG_4X_MEMCPY_THRESH (4) which without a bound may overflow.
The lack of lower-bound can be a correctness issue. The lack of
upper-bound cannot.
Fangrui Song [Wed, 15 Jun 2022 18:29:55 +0000 (11:29 -0700)]
elf: Remove ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA
If an executable has copy relocations for extern protected data, that
can only work if the library containing the definition is built with
assumptions (a) the compiler emits GOT-generating relocations (b) the
linker produces R_*_GLOB_DAT instead of R_*_RELATIVE. Otherwise the
library uses its own definition directly and the executable accesses a
stale copy. Note: the GOT relocations defeat the purpose of protected
visibility as an optimization, but allow rtld to make the executable and
library use the same copy when copy relocations are present, but it
turns out this never worked perfectly.
Without ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA, b.so accesses the copy
relocated data like a.so.
There is now a warning for copy relocation on protected symbol since
commit 7374c02b683b7110b853a32496a619410364d70b. It's extremely
unlikely anyone relies on the ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA
behavior, so let's remove it: this removes a check in the symbol lookup
code.