Paul Zimmermann [Mon, 3 May 2021 09:30:25 +0000 (11:30 +0200)]
add workload traces for cbrtl
These workload traces cover the whole "long double" range.
This patch was prepared with the help of Adhemerval Zanella. Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 10 May 2021 08:31:41 +0000 (10:31 +0200)]
nptl: Move changing of stack permissions into ld.so
All the stack lists are now in _rtld_global, so it is possible
to change stack permissions directly from there, instead of
calling into libpthread to do the change.
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 10 May 2021 08:31:41 +0000 (10:31 +0200)]
nptl: Move more stack management variables into _rtld_global
Permissions of the cached stacks may have to be updated if an object
is loaded that requires executable stacks, so the dynamic loader
needs to know about these cached stacks.
The move of in_flight_stack and stack_cache_actsize is a requirement for
merging __reclaim_stacks into the fork implementation in libc.
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 10 May 2021 08:31:41 +0000 (10:31 +0200)]
Linux: Simplify and fix the definition of SINGLE_THREAD_P
Always use __libc_multiple_threads if beneficial, and do not assume
the the dynamic loader is single-threaded. This assumption could
become incorrect by accident once more code is moved from libpthread
into it. The previous commit introducing the
NO_SYSCALL_CANCEL_CHECKING macro enables this change.
Do not hint to the compiler that multi-threaded programs are unlikely
(which is not quite true anymore).
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 10 May 2021 08:31:41 +0000 (10:31 +0200)]
Linux: Explicitly disable cancellation checking in the dynamic loader
Historically, SINGLE_THREAD_P is defined to 1 in the dynamic loader.
This has the side effect of disabling cancellation points. In order
to enable future use of SINGLE_THREAD_P for single-thread
optimizations in the dynamic loader (which becomes important once
more code is moved from libpthread), introduce a new
NO_SYSCALL_CANCEL_CHECKING macro which is always 1 for IS_IN (rtld),
indepdently of the actual SINGLE_THREAD_P value.
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 10 May 2021 08:31:41 +0000 (10:31 +0200)]
elf, nptl: Resolve recursive lock implementation early
If libpthread is included in libc, it is not necessary to delay
initialization of the lock/unlock function pointers until libpthread
is loaded. This eliminates two unprotected function pointers
from _rtld_global and removes some initialization code from
libpthread.
Tested-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Noah Goldstein [Tue, 4 May 2021 23:02:40 +0000 (19:02 -0400)]
x86: Add EVEX optimized memchr family not safe for RTM
No bug.
This commit adds a new implementation for EVEX memchr that is not safe
for RTM because it uses vzeroupper. The benefit is that by using
ymm0-ymm15 it can use vpcmpeq and vpternlogd in the 4x loop which is
faster than the RTM safe version which cannot use vpcmpeq because
there is no EVEX encoding for the instruction. All parts of the
implementation aside from the 4x loop are the same for the two
versions and the optimization is only relevant for large sizes.
powerpc64le: Fix ifunc selection for memset, memmove, bzero and bcopy
The hwcap2 check for the aforementioned functions should check for
both PPC_FEATURE2_ARCH_3_1 and PPC_FEATURE2_HAS_ISEL but was
mistakenly checking for any one of them, enabling isa 3.1 version of
the functions in incompatible processors, like POWER8.
Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
linux: Remove /proc/cpuinfo fallback on alpha and sparc
There is no much gain in fallback to cpuinfo if sysfs is no present,
usually on restricted environment neither will be present. It also
simplifies the code and make all architecture use the sched_getaffinity
as the sysfs fallback.
linux: Use sched_getaffinity for __get_nprocs (BZ #27645)
Both the sysfs and procfs parsing (through GET_NPROCS_PARSER) are
removed in favor the syscall. The initial scratch buffer should
fit to most of the common usage (1024 bytes with maps to 8192 CPUs).
Checked on x86_64-linux-gnu and aarch64-linux-gnu.
Remove architecture specific sched_cpucount optimizations
And replace the generic algorithm with the Brian Kernighan's one.
GCC optimize it with popcnt if the architecture supports, so there
is no need to add the extra POPCNT define to enable it.
This is really a micro-optimization that only adds complexity:
recent ABIs already support it (x86-64-v2 or power64le) and it
simplifies the code for internal usage, since i686 does not allow an
internal iFUNC call.
Checked on x86_64-linux-gnu, aarch64-linux-gnu, and
powerpc64le-linux-gnu.
Martin Sebor [Thu, 6 May 2021 16:56:25 +0000 (10:56 -0600)]
Annotate additional APIs with GCC attribute access.
This change continues the improvements to compile-time out of bounds
checking by decorating more APIs with either attribute access, or by
explicitly providing the array bound in APIs such as tmpnam() that
expect arrays of some minimum size as arguments. (The latter feature
is new in GCC 11.)
The only effects of the attribute and/or the array bound is to check
and diagnose calls to the functions that fail to provide a sufficient
number of elements, and the definitions of the functions that access
elements outside the specified bounds. (There is no interplay with
_FORTIFY_SOURCE here yet.)
Florian Weimer [Wed, 5 May 2021 15:15:57 +0000 (17:15 +0200)]
nptl: Move sem_close, sem_open into libc
The symbols were moved using move-symbol-to-libc.py.
Both functions are moved at the same time because they depend
on internal functions in sysdeps/pthread/sem_routines.c, which
are moved in this commit as well. Additional hidden prototypes
are required to avoid check-localplt failures.
Florian Weimer [Wed, 5 May 2021 15:15:57 +0000 (17:15 +0200)]
nptl: Consolidate async cancel enable/disable implementation in libc
Previously, the source file nptl/cancellation.c was compiled multiple
times, for libc, libpthread, librt. This commit switches to a single
implementation, with new __pthread_enable_asynccancel@@GLIBC_PRIVATE,
__pthread_disable_asynccancel@@GLIBC_PRIVATE exports.
The almost-unused CANCEL_ASYNC and CANCEL_RESET macros are replaced
by LIBC_CANCEL_ASYNC and LIBC_CANCEL_ASYNC macros. They call the
__pthread_* functions unconditionally now. The macros are still
needed because shared code uses them; Hurd has different definitions.
Florian Weimer [Wed, 5 May 2021 04:20:31 +0000 (06:20 +0200)]
elf, nptl: Initialize static TLS directly in ld.so
The stack list is available in ld.so since commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move stack list
variables into _rtld_global"), so it's possible to walk the stack
list directly in ld.so and perform the initialization there.
This eliminates an unprotected function pointer from _rtld_global
and reduces the libpthread initialization code.
Noah Goldstein [Mon, 3 May 2021 07:03:19 +0000 (03:03 -0400)]
x86: Optimize memchr-evex.S
No bug. This commit optimizes memchr-evex.S. The optimizations include
replacing some branches with cmovcc, avoiding some branches entirely
in the less_4x_vec case, making the page cross logic less strict,
saving some ALU in the alignment process, and most importantly
increasing ILP in the 4x loop. test-memchr, test-rawmemchr, and
test-wmemchr are all passing.
Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
Noah Goldstein [Mon, 3 May 2021 07:01:58 +0000 (03:01 -0400)]
x86: Optimize memchr-avx2.S
No bug. This commit optimizes memchr-avx2.S. The optimizations include
replacing some branches with cmovcc, avoiding some branches entirely
in the less_4x_vec case, making the page cross logic less strict,
asaving a few instructions the in loop return loop. test-memchr,
test-rawmemchr, and test-wmemchr are all passing.
Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
It operates similar to execve and it is is already used to implement
fexecve without requiring /proc to be mounted. However, different
than fexecve, if the syscall is not supported by the kernel an error
is returned instead of trying a fallback.
Checked on x86_64-linux-gnu and powerpc64le-linux-gnu.
Noah Goldstein [Mon, 3 May 2021 08:44:35 +0000 (04:44 -0400)]
Bench: Expand bench-memchr.c
No bug. This commit adds some additional cases for bench-memchr.c
including testing medium sizes and testing short length with both an
inbound match and out of bound match.
Alyssa Ross [Mon, 3 May 2021 08:34:11 +0000 (10:34 +0200)]
stdio: fix vfscanf with matches longer than INT_MAX (bug 27650)
Patterns like %*[ can safely be used to match a great many characters,
and it's quite realisitic to use them for more than INT_MAX characters
from an IO stream.
With the previous approach, after INT_MAX characters (v)fscanf would
return successfully, indicating an end to the match, even though there
wasn't one.
Florian Weimer [Mon, 3 May 2021 06:12:12 +0000 (08:12 +0200)]
nptl: Move tss_create into libc
The symbol was moved using scripts/move-symbol-to-libc.py.
__pthread_key_create@@GLIBC_2.34 is no longer needed by glibc
itself with this change, but __pthread_key_create is used by
libstdc++, so it still has to be exported as a public symbol.
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 3 May 2021 06:12:11 +0000 (08:12 +0200)]
nptl: Move mtx_init into libc
The symbol was moved using scripts/move-symbol-to-libc.py.
The calls to __pthread_mutex_init, __pthread_mutexattr_init,
__pthread_mutexattr_settype are now private and no longer need
to be exported. This allows the removal of the newly added
GLIBC_2.34 symbol versions for those functions.
Also clean up some weak declarations in <libc-lockP.h> for
these functions. They are not needed and potentially incorrect
for static linking of mtx_init.
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 3 May 2021 06:12:11 +0000 (08:12 +0200)]
nptl: Move mtx_destroy into libc
The symbol was moved using scripts/move-symbol-to-libc.py.
The __pthread_mutex_destroy@@GLIBC_2.34 symbol is no longer
neded because this commit makes __pthread_mutex_destroy@GLIBC_2.0
a compatibility symbol, so remove the new symbol version.
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
Florian Weimer [Mon, 3 May 2021 06:12:11 +0000 (08:12 +0200)]
nptl: Move call_once into libc
The symbol was moved using scripts/move-symbol-to-libc.py.
This change also turns __pthread_once into a compatibility symbol
because after the call_once move, an internal call to __pthread_once
can be used. This an adjustment to __libc_once: Outside libc (e.g.,
in nscd), it has to call pthread_once. With __pthread_once as a
compatibility symbol, it is no longer to add a new GLIBC_2.34
version after the move from libpthread, and this commit removes
the new __pthread_once@@GLIBC_2.34 version.
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
This implementation is based on __memset_power8 and integrates a lot
of suggestions from Anton Blanchard.
The biggest difference is that it makes extensive use of stxvl to
alignment and tail code to avoid branches and small stores. It has
three main execution paths:
a) "Short lengths" for lengths up to 64 bytes, avoiding as many
branches as possible.
b) "General case" for larger lengths, it has an alignment section
using stxvl to avoid branches, a 128 bytes loop and then a tail
code, again using stxvl with few branches.
c) "Zeroing cache blocks" for lengths from 256 bytes upwards and set
value being zero. It is mostly the __memset_power8 code but the
alignment phase was simplified because, at this point, address is
already 16-bytes aligned and also changed to use vector stores.
The tail code was also simplified to reuse the general case tail.
All unaligned stores use stxvl instructions that do not generate
alignment interrupts on POWER10, making it safe to use on
caching-inhibited memory.
On average, this implementation provides something around 30%
improvement when compared to __memset_power8.
Reviewed-by: Matheus Castanho <msc@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
This implementation is based on __memcpy_power8_cached and integrates
suggestions from Anton Blanchard.
It benefits from loads and stores with length for short lengths and for
tail code, simplifying the code.
All unaligned memory accesses use instructions that do not generate
alignment interrupts on POWER10, making it safe to use on
caching-inhibited memory.
The main loop has also been modified in order to increase instruction
throughput by reducing the dependency on updates from previous iterations.
On average, this implementation provides around 30% improvement when
compared to __memcpy_power7 and 10% improvement in comparison to
__memcpy_power8_cached.
This patch was initially based on the __memmove_power7 with some ideas
from strncpy implementation for Power 9.
Improvements from __memmove_power7:
1. Use lxvl/stxvl for alignment code.
The code for Power 7 uses branches when the input is not naturally
aligned to the width of a vector. The new implementation uses
lxvl/stxvl instead which reduces pressure on GPRs. It also allows
the removal of branch instructions, implicitly removing branch stalls
and mispredictions.
2. Use of lxv/stxv and lxvl/stxvl pair is safe to use on Cache Inhibited
memory.
On Power 10 vector load and stores are safe to use on CI memory for
addresses unaligned to 16B. This code takes advantage of this to
do unaligned loads.
The unaligned loads don't have a significant performance impact by
themselves. However doing so decreases register pressure on GPRs
and interdependence stalls on load/store pairs. This also improved
readability as there are now less code paths for different alignments.
Finally this reduces the overall code size.
3. Improved performance.
This version runs on average about 30% better than memmove_power7
for lengths larger than 8KB. For input lengths shorter than 8KB
the improvement is smaller, it has on average about 17% better
performance.
This version has a degradation of about 50% for input lengths
in the 0 to 31 bytes range when dest is unaligned.
Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
Joseph Myers [Thu, 29 Apr 2021 10:50:11 +0000 (10:50 +0000)]
Update kernel version to 5.12 in tst-mman-consts.py.
This patch updates the kernel version in the test tst-mman-consts.py
to 5.12. (There are no new MAP_* constants covered by this test in
5.12 that need any other header changes.)
Joseph Myers [Wed, 28 Apr 2021 17:19:24 +0000 (17:19 +0000)]
Update syscall lists for Linux 5.12.
Linux 5.12 has one new syscall, mount_setattr. Update
syscall-names.list and regenerate the arch-syscall.h headers with
build-many-glibcs.py update-syscalls.
git://sourceware.org / glibc.git / log