hurd: Don't migrate reply port into __init1_tcbhead
Properly differentiate between setting up the real TLS with
TLS_INIT_TP, and setting up the early TLS (__init1_tcbhead) in static
builds. In the latter case, don't yet migrate the reply port into the
TCB, and don't yet set __libc_tls_initialized to 1.
This also lets us move the __init1_desc assignment inside
_hurd_tls_init ().
Created tunable to force small pages on stack allocation.
Created tunable glibc.pthread.stack_hugetlb to control when hugepages
can be used for stack allocation.
In case THP are enabled and glibc.pthread.stack_hugetlb is set to
0, glibc will madvise the kernel not to use allow hugepages for stack
allocations.
DJ Delorie [Mon, 3 Apr 2023 21:33:03 +0000 (17:33 -0400)]
malloc: set NON_MAIN_ARENA flag for reclaimed memalign chunk (BZ #30101)
Based on these comments in malloc.c:
size field is or'ed with NON_MAIN_ARENA if the chunk was obtained
from a non-main arena. This is only set immediately before handing
the chunk to the user, if necessary.
The NON_MAIN_ARENA flag is never set for unsorted chunks, so it
does not have to be taken into account in size comparisons.
When we pull a chunk off the unsorted list (or any list) we need to
make sure that flag is set properly before returning the chunk.
Use the rounded-up size for chunk_ok_for_memalign()
Do not scan the arena for reusable chunks if there's no arena.
Account for chunk overhead when determining if a chunk is a reuse
candidate.
mcheck interferes with memalign, so skip mcheck variants of
memalign tests.
Reviewed-by: Carlos O'Donell <carlos@redhat.com> Tested-by: Carlos O'Donell <carlos@redhat.com>
_hurd_thread_sigstate () already handles finding an existing sigstate
before allocating a new one, so just use that. Bonus: this will only
lock the _hurd_siglock once.
Much as the comment says, things on _hurd_subinit assume that _hurd_pid
is already initialized by the time _hurd_subinit is run, so
_hurd_proc_subinit has to run before it. Specifically, init_dtable ()
calls _hurd_port2fd (), which uses _hurd_pid and _hurd_pgrp to set up
ctty handling. With _hurd_subinit running before _hurd_proc_subinit,
ctty setup was broken:
Fix this by running the _hurd_proc_subinit hook in the correct place --
just after _hurd_portarray is set up (so the proc server port is
available in its usual place) and just before running _hurd_subinit.
We must not use the user's reply port (scp->sc_reply_port) for any of
our own RPCs, otherwise various things break. So, use MACH_PORT_DEAD as
a reply port when destroying our reply port, and make sure to do this
after _hurd_sigstate_unlock (), which may do a gsync_wake () RPC.
It is common to have (some of) stdin, stdout and stderr point to the
very same port. We were making the ctty RPCs that _hurd_port2fd () does
for each one of them separately:
Instead, let's detect this case and duplicate the ctty port we already
have. This means we do 1 RPC instead of 3 (and create a single protid
on the server side) if the file is our ctty, and no RPCs instead of 1
if it's not. A clear win!
Optimize the fast paths (x < y) and (x/y < 2^12). Delay handling of special
cases to reduce the number of instructions executed before the fast paths.
Performance improvements for fmod:
Wilco Dijkstra [Tue, 21 Mar 2023 14:00:22 +0000 (14:00 +0000)]
Benchtests: Adjust timing
Adjust iteration counts so benchmarks don't run too slowly or quickly.
Ensure benchmarks take less than 10 seconds on older, slower cores and
more than 0.5 seconds on fast cores.
Sergey Bugaev [Sun, 19 Mar 2023 15:10:07 +0000 (18:10 +0300)]
hurd: Only check for TLS initialization inside rtld or in static builds
When glibc is built as a shared library, TLS is always initialized by
the call of TLS_INIT_TP () macro made inside the dynamic loader, prior
to running the main program (see dl-call_tls_init_tp.h). We can take
advantage of this: we know for sure that __LIBC_NO_TLS () will evaluate
to 0 in all other cases, so let the compiler know that explicitly too.
Also, only define _hurd_tls_init () and TLS_INIT_TP () under the same
conditions (either !SHARED or inside rtld), to statically assert that
this is the case.
Other than a microoptimization, this also helps with avoiding awkward
sharing of the __libc_tls_initialized variable between ld.so and libc.so
that we would have to do otherwise -- we know for sure that no sharing
is required, simply because __libc_tls_initialized would always be set
to true inside libc.so.
Now that the signal code no longer accesses it, the only real user of it
was mig-reply.c, so move the logic for managing the port there.
If we're in SHARED and outside of rtld, we know that __LIBC_NO_TLS ()
always evaluates to 0, and a TLS reply port will always be used, not
__hurd_reply_port0. Still, the compiler does not see that
__hurd_reply_port0 is never used due to its address being taken. To deal
with this, explicitly compile out __hurd_reply_port0 when we know we
won't use it.
Also, instead of accessing the port via THREAD_SELF->reply_port, this
uses THREAD_GETMEM and THREAD_SETMEM directly, avoiding possible
miscompilations.
Samuel Thibault [Tue, 11 Apr 2023 22:12:02 +0000 (00:12 +0200)]
aio: Fix freeing memory
The content of the pool array is initialized only until pool_size,
pointers between pool_size and pool_max_size were not initialized by the
realloc call in get_elem so they should not be freed.
This fixes aio tests crashing at their termination on GNU/Hurd.
Apparently we load libc.so (and thus start using its functions) before
calling TLS_INIT_TP, so libc.so functions should not actually assume
that TLS is always set up.
Sergey Bugaev [Sun, 19 Mar 2023 15:10:10 +0000 (18:10 +0300)]
hurd: Don't leak __hurd_reply_port0
Previously, once we set up TLS, we would implicitly switch from using
__hurd_reply_port0 to reply_port inside the TCB, leaving the former
unused. But we never deallocated it, so it got leaked.
Instead, migrate the port into the new TCB's reply_port slot. This
avoids both the port leak and an extra syscall to create a new reply
port for the TCB.
Sergey Bugaev [Sun, 19 Mar 2023 15:10:08 +0000 (18:10 +0300)]
hurd: Improve reply port handling when exiting signal handlers
If we're doing signals, that means we've already got the signal thread
running, and that implies TLS having been set up. So we know that
__hurd_local_reply_port will resolve to THREAD_SELF->reply_port, and can
access that directly using the THREAD_GETMEM and THREAD_SETMEM macros.
This avoids potential miscompilations, and should also be a tiny bit
faster.
Also, use mach_port_mod_refs () and not mach_port_destroy () to destroy
the receive right. mach_port_destroy () should *never* be used on
mach_task_self (); this can easily lead to port use-after-free
vulnerabilities if the task has any other references to the same port.
Sergey Bugaev [Sun, 19 Mar 2023 15:10:07 +0000 (18:10 +0300)]
hurd: Only check for TLS initialization inside rtld or in static builds
When glibc is built as a shared library, TLS is always initialized by
the call of TLS_INIT_TP () macro made inside the dynamic loader, prior
to running the main program (see dl-call_tls_init_tp.h). We can take
advantage of this: we know for sure that __LIBC_NO_TLS () will evaluate
to 0 in all other cases, so let the compiler know that explicitly too.
Also, only define _hurd_tls_init () and TLS_INIT_TP () under the same
conditions (either !SHARED or inside rtld), to statically assert that
this is the case.
Other than a microoptimization, this also helps with avoiding awkward
sharing of the __libc_tls_initialized variable between ld.so and libc.so
that we would have to do otherwise -- we know for sure that no sharing
is required, simply because __libc_tls_initialized would always be set
to true inside libc.so.
These are just regular local variables that are not accessed in any
funny ways, not even though a pointer. There's absolutely no reason to
declare them volatile. It only ends up hurting the quality of the
generated machine code.
If anything, it would make sense to decalre sigsp as *pointing* to
volatile memory (volatile void *sigsp), but evidently that's not needed
either.
Paul Eggert [Sat, 8 Apr 2023 20:51:26 +0000 (13:51 -0700)]
manual: update AddressSanitizer discussion
* manual/string.texi (Truncating Strings): Update obsolescent
reference and use the more-generic term “AddressSanitizer”.
Mention fortification, too. -fcheck-pointer-bounds is no longer
supported.
Paul Eggert [Sat, 8 Apr 2023 20:51:26 +0000 (13:51 -0700)]
manual: improve string section wording
* manual/string.texi: Editorial fixes. Do not say “text” when
“string” or “string contents” is meant, as a C string can contain
bytes that are not valid text in the current encoding.
When warning about strcat efficiency, warn similarly about strncat
and wcscat. “coping” → “copying”.
Mention at the start of the two problematic sections that problems
are discussed at section end.
H.J. Lu [Wed, 5 Apr 2023 16:21:26 +0000 (09:21 -0700)]
<bits/platform/x86.h>: Rename to x86_cpu_INDEX_7_ECX_15
Rename x86_cpu_INDEX_7_ECX_1 to x86_cpu_INDEX_7_ECX_15 for the unused bit
15 in ECX from CPUID with EAX == 0x7 and ECX == 0. Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
sysdeps/mach/hurd/htl/pt-pthread_self.c: New file.
htl/Makefile: .. Add it to libc routine.
sysdeps/mach/hurd/htl/pt-sysdep.c(__pthread_self): Remove it.
sysdeps/mach/hurd/htl/pt-sysdep.h(__pthread_self): Add hidden propertie.
htl/Versions(__pthread_self) Version it as private symbol.
htl/pt-nthreads.c: new file.
htl/Makefile: Add it to routine.
htl/Versions: version it as private libc symbol.
htl/pt-create.c: remove his definition here.
htl/pt-internal.h: add propertie to it declaration.
As indicated by sparc kernel-features.h, even though sparc64 defines
__NR_pause, it is not supported (ENOSYS). Always use ppoll or the
64 bit time_t variant instead.
math: Remove the error handling wrapper from fmod and fmodf
The error handling is moved to sysdeps/ieee754 version with no SVID
support. The compatibility symbol versions still use the wrapper
with SVID error handling around the new code. There is no new symbol
version nor compatibility code on !LIBM_SVID_COMPAT targets
(e.g. riscv).
The ia64 is unchanged, since it still uses the arch specific
__libm_error_region on its implementation. For both i686 and m68k,
which provive arch specific implementation, wrappers are added so
no new symbol are added (which would require to change the
implementations).
It shows an small improvement, the results for fmod:
This uses a new algorithm similar to already proposed earlier [1].
With x = mx * 2^ex and y = my * 2^ey (mx, my, ex, ey being integers),
the simplest implementation is:
mx * 2^ex == 2 * mx * 2^(ex - 1)
while (ex > ey)
{
mx *= 2;
--ex;
mx %= my;
}
With mx/my being mantissa of double floating pointer, on each step the
argument reduction can be improved 8 (which is sizeof of uint32_t minus
MANTISSA_WIDTH plus the signal bit):
while (ex > ey)
{
mx << 8;
ex -= 8;
mx %= my;
} */
The implementation uses builtin clz and ctz, along with shifts to
convert hx/hy back to doubles. Different than the original patch,
this path assume modulo/divide operation is slow, so use multiplication
with invert values.
I see the following performance improvements using fmod benchtests
(result only show the 'mean' result):
I also see similar improvements on arm-linux-gnueabihf when running on
the N1 aarch64 chips, where it a lot of soft-fp implementation (for
modulo, and multiplication):
This uses a new algorithm similar to already proposed earlier [1].
With x = mx * 2^ex and y = my * 2^ey (mx, my, ex, ey being integers),
the simplest implementation is:
mx * 2^ex == 2 * mx * 2^(ex - 1)
while (ex > ey)
{
mx *= 2;
--ex;
mx %= my;
}
With mx/my being mantissa of double floating pointer, on each step the
argument reduction can be improved 11 (which is sizeo of uint64_t minus
MANTISSA_WIDTH plus the signal bit):
while (ex > ey)
{
mx << 11;
ex -= 11;
mx %= my;
} */
The implementation uses builtin clz and ctz, along with shifts to
convert hx/hy back to doubles. Different than the original patch,
this path assume modulo/divide operation is slow, so use multiplication
with invert values.
I see the following performance improvements using fmod benchtests
(result only show the 'mean' result):
I also see similar improvements on arm-linux-gnueabihf when running on
the N1 aarch64 chips, where it a lot of soft-fp implementation (for
modulo, clz, ctz, and multiplication):
1. Subnormals: 128 inputs.
2. Normal numbers with large exponent difference (|x/y| > 2^8):
1024 inputs between FLT_MIN and FLT_MAX;
3. Close exponents (ey >= -103 and |x/y| < 2^8): 1024 inputs with
exponents between -10 and 10. Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Add three different dataset, from random floating point numbers:
1. Subnormals: 128 inputs.
2. Normal numbers with large exponent difference (|x/y| > 2^52):
1024 inputs between DBL_MIN and DBL_MAX;
3. Close exponents (ey >= -907 and |x/y| < 2^52): 1024 inputs with
exponents between -10 and 10. Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
H.J. Lu [Thu, 16 Mar 2023 00:42:54 +0000 (17:42 -0700)]
x86: Set FSGSBASE to active if enabled by kernel
Linux kernel uses AT_HWCAP2 to indicate if FSGSBASE instructions are
enabled. If the HWCAP2_FSGSBASE bit in AT_HWCAP2 is set, FSGSBASE
instructions can be used in user space. Define dl_check_hwcap2 to set
the FSGSBASE feature to active on Linux when the HWCAP2_FSGSBASE bit is
set.
Add a test to verify that FSGSBASE is active on current kernels.
NB: This test will fail if the kernel doesn't set the HWCAP2_FSGSBASE
bit in AT_HWCAP2 while fsgsbase shows up in /proc/cpuinfo. Reviewed-by: Florian Weimer <fweimer@redhat.com>
x86_64: Fix asm constraints in feraiseexcept (bug 30305)
The divss instruction clobbers its first argument, and the constraints
need to reflect that. Fortunately, with GCC 12, generated code does
not actually change, so there is no externally visible bug.
Suggested-by: Jakub Jelinek <jakub@redhat.com> Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
manual: Document __wur usage under _FORTIFY_SOURCE
The __warn_unused_result__ attribute is only enabled when fortification
is enabled. Mention that in the document. The rationale for this is
essentially to mitigate against CWE-252:
Sergey Bugaev [Sun, 19 Mar 2023 15:10:14 +0000 (18:10 +0300)]
hurd: Microoptimize _hurd_self_sigstate ()
When THREAD_GETMEM is defined with inline assembly, the compiler may not
optimize away the two reads of _hurd_sigstate. Help it out a little bit
by only reading it once. This also makes for a slightly cleaner code.
Sergey Bugaev [Sun, 19 Mar 2023 15:09:58 +0000 (18:09 +0300)]
hurd: Use uintptr_t for register values in trampoline.c
This is more correct, if only because these fields are defined as having
the type unsigned int in the Mach headers, so casting them to a signed
int and then back is suboptimal.
Also, remove an extra reassignment of uesp -- this is another remnant of
the ecx kludge.
Sergey Bugaev [Sun, 19 Mar 2023 15:09:54 +0000 (18:09 +0300)]
mach, hurd: Drop __libc_lock_self0
This was used for the value of libc-lock's owner when TLS is not yet set
up, so THREAD_SELF can not be used. Since the value need not be anything
specific -- it just has to be non-NULL -- we can just use a plain
constant, such as (void *) 1, for this. This avoids accessing the symbol
through GOT, and exporting it from libc.so in the first place.
Sergey Bugaev [Sun, 19 Mar 2023 15:09:51 +0000 (18:09 +0300)]
hurd: Disable O_TRUNC and FS_RETRY_MAGICAL in rtld
hurd/lookup-retry.c is compiled into rtld, the dynamic linker/loader. To
avoid pulling in file_set_size, file_utimens, tty/ctty stuff, more
string/memory code (memmove, strncpy, strcpy), and more strtoul/itoa
code, compile out support for O_TRUNC and FS_RETRY_MAGICAL when building
hurd/lookup-retry.c for rtld. None of that functionality is useful to
rtld during startup anyway. Keep support for FS_RETRY_MAGICAL("/"),
since that does not pull in much, and is required for following absolute
symlinks.
The large number of extra code being pulled into rtld was noticed by
reviewing librtld.map & elf/librtld.os.map in the build tree.
It is worth noting that once libc.so is loaded, the real __open, __stat,
etc. replace the minimal versions used initially by rtld -- this is
especially important in the Hurd port, where the minimal rtld versions
do not use the dtable and just pass real Mach port names as fds. Thus,
once libc.so is loaded, rtld will gain access to the full
__hurd_file_name_lookup_retry () version, complete with FS_RETRY_MAGICAL
support, which is important in case the program decides to
dlopen ("/proc/self/fd/...") or some such.
Sergey Bugaev [Sun, 19 Mar 2023 15:09:48 +0000 (18:09 +0300)]
hurd: Remove __hurd_threadvar_stack_{offset,mask}
Noone is or should be using __hurd_threadvar_stack_{offset,mask}, we
have proper TLS now. These two remaining variables are never set to
anything other than zero, so any code that would try to use them as
described would just dereference a zero pointer and crash. So remove
them entirely.
Sergey Bugaev [Sun, 19 Mar 2023 15:09:47 +0000 (18:09 +0300)]
hurd: Make exception subcode a long
On EXC_BAD_ACCESS, exception subcode is used to pass the faulting memory
address, so it needs to be (at least) pointer-sized. Thus, make it into
a long. This matches the corresponding change in GNU Mach.
Message-Id: <20230319151017.531737-5-bugaevc@gmail.com>