Bug 29167 - build failures with make --shuffle -j N
Summary: build failures with make --shuffle -j N
Status: RESOLVED INVALID
Alias: None
Product: glibc
Classification: Unclassified
Component: build (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-05-22 21:14 UTC by Fangrui Song
Modified: 2022-06-09 07:04 UTC (History)
2 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Fangrui Song 2022-05-22 21:14:15 UTC
See https://trofi.github.io/posts/238-new-make-shuffle-mode.html for a pending --shuffle option to GNU make to detect brittle dependencies.
I have patched my local repo ~/Dev/make with v6-0001-Add-shuffle-argument-support.patch

glibc currently has a number of missing dependency issues:

% mkdir -p out/play && cd out/play
% ../../configure --prefix=/tmp/glibc/play --enable-static-pie && ~/Dev/make/out/release/make --shuffle=1653253310 -j 20
...
gcc wcstombs.c -c -std=gnu11 -fgnu89-inline  -g -O2 -Wall -Wwrite-strings -Wundef -Werror -fmerge-all-constants -frounding-math -fno-stack-protector -fno-common -Wstrict-prototypes -Wold-style-definition -fmath-errno    -fpie   -ftls-model=initial-exec      -I../include -I/home/maskray/Dev/glibc/out/play/stdlib  -I/home/maskray/Dev/glibc/out/play  -I../sysdeps/unix/sysv/linux/x86_64/64  -I../sysdeps/unix/sysv/linux/x86_64  -I../sysdeps/unix/sysv/linux/x86/include -I../sysdeps/unix/sysv/linux/x86  -I../sysdeps/x86/nptl  -I../sysdeps/unix/sysv/linux/wordsize-64  -I../sysdeps/x86_64/nptl  -I../sysdeps/unix/sysv/linux/include -I../sysdeps/unix/sysv/linux  -I../sysdeps/nptl  -I../sysdeps/pthread  -I../sysdeps/gnu  -I../sysdeps/unix/inet  -I../sysdeps/unix/sysv  -I../sysdeps/unix/x86_64  -I../sysdeps/unix  -I../sysdeps/posix  -I../sysdeps/x86_64/64  -I../sysdeps/x86_64/fpu/multiarch  -I../sysdeps/x86_64/fpu  -I../sysdeps/x86/fpu  -I../sysdeps/x86_64/multiarch  -I../sysdeps/x86_64  -I../sysdeps/x86/include -I../sysdeps/x86  -I../sysdeps/ieee754/float128  -I../sysdeps/ieee754/ldbl-96/include -I../sysdeps/ieee754/ldbl-96  -I../sysdeps/ieee754/dbl-64  -I../sysdeps/ieee754/flt-32  -I../sysdeps/wordsize-64  -I../sysdeps/ieee754  -I../sysdeps/generic  -I.. -I../libio -I.  -D_LIBC_REENTRANT -include /home/maskray/Dev/glibc/out/play/libc-modules.h -DMODULE_NAME=libc -include ../include/libc-symbols.h  -DPIC     -DTOP_NAMESPACE=glibc -o /home/maskray/Dev/glibc/out/play/stdlib/wcstombs.o -MD -MP -MF /home/maskray/Dev/glibc/out/play/stdlib/wcstombs.o.dt -MT /home/maskray/Dev/glibc/out/play/stdlib/wcstombs.o
../sysdeps/unix/sysv/linux/x86_64/getcontext.S:22:10: fatal error: ucontext_i.h: No such file or directory
   22 | #include "ucontext_i.h"
      |          ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [/home/maskray/Dev/glibc/out/play/sysd-rules:41: /home/maskray/Dev/glibc/out/play/stdlib/getcontext.o] Error 1 --shuffle=1653253310
make[2]: *** Waiting for unfinished jobs....
../sysdeps/unix/sysv/linux/x86_64/makecontext.c:29:10: fatal error: ucontext_i.h: No such file or directory
   29 | #include "ucontext_i.h"
      |          ^~~~~~~~~~~~~~

In another fresh build with --shuffle=reverse 

% ~/Dev/make/out/release/make --shuffle=reverse -j 20
...
make[2]: Entering directory '/home/maskray/Dev/glibc/elf'
python3 -B ../scripts/dso-ordering-test.py --description-file dso-sort-tests-1.def --objpfx /home/maskray/Dev/glibc/out/reverse/elf/ --output-makefile /home/maskray/Dev/glibc/out/reverse/elf/dso-sort-tests-1.def.generated-makefileT
python3 -B ../scripts/dso-ordering-test.py --description-file dso-sort-tests-2.def --objpfx /home/maskray/Dev/glibc/out/reverse/elf/ --output-makefile /home/maskray/Dev/glibc/out/reverse/elf/dso-sort-tests-2.def.generated-makefileT
mv /home/maskray/Dev/glibc/out/reverse/elf/dso-sort-tests-1.def.generated-makefileT /home/maskray/Dev/glibc/out/reverse/elf/dso-sort-tests-1.def.generated-makefile
mv /home/maskray/Dev/glibc/out/reverse/elf/dso-sort-tests-2.def.generated-makefileT /home/maskray/Dev/glibc/out/reverse/elf/dso-sort-tests-2.def.generated-makefile
make[2]: *** No rule to make target '/home/maskray/Dev/glibc/out/reverse/stamp.oS', needed by '/home/maskray/Dev/glibc/out/reverse/libc_nonshared.a'.  Stop.
make[2]: Leaving directory '/home/maskray/Dev/glibc/elf'
make[1]: *** [Makefile:483: elf/others] Error 2 --shuffle=reverse
make[1]: Leaving directory '/home/maskray/Dev/glibc'
Comment 1 Andreas Schwab 2022-06-07 14:31:12 UTC
The toplevel Makefile uses .NOTPARALLEL: to force sequential build.
Comment 2 Fangrui Song 2022-06-09 07:04:25 UTC
Using v7 from https://savannah.gnu.org/bugs/index.php?62100 does not have the issue. Closing

Tested several configurations:
../../configure --prefix=/tmp/glibc/play --with-default-link=yes --enable-static-pie && /tmp/p/make/out/release/make --shuffle=1653253310 -j 20
../../configure --prefix=/tmp/glibc/play --with-default-link=yes --enable-static-pie && /tmp/p/make/out/release/make --shuffle=reverse -j 20
../../configure --prefix=/tmp/glibc/play --with-default-link=yes && /tmp/p/make/out/release/make --shuffle=reverse -j 40