[Bug debuginfod/28577] Make run-...-fd-prefetch-caches.sh test something
Mark Wielaard
mark@klomp.org
Fri May 27 16:20:47 GMT 2022
Hi Noah,
On Wed, 2022-05-25 at 14:44 -0400, Noah Sanci via Elfutils-devel wrote:
> PR28577: Make run-debuginfod-fd-prefetch-caches.sh test
> something
>
> Update to the run-debuginfod-fd-prefetch to make the test more sound.
Thanks. And according to Evgeny this makes the test pass for him:
https://sourceware.org/bugzilla/show_bug.cgi?id=29180#c2
But I find it a little hard to review, it would have been helpful to
get a bit more comments/documentation what exactly is being tested now.
> diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-
> debuginfod-fd-prefetch-caches.sh
> index 66a7a083..50ffc4ac 100755
> --- a/tests/run-debuginfod-fd-prefetch-caches.sh
> +++ b/tests/run-debuginfod-fd-prefetch-caches.sh
> @@ -21,22 +21,22 @@
> # for test case debugging, uncomment:
> set -x
> unset VALGRIND_CMD
> +mkdir R Z
OK, some boilerplate.
> -mkdir R
> -cp -rvp ${abs_srcdir}/debuginfod-rpms R
> -if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
> - rm -vrf R/debuginfod-rpms/fedora31
> -fi
This gets moved to below.
> -FDCACHE_MBS=$((1/1024/1024))
> +FDCACHE_MBS=1
> FDCACHE_FDS=1
> -PREFETCH_MBS=$((1/1024/1024))
> -PREFETCH_FDS=2
> +PREFETCH_MBS=1
> +PREFETCH_FDS=1
> PREFETCH=1
OK, new presets.
> # This variable is essential and ensures no time-race for claiming
> ports occurs
> # set base to a unique multiple of 100 not used in any other 'run-
> debuginfod-*' test
> base=8800
> get_ports
> +cp -rvp ${abs_srcdir}/debuginfod-rpms R
> +if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
> + rm -vrf R/debuginfod-rpms/fedora31
> +fi
> +cp -rvp ${abs_srcdir}/debuginfod-tars Z
OK, put rpms under R, tars under Z.
> DB=${PWD}/.debuginfod_tmp.sqlite
> tempfiles $DB
> @@ -47,88 +47,68 @@ export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
> ######
> rm -rf $DEBUGINFOD_CACHE_PATH
> rm -rf $DB
> +
> # Set mb values to ensure the caches certainly have enough space
> # To store the test files
> env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \
> - --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \
> - --fdcache-mbs=50 --fdcache-prefetch-mbs=50 \
> - --fdcache-prefetch=$PREFETCH > vlog$PORT1 2>&1 &
> + --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-fds=$PREFETCH_FDS -vvvvv -g 0 -t 0 -R R \
> + -Z .tar.bz2=bzcat Z --fdcache-mbs=50 --fdcache-prefetch-mbs=100 \
> + --fdcache-prefetch=$PREFETCH -F > vlog$PORT1 2>&1 &
Now run with --fdcache-fds=1 --fdcache-mbs=50 and
-fdcache-prefetch-fds=1 fdcache-prefetch=1 -fdcache-prefetch-mbs=100
It is these last 3 we want to test.
> PID1=$!
> tempfiles vlog$PORT1
> errfiles vlog$PORT1
> +
> # Server must become ready
> wait_ready $PORT1 'ready' 1
> ########################################################################
> -cp -rvp ${abs_srcdir}/debuginfod-rpms R
> -if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
> - rm -vrf R/debuginfod-rpms/fedora31
> -fi
> kill -USR1 $PID1
> wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
> +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
> +wait_ready $PORT1 'thread_busy{role="scan"}' 0
OK, boilerplate to wait for initial scan.
> # All rpms need to be in the index, except the dummy permission-000 one
> rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
> wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
> -kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
> -# Wait till both files are in the index and scan/index fully finished
> +kill -USR1 $PID1
> wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
> -export DEBUGINFOD_URLS="http://127.0.0.1:"$PORT1
> +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
> +wait_ready $PORT1 'thread_busy{role="scan"}' 0
Do another scan and wait for it to be ready.
> -archive_hits=5
> -SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
> -for i in $archive_hits
> -do
> - archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
> -done
> +export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/
> +# load prefetch cache
> +testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb
OK, which particular file is this?
> metrics=$(curl http://127.0.0.1:$PORT1/metrics)
> -regex="fdcache_op_count\{op=\"enqueue\"\} ([0-9]+).*fdcache_op_count\{op=\"evict\"\} ([0-9]+).*"
> -enqueue=0
> -if [[ $metrics =~ $regex ]]
> -then
> - enqueue=${BASH_REMATCH[1]}
> - evict=${BASH_REMATCH[2]}
> +regex="fdcache_prefetch_count ([0-9])+"
> +# Check to see if prefetch cache is loaded
> +if [[ $metrics =~ $regex ]]; then
> + if [[ ${BASH_REMATCH[1]} -ne 1 ]]; then
> + err
> + fi
Why the -ne 1 ? What number are we expecting and why?
> else
> - err
> -fi
> -# This is an ad-hoc test than only works when FDCACHE_FDS < "archive_test"s above (5)
> -# otherwise there will be no eviction
> -if [[ $(( $enqueue-$FDCACHE_FDS )) -ne $evict ]]
> -then
> - err
> -fi
> -# Test prefetch cache
> -archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
> + err
> +fi
> +# Ensure that searching for the source prefetched finds it in the prefetch cache
> +testrun ${abs_top_builddir}/debuginfod/debuginfod-find source cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c
> metrics=$(curl http://127.0.0.1:$PORT1/metrics)
> -regex="fdcache_prefetch_count ([0-9])+"
> -pf_count=0
> -for i in $FDCACHE_FDS
> -do
> - if [[ $metrics =~ $regex ]]; then
> -
> - if [[ $(( $pf_count+$PREFETCH )) -ne ${BASH_REMATCH[1]} ]]; then
> - err
> - else
> - pf_count=${BASH_REMATCH[1]}
> - fi
> -
> - else
> +regex="fdcache_op_count\{op=\"prefetch_access\"\} ([0-9])+"
> +if [[ $metrics =~ $regex ]]; then
> + if [[ ${BASH_REMATCH[1]} -ne 1 ]]; then
> err
> fi
> -done
> +else
> + err
> +fi
Again -ne 1? Shouldn't it be different from the above now?
> kill $PID1
> wait $PID1
> PID1=0
>
> -
> #########
> -# Test mb limit on fd and prefetch cache
> +# Test mb limit on fd cache
> #########
> -
> rm -rf $DEBUGINFOD_CACHE_PATH
> rm -rf $DB
> env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \
> - --fdcache-mbs=1 --fdcache-prefetch-mbs=1 --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \
> - > vlog2$PORT1 2>&1 &
> + --fdcache-mbs=$FDCACHE_MBS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R > vlog2$PORT1 2>&1 &
> PID1=$!
OK start a fresh debuginfod with -fdcache-mbs=1 --fdcache-mintmp 0
Is the fdcache-mintmp 0 meant to clear the cache immediately?
> tempfiles vlog2$PORT1
> errfiles vlog2$PORT1
> @@ -139,26 +119,26 @@ wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
> # All rpms need to be in the index, except the dummy permission-000 one
> rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
> wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
> -kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
> -# Wait till both files are in the index and scan/index fully finished
> +kill -USR1 $PID1
> wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
> +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
> +wait_ready $PORT1 'thread_busy{role="scan"}' 0
OK, boilerplate again for triggering and waiting for a new scan to
complete.
> -archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
> +# This many archives cause the fd cache to be loaded with just over 1mb of
> +# files. So we expect the 1mb cap off
> +SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
> archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
> archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
> archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
> archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
> archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
> archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
> -archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
OK, test that the hello.c source file is found in the scanned archives,
that the fdcache is used and that the sha1sum checks out.
> metrics=$(curl http://127.0.0.1:$PORT1/metrics)
> -regex="fdcache_bytes ([0-9]+).*fdcache_prefetch_bytes ([0-9]+)"
> +regex="fdcache_bytes ([0-9]+)"
> mb=$((1024*1024))
> if [[ $metrics =~ $regex ]]; then
> fdbytes=${BASH_REMATCH[1]}
> - pfbytes=${BASH_REMATCH[2]}
> - if [ $fdbytes -gt $mb ] || [ $pfbytes -gt $mb ]; then
> + if [ $fdbytes -gt $mb ] ; then
> err
> fi
OK, check that the prefetch cache isn't bigger than 10MB. Why that
number?
Thanks,
Mark
More information about the Elfutils-devel
mailing list