One has to use explicit 'set debug-file-directory local-directory' even after 'set sysroot remote:' because neither automatically nor explicit 'remote:' prefix for 'set debug-file-directory' works. That also means one has to have NFS-mounted target machine and the 'remote:' mechanism is not usable for 'set debug-file-directory'. remote$ gdbserver :1234 xz local$ gdb -ex 'set sysroot remote:' -ex 'target remote host1s:1234' -ex 'symbol-file remote:/usr/bin/xz' -ex 'l main' -> Reading symbols from remote:/usr/bin/xz...warning: the debug information found in "/usr/lib/debug/usr/bin//xz.debug" does not match "remote:/usr/bin/xz" (CRC mismatch). warning: Ignoring non-absolute filename: <remote:/usr/bin/xz> Missing separate debuginfo for remote:/usr/bin/xz Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/3b/f71a5d02162aefbe30f77a626e71b0b0a165d9.debug Reading symbols from remote:/usr/bin/xz...(no debugging symbols found)...done. (no debugging symbols found)...done. No symbol table is loaded. Use the "file" command. remote$ gdbserver :1234 xz local$ gdb -ex 'set sysroot remote:' -ex 'set debug-file-directory remote:/usr/lib/debug' -ex 'target remote host1s:1234' -ex 'symbol-file remote:/usr/bin/xz' -ex 'l main' -> Reading symbols from remote:/usr/bin/xz...warning: the debug information found in "/usr/lib/debug/usr/bin//xz.debug" does not match "remote:/usr/bin/xz" (CRC mismatch). warning: Ignoring non-absolute filename: <remote:/usr/bin/xz> warning: Ignoring non-absolute filename: <remote/.build-id/3b/f71a5d02162aefbe30f77a626e71b0b0a165d9.debug /usr/lib/debug/.build-id/3b/f71a5d02162aefbe30f77a626e71b0b0a165d9.debug> Missing separate debuginfo for remote:/usr/bin/xz Try: yum --enablerepo='*debug*' install remote/.build-id/3b/f71a5d02162aefbe30f77a626e71b0b0a165d9.debug /usr/lib/debug/.build-id/3b/f71a5d02162aefbe30f77a626e71b0b0a165d9.debug Reading symbols from remote:/usr/bin/xz...(no debugging symbols found)...done. (no debugging symbols found)...done. No symbol table is loaded. Use the "file" command. remote$ gdbserver :1234 xz local$ gdb -ex 'set sysroot remote:' -ex 'set debug-file-directory /host1/var/lib/mock/fedora-19-x86_64/root/usr/lib/debug' -ex 'target remote host1s:1234' -ex 'symbol-file remote:/usr/bin/xz' -ex 'l main' -> Reading symbols from remote:/usr/bin/xz...Reading symbols from /host1/var/lib/mock/fedora-19-x86_64/root/usr/lib/debug/usr/bin/xz.debug...done. done. [...] 142 int 143 main(int argc, char **argv)
There are probably multiple causes for this not working. One is that the colon character is used as a separator for debug-file-directory so instead of search for "target:/usr/lib/debug/<...>" it will search for "target/<...>" and "/usr/lib/debug/<...>" locally. And since the directory to search for a symbol file for a remote file already has a "target:" prefix (if I understood the code correctly), than it will be looking for /usr/lib/debug/target:/.../<...> locally anyhow. The "target:" prefix will be in the middle of the path. A newer gdb with "set debug separate-debug-file 1" can help show what is going on.
I hit this also in an attempt to use gdbserver within a container where the container's debug symbols could be easily installed such that we'd be testing/debugging in the same environment where the original problem experienced. FROM mariadb:10.6 RUN sed -i -e 's/$/ main\/debug/' /etc/apt/sources.list.d/mariadb.list \ && apt-get update \ && apt-get install -y mariadb-server-core-$MARIADB_MAJOR-dbgsym gdbserver \ && rm -rf /var/lib/apt/lists/* EXPOSE 2345 USER mysql CMD ["gdbserver", "localhost:2345", "mysqld"] After running the container file above: podman run --cap-add=SYS_PTRACE --volume mdbdata:/var/lib/mysql -p 2345:2345 --name mdb10.6 --rm mariadb_local:10.6-debug From the host: (gdb) set debug-file-directory target:/usr/lib/debug (gdb) target remote localhost:2345 Remote debugging using localhost:2345 Reading /usr/sbin/mariadbd from remote target... warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead. Reading /usr/sbin/mariadbd from remote target... Reading symbols from target:/usr/sbin/mariadbd... Reading /usr/sbin/e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading /usr/sbin/.debug/e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading target//usr/sbin/e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading target/usr/sbin//e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading target:target/usr/sbin//e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading /usr/lib/debug//usr/sbin/e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading /usr/lib/debug/usr/sbin//e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Reading target:/usr/lib/debug/usr/sbin//e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug from remote target... Missing separate debuginfo for target:/usr/sbin/mariadbd Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/b9/e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug ... Container is a ubuntu:focal based so the Fedora gdb path resolution is a bit mismatched, however if "set debug-file-directory target:/usr/lib/debug" was processed, then the next manual "symbol target:/usr/lib/debug/.build-id/b9/e5c6ca8600f8f6ec5fd85ef069200b1cdb3c7d.debug" to load it won't have been necessary. While getting the right overlayfs directory would avoid the "target:" need it quickly falls afoul of selinux and user id mapping complications. So it would be really nice if target: could actually mean target rather than a directory in this path, even if it complicates a bit of compatibility. Thanks for your consideration.