Augmenting ld.so Bootstrap Relocations for Unikernel Linux

[Carlos O'Donell]

On 5/9/24 08:20, Vance Raiti via Libc-help wrote:
> I am writing today to ask for guidance on augmenting the bootstrap
> relocations that ld.so performs.

Hello Vance!

I'm going to add libc-alpha (developer list) because this topic crosses boundaries
between help and net-new development.

Let me introduce you to the team at Red Hat that works on glibc (Florian, DJ, Arjun,
Patsy) for Red Hat Enterprise Linux (and the layered products that use RHEL).

> My name is Vance Raiti. I’m an undergraduate research assistant working
> with the Red Hat Collaboratory at Boston University on developing dynamic
> linkage support for Unikernel Linux (https://arxiv.org/pdf/2206.00789). The
> goal of this project is to allow any Linux application to run unmodified as
> a unikernel application, executing in supervisor mode and gaining access to
> kernel symbols.

Nice!

> In order to support this, we use a modified glibc (
> https://github.com/unikernelLinux/glibc/tree/ukl-dynamic) that, among other
> things, uses a special system call ABI: instead of issuing syscall
> instructions, glibc instead calls the kernel entry point (entry_SYSCALL_64)
> directly. For example, the following assembly is used in the
> internal_syscall# macros:
> 
>         asm volatile (
> 
>         "call entry_SYSCALL_64@PLT\n\t"
> 
>         : "=a" (resultvar)
> 
>         : "0" (number)
> 
>         : "memory", REGISTERS_CLOBBERED_BY_SYSCALL);
> 
> This symbol is provided to glibc by a shared object, libuklsyms.so. glibc
> is linked with this shared object by appending a few options to the
> build-shlib-helper and build-shlib definitions in glibc’s top-level
> Makerules file:

I assume that call doesn't follow the normal procedure call standard?

> define build-shlib-helper
> 
> $(LINK.o) -shared -static-libgcc -Wl,-O1 $(sysdep-LDFLAGS) \
> 
>           $(if $($(@F)-no-z-defs)$(no-z-defs),,-Wl,-z,defs) $(rtld-LDFLAGS)
> \
> 
>           $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx) \
> 
>           $(extra-B-$(@F:lib%.so=%).so) $(load-map-file) \
> 
>           -Wl,-soname=lib$(libprefix)$(@F:lib%.so=%).so$($(@F)-version) \
> 
>           $(LDFLAGS.so) $(LDFLAGS-lib.so) $(LDFLAGS-$(@F:lib%.so=%).so) \
> 
>           -L$(subst :, -L,$(rpath-link))
> -Wl,-rpath-link=$(rpath-link):$(common-objpfx).. \
> 
>           -L$(common-objpfx).. -Wl,-rpath=/data
> 
> endef
> 
> …
> 
> define build-shlib
> 
> $(build-shlib-helper) -o $@ $(shlib-lds-flags) \
> 
>           $(csu-objpfx)abi-note.o $(build-shlib-objlist) \
> 
>           -luklsyms
> 
> endef
> 
> We choose this approach because we can use a kernel module to update
> libuklsyms.so after the kernel has booted, allowing us to account for KASLR.

Makes sense.

> This works well for the normal glibc shared objects (libc.so,
> libpthread.so, etc.), however, it causes complications for the dynamic
> linker, ld.so. Since our system calls use the PLT, ld.so needs to relocate
> itself before issuing any. I’ve made it so that ld.so knows the address of
> entry_SYSCALL_64 ($r15 is initialized with its value by the kernel), so it
> is possible, in theory, for ld.so to relocate itself. I noticed that in
> _dl_start, one of the first things that ld.so does is perform bootstrap
> relocation on itself using boostrap_map. Would it be possible to add the
> entry_SYSCALL_64 address to boostrap_map such that ld.so could properly
> relocate entry_SYSCALL_64@PLT and make calls to it? If so, how could that
> be done?

Not easily IMO.

The problem is that you want to make syscalls before you can dlopen shared objects?

Is there any reason you can't extend the existing vDSO framework?

Is vDSO setup too late?

Do you need syscalls earlier?

You would likely have to create a fake link_map with all the information about libuklsyms.so,
make all the ld.so syscalls use PLT relocations, and then in the bootstrap map processing
the PLT relocations would be updated using the fake link_map. Then once you're past that point
you probably want to upgrade the fake link_map with any missing libuklsyms.so data. 

If you don't provide the fake link_map then you have to load libuklsyms.so... but I assume
it's already in memory somewhere. You'd want to create an internal-only fdlopen()-like interface
(dlopen from a memory-backed fd ala memfd_create()).

This seems more complicated than just providing the required minimal boostrap syscall set via
the vDSO interface whose information is passed via AT_SYSINFO/AT_SYSINFO_EHDR?

-- 
Cheers,
Carlos.