Re: ppc64 vDSO in mainline

[Steve Munroe <sjmunroe at us dot ibm dot com>]

Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote on 03/23/2005 
06:06:22 PM:

> On Wed, 2005-03-23 at 17:13 -0600, Steve Munroe wrote:
> 
> > I am starting to look at conecting glibc to to the VDSO 
> > __kernel_gettimeofday(). We have a few things to work out.
> > 
> > So far my attempts to use (almost) normal require that the application 
be 
> > linked to vdso*.so to use extern or even dlsym resolution. 
> > 
> > extern  int  __attribute__ ((weak)) __kernel_gettimeofday(struct 
timeval 
> > *, void *);
> > 
> > It seem that ld will not put the reference into the .dynsym table 
unless 
> > it has found the symbol in a dso at link time. Link to the vdso*.so 
also 
> > allow the extern/dlsym to resolve.
> 
> Can't libc.so itself link against the vDSO rather than the application ?
> I don't like the idea of requiring _applications_ to link against it.

Lets start by separating the mechanism from policy discussion. How any 
executable binds to the vdso functions is mechanism. Which executables we 
allow to bind to the vdso is policy. Letting (perceived) policy dictate 
mechanism is bad design.

> 
> > So this becomes the first decision point. For binding libc functions 
> > internally to vdso function we could do ld.so magic to get the entry 
> > points directly out of the ELF HDR or out of the link_map (BTW I would 

> > appreciate some pointers in that direction as Alan Modra is on 
vacation 
> > ... not which function to call & parms to use.) But that is not a 
general 
> > solution and is unsatisfactory for performance sensitive functions 
like 
> > memcpy/memcmp (because it implies another level of indirection)
> 
> Yes. Also beware that the vDSO doesn't expose function descriptors, but
> the symbols exposed are offsets from the beginning of the vDSO. You'll
> need your own 'branch island', if possible in asm. One advantage though
> is that the vDSO does use nor affects the TOC, thus a call to a vDSO
> function doesn't need a TOC load nor a TOC reload. That can probably be
> hanlded with appropriate inline asm glue
> 
If we want to use C language and avoid clumsy asm hackery, we need to use 
build and use a valid function descriptor and function pointer. And normal 
C call by function pointer will save and load a new TOC. I was trying to 
help you out here because a TOC point is kinda useful to address any data 
the vdso function might need.

> > The first thing we would like is vdso call via PLT. This requires that 
the 
> > VDSO exports enough info to resolve the PLT ref. This seems simple for 

> > ppc32 as the target function address is needed from the link map. For 
> > PPC64 (and other 64-bit platforms) the VDSO will have the export 
> > preresolved .opd entries (The .opd entry is simply copied to the PLT 
> > entry).
> 
> The vDSO will not expose anything that isn't completely static. The vDSO
> is mapped read-only and shared between every processes, but can be
> mapped at different virtual addresses (the later isn't yet implemented,
> but the ability to do it is a design requirement). Thus it can't
> expose .opd entries that have already been resolved to point to the
> right virtual address as those can be different for every process. This
> is why the vDSO has been changed from the early implementations to not
> expose function descriptors, but simple symbols that are offsets from
> the start of the vDSO. This was debated many times on this list already.
> 
I understand the requirement that the VDSO image is RO and thus ld.so can 
not relocate (change) the vdso! This not exactly that same thing as 
static.

Binding involves relocations in the calling executable based on 
informatation exported in the .dynsym table of the target shared object. 
For PPC64 binding, also copies data (the .opd entry) from the target to 
the calling executables PLT. Both of these are RO from the prespective of 
the caller and the dynamic linker. 

The symbols from the .dynsym table are exported to the process via 
AT_SYSINFO_EHDR and the ld.so has already added those symbols to the link 
map. So the dynamic linker already has enough information to resolve the 
callers PLT for a call to __kernel_gettimofday for PPC vdso32. This is 
very interresting because if the target function is withing the low order 
32MB of the address space (and vdso32.so currently is), 
elf_machine_fixup_plt will update the PLT with a single branch absolute 
instruction. There is no faster linkage in PPC architecture. 

The problem I was trying to point out is that we don't ennough information 
(for ld.so) to set up the callers PLT and ld will not allocate the PLT 
slot, unless that execuable was linked against against the vdso32.so. That 
is a glibc configure/make system issue (if we agree that PLT linking to 
VDSO is a good idea). 

> (The initial implementation did provide OPDs, and ld.so could remap them
> using mprotect to trigger COW capability of the ppc64 vDSO, but that
> required a change to ld.so to enable relocation on vDSOs that was
> rejected, besides, I agree that it's not the right thing to do for
> performances reasons).
> 
Yes if the vdso64.so has .odp entries then vdso64.so must avoid exporting 
any relocations against its .opd. Either the vdso64.so can be bound at a 
static address (which allows the .opd entry contents to be resolved at 
link time). Or the kernel must perform any required relocations in the 
vdso and must remove those relocations from ELF header before it is made 
visible to ld.so.

> > Ben:
> > 
> > I did manage to call __kernel_gettimeofday(), but
> > 
> > There is also a problem with the current ppc vdso64.so where it 
exports 
> > the actual function address instead of the expected function 
descriptor 
> > address (.opd entry). 
> 
> Not even the address, unless I did something wrong, but an offset.

You linked the vdso to a static address so it is the actual address. 
Anyway the dynmaic linkers symbol resolutions should do any required 
arthimatic. 

> 
> > This requires the caller to manufacture a function 
> > descriptor at runtime and disallows any thoughts of normal (PLT) call 
> > binding. It would be helpful if the vdso64.so followed the ABI but the 

> > kernel will have to insure that the .opd entries are pre-resolved.
> 
> See debates on this list monthes ago :) I did that upon request from
> Ulrich, Roland, ...
> 
See my comment above. I agreed to removing the .opd to get around a 
problem in the short term. Now we are discussing design which should be 
looking long term.

> > There is also the question of the TOC. The currently all functions are 

> > leaf routines so a TOC is not technically required. But normal binding 

> > will will (try to) load a new TOC for the VDSO anyway.
> 
> vDSO will _not_ and never use/require a TOC by design. The compiler of
> course doesn't know it, but if the symbol is to be called by glibc
> itself, an appropriate asm glue can be used that avoids these. If the
> vDSO symbol is to be directly "linked" to the application by ld.so, then
> OPD's have to be constructed on the fly by ld.so, the value of the TOC
> in them is irrelevant.
> 
Like I mentioned about there are reasons why the vdso will need the TOC 
pointer set. Even static leaf routines can use the TOC to access large 
constants and there is the issues of how gettimeofday accesses the magic 
timer conversation values (which change periodically).

> Note that Paulus and I have been wondering wether it would be worth
> defining a new relocation type for the vDSO symbols to avoid special
> casing in ld.so. That relocation type would be for use by the vDSO,
> defined as "offset to function with no TOC", and might make things
> easier for ld.so to deal with those ... 
> 
I don't think this would help.

Steven J. Munroe
Linux on Power Toolchain Architect
IBM Corporation, Linux Technology Center