Re: Handling pgoff in perf elf mmap/mmap2 elf info

[Milian Wolff <mail at milianw dot de>]

On Mittwoch, 26. September 2018 16:38:43 CEST Milian Wolff wrote:
> On Friday, September 21, 2018 3:07:29 PM CEST Mark Wielaard wrote:
> > On Wed, 2018-09-19 at 14:24 +0200, Ulf Hermann wrote:
> > > > We suspect perf to offset its recording-addresses of mmapped
> > > > dsos/executables starting with a specific section, such that they
> > > > denote
> > > > their pointers with this pg_offset parameter. (e.g. skipping a
> > > > library's
> > > > header and setting pgoff to the headersize). Although we are not
> > > > 100%
> > > > sure about this information.
> > > 
> > > According to my understanding, the pgoff is not perf's invention.
> > > Rather, the libary loader for the target application does not mmap()
> > > the full ELF file, but only the parts it's interested in. Those
> > > partial mappings are then reported through perf.
> > 
> > OK, so pgoff is like the offset argument of the mmap call?
> > Is it just recording all user space mmap events that have PROT_EXEC in
> > their prot argument? What about if the mapping was later changed with
> > mprotect? Or does PERF_RECORD_MMAP only map to some internal kernel
> > mmap action?
> > 
> > >  We then try to recreate the memory mapping with perfparser, but run
> > > 
> > > into problems because dwfl_report_elf() doesn't let us do partial
> > > mappings. You can only map complete files with that function. There
> > > probably is some way to manually map the relevant sections using
> > > other functions in libdw and libelf, but I haven't figured out how to
> > > do this, yet. If there is a simple trick I'm missing, I'd be happy to
> > > hear about it.
> > > 
> > > And, yes, a function that works like dwfl_report_elf, but takes a
> > > pgoff and length as additional parameters is sorely missing from the
> > > API.
> > 
> > dwfl_report_elf indeed does assume the whole ELF file is mapped in
> > according to the PHDRs in the file and the given base address. But what
> > you are actually seeing (I think, depending on the answers on the
> > questions above) is the dynamic loader mapping in the file in pieces.
> > And so you would like an interface where you can report the module
> > piece wise while it is being mapped in. So what would be most
> > convenient would be some kind of dwfl_report_elf_mmap function that you
> > can use to either get a new Dwfl_Module or to extend an existing one.
> > 
> > I have to think how that interacts with mprotect and mmunmap.
> 
> Hey Mark,
> 
> I can only second what Christoph and Ulf said so far. I want to add though
> that this limitation essentially makes elfutils unusable for usage in perf.
> I.e., perf can be build with either libunwind or elfutils for unwinding
> callstacks. Using the former works like a charm. The latter runs into the
> same problems like perfparser / hotspot. To reproduce, use a modern
> distribution with recent userland and kernel, then do something like this:
> 
> ~~~~~
> $ cat test.cpp
> #include <cmath>
> #include <complex>
> #include <iostream>
> #include <random>
> 
> using namespace std;
> 
> int main()
> {
>     uniform_real_distribution<double> uniform(-1E5, 1E5);
>     default_random_engine engine;
>     double s = 0;
>     for (int i = 0; i < 10000000; ++i) {
>         s += norm(complex<double>(uniform(engine), uniform(engine)));
>     }
>     cout << s << '\n';
>     return 0;
> }
> $ g++ -O2 -g test.cpp -o test
> $ perf record --call-graph dwarf ./test
> $ perf report --stdio -vv
> ...
> overlapping maps:
>  55a1cdd87000-55a1cdd89000 1000 /ssd/milian/projects/kdab/rnd/hotspot/tests/
> test-clients/cpp-inlining/test
>  55a1cdd85000-55a1cdd89000 0
> /ssd/milian/projects/kdab/rnd/hotspot/tests/test- clients/cpp-inlining/test
>  55a1cdd85000-55a1cdd87000 0
> /ssd/milian/projects/kdab/rnd/hotspot/tests/test- clients/cpp-inlining/test
> overlapping maps:
>  7fdda4690000-7fdda46af000 2000 /usr/lib/ld-2.28.so
>  7fdda468e000-7fdda46ba000 0 /usr/lib/ld-2.28.so
>  7fdda468e000-7fdda4690000 0 /usr/lib/ld-2.28.so
>  7fdda46af000-7fdda46ba000 0 /usr/lib/ld-2.28.so
> ...
> 57.14%     0.00%  test     [unknown]         [.] 0xe775b17d50ae8cff
> 
>             ---0xe775b17d50ae8cff
> ~~~~~
> 
> To build perf against elfutils, do:
> 
> ~~~~~
> git clone --branch=perf/core git://git.kernel.org/pub/scm/linux/kernel/git/
> acme/linux.git
> cd linux/tools/perf
> make NO_LIBUNWIND=1
> ~~~~~
> 
> The perf binary in the last folder can than be used as a drop-in
> replacement.
> 
> Since I consider this issue a serious blocker, I would like to see it fixed
> sooner rather than later. Would it maybe be possible for you to create a
> proof of concept for the new proposed dwfl_report_elf_mmap? I can then try
> to take it from there to fill in the missing bits and pieces and to make it
> actually work for our purposes.

Hey Mark,

any news on this? Today, I spend some time reading through dwfl_report_elf.c 
and it's far from trivial for me to go from here to a dwfl_report_elf_mmap or 
similar.

If you are unable to work on a POC, can you guide us a bit please? Here are 
some questions from my side:

- The address mapping seems to be handled via dwfl_report_module, so do we 
want to call this once per mmap we encounter?

- If so, then maybe what's actually missing is some API to allow *setting* an 
Elf for a Dwfl_Module, i.e. such that dwfl_report_module becomes useful for 
public consumption? This would basically boil down to two functions: One to 
open an elf file as is done currently in dwfl_report_elf/__libdw_open_file. 
Then another function to assign the opened Elf to the Dwfl_Module, cf. the 
tail of __libdwfl_report_elf below the call to __libdwfl_elf_address_range.

- But then, in such a per-mmap module, what values would one set for the 
following properties?

	m->main.vaddr = vaddr;
	m->main.address_sync = address_sync;
	m->main_bias = bias;

Do we set the same values everywhere, or do these values then depend on which 
mmap section we are looking at? Generally, I still don't have a good enough 
knowledge of the Elf API nor elfutils to really know what I'm talking about 
here...

Thanks
-- 
Milian Wolff
mail@milianw.de
http://milianw.de

Attachment: signature.asc
Description: This is a digitally signed message part.