Handling pgoff in perf elf mmap/mmap2 elf info

[Milian Wolff]

On Friday, September 21, 2018 3:07:29 PM CEST Mark Wielaard wrote:
> On Wed, 2018-09-19 at 14:24 +0200, Ulf Hermann wrote:
> > > We suspect perf to offset its recording-addresses of mmapped
> > > dsos/executables starting with a specific section, such that they
> > > denote
> > > their pointers with this pg_offset parameter. (e.g. skipping a
> > > library's
> > > header and setting pgoff to the headersize). Although we are not
> > > 100%
> > > sure about this information.
> > 
> > According to my understanding, the pgoff is not perf's invention.
> > Rather, the libary loader for the target application does not mmap()
> > the full ELF file, but only the parts it's interested in. Those
> > partial mappings are then reported through perf.
> 
> OK, so pgoff is like the offset argument of the mmap call?
> Is it just recording all user space mmap events that have PROT_EXEC in
> their prot argument? What about if the mapping was later changed with
> mprotect? Or does PERF_RECORD_MMAP only map to some internal kernel
> mmap action?
> 
> >  We then try to recreate the memory mapping with perfparser, but run
> > 
> > into problems because dwfl_report_elf() doesn't let us do partial
> > mappings. You can only map complete files with that function. There
> > probably is some way to manually map the relevant sections using
> > other functions in libdw and libelf, but I haven't figured out how to
> > do this, yet. If there is a simple trick I'm missing, I'd be happy to
> > hear about it. 
> > 
> > And, yes, a function that works like dwfl_report_elf, but takes a
> > pgoff and length as additional parameters is sorely missing from the
> > API.
> 
> dwfl_report_elf indeed does assume the whole ELF file is mapped in
> according to the PHDRs in the file and the given base address. But what
> you are actually seeing (I think, depending on the answers on the
> questions above) is the dynamic loader mapping in the file in pieces.
> And so you would like an interface where you can report the module
> piece wise while it is being mapped in. So what would be most
> convenient would be some kind of dwfl_report_elf_mmap function that you
> can use to either get a new Dwfl_Module or to extend an existing one.
> 
> I have to think how that interacts with mprotect and mmunmap.

Hey Mark,

I can only second what Christoph and Ulf said so far. I want to add though 
that this limitation essentially makes elfutils unusable for usage in perf. 
I.e., perf can be build with either libunwind or elfutils for unwinding 
callstacks. Using the former works like a charm. The latter runs into the same 
problems like perfparser / hotspot. To reproduce, use a modern distribution 
with recent userland and kernel, then do something like this:

~~~~~
$ cat test.cpp
#include <cmath>
#include <complex>
#include <iostream>
#include <random>

using namespace std;

int main()
{
    uniform_real_distribution<double> uniform(-1E5, 1E5);
    default_random_engine engine;
    double s = 0;
    for (int i = 0; i < 10000000; ++i) {
        s += norm(complex<double>(uniform(engine), uniform(engine)));
    }
    cout << s << '\n';
    return 0;
}
$ g++ -O2 -g test.cpp -o test
$ perf record --call-graph dwarf ./test
$ perf report --stdio -vv
...
overlapping maps:
 55a1cdd87000-55a1cdd89000 1000 /ssd/milian/projects/kdab/rnd/hotspot/tests/
test-clients/cpp-inlining/test
 55a1cdd85000-55a1cdd89000 0 /ssd/milian/projects/kdab/rnd/hotspot/tests/test-
clients/cpp-inlining/test
 55a1cdd85000-55a1cdd87000 0 /ssd/milian/projects/kdab/rnd/hotspot/tests/test-
clients/cpp-inlining/test
overlapping maps:
 7fdda4690000-7fdda46af000 2000 /usr/lib/ld-2.28.so
 7fdda468e000-7fdda46ba000 0 /usr/lib/ld-2.28.so
 7fdda468e000-7fdda4690000 0 /usr/lib/ld-2.28.so
 7fdda46af000-7fdda46ba000 0 /usr/lib/ld-2.28.so
...
57.14%     0.00%  test     [unknown]         [.] 0xe775b17d50ae8cff
            |
            ---0xe775b17d50ae8cff
~~~~~

To build perf against elfutils, do:

~~~~~
git clone --branch=perf/core git://git.kernel.org/pub/scm/linux/kernel/git/
acme/linux.git
cd linux/tools/perf
make NO_LIBUNWIND=1
~~~~~

The perf binary in the last folder can than be used as a drop-in replacement.

Since I consider this issue a serious blocker, I would like to see it fixed 
sooner rather than later. Would it maybe be possible for you to create a proof 
of concept for the new proposed dwfl_report_elf_mmap? I can then try to take 
it from there to fill in the missing bits and pieces and to make it actually 
work for our purposes.

Thanks
-- 
Milian Wolff
mail@milianw.de
http://milianw.de