Re: [RFC][PATCH 22/22] Add Infinity notes implementing td_ta_map_lwp2thr

[Gary Benson <gbenson at redhat dot com>]

Adhemerval Zanella wrote:
> > Em 21 de out de 2015, às 19:28, Gary Benson <gbenson@redhat.com> escreveu:
> > Adhemerval Zanella wrote:
> > > > On 21-10-2015 17:09, Gary Benson wrote:
> > > > Adhemerval Zanella wrote:
> > > > > AFAIK the only non-trivial build dependency for testing
> > > > > is perl and I do not see a way forward to add another
> > > > > interpreter dependency.  It will be even harder on
> > > > > cross-compiling environment to do native testing with such
> > > > > dependency.
> > > > 
> > > > I8C and I8X run on the host, not the target.  You only need
> > > > Python on the host.  It's probably already there :)
> > > > 
> > > > And on the host, if you have Python and pip installed then you
> > > > can install I8C and I8X with a single shell command.  It takes
> > > > about 20 seconds on my <5Mbit connection and you don't need
> > > > root access.
> > > 
> > > That is not the case where you are bootstrapping on a platform
> > > that does not have python enabled.
> > 
> > Don't people usually cross-build to bootstrap new platforms?
> 
> So the testing itself does not require to run Python natively? If it
> is the case where you can the i8x tests in the build machine I agree
> that it is not an impending reason.

Neither the compile step nor the test step require Python to run
natively.  I8C and I8X both run only on the build host.

> > > > > So the compelling reason I see so far to do not make
> > > > > assembly releases and to not add the python as build/tests
> > > > > requirement is the testcases.  How hard would be to add a
> > > > > non-python testcase on GLIBC to test such features?
> > > > 
> > > > I8X has 1230 statements (lines of code that do something, as
> > > > opposed to comments, blank lines and documentation).  I don't
> > > > have figures for how Python compares to C in terms of line
> > > > count but I figure you're looking at 2500-5000 lines of C and
> > > > that's not including the testcases themselves or something to
> > > > parse and execute them if you go that route.  Easily a month
> > > > of work, and the result would likely be less flexible.
> > > 
> > > I was thinking more like something a objdump or similar to just
> > > dump the notes in a meaningful way.
> > 
> > With respect that's not really the same thing at all.  The I8X
> > testcase executes the functions multiple times with the correct
> > endianness and wordsize for the target.  The generated functions
> > are different for each platform too--they have structure offsets
> > and other constants embedded in them, for example, and the note
> > in this patch has three different variants depending on how the
> > target does THREAD_SELF.  You can't even compare the operations
> > because if a field in a structure has a zero offset on one
> > platform and a nonzero offset on another then the platform with
> > the nonzero offset will have extra adds that the compiler
> > optimized away on the platform with the zero offset.
> 
> Unless it requires runtime information where you can't obtain prior
> hand due kernel/hardware interaction I do not see why this is not a
> matter of parse the correct fields and information in the notes.

Sure, but what you're talking about is essentially recreating the
note compiler.  I8C is currently 2699 Python statements plus a 917
statement testsuite, so over 3500 Python statements to translate
into a presumably less-expressive language.

> However I see that the infrastructure is already created using i8x,
> so it would be a duplicate effort.

Yeah.  And I8X is testing a different thing too.  It's not checking
the bytecode you got is the bytecode you expected, it's checking the
result of executing the bytecode is the result you expected.

> > > I think one strategy that we can use to decouple the need for a
> > > python binary in build is:
> > > 
> > > 1. Provide assembly generated notes from a defined I8X compiler
> > >   along with I8X sources.
> > > 2. Add a makefile rule to autogenerate the assembly file if a I8C
> > >   compiler is presented in the system.
> > > 3. For tests, only run is I8X is presented.
> > 
> > That's not really viable for the reasons I detailed above.  You
> > would have to ship generated assembly for every platform, and even
> > that is not enough.  For example, on some platforms "sizeof
> > (struct pthread)" is a constant that gets into notes.  If someone
> > adds a field to struct pthread and doesn't regenerate the notes
> > then you're shipping GLIBC with broken notes.  Are there CFLAGS
> > the user could set that would change structure offsets?  Setting
> > these would mean broken notes if you didn't have I8C installed.
> > There are constants in kernel headers that end up in notes.  If
> > one of these changes, you're shipping GLIBC with broken notes.
> > 
> > There are so many silent failure modes here--and broken notes are
> > far worse than no notes at all.  The only robust way to do this is
> > to build the notes with the constants and offsets that you are
> > building the rest of GLIBC with, and that means saying if you want
> > a GLIBC with notes then you need to install this package on your
> > build system.
> 
> This can be done with proper documentation and in the release cycle:
> the release maintainer will recreate the assembly annotations for
> the release architectures.  My only concern is the kernel notes
> dependency, however even those we need to take care since you also
> do not know prior hand which version you will end using.

That's fragile.  It's also burdensome for the release maintainer.
It also raises the barrier for people hacking on GLIBC, because not
only do they need to know all the other things they need to know, they
also now need to know about recreating notes.  Say somebody is hacking
on NPTL, they don't know about updating notes, so the first thing they
see is the debugger won't attach.  How long do they spend going round
in circles until somebody says "oh, you need to run this command in
this directory."?

It also doesn't solve the problem that if people are working without
I8C and I8X installed then they are creating GLIBCs containing
unchecked notes.  My position is users create GLIBCs containing
fully-tested notes or they create GLIBCs creating no notes at all.

In either setup there's a process to create notes.  What you're
saying is, "let's encode this process in documentation and let
people figure it out."  What I'm saying is, "let's encode this in
code and let the build host figure it out."

Thanks,
Gary

-- 
http://gbenson.net/