Re: [RFC][PATCH 22/22] Add Infinity notes implementing td_ta_map_lwp2thr

[Adhemerval Zanella <adhemerval dot zanella at linaro dot org>]

On 22-10-2015 06:56, Gary Benson wrote:
> Adhemerval Zanella wrote:
>>> Em 21 de out de 2015, às 19:28, Gary Benson <gbenson@redhat.com> escreveu:
>>> Adhemerval Zanella wrote:
>>>>> On 21-10-2015 17:09, Gary Benson wrote:
>>>>> Adhemerval Zanella wrote:
>>>>>> AFAIK the only non-trivial build dependency for testing
>>>>>> is perl and I do not see a way forward to add another
>>>>>> interpreter dependency.  It will be even harder on
>>>>>> cross-compiling environment to do native testing with such
>>>>>> dependency.
>>>>>
>>>>> I8C and I8X run on the host, not the target.  You only need
>>>>> Python on the host.  It's probably already there :)
>>>>>
>>>>> And on the host, if you have Python and pip installed then you
>>>>> can install I8C and I8X with a single shell command.  It takes
>>>>> about 20 seconds on my <5Mbit connection and you don't need
>>>>> root access.
>>>>
>>>> That is not the case where you are bootstrapping on a platform
>>>> that does not have python enabled.
>>>
>>> Don't people usually cross-build to bootstrap new platforms?
>>
>> So the testing itself does not require to run Python natively? If it
>> is the case where you can the i8x tests in the build machine I agree
>> that it is not an impending reason.
> 
> Neither the compile step nor the test step require Python to run
> natively.  I8C and I8X both run only on the build host.
> 
>>>>>> So the compelling reason I see so far to do not make
>>>>>> assembly releases and to not add the python as build/tests
>>>>>> requirement is the testcases.  How hard would be to add a
>>>>>> non-python testcase on GLIBC to test such features?
>>>>>
>>>>> I8X has 1230 statements (lines of code that do something, as
>>>>> opposed to comments, blank lines and documentation).  I don't
>>>>> have figures for how Python compares to C in terms of line
>>>>> count but I figure you're looking at 2500-5000 lines of C and
>>>>> that's not including the testcases themselves or something to
>>>>> parse and execute them if you go that route.  Easily a month
>>>>> of work, and the result would likely be less flexible.
>>>>
>>>> I was thinking more like something a objdump or similar to just
>>>> dump the notes in a meaningful way.
>>>
>>> With respect that's not really the same thing at all.  The I8X
>>> testcase executes the functions multiple times with the correct
>>> endianness and wordsize for the target.  The generated functions
>>> are different for each platform too--they have structure offsets
>>> and other constants embedded in them, for example, and the note
>>> in this patch has three different variants depending on how the
>>> target does THREAD_SELF.  You can't even compare the operations
>>> because if a field in a structure has a zero offset on one
>>> platform and a nonzero offset on another then the platform with
>>> the nonzero offset will have extra adds that the compiler
>>> optimized away on the platform with the zero offset.
>>
>> Unless it requires runtime information where you can't obtain prior
>> hand due kernel/hardware interaction I do not see why this is not a
>> matter of parse the correct fields and information in the notes.
> 
> Sure, but what you're talking about is essentially recreating the
> note compiler.  I8C is currently 2699 Python statements plus a 917
> statement testsuite, so over 3500 Python statements to translate
> into a presumably less-expressive language.
> 
>> However I see that the infrastructure is already created using i8x,
>> so it would be a duplicate effort.
> 
> Yeah.  And I8X is testing a different thing too.  It's not checking
> the bytecode you got is the bytecode you expected, it's checking the
> result of executing the bytecode is the result you expected.

That's why I asked if you need to run natively, since the expected results
will only vary if you have runtime constraints.  But I do see that changing
the offsets or internal structures will probably ending to adjust the
tests expectations also.

> 
>>>> I think one strategy that we can use to decouple the need for a
>>>> python binary in build is:
>>>>
>>>> 1. Provide assembly generated notes from a defined I8X compiler
>>>>   along with I8X sources.
>>>> 2. Add a makefile rule to autogenerate the assembly file if a I8C
>>>>   compiler is presented in the system.
>>>> 3. For tests, only run is I8X is presented.
>>>
>>> That's not really viable for the reasons I detailed above.  You
>>> would have to ship generated assembly for every platform, and even
>>> that is not enough.  For example, on some platforms "sizeof
>>> (struct pthread)" is a constant that gets into notes.  If someone
>>> adds a field to struct pthread and doesn't regenerate the notes
>>> then you're shipping GLIBC with broken notes.  Are there CFLAGS
>>> the user could set that would change structure offsets?  Setting
>>> these would mean broken notes if you didn't have I8C installed.
>>> There are constants in kernel headers that end up in notes.  If
>>> one of these changes, you're shipping GLIBC with broken notes.
>>>
>>> There are so many silent failure modes here--and broken notes are
>>> far worse than no notes at all.  The only robust way to do this is
>>> to build the notes with the constants and offsets that you are
>>> building the rest of GLIBC with, and that means saying if you want
>>> a GLIBC with notes then you need to install this package on your
>>> build system.
>>
>> This can be done with proper documentation and in the release cycle:
>> the release maintainer will recreate the assembly annotations for
>> the release architectures.  My only concern is the kernel notes
>> dependency, however even those we need to take care since you also
>> do not know prior hand which version you will end using.
> 
> That's fragile.  It's also burdensome for the release maintainer.
> It also raises the barrier for people hacking on GLIBC, because not
> only do they need to know all the other things they need to know, they
> also now need to know about recreating notes.  Say somebody is hacking
> on NPTL, they don't know about updating notes, so the first thing they
> see is the debugger won't attach.  How long do they spend going round
> in circles until somebody says "oh, you need to run this command in
> this directory."?
> 
> It also doesn't solve the problem that if people are working without
> I8C and I8X installed then they are creating GLIBCs containing
> unchecked notes.  My position is users create GLIBCs containing
> fully-tested notes or they create GLIBCs creating no notes at all.
> 
> In either setup there's a process to create notes.  What you're
> saying is, "let's encode this process in documentation and let
> people figure it out."  What I'm saying is, "let's encode this in
> code and let the build host figure it out."

Yes it is a tradeoff of adding external dependency and making GLIBC
build independently of them. Also the exported NPTL structures do
not change in a dynamic way (struct pthread, for instance, changes
it's size twice in seven years).

Also, currently if someone is hacking on GLIBC and he will also need
to adjust libpthread_db anyway. I know it is not the best approach
and I do want to see a better strategy to handle it, but adding an
external tools with python dependency IMHO is not the best one.

I would expect such tool to be included in binutils and have only 
binutils own dependencies as requisites. I would also expect notes
definition language stability, meaning that a notes defined with a
older version will always be handled by new compiler version
(so building a older glibc notes version with a new notes compiler
won't break the build).

> 
> Thanks,
> Gary
>