How to run multiple target boards

[Simon Marchi]

On 2020-05-26 6:56 a.m., Tom de Vries wrote:
> On 26-05-2020 12:39, Pedro Alves wrote:
>> On 5/26/20 9:49 AM, Tom de Vries wrote:
>>> [ was: Re: [committed][gdb/testsuite] Fix incorrect string concat in
>>> jit-elf.exp ]
>>>
>>> On 13-05-2020 15:45, Simon Marchi wrote:
>>>> FWIW, I just learned it's possible to pass a list of target boards to --target_board:
>>>>
>>>>   --target_board='unix native-gdbserver native-extended-gdbserver'
>>>>
>>>> And it gives a combined result at the end.  So I'll make myself an alias to test on all
>>>> the commonly used boards, probably:
>>>>
>>>> - cc-with-debug-names.exp
>>>> - cc-with-dwz.exp
>>>> - cc-with-dwz-m.exp
>>>> - cc-with-gdb-index.exp
>>>> - debug-types.exp
>>>> - dwarf4-gdb-index.exp
>>>> - fission-dwp.exp
>>>> - fission.exp
>>>> - native-extended-gdbserver.exp
>>>> - native-gdbserver.exp
>>>> - native-stdio-gdbserver.exp
>>>> - readnow.exp
>>
>> What are the advantages of doing this compared to a script that runs the
>> testsuite for each of the boards, separately?
>>
> 
> I'd say:
> - the target boards not overwriting the .log and .sum files of other
>   target boards
> - the joint summary.
> 
> Thanks,
> - Tom
> 
>>>>
>>>> When working on a single test, it's usually not to long to run them all.  Of course, it's
>>>> not really practical to run the complete testsuite twice (before and after) for each board
>>>> for every change we do...
>>>
>>> Hi Simon,
>>>
>>> I just tried this out.
>>>
>>> My test scripts use make check, but I didn't manage to make that work
>>> yet, so I tried out using runtest directly (and I may be missing
>>> something obvious, given that I haven't used this before):
>>> ...
>>> $ cd build/gdb/testsuite
>>> $ runtest gdb.base/gold-gdb-index.exp --target_board='cc-with-gdb-index
>>>  unix'
>>> ...
>>
>> One issue this with that I've run into in the past, is with global
>> variables leaking from one board to the other.
>>
>> And I've just tried a quick test, and lucky me I seem to have run into
>> something like that immediately:
>>
>>  $ make check RUNTESTFLAGS="--target_board='native-gdbserver unix' break.exp"
>>  ...
>>  Schedule of variations:
>>      native-gdbserver
>>      unix
>>  ...
>>  FAIL: gdb.base/break.exp: run until function breakpoint
>>  FAIL: gdb.base/break.exp: run until breakpoint set at a line number (the program is no longer running)
>>  FAIL: gdb.base/break.exp: run until file:function(6) breakpoint (the program is no longer running)
>>  FAIL: gdb.base/break.exp: run until file:function(5) breakpoint (the program is no longer running)
>>  (snip a bunch more)
>>
>> (Running break.exp for each board in isolation passes cleanly, of course.)
>>
>> Another thing is that the test messages in gdb.sum don't indicate which
>> board issued the PASS/FAIL (on each result), making it more difficult
>> to analyze.

Ok, so in retrospect, not such a good idea :(.

Simon