Checks that autotools generated files were re-generated correctly

[Martin Jambor]

Hi,

On Wed, Nov 15 2023, Mark Wielaard wrote:
> Hi! (adding gdb and binutils to the CC)
>
> On Thu, Nov 09, 2023 at 12:30:59AM +0100, Mark Wielaard wrote:
>> On Mon, Nov 06, 2023 at 06:04:48PM +0100, Martin Jambor wrote:
>> > I have inherited Martin Liška's buildbot script that checks that all
>> > sorts of autotools generated files, mainly configure scripts, were
>> > re-generated correctly when appropriate.  While the checks are hopefully
>> > useful, they report issues surprisingly often and reporting them feels
>> > especially unproductive.
>> 
>> Cool! A small python script cannot replace him of course. But it is
>> nice to get a small virtual mliska :)
>> 
>> > Could such checks be added to our server side push hooks so that commits
>> > introducing these breakages would get refused automatically.  While the
>> > check might be a bit expensive, it only needs to be run on files
>> > touching the generated files and/or the files these are generated from.
>> 
>> So this doesn't just need that script, but also an execution
>> environment that contains the right versions of the autotools. We
>> could install those of course, but running them from a git hook on
>> checkin indeed seems a little expensive.
>> 
>> Creating a container with the script and the right versions of all
>> tools might be the best thing. Then the script can be run from a cron
>> job or buildbot. Or even by someone hacking on the build/configure
>> scripts to make sure they are generating with the right tools.
>> 
>> builder.sourceware.org already contains various of such containers:
>> https://sourceware.org/cgit/builder/tree/builder/containers
>> https://sourceware.org/cgit/builder/tree/README_containers
>> 
>> Friday is Sourceware Open Office hour (#overseers on irc.libera.chat
>> at 18:00 UTC). We could hack something together then and see how to
>> hook it up.
>
> So we did indeed discuss and hack something together.

Wonderful, thanks to everyone working to improve the situation with
these errors.

Martin

>
> The container file is here:
> https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
>
> The script is here:
> https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
>
> A buildbot can then use that container to run that script and then
> check that git diff is empty on every commit to binutils-gdb.git and
> gcc.git. If it finds an issue it will sent email with the diff.
>
> It already found issues in both gcc and binutils-gdb. All fixed now.
>
> Thanks to Arsen and Sam for testing, fixing and updating the script
> (it now also runs aclocal).
>
> If you want to run this locally you can use the container file to
> create an image and then run autoregen.py on your local tree by bind
> mounting your git tree inside it.
>
> $ git clone https://sourceware.org/git/builder.git
> $ cd builder/
> $ podman build -t autotools -f builder/containers/Containerfile-autotools \
>   builder/containers
> $ cd .. # assuming your binutils-gdb directory is here
> $ podman run --privileged -u root -v $(pwd)/binutils-gdb:/binutils-gdb \
>   -ti --entrypoint "/bin/bash" autotools
> # then inside the container
>  cd /binutils-gdb
>  autoregen.py
>
> Change binutils-gdb to gcc if you are working on gcc.
> You should also be able to do the above with docker instead of podman.
>
> Cheers,
>
> Mark