In coredump-filter.c  there is this code:
void *ret = mmap (addr, size, prot, flags, fd, offset);
assert (ret != NULL);
The mmap function never returns NULL, on errors it returns MAP_FAILED (or -1). Thus this check is wrong and should probably be "ret != MAP_FAILED".
(Sidenote: asserts usually shouldn't be used for error checking, but this may be a design decision due to this being a test suite.)
Thanks for the report. Was this found using some static analysis tool, or you just stumbled on it?
I'll also fix the spots that check against -1 to use MAP_FAILED.
(In reply to Hanno Boeck from comment #0)
> (Sidenote: asserts usually shouldn't be used for error checking, but this
> may be a design decision due to this being a test suite.)
For test programs I think it's fine.
The master branch has been updated by Simon Marchi <email@example.com>:
Author: Simon Marchi <firstname.lastname@example.org>
Date: Wed May 20 10:50:39 2020 -0400
gdb/testsuite: check mmap ret val against MAP_FAILED
Fixup a few spots in the testsuite that use mmap to consistently check
the return value against MAP_FAILED.
One spot in gdb.base/coredump-filter.c checked against NULL, that is
wrong. The other spots either didn't check, or checked against -1.
MAP_FAILED has the value -1, at least on Linux, but it's better to check
against the documented define.
* gdb.base/coredump-filter.c (do_mmap): Check mmap ret val
* gdb.base/coremaker.c (mmapdata): Likewise.
* gdb.base/jit-reader-host.c (main): Likewise.
* gdb.base/sym-file-loader.c (load): Likewise.
(In reply to Simon Marchi from comment #1)
> Thanks for the report. Was this found using some static analysis tool, or
> you just stumbled on it?
So this is a very good question and the answer may surprise you :-)
I learned about this type of bug and I thought "I can write a trivial shellscript to find these kinds of bugs". So if my 6 lines of bash count as a static analysis tool then yes.
I plan to release it soon, I'll post a link.
Well, it worked well enough to find one bug, thanks!