Re: [PATCH 2/2] GDB test suite: Get core files on targets with systemd-coredump

[Pedro Alves <palves at redhat dot com>]

On 10/17/2017 06:36 PM, Andreas Arnez wrote:
> On Tue, Oct 17 2017, Pedro Alves wrote:
> 
>> On 09/18/2017 07:41 PM, Andreas Arnez wrote:
>>> So far the test suite skips tests if they need system-generated core files
>>> and the core files can not be found.  In particular this is usually the
>>> case on systems with an active systemd-coredump service.  On such systems,
>>> core files are not written into the local directory, but made accessible
>>> via a command-line utitily "coredumpctl" instead.
>>>
>>> This patch enables processing core files on such systems as well.  Note
>>> that there are a few quirks:
>>>
>>> * In my tests, after invoking a command that dumps core, it could happen
>>>   that "coredumpctl" did not find the dump immediately afterwards.  After
>>>   waiting a bit, the dump was found and could be accessed.  Thus the patch
>>>   performs a single wait-and-retry in case of failure.
>>>
>>> * There does not seem to be a way for a user to remove specific core dumps
>>>   from the journal.  Thus it can happen that "coredumpctl" returns an old
>>>   dump, and the test case continues with that instead of the new one.  It
>>>   might be possible to improve the logic here, by considering the time
>>>   stamps as well.  I leave that for a future patch.
>>>
>>> * On the system I've tested it on, the bigcore.exp test case still failed
>>>   because coredumpctl truncated the core file after 4G for some reason.
>>
>> I'm a bit unsure about whether this is the right approach,
>> expecially given the caveats above.  Also, this seems to mean that
>> running the testsuite on such a system clutters the system log on and on,
>> maybe even triggers dispatch of notifications to admins, etc.
> 
> The caveats above are really bugs/design flaws of systemd-coredump.  If
> nothing else, maybe this discussion helps addressing them.  Offering no
> way to prevent system log cluttering could be viewed as another flaw,
> see also below.
> 
>> I wonder whether there's a way to tell systemd-coredump to
>> let the core dumps be generated on the file system for the current
>> shell environment?  Like we try to run "ulimit -c unlimited".
> 
> I agree that there *should* be such a way -- but I haven't found any.
> And such a mechanism should probably allow suppressing the log entries
> as well.
> 
>> Failing that, it may be better to instead make the testsuite skip
>> the tests gracefully, and display a big and visible warning
>> if systemd-coredump is detected as active.
> 
> This might be the right trade-off if we expect test cases to be executed
> only on systems that the user has full control over.  But I consider
> this restriction too tight and would prefer a "best effort" approach
> instead.  Maybe we should emit a warning *and* try our best to execute
> the test?

Not sure, really.  It seems like the "best effort" results in
racy tests, e.g., if "coredumpctl" returns an old dump, or
if coredumpctl decides to rate-limit core dump generation (which
according to the docs, it does).  It very much sounds like that
can lead to hard to diagnose problems and send GDB hackers tilting
at windmills.

> 
>> I mean, you already have to tweak other things in the system in
>> order to be able to run the testsuite correctly.  For example,
>> you have to tweak /proc/sys/kernel/yama/ptrace_scope to make
>> attach tests work at all, for example.  systemd-coredump kind of
>> seems like more of the same.
> 
> So should we document a sequence of admin commands that makes a system
> debug-ready, or in particular ready for the GDB test suite?

IMO, yes.  We already have something like that, but it's mixed with
the instructions for setting up builders:

  https://sourceware.org/gdb/wiki/BuildBot#Fedora-specific_instructions

(Note we already suggest disabling ABRT and tweaking 
kernel.core_pattern.)

It'd be great to move that info to some specific page about setting
up an environment for developing and testing GDB.  Also, some of
the command sequences there could move to scripts under gdb/contrib/,
IMHO.

> 
> But I'm not so sure about this.  IMHO a default mainstream Linux
> installation should be suited for development- and debugging purposes
> *without* any tweaking.  Also, if there are good reasons for a security
> measure, we shouldn't rely on disabling it globally.

I think that battle is lost.  Mainstream Linux installations are 
already very much not suited for development OOTB.  You have to install 
a bunch of development  packages that are not installed by default, 
before you can build  anything, including compiler, etc.  If you can
install packages, then you can also disable a few features that
really are not meant for development environments.  What we're missing
is a simple "one-click button" way to adapt an installation / user
environment for development.

> 
> With respect to Yama's ptrace scope, the distributions seem to differ.
> For instance, Fedora does not activate it by default
> (https://fedoraproject.org/wiki/Security_Features_Matrix), while Ubuntu
> does (https://wiki.ubuntu.com/Security/Features).  And I wonder whether
> this feature couldn't be adjusted to be more debug-friendly either.

The whole point of the feature is to prevent debugging, so I don't
see how, off hand.  Note that on Fedora nowadays, if you install GDB, 
then Yama's ptrace scope is relaxed via a rpm package dependency.
systemd-coredump could also be treated in the same way -- disable
it for users that have GDB or other development tools installed.
I don't actually know whether it's possible to disable systemd-coredump for a
single user, though it seems to me like an obviously desirable feature.

Thanks,
Pedro Alves