Backtrace at assert:
...
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff4b85b01 in __GI_abort () at abort.c:79
#2  0x0000000000a9bd08 in dump_core () at /home/vries/gdb_versions/devel/src/gdb/utils.c:204
#3  0x0000000000a9c21a in internal_vproblem(internal_problem *, const char *, int, const char *, typedef __va_list_tag __va_list_tag *) (problem=0x15aa6c0 <internal_error_problem>, 
    file=0xe346e0 "/home/vries/gdb_versions/devel/src/gdb/infrun.c", line=6384, 
    fmt=0xe34269 "%s: Assertion `%s' failed.", ap=0x7fffffffcb98)
    at /home/vries/gdb_versions/devel/src/gdb/utils.c:414
#4  0x0000000000a9c2e2 in internal_verror (
    file=0xe346e0 "/home/vries/gdb_versions/devel/src/gdb/infrun.c", line=6384, 
    fmt=0xe34269 "%s: Assertion `%s' failed.", ap=0x7fffffffcb98)
    at /home/vries/gdb_versions/devel/src/gdb/utils.c:439
#5  0x0000000000d39725 in internal_error (
    file=0xe346e0 "/home/vries/gdb_versions/devel/src/gdb/infrun.c", line=6384, 
    fmt=0xe34269 "%s: Assertion `%s' failed.")
    at /home/vries/gdb_versions/devel/src/gdbsupport/errors.cc:55
#6  0x000000000074b047 in process_event_stop_test (ecs=0x7fffffffd270)
    at /home/vries/gdb_versions/devel/src/gdb/infrun.c:6383
#7  0x000000000074ad3b in handle_signal_stop (ecs=0x7fffffffd270)
    at /home/vries/gdb_versions/devel/src/gdb/infrun.c:6277
#8  0x0000000000749232 in handle_inferior_event (ecs=0x7fffffffd270)
    at /home/vries/gdb_versions/devel/src/gdb/infrun.c:5530
#9  0x00000000007456e4 in fetch_inferior_event ()
    at /home/vries/gdb_versions/devel/src/gdb/infrun.c:3912
#10 0x000000000072af38 in inferior_event_handler (event_type=INF_REG_EVENT)
    at /home/vries/gdb_versions/devel/src/gdb/inf-loop.c:42
#11 0x000000000078584c in handle_target_event (error=0, client_data=0x0)
    at /home/vries/gdb_versions/devel/src/gdb/linux-nat.c:4060
#12 0x0000000000d3a447 in handle_file_event (file_ptr=0x56c3aa0, ready_mask=1)
    at /home/vries/gdb_versions/devel/src/gdbsupport/event-loop.cc:575
#13 0x0000000000d3a9cf in gdb_wait_for_event (block=0)
    at /home/vries/gdb_versions/devel/src/gdbsupport/event-loop.cc:701
#14 0x0000000000d39859 in gdb_do_one_event ()
    at /home/vries/gdb_versions/devel/src/gdbsupport/event-loop.cc:212
#15 0x0000000000a26343 in wait_sync_command_done ()
    at /home/vries/gdb_versions/devel/src/gdb/top.c:526
#16 0x0000000000a263bb in maybe_wait_sync_command_done (was_sync=0)
    at /home/vries/gdb_versions/devel/src/gdb/top.c:543
#17 0x0000000000a26953 in execute_command (p=0x7fffffffe15d "", from_tty=0)
    at /home/vries/gdb_versions/devel/src/gdb/top.c:670
#18 0x00000000007ae648 in catch_command_errors (
    command=0xa263d4 <execute_command(char const*, int)>, arg=0x7fffffffe15c "n", from_tty=0)
    at /home/vries/gdb_versions/devel/src/gdb/main.c:448
#19 0x00000000007ae80b in execute_cmdargs (cmdarg_vec=0x7fffffffd830, file_type=CMDARG_FILE, 
    cmd_type=CMDARG_COMMAND, ret=0x7fffffffd80c)
    at /home/vries/gdb_versions/devel/src/gdb/main.c:533
#20 0x00000000007afa29 in captured_main_1 (context=0x7fffffffda30)
    at /home/vries/gdb_versions/devel/src/gdb/main.c:1203
#21 0x00000000007afc15 in captured_main (data=0x7fffffffda30)
    at /home/vries/gdb_versions/devel/src/gdb/main.c:1224
#22 0x00000000007afc80 in gdb_main (args=0x7fffffffda30)
    at /home/vries/gdb_versions/devel/src/gdb/main.c:1249
#23 0x0000000000417f6e in main (argc=28, argv=0x7fffffffdb38)
    at /home/vries/gdb_versions/devel/src/gdb/gdb.c:32
...

At assert:
...
(gdb) p ecs->event_thread->control.exception_resume_breakpoint
$1 = (breakpoint *) 0x0
(gdb) p ecs->event_thread->control
$2 = {step_resume_breakpoint = 0x64e4930, exception_resume_breakpoint = 0x0, 
  single_step_breakpoints = 0x0, step_range_start = 4292450, step_range_end = 4292462, 
  step_start_function = 0x2590110, may_range_step = 0, step_frame_id = {
    stack_addr = 140737488346048, code_addr = 4292391, special_addr = 0, 
    stack_status = FID_STACK_VALID, code_addr_p = 1, special_addr_p = 0, 
    artificial_depth = 0}, step_stack_frame_id = {stack_addr = 140737488346048, 
    code_addr = 4292391, special_addr = 0, stack_status = FID_STACK_VALID, code_addr_p = 1, 
    special_addr_p = 0, artificial_depth = 0}, trap_expected = 0, proceed_to_finish = 0, 
  in_infcall = 0, step_over_calls = STEP_OVER_ALL, stop_step = 0, stop_bpstat = 0x570db10, 
  stepping_command = 1}
...

Just disabling the assert doesn't work, it makes gdb hang.

Hmm, it seems this reproduces in the test-suite:
...
(gdb) PASS: gdb.cp/nextoverthrow.exp: post-check - next over a throw 3
next^M
/home/vries/gdb_versions/devel/src/gdb/infrun.c:6384: internal-error: void process_event_stop_test(execution_control_state*): Assertion `ecs->event_thread->control.exception_resume_breakpoint != NULL' failed.^M
A problem internal to GDB has been detected,^M
further debugging may prove unreliable.^M
Quit this debugging session? (y or n) FAIL: gdb.cp/nextoverthrow.exp: next past catch (GDB internal error)
...

(In reply to Tom de Vries from comment #4)
> Hmm, it seems this reproduces in the test-suite:
> ...
> (gdb) PASS: gdb.cp/nextoverthrow.exp: post-check - next over a throw 3
> next^M
> /home/vries/gdb_versions/devel/src/gdb/infrun.c:6384: internal-error: void
> process_event_stop_test(execution_control_state*): Assertion
> `ecs->event_thread->control.exception_resume_breakpoint != NULL' failed.^M
> A problem internal to GDB has been detected,^M
> further debugging may prove unreliable.^M
> Quit this debugging session? (y or n) FAIL: gdb.cp/nextoverthrow.exp: next
> past catch (GDB internal error)
> ...

I also managed to reproduce this at gdb-8.3-branch branchpoint.

I haven't seen this failure before, but it's caused by the backport for gcc PR97774 - "Incorrect line info for try/catch" ( https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97774 ) landing on the compiler I'm using.

So, this should be reproducible for anyone using gcc-10 or later.

So, minimal version:
...
     1  #include <iostream>
     2
     3  void
     4  function1 (int val)
     5  {
     6    throw val;
     7  }
     8
     9  void
    10  function3 (int val)
    11  {
    12    {
    13      try
    14        {
    15          function1 (val);
    16        }
    17      catch (...) 
    18        {
    19          std::cout << "Caught and handled function1 exception" << std::endl;
    20        }
    21    }
    22  }
    23
    24  int
    25  main (void) 
    26  { 
    27    function3 (3);
    28
    29    return 0;
    30  }
...

Compiled like:
...
$ $ g++ nextoverthrow.cc -g
...

Reproduces like so:
...
$ gdb -batch ./a.out -ex 'b 27' -ex run -ex next
Breakpoint 1 at 0x400ae4: file nextoverthrow.cc, line 27.

Breakpoint 1, main () at nextoverthrow.cc:27
27        function3 (3);
infrun.c:6384: internal-error: void process_event_stop_test(execution_control_state*): Assertion `ecs->event_thread->control.exception_resume_breakpoint != NULL' failed.
...

With -ex "set debug infrun 1", we see:
...
Breakpoint 1, main () at nextoverthrow.cc:27
27        function3 (3);
[infrun] infrun_async: enable=0
[infrun] clear_proceed_status_thread: process 4142
[infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT
[infrun] start_step_over: step-over queue now empty
[infrun] start_step_over: resuming [process 4142] for step-over
[infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=1, current thread [process 4142] at 0x400ae4
[infrun] infrun_async: enable=1
[infrun] prepare_to_wait: prepare_to_wait
[infrun] proceed: [process 4142] resumed
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x400ae9
[infrun] process_event_stop_test: stepping inside range [0x400ae4-0x400aee]
[infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [process 4142] at 0x400ae9
[infrun] prepare_to_wait: prepare_to_wait
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x400a83
[infrun] process_event_stop_test: stepped into subroutine
[infrun] insert_step_resume_breakpoint_at_sal_1: inserting step-resume breakpoint at 0x400aee
[infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [process 4142] at 0x400a83
[infrun] prepare_to_wait: prepare_to_wait
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x7ffff74af050
[infrun] process_event_stop_test: BPSTAT_WHAT_SET_LONGJMP_RESUME
[infrun] insert_exception_resume_breakpoint: exception resume at 400a9b
[infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=1, current thread [process 4142] at 0x7ffff74af050
[infrun] prepare_to_wait: prepare_to_wait
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x7ffff74af054
[infrun] process_event_stop_test: BPSTAT_WHAT_SET_LONGJMP_RESUME
[infrun] insert_exception_resume_from_probe: exception resume at 0x400a9b
[infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=1, current thread [process 4142] at 0x7ffff74af054
[infrun] prepare_to_wait: prepare_to_wait
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x7ffff74af055
[infrun] process_event_stop_test: step-resume breakpoint is inserted
[infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [process 4142] at 0x7ffff74af055
[infrun] prepare_to_wait: prepare_to_wait
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x400a9b
[infrun] process_event_stop_test: BPSTAT_WHAT_CLEAR_LONGJMP_RESUME
[infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [process 4142] at 0x400a9b
[infrun] prepare_to_wait: prepare_to_wait
[infrun] target_wait (-1.0.0, status) =
[infrun]   4142.4142.0 [process 4142],
[infrun]   status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_inferior_event: status->kind = stopped, signal = GDB_SIGNAL_TRAP
[infrun] handle_signal_stop: stop_pc=0x400a9b
[infrun] process_event_stop_test: BPSTAT_WHAT_CLEAR_LONGJMP_RESUME
/home/vries/gdb_versions/devel/src/gdb/infrun.c:6384: internal-error: void process_event_stop_test(execution_control_state*): Assertion `ecs->event_thread->control.exception_resume_breakpoint != NULL' failed.
...

So, the first time we do BPSTAT_WHAT_CLEAR_LONGJMP_RESUME, we delete ecs->event_thread->control.exception_resume_breakpoint.

Then we choose case 3, to keep going:
...
                /* Case 3.  */
                keep_going (ecs);
                return;
...

But it seems some somehow we hit again BPSTAT_WHAT_CLEAR_LONGJMP_RESUME at the same stop_pc. And then run into the assert, because we've deleted it the first time round.

So, this seems to fix it:
...
diff --git a/gdb/infrun.c b/gdb/infrun.c
index 2e5e837452d..3a57cf110a4 100644
--- a/gdb/infrun.c
+++ b/gdb/infrun.c
@@ -6410,6 +6410,7 @@ process_event_stop_test (struct execution_control_state *ecs)
            else
              {
                /* Case 3.  */
+               ecs->event_thread->stepping_over_breakpoint = 1;
                keep_going (ecs);
                return;
              }
...

Resulting in:
...
$ gdb -batch ./a.out -ex 'b 27' -ex run -ex next
Breakpoint 1 at 0x400ae4: file nextoverthrow.cc, line 27.

Breakpoint 1, main () at nextoverthrow.cc:27
27        function3 (3);
Caught and handled function1 exception
29        return 0;
...

Is this a proper fix?

(In reply to Tom de Vries from comment #7)
> So, this seems to fix it:
> ...
> diff --git a/gdb/infrun.c b/gdb/infrun.c
> index 2e5e837452d..3a57cf110a4 100644
> --- a/gdb/infrun.c
> +++ b/gdb/infrun.c
> @@ -6410,6 +6410,7 @@ process_event_stop_test (struct
> execution_control_state *ecs)
>             else
>               {
>                 /* Case 3.  */
> +               ecs->event_thread->stepping_over_breakpoint = 1;
>                 keep_going (ecs);
>                 return;
>               }
> ...
> 

FTR, survived regression test.

(In reply to Tom de Vries from comment #5)
> So, this should be reproducible for anyone using gcc-10 or later.

Confirmed, I reproduced this on openSUSE Tumbleweed with system gcc 10.2.1, and debuginfo for libgcc installed.

In the beginning of the test-case, we check for skip_unwinder_tests, which first checks for the presence of _Unwind_DebugHook.  Without debuginfo for libgcc, it's not found, with debuginfo for libgcc, it is.

FWIW, I found an apparently never committed gcc patch ( https://gcc.gnu.org/legacy-ml/gcc-patches/2010-04/msg01647.html ) to make the symbol part of the lib interface, such that it can be found without libgcc debuginfo installed.

Hmm, I start to wonder whether the problem is that we install two exception_master breakpoints:
- one using _Unwind_DebugHook
- one using probes.

In create_exception_master_breakpoint we have:
...
   for (objfile *objfile : current_program_space->objfiles ())
     {
        /* Try using probes.  */
        if (/* successful */)
          continue;

        /* Try using _Unwind_DebugHook */
     }
...

What happens is that we first encounter libgcc.debug. We try using probes, and this fails, so we try _Unwind_DebugHook, which succeeds.

Then we encounter libgcc.  We try using probes, and this succeeds.

I'm not sure why we keep trying after installing a master breakpoint.  Can there be one per objfile?

Either way, the root cause seems to be that the probes vs _Unwind_DebugHook preference mechanism does not take seperate debug info objfiles into account.

Created attachment 13094 [details]
tentative patch, splits loop in create_exception_master_breakpoint into two loops

Submitted patch: https://sourceware.org/pipermail/gdb-patches/2021-January/174712.html

The master branch has been updated by Tom de Vries <vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=1940319c0ef6271cc42a5d4c359be90063948710

commit 1940319c0ef6271cc42a5d4c359be90063948710
Author: Tom de Vries <tdevries@suse.de>
Date:   Fri Jan 8 11:11:16 2021 +0100

    [gdb] Fix internal-error in process_event_stop_test
    
    The function create_exception_master_breakpoint in gdb/breakpoint.c attempts
    to set a master exception breakpoint in each objfile.  It tries this using
    a libgcc/unwind probe, and if that fails then using the
    _Unwind_DebugHook symbol:
    ...
       for (objfile *objfile : current_program_space->objfiles ())
         {
            /* Try using probes.  */
            if (/* successful */)
              continue;
    
            /* Try using _Unwind_DebugHook */
         }
    ...
    
    The preference scheme works ok both if the objfile has debug info, and if it's
    stripped.
    
    But it doesn't work when the objfile has a .gnu_debuglink to a .debug file
    (and the .debug file is present).  What happens is that:
    - we first encounter objfile libgcc.debug
    - we try using probes, and this fails
    - so we try _Unwind_DebugHook, which succeeds
    - next we encounter objfile libgcc
    - we try using probes, and this succeeds.
    So, we end up with a master exception breakpoint in both libgcc (using probes)
    and libgcc.debug (using _Unwind_DebugHook).
    
    This eventually causes:
    ...
    (gdb) PASS: gdb.cp/nextoverthrow.exp: post-check - next over a throw 3
    next^M
    src/gdb/infrun.c:6384: internal-error: \
      void process_event_stop_test(execution_control_state*): \
      Assertion `ecs->event_thread->control.exception_resume_breakpoint != NULL' \
      failed.^M
    A problem internal to GDB has been detected,^M
    further debugging may prove unreliable.^M
    Quit this debugging session? (y or n) FAIL: gdb.cp/nextoverthrow.exp: next
    past catch (GDB internal error)
    ...
    
    To trigger this internal-error, we need to use gcc-10 or later to compile the
    test-case, such that it contains the fix for gcc PR97774 - "Incorrect line
    info for try/catch".
    
    Fix this by only trying to install the master exception breakpoint in
    libgcc.debug using the _Unwind_DebugHook method, if the install using probes
    in libgcc failed.
    
    Tested on x86_64-linux.
    
    gdb/ChangeLog:
    
    2021-01-08  Tom de Vries  <tdevries@suse.de>
    
            PR gdb/26881
            * breakpoint.c (create_exception_master_breakpoint_probe)
            (create_exception_master_breakpoint_hook): Factor out
            of ...
            (create_exception_master_breakpoint): ... here.  Only try to install
            the master exception breakpoint in objfile.debug using the
            _Unwind_DebugHook method, if the install using probes in objfile
            failed.

The gdb-10-branch branch has been updated by Tom de Vries <vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=360ecc531331f9e2709a08aabb38586eddae0d5d

commit 360ecc531331f9e2709a08aabb38586eddae0d5d
Author: Tom de Vries <tdevries@suse.de>
Date:   Fri Jan 8 11:58:41 2021 +0100

    [gdb] Fix internal-error in process_event_stop_test
    
    The function create_exception_master_breakpoint in gdb/breakpoint.c attempts
    to set a master exception breakpoint in each objfile.  It tries this using
    a libgcc/unwind probe, and if that fails then using the
    _Unwind_DebugHook symbol:
    ...
       for (objfile *objfile : current_program_space->objfiles ())
         {
            /* Try using probes.  */
            if (/* successful */)
              continue;
    
            /* Try using _Unwind_DebugHook */
         }
    ...
    
    The preference scheme works ok both if the objfile has debug info, and if it's
    stripped.
    
    But it doesn't work when the objfile has a .gnu_debuglink to a .debug file
    (and the .debug file is present).  What happens is that:
    - we first encounter objfile libgcc.debug
    - we try using probes, and this fails
    - so we try _Unwind_DebugHook, which succeeds
    - next we encounter objfile libgcc
    - we try using probes, and this succeeds.
    So, we end up with a master exception breakpoint in both libgcc (using probes)
    and libgcc.debug (using _Unwind_DebugHook).
    
    This eventually causes:
    ...
    (gdb) PASS: gdb.cp/nextoverthrow.exp: post-check - next over a throw 3
    next^M
    src/gdb/infrun.c:6384: internal-error: \
      void process_event_stop_test(execution_control_state*): \
      Assertion `ecs->event_thread->control.exception_resume_breakpoint != NULL' \
      failed.^M
    A problem internal to GDB has been detected,^M
    further debugging may prove unreliable.^M
    Quit this debugging session? (y or n) FAIL: gdb.cp/nextoverthrow.exp: next
    past catch (GDB internal error)
    ...
    
    To trigger this internal-error, we need to use gcc-10 or later to compile the
    test-case, such that it contains the fix for gcc PR97774 - "Incorrect line
    info for try/catch".
    
    Fix this by only trying to install the master exception breakpoint in
    libgcc.debug using the _Unwind_DebugHook method, if the install using probes
    in libgcc failed.
    
    Tested on x86_64-linux.
    
    gdb/ChangeLog:
    
    2021-01-08  Tom de Vries  <tdevries@suse.de>
    
            PR gdb/26881
            * breakpoint.c (create_exception_master_breakpoint_probe)
            (create_exception_master_breakpoint_hook): Factor out
            of ...
            (create_exception_master_breakpoint): ... here.  Only try to install
            the master exception breakpoint in objfile.debug using the
            _Unwind_DebugHook method, if the install using probes in objfile
            failed.

Fixed in both master and gdb 10.2.  Marking resolved-fixed.