Bug 11094 - cannot reexecute programs using the GDB JIT interface in shared libs
Summary: cannot reexecute programs using the GDB JIT interface in shared libs
Status: RESOLVED FIXED
Alias: None
Product: gdb
Classification: Unclassified
Component: breakpoints (show other bugs)
Version: 7.1
: P2 normal
Target Milestone: 7.12
Assignee: Pedro Alves
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-12-15 21:01 UTC by Török Edwin
Modified: 2016-10-06 12:07 UTC (History)
3 users (show)

See Also:
Host: x86_64-linux-gnu
Target: x86_64-linux-gnu
Build: x86_64-linux-gnu
Last reconfirmed: 2016-08-24 00:00:00


Attachments
testcase.sh (188 bytes, text/plain)
2009-12-15 21:01 UTC, Török Edwin
Details
bp_minus4.c (43 bytes, text/plain)
2009-12-15 21:02 UTC, Török Edwin
Details
shared.c (442 bytes, text/plain)
2009-12-15 21:03 UTC, Török Edwin
Details
proposed patch to fix the bug (524 bytes, patch)
2010-02-20 21:33 UTC, Reid Kleckner
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Török Edwin 2009-12-15 21:01:08 UTC
When trying to debug a program (for example ClamAV) that uses the GDB JIT inside
a shared lib (for example LLVM in libclamavc++.so), then I cannot reexecute the
program.
This happens when no code was registered with the JIT interface, such as when I
simply put a breakpoint on main, and try to run twice, the 2nd run will fail
telling me it couldn't reset breakpoint -4 (or -2).

I've created a small testcase to show this problem (will attach), here is the
output:
--- shared.c:                                                                
#include <stdint.h>                                                          
typedef enum                                                                 
{                                                                            
  JIT_NOACTION = 0,                                                          
  JIT_REGISTER_FN,                                                           
  JIT_UNREGISTER_FN                                                          
} jit_actions_t;                                                             

struct jit_code_entry
{                    
  struct jit_code_entry *next_entry;
  struct jit_code_entry *prev_entry;
  const char *symfile_addr;         
  uint64_t symfile_size;            
};                                  

struct jit_descriptor
{                    
  uint32_t version;  
  /* This type should be jit_actions_t, but we use uint32_t
     to be explicit about the bitwidth.  */                
  uint32_t action_flag;                                    
  struct jit_code_entry *relevant_entry;                   
  struct jit_code_entry *first_entry;                      
};                                                         

/* GDB puts a breakpoint in this function.  */
void __attribute__((noinline)) __jit_debug_register_code() { };

/* Make sure to specify the version statically, because the
   debugger may check the version before we can set it.  */
struct jit_descriptor __jit_debug_descriptor = { 1, 0, 0, 0 };
int foo() {                                                   
    return 4;                                                 
}                                                             

--- bp_minus4.c:
int foo();      
int main()      
{               
    return foo();
}                

gcc (Debian 4.4.2-5) 4.4.2
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

GNU gdb (GDB) 7.0-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/edwin/gdbbug/bp_minus4...done.
Breakpoint 1 at 0x4005f8: file bp_minus4.c, line 4.

Breakpoint 1, main () at bp_minus4.c:4
4           return foo();
/home/edwin/gdbbug/gdb.tmp:3: Error in sourced command file:
Warning:
Cannot insert breakpoint -2.
Error accessing memory address 0x7ffff7dfa5dc: Input/output error.

(gdb) maint info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004005f8 in main at bp_minus4.c:4
-2      jit events     keep y   0x00007ffff7dfa5dc
-9      shlib events   keep y   0x000000381da0dcc0 <*__GI__dl_debug_state>
(gdb) quit
A debugging session is active.

        Inferior 2 [process 18147] will be killed.

Quit anyway? (y or n) y
Comment 1 Török Edwin 2009-12-15 21:01:37 UTC
Created attachment 4472 [details]
testcase.sh

script to compile testcase and run gdb
Comment 2 Török Edwin 2009-12-15 21:02:01 UTC
Created attachment 4473 [details]
bp_minus4.c

the file containing main()
Comment 3 Török Edwin 2009-12-15 21:03:56 UTC
Created attachment 4474 [details]
shared.c

the file for the shared lib, this "implements" the JIT interface.
It doesn't actually implement anything, just declares the __jit* symbols, the
code is a copy of the one in GDB online manual.

LLVM has code that actually registers something with the JIT, but this bug can
be triggered without actually registering any code, hence this simple testcase.


Note that if the file is not compiled as a shared lib, then I can rerun the
program without any errors.
Comment 4 Török Edwin 2009-12-22 14:14:02 UTC
As discussed with rnk on IRC this might be the cause:
rnk: edwin, so the problem is the shared library gets loaded at a different
address on the next run?
rnk: and GDB doesn't update the address of the breakpoint it's trying to set?
edwin: rnk: thats very possible
edwin: the default is ASLR
edwin: so no shared lib will get loaded to same address
Comment 5 Paul Pluzhnikov 2009-12-22 16:32:59 UTC
Note that GDB turns off ASLR by default:

gdb64-cvs -nx -ex 'show disable-randomization' -ex 'quit'
GNU gdb (GDB) 7.0.50.20091211-cvs
...
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Disabling randomization of debuggee's virtual address space is on.
Comment 6 Török Edwin 2009-12-22 17:00:22 UTC
(In reply to comment #5)
> Note that GDB turns off ASLR by default:
> 
> gdb64-cvs -nx -ex 'show disable-randomization' -ex 'quit'
> GNU gdb (GDB) 7.0.50.20091211-cvs
> ...
> This GDB was configured as "x86_64-unknown-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Disabling randomization of debuggee's virtual address space is on.
> 

Same here.

I think it tries to set the breakpoint too early, before the shared lib is loaded.

First run:
(gdb) b main
Breakpoint 1 at 0x4005f8: file bp_minus4.c, line 4.
(gdb) r
Starting program: /home/edwin/gdbbug/bp_minus4

Breakpoint 1, main () at bp_minus4.c:4
4           return foo();
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x000000381da00af0  0x000000381da16a24  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00007ffff7dfa510  0x00007ffff7dfa628  Yes         ./shared.so
0x000000381de1e730  0x000000381df0292c  Yes (*)     /lib/libc.so.6
(*): Shared library is missing debugging information.

Second run:
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/edwin/gdbbug/bp_minus4
Warning:
Cannot insert breakpoint -2.
Error accessing memory address 0x7ffff7dfa5dc: Input/output error.

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x000000381da00af0  0x000000381da16a24  Yes (*)     /lib64/ld-linux-x86-64.so.2
(*): Shared library is missing debugging information.

And here is /proc/maps for the process
$ ps aux|grep gdb
edwin    10001  0.2  0.2  37260 10264 pts/1    S+   18:58   0:00 gdb ./bp_minus4
edwin    10020  0.0  0.0    220    24 pts/1    T    18:59   0:00
/home/edwin/gdbbug/bp_minus4
edwin    10033  0.0  0.0   7288   784 pts/2    S+   18:59   0:00 grep gdb

$ cat /proc/10020/maps
00400000-00401000 r-xp 00000000 fd:02 5389238                           
/home/edwin/gdbbug/bp_minus4
00600000-00601000 rw-p 00000000 fd:02 5389238                           
/home/edwin/gdbbug/bp_minus4
381da00000-381da1d000 r-xp 00000000 09:03 260632                        
/lib/ld-2.10.2.so
381dc1c000-381dc1e000 rw-p 0001c000 09:03 260632                        
/lib/ld-2.10.2.so
7ffff7ffe000-7ffff7fff000 r-xp 00000000 00:00 0                          [vdso]
7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Comment 7 Paul Pluzhnikov 2009-12-22 23:39:27 UTC
I am guessing that bp_jit_event breakpoints must be disabled here
(in addition to bp_breakpoint and bp_hardware_breakpoint):

// gdb/breakpoint.c

static void
disable_breakpoints_in_unloaded_shlib (struct so_list *solib)
{
...
  ALL_BP_LOCATIONS (loc, locp_tmp)
  {
    struct breakpoint *b = loc->owner;
    if ((loc->loc_type == bp_loc_hardware_breakpoint
...
        && (b->type == bp_breakpoint || b->type == bp_hardware_breakpoint)
        && solib_contains_address_p (solib, loc->address))
      {
        loc->shlib_disabled = 1;

[There are likely several more places where bp_jit_event should be treated same
as bp_breakpoint.]
Comment 8 Reid Kleckner 2010-02-20 21:33:08 UTC
Created attachment 4614 [details]
proposed patch to fix the bug
Comment 9 Reid Kleckner 2010-02-20 21:47:44 UTC
(In reply to comment #8)

This suddenly became important for unladen swallow as soon as we tried to link
to LLVM as a shared library.  :)

I tested my patch with unladen, and it correctly registers JITed code on the
first run and any subsequent run.  Can someone review and commit this patch for
me?  Thanks!
Comment 10 Török Edwin 2010-03-21 20:39:33 UTC
This is still an issue with GDB 7.1:

(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
The program being debugged has been started already.
Starting program: /home/edwin/clam/git/builds/debug/clambc/.libs/lt-clambc
/tmp/foo.cbc
Warning:
Cannot insert breakpoint -15.
Error accessing memory address 0x7ffff7735510: Input/output error.

Good news is that GDB 7.1 doesn't crash anymore when trying to rerun the program
(it happened quite often with 7.0).
Comment 11 cvs-commit@gcc.gnu.org 2010-03-22 17:36:46 UTC
Subject: Bug 11094

CVSROOT:	/cvs/src
Module name:	src
Changes by:	tromey@sourceware.org	2010-03-22 17:36:28

Modified files:
	gdb            : ChangeLog breakpoint.c 

Log message:
	2010-03-22  Reid Kleckner  <reid@kleckner.net>
	
	PR gdb/11094
	* breakpoint.c (disable_breakpoints_in_unloaded_shlib): Add
	bp_jit_event.
	(disable_breakpoints_in_shlibs): Likewise.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog.diff?cvsroot=src&r1=1.11510&r2=1.11511
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/breakpoint.c.diff?cvsroot=src&r1=1.462&r2=1.463

Comment 12 Tom Tromey 2010-03-22 17:37:54 UTC
I checked in Reid's fix to cvs trunk.
It will show up in 7.2.
If you try it and still have problems, please reopen this PR, thanks.
Comment 13 laruence 2016-08-24 09:02:36 UTC
this bug still exists in 7.10 and latest 7.11
Comment 14 Pedro Alves 2016-08-24 13:12:45 UTC
With master, GDB crashes with a seg fault:

Program received signal SIGSEGV, Segmentation fault.
0x00000000005ee894 in event_location_to_string (location=0x0) at /home/pedro/gdb/mygit/src/gdb/location.c:412
412       if (EL_STRING (location) == NULL)
(top-gdb) bt
#0  0x00000000005ee894 in event_location_to_string (location=0x0) at /home/pedro/gdb/mygit/src/gdb/location.c:412
#1  0x000000000057411a in print_breakpoint_location (b=0x18288e0, loc=0x0) at /home/pedro/gdb/mygit/src/gdb/breakpoint.c:6201
#2  0x000000000057483f in print_one_breakpoint_location (b=0x18288e0, loc=0x182cf10, loc_number=0, last_loc=0x7fffffffd258, allflag=1)
    at /home/pedro/gdb/mygit/src/gdb/breakpoint.c:6473
#3  0x00000000005751e1 in print_one_breakpoint (b=0x18288e0, last_loc=0x7fffffffd258, allflag=1) at /home/pedro/gdb/mygit/src/gdb/breakpoint.c:6707
#4  0x000000000057589c in breakpoint_1 (args=0x0, allflag=1, filter=0x0) at /home/pedro/gdb/mygit/src/gdb/breakpoint.c:6947
#5  0x0000000000575aa8 in maintenance_info_breakpoints (args=0x0, from_tty=0) at /home/pedro/gdb/mygit/src/gdb/breakpoint.c:7026
#6  0x00000000004b74c4 in do_cfunc (c=0xedc760, args=0x0, from_tty=0) at /home/pedro/gdb/mygit/src/gdb/cli/cli-decode.c:105
#7  0x00000000004ba420 in cmd_func (cmd=0xedc760, args=0x0, from_tty=0) at /home/pedro/gdb/mygit/src/gdb/cli/cli-decode.c:1888
#8  0x000000000073f089 in execute_command (p=0x16a4426 "", from_tty=0) at /home/pedro/gdb/mygit/src/gdb/top.c:668
#9  0x0000000000624191 in command_handler (command=0x16a4410 "maint info breakpoints") at /home/pedro/gdb/mygit/src/gdb/event-top.c:620
#10 0x000000000073ec9c in read_command_file (stream=0x1715db0) at /home/pedro/gdb/mygit/src/gdb/top.c:481
#11 0x00000000004bd42e in script_from_file (stream=0x1715db0, file=0x7fffffffdca5 "/home/pedro/tmp/pr11094/gdb.tmp") at /home/pedro/gdb/mygit/src/gdb/cli/cli-script.c:1703
#12 0x00000000004be16e in source_script_from_stream (stream=0x1715db0, file=0x7fffffffdca5 "/home/pedro/tmp/pr11094/gdb.tmp", 
    file_to_open=0x7fffffffdca5 "/home/pedro/tmp/pr11094/gdb.tmp") at /home/pedro/gdb/mygit/src/gdb/cli/cli-cmds.c:578
#13 0x00000000004be22f in source_script_with_search (file=0x7fffffffdca5 "/home/pedro/tmp/pr11094/gdb.tmp", from_tty=1, search_path=0)
    at /home/pedro/gdb/mygit/src/gdb/cli/cli-cmds.c:618
#14 0x00000000004be262 in source_script (file=0x7fffffffdca5 "/home/pedro/tmp/pr11094/gdb.tmp", from_tty=1) at /home/pedro/gdb/mygit/src/gdb/cli/cli-cmds.c:628
#15 0x000000000061b0f5 in catch_command_errors_const (command=0x4be23d <source_script(char const*, int)>, arg=0x7fffffffdca5 "/home/pedro/tmp/pr11094/gdb.tmp", from_tty=1)
    at /home/pedro/gdb/mygit/src/gdb/main.c:402
#16 0x000000000061c469 in captured_main (data=0x7fffffffd720) at /home/pedro/gdb/mygit/src/gdb/main.c:1122
#17 0x000000000061c50a in gdb_main (args=0x7fffffffd720) at /home/pedro/gdb/mygit/src/gdb/main.c:1159
#18 0x000000000040d38d in main (
During symbol reading, cannot get low and high bounds for subprogram DIE at 24065.
argc=5, argv=0x7fffffffd828) at /home/pedro/gdb/mygit/src/gdb/gdb.c:32
(top-gdb)
Comment 15 Pedro Alves 2016-08-24 13:17:15 UTC
This is GDB trying to print the location spec of the jit breakpoint, but that's an internal breakpoint without one.

If I add a NULL check, then we see that the JIT breakpoint ended up pending.  

But that's incorrect.  GDB should have managed to recreate the JIT breakpoint's locations for the second run.  So the problem is earlier.

The problem seems to me to be that we don't discard the breakpoint when the  objfile with the JIT is unloaded.
Comment 16 Pedro Alves 2016-08-24 13:18:30 UTC
The fix seems quite small and safe.  Tentatively setting target milestone to 7.12.
Comment 17 Pedro Alves 2016-08-24 15:51:49 UTC
Fix posted:
  https://sourceware.org/ml/gdb-patches/2016-08/msg00249.html
Comment 18 laruence 2016-08-26 07:21:26 UTC
(In reply to Pedro Alves from comment #17)
> Fix posted:
>   https://sourceware.org/ml/gdb-patches/2016-08/msg00249.html

Thanks, it's great.
Comment 19 Pedro Alves 2016-10-06 12:07:35 UTC
Fixed in master and 7.12.