Bug 22380 - Inconsistency detected by ld.so: dl-open.c: 717: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!
Summary: Inconsistency detected by ld.so: dl-open.c: 717: _dl_open: Assertion `_dl_deb...
Status: ASSIGNED
Alias: None
Product: glibc
Classification: Unclassified
Component: dynamic-link (show other bugs)
Version: unspecified
: P2 critical
Target Milestone: ---
Assignee: Florian Weimer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-11-01 07:05 UTC by Prasanth
Modified: 2021-03-26 17:39 UTC (History)
8 users (show)

See Also:
Host: ubuntu
Target:
Build:
Last reconfirmed: 2017-11-01 00:00:00
fweimer: security-


Attachments
attachment-17785-0.html (958 bytes, text/html)
2017-11-01 19:01 UTC, Prasanth
Details
sprof flag test (308 bytes, patch)
2020-02-23 13:23 UTC, Felipe Rodriguez
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Prasanth 2017-11-01 07:05:10 UTC
Hi, 
 I am trying to profile my libraries, I tried the example given in the MAN page of http://manpages.ubuntu.com/manpages/xenial/man1/sprof.1.html
  
It is giving error Inconsistency detected by ld.so: dl-open.c: 717: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed! 
with glibc 2.23, 2.18, 2.19 version.

I could see the #Bug 4578 which is listing the same error, but I couldn't make use of the patch in my host.
 
Can I get the fix for glibc version 2.19?
Comment 1 Carlos O'Donell 2017-11-01 18:58:58 UTC
(In reply to Prasanth from comment #0)
> Hi, 
>  I am trying to profile my libraries, I tried the example given in the MAN
> page of http://manpages.ubuntu.com/manpages/xenial/man1/sprof.1.html
>   
> It is giving error Inconsistency detected by ld.so: dl-open.c: 717:
> _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state ==
> RT_CONSISTENT' failed! 
> with glibc 2.23, 2.18, 2.19 version.
> 
> I could see the #Bug 4578 which is listing the same error, but I couldn't
> make use of the patch in my host.

Does your program multithreaded? Does it call fork? Does it use non-AS-safe functions in the forking thread *after* calling fork?
Comment 2 Prasanth 2017-11-01 19:01:58 UTC
Created attachment 10565 [details]
attachment-17785-0.html

No, it's not multi threading program, you could see in the man page it's
suit simple c program

On 02-Nov-2017 12:29 AM, "carlos at redhat dot com" <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=22380
>
> Carlos O'Donell <carlos at redhat dot com> changed:
>
>            What    |Removed                     |Added
> ------------------------------------------------------------
> ----------------
>                  CC|                            |allan at archlinux dot
> org,
>                    |                            |carlos at redhat dot com
>
> --- Comment #1 from Carlos O'Donell <carlos at redhat dot com> ---
> (In reply to Prasanth from comment #0)
> > Hi,
> >  I am trying to profile my libraries, I tried the example given in the
> MAN
> > page of http://manpages.ubuntu.com/manpages/xenial/man1/sprof.1.html
> >
> > It is giving error Inconsistency detected by ld.so: dl-open.c: 717:
> > _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state ==
> > RT_CONSISTENT' failed!
> > with glibc 2.23, 2.18, 2.19 version.
> >
> > I could see the #Bug 4578 which is listing the same error, but I couldn't
> > make use of the patch in my host.
>
> Does your program multithreaded? Does it call fork? Does it use non-AS-safe
> functions in the forking thread *after* calling fork?
>
> --
> You are receiving this mail because:
> You reported the bug.
Comment 3 Carlos O'Donell 2017-11-01 20:22:57 UTC
(In reply to Prasanth from comment #2)
> Created attachment 10565 [details]
> attachment-17785-0.html
> 
> No, it's not multi threading program, you could see in the man page it's
> suit simple c program

Prasanth,

Thank you.

The loading of the shared library for analysis fails, and we are in RT_ADD state instead of a consistent state:

706	  assert (_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT);
(gdb) p _r_debug
$1 = {r_version = 1, r_map = 0x7ffff7ffe130, r_brk = 140737351944064, r_state = RT_ADD, 
  r_ldbase = 140737351872512}

Then in _dl_find_dso_for_object:

241   if (__glibc_unlikely (mode & __RTLD_SPROF))
242     /* This happens only if we load a DSO for 'sprof'.  */
243     return;

We stop doing any work before marking the object as consistent. Which seems odd, perhaps I've done something wrong in my analysis.

This looks like sprof has been broken for a long time if this is true, and we've not had any regression test to cover this.

Florian,

Given, that you're poking around the dynamic loader, would you be able to provide any input on what might be wrong?

With upstream master I see:
"sprof: error while loading shared libraries: sprof: cannot open shared object file"

That doesn't make much sense though since the shared library is in the path.
Comment 4 Prasanth 2017-11-02 05:01:55 UTC
(In reply to Carlos O'Donell from comment #3)
> (In reply to Prasanth from comment #2)
> > Created attachment 10565 [details]
> > attachment-17785-0.html
> > 
> > No, it's not multi threading program, you could see in the man page it's
> > suit simple c program
> 
> Prasanth,
> 
> Thank you.
> 
> The loading of the shared library for analysis fails, and we are in RT_ADD
> state instead of a consistent state:
> 
> 706	  assert (_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT);
> (gdb) p _r_debug
> $1 = {r_version = 1, r_map = 0x7ffff7ffe130, r_brk = 140737351944064,
> r_state = RT_ADD, 
>   r_ldbase = 140737351872512}
> 
> Then in _dl_find_dso_for_object:
> 
> 241   if (__glibc_unlikely (mode & __RTLD_SPROF))
> 242     /* This happens only if we load a DSO for 'sprof'.  */
> 243     return;
> 
> We stop doing any work before marking the object as consistent. Which seems
> odd, perhaps I've done something wrong in my analysis.
> 
> This looks like sprof has been broken for a long time if this is true, and
> we've not had any regression test to cover this.
> 
> Florian,
> 
> Given, that you're poking around the dynamic loader, would you be able to
> provide any input on what might be wrong?
> 
> With upstream master I see:
> "sprof: error while loading shared libraries: sprof: cannot open shared
> object file"
> 
> That doesn't make much sense though since the shared library is in the path.

Hi carlos,

 I too tried with the upstream patch, That end-up with the "Can't find the object file error"
Comment 5 Prasanth 2017-11-28 05:50:45 UTC
Hi Florian,
 Have you got any workaround on this?
Comment 6 Alex Speasmaker 2018-06-12 19:42:02 UTC
I am also having this issue.

Ubuntu Xenial

$ ldd --version 

ldd (Ubuntu GLIBC 2.23-0ubuntu10) 2.23

running sprof on a generated profile data produces the following error:

Inconsistency detected by ld.so: dl-open.c: 717: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!
Comment 7 Walter Mollica 2018-08-05 16:41:46 UTC
Same issue here,
Ubuntu artful

$ ldd --version

ldd (Ubuntu GLIBC 2.26-0ubuntu2.1) 2.26

And running sprof on generated .profile data returns:

Inconsistency detected by ld.so: dl-open.c: 717: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!
Comment 8 Felipe Rodriguez 2020-02-23 13:20:18 UTC
I encountered this bug with sprof on my patched glibc-2.20 and confirmed that the same issue exists on Ubuntu 16.  I tried the earlier patch attached to this Bug but encountered a deadlock later in my system build processes.  Modifying logic which works fine except when a shared object is loaded by sprof's magic seems wrong so I removed the patch.  The Bug only occurs with sprof's loading so simply disabling the assertion when an object is loaded with the sprof flag __RTLD_SPROF allows me to use sprof while avoiding changes that might disturb the normal shared object loading mechanism.

+  if (!(__glibc_unlikely (mode & __RTLD_SPROF))) {
   assert (_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT);
+  }

I have attached a patch for glibc-2.20.
Comment 9 Felipe Rodriguez 2020-02-23 13:23:43 UTC
Created attachment 12314 [details]
sprof flag test

Disable consistency check assertion when loading with sprof
Comment 10 Ramesh Errabolu 2021-03-26 17:39:45 UTC
Wondering if there is a patched library I can get for Ubuntu 18.04.5 LTS for x86_64. The ldd version I have is: ldd (Ubuntu GLIBC 2.27-3ubuntu1.3) 2.27

If not can someone advise as to when a fix will be available.