open multiple instances of same library with dlmopen

[Ashish Sangwan <ashishsangwan2 at gmail dot com>]

I am using a shared library which has quite a few global variables,
used in almost
all the exported functions so the library functions are not thread safe.
My application creates multiple threads and each threads dynamically open this
library and to avoid using any synchronization between parallel calls
to the exported
functions, I copied the library multiple times with different names on disk with
each thread opening its own copy. This works OK, but obviously, the
multiple copies of the text section and
the read-private section unnecessary occupies the memory, whereas I
want each thread
to have just its own private data section.

Seems in this situation, dlmopen will be apt, so I tried using the
dlmopen with LM_ID_NEWLM instead of dlopen.
LM_ID_NEWLM    Cause the object to create a new link-map list as part
of loading.  Objects
        that  are opened on a new link-map list must express all of
their dependen-
        cies.
A lot of doubts are there after some (3-4)observations I made on the
dlmopen behavior.

For testing the behavior of "dlopen vs dlmopen" I used the following
trivial code
The libfoo code:
#include <stdio.h>
int global = 10;
void foo(void)
{
    global++;
    printf("Hello, I'm a shared library\n");
}

The app_dlopen will open 2 instance of this library (different inode numbers)
void (*foo1)(void);
int main(void)
{
    int i;
    void *handle;
    char name[40] = "\0";
    for (i = 0; i < 2 ; i++) {
        sprintf(name, "%s.%d", "libfoo.so", i);
        handle = dlopen(name,
(RTLD_NOW|RTLD_LOCAL|RTLD_DEEPBIND|RTLD_NODELETE));
        if (!handle) {
            printf("error %s\n", dlerror());
            printf("dlopen failed for %s\n", name);
            break;
        }
        foo1 = dlsym(handle, "foo");
        foo1();
    }
    scanf("%d", &i);
    return 0;
}

The app_dlmopen will open the same library twice (same inum)
void (*foo1)(void);
int main(void)
{
    int i;
    void *handle;

    for (i = 0; i < 2 ; i++) {
        handle = dlmopen(LM_ID_NEWLM, "libfoo.so", (RTLD_LOCAL |RTLD_LAZY));
        if (!handle) {
            printf("%s\n", dlerror());
            printf("dlopen failed for %d\n", i);
            break;
        }
        foo1 = dlsym(handle, "foo");
        foo1();
    }
    scanf("%d", &i);
    return 0;
}

First, I run the app_dlopen, the /proc/pid/maps is as expected:
(Only showing the mmaped libraries, also excluded are ld and libdl mappings)
7fa3b753b000-7fa3b753c000 r-xp 00000000 08:01 34603119
  /home/ashish/demo/libfoo.so.1
7fa3b753c000-7fa3b773b000 ---p 00001000 08:01 34603119
  /home/ashish/demo/libfoo.so.1
7fa3b773b000-7fa3b773c000 r--p 00000000 08:01 34603119
  /home/ashish/demo/libfoo.so.1
7fa3b773c000-7fa3b773d000 rw-p 00001000 08:01 34603119
  /home/ashish/demo/libfoo.so.1
7fa3b773d000-7fa3b773e000 r-xp 00000000 08:01 34603118
  /home/ashish/demo/libfoo.so.0
7fa3b773e000-7fa3b793d000 ---p 00001000 08:01 34603118
  /home/ashish/demo/libfoo.so.0
7fa3b793d000-7fa3b793e000 r--p 00000000 08:01 34603118
  /home/ashish/demo/libfoo.so.0
7fa3b793e000-7fa3b793f000 rw-p 00001000 08:01 34603118
  /home/ashish/demo/libfoo.so.0
7fa3b793f000-7fa3b7afa000 r-xp 00000000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7fa3b7afa000-7fa3b7cf9000 ---p 001bb000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7fa3b7cf9000-7fa3b7cfd000 r--p 001ba000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7fa3b7cfd000-7fa3b7cff000 rw-p 001be000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so

When I check the smaps, I can see the r-xp areas of the mmaped libc
being shared among different instances:
7fa3b793f000-7fa3b7afa000 r-xp 00000000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
Size:               1772 kB
Rss:                1088 kB
Pss:                   9 kB < This
Shared_Clean:       1088 kB < And This
Everything is per expectation till now.

The r--p mmaped area is strange
7fa3b7cf9000-7fa3b7cfd000 r--p 001ba000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
Size:                 16 kB
Rss:                  16 kB < All of it is in memory
Pss:                  16 kB < Even though it is read only, but not shared
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB < The read only mappings are dirty?
Referenced:           16 kB
Anonymous:            16 kB < We can see the inode number, but still
this vma counted as anon?

While looking at the strace output, I somehow managed to relate the
above log with mmap calls.
mmap(0x7fa3b7cf9000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ba000) = 0x7fa3b7cf9000
mprotect(0x7fa3b7cf9000, 16384, PROT_READ) = 0
Seems a read/write private mapping is 1st written upon which sets the
anon flag, followed by mprotect making it read only.

Question 1: What's written in the 16kb area?
Is it always 16kb or could be larger? Trying to prevent overwrite over
dirty pages?

Finally gets some status information for this application, we will
compare it with dlmopen.
cat /proc/3769/status:
VmRSS:        1364 kB
VmLib:        1932 kB

Second, run the dlmopen app:
7f586528a000-7f5865445000 r-xp 00000000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865445000-7f5865644000 ---p 001bb000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865644000-7f5865648000 r--p 001ba000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865648000-7f586564a000 rw-p 001be000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f586564a000-7f586564f000 rw-p 00000000 00:00 0
7f586564f000-7f5865650000 r-xp 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865650000-7f586584f000 ---p 00001000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f586584f000-7f5865850000 r--p 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865850000-7f5865851000 rw-p 00001000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865851000-7f5865a0c000 r-xp 00000000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865a0c000-7f5865c0b000 ---p 001bb000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865c0b000-7f5865c0f000 r--p 001ba000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865c0f000-7f5865c11000 rw-p 001be000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865c11000-7f5865c16000 rw-p 00000000 00:00 0
7f5865c16000-7f5865c17000 r-xp 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865c17000-7f5865e16000 ---p 00001000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865e16000-7f5865e17000 r--p 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865e17000-7f5865e18000 rw-p 00001000 08:01 34603117
  /home/ashish/demo/libfoo.so
7f5865e18000-7f5865fd3000 r-xp 00000000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f5865fd3000-7f58661d2000 ---p 001bb000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f58661d2000-7f58661d6000 r--p 001ba000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so
7f58661d6000-7f58661d8000 rw-p 001be000 08:01 25694285
  /lib/x86_64-linux-gnu/libc-2.19.so


OK, as expected of dlmopen, libfoo.so is mapped twice with the same
inode number in the same process.
The r-xp mapped area of the libfoo is shared
7f586564f000-7f5865650000 r-xp 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          4 kB

Again, the r--p mmaped area of libfoo is mapped anonymously and dirty,
so this part is not shared
memory and consumes memory privately.
7f586584f000-7f5865850000 r--p 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
7f5865e16000-7f5865e17000 r--p 00000000 08:01 34603117
  /home/ashish/demo/libfoo.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB

Obviously, rw-p should be different. And the ---p has the RSS 0, so
need not worry about that.
7f5865c17000-7f5865e16000 ---p 00001000 08:01 34603117
  /home/ashish/demo/libfoo.so
Size:               2044 kB
Rss:                   0 kB

The other difference I observed is that the number of libc mappings is
3 times compared to dlopen.
In app_dlopen, libc was mapped in 4 places where in this case it is
mapped at 12 places.

Question2: Why is that? If my app would have been using more
libraries, all of them would be
remapped or is it just the libc?

Due to this increased number of libc mappings, the overall mapped area
and RSS area is
surprisingly increased compared to the dlopen case:
VmRSS:        2720 kB
VmLib:        5476 kB

Question3: I understand the demo library I used is pretty small, so if
the actual
library is having large text section, can I expect an overall reduction in the
actual memory occupied by the complete process when using dlmopen?

I would really appreciate if someone could answer or give some pointers.

Thanks!!