Bug 22390

The crashing command is run from the localedata directory in the source tree.  The full command line looks like this:

/builddir/build/BUILD/glibc-2.26-65-ga76376df7c/build-ppc64-redhat-linux/locale/localedef --alias-file=../intl/locale.alias --no-archive -i locales/nl_AW -c -f charmaps/UTF-8 --prefix=/builddir/build/BUILDROOT/glibc-2.26-16.fc27.ppc64 nl_AW

It is executed with the built glibc, not the installed one, although both versions are quite close in this case.

And it is necessary to set I18NPATH=. before running this command.

Also seen on arm and ppc64le.

Other crashes at the same place, with:

        0x00007ffffca60000 - 0x0000800000030000 is load15
remainder = 0x800000004000
r9 = 0x2c001
r31 = 0x800000004000

        0x00007ffffc910000 - 0x0000800000060000 is load16
remainder = 0x800000032c00
r9 = 0x2d401
r31 = 0x800000032c00

So far, I have only seen this with an explicit loader invocation.  But with that, it also reproduces with the installed glibc.

Created attachment 10571 [details]
log.4332

strace log from crash

Created attachment 10572 [details]
core.4332.xz

coredump for log.4332 run

Backtrace from core.4332:

#0  sysmalloc (nb=nb@entry=7340064, av=av@entry=0x7fffbe610ee8 <main_arena>)
    at malloc.c:2768
#1  0x00007fffbe4a9984 in _int_malloc (av=0x7fffbe610ee8 <main_arena>, 
    bytes=7340049) at malloc.c:4134
#2  0x00007fffbe4af92c in _int_realloc (nb=7340064, oldsize=3670032, 
    oldp=0x7fffffa35800, av=0x7fffbe610ee8 <main_arena>) at malloc.c:4626
#3  __GI___libc_realloc (oldmem=0x7fffffa35810, bytes=7340048) at malloc.c:3245
#4  0x0000000010049684 in xrealloc (p=<optimized out>, n=<optimized out>)
    at programs/xmalloc.c:102
#5  0x0000000010006938 in idx_table_add (value=142819, wc=<optimized out>, 
    t=0x7fffff267408) at programs/3level.h:127
#6  find_idx (ctype=ctype@entry=0x7fffff2673f0, 
    table=table@entry=0x7fffff267978, max=max@entry=0x7fffff267980, 
    act=act@entry=0x7fffff267988, idx=<optimized out>)
    at programs/ld-ctype.c:1234
#7  0x0000000010010934 in find_idx (idx=<optimized out>, act=<optimized out>, 
    max=<optimized out>, table=<optimized out>, ctype=<optimized out>)
    at programs/ld-ctype.c:1388
#8  ctype_read (ldfile=<optimized out>, result=<optimized out>, 
    charmap=0x7ffffd801610, repertoire_name=<optimized out>, 
    ignore_content=<optimized out>) at programs/ld-ctype.c:2312
#9  0x000000001003c3ec in locfile_read (result=0x7fffff266ea0, 
    charmap=0x7ffffd801610) at programs/locfile.c:173
#10 0x000000001000490c in load_locale (category=<optimized out>, 
    name=0x7fffff266e60 "i18n", repertoire_name=0x0, charmap=0x7ffffd801610, 
    copy_locale=0x0) at programs/localedef.c:621
#11 0x0000000010010c9c in ctype_read (ldfile=0x7fffff266ce0, 
    result=0x7fffff266950, charmap=0x7ffffd801610, repertoire_name=0x0, 
    ignore_content=<optimized out>) at programs/ld-ctype.c:2141
#12 0x000000001003c3ec in locfile_read (result=0x7fffff266950, 
    charmap=0x7ffffd801610) at programs/locfile.c:173
#13 0x000000001000490c in load_locale (category=<optimized out>, 
    name=0x7fffff266910 "nl_NL", repertoire_name=0x0, charmap=0x7ffffd801610, 
    copy_locale=0x0) at programs/localedef.c:621
#14 0x0000000010010c9c in ctype_read (ldfile=0x7ffffd801510, 
    result=0x7fffe4459b00, charmap=0x7ffffd801610, repertoire_name=0x0, 
    ignore_content=<optimized out>) at programs/ld-ctype.c:2141
#15 0x000000001003c3ec in locfile_read (result=0x7fffe4459b00, 
    charmap=0x7ffffd801610) at programs/locfile.c:173
#16 0x0000000010003720 in main (argc=<optimized out>, argv=0x7fffe445a120)
    at programs/localedef.c:252

remainder = 0x8000004b5830
r9 = 0x2a7d1
r31 = 0x8000004b5830

Memory map:

        0x0000000010000000 - 0x0000000010010000 is load1a
        0x0000000010010000 - 0x0000000010010000 is load1b
        0x0000000010060000 - 0x0000000010070000 is load2
        0x0000000010070000 - 0x0000000010080000 is load3
        0x00007fffb3e60000 - 0x00007fffb5300000 is load4
        0x00007fffb6110000 - 0x00007fffb75b0000 is load5
        0x00007fffb77d0000 - 0x00007fffb77d0000 is load6
        0x00007fffb7800000 - 0x00007fffb7800000 is load7
        0x00007fffb7810000 - 0x00007fffb7810000 is load8
        0x00007fffbe3e0000 - 0x00007fffbe3f0000 is load9a
        0x00007fffbe3f0000 - 0x00007fffbe3f0000 is load9b
        0x00007fffbe5f0000 - 0x00007fffbe5f0000 is load10
        0x00007fffbe600000 - 0x00007fffbe610000 is load11
        0x00007fffbe610000 - 0x00007fffbe620000 is load12
        0x00007fffbe620000 - 0x00007fffbe620000 is load13
        0x00007fffbe630000 - 0x00007fffbe650000 is load14
        0x00007fffbe650000 - 0x00007fffbe660000 is load15a
        0x00007fffbe660000 - 0x00007fffbe660000 is load15b
        0x00007fffbe690000 - 0x00007fffbe6a0000 is load16
        0x00007fffbe6a0000 - 0x00007fffbe6b0000 is load17
        0x00007fffe4430000 - 0x00007fffe4460000 is load18
        0x00007ffffd800000 - 0x00008000004e0000 is load19

If I'm not mistaken, 0x8000004b5830 + 16 is still within the 0x00007ffffd800000 - 0x00008000004e0000 mapping, and brk actually returned 0x00008000004e0000.  So this seems like a kernel bug.

Comments?

I forgot to mention that I see this with kernel-4.13.9-200.fc26.ppc64 (from Fedora).

I submitted this to the kernel people:

https://marc.info/?l=linux-mm&m=150972872411797&w=2
https://lists.ozlabs.org/pipermail/linuxppc-dev/2017-November/165567.html

(In reply to Andreas Schwab from comment #3)
> Also seen on arm and ppc64le.

The arm issue is likely different.  ppc64le is the same (I see the crash there as well).  The upstream kernel maintainers identified the 128 TB memory layout by default as the likely cause of this regression.

Created attachment 10574 [details]
reproducer.c

I'm attaching a simplified reproducer.  It needs to be run with an explicit linker invocation, so that the heap is placed above ld.so.

The reproducer is still probabilistic.  The increment might be so large that the second sbrk call fails.  But if the sbrk succeeds, the assignment at the end will always segfault when running under a kernel which has the bug.

Identified as a kernel bug.