[PATCH][gdb/guile] Don't allow libguile to change libgmp mem fns

[Ludovic Courtès]

Hi Tom,

Tom de Vries <tdevries@suse.de> skribis:

> The fact that libguile tries to set the libgmp memory functions is a bug which
> should be fixed starting version v3.0.6.

Yes.  Building Guile with mini-GMP is recommended in 3.0.6 and later.

> +++ b/gdb/guile/guile.c
> @@ -662,10 +662,32 @@ gdbscm_initialize (const struct extension_language_defn *extlang)
>    {
>      gdb::block_signals blocker;
>  
> +    /* There are libguile versions (f.i. v3.0.5) that by default call
> +       mp_get_memory_functions during initialization to install custom
> +       libgmp memory functions.  This is considered a bug and should be
> +       fixed starting v3.0.6.
> +       Before gdb commit 880ae75a2b7 "gdb delay guile initialization until
> +       gdbscm_finish_initialization", that bug had no effect for gdb,
> +       because gdb subsequently called mp_get_memory_functions to install
> +       its own custom functions in _initialize_gmp_utils.  However, since
> +       aforementioned gdb commit the initialization order is reversed,
> +       allowing libguile to install a custom malloc that is incompatible
> +       with the custom free as used in gmp-utils.c, resulting in a
> +       "double free or corruption (out)" error.
> +       Work around the libguile bug by saving the libgmp memory functions
> +       before guile initialization, and restoring them afterwards.  */
> +    void *(*alloc_func) (size_t);
> +    void *(*realloc_func) (void *, size_t, size_t);
> +    void (*free_func) (void *, size_t);
> +    mp_get_memory_functions (&alloc_func, &realloc_func, &free_func);
> +
>      /* scm_with_guile is the most portable way to initialize Guile.  Plus
>         we need to initialize the Guile support while in Guile mode (e.g.,
>         called from within a call to scm_with_guile).  */
>      scm_with_guile (call_initialize_gdb_module, NULL);
> +
> +    /* Restore libgmp memory functions.  */
> +    mp_set_memory_functions (alloc_func, realloc_func, free_func);

This code would lead to memory leaks because Guile would still think it
has its memory functions installed so it would never explicitly free GMP
memory.

Instead, you can do:

  scm_install_gmp_memory_functions = 0;

before the first call to ‘scm_with_guile’.  That works with 3.0, 2.2,
and 2.0.

HTH!

Ludo’.