During fclose() of an open_memstream() FILE*, _IO_mem_finish() does
*mp->bufloc = (char *) realloc (fp->_IO_write_base,
fp->_IO_write_ptr - fp->_IO_write_base + 1);
If this realloc() fails, fclose() still returns 0 (success), but the buffer will now be NULL, which is unexpected.
I realize the new size is not bigger than the current size of the buffer, so a "sane" realloc() implementation probably won't fail here. But realloc() is allowed to be replaced by the user, and there's no reason it can't fail arbitrarily. In my case I'm trying to test my own error paths by injecting allocation failures, but I'm encountering this bug through systemd's libnss plugin, manifesting as this assertion failure:
Assertion 's' failed at src/shared/json.c:1760, function json_variant_format(). Aborting.
The relevant systemd code is here: https://github.com/systemd/systemd/blob/908dbc70d6abeb9f65624d89fb5ca021815d69ae/src/shared/json.c#L1745-L1773
systemd does not check the return value of fclose(), assuming the previous fflush() is enough to catch any errors. So rather than propagating the error in this case, I think the ideal fix would be to use fp->_IO_write_base if the realloc() fails.