Re: [RFC][BZ #16009] Memory handling in strxfrm_l()

[Rich Felker <dalias at libc dot org>]

On Thu, Nov 20, 2014 at 12:10:29PM +0100, Leonhard Holz wrote:
> 4. Line 156: Handling of failed malloc(). If malloc fails strxfrm_l
> tries to allocate the memory on the stack. This looks a bit weird to
> me since if the needed memory is too large for the heap then it's
> probably too large for the stack. Also alloca has a bad fail
> behaviour, best case it terminates the program with a stack overflow
> error, otherwise the "program behavior is undefined" (man page). So
> I think one should avoid allocation attempts larger than
> __libc_use_alloca recommends in any case.

Indeed, this is a serious bug.

> 5. Handling of failed memory allocation. Since the last ressort in
> the current implementation is a stack overflow there is currently no
> "function could not compute" path. I'd like to add that in case the
> needed buffer is too large at all (regarding size_t) or too large
> for alloca + malloc fails. For the return value the man page states:
> "The strxfrm() function returns the number of bytes required to
> store the transformed string in dest excluding the terminating null
> byte ('\0'). If the value returned is n or more, the contents of
> dest are indeterminate." - Does that imply that returning n is a
> good idea to communicate an error?

No, the function is not permitted to return an error; it's required by
ISO C to produce a result. Falsely reporting that it needs more space
for the result, and thereby causing the caller to keep allocating
larger and larger buffers until it runs out of memory itself, is not
valid; in particular, it could report different needed lengths for the
same string at different calls in the ame program with the same
locale.

If strcoll_l is using an algorithm that requires allocation, this
needs to be fixed -- there's no fundamental reason it needs to
allocate.

> 6. Use the given parameter n instead of srclen = strlen(src) for
> computing the buffer size. Since only n bytes are written into dst
> the number of weights and rules to cache should be limited by this
> parameter and calling strlen could be avoided. Does this sound
> reasonable?

I'm not sure about this; I think you should check carefully if it's
actually valid.

Rich