Created attachment 8867 [details]
mbsrtowcs and wcsrtombs fail with len=SIZE_MAX

The len argument of the mbsrtowcs and wcsrtombs functions limits the number of elements stored into the destination array but it could be greater than the size of the array. Hence additions in the following codes fragments are invalid C (exhibit UB):

https://sourceware.org/git/?p=glibc.git;a=blob;f=wcsmbs/mbsrtowcs_l.c;h=d71934117d4e8aa894aa8d5f33ba2c308a7bf3d6;hb=HEAD#l107

 107       data.__outbuf = (unsigned char *) dst;
 108       data.__outbufend = data.__outbuf + len * sizeof (wchar_t);

https://sourceware.org/git/?p=glibc.git;a=blob;f=wcsmbs/wcsrtombs.c;h=ae303683383c3116a695bd0269c836a4f92c4cb9;hb=HEAD#l107

 107       data.__outbuf = (unsigned char *) dst;
 108       data.__outbufend = (unsigned char *) dst + len;

The functions also return wrong results wrongly return 0 when called with large len, e.g., with len=SIZE_MAX. The attached program prints "0 0" instead of "2 2".

Please note that fixing only wrong result is not enough, the root problem is invalid pointer arithmetic. See pr19411 for comparison.