[PATCH] Fix undefined behaviour inconsistent for strtok

[Wilco Dijkstra]

Hi,

>+  if ((s == NULL) && ((s = olds) == NULL))
>+    return NULL;
>
> What is the benefit of this given:
>
>  if (s == NULL)
>    /* This token finishes the string.  */
>    olds = __rawmemchr (token, '\0');

Right, looking at this a bit more, if we call strcspn rather than strpbrk,
we get the end of the string for free and avoid 2 calls.

It looks like the reason behind the (s = olds) == NULL check is purely
performance. Checking *s == '\0' at the start is slightly faster still (and
would seem better as it catches incorrect use of strtok).

With these changes you get > 2x speedup for most cases. I'll post a patch.

Wilco