Re: [PATCH] glibc: Remove CPU set size checking from affinity functions [BZ #19143]

[Florian Weimer <fweimer at redhat dot com>]

On 03/02/2016 03:12 PM, Michael Kerrisk wrote:

> With this change, I wonder if some pieces in the sched_setaffinity(2)
> man page (http://man7.org/linux/man-pages/man2/sched_setaffinity.2.html)
> may be in order. Below, I've quoted some relevant pieces from the man
> page. Anything there that needs tweaking?
> 
>    ERRORS
>        EINVAL The affinity bit mask mask contains no processors that  are
>               currently  physically  on  the  system and permitted to the
>               thread according to any restrictions that may be imposed by
>               the "cpuset" mechanism described in cpuset(7).
> 
>        EINVAL (sched_getaffinity()   and,   in   kernels   before  2.6.9,
>               sched_setaffinity()) cpusetsize is smaller than the size of
>               the affinity mask used by the kernel.
>    [...]
>    NOTES
>    [...]
>      Handling systems with large CPU affinity masks
>        The  underlying  system  calls  (which  represent CPU masks as bit
>        masks of type unsigned long *) impose no restriction on  the  size
>        of  the  CPU mask.  However, the cpu_set_t data type used by glibc
>        has a fixed size of 128 bytes, meaning that the maximum CPU number
>        that  can be represented is 1023.  If the kernel CPU affinity mask
>        is larger than 1024, then calls of the form:
> 
>            sched_getaffinity(pid, sizeof(cpu_set_t), &mask);
> 
>        will fail with the error EINVAL, the error produced by the  underâ
>        lying  system  call  for the case where the mask size specified in
>        cpusetsize is smaller than the size of the affinity mask  used  by
>        the  kernel.   (Depending  on  the system CPU topology, the kernel
>        affinity mask can be  substantially  larger  than  the  number  of
>        active CPUs in the system.)

That's still true.

>        When  working on systems with large kernel CPU affinity masks, one
>        must dynamically allocate the mask argument.  Currently, the  only
>        way  to  do  this  is by probing for the size of the required mask
>        using sched_getaffinity() calls with increasing mask sizes  (until
>        the call does not fail with the error EINVAL).

I think this needs to reference the CPU_ALLOC manual page.

One caveat is that sched_getaffinity can set bits beyond the requested
allocation size (in bits) because the kernel gets a padded CPU vector
and sees a few additional bits.  The fix for that is to iterate over the
bits, counting those which are set, and stop if you reach the value of
CPU_COUNT, rather than iterating over the bits you allocated.

Florian