Principles for API sources

[Joseph Myers <joseph at codesourcery dot com>]

Various recent proposals for new APIs taken from existing sources of APIs 
seem to have got stuck for similar reasons.  This includes Linux kernel 
syscall wrappers as well as explicit_bzero and strlcpy.  I'd like to 
propose some general principles for such APIs in the hopes of unsticking 
those discussions - if we can get consensus on principles (consensus does 
not mean unanimity; consensus does not mean waiting indefinitely for 
people who might want to contribute to the discussion at some 
indeterminate point in the future) then we don't need to redebate the 
principles for each proposed API, just consider how they apply to that 
API.

All of the following is only about APIs, for the OS-independent GNU API (I 
think the idea of libinux-syscalls.so.1 for OS-specific Linux syscall 
wrappers was uncontroversial).  It does not address what library an API 
goes in, or other ABI issues, although in most cases the choice of library 
will be obvious.  And all the usual patch requirements apply to any API 
addition.

* Required ISO C and POSIX APIs (in current versions, and in previous 
versions with very rare exceptions where there is particular difficulty in 
supporting an old API) are all assumed appropriate for inclusion in glibc 
(I think this is uncontroversial).

* Optional ISO C and POSIX APIs, and APIs from ISO C TRs / TSs / standards 
outside the main C standard, require a bit more consideration, but that 
should generally be at the level of considering a complete set of APIs 
together, and with a starting point in favour of inclusion if the nature 
of the APIs is very closely tied to existing glibc APIs or aspects of them 
make them hard to implement correctly independently of libc.  (So we might 
discuss the desirability of C11 Annex K support in glibc, for example, and 
whether such APIs should generally go in a separate libc_s shared library 
built by glibc, but should not get into piecemeal debate about individual 
APIs there.)

* Historical practice was that glibc adopted both BSD and SVID APIs, 
including multiple APIs for the same thing, and until recently adopted 
APIs for most Linux syscalls (although some of those went in libc.so for 
GNU/Linux only).  My claim is that we have now swung much too far the 
other way in conservatism about adopting existing APIs, and that this is 
not beneficial to glibc or glibc users.

* Specifically, I think the following should be considered appropriate 
sources for APIs to be added to the OS-independent GNU API in glibc, where 
the APIs in question are the general sort of functionality traditionally 
included in libc and associated libraries on Unix-like systems: Linux 
kernel syscalls, where the functionality would plausibly be meaningful 
under other kernels; modern BSD systems; other Unix-like operating 
systems.  This is in order of preference: APIs from Linux syscalls have a 
stronger expectation of appropriateness than those from modern BSD 
systems, which have a stronger expectation of appropriateness than those 
from other Unix-like operating systems.

* The fact that an API already exists and is being used in free software 
(in a context where it isn't deprecated) should be considered a point in 
favour of inclusion of that API, that can counterbalance arguments about 
imperfections in the API design; the more widespread the usage is, the 
more important it is as an argument for inclusion.  Deficiencies in API 
design, or issues of consistency with other APIs in glibc, can still be 
relevant issues.  But whether they outweigh usage depends on how serious 
the deficiencies are, and how widespread the usage is.

* Where there are choices of multiple APIs in an area, usage information 
(such as from Zack's analysis for explicit_bzero) is valuable for choosing 
between them.  (Where multiple APIs are in use there may sometimes be a 
case for supporting more than one of them, but that needs a stronger case 
than for adding one API in an area from an existing API source.)

* The following are all points in favour of inclusion of an API because 
they would make a good external implementation more difficult: (a) natural 
integration with fortification support; (b) natural integration with 
thread cancellation; (c) implementation involves low-level information 
about syscall calling conventions; (d) any other way in which the function 
is naturally integrated with libc infrastructure rather than being 
definable completely independently.

* That an API is not fully general - that there is some problem in the 
area of the API that it does not fully solve - is only a weak point 
against that API.  This is especially the case if the API (notwithstanding 
its limitations) is already widely used.  This is also especially the case 
for APIs related to security hardening (making existing code more robust 
against exploitation in the presence of bugs, through local changes), 
where in general absolute protection does not exist.

* We should respect that other communities do have good processes for API 
review, in particular as regards Linux kernel syscalls added in the past 
few years, so that most of the design review for a new API has already 
been done if that API comes to us from an external source.

* On the above basis I would accept most non-deprecated Linux syscalls, 
except where particularly OS-specific, while much more selectively 
accepting widely-used BSD APIs, especially in the security hardening area 
(including explicit_bzero, strlcpy and strlcat, for which I think the 
widespread use, and greater suitability for hardening existing code than 
redesigning its memory allocation strategies, outweighs the imperfections 
of the APIs).

-- 
Joseph S. Myers
joseph@codesourcery.com