openat2 (BZ 31664) vs FreeBSD-style O_RESOLVE_BENEATH

[Paul Eggert]

When I changed GNU tar to use openat2[1], I found that the only new 
feature that I used was its RESOLVE_BENEATH flag.

FreeBSD did this in a different way, by adding O_RESOLVE_BENEATH as a 
flag to plain openat, and by adding AT_RESOLVE_BENEATH as a flag for 
fstatat, faccessat, funlinkat, fchmodat, utimensat, linkat, and some 
other FreeBSD-specific syscalls.

In some ways the FreeBSD approach is better: it doesn't require a new 
syscall (it merely extends old ones), it improves on syscalls other than 
just openat, and it would have been a bit easier to modify GNU tar to 
use the FreeBSD approach than to use openat2. However, the FreeBSD 
approach does have some limitations, in that some syscalls (e.g., 
mkdirat, mknodat) that lack a flags argument don't have a 
AT_RESOLVE_BENEATH option. In the end, I modified GNU Tar, via Gnulib, 
to use FreeBSD's O_RESOLVE_BENEATH when available, to use openat2's 
RESOLVE_BENEATH when available, and to emulate openat2's RESOLVE_BENEATH 
on other platforms, and to use these new options only to open the parent 
directories of the files that I actually wanted to access (so that I 
didn't need to worry about fstatat etc. messing up).

One possibility to simplify portability for other apps would be to add 
O_RESOLVE_BENEATH support to glibc, for compatibility with FreeBSD. This 
should be easy to do by using openat2 internally. Supporting 
AT_RESOLVE_BENEATH would be harder, as it would require help from the 
Linux kernel; and anyway as mentioned above AT_RESOLVE_BENEATH doesn't 
solve the whole problem.

Even if glibc doesn't add O_RESOLVE_BENEATH, the glibc manual should 
mention the connection between openat2's RESOLVE_BENEATH and FreeBSD 
O_RESOLVE_BENEATH, in a portability section under openat and/or openat2. 
The Linux man page for openat2 does this, so you can look to it for 
wording ideas.

I don't know why the FreeBSD approach was rejected by the Linux kernel 
developers.

[1]: 
https://cgit.git.savannah.gnu.org/cgit/tar.git/commit/?id=75b03fdff48916bd0654677ed21379bdb0db016d