Re: Standards References

[Rical Jasan <ricaljasan at pacific dot net>]

On 10/13/2016 11:32 PM, Carlos O'Donell wrote:
> On 10/14/2016 01:22 AM, Rical Jasan wrote:
>> Point being, that list is by no means complete, but should work for the
>> following question:
>>
>>  * Where can I find the references for these standards?
> 
> That is a fairly broad and difficult to answer question.

I couldn't write it any other way without fork-bombing into all the
things I did and didn't already know.  Thank you for answering
regardless; many of those are new to me, or help clarify what's
important, giving a context I was unaware of.

> Many of the older standards are very difficult to get today.

I've noticed.  Working on sockets, the opengroup stuff has been great,
but when trying to trace many things back to their origins, I've been
stonewalled by "Carried over from Issue 4", but that not being available.

> Michael Kerrisk is probably the best keeper of the flame, and if you
> find any more information I would submit patches to the linux man
> pages project to update this page:
> 
> http://man7.org/linux/man-pages/man7/standards.7.html

That is awesome!  I was not aware of that page.

To start fleshing out 1b below, would you want to use those names, or
names from feature_test_macros(7), or both?  I think I would lean
towards correlating the two, so rendered names might be more friendly
like standards(7), but were connected to the names in
feature_test_macros(7), allowing us to eventually say things like, "foo
is standardized by Nicely Formatted Standard Name, and made available
whenever _NFSN is defined.", or however we agree on the rendering.

> The Open Group has the largest collection that I am aware of and several
> of them are free to access (you may need an account):
> http://www.opengroup.org/standards/unix
> http://pubs.opengroup.org/onlinepubs/007908799/
> http://www.unix.org/version2/

So far I've been going to pubs.opengroup.org primarily (no account), so
I'm glad I was on the right track.

> The ISO standards are accessible also directly from WG14:
> http://www.open-std.org/jtc1/sc22/wg14/www/standards
> 
> The three most important standards for glibc are:
> * POSIX Issue 7:
> http://pubs.opengroup.org/onlinepubs/9699919799/
> * ISO C11 (N1570)
> * IEEE 754-2008 (only for sale from the IEEE)
>   https://standards.ieee.org/findstds/standard/754-2008.html

Thank you; good to know.

>> Based on elements of the description of the framework in [1], it seems
>> that in an ideal manual, we'd have definitive annotations for all the
>> standards a given item fell under; e.g., different feature nest macros
>> may independently yield a particular function's declaration.
> 
> Yes.
> 
>> It would also seem that standards' versions would be relevant here, as I
>> believe some header tests check for them (and who knows, maybe some
>> should, but don't).  I would think we would want the earliest version of
>> the standard in which something was present, but I can see doing that
>> exhaustively getting messy, and mind-numbing, quick.
> 
> Right.
> 
>> This is obviously work for a historian, and unfortunately I haven't
>> lived this particular history, so it's an actual study.  Recognizing
>> that, I think it'd be more practical to ignore the encyclopaedic utopia
>> of 20 years from now [3] and focus on the necessary legwork that needs
>> to be done.  So my next question:
> 
> Agreed.
> 
>>  * How do the following first steps sound?
>>
>>  1) Refactor all the source-level standards @comments into
>>   a) a consistent ordering wrt. line placement (@comment before @def)
>>   b) a consistent nomenclature (no more XOPEN and X/Open)
> 
> That makes sense.
> 
>> which should put us in a good place to:
>>
>>  2) Implement a basic macro framework by
>>   a) converting @comments to macros and
>>   b) removing the free-form comments already in descriptions
> 
> Agreed.
> 
>> Getting through Step 2 seems achievable (for me) in the reasonably short
>> term, and gets the manual to a point where we have a consistent
>> framework for annotating standards in the manual.  At that point,
>> further work could be done to:
>>
>>  3) Fill in missing standards and
>>  4) Vet standards references for
>>   a) correctness,
>>   b) completeness, and even
>>   c) versions
> 
> And enforce that all functions need a standard reference.

Good point.  For the record, I tend to use "function" as a catch-all for
functions, data types (e.g., structs), macros, and anything else
documented by the manual, so I consider this work to apply to all those
things.

>> I imagine those steps would apply to both the manual and the headers.
> 
> Yes.
> 
>> The proposal in [1] envisions perhaps a greater level of interoperation
>> than I could provide in the short term, but I would like to see the
>> standards references cleaned up, period.  As described in [4] (looking
>> now at the notes mentioned there, they are from March 2015), I think a
>> simple macro framework would be nice, and would provide much-needed
>> consistency, at least within the manual itself.  I also don't think it
>> conflicts with the extended vision in [1] (e.g., automated checking of
>> all macros and headers with their documented standards, etc.) because
>> that could come after Step 2 or supplant it entirely, and Steps 3 & 4
>> are really just the long-term historian's work that will plug in to
>> whatever framework exists (including the current one, if Steps 1 and/or
>> 2 never happen).  I've tried to pick steps that do something useful
>> regardless of how many or which are taken, always improving something,
>> keeping our options open, and easing any future work.
> 
> Agreed, that sounds great.
> 
> It reminds me a bit of the MT/AS/AC annotations work that we did, but
> extending the manual again into the realm of annotated standards
> compliance.

That is definitely an inspiration.

Even if the licensing issue you mention below is a problem for
src-to-doc automation, being able to say @gnuext{} (for example), like
@theglibc{}, instead of writing a free-form sentence or paragraph to the
same effect, will go a long way in making the documentation both more
consistent and easier to write and maintain.

> One thing to note is that there is a difference between 'the name of
> the function comes from standard X' and 'the function implements what
> standard X says it does', and these two don't always match. Like the
> POSIX scheduling functions do not do what POSIX says they do, instead
> they match more closely what Linux does. So it's not entirely clear
> if those functions are buggy, should get fixed to comply, or should
> not be defined if you're goal is to use only functions from a given
> standard.

Interesting.  Is there precedence within the feature test macros, where
the Linux standard supersedes POSIX, or vice versa, or is it possible to
dictate that behaviour?  This seems orthogonal, like a question that
should be answered in the implementation/headers, ultimately.  If that
conflict already exists in practice, do you consider it buggy, broken,
or worth omitting?  I imagine we could devise a syntax for such cases,
though.

Referring back to _NFSN/Nice Name above, but also keeping the stepwise
progress in mind, I envision the initial macros to be fairly simple,
like @stdfn{...} rendering as, "This function is defined by ...", which
should make extending the framework into a more complicated scheme
easier than it would be as things stand now, which meets the goals of
always improving the current mess and making it easier to do better later.

If precedence was definite, something like @stdfn{POSIX, Linux} could
render as, "This function is defined by POSIX, but implemented
differently by Linux.", and maybe @stdfn{POSIX, Linux} and @stdfn{Linux,
POSIX} would render differently.  That might not work as nicely with
multiple, but equal, standards, though.  Like if we ever got to the
point of, "This function was originally defined as a GNU extension, but
standardized in POSIX.2." (which happens).

>> Well, that's enough for now.  Thank you for your time.
> 
> A final note about copyright and licensing. The manual is GFDL and the
> source code is GPLv2, so you cannot copy any legally significant text
> from one to the other without relicensing permission by the FSF.

When I do cull my knowledge from the sources, I don't copy/paste
anything.  If I can't restate it in a meaningful way from my own
understanding, I don't think I should writing the documentation for it
anyway.  But thank you for making that explicit, as I wouldn't have been
aware otherwise.

Should I avoid reading man pages for functions that aren't documented in
the glibc manual but that I intend on documenting because they are
present in the source, but absent from the manual?

What about other forms of documentation, like tutorials or forums?

> I believe that Joseph Myers has such an authority in GCC to do just that,
> but we don't in glibc. It would be required if we had some closer source
> to manual integration. We would need someone who was blessed to legally
> run a script to update the manual from the source or vice-versa.

Crazy.  I was reading the GnuTLS manual the other day, and was having a
helluva time finding where they documented their functions (they
conveniently include prototypes wherever references are needed)---lo and
behold, all they did was include some kind of magical reference to
"include the prototype for this function here", and everywhere that
happened, the source of the rendered text came from the source-level
comments of the function itself.  I couldn't believe it.  They
documented the function /in the source code/, and the generated
documentation was based purely on that, no matter where or how many
times it occurred within their manual.

I really don't care that glibc does it differently and I'm more than
happy to help improve things here, but I figured the infrastructures
were just so different it was too much work to do it like that.  If I'm
understanding you correctly, we actually can't legally synchronize our
source code and documentation (without approval and a designated agent)?

FWIW, converting the entirety of glibc to be self-documenting from its
own source code is not something that sounds fun.

So not fun, I probably couldn't resist doing it anyway...  It's hard to
turn down a challenge.

> Technically I think we violate this with the errno list generation scripts
> we have that integrate with the manual, but perhaps we can call that
> "not legally significant."

Oh good, we have loopholes.  :)

Rical