Re: primitive documentation conventions

[Greg Badros <gjb@cs.washington.edu>]

Jim Blandy <jimb@red-bean.com> writes:

> > I would like to suggest first off that we should design and implement
> > the scwm documentation extraction system with the intention of it
> > being eventually incorporated in Guile, perhaps with some changes.
> 
> Exactly.  This is a problem that all Guile users will face, so if we
> can solve it thoroughly, that's a big win.
> 
> I agree that we should implement the extractor in Guile.  It may be
> slow at first, but speeding up Guile's I/O is on the list, and I'd
> like to have some actual text-processing applications to test my
> changes.

I agree, but to the extent that I work on a prototype system, the
initial versions will probably be in perl.  After things are stable, I
can rewrite in guile.  (Partially a chick & egg problem -- guile's
documentation is still sparse enough that I find it difficult to learn
about aspects which I haven't used before).

> > * Output: We should, for now at least, output each of DocBook SGML,
> > TexInfo, and an on-disk documentation string database.
> 
> Sounds reasonable.

Ditto.

> > * How to get at docstrings for functions written in C: I suggest that
> > we try for a system where they are not kept in core straight off.
> 
> Emacs does this.  Docstrings are included as an argument to a macro
> which drops them when generating normal Emacs code.  Then there's
> another function which scans the source code for these macros and
> extracts the docstrings into a file, labeled with the function name.
> 
> I seem to remember long-standing bugs or FAQ's about this process; the
> installation procedure wasn't quite right, and the docstring files
> were always out of date with respect to the executable.  I haven't
> thought about this at all, but if all Guile users are going to follow
> this procedure, we should look for something foolproof.
> 
> > ** For C files, there are two alternatives that are, IMO, more or less
> > equally acceptable. One is to use specially distinguished comments,
> > the other is to add an extra string parameter to SCM_PROC which
> > guile-snarf would ignore, but which the doc extractor would use for
> > it's own purposes. Jim Blandy seemed suspcious of the concept of
> > looking through the comments; however, I'm not sure it is that
> > bad. It's certainly more expandable to documenting concepts and
> > specially distinguished variables, and not just procedures.
> 
> I'm not so suspicious of it; I've done similar things before.
> However, you can't make comments conditional on configure-generated
> #defines, so your extracted docstrings may not always correspond to
> the source that got compiled.  Something that operates post-CPP
> wouldn't have this problem.

I hadn't thought of the CPP issue before (ironic, since some of my
research in the past couple of years has related to it).  In any case,
we could run the preprocessor w/o the SCWM_PROC macro defined and w/o
discarding comments and use that as the source if we really cared.

I like comments as comments instead of strings, and it'd be a bit
annoying to have to deal with the \n\ at the end of each comment line
(though I'm sure an emacs command to reformat doc strings would be easy
enough).

> Note that, contrary to Knuth's claims, you can't generate decent
> internals documentation by extracting stuff from the source.  Source
> code and manuals are laid out differently.  This system is only going
> to be providing point documentation, not a full manual.

One of the things I am going for, though, is the ability to embed
concept documentation in the source, too. E.g., the key-specifier form
in SCWM is determined by a single C support procedure of the key-binding 
and unbinding primitives.  I'd like comments near that support procedure 
to be woven into a section on the appropriate form for a key-specifier,
and have the related primitives reference that section.

> That is, the FSF does the right thing in providing *both* docstrings
> and a full Emacs Lisp manual, and having them be separate.  A manual
> entry knows a lot more about its context than a docstring does, so the
> best docstrings rely little on context, while the best manuals have
> overview sections, explanations of common concepts, and then function
> descriptions which operate in that context.

Some of the "concept" sections extracted from the source would be
appropriate for a manual, IMHO.  Tight coupling of documentation and the 
code that it documents is essential to keep them in sync.

> > * Markup: We may want extra markup in the docs in addition to the
> > presumably automatically generated function signature. If this is ever
> > supported, I suggest we use DocBook tags, as opposed to TexInfo or
> > worse yet, something ad-hoc. It seems to me that DocBook->TexInfo is
> > likely to be the easier translation, and if there's ever a DocBook
> > system tht generates texi automatically, we won't have to worry about
> > it ourselves at all.
> 
> DocBook is fine, but we'll need to specify which tags we allow in
> docstrings, right?  I'm familiar with the general idea of SGML, DTD's,
> style sheets, and DocBook, but I don't know the details.

I'd like the documentation to be as easy and unobtrusive to write as
possible, and would prefer to use conventions (yes, ad hoc conventions)
for much of the markup.  e.g., my current system scans the argument list 
of the procedure and determines the formal names.  Then, any time a
formal name appears in the comment in all uppercase, it can be marked as
a formal.  I think:

SCWM_PROC(unbind_key, "unbind-key", 2, 0, 0,
          (SCM contexts, SCM key))
     /** Remove any bindings attached to KEY in given CONTEXTS.
CONTEXTS is a list of event-contexts (e.g., '(button1 sidebar))
KEY is a string giving the key-specifier (e.g., M-Delete for META+Delete) */

Is a lot easier to write and read (in the source code) than:

SCWM_PROC(unbind_key, "unbind-key", 2, 0, 0,
          (SCM contexts, SCM key))
     /** Remove any bindings attached to <formal-argument
pos=2>KEY</formal-argument> in given <formal-argument
pos=1>CONTEXTS</formal-argument>.  <formal-argument
pos=1>CONTEXTS</formal-argument> is a list of event-contexts (e.g.,
'(button1 sidebar)) <formal-argument pos=2>KEY</formal-argument> is a
string giving the key-specifier (e.g., M-Delete for META+Delete) */

I'm exaggerating a bit, but I'd like the extraction system to have a
certain degree of "smarts" from some simple conventions for the standard 
cases.

Beyond that, a well-defined subset of DocBook tags could be used for the 
less common cases.  In particular, we'll need cross-referencing.

We need to get the big decisions right about what the markup should look 
like in the source code, but once we do, we need to start documenting
and we can deal with the extraction tools iteratively.  The current
sparse docs are way too limiting.

Greg