(PR11207) Macroprocessor discussion

[Serguei Makarov <smakarov at redhat dot com>]

This is a thread to keep track of further back and forth on PR11207 (http://sourceware.org/bugzilla/show_bug.cgi?id=11207), an attempt to develop (or steal) some kind of solution for preprocessor macros in systemtap code. (Trading large chunks of text back and forth on a mailing list is more convenient than trading text back and forth on a Bugzilla page.)

To begin with, here is the current ultra-custom version of the...

# SystemTap Macro Processor Syntax Proposal

The macroprocessor has both an embedded mode (which feeds text directly to the systemtap parser, retaining the original line and column coordinates for error reporting purposes) and a standalone mode (which produces text output that can be used e.g. for further processing of docstrings into documentation).

# Example Usage - Defining Multiple Probes

    /** ... doc comments common to all ip probes ... */
    %define(ipprobes, probe_name, hook4_name, hook6_name,(
      %grab_invocation_docs(probe_name, ipprotocol_name)
      %grab_definition_docs
    
      %defloc(ipprobe_body_common,(
        ... stuff common to both ipv4 and ipv6 ...
      ))
    
      /** %invocation_docs(ip,IP)
       * %definition_docs
       * ... doc comments specific to ip ... */
      probe netfilter.ip.%probe_name = netfilter.ipv4.%probe_name,
              netfilter.ipv6.%probe_name { }
    
      /** %invocation_docs(ip4,IPv4)
       * %definition_docs
       * ... doc comments specific to ipv4 ... */
      probe netfilter.ipv4.%probe_name
              = netfilter.pf("NFPROTO_IPV4").hook(%hook4_name) {
        %ipprobe_body_common
        ... stuff specific to ipv4 ...
      }

      /** %invocation_docs(ip6,IPv6)
       * %definition_docs
       * ... doc comments specific to ipv6 ... */
      probe netfilter.ipv6.%probe_name
              = netfilter.pf("NFPROTO_IPV6").hook(%hook4_name) {
        %ipprobe_body_common
        ... stuff specific to ipv6 ...
      }
    ))

    /** probe netfilter.%probe_name.pre_routing - Called before an %ipprotocol_name packet is routed */
    %ipprobes(pre_routing,"NF_INET_PRE_ROUTING","NF_IP6_PRE_ROUTING")

# Example Usage - Defining a Shorthand for a Cast Operation

    %define(FOO,ptr, @cast(ptr, "struct foo", "/path/to/app:<sys/foo.h") )
    bar = %FOO(p)->bar
    baz = %FOO(p)->baz

# Example Usage - Equivalent to a Standard CPP #define Macro

    %define(AREA,base,height, ((base)*(height)/2.0) )
    a = AREA(2+2,5) // correctly handles precedence

# Detailed Explanation

Preprocessor Activation Sequences
    %ident  -- parameterless macro invocation
    %ident( -- macro invocation with parameter list
    %(      -- conditional expression of the form %( ... %? ... %: ... %)
    /**     -- docstring enclosure -- we can use macro invocations inside

Items inside a macro invocation's parameter list are separated by commas. To allow complex parameters which stretch over multiple lines and in turn contain commas, the macro processor respects nested parens "()" and does not end a parameter if it encounters commas inside the parens. If an opening paren "(" is the very first character of the parameter, the outermost parens are discarded:
    %ident(p1,p2) -- calls macro ident with two parameters "p1" and "p2"
    %ident((p1,p2)) -- calls macro ident with one parameter "p1,p2" (discarding outermost parens)
    %ident(p1,(p2,p3)) -- calls macro ident with two parameters "p1" and "p2
    %ident(((param))) -- calls macro ident with one parameter "(p1,p2)" (discarding outermost parens)
    %ident(foo(p1,p2)) -- calls macro ident with one parameter "foo(p1,p2)"
    %ident( (p1,p2)) -- calls macro ident with one parameter " (p1,p2)" (keeping outermost parens)

This business with the parens allows us to write complex multiline definitions:
    %define(multiline_macro,(
      ... multiple lines of stuff go here ...
    ))

Behaviour currently undecided for the following examples:
    %ident((p1,p2)   ) -- PROBABLY DISCARD PARENS
    %ident((p1,p2)foobar) -- BEHAVIOUR UNDECIDED

The macro processor is aware of docstrings. When invoking a macro, it keeps track of the most recent docstring it saw before the invocation. If the docstring is accessed via %grab_invocation_docs or %grab_definition_docs, it is pulled out of the original source text (as though it was never there) and made available to be re-inserted elsewhere using %invocation_docs and %definition_docs. There are a few tricky issues to handle with the fact that text inside docstrings traditionally prefixes each new line with " * ", but it should be feasible to magically do the right thing to allow use of regular macros inside docstrings, e.g:

    %define(code_snippet,(
    printf("This is sample code!\n")
    printf("It extends over multiple lines!\n")
    ))

    /** This is a docstring, handled specially by the macroprocessor.
     *
     * %code_snippet
     *
     * The snippet above magically has each line start with ' * '. */
    probe foo {
          // But we can also employ the multiline macro in regular code!
          %code_snippet
    }

    /* The IMPORTANT thing though (the actual reason for making this work)
       is that the DEFINITION of code_snippet DOES NOT have to have each
       line except the first start with " * ". */

Obviously, some nasty magic is required behind the scenes to make this work transparently. My opinion is that it can be made to work with some thought. (XXX: As a bonus, similar magic could also be performed to ensure that the indentation of macro output is at least as much as the indentation of surrounding code.)

XXX: some macro %defraw or similar can be used to bypass all the magic?

Docstring considerations are only relevant for the standalone (text-generating) version of the macroprocessor. The embedded version can obviously discard all comments and special comment-handling macros.

Predefined Macros (possibility)
    %define(name, param_1, param_2, ..., param_n, macro_body)
    %defloc(name, param_1, param_2, ..., param_n, macro_body)
    %grab_invocation_docs(param_1, param_2, ..., param_n)
    -- defines a local %invocation_docs(...) macro
    %grab_definition_docs(param_1, param_2, ..., param_n)
    -- defines a local %definition_docs(...) macro
    %undef(name)

XXX: alternate naming scheme for the macros?

%define and %undef work in the obvious manner. param_1, param_2, ... give names of parameters, which are available as zero parameter macros inside the definition body:

    %define(foo,x, ((%x) + 2) )
    // Note the style of putting spaces around the body of a one-line macro.
    // This ensures the parentheses are kept, while still looking reasonable.

%defloc is a bit tricky. When it is encountered inside a macro body, the resulting definition is only valid for the duration of the macro body expansion; afterwards, the old definition is restored. It is effectively a more compact version of the m4 idiom where we 'pushdef' a macro at the beginning of a macro body and 'popdef' it at the end. (XXX: We may want to define pushdef and popdef as well at a later point.)

%grab_invocation_docs inside a macro body grabs the docstring closest to the point where the macro is invoked and makes it available to instead be inserted whenever we use the macro %invocation_docs. (If %invocation_docs is used *inside* a docstring, it seamlessly pastes the content of one docstring into the other.)

%grab_definition_docs inside a macro body works similarly with the docstring attached to the invocation of %define that created the macro.

The reason both of the latter macros have parameters is because we might want to repeat the same docstring several times with custom substitutions (see above). To make this work we could (XXX: either delay evaluation of docstring contents until we know the context it is evaluated in -- probably this is the better solution, OR XXX: or simply say that undefined macro invocations are left untouched). Hence docstrings can be constructed as follows:

    %define(cakery,(
      %grab_invocation_docs(foodstuff)

      /** EXTRA BLAH BLAH %invocation_docs(CHEESE) */
      probe cheese { ... }

      %invocation_docs(BEER) // this also works
      probe beer { ... }
    ))

    /** BLAH BLAH BLAH DOCUMENTATION ABOUT %foodstuff */
    %cakery

This would produce output similar to the following:

    /** EXTRA BLAH BLAH BLAH BLAH BLAH DOCUMENTATION ABOUT CHEESE */
    probe cheese { ... }

    /** BLAH BLAH BLAH DOCUMENTATION ABOUT BEER */
    probe beer { ... }