Re: (PR11207) Macroprocessor discussion

[Serguei Makarov <smakarov at redhat dot com>]

Frank,

Thanks for taking the time to offer feedback. There's really one major issue that's blocking for the design, which I discuss below. (As for the docstring-manipulation stuff not being appealing -- it's all right, that won't be in the initial prototype anyway, and we can live without it if I can't think of a nicer way to get the preprocessor to do what I wanted from that feature...)

> This is a problem.  One of the expected uses of this mechanism was to
> let tapsets define macros for use by other scripts, such as for performing
> kernel-flavoured offset_of(), container_of() type operations.

Hm, based on my reading of the code I felt that implementing this makes for a thorny problem with the way tapset inclusion is currently arranged. From what I know, identifiers across files are resolved as follows:
- all of the tapset files are parsed independently of each other (see main.cxx:L) and collected in s.library_files (this implies that macroexpansion is performed on each one)
- during semantic analysis, we resolve identifiers by finding the files containing referenced identifiers and adding them to s.files, then checking which tapsets *those* import in a transitive closure sorta thing
- everything that ends up in the transitive closure is compiled into the final module

This is a chicken-and-egg problem, since tapsets obviously are allowed to build on one another, and to properly parse and macroexpand a file we need to know what macros are in other files, so we need to macroexpand those files, which requires knowing what macros are pulled in from other files for *that* file...

One hack I can think of for getting around the issue is to force tapset writers to explicitly put directives such as

%include_macros("other_tapset.stp")

if they want to use macros from that tapset, in order for stap to have the information to parse files in the correct order. (The most obvious solution for this requires the parser to be able to suspend parsing one file, go off and parse another file, and then return to the first one where it left off. This seems reasonably, but some double-checking is required to be sure that nothing in parse.cxx assumes files are handled one at a time...)

[There was originally a digression here about how much time is spent parsing the tapset files sequentially, but it's not that much time as a proportion of the script run, so the efficiency concerns I was thinking about are basically irrelevant. (They may be applicable to Pass 2, but that's a discussion for a different day.) And pass timings are automatically printed for verbose > 0. :-)]

The final script being compiled is exempted from the need to use explicit macro inclusion directives, since by that point there is no chicken-and-egg ambiguity.

(As a minor detail, we also have to flip the order of passes 1a and 1b. 1b parses the tapsets, while 1a parses the final script.)

That's not a very satisfying solution. However, unless I think of something else, our design space for this corner of the problem seems to be very limited...

- Serhei

(PS: Argh, I keep forgetting to use reply to all...)