Re: [pcp] suitability of PCP for event tracing

[Ken McDonell <kenj at internode dot on dot net>]

On Thu, 2010-11-25 at 09:58 +1100, nathans@aconex.com wrote:
> ... 
> 
> > > | typedef struct {
> > > |     struct timeval	er_timestamp;
> > > |     int		er_nparams;	/* number of er_param[] entries */
> > > |     pmEventParameter	er_param[1];
> > > | } pmEventRecord;
> > > 
> > 
> > I considered er_flags but this seems to be dependent on prior
> > agreement of the producer and consumer, 
> 
> ...?  To my mind, its the opposite.  The idea is that these flags
> encapsulate the generic characteristics of traces, which allows a
> generic client to make more sense of them.
> 
> For example, pmchart could do a decent job of displaying traces with
> these flags, but it has no chance without them (cannot tell what is
> a start event, whats an end event, nor how to build a hierarchy ...
> for an arbitrary trace).  Thinking specifically of the gantt chart
> (e.g. http://www.bootchart.org/images/bootchart.png) class of trace
> visualisation here, if that helps clarify wtf I'm on about.  That is
> feasible (and useful!) for arbitrary traces via er_flags, I think.
> 
> >  Also, there is no obvious place to store the
> > value when the packed array of event records is expanded (it cannot
> > go
> > in the pmResult, so needs another allocation and return parameter for
> > pmUnpackEventRecords (as you indicated later).
> 
> So, this seems to be more of an issue, yes it complicates the API a
> bit ... but its the difference between something bland and something
> alot more useful IMO.
> 
> > I'd prefer to see event "flags" or event "type" encoded in a specific
> > parameter of the event record.  This also needs prior agreement by
> > the
> > consumer and the producer, enables all of the same functionality that
> > er_flags would, and requires no additional API or data structure
> > changes.
> 
> The problem there, as you say, is that it means client tools must all
> be custom written (prior agreement)... theres really not much useful/
> generic that pmchart could do with this.

I've been mulling over this issue (and the related matter of "missed"
event records, where I don't think the implementation is sufficiently
flexible).

I think _anything_ related to event records has to map onto a
pmResult ... this is the only way avoid auxiliary data structures for
pmUnpackEventRecords(), but _more_ importantly when pmlogextract knows
how to unroll an archive and get all of the original pmResults and the
event records pmResults temporally sorted in a single output archive the
_only_ place to hide information will be in a pmResult.

Try this one for size ...

Add a new sort of "anonymous" PCP metric, such that the PMID is
hard-coded for a small number of anon metrics as is the pmDesc (I'm
thinking one per simple data type, semantics discrete, no units, no
instance domain) and the "name", e.g. anon.u32 for a PM_TYPE_U32 metric
with pmid 0:0:1.

Domain #0 is unused, so we have a convenient place in the range of
PMIDs.

I think the only places where pmids and names are used is these routines
- pmLookupName, pmNameID, pmNameAll, pmLookupDesc, pmFetch and
pmLookupText - and I can see how to support anon metrics in all of these
places locally in libpcp with no reference to pmcd or an underlying
archive.  So we can stuff anon metrics into a pmResult without needing
any particular PMDA to provide the metadata for the anon metrics.

Now, pmUnpackEventRecords could _add_ anon.u32 to hold the er_flags
value of er_flags is not zero.

So pmchart calls pmUnpackEventRecords and checks each pmResult, if
anon.u32 is the first metric, then the value is er_flags, and Nathan is
off to the races.

And I could use something similar to encode ea_nmissed insitu so the
correct place in the event array could be marked ... I can see
implementation use cases where there are "missed" event records at the
start of the array (tail buffering), end of the array (quota buffering),
or middle of the array (resource pressure or other throttling of the
rate of event capture). 

I'd appreciate thoughts on this.