Implementing a generic binary trace interface.

[Jose R. Santos]

Masami Hiramatsu wrote:

>Hi,
>
>Jose R. Santos wrote:
>> Hi folks,
>>
>> My team is currently implementing a trace tool using SystemTap that
>> currently does logging by means of printf mechanism.  We want to move to
>> a binary trace format but there is no such mechanism on SystemTap for
>> doing this.  I've looked at what the folks at Hitachi have done with
>> BTI, but this seems to force a specific trace format that is not
>> suitable for what we need.  Ideally, the trace format should be left to
>> the tapset using the interface not the BTI.  I propose to slightly alter
>> the BTI from Hitachi to allow other trace implementations to use the
>> trace format that's most convenient for the people implementing them.
>
>We had recognized the problems of previous BTI implementation. And we
>have developed more generic method called gBTI (generic Binary Transport
>Interface) which can be used with current ATI.
>gBTI itself uses no formatted structure but an array of long (64bit integer).
>It just transports those several binary data from tapset to user daemon. It
>works like IP packet envelope.
>
>To support this gBTI, I think, systemtap just introduce only three features below:
>1. _stp_binary_write() function in runtime.
>2. stpd enhancement to handle binary data correctly.
>3. some wrapping functions for user scripts. (i.e. lket_trace(), binary_log(), etc)
>So I think supporting gBTI is much easier and simpler than supporting other
>method.
>
>1. runtime function
>gBTI provides a runtime function called _stp_binary_write() for tapsets.
>_stp_binary_write() should be invoked with a number as the first argument and
>variable argument lists. In other words, the synopsis is:
>
> void _stp_binary_write(int num, ...);
>
>The "num" argument specifies the number of arguments following this argument.
>The type of other variables must be int64_t.
>For example, you can use any format of followings;
>
> _stp_binary_write(3, (int64_t)arg1, (int64_t)arg2, (int64_t)arg3);
> _stp_binary_write(2, (int64_t)current->pid, (int64_t)current);
> _stp_binary_write(5, (int64_t)lkst_header1(), (int64_t)lkst_header2(),
>                  (int64_t)etype, (int64_t)arg1, (int64_t)arg2);
>
> And so on.
>
>The _stp_binary_write() function writes the data as a binary packet into a
>relayfs channel which is shared by current ATI. The gBTI also share the
>sequential-ID with the ATI by using _stp_seq_inc(). The packet format is
>described below.
>
>gBTI packet:
>[seq-id][\0][num][arg1][arg2]....[arg(num-1)]
>
>In the other hands, ATI packet is;
>[seq-id][string][\0]
>(Length of string is greater than 0, this is checked by runtime functions.)
>
>Thus, gBTI can share the channel safely, because if the packet whose first
>character of the string is '\0', it is binary data packet.
>  
>

Nice...

This implementation is more generic and its available now.  While this 
is suitable for our current trace work, I still agree with Tom that this 
approach may not be as flexible for gathering other data types.  While 
having fixed data types make it easy to decode the binary trace, there 
may be situations were we may want to grab data that may not fit on 
either a size long arg or the 128 arg limit.  Also, while possible to 
write a function call with 128 args, this would look really ugly and 
seems error prone.  Having a basic struct data type on SystemTap would 
solve some of these issues.

>2. stpd enhancement
>We should enhance stpd daemon to handle both ascii packet and binary packet
>correctly. But it is not so difficalt.
>  
>
I claim complete ignorance on how the stpd daemon works.  Can you 
elaborate on what the problem and possible enhancements are?  Does this 
still require your prohibit merging patch?

>3. wrapping functions
>I also think we can define the various interfaces for example LKST, LKET,
>or more generic binary_log() interface over the gBTI runtime.
>  
>
This I like.  I would use this if it were available now on SystemTap.

>What would you think about the gBTI?
>  
>
Its a good solution and seems like we could port our trace to use this 
mechanism easily (which we would had to do anyway if we implemented our 
own BTI).  This could later be enhance with what Tom is proposing.  I 
would like to see this or an equivalent functionality as part of 
SystemTap though.  Since you move the functionality to a tapset, are you 
looking at maintaining this as part of LKST or do you have plans to get 
this into SystemTap itself?

>I have already a concrete implementation of gBTI(for relayfs). I developed
>it as a tapset script including runtime part.
>I attach two files to this mail.
>
>- gbti.stp:      gBTI core runtime (_stp_binary_write())
>- lkst_gbti.stp: LKST compatible wrapping function of gBTI
>
>Please review it.
>
>Best regards,
>  
>
I've done a visual inspection and don't see any mayor issues with it.  I 
will play with it a bit and see how it goes.

What is the purpose of having gbti_init()?

Thanks

-JRS