py uiout was: Re: [PATCH 3/3] PR gdb/13860: don't lose '-interpreter-exec console EXECUTION_COMMAND''s output in async mode.

[Matt Rice]

On Fri, May 18, 2012 at 10:05 AM, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Matt" == Matt Rice <ratmice@gmail.com> writes:
>
> Matt> FWIW the attachments to PR 11688 should probably be able to see if
> Matt> this is an issue (I would also imagine it seems like it would)
>
> Matt> the main issue is that the current ui-out types do not contain
> Matt> adequate information to allow one to pre-allocate a python object of
> Matt> e.g. a dictionary or a list, instead we must wait after starting the
> Matt> list or dictionary until the first item is added to know if it is
> Matt> key/value or merely values,
>
> Could you walk me through it?
> I am still having trouble understanding the issue.

Sure, i'll give it a shot

> It seems to me that ui-out tables would map to a list of objects; and
> lists or tuples would generally map to lists, except if they are a
> direct child of a table.

Yeah, my first attempt was to use lists->lists, lists of tuples->list
of 2-tuples which led to the following stuff, if you were thinking
something better along these lines, i'm open to ideas.

this lead to the following type of output:
(gdb) py print gdb.mi_execute("break-list")
[('BreakpointTable', [('nr_rows', '1'), ('nr_cols', '6'), ('hdr',
[[('width', '7'), ('alignment', '-1'), ('col_name', 'number'),
('colhdr', 'Num')], [('width', '14'), ('alignment', '-1'),
('col_name', 'type'), ('colhdr', 'Type')], [('width', '4'),
('alignment', '-1'), ('col_name', 'disp'), ('colhdr', 'Disp')],
[('width', '3'), ('alignment', '-1'), ('col_name', 'enabled'),
('colhdr', 'Enb')], [('width', '18'), ('alignment', '-1'),
('col_name', 'addr'), ('colhdr', 'Address')], [('width', '40'),
('alignment', '2'), ('col_name', 'what'), ('colhdr', 'What')]]),
('body', [('bkpt', [('number', '1'), ('type', 'breakpoint'), ('disp',
'keep'), ('enabled', 'y'), ('addr', '0x000000000048d440'), ('func',
'main'), ('file', '../../gdb.updated/gdb/gdb/gdb.c'), ('fullname',
'/home/ratmice/git/gdb.updated/gdb/gdb/gdb.c'), ('line', '26'),
('times', '0'), ('original-location', 'main')])])])]

(gdb) py print gdb.mi_execute("break-list")[0][1][3][1][0][1][0]
('number', '1')

you could safely assume I wasn't all too happy with the output, or the
resulting code to work with the output, so on to the 2nd attempt.
(p.s. please ignore the numbers are strings, that is easily fixed)

In the 2nd attempt I tried a mapping of
list->list,
list of tuples->dictionary

I don't have a link handy for the MI docs (currently a 404), but it contains:
`RESULT ==>'
     ` VARIABLE "=" VALUE'
`VALUE ==>'
     ` CONST | TUPLE | LIST '
`LIST ==>'
` "[]" | "[" VALUE ( "," VALUE )* "]" | "[" RESULT ( "," RESULT )* "]" '

Notes:
* New GDB/MI commands should only output LISTS containing VALUES.

so you'll never see a mixed list, e.g. containing some VALUES and some
RESULTS, so in theory a result list->dictionary mapping should be
possible.

but it is impossible to know if we are working with a list of values,
or a list of tuples until the 1st item is added to the list (both are
started as ui_out_type_list).  Meaning we don't know if the resulting
object is a dictionary or a list until adding something, thus instead
of an empty list or dictionary, a list/dict with no items returns
None,
so, there is a small burden on the calling code, must check for None
before attempting to iterate or whatever.

below is the equivalent output of the first example using a
dictionary/list mapping as described above.

(gdb) break main
Breakpoint 1 at 0x48d490: file ../../gdb.updated/gdb/gdb/gdb.c, line 26.
(gdb) py print gdb.mi_execute("break-list")
{'body': {'bkpt': {'disp': 'keep', 'file':
'../../gdb.updated/gdb/gdb/gdb.c', 'addr': '0x000000000048d490',
'enabled': 'y', 'number': 1, 'times': 0, 'func': 'main', 'fullname':
'/home/ratmice/git/gdb.updated/gdb/gdb/gdb.c', 'type': 'breakpoint',
'line': 26, 'original-location': 'main'}}, 'hdr': [{'width': 7,
'colhdr': 'Num', 'col_name': 'number', 'alignment': -1}, {'width': 14,
'colhdr': 'Type', 'col_name': 'type', 'alignment': -1}, {'width': 4,
'colhdr': 'Disp', 'col_name': 'disp', 'alignment': -1}, {'width': 3,
'colhdr': 'Enb', 'col_name': 'enabled', 'alignment': -1}, {'width':
18, 'colhdr': 'Address', 'col_name': 'addr', 'alignment': -1},
{'width': 40, 'colhdr': 'What', 'col_name': 'what', 'alignment': 2}],
'nr_cols': 6, 'nr_rows': 1}
(gdb) py print gdb.mi_execute("break-list")['body']['bkpt']['number']
1

It's much nicer IMO, but still a bit wonky due to the use of
introspection e.g. with no initial breakpoints:

(gdb) py print gdb.mi_execute("break-list")['body']
None
(gdb) py print gdb.mi_execute("break-list")['body']['bkpt']['number']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'
Error while executing Python code.
(gdb) break main
Breakpoint 1 at 0x48d490: file ../../gdb.updated/gdb/gdb/gdb.c, line 26.
(gdb) py print gdb.mi_execute("break-list")['body']['bkpt']['number']
1

to get rid of this limitation we could add
ui_out_type_value_list/ui_out_type_result_list.  With that we could
create a list/dictionary before the first item is added, both list
types would handled with the existing code that handles
ui_out_type_list for gdb/mi uiout but for the py uiout would create a
list or a dictionary.  (haven't actually done this)

the 2nd issue which we run across is when a RESULT contains VARIABLES
of duplicate names.  if we add a 2nd breakpoint, the result list will
have multiple RESULTS with VARIABLES named 'bkpt'.
using introspection again we can look for an existing key, and promote
it to a list (if it isn't already one):

(gdb) break main
Note: breakpoint 1 also set at pc 0x48d490.
Breakpoint 2 at 0x48d490: file ../../gdb.updated/gdb/gdb/gdb.c, line 26.
(gdb) py print gdb.mi_execute("break-list")['body']['bkpt']['number']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: list indices must be integers, not str
Error while executing Python code.
(gdb) py for bkpt in gdb.mi_execute("break-list")['body']['bkpt']:
print bkpt['number']
1
2

and to get rid of this we'd have to have another type of list, e.g.
ui_out_type_duplicate_key_result_list, to specify a dictionary with
lists as values.  These would still have to check that the dictionary
contains the key to call, but thats standard.  The difference is that
if they do contain the key, they can safely assume that it'll be an
array even if it only contains a single object.

in essence by using introspection in the py-uiout layer, i have pushed
that introspection into the calling code, the data structures produced
when dynamic differ at various points on one hand its easier to read
what the code is doing, but harder to know what cases it might break.

adding new ui_out_types should resolve these introspection/type
stability issues, but means changing a lot of commands, and since this
is new functionality would it make sense to expedite the process by
allowing:

make ui_out_type_list assume a 'default' value (the non-deprecated one
a list of VALUES), py-uiout would throw an exception on the other

make the py-uiout throw an exception when it runs across a key for a
non duplicate_key dictionary when it a key exists
make a best effort to fix up commands,
fix the more obscure issues as people attempt to use them from
py-uiout and trip over them.

this plan somewhat risks tripping over some bit of code which creates
a list, but doesn't know whether the contents is a result/value list,
but a complete audit of all command list creation seems somewhat
difficult and time consuming.

hopefully at least this explains all in one place what i've
tried/problems encountered in trying to get py-uiout as nice as
possible.