[PATCH v3 2/2] D: Move classification of symbols from the grammar to the lexer

[Doug Evans]

Iain Buclaw <ibuclaw@gdcproject.org> writes:
> This patch rewrites the lexer to be just like C++'s, where it will
> attempt to classify and reduce any alternating '.' and symbol-like
> tokens.
>
> There is a little added complexity as existing D compilers will only
> write the full module name to debug.  For instance, given
> 'package.and.another.foo', the first three parts will return as
> unknown/non-existent symbols.   While it would be nice to fix this in
> my compiler, there does not seem to be a nice way to chain package and
> module names together that is understood by GDB (I think language_go
> has a similar problem to this).
>
> However, now I'm happy.  As I can finally begin to start supporting
> and implementing exp.property style syntax in the grammar without
> running into shift reduce conflicts (starting with .sizeof and .typeof
> :-)
>
> Regards
> Iain.
> ---
>
> 2015-08-03  Iain Buclaw  <ibuclaw@gdcproject.org>
>     
>     	* d-exp.y (%union): Add voidval.
>     	(%token): Add UNKNOWN_NAME as a token to represent an unclassified
>     	name in the lexing stage.
>     	(PostfixExpression): Move symbol completion handling in grammar here
>     	from PrimaryExpression.
>     	(PrimaryExpression): Move routines to handle resolving identifier
>     	tokens in the grammar here from push_expression_name.
>     	(IdentifierExp): Remove the handling of alternating '.' and identifier
>     	tokens.
>     	(TypeExp): Allow TypeExp to be wrapped in parenthesis in the grammar.
>     	(BasicType): Remove C-style typename rules.
>     	(d_type_from_name, d_module_from_name, push_variable)
>     	(push_fieldnames, push_type_name, push_module_name)
>     	(push_expression_name): Remove.
>     	(lex_one_token): Rename from yylex.  Replace pstate with par_state.
>     	(token_and_value): New type.
>     	(token_fifo, popping, name_obstack): New globals.
>     	(classify_name): New function.
>     	(classify_inner_name): Likewise.
>     	(yylex): Likewise.
>     	(d_parse): Initialize token_fifo, popping and name_obstack.

Hi.

I didn't have time to give this more than a glance.
Since it's all just pure D stuff, I'd say go for it.