Extend syntax

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
pohmelie
Posts: 55
Joined: Mon Nov 23, 2015 6:31 pm

Extend syntax

Post by pohmelie » Wed Dec 16, 2015 10:46 pm

Hi, I want to extend syntax with async def/await. At first, just as sugar for mark function as generator and replace yield from with await. If so I will understand how hard it should be to implement async for/async with/awaitable objects etc.
Lets say I want async def:
1. Add MP_TOKEN_KW_ASYNC, to lexer.h right after MP_TOKEN_KW_YIELD.
2. Add "async", to lexer.c right after "yield",
3. Add rule to grammar.h (almost copy of function rule)

Code: Select all

DEF_RULE(asyncfuncdef, c(asyncfuncdef), blank | and(9), tok(KW_ASYNC), tok(KW_DEF), tok(NAME), tok(DEL_PAREN_OPEN), opt_rule(typedargslist), tok(DEL_PAREN_CLOSE), opt_rule(funcdefrettype), tok(DEL_COLON), rule(suite))
Here I understand, that there should be some changes for decorating async def, but lets omit this right now.
4. Add functions to compile.c

Code: Select all

STATIC void compile_asyncfuncdef(compiler_t *comp, mp_parse_node_struct_t *pns) {
    qstr fname = compile_asyncfuncdef_helper(comp, pns, comp->scope_cur->emit_options);
    // store function object into function name
    compile_store_id(comp, fname);
}
...
STATIC qstr compile_asyncfuncdef_helper(compiler_t *comp, mp_parse_node_struct_t *pns, uint emit_options) {
    if (comp->pass == MP_PASS_SCOPE) {
        // create a new scope for this function
        scope_t *s = scope_new_and_link(comp, SCOPE_FUNCTION, (mp_parse_node_t)pns, emit_options);
        // store the function scope so the compiling function can use it at each pass
        pns->nodes[4] = (mp_parse_node_t)s;
    }

    // get the scope for this function
    scope_t *fscope = (scope_t*)pns->nodes[4];
    fscope->scope_flags |= MP_SCOPE_FLAG_GENERATOR;

    // compile the function definition
    compile_funcdef_lambdef(comp, fscope, pns->nodes[1], PN_typedargslist);

    // return its name (the 'f' in "def f(...):")
    return fscope->simple_name;
}
I thought it is enough and it even compiles, but...

Code: Select all

>>> async def yoba(): pass
Traceback (most recent call last):
  File "<stdin>", line 1
SyntaxError: invalid syntax
>>>
So, what I missed? And it will be great if someone will explain the logic/flow of micropython compiling process.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Extend syntax

Post by pfalcon » Fri Dec 18, 2015 5:26 pm

Couple of generic comments here:

1. It's a bit of big jump from "I'm not familiar with Makefile" (and you've dealt with it in great way, kudos, and hope you enjoyed it) to "I want to add new syntax to language". I'd suggest following smoother curve with intermediate steps. We have http://forum.micropython.org/viewtopic.php?f=3&t=779 , then just going thru whatever implemented already, giving it try, testing it manually, providing feedback, establishing automated means to test it is a great way to get involved and do something good for MicroPython (then if critical mass of people starts to do that, that "any issue is shallow").

2. If you followed our communication, you know that basic async/await support was implemented (well, coded). So, instead of trying to duplicate effort (which is still kinda ok, especially if you have time for that - it just doesn't move community anywhere), you could rather search PR tracker to find an older patch, test it, forward port, test it more, resubmit, argue for actual merging. (Be sure to search in closed issues too if you're interested to follow this way.)
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: Extend syntax

Post by Damien » Fri Dec 18, 2015 6:31 pm

+1 to both points made by @pfalcon.

Your syntax error is probably because you didn't add asyncfuncdef to the list of statements in the grammar.

pohmelie
Posts: 55
Joined: Mon Nov 23, 2015 6:31 pm

Re: Extend syntax

Post by pohmelie » Fri Dec 18, 2015 11:23 pm

It's a bit of big jump
"I'm not familiar with Makefile" says nothing about my programming skills/level. It mean right what it mean.
If you followed our communication, you know that basic async/await support was implemented (well, coded). So, instead of trying to duplicate effort (which is still kinda ok, especially if you have time for that - it just doesn't move community anywhere), you could rather search PR tracker to find an older patch, test it, forward port, test it more, resubmit, argue for actual merging. (Be sure to search in closed issues too if you're interested to follow this way.)
Wow, did not know about async/await PR, thanks!

<after two hours>


Damien, your code help me a lot! Thanks! I've made async def as I want (just mark functions as generators ;) ) and made available them to be decorated. Here is diff:

Code: Select all

diff --git a/py/compile.c b/py/compile.c
index f349dca..bab638f 100644
--- a/py/compile.c
+++ b/py/compile.c
@@ -717,6 +717,14 @@ STATIC qstr compile_funcdef_helper(compiler_t *comp, mp_parse_node_struct_t *pns
     return fscope->simple_name;
 }
 
+STATIC qstr compile_async_funcdef_helper(compiler_t *comp, mp_parse_node_struct_t *pns, uint emit_options) {
+    assert(MP_PARSE_NODE_IS_STRUCT(pns->nodes[0]));
+    mp_parse_node_struct_t *pns0 = (mp_parse_node_struct_t*)pns->nodes[0];
+    qstr afname = compile_funcdef_helper(comp, pns0, emit_options);
+    scope_t *fscope = (scope_t*)pns0->nodes[4];
+    fscope->scope_flags |= MP_SCOPE_FLAG_GENERATOR;
+    return afname;
+}
 // leaves class object on stack
 // returns class name
 STATIC qstr compile_classdef_helper(compiler_t *comp, mp_parse_node_struct_t *pns, uint emit_options) {
@@ -829,6 +837,8 @@ STATIC void compile_decorated(compiler_t *comp, mp_parse_node_struct_t *pns) {
     qstr body_name = 0;
     if (MP_PARSE_NODE_STRUCT_KIND(pns_body) == PN_funcdef) {
         body_name = compile_funcdef_helper(comp, pns_body, emit_options);
+    } else if (MP_PARSE_NODE_STRUCT_KIND(pns_body) == PN_async_funcdef) {
+        body_name = compile_async_funcdef_helper(comp, pns_body, emit_options);
     } else {
         assert(MP_PARSE_NODE_STRUCT_KIND(pns_body) == PN_classdef); // should be
         body_name = compile_classdef_helper(comp, pns_body, emit_options);
@@ -849,6 +859,12 @@ STATIC void compile_funcdef(compiler_t *comp, mp_parse_node_struct_t *pns) {
     compile_store_id(comp, fname);
 }
 
+STATIC void compile_async_funcdef(compiler_t *comp, mp_parse_node_struct_t *pns) {
+    qstr fname = compile_async_funcdef_helper(comp, pns, comp->scope_cur->emit_options);
+    // store function object into function name
+    compile_store_id(comp, fname);
+}
+
 STATIC void c_del_stmt(compiler_t *comp, mp_parse_node_t pn) {
     if (MP_PARSE_NODE_IS_ID(pn)) {
         compile_delete_id(comp, MP_PARSE_NODE_LEAF_ARG(pn));
@@ -1405,7 +1421,7 @@ STATIC void compile_for_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) {
     // for viper it will be much, much faster
     if (/*comp->scope_cur->emit_options == MP_EMIT_OPT_VIPER &&*/ MP_PARSE_NODE_IS_ID(pns->nodes[0]) && MP_PARSE_NODE_IS_STRUCT_KIND(pns->nodes[1], PN_power)) {
         mp_parse_node_struct_t *pns_it = (mp_parse_node_struct_t*)pns->nodes[1];
-        if (MP_PARSE_NODE_IS_ID(pns_it->nodes[0]) 
+        if (MP_PARSE_NODE_IS_ID(pns_it->nodes[0])
             && MP_PARSE_NODE_LEAF_ARG(pns_it->nodes[0]) == MP_QSTR_range
             && MP_PARSE_NODE_IS_STRUCT_KIND(pns_it->nodes[1], PN_trailer_paren)
             && MP_PARSE_NODE_IS_NULL(pns_it->nodes[2])) {
diff --git a/py/grammar.h b/py/grammar.h
index b7036c8..741c034 100644
--- a/py/grammar.h
+++ b/py/grammar.h
@@ -46,8 +46,9 @@ DEF_RULE(eval_input_2, nc, and(1), tok(NEWLINE))
 
 // decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
 // decorators: decorator+
-// decorated: decorators (classdef | funcdef)
+// decorated: decorators (classdef | funcdef | async_funcdef)
 // funcdef: 'def' NAME parameters ['->' test] ':' suite
+// async_funcdef: 'async' funcdef
 // parameters: '(' [typedargslist] ')'
 // typedargslist: tfpdef ['=' test] (',' tfpdef ['=' test])* [',' ['*' [tfpdef] (',' tfpdef ['=' test])* [',' '**' tfpdef] | '**' tfpdef]] | '*' [tfpdef] (',' tfpdef ['=' test])* [',' '**' tfpdef] | '**' tfpdef
 // tfpdef: NAME [':' test]
@@ -57,8 +58,9 @@ DEF_RULE(eval_input_2, nc, and(1), tok(NEWLINE))
 DEF_RULE(decorator, nc, and(4), tok(DEL_AT), rule(dotted_name), opt_rule(trailer_paren), tok(NEWLINE))
 DEF_RULE(decorators, nc, one_or_more, rule(decorator))
 DEF_RULE(decorated, c(decorated), and(2), rule(decorators), rule(decorated_body))
-DEF_RULE(decorated_body, nc, or(2), rule(classdef), rule(funcdef))
+DEF_RULE(decorated_body, nc, or(3), rule(classdef), rule(funcdef), rule(async_funcdef))
 DEF_RULE(funcdef, c(funcdef), blank | and(8), tok(KW_DEF), tok(NAME), tok(DEL_PAREN_OPEN), opt_rule(typedargslist), tok(DEL_PAREN_CLOSE), opt_rule(funcdefrettype), tok(DEL_COLON), rule(suite))
+DEF_RULE(async_funcdef, c(async_funcdef), and(2), tok(KW_ASYNC), rule(funcdef))
 DEF_RULE(funcdefrettype, nc, ident | and(2), tok(DEL_MINUS_MORE), rule(test))
 // note: typedargslist lets through more than is allowed, compiler does further checks
 DEF_RULE(typedargslist, nc, list_with_end, rule(typedargslist_item), tok(DEL_COMMA))
@@ -157,7 +159,7 @@ DEF_RULE(name_list, nc, list, tok(NAME), tok(DEL_COMMA))
 DEF_RULE(assert_stmt, c(assert_stmt), and(3), tok(KW_ASSERT), rule(test), opt_rule(assert_stmt_extra))
 DEF_RULE(assert_stmt_extra, nc, ident | and(2), tok(DEL_COMMA), rule(test))
 
-// compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
+// compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_funcdef
 // if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
 // while_stmt: 'while' test ':' suite ['else' ':' suite]
 // for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
@@ -168,7 +170,7 @@ DEF_RULE(assert_stmt_extra, nc, ident | and(2), tok(DEL_COMMA), rule(test))
 // with_item: test ['as' expr]
 // suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
 
-DEF_RULE(compound_stmt, nc, or(8), rule(if_stmt), rule(while_stmt), rule(for_stmt), rule(try_stmt), rule(with_stmt), rule(funcdef), rule(classdef), rule(decorated))
+DEF_RULE(compound_stmt, nc, or(9), rule(if_stmt), rule(while_stmt), rule(for_stmt), rule(try_stmt), rule(with_stmt), rule(funcdef), rule(classdef), rule(decorated), rule(async_funcdef))
 DEF_RULE(if_stmt, c(if_stmt), and(6), tok(KW_IF), rule(test), tok(DEL_COLON), rule(suite), opt_rule(if_stmt_elif_list), opt_rule(else_stmt))
 DEF_RULE(if_stmt_elif_list, nc, one_or_more, rule(if_stmt_elif))
 DEF_RULE(if_stmt_elif, nc, and(4), tok(KW_ELIF), rule(test), tok(DEL_COLON), rule(suite))
diff --git a/py/lexer.c b/py/lexer.c
index 89ecc38..e67840a 100644
--- a/py/lexer.c
+++ b/py/lexer.c
@@ -261,6 +261,7 @@ STATIC const char *tok_kw[] = {
     "while",
     "with",
     "yield",
+    "async",
     "__debug__",
 };
 
diff --git a/py/lexer.h b/py/lexer.h
index 36d1e99..e3017bf 100644
--- a/py/lexer.h
+++ b/py/lexer.h
@@ -90,6 +90,7 @@ typedef enum _mp_token_kind_t {
     MP_TOKEN_KW_WHILE,
     MP_TOKEN_KW_WITH,
     MP_TOKEN_KW_YIELD,
+    MP_TOKEN_KW_ASYNC,
 
     MP_TOKEN_OP_PLUS,               // 47
     MP_TOKEN_OP_MINUS,
diff --git a/py/repl.c b/py/repl.c
index 182961b..4b3dcf9 100644
--- a/py/repl.c
+++ b/py/repl.c
@@ -57,6 +57,7 @@ bool mp_repl_continue_with_input(const char *input) {
         || str_startswith_word(input, "with")
         || str_startswith_word(input, "def")
         || str_startswith_word(input, "class")
+        || str_startswith_word(input, "async")
         ;
 
     // check for unmatched open bracket, quote or escape quote
I will continue with await as sugar for yield from later 8-)

pohmelie
Posts: 55
Joined: Mon Nov 23, 2015 6:31 pm

Re: Extend syntax

Post by pohmelie » Tue Dec 22, 2015 5:41 pm

I'm almost done with "await" as synonym for "yield from". The problem is with dispatching(?) between await_stmt and await_expr. It looks like it always await_expr. I add/change such rules to grammar.h:

Code: Select all

DEF_RULE(flow_stmt, nc, or(6), rule(break_stmt), rule(continue_stmt), rule(return_stmt), rule(raise_stmt), rule(yield_stmt), rule(await_stmt))
DEF_RULE(await_stmt, c(await_stmt), and(1), rule(await_expr))
DEF_RULE(expr_stmt_6, nc, or(3), rule(yield_expr), rule(await_expr), rule(testlist_star_expr))
DEF_RULE(atom_2b, nc, or(3), rule(yield_expr), rule(await_expr), rule(testlist_comp))
DEF_RULE(await_expr, c(await_expr), and(2), tok(KW_AWAIT), rule(test))
and such functions to compile.c:

Code: Select all

STATIC void compile_await_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) {
    compile_node(comp, pns->nodes[0]);
    EMIT(pop_top);
}
STATIC void compile_await_expr(compiler_t *comp, mp_parse_node_struct_t *pns) {
    if (comp->scope_cur->kind != SCOPE_FUNCTION && comp->scope_cur->kind != SCOPE_LAMBDA) {
        compile_syntax_error(comp, (mp_parse_node_t)pns, "'await' outside function");
        return;
    }
    compile_node(comp, pns->nodes[0]);
    EMIT(get_iter);
    EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE);
    EMIT(yield_from);
}
Those are just copies from compile_yield_stmt and compile_yield_expr (partial copy for last).
And if I write:

Code: Select all

r = await coro()
everything is fine, and compile_await_expr called.
But if I ommit "r ="

Code: Select all

await coro()
then there is no call of compile_await_stmt (which will pop "await" result out), but only compile_await_expr. And I got "ERROR: stack size not back to zero; got 1" message.

Also some questions about grammar:
  • what is "nc" argument? No compile?
  • what is rule(test)?

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: Extend syntax

Post by Damien » Tue Dec 22, 2015 11:53 pm

Try printing some debugging info. There is an option near the top of parse.c to print names of parse nodes. Then in unix/main.c you can uncomment the code that prints the parse tree.

nc means no compile and is used for parse nodes that are either never emitted (they are optimised away by the parser, eg or nodes) or are handled in a special way by the compiler.

rule(X) means try to match rule X. "test" is just one of the rules (top level rule for expressions).

pohmelie
Posts: 55
Joined: Mon Nov 23, 2015 6:31 pm

Re: Extend syntax

Post by pohmelie » Thu Dec 24, 2015 1:30 pm

Damien, thanks for debug mode option. It works just fine since now. 8-)

As I said, the next step is to try to implement "async with", "async for". And here is my idea: just do this as pep says:
https://www.python.org/dev/peps/pep-0492/#new-syntax
https://www.python.org/dev/peps/pep-0492/#id10
So don't need to implement something new on core level. Just emit sequence of commands and compile body. Is this way ok or you have some better ideas to implement this?

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: Extend syntax

Post by Damien » Sun Dec 27, 2015 10:30 pm

pohmelie wrote: As I said, the next step is to try to implement "async with", "async for". And here is my idea: just do this as pep says:
https://www.python.org/dev/peps/pep-0492/#new-syntax
https://www.python.org/dev/peps/pep-0492/#id10
So don't need to implement something new on core level. Just emit sequence of commands and compile body. Is this way ok or you have some better ideas to implement this?
The usual way I work is I try a few different approaches (some more complete than others), evaluate the advantages of each approach, and then make a decision what the best way is to move forward. So I can't say if the way you propose will be the best! Confining the changes to just the compiler is a good idea since it doesn't add complexity to the VM, or add new bytecodes, and native code would work as well (if only it had generators...). Best advice I can give then is to just try it and see how far you get, then report results.

pohmelie
Posts: 55
Joined: Mon Nov 23, 2015 6:31 pm

Re: Extend syntax

Post by pohmelie » Sun Dec 27, 2015 10:47 pm

Damien wrote:Best advice I can give then is to just try it and see how far you get, then report results
This is good. When I post my previous message, I expect that there is something I certainly miss (as it was in another threads and issues on github :lol: ). I will report progress ASAP.

pohmelie
Posts: 55
Joined: Mon Nov 23, 2015 6:31 pm

Re: Extend syntax

Post by pohmelie » Tue Dec 29, 2015 2:12 pm

I'm stuck with async with. Trying to implement combo of compile_with_stmt_helper and compile_try_except for catching and finalization. It looks like first part works fine, since I've just add yield from of magic method __aenter__. So, the try/except/finally part is hard, and I have some questions:
1. comp_next_label. Are labels a compiletime markers? So we can put this code here, and that code there. Not just as straight block by block. Am I right?
2. pop_block. Time to understand what it is. pop_top is ok, it like pop results from stack, but what is pop_block?
3. What is the core of exception emitters? I mean, lets say I want just "try: block1 except: block2", what will be the emitters/compile_node sequence? It's just to understand the pattern.
4. Is it possible to take object without name in except? With "with" we have builtin(?) mechanic for cleanup, but this time this should be made by hand. So, we have

Code: Select all

async with AWith() as foo:
    print(foo)
This translates to something like

Code: Select all

foo = await AWith().__aenter__()
try:
    print(foo)
except:
    if not await (?).__aexit__(...):
        raise
finally:
    ...
And AWith instance have no name. How can I do async call it method in except/finally cases?

Post Reply