Extend syntax

[Damien]

I'm stuck with async with.

I'm not surprised... it's difficult! There's a reason why it's not already been done

1. comp_next_label. Are labels a compiletime markers? So we can put this code here, and that code there. Not just as straight block by block. Am I right?

It's just like a label in assembler, or a goto label in C. Nothing to do with blocks.

2. pop_block. Time to understand what it is. pop_top is ok, it like pop results from stack, but what is pop_block?

Look at how the bytecode is implemented in vm.c. It pops an exception handler from the exception stack.

3. What is the core of exception emitters? I mean, lets say I want just "try: block1 except: block2", what will be the emitters/compile_node sequence? It's just to understand the pattern.

Make a simple test script and run it with unix version of micropython, using "-v -v" as command line arguments. This will show you the generated bytecode, and from this you can inderstand how all code is compiled.

4. Is it possible to take object without name in except?

Yes, easily. Just look at how a simple "try: pass except: pass" is compiled.

And AWith instance have no name. How can I do async call it method in except/finally cases?

Store the instance on the stack and access it that way (a bit difficult, but the only way to do it properly).

[pohmelie]

Thanks, Damien, "-v -v" option is just awesome! I've found and fix some bug in "await" with it.
Now I'm trying to understand how to put object on stack. So, I see "compile_store_id" c-function call for "x = foo()" python code, which stores result with

Code: Select all

STORE_FAST NUMBER

Number is growing up index of object on some stack/array. But it is still bound with name. I can't find function, which will put/append some object to stack/array without binding it with name. I see some counter inside "binding":

Code: Select all

id_info_t *id = scope_find(scope, qst);
id->local_num

Which is number for "LOAD_FAST" too.
And there definetly should be increment of some scope counter for this. But I can't find it

To be more clear. I've got python code

Code: Select all

async def yoba():

    AWith()

Code: Select all

$ ./micropython -v -v test-try-short.py 
----------------
[   1] async_stmt(90) (n=1)
[   1]   funcdef(10) (n=5)
           id(yoba)
           NULL
           NULL
[   3]     expr_stmt(32) (n=2)
[   3]       power(123) (n=3)
               id(AWith)
[   3]         trailer_paren(139) (n=1)
                 NULL
               NULL
             NULL
           NULL
----------------
File test-try-short.py, code block '<module>' (descriptor: 0x7fe5a3637f40, bytecode @0x7fe5a3638380 37 bytes)
Raw bytecode (code_info_size=10, bytecode_size=27):
 01 00 00 00 00 00 0a 82 33 83 7b 00 00 00 00 00
 ff 60 00 00 00 00 00 00 00 83 63 a3 e5 7f 00 00
 24 83 7c 11 5b

arg names:(N_STATE 1)
(N_EXC_STACK 0)
  bc=-1 line=1
00 MAKE_FUNCTION 0x7fe5a3638300
15 STORE_NAME yoba
18 LOAD_CONST_NONE
19 RETURN_VALUE
File test-try-short.py, code block 'yoba' (descriptor: 0x7fe5a3638300, bytecode @0x7fe5a36383c0 26 bytes)
Raw bytecode (code_info_size=10, bytecode_size=16):
 01 00 04 00 00 00 0a 83 7c 83 7b 41 00 00 00 00
 ff 1d 83 7d 00 64 00 32 11 5b

arg names:(N_STATE 1)
(N_EXC_STACK 0)
  bc=-1 line=1
  bc=0 line=3
00 LOAD_GLOBAL AWith (cache=0)
04 CALL_FUNCTION n=0 nkw=0
06 POP_TOP
07 LOAD_CONST_NONE
08 RETURN_VALUE
mem: total=3034, current=910, peak=2403
stack: 32 out of 80000
GC: total: 2072832, used: 992, free: 2071840
 No. of 1-blocks: 9, 2-blocks: 2, max blk sz: 6

Interesting part of bytecode is

Code: Select all

00 LOAD_GLOBAL AWith (cache=0)
04 CALL_FUNCTION n=0 nkw=0
06 POP_TOP

I need to replace "POP_TOP" with "STORE_FAST GOOD_NEW_INDEX", and want to use this "GOOD_NEW_INDEX" later for "LOAD_FAST". Which c-function will help me?

[pohmelie]

I've some success with "async with":

Code: Select all

STATIC void compile_async_with_stmt_helper(compiler_t *comp, int n, mp_parse_node_t *nodes, mp_parse_node_t body) {
    if (n == 0) {
        // no more pre-bits, compile the body of the with
        compile_node(comp, body);
    } else {
        uint label_exception = comp_next_label(comp);
        uint label_no_reraise = comp_next_label(comp);
        uint label_success = comp_next_label(comp);
        uint label_end = comp_next_label(comp);
        qstr context;

        if (MP_PARSE_NODE_IS_STRUCT_KIND(nodes[0], PN_with_item)) {
            // this pre-bit is of the form "a as b"
            mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)nodes[0];
            compile_node(comp, pns->nodes[0]);
            context = MP_PARSE_NODE_LEAF_ARG(pns->nodes[0]);
            compile_store_id(comp, context);
            compile_load_id(comp, context);
            EMIT_ARG(load_method, MP_QSTR___aenter__);
            EMIT_ARG(call_method, 0, 0, 0);
            EMIT(get_iter);
            EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE);
            EMIT(yield_from);
            c_assign(comp, pns->nodes[1], ASSIGN_STORE);
        } else {
            // this pre-bit is just an expression
            compile_node(comp, nodes[0]);
            context = MP_PARSE_NODE_LEAF_ARG(nodes[0]);
            compile_store_id(comp, context);
            compile_load_id(comp, context);
            EMIT_ARG(load_method, MP_QSTR___aenter__);
            EMIT_ARG(call_method, 0, 0, 0);
            EMIT(get_iter);
            EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE);
            EMIT(yield_from);
            EMIT(pop_top);
        }
        EMIT_ARG(setup_except, label_exception);
        compile_increase_except_level(comp);
        // compile additional pre-bits and the body
        compile_async_with_stmt_helper(comp, n - 1, nodes + 1, body);
        // finish this with block
        EMIT(pop_block);
        EMIT_ARG(jump, label_success); // jump over exception handler

        EMIT_ARG(label_assign, label_exception); // start of exception handler
        EMIT(start_except_handler);

        EMIT(pop_top);
        EMIT(pop_top);
        EMIT(pop_top);

        compile_load_id(comp, context);
        EMIT_ARG(load_method, MP_QSTR___aexit__);
        EMIT_ARG(call_method, 0, 0, 0);             // TODO: add *sys.exc_info()
        EMIT(get_iter);
        EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE);
        EMIT(yield_from);
        EMIT_ARG(pop_jump_if, true, label_no_reraise);
        EMIT_ARG(raise_varargs, 0);

        EMIT_ARG(label_assign, label_no_reraise);
        EMIT(pop_except);
        compile_decrease_except_level(comp);
        EMIT(end_except_handler);
        EMIT_ARG(jump, label_end);
        EMIT(end_finally);

        EMIT_ARG(label_assign, label_success);
        compile_load_id(comp, context);
        EMIT_ARG(load_method, MP_QSTR___aexit__);
        EMIT_ARG(call_method, 0, 0, 0);             // TODO: add *sys.exc_info()
        EMIT(get_iter);
        EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE);
        EMIT(yield_from);
        EMIT(pop_top);
        EMIT_ARG(label_assign, label_end);

    }
}

STATIC void compile_async_with_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) {
    // get the nodes for the pre-bit of the with (the a as b, c as d, ... bit)
    mp_parse_node_t *nodes;
    int n = mp_parse_node_extract_list(&pns->nodes[0], PN_with_stmt_list, &nodes);
    assert(n > 0);

    // compile in a nested fashion
    compile_async_with_stmt_helper(comp, n, nodes, pns->nodes[1]);
}

#include <stdio.h>
STATIC void compile_async_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) {
    assert(MP_PARSE_NODE_IS_STRUCT(pns->nodes[0]));
    mp_parse_node_struct_t *pns0 = (mp_parse_node_struct_t*)pns->nodes[0];
    if (MP_PARSE_NODE_STRUCT_KIND(pns0) == PN_funcdef) {
        // async def
        compile_funcdef(comp, pns0);
        scope_t *fscope = (scope_t*)pns0->nodes[4];
        fscope->scope_flags |= MP_SCOPE_FLAG_GENERATOR;
    } else if (MP_PARSE_NODE_STRUCT_KIND(pns0) == PN_for_stmt) {
        // async for
        // TODO implement me
        printf("async for\n");
        // compile_node(comp, pns->nodes[0]);
    } else {
        // async with
        // TODO implement me
        printf("async with\n");
        assert(MP_PARSE_NODE_STRUCT_KIND(pns0) == PN_with_stmt);
        compile_async_with_stmt(comp, (mp_parse_node_struct_t *)pns->nodes[0]);
    }
}

Above code compile

Code: Select all

async def yoba():

    async with AWith() as foo:

        print(foo)

into exactly same bytecode as

Code: Select all

async def yoba():

    mgr = AWith()
    foo = await mgr.__aenter__()
    try:

        print(foo)

    except:

        if not await mgr.__aexit__():

            raise

    else:

        await mgr.__aexit__()

Except, that there is no «mgr» name in scope.

Code: Select all

----------------
[   1] async_stmt(90) (n=1)
[   1]   funcdef(10) (n=5)
           id(yoba)
           NULL
           NULL
[   3]     async_stmt(90) (n=1)
[   3]       with_stmt(86) (n=2)
[   3]         with_item(88) (n=2)
[   3]           power(123) (n=3)
                   id(AWith)
[   3]             trailer_paren(139) (n=1)
                     NULL
                   NULL
                 id(foo)
[   5]         expr_stmt(32) (n=2)
[   5]           power(123) (n=3)
                   id(print)
[   5]             trailer_paren(139) (n=1)
                     id(foo)
                   NULL
                 NULL
           NULL
----------------
async with
File test-awith-short.py, code block '<module>' (descriptor: 0x7f921f59bf40, bytecode @0x7f921f59c420 37 bytes)
Raw bytecode (code_info_size=10, bytecode_size=27):
 01 00 00 00 00 00 0a 82 33 83 7b 00 00 00 00 00
 ff 60 00 00 00 00 00 00 80 c3 59 1f 92 7f 00 00
 24 83 7c 11 5b

arg names:(N_STATE 1)
(N_EXC_STACK 0)
  bc=-1 line=1
00 MAKE_FUNCTION 0x7f921f59c380
15 STORE_NAME yoba
18 LOAD_CONST_NONE
19 RETURN_VALUE
async with
ERROR: stack size not back to zero; got -3
async with
ERROR: stack size not back to zero; got -3
async with
ERROR: stack size not back to zero; got -3
File test-awith-short.py, code block 'yoba' (descriptor: 0x7f921f59c380, bytecode @0x7f921f59c460 80 bytes)
Raw bytecode (code_info_size=10, bytecode_size=70):
 07 01 04 00 00 00 0a 83 7c 83 7b 41 53 00 00 00
 ff 1d 83 7d 00 64 00 c0 b0 1f 18 66 00 42 11 5e
 c1 3f 0c 00 1d 81 20 00 b1 64 01 32 44 35 15 80
 32 32 32 b0 1f 19 66 00 42 11 5e 36 02 80 5c 00
 45 35 0a 80 41 b0 1f 19 66 00 42 11 5e 32 11 5b

arg names:(N_STATE 7)
(N_EXC_STACK 1)
  bc=-1 line=1
  bc=0 line=3
  bc=19 line=5
00 LOAD_GLOBAL AWith (cache=0)
04 CALL_FUNCTION n=0 nkw=0
06 STORE_FAST 0
07 LOAD_FAST 0
08 LOAD_METHOD __aenter__
10 CALL_METHOD n=0 nkw=0
12 GET_ITER
13 LOAD_CONST_NONE
14 YIELD_FROM
15 STORE_FAST 1
16 SETUP_EXCEPT 31
19 LOAD_GLOBAL print (cache=0)
23 LOAD_FAST 1
24 CALL_FUNCTION n=1 nkw=0
26 POP_TOP
27 POP_BLOCK
28 JUMP 52
31 POP_TOP
32 POP_TOP
33 POP_TOP
34 LOAD_FAST 0
35 LOAD_METHOD __aexit__
37 CALL_METHOD n=0 nkw=0
39 GET_ITER
40 LOAD_CONST_NONE
41 YIELD_FROM
42 POP_JUMP_IF_TRUE 47
45 RAISE_VARARGS 0
47 POP_EXCEPT
48 JUMP 61
51 END_FINALLY
52 LOAD_FAST 0
53 LOAD_METHOD __aexit__
55 CALL_METHOD n=0 nkw=0
57 GET_ITER
58 LOAD_CONST_NONE
59 YIELD_FROM
60 POP_TOP
61 LOAD_CONST_NONE
62 RETURN_VALUE
mem: total=3213, current=945, peak=2496
stack: 32 out of 80000
GC: total: 2072832, used: 1056, free: 2071776
 No. of 1-blocks: 8, 2-blocks: 2, max blk sz: 6

But, I get "micropython: ../py/emitbc.c:486: emit_bc_pre: Assertion `(mp_int_t)emit->stack_size + stack_size_delta >= 0' failed." error.
What did I miss?

[pohmelie]

Looks like I'm almost done with "async with". It was interesting. So, error was in understanding of flow. And even if bytecode is same, there is some "magic" (or magic just for me) how unreachable code affect stack and state.
The last problem is *sys.exc_info() for __aexit__ call. I have this three values on stack right after enter exception, so, I need to use them. But function must be before them on stack to call. I mean I have [a1, a2, a3] on stack and if I load method this become [a1, a2, a3, method], but to make proper call I need [method, a1, a2, a3].
Solutions:

• load method not to top (is this possible?).
• pop arguments and store them to locals, then push method and push arguments back.
• implement MP_BC_ROT_FOUR. (extend vm, bad idea?)
• load method before arguments appear on stack (which place this should be done? before "try"?)
• use same code as sys.exc_info()

And probably there will be disambiguation with variables order, [a1, a2, a3] or [a3, a2, a1].

[pohmelie]

This was not so hard as I expected. "load method before arguments" works just fine. I was not sure, that there is nothing between loaded method and exception arguments. But hopefully this is true. So I load method before try and right after except my stack looks like [method, a3, a2, a1], so I just made rot_three and rot_two to change arguments order to [method, a1, a2, a3]. And now it looks like it fully worked as pep492 said.
There is also same (as cpython) behaviour for

Code: Select all

async with Foo() as foo, Bar() as bar:

And exception raises in Bar().__aenter__, so async call to foo.__exit__ made.

[pohmelie]

"async for" was pretty easy after "async with" and diggin into µpython code. It looks like, everything works fine. I will try to minimize code, refactor a bit and end with pull request…