Callstack in Micropython

[naums]

Hello,

The callstack is for a project of mine very important and I would like to read and modify it. For clarification: the callstack is the place, where the interpreter stores where to return when a return-statement is executed. It could also be used when raising (and catching) an Exception to determine what happened and print a traceback.

Are there any modules integrated in Micropython which are there to read the callstack? In CPython there is sys._getframe(), which could be helpful to read the frames on the stack. Are there any facilities that would allow me to modify the currently used stack, either by writing a previously saved stack in that spot or by declaring a different stack as the currently used one?

Where is the stack located? Is it in the heap or seperate to that? I haven't looked that closely in the code so: can someone explain what the format is, that is used and when the data is updated to represent the current state? I have already found, that on calling a bytecode function there are some information written onto the stack. Can I read and write these from Python-level or do I have to write my own module for that at the moment?

Code: Select all

// py/objfun.c
// line 213
#define INIT_CODESTATE(code_state, _fun_bc, n_args, n_kw, args) \
    code_state->fun_bc = _fun_bc; \
    code_state->ip = 0; \
    mp_setup_code_state(code_state, n_args, n_kw, args); \
    code_state->old_globals = mp_globals_get();

// line 251
STATIC mp_obj_t fun_bc_call(mp_obj_t self_in, size_t n_args, size_t n_kw, const mp_obj_t *args) {
    MP_STACK_CHECK();

    DEBUG_printf("Input n_args: " UINT_FMT ", n_kw: " UINT_FMT "\n", n_args, n_kw);
    DEBUG_printf("Input pos args: ");
    dump_args(args, n_args);
    DEBUG_printf("Input kw args: ");
    dump_args(args + n_args, n_kw * 2);
    mp_obj_fun_bc_t *self = MP_OBJ_TO_PTR(self_in);
    DEBUG_printf("Func n_def_args: %d\n", self->n_def_args);

    size_t n_state, state_size;
    DECODE_CODESTATE_SIZE(self->bytecode, n_state, state_size);

    // allocate state for locals and stack
    mp_code_state_t *code_state = NULL;
    #if MICROPY_ENABLE_PYSTACK
    code_state = mp_pystack_alloc(sizeof(mp_code_state_t) + state_size);
    #else
    if (state_size > VM_MAX_STATE_ON_STACK) {
        code_state = m_new_obj_var_maybe(mp_code_state_t, byte, state_size);
    }
    if (code_state == NULL) {
        code_state = alloca(sizeof(mp_code_state_t) + state_size);
        state_size = 0; // indicate that we allocated using alloca
    }
    #endif

    INIT_CODESTATE(code_state, self, n_args, n_kw, args);
    // ...

Update: I found out, that besides the code-state above, the Pystack is also used for storing references to the arguments to methods (objboundmeth.c), but not for arguments to normal functions(?). Or are functions in itself methods of itself in a way, so that when calling a function it is always seen as a method call? Can someone confirm?

PS: I assume MICROPY_ENABLE_PYSTACK to be set.

[stijn]

Unless I'm mistaken there is no built-in functionaliity whatsoever to access, let alone modify, the callstack.
If you want to studoy out how it works, maybe mp_obj_exception_add_traceback calls in vm.c are a good way to start since they are used to populate the exception traceback with human readable information.

MICROPY_ENABLE_PYSTACK is just about how to select where memory allocation for arguments etc occurs. Without it those are allocated using alloca or else from the GC heap. With it they are allocated from whatever mp_pystack_init was called with, possibly a piece of the C stack.

[naums]

Hello.

I'm currently looking into that, I found out, that with Pystack enabled the stack is located continously in memory, whereas with the stack-less option it may be scattered all over the place. Okay, both form a stack, more or less.

I am assuming PYSTACK to be enabled for simplicity reasons. And I found out, that I can (in a micropython module) read the Pystack, but I am note quite sure whether I am reading an argument to a function, a code_state structure or nothing (pad-bytes).

The really interesting thing is for me also - what happens when an exception occurs - how can I access the OLD stack before that exception was raised. If I could access that stack, I could store it somewhere in memory and restore it later on to jump back to where I was before raising the exception. Basically that behaviour is comparable to trapping into the OS-kernel in a C-program.

[pythoncoder]

...how can I access the OLD stack before that exception was raised. If I could access that stack, I could store it somewhere in memory and restore it later on to jump back to where I was before raising the exception...

That sounds quite hairy. I'd use

Code: Select all

while True:
    # prepare for running foo
    try:
        foo()
    except MyException:
        continue  # Back to any preparatory code
    break

Or am I missing something?

[naums]

Yeah, that is basically what I want to do, but I want to execute several different python programms interleaved, i.e. I want to have several processes of python code, which can at any point in time give up the CPU-time themselves, by raising a certain exception, then I would like to store the Exception callstack somewhere, restore a different one, switch heaps (i.e. tell Micropython that it should use a heap from A to B from now on) and return, to return from the function raising the exception.

I try to visualize:

Code: Select all

# "OS"
while 1:
	try:
		process = findNextProcess()
		restoreStack( process )
		restoreHeap ( process )
		# on the new stack, the top entry should be the place, where we jumped into yield earlier
		return
	except RescheduleException:
		storeExceptionStack ( currentlyRunningProcess )

# process 1:
def yield:
	raise RescheduleException

<bla some code bla>
yield ()

# process 2:
def yield:
	raise RescheduleException

<bla some code bla>
yield ()

When overcoming the hurdle of setting up the callstack when creating the processes, and filling out the blanks (i.e. functions to store/restore the heap and stack) these two processes might become alternating. At least that is the plan.

So basically what I'm looking for is information on how to switch heaps (from one heap to another one, without invalidating any information inside them), and storing and restoring the (Exception)Stack.

While we are at it: I would also need to change the list of loaded modules, but I assume, that they would also reside in the heap, right?

[pythoncoder]

This sounds ambitious. I assume you've considered and rejected uasyncio.