Exiting the script after creating threads will crash the system ?

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
BramPeeters
Posts: 48
Joined: Wed Jan 31, 2018 3:10 pm

Exiting the script after creating threads will crash the system ?

Post by BramPeeters » Wed Sep 12, 2018 2:17 pm

Hi,

After analyzing execute_from_str, it seems that it is not allowed to create some helper threads and then exit the script, because the entire bytecode (or whatever you call the output of the compilation process) will be cleaned up by the garbage collector as there is no longer a reference to it. As long as the script is running, the reference is on the stack of thread calling execute_from_str in the 'module_fun' variable, but as soon as you exit the script mp_call_function_0 will return and you will leave the scope where module_fun exists, hence the memory allocated in mp_obj_new_fun_bc (see code below) will be up for freeing (?)
Or is this module_fun object added to the root pointers via some other mechanism ?

Code: Select all

mp_obj_t execute_from_str(const char *str) {
    nlr_buf_t nlr;
    if (nlr_push(&nlr) == 0) {
        mp_lexer_t *lex = mp_lexer_new_from_str_len(0/*MP_QSTR_*/, str, strlen(str), false);
        mp_parse_tree_t pt = mp_parse(lex, MP_PARSE_FILE_INPUT);
        mp_obj_t module_fun = mp_compile(&pt, lex->source_name, MP_EMIT_OPT_NONE, false);
        mp_call_function_0(module_fun);
        nlr_pop();
        return 0;
    } else 
...
}

mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, uint emit_opt, bool is_repl) {
	...
	return mp_make_function_from_raw_code(...)
}

mp_obj_t mp_make_function_from_raw_code(const mp_raw_code_t *rc, mp_obj_t def_args, mp_obj_t def_kw_args) {
{
	...
	fun = mp_obj_new_fun_bc(def_args, def_kw_args, rc->data.u_byte.bytecode, rc->data.u_byte.const_table);
	...
	return fun;	
}
Related to my other issue that stacks which are not in the GC heap are even not inspected for references (viewtopic.php?f=3&t=5245 ), this means that eg on the stm32/cc3200/... ports the entire compiled bytecode area will be 'collected' as soon as the garbage collector runs....might explain some crashes i was seeing :).

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: Exiting the script after creating threads will crash the system ?

Post by jickster » Wed Sep 12, 2018 4:54 pm

BramPeeters wrote:
Wed Sep 12, 2018 2:17 pm
Hi,

After analyzing execute_from_str, it seems that it is not allowed to create some helper threads and then exit the script
Why would you exit the VM while threads are still executing? Shouldn't you wait until they're done or kill them?

BramPeeters
Posts: 48
Joined: Wed Jan 31, 2018 3:10 pm

Re: Exiting the script after creating threads will crash the system ?

Post by BramPeeters » Wed Sep 12, 2018 5:05 pm

There is no 'hard' reason really.
Just the way i made some scripts for testing thread creation where the treads i created continued to do things for a longer time than the main thread which already had exited, simply because i was not aware of the limitation.

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: Exiting the script after creating threads will crash the system ?

Post by jickster » Wed Sep 12, 2018 5:09 pm

BramPeeters wrote:
Wed Sep 12, 2018 5:05 pm
There is no 'hard' reason really.
Just the way i made some scripts for testing thread creation where the treads i created continued to do things for a longer time than the main thread which already had exited, simply because i was not aware of the limitation.
I never said it was a limitation.

I BELIEVE it's a limitation not because of a code reason - yet - but because of this reasoning:
  • you're creating a Virtual MACHINE. If you shutdown a real machine, you wouldn't expect threads to continue executing
  • in higher-level OS programming, when you create a THREAD, it only lives as long as the parent THREAD lives

pfalcon
Posts: 1089
Joined: Fri Feb 28, 2014 2:05 pm

Re: Exiting the script after creating threads will crash the system ?

Post by pfalcon » Wed Sep 12, 2018 8:49 pm

After analyzing execute_from_str
Analyzing confusion can be no less fun than analyzing execute_from_str. Here's my take. BramPeeters, when you say that you analyze execute_from_str, you don't tell the whole truth. You forget to tell that you analyze execute_from_str in examples/embedding/hello-embed.c. I don't know if "example/embedding" part, I don't know if the start of README.md there "This directory contains a (very simple!) example", I don't know if lack of threading enabled in the included mpconfigport.h all ring any bells, but the news I have is that the example is supposed to show the absolutely minimal way to get something executed via embedded MicroPython interpreter, and not necessarily anything else.

I wouldn't be surprised if threading doesn't work with it (it was written before threading was implemented in uPy), and if I wanted to test/try/use threading, I'd use production interpreter and port. I also wouldn't be surprised if that example bitrotted somewhat and even if it contained bugs - it didn't get much attention after its inception (unlike the production interpreters). For the purpose it was written - print something via Python code - it worked well and does work well still. Doing anything more complex may require doing something more complex that it so far does (d'oh). Thanks for bringing up these issues though - they will definitely help to breathe new life into it.
Awesome MicroPython list
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
Oftentimes more up to date docs than mainline - http://pycopy.readthedocs.io/

BramPeeters
Posts: 48
Joined: Wed Jan 31, 2018 3:10 pm

Re: Exiting the script after creating threads will crash the system ?

Post by BramPeeters » Wed Sep 12, 2018 9:43 pm

You forget to tell that you analyze execute_from_str in examples/embedding/hello-embed.c.
I indeed took that sequence from the example, as an temporary intermediate step because ultimately i would like to run precompiled bytecode but that did not seem that trivial (cfr viewtopic.php?f=2&t=4473).

But if I compare the sequence of the example with that of micropython\lib\utils\pyexec.c , the

Code: Select all

STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input_kind, int exec_flags)
function in that contains the exact same sequence (for MICROPY_ENABLE_COMPILER ) of

Code: Select all

mp_lexer_t *lex = xxx
mp_parse_tree_t pt = xxx
mp_obj_t module_fun = xxx
mp_call_function_0(module_fun);
steps (along with some extra steps I do not need, being control C support and a time measurement)

So if you want i can reformulate it in terms of micropython\lib\utils\pyexec.c and stm32, the conclusion stays the same:
if you quit the main thread in parse_compile_execute after creating some extra threads for the stm32 boards, it will lead to the GC cleaning up the module_fun object code while the extra threads are still running that code.

[ I assume this is what you mean by looking at a 'production interpreter and port' , parse_compile_execute is used by the stm32 port (main.c) to run main.py via pyexec_file(main_py); ?)

Apart from that, threading seems to work fine actually (i am using cc3200 style threads on top of freertos on an stm32), I just need to play nice and not exit :).
Probably it is possible to rewrite some code so that module_fun is added to the root pointers, so that exiting the main thread becomes a possibility (because as long as i did not start gc.collect after exiting the script with initial thread, everything worked fine).
I wonder what would happen if you then run mp_call_function_0(module_fun); a second time starting a second bunch of threads :)

pfalcon
Posts: 1089
Joined: Fri Feb 28, 2014 2:05 pm

Re: Exiting the script after creating threads will crash the system ?

Post by pfalcon » Wed Sep 12, 2018 10:25 pm

... So, if you have a testcase (small .py file) which doesn't work as it should with the unchanged Unix or STM32 ports, feel free to open a bug ticket at https://github.com/micropython/micropython/issues .

(Note the responses try to address all your recent threads, I just posted it to the one with least messages, to not add noise to more populated).
Awesome MicroPython list
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
Oftentimes more up to date docs than mainline - http://pycopy.readthedocs.io/

Post Reply