[SOLVED]MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

[SOLVED]MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

Post by jickster » Wed Nov 15, 2017 4:23 pm

My RAM is extremely limited so I need to execute from flash; which method of code storage can do this?
Last edited by jickster on Thu Nov 23, 2017 2:32 am, edited 1 time in total.

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

Post by Damien » Thu Nov 16, 2017 5:48 am

FROZEN_STR will store the scripts as strings in flash and when imported compile them at runtime and store the compiled bytecode in RAM. So that won't help to reduce RAM usage, it's instead a nice way to have scripts and resources in flash without having a filesystem).

FROZEN_MPY is what you want: it will precompile the scripts offline (during the "make" process), freeze the bytecode into flash, and the execution of bytecode happens from flash. You'll need to use mpy-cross to precompile the scripts, and tools/mpy-tool.py to freeze the data. See the ports/minimal example code for use of FROZEN_MPY, and also ports/stm32 (and ports/esp8266) for a more sophisticated use where the precompilation is all done automatically through the Makefile.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

Post by jickster » Thu Nov 16, 2017 3:27 pm

Damien wrote:FROZEN_STR will store the scripts as strings in flash and when imported compile them at runtime and store the compiled bytecode in RAM. So that won't help to reduce RAM usage, it's instead a nice way to have scripts and resources in flash without having a filesystem).

FROZEN_MPY is what you want: it will precompile the scripts offline (during the "make" process), freeze the bytecode into flash, and the execution of bytecode happens from flash. You'll need to use mpy-cross to precompile the scripts, and tools/mpy-tool.py to freeze the data. See the ports/minimal example code for use of FROZEN_MPY, and also ports/stm32 (and ports/esp8266) for a more sophisticated use where the precompilation is all done automatically through the Makefile.
Thanks!

I have followed the FROZEN_MPY workflow successfully; good to know it executes from flash.

Regarding the MPY, why do I need to convert the .mpy file - which is pure binary - to the frozen-mpy-in-C format?

I guess I’m asking: is the .mpy format useful in itself either now or in the future (ie without converting to mpy-in-C format)?






Sent from my iPhone using Tapatalk

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

Post by Damien » Fri Nov 17, 2017 12:56 am

jickster wrote:
Thu Nov 16, 2017 3:27 pm
Regarding the MPY, why do I need to convert the .mpy file - which is pure binary - to the frozen-mpy-in-C format?

I guess I’m asking: is the .mpy format useful in itself either now or in the future (ie without converting to mpy-in-C format)?
Yes .mpy is useful. There are 3 main ways to import and execute a script:
  • A textual .py file (either from a filesystem or FROZEN_STR) is parsed and compiled into bytecode which executes from RAM
  • A precompiled .mpy binary file (from a filesystem) is loaded directly into RAM for execution; benefits over .py: saves RAM and time because compilation is done offline, also these files are usually portable to many different boards/ports and don't require rebuilding the firmware
  • A frozen .mpy file has the bytecode converted to a C data structure and compiled into the firmware, where it can be executed from flash/ROM; benefits over normal (unfrozen) .mpy: almost zero time and RAM overhead for the import, can execute from flash; downside is that changing the script requires rebuilding the firmware
Note that there are ongoing efforts to dynamically freeze .mpy files at runtime, eg https://github.com/micropython/micropython/issues/2709

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

Post by jickster » Fri Nov 17, 2017 3:02 pm

Damien wrote:
Fri Nov 17, 2017 12:56 am
jickster wrote:
Thu Nov 16, 2017 3:27 pm
Regarding the MPY, why do I need to convert the .mpy file - which is pure binary - to the frozen-mpy-in-C format?

I guess I’m asking: is the .mpy format useful in itself either now or in the future (ie without converting to mpy-in-C format)?
Yes .mpy is useful. There are 3 main ways to import and execute a script:
  • A textual .py file (either from a filesystem or FROZEN_STR) is parsed and compiled into bytecode which executes from RAM
  • A precompiled .mpy binary file (from a filesystem) is loaded directly into RAM for execution; benefits over .py: saves RAM and time because compilation is done offline, also these files are usually portable to many different boards/ports and don't require rebuilding the firmware
  • A frozen .mpy file has the bytecode converted to a C data structure and compiled into the firmware, where it can be executed from flash/ROM; benefits over normal (unfrozen) .mpy: almost zero time and RAM overhead for the import, can execute from flash; downside is that changing the script requires rebuilding the firmware
Note that there are ongoing efforts to dynamically freeze .mpy files at runtime, eg https://github.com/micropython/micropython/issues/2709
Why can't you execute from flash the code that is saved using persistent.c: mp_raw_code_save()?

Is it simply a matter of formatting it so it matches the generated _frozen_mpy.c?

Code: Select all

python ../tools/mpy-tool.py -f -q build/genhdr/qstrdefs.preprocessed.h -mlongint-impl=none frozentest.mpy > build/_frozen_mpy.c

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: MP_FROZEN_STR, MP_FROZEN_MPY - which can execute from flash?

Post by jickster » Fri Nov 17, 2017 7:07 pm

Update:

The normal function called in the process of execution is

Code: Select all

mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, uint emit_opt, bool is_repl) {
    mp_raw_code_t *rc = mp_compile_to_raw_code(parse_tree, source_file, emit_opt, is_repl);
    // return function that executes the outer module
    return mp_make_function_from_raw_code(rc, MP_OBJ_NULL, MP_OBJ_NULL);
}
Since rc is not modified in mp_make_function_from_raw_code(const mp_raw_code_t *rc, mp_obj_t def_args, mp_obj_t def_kw_args);, then you should be able to save rc to flash and pass in the new pointer like so:

Code: Select all

mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, uint emit_opt, bool is_repl) {
    mp_raw_code_t *rc = mp_compile_to_raw_code(parse_tree, source_file, emit_opt, is_repl);
    // return function that executes the outer module
    newRC =   mp_save_raw_code(rc); // modify to return new pointer to flash memory
    return mp_make_function_from_raw_code(newRC, MP_OBJ_NULL, MP_OBJ_NULL);
}
There is an issue. Often when writing to flash, you place a header at the beginning of each flash-page, breaking the contiguity of the data structure you're saving. For mp_raw_code_t to be executable from flash, there can be no additional meta-data that breaks up the contiguity of the mp_raw_code_t .

Post Reply