Difference in mpy output on target versus host

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
BramPeeters
Posts: 54
Joined: Wed Jan 31, 2018 3:10 pm

Difference in mpy output on target versus host

Post by BramPeeters » Mon Jan 06, 2020 1:54 pm

Hi,

Due to running into memory size constraints with on target compilation of scripts I am looking into compiling from py to mpy on an external host and transferring the compiled program to the target.
At first sight it actually seems to work but if I compare an mpy file compiled on the target versus one compiled on the host pc there are a lot of differences between them. One I can decipher in the bytecode is that some names are postfixed with the filename in the host pc version (so eg enter versus enter*main.py), but there are many more.
I figured this maybe had to do with qstrings and i added all the generated qstrings of the target to the qstrdefsport.h file in mpy-cross and recompiled mpy-cross but as far as i can see these have been completely ignored (they do not end up in /mpy-cross/build/genhdr/qstrdefs.generated.h).
I also do not think it has to do with optimization levels as both files have about the same size after compilation (85582 on target versus 86565 on host).

On the host i compile with ./mpy-cross main.py
On the target i compile/save with (with the program loaded into the str variable) :

Code: Select all

        qstr src_name = MP_QSTR_;
        mp_lexer_t *lex = mp_lexer_new_from_str_len( src_name, str, strlen(str), false);
        mp_parse_tree_t pt = mp_parse(lex, MP_PARSE_FILE_INPUT);
        mp_raw_code_t *rc = mp_compile_to_raw_code(&pt, src_name, MP_EMIT_OPT_NONE, false);
        mp_raw_code_save_file(rc, output_file);
Sadly I am way out of sync with the latest releases and i am still running 1.9.3

Anyone any insight in where the differences might come from (and if I should be worried about them ) ?

Thanks
Bram

jedie
Posts: 245
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: Difference in mpy output on target versus host

Post by jedie » Mon Jan 06, 2020 2:07 pm

I don't understand what do you mean with "on target compilation of scripts"

It sounds to me, that you old solution, compiles py -> mpy files directly on the device ?!? And now you would like to compile it on PC and send it to device?!?

btw. my experience on ESP8266 is, that .mpy compilation will not save enough RAM, see my long journey: viewtopic.php?f=2&t=7345

BramPeeters
Posts: 54
Joined: Wed Jan 31, 2018 3:10 pm

Re: Difference in mpy output on target versus host

Post by BramPeeters » Mon Jan 06, 2020 4:04 pm

It sounds to me, that you old solution, compiles py -> mpy files directly on the device ?!? And now you would like to compile it on PC and send it to device?!?
Exactly, target == device
btw. my experience on ESP8266 is, that .mpy compilation will not save enough RAM
It seems on our board it would make quite a difference, but memory is probably organised a bit different than on your ESP8266 (I assume, not familiar with ESP8266). I have an external 512kb sram. About half of this is reserved for uPython heap and half for normal IAR heap ( I run uPython on top of freertos with an IAR toolchain). Freertos heap is in internal cpu memory along with all the variables.

We run into troubles compiling scripts with ~2000 LOC either because of uPython stack or heap issues (but after increasing the stack now it seems heap-related). Since increasing uPython heap comes at the cost of decreasing IAR heap I want to avoid doing that if the extra heap is only needed for compilation (IAR heap is also in high demand, we also run sqlite in IAR heap which eats kb like candy) . Once the mpy files are compiled they are stored in a filesystem and loaded from the filesystem sits in IAR heap so for that part uPython heap is not used (of course if the files get much big we will start running more and more into IAR heap problems too), and uPython heap is only used for objects really created by the code running which we can control by not doing too much in parallel.

Freezing (which embeds scripts into the firmware if I understand correctly) is not an option, we have a device which runs common firmware to which the client then sends scripts to do specific things (different scripts on different devices). ( If we have no other options left we might try to identify some often recurring parts and freeze them, but probably then it would be even better to just code that functionality in C land)

jedie
Posts: 245
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: Difference in mpy output on target versus host

Post by jedie » Mon Jan 06, 2020 4:13 pm

BramPeeters wrote:
Mon Jan 06, 2020 4:04 pm
Freezing (which embeds scripts into the firmware if I understand correctly) is not an option, we have a device which runs common firmware to which the client then sends scripts to do specific things (different scripts on different devices). ( If we have no other options left we might try to identify some often recurring parts and freeze them, but probably then it would be even better to just code that functionality in C land)
Yes, freezing embeds compiled script in firmware. The difference is: micropython can run these script directly, without need to copy .mpv from flash to RAM... at least that's what on ESP32/8266 happens, but i think it works in the same way on every platform. See also: viewtopic.php?f=2&t=7345&p=42825#p42807

So, you can't freeze everything, because you will not compile to many different firmwares for the different devices, right?
There will probably be some scripts that are the same everywhere. These can be frozen.

In my https://github.com/jedie/micropython-sonoff-webswitch project i doe this:
  • freeze everything (except boot.py and main.py)
  • send via "soft" OTA updates missing or changed files as .mpy to the device
I also change the sys.path, so that first .mpy files loaded from flash filesystem and then from freeze modules (if not found on flash)

BramPeeters
Posts: 54
Joined: Wed Jan 31, 2018 3:10 pm

Re: Difference in mpy output on target versus host

Post by BramPeeters » Mon Jan 06, 2020 5:46 pm

Potentially we could move to something like this in the future (and always nice to have something to start from :P) , but for now as step 1 I would like the understand why the PC compiled files are different from the target/device compiled ones :)

jedie
Posts: 245
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: Difference in mpy output on target versus host

Post by jedie » Mon Jan 06, 2020 6:29 pm

BramPeeters wrote:
Mon Jan 06, 2020 5:46 pm
I would like the understand why the PC compiled files are different from the target/device compiled ones :)
Can you explain how you compile the files on device? How do you compare? And witch board do you use?

User avatar
jimmo
Posts: 1204
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Difference in mpy output on target versus host

Post by jimmo » Mon Jan 06, 2020 8:40 pm

jedie wrote:
Mon Jan 06, 2020 6:29 pm
Can you explain how you compile the files on device? How do you compare? And witch board do you use?
They already did -- the code snippet is in the first post -- you basically just run the same code that mpy-cross does. When you have MICROPY_PERSISTENT_CODE_SAVE enabled, you can do this. However it's likely they to make some other minor changes too (I'm guessing you provided your own implementation of mp_raw_code_save_file to write to the IAR filesystem? Or do you have a way to map the IAR filesystem to the MPY VFS?
BramPeeters wrote:
Mon Jan 06, 2020 1:54 pm
Sadly I am way out of sync with the latest releases and i am still running 1.9.3
Just to confirm that you're running also the 1.9.3 version of mpy-cross, and that despite the differences in the files, both versions work fine?

I'll need to go back to 1.9.3 and test this out for myself.

BramPeeters
Posts: 54
Joined: Wed Jan 31, 2018 3:10 pm

Re: Difference in mpy output on target versus host

Post by BramPeeters » Mon Jan 06, 2020 9:04 pm

Can you explain how you compile the files on device? How do you compare? And witch board do you use?
There are various ways I can access the device's file system (let it act as usb mass storage device, or over a network connection, ... )
So current procedure was to place a script text file in the file system which the device then compiles via the calls given in the original post.
I can then extract that compiled and saved mpy file via the same way and thus compare it to the pc compiled version.
Maybe i am the first to do this and realize there is in fact a difference :P . To be sure i should either fill in the filename in qstr src_name = MP_QSTR_; on the device or clear the filename on the pc version.

The board is a proprietary board based on stm32f4.

Edit: ok already answered by jimmo but this contains a bit more details about the practical side so i will leave it in
(I'm guessing you provided your own implementation of mp_raw_code_save_file to write to the IAR filesystem? Or do you have a way to map the IAR filesystem to the MPY VFS?
Haha yes, but to make life complex we are actually using fatfs, but the original one, so not uPython's altered oofatfs variant.
Mostly because originally we already had other components accessing the file system via fatfs and it seemed unneeded or even dangerous to have raw file access from within uPython at the time. Of course later on there was actually a need for file access from upython and I had to glue fatfs to uPython's expectations of oofatfs. Fun times(still not sure which option was best, move everything to oofatfs or move everything fatfs ).
Just to confirm that you're running also the 1.9.3 version of mpy-cross, and that despite the differences in the files, both versions work fine?
Yes, correct, though I only have done limited testing with the PC based version so issues might still pop up. So far it looks promising though.
As already mentioned above I will have to test what happens if I make the file name part equal to cancel out any differences that might cause (will be for tomorrow).

User avatar
jimmo
Posts: 1204
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Difference in mpy output on target versus host

Post by jimmo » Tue Jan 07, 2020 3:58 am

I totally missed that you were setting src_name to empty, yes that would most likely explain what you described.

Things have changed a bit since 1.9 but it's important that mpy-cross runs with the same settings as what the device is using. In 1.12 this is things like -march and bytecode features -- see http://docs.micropython.org/en/latest/r ... files.html for details. Before 1.12 not all these settings were encoded into the .mpy file so you could end up accidentally loading an incompatible file.

BramPeeters
Posts: 54
Joined: Wed Jan 31, 2018 3:10 pm

Re: Difference in mpy output on target versus host

Post by BramPeeters » Tue Jan 07, 2020 3:10 pm

Sadly fixing the name difference does not make the result equal (the names are the same now, i put an empty filename in the host cross compiler.

Eg in the first bytes there are already differences (I have added image with birds eye view of differences)
mpy-cross: 4D 03 02 1F AF 16 28 02 00 00 00 00 83 65 34 00
on target : 4D 03 02 1F AE 3D 28 02 00 00 00 00 83 62 23 00

So i probably have different settings.
Using sys_mpy = sys.implementation.mpy as in the page you linked does not work because 1.9.3 does not yet have support for this attribute.

Any idea how to configure these things in the xcompiler ? I just spend some time trying to match the parameters in mpconfigport.h but
- Either i cant make them equal eg
#define MICROPY_EMIT_THUMB (1)
#define MICROPY_EMIT_INLINE_THUMB (1)
leads to
gccollect.c: In function ‘gc_collect’:
gccollect.c:147:5: error: implicit declaration of function ‘mp_unix_mark_exec’ [-Werror=implicit-function-declaration]
mp_unix_mark_exec();
^~~~~~~~~~~~~~~~~
(and it is not only undeclared, it does not exist in mpy-cross)

- Or it does not seem to make any difference (eg when i change MICROPY_FLOAT_IMPL_DOUBLE to MICROPY_FLOAT_IMPL_FLOAT as on my target )

So i am wondering if I am looking at the right place.
Attachments
Diff.jpg
Diff.jpg (81.39 KiB) Viewed 234 times

Post Reply