mp-cross and precompiled but not frozen scripts

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
BramPeeters
Posts: 12
Joined: Wed Jan 31, 2018 3:10 pm

mp-cross and precompiled but not frozen scripts

Post by BramPeeters » Thu Mar 01, 2018 1:42 pm

Hi,

I have a scenario where i would like to precompile a .py script on a host machine, send it over to a target and then execute the precompiled script.

Initially I was under the impression that this would be trivial but after reading the forums I found the following discussion
https://github.com/micropython/micropython/issues/2709
and now i am not so sure anymore :)

This discussion has the additional requirement though that the mpy script should be able to run from flash, if i understand it correctly it becomes easier if the mpy code can stay in ram ?

I probably do not understand the full extend of this discussion though, to start i have a few additional (probably noob, sorry) questions.

[0] Is this 'simpler' scenario already possible with 1.9.3 ? Or I am I going to run into the same problems as mentioned in the thread. For one I see QSTR's being mentioned, do i need to do something special for strings in the mpy or is this taken care of when calling mp_make_function_from_raw_code ?

[1] Looking at the mp_make_function_from_raw_code function, there are several kinds of 'precompiled code':
typedef enum {
MP_CODE_UNUSED,
MP_CODE_RESERVED,
MP_CODE_BYTECODE,
MP_CODE_NATIVE_PY,
MP_CODE_NATIVE_VIPER,
MP_CODE_NATIVE_ASM,
} mp_raw_code_kind_t;
What is native py ?

[2] How do the mp-cross options map to these types ?
Implementation specific options:
emit={bytecode,native,viper} -- set the default code emitter

[3] I tried precompiling a script with mp-cross using
./mpy-cross -X emit=bytecode frozentest.py
./mpy-cross -X emit=native frozentest.py
and I get 2 times the same output. How is that possible? Am i using the emit option wrong ?

[4] I was surprised there is even a native option. How does mpy-cross cross know what the native target format is ? It is build using "make -C mpy-cross" and you cannot supply the target's architecture via the command line. Am i misunderstanding what native means ( = opcodes directly executable by the target's mcu bypassing the VM ?)?

Online
User avatar
pythoncoder
Posts: 2804
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: mp-cross and precompiled but not frozen scripts

Post by pythoncoder » Thu Mar 01, 2018 5:46 pm

It is certainly possible - in fact easy - to cross-compile a Python script, copy the resultant mpy file to the target's filesystem and run it. In this instance it runs from RAM.

Regarding the code emitters these are discussed here and here and in the official docs.

As far as I'm aware you can't cross-compile to native code for the reason you give. For the same reason you can't cross compile code which includes native or viper decorators, nor can you freeze such code as bytecode.
Peter Hinch

BramPeeters
Posts: 12
Joined: Wed Jan 31, 2018 3:10 pm

Re: mp-cross and precompiled but not frozen scripts

Post by BramPeeters » Thu Mar 01, 2018 9:04 pm

Thanks for the links, i already found the official docs one but the others are more in depth.

I do notice however that if i use mp-cross to generate an mpy file and then load that file as 'byte array' (currently for testing i just compile the resulting mpy in as a comma separated byte array, in the end this should be transmitted via some protocol) and run it through the mp_make_function_from_raw_code, rc->kind goes into the MP_CODE_NATIVE_ASM case and not the MP_CODE_BYTECODE case....

If i look at the generated mpy file i get
0x4d, 0x03, 0x02, 0x1f, 0x5c, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0d,
0x34, 0x00, 0xf3, 0x00, 0x2a, 0x28, 0x28, 0x28, 0x2b, 0x28, 0x00, 0x00,
0xff, 0x1b, 0xc3, 0x00, 0x16, 0xf4, 0x00, 0x64, 0x01, 0x32, 0x1b, 0xc3,
0x00, 0x17, 0x00, 0x64, 0x01, 0x32, 0x1b, 0xc3, 0x00, 0x17, 0x01, 0x64,
<etc>

If i understand things correcly the _mp_raw_code_t is mapped directly on this.
With
typedef struct _mp_raw_code_t {
mp_raw_code_kind_t kind : 3;
mp_uint_t scope_flags : 7;
mp_uint_t n_pos_args : 11;

this means i need to look at the first 3 bits of the generated mpy bytes

So i have 4D 03 or 0100 1101 0000 0011 but now the question becomes where these 3 bits are located...
Since i end up in MP_CODE_NATIVE_ASM = 5 on my stm32 target i suppose he looks at 101.
If mp-cross is always supposed to generate bytecode = 2 then maybe he should look at the 010 ?
So i have some weird bit endianess problem ( I am compiling with IAR, not sure how well defined/compiler dependend C bit fields and endianess are)?
Or is he looking at the right bits and is the code generated by mp-cross not ok ?

Online
User avatar
pythoncoder
Posts: 2804
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: mp-cross and precompiled but not frozen scripts

Post by pythoncoder » Fri Mar 02, 2018 7:36 am

I haven't studied the file format so I can't offer anything there.

It might be worth stepping back and considering what can be done with existing utilities, which could be scripted to perhaps do what you have in mind.

Dave Hylands' rshell can copy a precompiled file to a target's filesystem. The official pyboard.py can remote-execute a MicroPython script on a Pyboard. It doesn't seem to work with .mpy files but you could use pyboard.py to transfer and execute a one line MicroPython script which imports the .mpy file.
Peter Hinch

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: mp-cross and precompiled but not frozen scripts

Post by jickster » Wed May 16, 2018 2:12 pm

pythoncoder wrote:
Thu Mar 01, 2018 5:46 pm

As far as I'm aware you can't cross-compile to native code for the reason you give.
What is that reason?

Online
User avatar
pythoncoder
Posts: 2804
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: mp-cross and precompiled but not frozen scripts

Post by pythoncoder » Wed May 16, 2018 4:09 pm

From the OP:
How does mpy-cross cross know what the native target format is ?
As far as I know there is no way to tell it the target machine code architecture.
Peter Hinch

Post Reply