How does VM handle source code?

[Pingza]

Hi all

I am trying to wrap my head around the internal mechanism that the translation of source to machine code and can't seem to find a clear answer based on the fragments I have found online.

- Does the VM translate source line by line only i.e treat source like REPL?

- Can the VM compile the whole program up front?

- Does VM translate to an internal bytecode and then the VM executes the bytecode?

- If the answer is yes to bytecode then does the VM support JIT to actual machine code?

- Can the VM translate source to machine code directly I.e compile the python to run as a bare metal application?

Finally if all the source translation is happening on the micro doesn't this kill resources?

Thanks. Please advise if my questions aren't clear as I have only had one cup of coffee.

[jickster]

Hi all

I am trying to wrap my head around the internal mechanism that the translation of source to machine code and can't seem to find a clear answer based on the fragments I have found online.

- Does the VM translate source line by line only i.e treat source like REPL?

- Can the VM compile the whole program up front?

- Does VM translate to an internal bytecode and then the VM executes the bytecode?

- If the answer is yes to bytecode then does the VM support JIT to actual machine code?

- Can the VM translate source to machine code directly I.e compile the python to run as a bare metal application?

Finally if all the source translation is happening on the micro doesn't this kill resources?

Thanks. Please advise if my questions aren't clear as I have only had one cup of coffee.

First some clarification on terminology.

Personally, when I say VM I am referring to the functionality in vm.c which only executes the bytecode.
I see you're using "VM" to refer to the entire compilation+execution process which one could argue is correct . . . but be aware of this difference in terminology.

- Does the VM translate source line by line only i.e treat source like REPL?

Absolutely not. I also used to think you could execute a .py file by compiling+exec one line at a time but there's certain cases where that doesn't work.

See this https://stackoverflow.com/questions/464 ... atomically

- Can the VM compile the whole program up front?

That IS exactly what happens.

- Does VM translate to an internal bytecode and then the VM executes the bytecode?

Yes. See vm.c. Everything in ENTRY(...) is a bytecode.

- If the answer is yes to bytecode then does the VM support JIT to actual machine code?

No. All the code to be executed must be compiled before it is executed.

Python is not like Java in that Python is a scripting language where each identifier has no type.
In Java, "int bob" can only be assigned values that are "int"s or "int"-like.
In Python, "bob" can hold an "int" and in the very next line, it can hold a object-type.
Admittedly, I don't know why this makes a difference regarding JIT and compilation but I'm 90% sure it does.


- Can the VM translate source to machine code directly I.e compile the python to run as a bare metal application?

Kind of and in some cases; see
http://docs.micropython.org/en/v1.9.3/p ... de-emitter.

The 3 different code emitters
https://www.kickstarter.com/projects/21 ... sts/664832

If you limit your Python functions to use a subset of Python, the compiler will emit native instructions but they'll still have a header/footer that vm.c will use to execute the native code.
So if by "bare metal" you mean "without vm.c", then no.

Caveat. Technically, you could (probably?) take only the native instructions emitted and execute those yourself but then you'd have to do whatever setup vm.c does (if it does anything) to do the same thing.

Finally if all the source translation is happening on the micro doesn't this kill resources?

Yes it does pose an issue. If your .py is too big, the compilation step could fail due to not enough RAM. Also the compiled bytecode is in RAM.
However, there is another workflow.

There is an optional step in the image compilation that you can enable with a flag called FROZEN_MPY.
You can "compile" the .py to bytecode format on your PC - which then is placed in a .c file - and then include the resulting c-bytecode file during the build of the image.
After you flash the image, micropython can execute the bytecode from flash.

This eliminates the on-board compilation step (and associated RAM required) AND because the bytecode resides in flash, you again save RAM.

[jickster]

If this answers your question, prepend [SOLVED] to the title of your question.

[pythoncoder]

...However, there is another workflow...

There are two other options for dealing with large source files. One is to split the file into smaller modules, and the second is to cross-compile the file on a PC and copy the bytecode to the target. Cross-compiling avoids the need for building the firmware, so turnround can be quick, but as you point out frozen bytecode is the most efficient in that the bytecode is stored in flash.

Well, that is the case for platforms which can execute the compiled c-bytecode from flash. I gather some chips can only execute code in RAM (ESP8266?).