How to implement frozen bytecode?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
appels
Posts: 13
Joined: Thu Dec 03, 2020 3:14 pm

How to implement frozen bytecode?

Post by appels » Sat Jul 10, 2021 8:18 pm

Hi,

I've been looking for some information on how to implement frozen bytecode, and I found the following in the docs:
https://docs.micropython.org/en/latest/ ... h-freezing
Using frozen bytecode requires building the executable (firmware) for a given MicroPython port from the C source code. Consequently, the process is:

1. Follow the instructions for a particular port on setting up a toolchain and building the port. For example, for ESP8266 port, study instructions in ports/esp8266/README.md and follow them. Make sure you can build the port and deploy the resulting executable/firmware successfully before proceeding to the next steps.
2. Build MicroPython Unix port and make sure it is in your PATH and you can execute micropython.
3. Change to port’s directory (e.g. ports/esp8266/ for ESP8266).
4. Run make clean-frozen. This step cleans up any previous modules which were installed for freezing (consequently, you need to skip this step to add additional modules, instead of starting from scratch).
5. Run micropython -m upip install -p modules <packages>... to install packages you want to freeze.
6. Run make clean.
7. Run make.
Firstly I was wondering if this makes frozen bytcode or frozen modules? Because you don't use the cross-compiler to make a .mpy file, so I would assume this makes frozen modules? But if it does make frozen bytecode, how do you make frozen modules?

Then after some more research and looking around I found the following link to the github repo of Peter Hinch in a post on this forum.
https://github.com/peterhinch/micropyth ... -manifests
Which explains that as of October 2019 the method of specifying modules to be frozen has changed and now uses a manifest.py file.

Another post I saw on this forum confirmed this. It was an update by Damien George on the release of MicroPython 1.12 that said:
viewtopic.php?f=8&t=7430
A new mechanism to freeze scripts into firmware is provided - a frozen manifest - whereby scripts to freeze are listed in a Python file (eg manifest.py). All ports are updated to use this new feature.
So I tried to find some more information on this new method of freezing bytecode, but I couldn't find anything in the docs or in the Github repo of MicroPython that explained this process.

The only thing I found was the explanation in the github repo of Peter Hinch:
https://github.com/peterhinch/micropyth ... -manifests

So my questions:
-Should I use the method explained in the docs or should I use this manifest.py method?
-If I should use the manifest.py method, is there some more information about this?
Because I still don't understand it completely:
*Do you have to make this manifest.py yourself? Because I don't see it in the micropython/ports/stm32/boards/PYBV11 folder?
*Should I always write an include at the beginning in the manifest.py file or is it a special feature you use when you want to import
another manifest.py file, for some reason.
*Do you have to put the dollar sign before the path in the freeze() function? Because it is done like this in the first two freeze()
examples but not in the third.
*In the examples all the files used by the freeze() function are .py files. Can you also use .mpy files and gives this frozen bytecode?


Thanks in advance!

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: How to implement frozen bytecode?

Post by jimmo » Sun Jul 11, 2021 2:25 pm

appels wrote:
Sat Jul 10, 2021 8:18 pm
Firstly I was wondering if this makes frozen bytcode or frozen modules? Because you don't use the cross-compiler to make a .mpy file, so I would assume this makes frozen modules? But if it does make frozen bytecode, how do you make frozen modules?
Frozen just refers to the process of putting code in ROM.
This can either be the python text, or bytecode. However, by default, .py files will be automatically cross-compiled during the freezing process (the assumption being that you always want to freeze bytecode). .mpy files get frozen directly.

Typically it's best to let the freezing process do the cross-compiling for you because it will ensure the right architecture settings are passed to the cross compiler.

(You can use freeze_as_str("foo.py") to force it to freeze as text instead).

Although you can ask the manifest to freeze an .mpy file
appels wrote:
Sat Jul 10, 2021 8:18 pm
-Should I use the method explained in the docs or should I use this manifest.py method?
Definitely the manifest.py way.

We definitely need to write some documentation for this, but conceptually the idea is that if you're doing any sort of customisation to the build, the way you do that is by writing your own board definition (either from scratch, or using an existing one as a template).

Damien did just publish a repo with some example board definitions showing some different ways that boards can add their own frozen modules and user defined C modules. https://github.com/micropython/micropyt ... ple-boards
appels wrote:
Sat Jul 10, 2021 8:18 pm
*Do you have to make this manifest.py yourself? Because I don't see it in the micropython/ports/stm32/boards/PYBV11 folder?
If you want to customise anything, yes. The PYBV11 doesn't have its own manifest because it just uses the STM32 one (common to all STM32 boards) in ports/stm32/boards/manifest.py

The way this works is that the Makefile sets the path to FROZEN_MANIFEST (which on stm32 defaults to ports/stm32/boards/manifest.py). You can override this in the board's mpconfigboard.mk.
appels wrote:
Sat Jul 10, 2021 8:18 pm
*Should I always write an include at the beginning in the manifest.py file or is it a special feature you use when you want to import
another manifest.py file, for some reason.
It depends. If you're "extending" (or "inherting") from another manifest, then yes.

So for example, if someone made a stm32-based board with their own custom drivers written in Python, then they'd:
- Set FROZEN_MANIFEST ?= $(BOARD_DIR)/manifest.py
- Make their manifest.py do include("$(PORT_DIR)/boards/manifest.py") to get any port-generic features or drivers.

You can also use include to add a manifest for another library. (stm32/boards/manifest.py does this to get uasyncio)

For a good example of this, see the ports/esp32/TINYPICO board. (Although the ESP32 port now uses cmake, so maybe look at the 1.14 version if you want to do the STM32 equivalent -- https://github.com/micropython/micropyt ... anifest.py )

The example in Damien's new repo isn't quite right I think... https://github.com/micropython/micropyt ... TOM_PYBV11 because it doesn't inherit from the common stm32 manifest.
appels wrote:
Sat Jul 10, 2021 8:18 pm
Do you have to put the dollar sign before the path in the freeze() function? Because it is done like this in the first two freeze()
examples but not in the third.
The dollar sign is for the variable expansion -- e.g. $(PORT_DIR) or $(BOARD_DIR).

The third example is where you're providing the path directly (typically this would be an absolute path).
appels wrote:
Sat Jul 10, 2021 8:18 pm
*In the examples all the files used by the freeze() function are .py files. Can you also use .mpy files and gives this frozen bytecode?
See above -- the .py files will be cross compiled automatically, but you can give a .mpy directly if you want.

appels
Posts: 13
Joined: Thu Dec 03, 2020 3:14 pm

Re: How to implement frozen bytecode?

Post by appels » Mon Jul 12, 2021 4:02 pm

Thanks for the reply,

This is information is very helpfull.

Post Reply