Building an IoT device: minizing module import time to conserve battery

[liudr]

I'm building an IoT device that takes data every few minutes and sleeps between taking data. The ESP32 deepsleep function causes the woken CPU to reset so I practically rerun the whole import process after each deepsleep.

I think micropython takes about 2 seconds to boot. That's the time I can't optimize away. But I wrote a number of modules to handle different sensors and upload data to server. This is where I'd like some advice minimizing import time.

Right now my main.py contains the main code and it imports several other files. I've heard of compiled code also frozen code. My ESP32 has SPIRAM so there is enough memory. I would like to minimize the load time and don't mind using more RAM in the process.

It's worth pointing out that my modules import other lower-level modules to handle communication so I don't know if (cross-)compiling or freezing those modules works because I don't have basics of how python bytecode. I could do some reading on that.

I've done some reading on freezing modules or bytecode:

https://mpython.readthedocs.io/en/maste ... ained.html

It seems that freezing means to include a module (source) or compiled bytecode inside the firmware partition, which means compiling it for ESP32. I do have ESP-IDF V4 but it's carefully set up for an important project and I don't wish to disturb it. Also having to compile the firmware isn't my best option anyway if I can find other ways to include my modules and keep MP firmware generic. I also looked at how MP port on ESP32 sets up partition tables. From what I see, the partition table is fixed at 1.5MB so I have to adjust that if by adding more content I go over 1.5MB.

So I guess I'm leaning towards using the micropython cross compiler to precompile my modules into byte code and copy into the flash file system. I think this saves time to compile when importing and should not be slower than frozen bytecode by much. I can also copy the flash content of a complete system (MP+files I saved) and clone my devices. So am I on the right track? Thanks.

[curt]

I have been testing pyminify to compress the python source. With the default options it can reduce the source by about 30%. I would be interested hearing anyone's good/bad experiences with this utility and micropython.

Curt

[liudr]

Thanks Curt. Could you provide a link to your tool? I'd like to try it. I'm thinking, if I strip all the comments, that should speed up load time a bit, right?

[curt]

Here is the pyminify doc:
https://dflook.github.io/python-minifier/

Makes sense the less file source you have to read the quicker the app starts up and the less space taken up on the file system.

Curt

[dhylands]

You can also pre-compile your python code to bitcode and store the .mpy files (i.e. bytecode files) and then it won’t have to convert the python to bytecode every time.

[liudr]

You can also pre-compile your python code to bitcode and store the .mpy files (i.e. bytecode files) and then it won’t have to convert the python to bytecode every time.

Is the .mpy file byte code the same as regular python bytecode?

If not, can I compile code with REPL? I'm hesitant to attempt to install the cross compiler due to the work that I don't understand. I've read regular python has compile() and experimented it on ESP32 MP. It works but I can't write the bytecode out. I wonder what option to use to convert the compiled code into bytearray.

[liudr]

So I found this python tool called mpy-cross that can cross compile and easy to install with pip.

https://pypi.org/project/mpy-cross/

So I installed on my PC and did the cross compile. Honestly if I compare .mpy loading speed with .py, there is a slight improvement. If I compare .mpy with minified .py, the improvement is even less noticeable. What I did instead was rearranging where I import things. Because connecting to wifi may take time to yield a valid IP and connecting to another MCU also takes some time to settle, I had time.sleep() for that. Now I rearranged my code to do those starting wifi and starting another MCU first, THEN I import a number of modules. So the time importing those modules dub as time to wait for IP address and other MCU to boot up. I think json reading config file may also be too time consuming so I might elect to just use eval() to read in config.

About really cross-compiling that yields ESP32 machine code, I'm still somewhat interested because that will be fastest. I tried giving machine architecture to the mpy-cross tool but the resultant .mpy is identical to without arch option. Sure, a simple pip-installable tool doesn't have the full ESP-IDF to compile anything! I wonder how gruesome it would be to set up this thing from micropython repo. I first must compile it under a nix system so the executable would run on my PC. Then I must tell it somehow where my ESP-IDF tools are. This sounds impossible to do. Is it?

[pythoncoder]

The @micropython.native decorator causes the compiler to emit machine code for a specific function. There is also an option for mpy-cross to emit machine code - try the -h option to list options.

In my own micro-power project (on a Pyboard) I found the most power efficient way to wake up was to store my code as frozen bytecode.

[liudr]

Thanks. I'll try the decorator but I'm still not sure how the pip-installed tool can compile an ESP-IDF project to get binary code for ESP32.

Is the arch parameter for ESP32 extensawin or extensa?

[scruss]

One other thing about mpy files: they have to match the version of the MicroPython interpreter you're running.