get more free RAM...

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

get more free RAM...

Post by jedie » Wed Dec 04, 2019 8:58 am

I always run into the limits of the small RAM on the ESP8266 in my project: https://github.com/jedie/micropython-sonoff-webswitch

I've already tried to split code parts, delete modules, etc. But all that doesn't help. Only 2-3KB Ram remain free and quickly there is a MemoryError.
jimmo wrote:
Sun Oct 06, 2019 2:32 am
...when you run Python code off the flash filesystem, it needs temporary RAM to load the text, then it stores the compiled bytecode in RAM. When you freeze a module, neither the text nor the compiled bytecode need to live in RAM, it's executed directly off the flash.

Compiling to an .mpy helps a bit because the program text doesn't need to be loaded into RAM, but it still means the bytecode runs from RAM.
If I understood it right, there are two possibilities: "Freeze modules" and "precompiling to .mpy", right?
Both seem quite complex to me. It's just something else than only saving the .py files on the device.

Why are .mpy files not created and stored in the flash on a device with file system? That would help a lot, wouldn't it?

Where can I find instructions for "Freeze modules" and "precompiling to .mpy" ? I only found: http://docs.micropython.org/en/latest/r ... h-freezing which leaves many questions unanswered for me.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: get more free RAM...

Post by jimmo » Wed Dec 04, 2019 9:27 am

jedie wrote:
Wed Dec 04, 2019 8:58 am
If I understood it right, there are two possibilities: "Freeze modules" and "precompiling to .mpy", right?
Both seem quite complex to me. It's just something else than only saving the .py files on the device.
Yes. Both are alternatives to saving the .py files on the device.

You can generate an .mpy from your .py using mpy-cross. Then you place the .mpy on the filesystem instead. But this will give you a very small RAM saving.

Freezing is the process of putting the .mpy into the actual firmware image, so that it can be loaded from ROM directly. But this is not straightforward because you need to compile your own firmware image.

So it sounds like what you want is freezing. The simplest way to test this out is to put your .py files into ports/esp8266/modules and then compile the firmware -- any .py files in this directory will be automatically converted to .mpy and compiled into the firmware. i.e. other than the step of copying the files into the modules directory (or you can symlink them too), the instructions you need are how to build the firmware for esp8266. (see ports/esp8266/README.md, although I'd recommend using docker rather than the instructions there -- see viewtopic.php?f=15&t=7136#p40603 )

(And a more detailed docker guide for ESP32 at viewtopic.php?f=3&t=7307 )
jedie wrote:
Wed Dec 04, 2019 8:58 am
Why are .mpy files not created and stored in the flash on a device with file system? That would help a lot, wouldn't it?
Think of a .mpy as a more compact representation of your .py file. Mostly it's useful to save flash space. I'm not sure this is something you'd want to happen automatically on the device.

The thing you actually want is to move the .mpy file out of the filesystem and into the firmware (which is what's happening during the freezing process) so that it can be loaded directly without copying into RAM. This is a feature that's been discussed, and it would be useful. It's just not trivial :)

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: get more free RAM...

Post by pythoncoder » Wed Dec 04, 2019 9:34 am

Freezing bytecode is by far the best way to run substantial modules on the ESP8266. I have a long running test of a fairly large (~600LOC) program which reports 21KiB of free RAM.

The first step is to prove you can build the firmware from source. Follow the instructions here to create a standard build, install it and ensure it runs. Then freezing code is a matter of following the instructions you linked above.
Peter Hinch
Index to my micropython libraries.

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Dec 04, 2019 11:16 am

I didn't see how I can easy develop if I must build and reflash the firmware for every code change to test the changes... OTA updates is also not possible for this, isn't it? Or can micropython manipulate the flash ROM by himself?

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: get more free RAM...

Post by kevinkk525 » Wed Dec 04, 2019 11:24 am

I do it that way all the time because my project is way too big to not freeze it.
However you could freeze everything into the firmware except the module you are currently working on. If it is small enough, you can still just copy the file and load it as .py

Another thing for development is using the unix port (if you are not testing hardware specific modules of course).
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Dec 04, 2019 11:29 am

Hm. Think OTA updates are really needed.
Whats about the WIP firmware and using yaota8266 in combination with own building/freezing?

Has anybody a working pipeline for this?

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: get more free RAM...

Post by kevinkk525 » Wed Dec 04, 2019 2:19 pm

OTA would still need you to build the firmware again, let the esp download it and restart. It's not really faster than just building the firmware and flashing it directly.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Dec 04, 2019 3:30 pm

kevinkk525 wrote:
Wed Dec 04, 2019 2:19 pm
OTA would still need you to build the firmware again, let the esp download it and restart. It's not really faster than just building the firmware and flashing it directly.
The speed is not relevant as long as it is not infinitely slow ;)

It's the practical aspect: My current device is a Sonoff WiFi Smart Socket. I would like to update it without having to unscrew it again to connect it to the computer ;)
An open device on the computer to develop is not a problem. But I would also like to update other, closed devices, too.

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Dec 04, 2019 3:31 pm

pythoncoder wrote:
Wed Dec 04, 2019 9:34 am
Freezing bytecode is by far the best way to run substantial modules on the ESP8266. I have a long running test of a fairly large (~600LOC) program which reports 21KiB of free RAM.
Wow i'm far away from 21KB :?

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Dec 04, 2019 3:34 pm

jimmo wrote:
Wed Dec 04, 2019 9:27 am
You can generate an .mpy from your .py using mpy-cross. Then you place the .mpy on the filesystem instead. But this will give you a very small RAM saving.
What magnitudes are we talking about here? Not relevant? Because I guess .mpy files, I can update with my existing OTA solution.
Freezing is a complete new workflow.

Post Reply