RAM mystery

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Posts: 42
Joined: Tue Aug 09, 2016 10:58 am

RAM mystery

Postby cefn » Sun Jul 16, 2017 5:08 pm

I have a puzzle, that whenever I import one of our Milecastles 'story' modules, it seems to take a huge amount of ram (something like 25kB).

I wonder if anyone can cast an eye over the source and spot any reasons that the string content wouldn't make it into program memory through use of Frozen modules.

All dependent libraries have already been loaded before loading the story, so the issue is RAM used on loading the story module itself.

Although there are certainly in-RAM structures constructed during the evaluation of each story module (there are 'story nodes' and they point to each other and to template text) pretty much all the values are strings, and I would expect them to remain in a frozen module not make it into RAM. I can't figure out how the memory used approximates to anything close to 25000 bytes. That is, unless it includes the template text, which would add up to almost exactly 25k.

However, that text should be frozen according to...
https://docs.micropython.org/en/latest/ ... ained.html
...and the fact that we build a frozen module filesystem, and don't even run VFS at all!

The number of nodes in our largest story is less than 100, so it seems each node seems to be using ~250 bytes, even though a typical node points to just 4 string members!

Incidentally, this text is not directly read at runtime in Micropython anyway, as it has already been handled as a Jinja2-style template and precompiled into generator-style templates also stored as Frozen modules, using a very cool process pioneered by @pfalcon for his utemplate. However, setting these unused template text values to None at runtime doesn't appear to reclaim the memory and the co-authoring of the text and the logic (story network) is really important for testing and consistency so we can't easily afford to manually go through maintaining a copy of story logic without template text.

Here's an example Story...
https://github.com/cefn/avatap/blob/mas ... ieHouse.py

...and stories are made up of classes from this module...
https://github.com/cefn/avatap/blob/mas ... castles.py

Any thoughts on how to reduce the RAM footprint without fundamentally changing the architecture would be valuable.

User avatar
Posts: 1072
Joined: Thu May 28, 2015 5:54 pm

Re: RAM mystery

Postby deshipu » Mon Jul 17, 2017 7:06 am

Not sure I understand your explanations, but did I get it right that you are processing the template text at loading time? Because any string generated at run time will itself have to live in RAM.

Posts: 42
Joined: Tue Aug 09, 2016 10:58 am

Re: RAM mystery

Postby cefn » Mon Jul 17, 2017 7:15 am

In fact, the template loading and running is decoupled from loading the story structure. That's one reason RAM use is such a mystery. Loading of individual named templates from their frozen modules is triggered lazily by calculating their module named and loading on demand when a generator factory is needed for that template. The generators all use yield on static strings, without even assignment to named variables, using code from utemplate.

User avatar
Posts: 1635
Joined: Fri Jul 18, 2014 8:01 am

Re: RAM mystery

Postby pythoncoder » Mon Jul 17, 2017 7:50 am

I've not had time to study your code but it's likely that the problem results from unexpected copying. I did some work on storing fonts as frozen bytecode and initially had similar issues. In my case the solution was to include a memoryview object. An example may be seen here https://github.com/peterhinch/micropython-lcd160cr-gui/blob/master/font10.py but in essence it works as follows.

File dummy_font.py

Code: Select all

_font =\
b'\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'\  # and more ad nauseam

_mvfont = memoryview(_font)
def get_ch(ch):
    # do some maths
    return _mvfont[offset + 2:next_offs]  # and other data

The key point here is that using the memoryview to access the data avoids copying which otherwise seems to occur when bytes objects are accessed. I think this takes place because bytes and strings are immutable, which Python enforces by copying on access.

It's worth running some tests accessing your data and measuring RAM use before and after. It is definitely possible to access data such as this with very low incremental RAM usage.
Peter Hinch

Return to “ESP8266 boards”

Who is online

Users browsing this forum: No registered users and 1 guest