RAM mystery

[cefn]

I have a puzzle, that whenever I import one of our Milecastles 'story' modules, it seems to take a huge amount of ram (something like 25kB).

I wonder if anyone can cast an eye over the source and spot any reasons that the string content wouldn't make it into program memory through use of Frozen modules.

All dependent libraries have already been loaded before loading the story, so the issue is RAM used on loading the story module itself.

Although there are certainly in-RAM structures constructed during the evaluation of each story module (there are 'story nodes' and they point to each other and to template text) pretty much all the values are strings, and I would expect them to remain in a frozen module not make it into RAM. I can't figure out how the memory used approximates to anything close to 25000 bytes. That is, unless it includes the template text, which would add up to almost exactly 25k.

However, that text should be frozen according to...
https://docs.micropython.org/en/latest/ ... ained.html
...and the fact that we build a frozen module filesystem, and don't even run VFS at all!

The number of nodes in our largest story is less than 100, so it seems each node seems to be using ~250 bytes, even though a typical node points to just 4 string members!

Incidentally, this text is not directly read at runtime in Micropython anyway, as it has already been handled as a Jinja2-style template and precompiled into generator-style templates also stored as Frozen modules, using a very cool process pioneered by @pfalcon for his utemplate. However, setting these unused template text values to None at runtime doesn't appear to reclaim the memory and the co-authoring of the text and the logic (story network) is really important for testing and consistency so we can't easily afford to manually go through maintaining a copy of story logic without template text.

Here's an example Story...
https://github.com/cefn/avatap/blob/mas ... ieHouse.py

...and stories are made up of classes from this module...
https://github.com/cefn/avatap/blob/mas ... castles.py

Any thoughts on how to reduce the RAM footprint without fundamentally changing the architecture would be valuable.

[deshipu]

Not sure I understand your explanations, but did I get it right that you are processing the template text at loading time? Because any string generated at run time will itself have to live in RAM.

[cefn]

In fact, the template loading and running is decoupled from loading the story structure. That's one reason RAM use is such a mystery. Loading of individual named templates from their frozen modules is triggered lazily by calculating their module named and loading on demand when a generator factory is needed for that template. The generators all use yield on static strings, without even assignment to named variables, using code from utemplate.

[pythoncoder]

I've not had time to study your code but it's likely that the problem results from unexpected copying. I did some work on storing fonts as frozen bytecode and initially had similar issues. In my case the solution was to include a memoryview object. An example may be seen here https://github.com/peterhinch/micropyth ... /font10.py but in essence it works as follows.

File dummy_font.py

Code: Select all

_font =\
b'\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'\  # and more ad nauseam

_mvfont = memoryview(_font)
def get_ch(ch):
    # do some maths
    return _mvfont[offset + 2:next_offs]  # and other data

The key point here is that using the memoryview to access the data avoids copying which otherwise seems to occur when bytes objects are accessed. I think this takes place because bytes and strings are immutable, which Python enforces by copying on access.

It's worth running some tests accessing your data and measuring RAM use before and after. It is definitely possible to access data such as this with very low incremental RAM usage.

[cefn]

@pythoncoder I think I eventually found a post from you which clarified that for ESP8266 frozen modules weren't actually executed from program memory, and therefore code needed to be loaded into RAM. If this is also true for QStrings, then that would fully explain all the RAM use, and means the difference between .mpy and frozen is minimal for ESP8266. Is this a settled conclusion?

[pythoncoder]

I read this statement somewhere, alas I forget where; unfortunately I passed it on without thinking it through. The element of truth is that the ESP8266 can't execute code from Flash but has to copy it to RAM.

Later it struck me that frozen Python bytecode produced by the normal emitter is not machine code. It is data which is interpreted by the Python VM, so surely shouldn't be subject to this architectural constraint.

A definitive answer to this one would be good. @Damien? @pfalcon?

Regarding qstrs have you read http://docs.micropython.org/en/latest/p ... ight=qstrs - the para "storing strings in flash"? I tried this some time ago on the Pyboard but haven't attempted to apply it to the ESP8266. Please report back any results

[pythoncoder]

@Roberthh has provided the answer here viewtopic.php?p=21271#p21271.