RAM mystery
Posted: Sun Jul 16, 2017 5:08 pm
I have a puzzle, that whenever I import one of our Milecastles 'story' modules, it seems to take a huge amount of ram (something like 25kB).
I wonder if anyone can cast an eye over the source and spot any reasons that the string content wouldn't make it into program memory through use of Frozen modules.
All dependent libraries have already been loaded before loading the story, so the issue is RAM used on loading the story module itself.
Although there are certainly in-RAM structures constructed during the evaluation of each story module (there are 'story nodes' and they point to each other and to template text) pretty much all the values are strings, and I would expect them to remain in a frozen module not make it into RAM. I can't figure out how the memory used approximates to anything close to 25000 bytes. That is, unless it includes the template text, which would add up to almost exactly 25k.
However, that text should be frozen according to...
https://docs.micropython.org/en/latest/ ... ained.html
...and the fact that we build a frozen module filesystem, and don't even run VFS at all!
The number of nodes in our largest story is less than 100, so it seems each node seems to be using ~250 bytes, even though a typical node points to just 4 string members!
Incidentally, this text is not directly read at runtime in Micropython anyway, as it has already been handled as a Jinja2-style template and precompiled into generator-style templates also stored as Frozen modules, using a very cool process pioneered by @pfalcon for his utemplate. However, setting these unused template text values to None at runtime doesn't appear to reclaim the memory and the co-authoring of the text and the logic (story network) is really important for testing and consistency so we can't easily afford to manually go through maintaining a copy of story logic without template text.
Here's an example Story...
https://github.com/cefn/avatap/blob/mas ... ieHouse.py
...and stories are made up of classes from this module...
https://github.com/cefn/avatap/blob/mas ... castles.py
Any thoughts on how to reduce the RAM footprint without fundamentally changing the architecture would be valuable.
I wonder if anyone can cast an eye over the source and spot any reasons that the string content wouldn't make it into program memory through use of Frozen modules.
All dependent libraries have already been loaded before loading the story, so the issue is RAM used on loading the story module itself.
Although there are certainly in-RAM structures constructed during the evaluation of each story module (there are 'story nodes' and they point to each other and to template text) pretty much all the values are strings, and I would expect them to remain in a frozen module not make it into RAM. I can't figure out how the memory used approximates to anything close to 25000 bytes. That is, unless it includes the template text, which would add up to almost exactly 25k.
However, that text should be frozen according to...
https://docs.micropython.org/en/latest/ ... ained.html
...and the fact that we build a frozen module filesystem, and don't even run VFS at all!
The number of nodes in our largest story is less than 100, so it seems each node seems to be using ~250 bytes, even though a typical node points to just 4 string members!
Incidentally, this text is not directly read at runtime in Micropython anyway, as it has already been handled as a Jinja2-style template and precompiled into generator-style templates also stored as Frozen modules, using a very cool process pioneered by @pfalcon for his utemplate. However, setting these unused template text values to None at runtime doesn't appear to reclaim the memory and the co-authoring of the text and the logic (story network) is really important for testing and consistency so we can't easily afford to manually go through maintaining a copy of story logic without template text.
Here's an example Story...
https://github.com/cefn/avatap/blob/mas ... ieHouse.py
...and stories are made up of classes from this module...
https://github.com/cefn/avatap/blob/mas ... castles.py
Any thoughts on how to reduce the RAM footprint without fundamentally changing the architecture would be valuable.