Reclaiming last few flakes of RAM - need expert help

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Reclaiming last few flakes of RAM - need expert help

Post by cefn » Tue Apr 25, 2017 11:56 am

Very grateful for help from Micropython experts, if there are any firmware build tricks or code architecture tricks we have missed to minimise RAM usage.

It would be very sad to have to switch to a Pi this late in the game, as well as it being impossible to buy bulk Pi Zero boards for our project, (meaning a budget explosion of Pi Model A boards given the number of boxes we want as well as reopening a hella can of worms on power issues for battery-powered ones, and delaying our imminent install).


CANDIDATES FOR GAINING RAM

I speculate I can gain RAM from the following...

1) Shutting down functions/configurations not needed. E.g. can we save RAM by...
* ...disabling networking/Wifi? Our deployment doesn't use it. If so, what steps are needed to reclaim this RAM. I can't find any relevant options.
* ...dropping line-number or other debugging metadata? Once the boxes are running, debug is irrelevant. Is reclaiming RAM from unnecessary bytecode metadata possible, how?
* ...disabling the VFAT filesystem? Assuming I can use frozen _boot.py for startup I don't need it. Is MICROPY_FATFS = 0 in the Makefile the only change needed?

2) Re-architecting where I've misunderstood which operations use RAM.
* See also THINGS I AM ALREADY DOING and THINGS I BELIEVE below.

3) Other interventions I haven't even thought of
* Are there standard things which people drop to gain themselves an extra few bytes of RAM given the freedom to modify firmware image?


THINGS I AM ALREADY DOING

I have spent an extraordinarily long time optimising our RFID text adventure to minimise RAM use on a NodeMCUv2 running a dedicated Micropython firmware image built from Git. This image includes as many optimisations I can identify and I'm sure I've missed some.

At this point I'm so close to RAM limits that the final features I need to add will tip over the limit. I really hope I have missed some obvious solutions to reclaim RAM as we're so close. Currently we can load screen and RFID reader, write to the screen and run the text adventures from the console, but reading and writing RFID needs a bunch more RAM (the RFID cards are 1k, currently I have < 4k left with smaller text adventures and the larger text adventure stories we have are impossible to load).

I've had a good look at https://docs.micropython.org/en/latest/ ... ained.html and http://docs.micropython.org/en/v1.8.6/p ... ython.html and I think I've followed all of those guidelines as much as I can.

In my code I am already using...
* only Frozen Modules
* trying to import in dependency order, and running a gc.collect() after every import
* bytes objects and strings for data to avoid loading them into RAM e.g. for typography data in my bitfont library ( https://github.com/ShrimpingIt/bitfont )
* memoryview e.g. for SPI send buffer in my ST7920 library ( code example https://github.com/ShrimpingIt/micropyt ... /st7920.py)
* .format() as a substitute for concatenation of strings by plus operator
* incremental gc triggered when I suspect a large allocation has been made (e.g. on import)
* precompiled templates using https://github.com/pfalcon/utemplate to minimise eval at runtime

In my firmware build (based on fortnight-old micropython and esp_open_sdk git checkout) I'm already...
* disabling Berkeley Tree Database - in Makefile: MICROPY_PY_BTREE = 0
* increasing RAM limit to 40k - main.c: STATIC char heap[40 * 1024];
* increasing the size of irom - esp8266.ld: irom0_0_seg : org = 0x40209000, len = 0xb7000 AND in modesp.c: return MP_OBJ_NEW_SMALL_INT(0xc0000);
* populating known qstrings - qstrdefsport.h contains all dynamically loaded utemplate module names


THINGS I BELIEVE

* RAM used by the Stack (function-local variables) is garbage-collected immediately on return from a function
* Frozen modules use no RAM for the following, which remain in program memory ...
- class and function definitions
- literal strings and bytes
* RAM is allocated by a frozen module when...
- module is loaded (in-RAM metadata describing the module)
- constructing primitives or objects and storing to heap during execution of class and function definitions


THINGS I'M CONFUSED BY

Lots of RAM is used by loading stories like https://github.com/cefn/avatap/blob/mas ... enhouse.py even though they are frozen modules, all recursively-imported modules were already imported, and the stories are overwhelmingly composed of strings that I would expect to remain in ROM. You can see this in the reference code loading results below, where the step of loading the senhouse story permanently uses up 8kB of RAM!

I don't know the cost of the dirty hack of increasing RAM described at https://gerfficient.com/2016/10/19/fix- ... n-esp8266/ There must be some reason this is not the default configuration.

I found that spi sends triggered by st7920.Screen#redraw() actually DO reduce available RAM. I can't figure why (each write is backed by a locally-scoped sub-memoryview presumably disposed at the end of the function call, mapped to a pre-allocated command buffer bytearray). See example code at https://github.com/ShrimpingIt/micropyt ... /st7920.py and the memory_test reference code described below.


SUMMARY

Any thoughts on tricks and tips I have missed would be like gold-dust at this stage of proceedings. I really appreciate any effort people can make to scan our approach for any obvious issues we can address. Thanks all!


REFERENCE CODE

A reference example for the routine we are running is at https://github.com/cefn/avatap/blob/mas ... ry_test.py

The

Code: Select all

report_import
routine from https://github.com/cefn/avatap/blob/mas ... gnostic.py imports a module, and outputs data about post-load garbage collection.

This results in the following log from a clean boot of our firmware image, the report lines are understood as {RAM free after import}+{amount reclaimed by gc}={total RAM still available} ...

Code: Select all

>>> from regimes.memory_test import *
faces.font_5x7
26624+288=26880
st7920
25984+272=26224
machine
26160+80=26208
15856+7808=23632
milecastles
18864+736=19568
boilerplate
18720+176=18864
engine
17968+224=18160
consoleEngine
17456+208=17632
stories.senhouse
9616+48=9632
7056+2016=9040
mfrc522
8240+112=8320
Subsequently loading of up to 50 template generation modules progressively reduce the available RAM until it stabilises around 4kB. Unfortunately, we will need just a bit more to do reading/writing of RFID tags, and writing to the screen.

Immediately after loading, the memory map looks like this...

Code: Select all

>>> micropython.mem_info(1)
stack: 2144 out of 8192
GC: total: 40320, used: 32592, free: 7728
 No. of 1-blocks: 569, 2-blocks: 47, max blk sz: 370, max free sz: 338
GC memory layout; from 3ffef2e0:
00000: MDhhhBMhTDMDhDhB=BBBh===h====h=hhh==h===========================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: ============================================BAMhDhBh=Bh==hh=hh==
01400: =====h==========================================================
01800: ================================================================
01c00: ================================================================
02000: ================================================================
02400: ================================================================
02800: =======================================================TDh=hBT=M
02c00: DhMMBBDMDhBBBAMDShh==DAhDhh=h========h=Bh=BBh===B==BBBBBMB=LB=DD
03000: hhBBBDh=BBBh==================BBBB=B=B=B=BhB=hB=h=====h===Lhh=Bh
03400: =B=hh=========================h=======================h===hhh===
03800: =====MDSMDBhTB=BBBBhhBhhDhhBhhhBhhBhhBhhBhhBhDhhhhhBhhBhhhhhBhhD
03c00: hhhhhBhhBhhhhhhMhhDhhDhhBhhDhhBhhBhhBhhBhhDhhDhhDhhhhhh=========
04000: ======hDhhhhhBhhhhhBhhBhhBhhBhhBhhBhhBhhBhhBhhBhhDhhBhhBhhDhhhhh
04400: BhhhhhBhhBhhDhhhhhLhhhhhBhhh=hh===h===h=h===h====h==h=.h===h===B
04800: Bh==h===h===h=h==h===h===h===h==h===h========BDBBhSLhh=BBBBBBBh=
04c00: ==BDBh=====Bh=Bh===BDBh====BBh===BDhBBhBBDh===h===h=============
05000: =h===h==h===hDBBhLhBB=hh===h===DBh=Dh===hDhBhBhBBhBBhh====h===B=
05400: Bh===h===h========h===MMDhDDBhBh===hDhhDh==h========B=h===h=====
05800: ===hhhDhh===hBh=h===h=====h========h=h=.hDShDh==hDhhhDhhhDhhhhhD
05c00: DhDShh..BhhBBBhDhhDhDhDhhhhBBBBB.Bh==B=hDBLBBBBBh=Dhh.Sh=h==LLhh
06000: LhLhh==hLh====LhhhLLLLLh=LhLhh===LhLhhLLh====LLh.hSh===h===h=h==
06400: ======hLhhh=Lh==hLhLhhLhh===hhh=hh==h====================h==LhLL
06800: hhhLhh====h===hh.hhLhhhh===LLDh===================hhh==Lhh=h===h
06c00: ===LLhh==h====h===hDLh===h===.LhLh===hh====h.h===DhDhh.hh====h==
07000: ================.LhDhhD.hh=Sh===h====h====Lhh===h==h===h===..Lhh
07400: ===.h...hh==================h===...h=================h===h===h==
07800: =.h==h====h====....D...Dhh.hD..hh===.....Dhh..Lh..Dhhh=h......h.
07c00: ....Dh=T...........h==Dhhh==================hh....hh==h=========
08000: =====......Sh===========...............h=h==============........
08400: ...............h=================.........h==...................
08800: .....h..........................................................
       (4 lines all free)
09c00: ........................

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Reclaiming last few flakes of RAM - need expert help

Post by cefn » Tue Apr 25, 2017 3:26 pm

OK, from rummaging I have now tried disabling MICROPY_PY_USSL and MICROPY_SSL_AXTLS within the micropython/esp8266/Makefile before building with frozen modules and sneakily tried to increase the STATIC char heap in main.c to 48 x 1024 which so far has not broken anything. Before the SSL/TLS build change, trying to get a 48k heap created a crash on load.

Can there be something going badly wrong in invisible ways which could hurt later? How do I know how much I can afford to allocate to STATIC char heap after making changes elsewhere and what other productive changes would gain me some RAM (assuming I just need standalone micropython with frozen modules and SPI?

Is it a YMMV issue where it depends what transactions the other non-micropython code has to handle, whether its reduced available memory would trigger a crash?

Given this build change, with a 'short' text adventure story, I now have approx 16000 bytes to play with and I've successfully allocated bytearrays almost all the way to the ceiling without crash. I'll have to see whether the screen writing/RFIDread+write and the larger stories cause me to butt against the ceiling again.

I would still benefit hugely from any other suggestions what build flags to use or other configuration changes to configure the NodeMCUv2 as much as possible as a standalone micropython IO engine.

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Reclaiming last few flakes of RAM - need expert help

Post by cefn » Tue Apr 25, 2017 4:48 pm

OK, I've now succeeded at gaining even more RAM.

The trick was to modify boot.py as follows - note everything to do with flash and filesystem is commented out on load.

Code: Select all

import gc
gc.threshold((gc.mem_free() + gc.mem_alloc()) // 4)
import uos
"""
from flashbdev import bdev

try:
    if bdev:
        vfs = uos.VfsFat(bdev)
        uos.mount(vfs, '/flash')
        uos.chdir('/flash')
except OSError:
    import inisetup
    vfs = inisetup.setup()
"""
gc.collect()
This results in the following on boot...

Code: Select all

>>> import gc
>>> gc.collect()
>>> gc.mem_free()
48080
...and after loading all the text adventure logic, screen driver, RFID driver etc, it gives me 21392 bytes available, which I hope will give me the headroom I need to run everything on NodeMCUv2 with Micropython.

I still welcome other ideas how to turn off functions and save power or more importantly RAM.

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Reclaiming last few flakes of RAM - need expert help

Post by cefn » Wed Apr 26, 2017 6:06 am

I have committed a rollup of the changes I made into a fork in our micropython repo https://github.com/micropython/micropyt ... :memorymax

I attempted these changes to increase availability of storage for frozen modules and for availability of RAM as part of our Avatap project https://github.com/cefn/avatap.

Note, this would not be for everyone - they disable the VFAT filesystem (preventing writes of e.g. a main.py file, requiring the _boot.py frozen module to be edited for startup behaviours). They mess with various things in ways which are specific to our project, but may offer a handy reference for the bits of the firmware build that can be messed with to gain some advantages, but YMMV.

Also take account there is well-informed scepticism from one of the micropython experts that the Makefile build flags I changed have any impact on RAM at all...
https://github.com/micropython/micropyt ... -297091292
...but they seem to do no harm in our case...

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: Reclaiming last few flakes of RAM - need expert help

Post by manseekingknowledge » Mon Aug 26, 2019 6:23 am

cefn wrote:
Wed Apr 26, 2017 6:06 am
I have committed a rollup of the changes I made into a fork in our micropython repo https://github.com/micropython/micropyt ... :memorymax

I attempted these changes to increase availability of storage for frozen modules and for availability of RAM as part of our Avatap project https://github.com/cefn/avatap.

Note, this would not be for everyone - they disable the VFAT filesystem (preventing writes of e.g. a main.py file, requiring the _boot.py frozen module to be edited for startup behaviours). They mess with various things in ways which are specific to our project, but may offer a handy reference for the bits of the firmware build that can be messed with to gain some advantages, but YMMV.

Also take account there is well-informed scepticism from one of the micropython experts that the Makefile build flags I changed have any impact on RAM at all...
https://github.com/micropython/micropyt ... -297091292
...but they seem to do no harm in our case...
Having performed some research myself I have to concur with the "micropython experts" you mentioned. Please read @jimmo's post here for instructions on how to determine if disabling a feature actually reclaimed any RAM or not.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Reclaiming last few flakes of RAM - need expert help

Post by kevinkk525 » Mon Aug 26, 2019 2:21 pm

You are answering a 2 year old post...
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: Reclaiming last few flakes of RAM - need expert help

Post by manseekingknowledge » Mon Aug 26, 2019 3:10 pm

kevinkk525 wrote:
Mon Aug 26, 2019 2:21 pm
You are answering a 2 year old post...
Yup. Just because a thread hasn't been active for 2 years doesn't mean people don't read it. This particular thread has misleading information, and my reply directs people to the correct information.

Post Reply