get more free RAM...

[jedie]

While we're on the subject: What's about gc.threshold() ?

This is related to the above. It changes the code that I described above where it will do a gc.collect() if it fails to find enough free heap, into preemptively collecting if the available heap drops below a threshold.

i.e. think of it as the default threshold is "100%".

The question is, can't I set the threshold to a value where every action leads to an "not enough heap" ? So that gc.collect() is always executed... Just as I add gc.collect() everywhere?

[jimmo]

The question is, can't I set the threshold to a value where every action leads to an "not enough heap" ? So that gc.collect() is always executed... Just as I add gc.collect() everywhere?

I'm not quite sure I understand what you're asking?

You can set the threshold to zero, in which case _every_ allocation will do a collection first. (But you probably don't ever want this, it will be super slow! Remember GC performance scales with both the size of the working set, _and_ the total RAM).

[pythoncoder]

@jimmo Thanks for the explanation. Like so many of these things, it's obvious. After someone has explained it to you

[jedie]

You can set the threshold to zero, in which case _every_ allocation will do a collection first. (But you probably don't ever want this, it will be super slow! Remember GC performance scales with both the size of the working set, _and_ the total RAM).

Thanks, then I got it right. That brings me to my conclusion:

Lower threshold value == less RAM fragmentation == slower, but maybe no memory errors.

In the meantime, I've run some tests: Threshold values between 500-1000 seem like a good compromise. At least on an ESP8266.

[jedie]

I change my "minimal" uasyncio test server: https://github.com/jedie/micropython-so ... bserver.py

Now you can set the gc.threshold() on this test web server page. I also add the render time, so you can easy see with threshold value are going to be very slow and you see the RAM usage...

I would say this is a very good "real world" scenario. At least for my requirements

Here some results, i collected on a ESP8266 always submit the "GET test form":

threshold: =>16384 -> ~1.7KB free RAM, render times between 190-250ms
threshold: 8192 -> ~6,9KB free RAM, render times ~180-270ms
threshold: 4096 -> ~6,9KB free RAM, render times ~230ms
threshold: 2048 -> ~8,4KB free RAM, render times ~200-300ms
threshold: 1024 -> ~8,9KB free RAM, render times ~260-330ms
threshold: 512 -> ~9.0KB free RAM, render times ~270-370ms
threshold: 256 -> ~9.2KB free RAM, render times ~400ms
threshold: 128 -> ~9.2KB free RAM, render times ~500ms
threshold: 64 -> ~9.2KB free RAM, render times ~610ms
threshold: 32 -> ~9.1KB free RAM, render times ~880ms
threshold: 0 -> ~9.3KB free RAM, render times ~1350ms

The values vary widely. When I have more time, I will measure and average the whole thing with an external Python script.

[jimmo]

Here some results, i collected on a ESP8266 always submit the "GET test form":
...

That's really interesting! I'm not sure I quite expected the curve to look like that but it does make sense. Thanks for sharing

[jedie]

On the other side: Lower values seems not to work good with real code, because reset via wdt...

I now have a memory information after a crash:

Code: Select all

Watchdog.garbage_collection():
freed up 608 bytes -> 3088 bytes free
Traceback (most recent call last):
  File "src/webswitch.py", line 108, in request_handler
  File "src/webswitch.py", line 99, in send_response
  File "src/webswitch.py", line 78, in call_module_func
  File "src/http_set_timer.py", line 31, in get_form
  File "src/webswitch.py", line 29, in send_html_page
  File "src/device_name.py", line 34, in get_device_name
  File "src/config_files.py", line 31, in <module>
MemoryError: memory allocation failed, allocating 80 bytes
redirect to: '/main/menu/' OK
stack: 3184 out of 8192
GC: total: 37952, used: 35632, free: 2320
 No. of 1-blocks: 409, 2-blocks: 91, max blk sz: 264, max free sz: 49
GC memory layout; from 3ffeedd0:
00000: hMMhhhhDBBDhhhDhB=hB=BhMMSB=h====h===hh==h======================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: =================================================Dhhhh=hh====hhh
01400: Sh=====h===hh=======h=======hMDhh=======h=====h====h=hShSh=h====
01800: ===h================h=======Sh=ShBSh=======h==========DDhhShDhDh
01c00: =h====h=hh==h================h====h=======================h=====
02000: ==ShSh=Sh=Sh=Sh=Sh=ShSShSBhSDDBBST=h===h=hSh===========h========
02400: =======================h====================h====h==BBh======h==
02800: =====h=h====hMh===hDh===hhh==hSh==hh=======h=B=BBBBhBhBhhB=h===h
02c00: h=====h==h===Th===h==T=h=Mh===h===Dh===h=Dh=====Bh==hh====BhBBh=
03000: ============================DhM..B.MhDh==h===========BShh===h==S
03400: hhhMDhhh=======h===LMh=h===h==Sh=h===ShhBBBhDShShS.hSShh.h======
03800: =h=======h=SShh====h====h===h==================h=======h========
03c00: ===h=============h======hh======================================
04000: ==h================h=h=====ShShSh=====SMh===============hh=ShSMh
04400: =======DSh======TMDB.Dh=SSDBDBhBBDBBhDBDhBBh===Bh=BDh===DBhB=BBB
04800: Bhh===BB=hDhBBDh===hh========h=h===DB=hBDhBBh===Bh===BDhBh===hDh
04c00: hh===DDhhh===h=h===B=B=B=Shh==h===B=hh===h===hBh=h==============
05000: ====hDhh===h===BhDhhh=Sh===Bhhh=hh====h=Bh=B=h===h===h=h========
05400: ===============h===h=MhDST==h====h================h=h==h=====h==
05800: =h==....DhSh=T=hhhMh=======h==Shh==h=====.h===========h=Sh=hSShh
05c00: ==hh============hThSh=hSh======ShShSh=hh=h=DSh=======Dhhhhh==hhh
06000: =h=h==hhhh==h=hh=======h=hMDh=h=======Mhh======.hShSh..ShMDB.B..
06400: .h=Sh....hhSSST=h==h===Sh=============h===h=============h=SD.Shh
06800: =hhh==h=hSh=hSh=hhh====SSh=Sh=Sh=====h===hh=h=======h===Shh=h=Sh
06c00: ===hSh======h=h==================h=h........h=h=.h=============h
07000: ==h=SS.h=========h=h=h=h========hTh=======h====h=Shhh..h........
07400: h=============h==h=.B..h==h=.Sh==h====hSh====.hS.BBTh=======Th==
07800: ==Sh====...hh==hSTTTThhShh====h=Sh=h=.hhh===h=h====hhh==hS.hBh.h
07c00: ========h==============hBBBh==============Sh====================
08000: =h=======================BBB.D.....Bh================h==========
08400: =h=======........h========h==h===h==h===h==h====================
08800: =======h=h=B.......Bh====h===========h==..h===.Bh====BMDh.Bh====
08c00: ==h==Sh====h=============h===.S.h===..hTh=====h===h====h=.......
09000: .h====.................................................hShSB..h=
09400: ==..
Reset reason: 'MemoryError: memory allocation failed, allocating 80 bytes'

[MostlyHarmless]

Thinking about this again I reckon the really underutilized statement in an embedded environment would be del.

I would expect

Code: Select all

del a

to be even more efficient than

Code: Select all

a = None

Thoughts?

[jimmo]

Thinking about this again I reckon the really underutilized statement in an embedded environment would be del.

Yes. For the issue described here, "del a" (where "a" is a function local) is equivalent, and will be marginally faster and be represented in less bytecode. (Internally it NULLs the element in the locals table.. i.e. it's a DELETE_FAST op compared to LOAD_CONST_NONE+STORE_FAST).

Note that "del" doesn't immediately free any memory though as it doesn't know whether there are other references to it. So from that perspective it's identical to setting it to None.

But I agree - stylistically and because it's marginally more efficient, using "del a" would be preferable to "a = None" for this use case. (Although the C++ programmer in me hates using it because it's the opposite behavior of the "delete" operator).

[jedie]

I have build my first firmware... and wow... i get many free RAM:

screenshot.png (86.94 KiB) Viewed 5759 times