The question is, can't I set the threshold to a value where every action leads to an "not enough heap" ? So that gc.collect() is always executed... Just as I add gc.collect() everywhere?jimmo wrote: ↑Mon Dec 23, 2019 12:35 amThis is related to the above. It changes the code that I described above where it will do a gc.collect() if it fails to find enough free heap, into preemptively collecting if the available heap drops below a threshold.
i.e. think of it as the default threshold is "100%".
get more free RAM...
Re: get more free RAM...
Re: get more free RAM...
I'm not quite sure I understand what you're asking?
You can set the threshold to zero, in which case _every_ allocation will do a collection first. (But you probably don't ever want this, it will be super slow! Remember GC performance scales with both the size of the working set, _and_ the total RAM).
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: get more free RAM...
@jimmo Thanks for the explanation. Like so many of these things, it's obvious. After someone has explained it to you
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: get more free RAM...
Thanks, then I got it right. That brings me to my conclusion:
Lower threshold value == less RAM fragmentation == slower, but maybe no memory errors.
In the meantime, I've run some tests: Threshold values between 500-1000 seem like a good compromise. At least on an ESP8266.
Re: get more free RAM...
I change my "minimal" uasyncio test server: https://github.com/jedie/micropython-so ... bserver.py
Now you can set the gc.threshold() on this test web server page. I also add the render time, so you can easy see with threshold value are going to be very slow and you see the RAM usage...
I would say this is a very good "real world" scenario. At least for my requirements
Here some results, i collected on a ESP8266 always submit the "GET test form":
threshold: =>16384 -> ~1.7KB free RAM, render times between 190-250ms
threshold: 8192 -> ~6,9KB free RAM, render times ~180-270ms
threshold: 4096 -> ~6,9KB free RAM, render times ~230ms
threshold: 2048 -> ~8,4KB free RAM, render times ~200-300ms
threshold: 1024 -> ~8,9KB free RAM, render times ~260-330ms
threshold: 512 -> ~9.0KB free RAM, render times ~270-370ms
threshold: 256 -> ~9.2KB free RAM, render times ~400ms
threshold: 128 -> ~9.2KB free RAM, render times ~500ms
threshold: 64 -> ~9.2KB free RAM, render times ~610ms
threshold: 32 -> ~9.1KB free RAM, render times ~880ms
threshold: 0 -> ~9.3KB free RAM, render times ~1350ms
The values vary widely. When I have more time, I will measure and average the whole thing with an external Python script.
Now you can set the gc.threshold() on this test web server page. I also add the render time, so you can easy see with threshold value are going to be very slow and you see the RAM usage...
I would say this is a very good "real world" scenario. At least for my requirements
Here some results, i collected on a ESP8266 always submit the "GET test form":
threshold: =>16384 -> ~1.7KB free RAM, render times between 190-250ms
threshold: 8192 -> ~6,9KB free RAM, render times ~180-270ms
threshold: 4096 -> ~6,9KB free RAM, render times ~230ms
threshold: 2048 -> ~8,4KB free RAM, render times ~200-300ms
threshold: 1024 -> ~8,9KB free RAM, render times ~260-330ms
threshold: 512 -> ~9.0KB free RAM, render times ~270-370ms
threshold: 256 -> ~9.2KB free RAM, render times ~400ms
threshold: 128 -> ~9.2KB free RAM, render times ~500ms
threshold: 64 -> ~9.2KB free RAM, render times ~610ms
threshold: 32 -> ~9.1KB free RAM, render times ~880ms
threshold: 0 -> ~9.3KB free RAM, render times ~1350ms
The values vary widely. When I have more time, I will measure and average the whole thing with an external Python script.
Re: get more free RAM...
That's really interesting! I'm not sure I quite expected the curve to look like that but it does make sense. Thanks for sharing
Re: get more free RAM...
On the other side: Lower values seems not to work good with real code, because reset via wdt...
I now have a memory information after a crash:
I now have a memory information after a crash:
Code: Select all
Watchdog.garbage_collection():
freed up 608 bytes -> 3088 bytes free
Traceback (most recent call last):
File "src/webswitch.py", line 108, in request_handler
File "src/webswitch.py", line 99, in send_response
File "src/webswitch.py", line 78, in call_module_func
File "src/http_set_timer.py", line 31, in get_form
File "src/webswitch.py", line 29, in send_html_page
File "src/device_name.py", line 34, in get_device_name
File "src/config_files.py", line 31, in <module>
MemoryError: memory allocation failed, allocating 80 bytes
redirect to: '/main/menu/' OK
stack: 3184 out of 8192
GC: total: 37952, used: 35632, free: 2320
No. of 1-blocks: 409, 2-blocks: 91, max blk sz: 264, max free sz: 49
GC memory layout; from 3ffeedd0:
00000: hMMhhhhDBBDhhhDhB=hB=BhMMSB=h====h===hh==h======================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: =================================================Dhhhh=hh====hhh
01400: Sh=====h===hh=======h=======hMDhh=======h=====h====h=hShSh=h====
01800: ===h================h=======Sh=ShBSh=======h==========DDhhShDhDh
01c00: =h====h=hh==h================h====h=======================h=====
02000: ==ShSh=Sh=Sh=Sh=Sh=ShSShSBhSDDBBST=h===h=hSh===========h========
02400: =======================h====================h====h==BBh======h==
02800: =====h=h====hMh===hDh===hhh==hSh==hh=======h=B=BBBBhBhBhhB=h===h
02c00: h=====h==h===Th===h==T=h=Mh===h===Dh===h=Dh=====Bh==hh====BhBBh=
03000: ============================DhM..B.MhDh==h===========BShh===h==S
03400: hhhMDhhh=======h===LMh=h===h==Sh=h===ShhBBBhDShShS.hSShh.h======
03800: =h=======h=SShh====h====h===h==================h=======h========
03c00: ===h=============h======hh======================================
04000: ==h================h=h=====ShShSh=====SMh===============hh=ShSMh
04400: =======DSh======TMDB.Dh=SSDBDBhBBDBBhDBDhBBh===Bh=BDh===DBhB=BBB
04800: Bhh===BB=hDhBBDh===hh========h=h===DB=hBDhBBh===Bh===BDhBh===hDh
04c00: hh===DDhhh===h=h===B=B=B=Shh==h===B=hh===h===hBh=h==============
05000: ====hDhh===h===BhDhhh=Sh===Bhhh=hh====h=Bh=B=h===h===h=h========
05400: ===============h===h=MhDST==h====h================h=h==h=====h==
05800: =h==....DhSh=T=hhhMh=======h==Shh==h=====.h===========h=Sh=hSShh
05c00: ==hh============hThSh=hSh======ShShSh=hh=h=DSh=======Dhhhhh==hhh
06000: =h=h==hhhh==h=hh=======h=hMDh=h=======Mhh======.hShSh..ShMDB.B..
06400: .h=Sh....hhSSST=h==h===Sh=============h===h=============h=SD.Shh
06800: =hhh==h=hSh=hSh=hhh====SSh=Sh=Sh=====h===hh=h=======h===Shh=h=Sh
06c00: ===hSh======h=h==================h=h........h=h=.h=============h
07000: ==h=SS.h=========h=h=h=h========hTh=======h====h=Shhh..h........
07400: h=============h==h=.B..h==h=.Sh==h====hSh====.hS.BBTh=======Th==
07800: ==Sh====...hh==hSTTTThhShh====h=Sh=h=.hhh===h=h====hhh==hS.hBh.h
07c00: ========h==============hBBBh==============Sh====================
08000: =h=======================BBB.D.....Bh================h==========
08400: =h=======........h========h==h===h==h===h==h====================
08800: =======h=h=B.......Bh====h===========h==..h===.Bh====BMDh.Bh====
08c00: ==h==Sh====h=============h===.S.h===..hTh=====h===h====h=.......
09000: .h====.................................................hShSB..h=
09400: ==..
Reset reason: 'MemoryError: memory allocation failed, allocating 80 bytes'
- MostlyHarmless
- Posts: 166
- Joined: Thu Nov 21, 2019 6:25 pm
- Location: Pennsylvania, USA
Re: get more free RAM...
Thinking about this again I reckon the really underutilized statement in an embedded environment would be del.
I would expect
to be even more efficient than
Thoughts?
I would expect
Code: Select all
del a
Code: Select all
a = None
Thoughts?
Re: get more free RAM...
Yes. For the issue described here, "del a" (where "a" is a function local) is equivalent, and will be marginally faster and be represented in less bytecode. (Internally it NULLs the element in the locals table.. i.e. it's a DELETE_FAST op compared to LOAD_CONST_NONE+STORE_FAST).MostlyHarmless wrote: ↑Mon Dec 23, 2019 9:22 pmThinking about this again I reckon the really underutilized statement in an embedded environment would be del.
Note that "del" doesn't immediately free any memory though as it doesn't know whether there are other references to it. So from that perspective it's identical to setting it to None.
But I agree - stylistically and because it's marginally more efficient, using "del a" would be preferable to "a = None" for this use case. (Although the C++ programmer in me hates using it because it's the opposite behavior of the "delete" operator).
Re: get more free RAM...
I have build my first firmware... and wow... i get many free RAM: