get more free RAM...

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: get more free RAM...

Post by kevinkk525 » Wed Dec 04, 2019 3:41 pm

jedie wrote:
Wed Dec 04, 2019 3:30 pm
kevinkk525 wrote:
Wed Dec 04, 2019 2:19 pm
OTA would still need you to build the firmware again, let the esp download it and restart. It's not really faster than just building the firmware and flashing it directly.
The speed is not relevant as long as it is not infinitely slow ;)

It's the practical aspect: My current device is a Sonoff WiFi Smart Socket. I would like to update it without having to unscrew it again to connect it to the computer ;)
An open device on the computer to develop is not a problem. But I would also like to update other, closed devices, too.
I understand that, however you were asking about development.
For deployment I do have the same problem, my sonoff devices control my heating units, can't just remove them either and unscrew them every time I want to update.. But got no solution to that problem either.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Dec 04, 2019 5:39 pm

kevinkk525 wrote:
Wed Dec 04, 2019 3:41 pm
For deployment I do have the same problem, my sonoff devices control my heating units, can't just remove them either and unscrew them every time I want to update.. But got no solution to that problem either.
Have you tried the OTA Firmware builds with with the yaota8266 ota-client script?
Last commit to yaota8266 is 3 years ago, so i ask me if this project is alive and create: https://github.com/pfalcon/yaota8266/issues/25

I'm successful on my project: Now i compile .mpy files and send them via my OTA client/server to the device. Commit: https://github.com/jedie/micropython-so ... 686772d812

The savings are small. Instead of 2-3KB I now have 3-4KB free. But the MemoryErrors are gone :D and the effort to get this running is very small.

I changed my OTA server: It now compiles on every run the .mpy files new.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: get more free RAM...

Post by kevinkk525 » Wed Dec 04, 2019 9:04 pm

I havent tried those OTA builds and since the last commit is long ago, I won't try it.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
MostlyHarmless
Posts: 166
Joined: Thu Nov 21, 2019 6:25 pm
Location: Pennsylvania, USA

Re: get more free RAM...

Post by MostlyHarmless » Wed Dec 04, 2019 9:55 pm

It will help to divide the code into "stable" and "volatile" modules. The stable parts would be deployed as frozen modules (hard to update), the volatile parts would be precompiled .mpy file (slightly easier to update).


Just a thought, Jan

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: get more free RAM...

Post by kevinkk525 » Thu Dec 05, 2019 6:19 am

The problem is that you also need some OTA mechanism, even if only by webrepl. This mechanism takes RAM as well...
With the recent addition of LittleFS on Esp8266 you can now get 4kB of RAM additionally if you format your filesystem with littlefs. This could make enough RAM available for such a mechanism and using a few more .mpy files.

But if someone is really worried about not being able to update a device often enough and has not enough RAM available, I'd recommend using the ESP32. It might be twice the cost of an esp8266 but is still very cheap and the higher RAM makes it easy to only use .mpy files and some OTA mechanism or ftp server.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Thu Dec 05, 2019 8:15 am

MostlyHarmless wrote:
Wed Dec 04, 2019 9:55 pm
It will help to divide the code into "stable" and "volatile" modules. The stable parts would be deployed as frozen modules (hard to update), the volatile parts would be precompiled .mpy file (slightly easier to update).
I'd thought about something like that, too. ;)
kevinkk525 wrote:
Thu Dec 05, 2019 6:19 am
The problem is that you also need some OTA mechanism, even if only by webrepl. This mechanism takes RAM as well...
My current solution will go into separate code branches in main.py between OTA code <-> web server code
See:

Code: Select all

if rtc.d.get(_RTC_KEY_RUN) == _RUN_WEB_SERVER:
    print('start webserver')
    rtc.save(data={_RTC_KEY_RUN: _RUN_OTA_UPDATE}) # next boot -> start OTA
    from webswitch import WebServer
    from watchdog import Watchdog
    from power_timer import AutomaticTimer

    gc.collect()
    WebServer(
        pins=pins, rtc=rtc,
        watchdog=Watchdog(wifi=wifi, rtc=rtc, auto_timer=AutomaticTimer(rtc=rtc, pins=pins)),
        version=__version__
    ).run()
else:
    print('start OTA')
    rtc.save(data={_RTC_KEY_RUN: _RUN_WEB_SERVER}) # next boot -> start web server
    from ota_client import OtaUpdate

    gc.collect()
    OtaUpdate().run()
That's why every code part has the full RAM available.

kevinkk525 wrote:
Thu Dec 05, 2019 6:19 am
With the recent addition of LittleFS on Esp8266 you can now get 4kB of RAM additionally if you format your filesystem with littlefs. This could make enough RAM available for such a mechanism and using a few more .mpy files.
Thanks, that's great. 4kB is a lot. It's worth it. I didn't know that littlefs existed yet.

kevinkk525 wrote:
Thu Dec 05, 2019 6:19 am
But if someone is really worried about not being able to update a device often enough and has not enough RAM available, I'd recommend using the ESP32. It might be twice the cost of an esp8266 but is still very cheap and the higher RAM makes it easy to only use .mpy files and some OTA mechanism or ftp server.
I have some ESP32 devices here. e.g. also the OdroidGo. But I don't have a useful job for it, except to experiment with micropython ;)

But it's more practical to have "ready-made" devices that can really do something. That's why I used the Sonoff WiFi Smart Socket.

I opened a separate thread. With the question which end-user devices have an ESP32: viewtopic.php?f=2&t=7352

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: get more free RAM...

Post by pythoncoder » Thu Dec 05, 2019 9:16 am

@jedie In my view frozen bytecode is the only way to get serious amounts of code running on an ESP8266. You therefore need to define your workflow accordingly.

Further to the stable/volatile model I would reserve one hardware unit purely for development. Then you only deploy to the working units when you have a new release you've thoroughly tested.
Peter Hinch
Index to my micropython libraries.

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Fri Dec 20, 2019 5:14 pm

I have made some tests... Think i have not a problem with less RAM. I have a problem with fragmentation :?

e.g.:

Code: Select all

freed up 1712 bytes -> 3168 bytes free
Set state from timer
Traceback (most recent call last):
  File "src/webswitch.py", line 1, in request_handler
  File "src/webswitch.py", line 1, in send_response
  File "src/webswitch.py", line 1, in call_module_func
  File "src/http_set_timer.py", line 1, in get_form
  File "src/power_timer.py", line 1, in update_power_timer
  File "src/power_timer.py", line 1, in get_info_text
  File "src/times_utils.py", line 1, in <module>
MemoryError: memory allocation failed, allocating 136 bytes
There are 3KB free, but could not allocate 136 bytes :evil:

There is no way to "defragmentation" the RAM, right?
gc.collect() will only destroy unused parts, but will not move/optimize the RAM, or?

bitninja
Posts: 165
Joined: Thu Sep 15, 2016 4:09 pm
Location: Spring, Texas

Learn How To Freeze Your Modules

Post by bitninja » Fri Dec 20, 2019 10:31 pm

Hi,

I just want to back the notion that freezing modules into the firmware is really a good idea. It's not that difficult and you gain so much.

https://www.youtube.com/watch?v=jG7WBY_vmpE

While it is really reserved for thoroughly debugged code, the edit-compile-upload cycle is not that convoluted once you have it setup.

Hope this helps.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: get more free RAM...

Post by jimmo » Sat Dec 21, 2019 1:40 am

jedie wrote:
Fri Dec 20, 2019 5:14 pm
There is no way to "defragmentation" the RAM, right?
gc.collect() will only destroy unused parts, but will not move/optimize the RAM, or?
That's right.

Does this happen when your program first starts or after some amount of time? If it's after some amount of time, then in many cases fragmentation is actually a symptom of a memory leak -- i.e. holding references to things that don't need to be held. If your program also generates a lot of short strings, then that can be a problem too (due to string interning).

you can use

Code: Select all

import micropython
micropython.mem_info(1)
to look at the heap layout and see what sort of objects are taking up space (if you want to copy that into a gist or pastebin or something we can take a look).

Related, there may be some code changes you can make to reduce working set size -- this is a simple example, but consider that

Code: Select all

a = ...something that allocates...
a = ...something else that allocates...

Code: Select all

a = ...something that allocates...
a = None
a = ...something else that allocates...
will have very different memory usage as the latter affords an opportunity to collect between the two.

Post Reply