Remove imported module from RAM

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Tue Sep 18, 2018 7:39 pm

kevinkk525 wrote:
Tue Sep 18, 2018 7:39 pm
Thanks for looking into this. I don't know what changed, but now it looks like most of the RAM is freed after removing of modules.. bit weird.
Mark it as "[SOLVED]"

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Tue Sep 18, 2018 7:58 pm

Well, wanted to when I realized that it is behaving completely random and often even does not work correctly:

Code: Select all

>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
4272
>>> gc.collect();gc.mem_free()
4240
>>> import pysmartnode.testmodule
testmodule imported
hi there
>>> gc.collect();gc.mem_free()
3824
>>> gc.collect();gc.mem_free()
3824
>>> del pysmartnode.testmodule
>>> del sys.modules["pysmartnode.testmodule"]
>>> gc.collect();gc.mem_free()
4000
>>> gc.collect();gc.mem_free()
4000
>>> gc.collect();gc.mem_free()
3952
>>>
It's the same test that I did before, which worked at that time.
The testmodule just contains this:

Code: Select all

print("testmodule imported")

print("hi there")


class Lock:
    def __init__(self):
        self._locked = False

    async def __aenter__(self):
        while True:
            if self._locked:
                await asyncio.sleep_ms(_DEFAULT_MS)
            else:
                self._locked = True
                break

    async def __aexit__(self, *args):
        self._locked = False
        await asyncio.sleep_ms(_DEFAULT_MS)

    def locked(self):
        return self._locked


l = Lock()
It behaves completely random. At one point, it frees up everything except a couple Bytes, the next time it frees up everything except 130Bytes (and that's only a small module).
Oh and I can't mark it solved as it is not my thread (and now it is not completeley solved in my opinion, although I don't know what to do with this problem..)
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Tue Sep 18, 2018 8:08 pm

kevinkk525 wrote:
Tue Sep 18, 2018 7:58 pm
Well, wanted to when I realized that it is behaving completely random and often even does not work correctly:

Code: Select all

>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
4272
>>> gc.collect();gc.mem_free()
4240
>>> import pysmartnode.testmodule
testmodule imported
hi there
>>> gc.collect();gc.mem_free()
3824
>>> gc.collect();gc.mem_free()
3824
>>> del pysmartnode.testmodule
>>> del sys.modules["pysmartnode.testmodule"]
>>> gc.collect();gc.mem_free()
4000
>>> gc.collect();gc.mem_free()
4000
>>> gc.collect();gc.mem_free()
3952
>>>
It's the same test that I did before, which worked at that time.
The testmodule just contains this:

Code: Select all

print("testmodule imported")

print("hi there")


class Lock:
    def __init__(self):
        self._locked = False

    async def __aenter__(self):
        while True:
            if self._locked:
                await asyncio.sleep_ms(_DEFAULT_MS)
            else:
                self._locked = True
                break

    async def __aexit__(self, *args):
        self._locked = False
        await asyncio.sleep_ms(_DEFAULT_MS)

    def locked(self):
        return self._locked


l = Lock()
It behaves completely random. At one point, it frees up everything except a couple Bytes, the next time it frees up everything except 130Bytes (and that's only a small module).
Oh and I can't mark it solved as it is not my thread (and now it is not completeley solved in my opinion, although I don't know what to do with this problem..)
After import pysmartnode.testmodule, insert a dir()

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Tue Sep 18, 2018 8:46 pm

I tried it again on an empty setup (just in case) and it behaves similar.

Code: Select all

>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
22480
>>> import pysmartnode.testmodule
testmodule imported
hi there
>>> gc.collect();gc.mem_free()
22080
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main']
>>> gc.collect();gc.mem_free()
22048
>>> del pysmartnode.testmodule
>>> del sys.modules["pysmartnode.testmodule"]
>>> gc.collect();gc.mem_free()
22288
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main', 'a']
>>> del pysmartnode
>>> del sys.modules["pysmartnode"]
>>> gc.collect();gc.mem_free()
22320
>>> gc.collect();gc.mem_free()
22368
In your example you had only ~50Bytes not freed up, I have 100-200Bytes not freed. Multiple tests did not change that.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Tue Sep 18, 2018 8:54 pm

kevinkk525 wrote:
Tue Sep 18, 2018 8:46 pm
I tried it again on an empty setup (just in case) and it behaves similar.

Code: Select all

>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
22480
>>> import pysmartnode.testmodule
testmodule imported
hi there
>>> gc.collect();gc.mem_free()
22080
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main']
>>> gc.collect();gc.mem_free()
22048
>>> del pysmartnode.testmodule
>>> del sys.modules["pysmartnode.testmodule"]
>>> gc.collect();gc.mem_free()
22288
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main', 'a']
>>> del pysmartnode
>>> del sys.modules["pysmartnode"]
>>> gc.collect();gc.mem_free()
22320
>>> gc.collect();gc.mem_free()
22368
In your example you had only ~50Bytes not freed up, I have 100-200Bytes not freed. Multiple tests did not change that.
You're never gonna get 100% of the memory freed because when you run commands in REPL, like dir(), that will also allocate memory.
When you call dir(), a list is allocated and populated which uses heap.

And I see you've recognized that import a.b means you need to call del b and not del a.b (and same thing with del sys.modules["b"])

The only thing you can do to free memory occupied by mod is:

Code: Select all

del mod
del sys.modules["mod"]
gc.collect()
Things can get very tricky.

Code: Select all

import mod
a = mod
del mod
del sys.modules["mod"]
gc.collect()
You would not be deleting anything in this case because "a = mod" creates a reference to "mod"

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

[Solved] Re: Remove imported module from RAM

Post by kevinkk525 » Wed Sep 19, 2018 8:46 am

I understand that I don't get 100% back and especially in repl there can be allocations. Also I tried to not have any references left to the module I remove.
I tried to implement that strategy into my bigger programs and sometimes I end up getting almost no RAM back, which many of my smaller tests confirmed while others worked perfectly. Just tried it with a bigger (1kb) module and got everything back except 60Bytes. So it does theoretically work.

Guess I have to check it more carefully in my project, maybe I do have references I'm not aware of.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Sep 21, 2018 7:20 pm

I'm having troubles again.. Thanks for looking at it in advance.

Was trying to implement unloading modules into my project so I could run some functions and then unload that module to free up the RAM as I won't need that functions again. So I split one module into 2 in order to unload one after usage.
But apparently I'm completely unable to unload that module even when I only load this module directly.
In normal usage RAM even drops by almost 1kB. The manual tests reveals that RAM drops too, unless I remove the root package "pysmartnode" which is used by other loaded modules as well, so I can't juist delete the root package from sys.modules (I guess?).

Code: Select all

>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
22416
>>> sys.modules
{'pysmartnode': <module 'pysmartnode'>, 'pysmartnode.main': <module 'pysmartnode.main'>, 'main': <module 'main'>}
>>> gc.collect();gc.mem_free()
22368
>>> from pysmartnode.networking import mqtt_receive_config
>>> gc.collect();gc.mem_free()
18048
>>> gc.collect();gc.mem_free()
18048
>>> del mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> gc.collect();gc.mem_free()
17952
>>> gc.collect();gc.mem_free()
17952
>>> del sys.modules["pysmartnode.networking"]
>>> gc.collect();gc.mem_free()
17920
>>> del sys.modules["pysmartnode"]
>>> gc.collect();gc.mem_free()
18400
The content of pysmartnode.networking.mqtt_receive_config is this:

Code: Select all

import uasyncio as asyncio
import time
import json

_mqtt = None
_log = None

_awaiting_config = False
_has_failed = False
_has_succeeded = False
_config = None
_pyconfig = None


async def requestConfig(config, mqtt, log):
    global _log
    _log = log
    global _mqtt
    _mqtt = mqtt
    global _pyconfig
    _pyconfig = config
    if _awaiting_config:
        return
    asyncio.get_event_loop().create_task(_receiveConfig(log))
    while True:
        await asyncio.sleep_ms(200)
        if _has_failed:
            return False
        elif _has_succeeded:
            return _config


async def _awaitConfig(topic, msg, retain):
    _log.info("Building components", local_only=True)
    await _mqtt.unsubscribe("{!s}/login/".format(_mqtt.mqtt_home) + _mqtt.id)
    if type(msg) != dict:
        _log.critical("Received config is no dict")
        msg = None
    if msg is None:
        _log.error("Empty configuration received")
        global _has_failed
        _has_failed = True
        return
    else:
        _log.info("received config: {!s}".format(msg), local_only=True)
        # saving components
        _pyconfig.saveComponentsFile(msg)
        global _config
        _config = json.dumps(msg)
        global _has_succeeded
        _has_succeeded = True
        return


async def _receiveConfig(log):
    global _awaiting_config
    _awaiting_config = True
    while True:
        log.info("Receiving config", local_only=True)
        await _mqtt.subscribe("{!s}/login/{!s}".format(_mqtt.mqtt_home, _mqtt.id), _awaitConfig, qos=1,
                              check_retained=False)
        log.debug("waiting for config", local_only=True)
        await _mqtt.publish("{!s}/login/{!s}/set".format(_mqtt.mqtt_home, _mqtt.id), _pyconfig.VERSION, qos=1)
        t = time.ticks_ms()
        while (time.ticks_ms() - t) < 10000:
            if not _has_succeeded:
                await asyncio.sleep_ms(200)
            else:
                _awaiting_config = False
                return
        if _has_failed:
            _awaiting_config = False
            return
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Sep 21, 2018 7:24 pm

I don’t know what your question is.


Sent from my iPhone using Tapatalk Pro

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Sep 21, 2018 7:29 pm

my question is:

What am I doing wrong that I don't get any RAM back from this module?
(Unless I remove the root package "pysmartnode" but this is used by every other module too, so I don't know if I should remove that? But even doing so does not give me the RAM back when outside the manual test. I now splitted a module into 2 so I can unload one module but instead I end up with 1kB less free RAM than before when it was one module.)
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Sep 21, 2018 7:34 pm

kevinkk525 wrote:my question is:

What am I doing wrong that I don't get any RAM back from this module?
(Unless I remove the root package "pysmartnode" but this is used by every other module too, so I don't know if I should remove that? But even doing so does not give me the RAM back when outside the manual test. I now splitted a module into 2 so I can unload one module but instead I end up with 1kB less free RAM than before when it was one module.)

We went over this. Scroll up.

If you

import some.thing

you have to do

del thing
del sys.modules[“thing”]






Sent from my iPhone using Tapatalk Pro

Post Reply