Mark it as "[SOLVED]"kevinkk525 wrote: ↑Tue Sep 18, 2018 7:39 pmThanks for looking into this. I don't know what changed, but now it looks like most of the RAM is freed after removing of modules.. bit weird.
Remove imported module from RAM
Re: Remove imported module from RAM
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Remove imported module from RAM
Well, wanted to when I realized that it is behaving completely random and often even does not work correctly:
It's the same test that I did before, which worked at that time.
The testmodule just contains this:
It behaves completely random. At one point, it frees up everything except a couple Bytes, the next time it frees up everything except 130Bytes (and that's only a small module).
Oh and I can't mark it solved as it is not my thread (and now it is not completeley solved in my opinion, although I don't know what to do with this problem..)
Code: Select all
>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
4272
>>> gc.collect();gc.mem_free()
4240
>>> import pysmartnode.testmodule
testmodule imported
hi there
>>> gc.collect();gc.mem_free()
3824
>>> gc.collect();gc.mem_free()
3824
>>> del pysmartnode.testmodule
>>> del sys.modules["pysmartnode.testmodule"]
>>> gc.collect();gc.mem_free()
4000
>>> gc.collect();gc.mem_free()
4000
>>> gc.collect();gc.mem_free()
3952
>>>
The testmodule just contains this:
Code: Select all
print("testmodule imported")
print("hi there")
class Lock:
def __init__(self):
self._locked = False
async def __aenter__(self):
while True:
if self._locked:
await asyncio.sleep_ms(_DEFAULT_MS)
else:
self._locked = True
break
async def __aexit__(self, *args):
self._locked = False
await asyncio.sleep_ms(_DEFAULT_MS)
def locked(self):
return self._locked
l = Lock()
Oh and I can't mark it solved as it is not my thread (and now it is not completeley solved in my opinion, although I don't know what to do with this problem..)
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Re: Remove imported module from RAM
After import pysmartnode.testmodule, insert a dir()kevinkk525 wrote: ↑Tue Sep 18, 2018 7:58 pmWell, wanted to when I realized that it is behaving completely random and often even does not work correctly:
It's the same test that I did before, which worked at that time.Code: Select all
>>> import gc >>> import sys >>> gc.collect();gc.mem_free() 4272 >>> gc.collect();gc.mem_free() 4240 >>> import pysmartnode.testmodule testmodule imported hi there >>> gc.collect();gc.mem_free() 3824 >>> gc.collect();gc.mem_free() 3824 >>> del pysmartnode.testmodule >>> del sys.modules["pysmartnode.testmodule"] >>> gc.collect();gc.mem_free() 4000 >>> gc.collect();gc.mem_free() 4000 >>> gc.collect();gc.mem_free() 3952 >>>
The testmodule just contains this:
It behaves completely random. At one point, it frees up everything except a couple Bytes, the next time it frees up everything except 130Bytes (and that's only a small module).Code: Select all
print("testmodule imported") print("hi there") class Lock: def __init__(self): self._locked = False async def __aenter__(self): while True: if self._locked: await asyncio.sleep_ms(_DEFAULT_MS) else: self._locked = True break async def __aexit__(self, *args): self._locked = False await asyncio.sleep_ms(_DEFAULT_MS) def locked(self): return self._locked l = Lock()
Oh and I can't mark it solved as it is not my thread (and now it is not completeley solved in my opinion, although I don't know what to do with this problem..)
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Remove imported module from RAM
I tried it again on an empty setup (just in case) and it behaves similar.
In your example you had only ~50Bytes not freed up, I have 100-200Bytes not freed. Multiple tests did not change that.
Code: Select all
>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
22480
>>> import pysmartnode.testmodule
testmodule imported
hi there
>>> gc.collect();gc.mem_free()
22080
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main']
>>> gc.collect();gc.mem_free()
22048
>>> del pysmartnode.testmodule
>>> del sys.modules["pysmartnode.testmodule"]
>>> gc.collect();gc.mem_free()
22288
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main', 'a']
>>> del pysmartnode
>>> del sys.modules["pysmartnode"]
>>> gc.collect();gc.mem_free()
22320
>>> gc.collect();gc.mem_free()
22368
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Re: Remove imported module from RAM
You're never gonna get 100% of the memory freed because when you run commands in REPL, like dir(), that will also allocate memory.kevinkk525 wrote: ↑Tue Sep 18, 2018 8:46 pmI tried it again on an empty setup (just in case) and it behaves similar.
In your example you had only ~50Bytes not freed up, I have 100-200Bytes not freed. Multiple tests did not change that.Code: Select all
>>> import gc >>> import sys >>> gc.collect();gc.mem_free() 22480 >>> import pysmartnode.testmodule testmodule imported hi there >>> gc.collect();gc.mem_free() 22080 >>> dir() ['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main'] >>> gc.collect();gc.mem_free() 22048 >>> del pysmartnode.testmodule >>> del sys.modules["pysmartnode.testmodule"] >>> gc.collect();gc.mem_free() 22288 >>> dir() ['pysmartnode', 'gc', '__name__', 'uos', 'sys', 'main', 'a'] >>> del pysmartnode >>> del sys.modules["pysmartnode"] >>> gc.collect();gc.mem_free() 22320 >>> gc.collect();gc.mem_free() 22368
When you call dir(), a list is allocated and populated which uses heap.
And I see you've recognized that import a.b means you need to call del b and not del a.b (and same thing with del sys.modules["b"])
The only thing you can do to free memory occupied by mod is:
Code: Select all
del mod
del sys.modules["mod"]
gc.collect()
Code: Select all
import mod
a = mod
del mod
del sys.modules["mod"]
gc.collect()
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
[Solved] Re: Remove imported module from RAM
I understand that I don't get 100% back and especially in repl there can be allocations. Also I tried to not have any references left to the module I remove.
I tried to implement that strategy into my bigger programs and sometimes I end up getting almost no RAM back, which many of my smaller tests confirmed while others worked perfectly. Just tried it with a bigger (1kb) module and got everything back except 60Bytes. So it does theoretically work.
Guess I have to check it more carefully in my project, maybe I do have references I'm not aware of.
I tried to implement that strategy into my bigger programs and sometimes I end up getting almost no RAM back, which many of my smaller tests confirmed while others worked perfectly. Just tried it with a bigger (1kb) module and got everything back except 60Bytes. So it does theoretically work.
Guess I have to check it more carefully in my project, maybe I do have references I'm not aware of.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Remove imported module from RAM
I'm having troubles again.. Thanks for looking at it in advance.
Was trying to implement unloading modules into my project so I could run some functions and then unload that module to free up the RAM as I won't need that functions again. So I split one module into 2 in order to unload one after usage.
But apparently I'm completely unable to unload that module even when I only load this module directly.
In normal usage RAM even drops by almost 1kB. The manual tests reveals that RAM drops too, unless I remove the root package "pysmartnode" which is used by other loaded modules as well, so I can't juist delete the root package from sys.modules (I guess?).
The content of pysmartnode.networking.mqtt_receive_config is this:
Was trying to implement unloading modules into my project so I could run some functions and then unload that module to free up the RAM as I won't need that functions again. So I split one module into 2 in order to unload one after usage.
But apparently I'm completely unable to unload that module even when I only load this module directly.
In normal usage RAM even drops by almost 1kB. The manual tests reveals that RAM drops too, unless I remove the root package "pysmartnode" which is used by other loaded modules as well, so I can't juist delete the root package from sys.modules (I guess?).
Code: Select all
>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
22416
>>> sys.modules
{'pysmartnode': <module 'pysmartnode'>, 'pysmartnode.main': <module 'pysmartnode.main'>, 'main': <module 'main'>}
>>> gc.collect();gc.mem_free()
22368
>>> from pysmartnode.networking import mqtt_receive_config
>>> gc.collect();gc.mem_free()
18048
>>> gc.collect();gc.mem_free()
18048
>>> del mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> gc.collect();gc.mem_free()
17952
>>> gc.collect();gc.mem_free()
17952
>>> del sys.modules["pysmartnode.networking"]
>>> gc.collect();gc.mem_free()
17920
>>> del sys.modules["pysmartnode"]
>>> gc.collect();gc.mem_free()
18400
Code: Select all
import uasyncio as asyncio
import time
import json
_mqtt = None
_log = None
_awaiting_config = False
_has_failed = False
_has_succeeded = False
_config = None
_pyconfig = None
async def requestConfig(config, mqtt, log):
global _log
_log = log
global _mqtt
_mqtt = mqtt
global _pyconfig
_pyconfig = config
if _awaiting_config:
return
asyncio.get_event_loop().create_task(_receiveConfig(log))
while True:
await asyncio.sleep_ms(200)
if _has_failed:
return False
elif _has_succeeded:
return _config
async def _awaitConfig(topic, msg, retain):
_log.info("Building components", local_only=True)
await _mqtt.unsubscribe("{!s}/login/".format(_mqtt.mqtt_home) + _mqtt.id)
if type(msg) != dict:
_log.critical("Received config is no dict")
msg = None
if msg is None:
_log.error("Empty configuration received")
global _has_failed
_has_failed = True
return
else:
_log.info("received config: {!s}".format(msg), local_only=True)
# saving components
_pyconfig.saveComponentsFile(msg)
global _config
_config = json.dumps(msg)
global _has_succeeded
_has_succeeded = True
return
async def _receiveConfig(log):
global _awaiting_config
_awaiting_config = True
while True:
log.info("Receiving config", local_only=True)
await _mqtt.subscribe("{!s}/login/{!s}".format(_mqtt.mqtt_home, _mqtt.id), _awaitConfig, qos=1,
check_retained=False)
log.debug("waiting for config", local_only=True)
await _mqtt.publish("{!s}/login/{!s}/set".format(_mqtt.mqtt_home, _mqtt.id), _pyconfig.VERSION, qos=1)
t = time.ticks_ms()
while (time.ticks_ms() - t) < 10000:
if not _has_succeeded:
await asyncio.sleep_ms(200)
else:
_awaiting_config = False
return
if _has_failed:
_awaiting_config = False
return
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Re: Remove imported module from RAM
I don’t know what your question is.
Sent from my iPhone using Tapatalk Pro
Sent from my iPhone using Tapatalk Pro
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Remove imported module from RAM
my question is:
What am I doing wrong that I don't get any RAM back from this module?
(Unless I remove the root package "pysmartnode" but this is used by every other module too, so I don't know if I should remove that? But even doing so does not give me the RAM back when outside the manual test. I now splitted a module into 2 so I can unload one module but instead I end up with 1kB less free RAM than before when it was one module.)
What am I doing wrong that I don't get any RAM back from this module?
(Unless I remove the root package "pysmartnode" but this is used by every other module too, so I don't know if I should remove that? But even doing so does not give me the RAM back when outside the manual test. I now splitted a module into 2 so I can unload one module but instead I end up with 1kB less free RAM than before when it was one module.)
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Re: Remove imported module from RAM
kevinkk525 wrote:my question is:
What am I doing wrong that I don't get any RAM back from this module?
(Unless I remove the root package "pysmartnode" but this is used by every other module too, so I don't know if I should remove that? But even doing so does not give me the RAM back when outside the manual test. I now splitted a module into 2 so I can unload one module but instead I end up with 1kB less free RAM than before when it was one module.)
We went over this. Scroll up.
If you
import some.thing
you have to do
del thing
del sys.modules[“thing”]
Sent from my iPhone using Tapatalk Pro