Remove imported module from RAM

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Sep 21, 2018 8:24 pm

ok I ran it:

Code: Select all

dir()
['uos', 'gc', '__name__', 'main']
>>> sys.modules
{'main': <module 'main'>}
>>> from pysmartnode.networking import mqtt_receive_config
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys', 'mqtt_receive_config']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking.mqtt_receive_config': <module 'pysmartnode.networking.mqtt_receive_config'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
>>> del mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
>>> del sys.modules["pysmartnode.networking"]
>>> del sys.modules["pysmartnode"]
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'main': <module 'main'>}
>>>
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Sep 21, 2018 8:26 pm

kevinkk525 wrote:
Fri Sep 21, 2018 8:24 pm
ok I ran it:

Code: Select all

dir()
['uos', 'gc', '__name__', 'main']
>>> sys.modules
{'main': <module 'main'>}
>>> from pysmartnode.networking import mqtt_receive_config
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys', 'mqtt_receive_config']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking.mqtt_receive_config': <module 'pysmartnode.networking.mqtt_receive_config'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
>>> del mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
>>> del sys.modules["pysmartnode.networking"]
>>> del sys.modules["pysmartnode"]
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'main': <module 'main'>}
>>>
Do it without

Code: Select all

del sys.modules["pysmartnode"]
We're trying to see if you can delete only submodules.

kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Sep 21, 2018 8:28 pm

Sorry that I added these lines below.. it was all there:

Code: Select all

dir()
['uos', 'gc', '__name__', 'main']
>>> sys.modules
{'main': <module 'main'>}
>>> from pysmartnode.networking import mqtt_receive_config
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys', 'mqtt_receive_config']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking.mqtt_receive_config': <module 'pysmartnode.networking.mqtt_receive_config'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
>>> del mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Sep 21, 2018 8:30 pm

kevinkk525 wrote:
Fri Sep 21, 2018 8:28 pm
Sorry that I added these lines below.. it was all there:

Code: Select all

dir()
['uos', 'gc', '__name__', 'main']
>>> sys.modules
{'main': <module 'main'>}
>>> from pysmartnode.networking import mqtt_receive_config
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys', 'mqtt_receive_config']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking.mqtt_receive_config': <module 'pysmartnode.networking.mqtt_receive_config'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
>>> del mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> dir()
['uos', '__name__', 'main', 'gc', 'sys']
>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'pysmartnode.networking': <module 'pysmartnode.networking'>, 'main': <module 'main'>, 'pysmartnode': <module 'pysmartnode'>}
Ok progress. So we are allowed to del submodules.

Question is why isn't the RAM freed.

Are all the modules that you import .py files (and not builtin C-code)?

kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Sep 21, 2018 8:32 pm

All files are frozen bytecode.
If you look at the thread I linked, I described that

Code: Select all

 import some.module
and then deleting some.module does work in contrast to

Code: Select all

from some import module
. Even that does not work in my program but that's another thing..
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Sep 21, 2018 8:51 pm

kevinkk525 wrote:
Fri Sep 21, 2018 8:32 pm
All files are frozen bytecode.
If you look at the thread I linked, I described that

Code: Select all

 import some.module
and then deleting some.module does work in contrast to

Code: Select all

from some import module
. Even that does not work in my program but that's another thing..
Wow. I just missed the forest for the trees.

Code: Select all

>>> sys.modules
{'uasyncio': <module 'uasyncio'>, 'uasyncio.core': <module 'uasyncio.core'>, 'main': <module 'main'>}
>>>
Look at those modules in memory!!!!

Those are present even after you delete pysmartnode.networking.mqtt_receive_config

kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Sep 21, 2018 8:58 pm

Of course they are, because I did not remove them after unloading the module. But they have nothing to do with no RAM getting freed at all when removing the module. But when importing like "import pysmartnode.networking.mqtt_receive_config", I'm able to unload the module and get some RAM back.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Sep 21, 2018 9:09 pm

kevinkk525 wrote:
Fri Sep 21, 2018 8:58 pm
Of course they are, because I did not remove them after unloading the module. But they have nothing to do with no RAM getting freed at all when removing the module.
Do you know how much RAM the code of pysmartnode.networking.mqtt_receive_config takes?
Do you know how much RAM the import statements in pysmartnode.networking.mqtt_receive_config take?

It's very possible the code of pysmartnode.networking.mqtt_receive_config takes almost no RAM but the import statements take a ton.

To deduce the relative ratio of RAM usage, need to create experiments on pysmartnode.networking.mqtt_receive_config
(A) comment out everything except the import statements.
Only the import statements will take RAM because there is literally no code!
(B) comment out the import statements - and the rest of your code since it won't work without the import statements - and put a dummy function that takes A LOT of RAM like this:

Code: Select all

def b():
    b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b(); return 0
Then rerun the sequence of commands for each case and reply with results for A and B.

kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Sat Sep 22, 2018 7:23 am

The code of pysmartnode.networking.mqtt_receive_config takes about 200Bytes. The import statements a few kB which is asyncio.
Of course I did not intend to get to the same amount of free RAM as before the import of the module as asyncio of course stays in RAM. My problem is, that no RAM gets freed if I unload the module. I should get at least 100Bytes back (which I do if I import it like "import pysmartnode.networking.mqtt_receive_config" which I described in the other thread. This of course is now more fitting to what I described in the other thread but we were not discussing that curiosity here before).

Your dummy function did work the last time when I placed it into a module in the root directory. If I put that function (and only that function) in the pysmartnode.networking.mqtt_receive_config, that same thing happens as before, no RAM gets free if I unload the module. Only once I unload the parent packages "pysmartnode.networking" and "pysmartnode", I get the RAM back.
More interestingly the other method importing "pysmartnode.networking.mqtt_receive_config", which did release the RAM of the module before, now completely refuses to release any RAM even after removing parent packages and pysmartnode stays in dir() which is weird:

Code: Select all

>>> import gc
>>> import sys
>>> gc.collect();gc.mem_free()
22464
>>> import pysmartnode.networking.mqtt_receive_config
>>> gc.collect();gc.mem_free()
22080
>>> del pysmartnode.networking.mqtt_receive_config
>>> del sys.modules["pysmartnode.networking.mqtt_receive_config"]
>>> gc.collect();gc.mem_free()
22032
>>> del sys.modules["pysmartnode.networking"]
>>> gc.collect();gc.mem_free()
22000
>>> del sys.modules["pysmartnode"]
>>> gc.collect();gc.mem_free()
22032
>>> sys.modules
{'main': <module 'main'>}
>>> dir()
['pysmartnode', 'gc', '__name__', 'uos', 'a', 'main', 'sys']
>>>
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Remove imported module from RAM

Post by jickster » Sun Sep 23, 2018 2:25 am

If you want to save ram you can use FROZEN_MPY.

That’s different than FROZEN_STR.

FROZEN_MPY pre-compiles to bytecode which doesn’t take up RAM when being executed.

Seems like this would solve your problem of RAM usage.

Sent from my iPhone using Tapatalk Pro

Post Reply