Remove imported module from RAM

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Remove imported module from RAM

Post by jickster » Fri Aug 24, 2018 2:48 am

kevinkk525 wrote:I'm having problems with unloading modules, the RAM does not get freed.

I'm using this function to delete a module:

Code: Select all

def unloadModule(mod):
    # removes module from the system
    mod_name = mod.__name__
    if mod_name in sys.modules:
        del sys.modules[mod_name]
I am importing a module and call one function. Then I delete it like that:

Code: Select all

from pysmartnode.helpers import helper
helper.help()
unloadModule(helper)
del helper
But the RAM is only a small amount higher than with the module imported, far away from the free RAM before I imported the module.
I’m not saying your approach is wrong or right but if it has any chance of working, you have to call gc.collect() are you call del

del by itself just removes an entry from the dict.

If you want the memory it used to point to to be freed, you have to call gc.collect()

However I’m not guaranteeing your approach will actually free the memory; gotta look at the C source code.
Last edited by jickster on Fri Aug 24, 2018 2:48 am, edited 1 time in total.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Aug 24, 2018 2:52 am

curlyz wrote:Yea , I dont think it is that simple guys .
Interned string of that module will still exist in RAM

import micropython
micropython.qstr_info(1)

There exist every string , the memory that didnt get freed is here.
How to delete interned string may you ask , working on it :))
Deleting the bytecode but not the qstrings for a module will still save you the space the bytecode occupies.

kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Fri Aug 24, 2018 5:36 pm

Thanks for looking at it. Of course I'm calling gc.collect() many times after deleting a module. Sadly I was never able to get more than a few Bytes back, even if the module was a few kB.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Aug 24, 2018 5:39 pm

kevinkk525 wrote:
Fri Aug 24, 2018 5:36 pm
Thanks for looking at it. Of course I'm calling gc.collect() many times after deleting a module. Sadly I was never able to get more than a few Bytes back, even if the module was a few kB.
Interesting. I'm gonna look further into this.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Fri Aug 24, 2018 6:00 pm

curlyz wrote:
Sun Aug 05, 2018 10:36 am
Yea , I dont think it is that simple guys .
Interned string of that module will still exist in RAM

import micropython
micropython.qstr_info(1)

There exist every string , the memory that didnt get freed is here.
How to delete interned string may you ask , working on it :))
Create a forum topic so we can follow your progress.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Remove imported module from RAM

Post by jickster » Sat Aug 25, 2018 1:42 pm

deshipu wrote:
markxr wrote: If we delete a module, then import it again later, does it leak? (I don't know the answer to this).

It seems reasonable to import a module, do some setup work (e.g. at reset), then when finished, delete the module to allow more ram for later usage.
the rest is allocated dynamically as you use it.
That is not true.

When you import a module, the bytecode for it is created in RAM immediately before you even use it.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Tue Aug 28, 2018 9:36 pm

kevinkk525 wrote:
Fri Jul 13, 2018 6:34 pm
I'm having problems with unloading modules, the RAM does not get freed.

I'm using this function to delete a module:

Code: Select all

def unloadModule(mod):
    # removes module from the system
    mod_name = mod.__name__
    if mod_name in sys.modules:
        del sys.modules[mod_name]
I am importing a module and call one function. Then I delete it like that:

Code: Select all

from pysmartnode.helpers import helper
helper.help()
unloadModule(helper)
del helper
But the RAM is only a small amount higher than with the module imported, far away from the free RAM before I imported the module.
That should work. Here's exactly what I did.

Created a useless module which allocates a bunch of memory

mod_test.py

Code: Select all

a = list(range(500))
and compiled it into the firmware as a frozen_str

In Python console

Code: Select all

import gc
import sys
# note the memory
gc.collect(); gc.mem_free()

import mod_test
# free memory dropped by a lot
gc.collect(); gc.mem_free()

del mod_test
# free memory barely changed
gc.collect(); gc.mem_free()

del sys.modules['mod_test']
# huge free memory increase
gc.collect(); gc.mem_free()
Order of sys.modules dict vs main dict doesn't matter.

This will not free qstrings. Currently qstrings are never freed.

This confirms that memory taken up by variables inside modules are deleted.
Next test would be to import a module that has a lot of CODE but initializes few variables.


This would test whether the code associated with a module name is correctly deleted.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Wed Aug 29, 2018 12:37 am

kevinkk525 wrote:
Fri Jul 13, 2018 6:34 pm
I'm having problems with unloading modules, the RAM does not get freed.

I'm using this function to delete a module:

Code: Select all

def unloadModule(mod):
    # removes module from the system
    mod_name = mod.__name__
    if mod_name in sys.modules:
        del sys.modules[mod_name]
I am importing a module and call one function. Then I delete it like that:

Code: Select all

from pysmartnode.helpers import helper
helper.help()
unloadModule(helper)
del helper
But the RAM is only a small amount higher than with the module imported, far away from the free RAM before I imported the module.
Don't know what to tell you cause it works for me.

I created mod_test.py so that the compiled code would occupy lots of RAM but there's no data that gets created upon import unlike my previous post

Code: Select all

def b():
	b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();b();	
	return 1

Then I executed the following on the console

Code: Select all

>>> import gc
Type "help()" for more information.

>>> import sys

>>> gc.collect(); gc.mem_free()
4800

>>> import mod_test

>>> gc.collect(); gc.mem_free()
4288

>>> del mod_test

>>> gc.collect(); gc.mem_free()
4400

>>> del sys.modules['mod_test']

>>> gc.collect(); gc.mem_free()
4752

>>>
You can see almost all is reclaimed.

So, I don't know why it doesn't work for you.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Remove imported module from RAM

Post by jickster » Wed Aug 29, 2018 4:09 pm

curlyz wrote:
Sun Aug 05, 2018 10:36 am
Yea , I dont think it is that simple guys .
Interned string of that module will still exist in RAM

import micropython
micropython.qstr_info(1)

There exist every string , the memory that didnt get freed is here.
How to delete interned string may you ask , working on it :))
Let's brainstorm here
https://github.com/micropython/micropython/issues/4053

kevinkk525
Posts: 876
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove imported module from RAM

Post by kevinkk525 » Tue Sep 18, 2018 7:39 pm

Thanks for looking into this. I don't know what changed, but now it looks like most of the RAM is freed after removing of modules.. bit weird.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Post Reply