Page 1 of 2

Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 7:02 pm
by oserror
My main.py is as follows:

Code: Select all

import gc

def do_beginning():
    import weather_station

    return weather_station.main()

def do_end(data):
    import send_and_finish
    
    send_and_finish.main(data)

data = do_beginning()
gc.collect()
do_end(data)
My problem is that weather_station.mpy does not get garbage collected. I don't know a lot about garbage collection, but I thought that since do_beginning() was finished, that weather_station.mpy would be released.

What is the best way forward here? I split up the weather station code into several .mpy modules that are called by functions like do_beginning() and do_end() (from within weather_station.mpy). I thought that once the imported module went out of scope that it would be gone from memory, or at least that it would mostly be gone.

If I can figure out how to free up the memory, this thing will be ready for production. Of course, I'm still waiting on a few parts from China, but that's another issue. :)

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 7:33 pm
by dhylands
When you import a module, a reference to the module is stored in sys.modules.

So you need to do something like:

Code: Select all

import sys
del sys.modules['weather_station']
before doing the gc.collect

I did the following:

Code: Select all

import gc
import sys

def do_beginning():
    gc.collect()
    print('mem_free before =', gc.mem_free())
    import weather_station
    x = weather_station.main()
    del sys.modules['weather_station']
    gc.collect()
    print('mem_free after  =', gc.mem_free())
    return x
    
data = None
data = do_beginning()
gc.collect()
print('mem_free after  =', gc.mem_free(), 'data allocated')
data = None
gc.collect()
print('mem_free after  =', gc.mem_free(), 'data freed')
and got these results:

Code: Select all

>>> import test_ws
mem_free before = 100912
waether_station.py
weather_station.py main called
mem_free after  = 100608
mem_free after  = 100784 data allocated
mem_free after  = 100912 data freed
>>>  

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 8:07 pm
by oserror
Thanks a lot, Dave. I went through and did a "del sys.modules['module_to_remove'] and did the "gc.collect()" as well but I was still unable to do a https:// PUT with urequests. I also had trouble using the ssl module when I hacked together a manual request.

It seems to me that I had some problems doing this last night from the prompt. Could it be the case that ssl might not work well with the esp8266? I did get it to work a few times but there were a few machine resets as well.

I like doing things with ssl but in this case it's not critical.

Thanks again for your help!

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 8:30 pm
by dhylands
Be aware that when you're doing things from the REPL, the REPL keeps a bunch of history, including results from the previous command. These won't get gc'd until they get removed from history.

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 9:33 pm
by oserror
Okay, I got it working. I am not 100% on why this works.

Code: Select all

for i, value in enumerate(result):
    res = urequests.put(url='%s/%s/update/V%d' % (blynk_host, blynk_auth, i), json=['%s' % value])
    if res.status_code != 200:
        print('Error updating pin V%d: %s' % (i, res.text))
    else:
        print('Updated pin V%d' % i)
    del res
    sleep(.1)
I guess it's making two objects if I get rid of the 'del res' line. I had some Computer Science in college but I didn't go the whole way through. Can you explain what happened here?

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 10:06 pm
by jimmo
oserror wrote:
Thu Jun 06, 2019 9:33 pm
I guess it's making two objects if I get rid of the 'del res' line.
There's a brief moment where there are two objects. Think of it like this.

Code: Select all

a = big_thing()
a = big_thing()
is really the same as

Code: Select all

tmp = big_thing()
a = tmp
del tmp
tmp = big_thing() # now we have a and tmp pointing to separate big things
a = tmp
del tmp
You never see tmp, but the important thing is that creating the new object happens before it replaces the old value in `a` so both objects will be briefly live. (But all it takes is one instant for you to need the memory).

What you've done with `del req` is the easiest way to solve this. The other thing you could do is turn the inside of the loop into a function, and let scoping sort this out for you too.

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 10:27 pm
by oserror
Thanks a lot. That makes sense.

I'll have to be careful when handling large variables/objects. I have been lazy working on modern-day machines where one can be sloppy.

I'm just glad I got the ssl working, albeit a tight squeeze! I'll see what else I can clean up in my code.

Re: Need help freeing memory from unused modules

Posted: Thu Jun 06, 2019 10:40 pm
by jimmo
If you're feeling adventurous, frozen modules are a good way to claim some RAM back, but you will need to build your own firmware images.

Re: Need help freeing memory from unused modules

Posted: Tue Jun 11, 2019 5:03 pm
by oserror
I worked on this for a few days. I managed to free up a couple of kilobytes of RAM by doing manual garbage collection and further breaking down things into separate modules. I also tried using urllib.urequest.urlopen() using ssl. My program runs every 10 minutes, doing a deep sleep between cycles. I was getting intermittent errors running 'urequests' with ssl, and lately it was doing that with every ssl request. urllib.urequest.urlopen() works consistently. I'm not sure if I really need ssl, as I said before, but I like to use it when I can.

What I could do is modify urllib.urequest.urlopen to do a json PUT request and freeze that module. It would be good for another process where I send data and it would be an exercise. That is one part of my code that is not likely to change.

The only reason I wouldn't freeze more modules is that I might have to modify the code in the future, and I'm not sure how easily I will be able to access the Wemos D1 Mini once I put everything in the case. I've been waiting for a couple of things to arrive on the slow boat from China and I just have the circuit on the breadboard for the time being.

When I have this running for a few days and I have the code very presentable, I'll publish the GitHub code. The project is based on an Instructable for a solar powered weather station. I ported the code and tried to make it more 'pythonistic'. Right now it sends data to Blynk, Thingspeak and to MQTT (to my Home Assistant instance).

Thank you dhylands and jimmo for your help.

Re: Need help freeing memory from unused modules

Posted: Tue Jun 11, 2019 5:31 pm
by dhylands
By setting up your sys.path appropriately you can freeze everything and override just the portions that need to be overridden.

sys.path contains a list of places to look for a module. The empty string is the place that it will look for frozen modules. So you can put a directory containing your overrides earlier in sys.path than the empty string and then your override module will get loaded instead of the empty one.

See: viewtopic.php?f=2&t=4475&p=26209#p26209 for a more detailed discussion along with an example.