Garbage Collection Not Working

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Loxy618
Posts: 21
Joined: Wed Apr 24, 2019 2:01 am

Garbage Collection Not Working

Post by Loxy618 » Mon Jan 20, 2020 9:50 pm

Hello,
I'm running a simple script.

a=[1]*1000

The heap is getting used for the allocation (32K heap). At the end of the script I'm looking at the heap and see the memory is still allocated. I can run the same script infinite more times without any additional memory being allocated on the heap. It seems like the script isn't going out of scope such that the gc would free the used memory on the heap. Maybe I am not understanding how the gc works on MicroPython, I'm new at this. Any additional tips/examples would be appreciated.

Thanks

User avatar
tve
Posts: 216
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: Garbage Collection Not Working

Post by tve » Mon Jan 20, 2020 9:54 pm

Maybe a complete transcript would be helpful...
If you're creating a global 'a' at the repl, how do you expect it to go out of scope?

Loxy618
Posts: 21
Joined: Wed Apr 24, 2019 2:01 am

Re: Garbage Collection Not Working

Post by Loxy618 » Mon Jan 20, 2020 10:07 pm

So I'm probably not using MP in the most common way. I'm running it on a M7 that normally will run a basic while(1) with c-code. We have scripts that can be remotely added and stored in flash. There are some events that occur that may require a script to be executed. When these events occur, we will kick off MP, initialize it (if it hasn't been previously) and then execute the script. Then its still idle until another event occurs that has to run another script. If the script needs to use the heap, the memory is not freed after the script is done running.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Garbage Collection Not Working

Post by stijn » Tue Jan 21, 2020 9:14 am

Loxy618 wrote:
Mon Jan 20, 2020 10:07 pm
execute the script. Then its still idle
I assume this means you use execute_from_lexer or similar? Would really help to see actual code here, or at least some pseudocode showing what goes on (for starters: where does the GC call occur?). But in any case it's most likely as tve says: assigning a variable and then forcing GC won't do anything because the variable is still in the scope. But even if it weren't, bigger question is: what do you really want to know? Do you have a problem running out of memory and/or do you just want to study how the GC works? Normally you shouldn't have to care about what the GC is doing. It's tried and tested so you can assume it functions properly despite being a bit of a black box. Or, if you want to learn what it does, have a look at gc.c and the likes: define MICROPY_DEBUG_VERBOSE and set EXTENSIVE_HEAP_PROFILING to 1 and look at the generated output.

Loxy618
Posts: 21
Joined: Wed Apr 24, 2019 2:01 am

Re: Garbage Collection Not Working

Post by Loxy618 » Tue Jan 21, 2020 11:46 am

Thanks stijn. So after further testing I think you are correct that the variables aren't going out of scope after the script finishes. Some of the code is as follows.

const char test_script[] = "a=[1]*1000";
mp_script_exec(test_script);
gc_collect_start();
gc_collect_end();

The variables in the script uses heap space during execution. When the script is finished I would have expected the gc to free up the heap space used by the script but I don't quite understand well enough how the gc works.

In general my system allows users to add/execute/delete different scripts from memory as they choose and over time I've been seeing the available heap space decrease so I assumed it was a gc collector problem. If MP still thinks the scripts/variables are in scope even when the user deletes them from memory that seems like the problem. Is there a way to 'kill' or force the script to go out of scope once it's finished executing to allow the gc to free up the heap?

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Garbage Collection Not Working

Post by stijn » Tue Jan 21, 2020 12:40 pm

Is there a way to 'kill' or force the script to go out of scope
Yes (couple of ways, e.g. wrap the whole thing in a function and execute the function instead of using global scope), but also make sure that is really what you want? I mean: you allow a user to execute multiple statements consecutively. Then if a user first says 'a = 1' it could be rather confusing/suprising for the user that when running the next statement, say 'b = a', suddenly a doesn't exist anymore? On the other hand if each run of a script is meant to be 'standalone' then that does make sense.

Loxy618
Posts: 21
Joined: Wed Apr 24, 2019 2:01 am

Re: Garbage Collection Not Working

Post by Loxy618 » Tue Jan 21, 2020 1:48 pm

Thanks, so each script will be standalone. If there is no way to free up memory after running a global script then we can work around that using functions. I tested by loading the following function into flash, then executed.

Code: Select all

def test():
 try:
  c=[1]*1000
  c[6]=6
  if c[6]==6:
   c[8]=8
 except:
  exit(0)
test()
I read the heap info prior to running and 432 bytes were used (this usage is expected). After running successfully, the heap info said that now 528 bytes were used. Can you explain why the heap still isn't going back to the initial 428 bytes used? Is that because the test() function call at the end is global? Some other reason?

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Garbage Collection Not Working

Post by stijn » Tue Jan 21, 2020 4:27 pm

528 bytes left still means the memory for c was reclaimed otherwise much more would be left, no? The rest can be memory from the parser or so, pretty hard to tell without logging everything which is on the heap. Btw if I just run this from a file the result is different:

Code: Select all

import gc

def Test():
  try:
    c = [1] * 1000
    c[6] = 6
    if c[6] == 6:
      c[8] = 8
  except:
    exit(0)

print('pre run', gc.mem_alloc())
Test()
print('post run', gc.mem_alloc())
print('collect', gc.collect())
print('post run', gc.mem_alloc())
shows for me

Code: Select all

pre run 3008
post run 11104
collect 5
post run 2976

Loxy618
Posts: 21
Joined: Wed Apr 24, 2019 2:01 am

Re: Garbage Collection Not Working

Post by Loxy618 » Tue Jan 21, 2020 6:38 pm

Okay, I assume you had MICROPY_PY_GC enabled to use "import gc". I am trying to use MICROPY_ENABLE_GC. I will try your way and see. I'm not sure if that will make a difference

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Garbage Collection Not Working

Post by stijn » Tue Jan 21, 2020 7:20 pm

In principle it doesn't matter if you call gc_collect via Python or C, that should both do the same. But note each port has it's own implementation of gc_collect and that should work properly else results might not be ok.

Post Reply