Memory Leak on Micropython Port

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
mfrsi
Posts: 2
Joined: Tue Jan 10, 2017 8:10 pm

Memory Leak on Micropython Port

Post by mfrsi » Wed Jan 11, 2017 12:35 am

Hello

My team is currently working on porting MicroPython to our own Spansion MCU board. We were able to get the interpreter working, but we are running into a memory leak. After invoking do_str() around 30 times without resetting, our board crashes.

We currently suspect that some memory is not freed during parsing. Any help would be appreciated. The rest of the team will be available in a few hours and can go over the issue in more detail.

Thank you

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Memory Leak on Micropython Port

Post by pythoncoder » Wed Jan 11, 2017 7:19 am

It's hard to comment without seeing code but could it be related to this known issue https://github.com/micropython/micropython/issues/2280?
Peter Hinch
Index to my micropython libraries.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Memory Leak on Micropython Port

Post by stijn » Wed Jan 11, 2017 9:18 am

Indeed hard to tell for sure what the issue is without seeing any code which can reproduce the issue; preferrably on e.g. the unix port as that is easily debuggable. If you use the same code and build it for unix (while tuning heap size so it's the same as for the board), does it crash as well? If so then at least you can figure out what 'crash' entails.

chitturi123
Posts: 7
Joined: Mon Jan 16, 2017 6:11 am

Re: Memory Leak on Micropython Port

Post by chitturi123 » Mon Jan 16, 2017 10:18 am

Hi
I am the member of the team working on the MycroPython project ported on to Cortex M4.
The following is the detailed description of the problem i am facing.
I have kept debug trace of memory allocation and freeing, and found that some of the

addresses allocated are not freed.
These allocations are happening in mp_parse, mp_compile and mp_call_function_0 in do_str()
function and some addresses allocated are not freed.

I need some clarifications from the observations i had while executing the code.
1) My gc_collect function in mycropython is not triggered(In the debug print i don't see
any invocation of this function.)while running Is it a problem? or some thing i have not enabled in mpconfig.h or mpconfigport.h
files or in malloc.c file ?
2) Is the memory freeing problem due to gc_collect not triggered?
3) MICROPY_PY_GC in mpconfigport.h file to be enabled or not?
4) in malloc.c file MICROPY_ENABLE_FINALISER,
MICROPY_MEM_STATS,MICROPY_MALLOC_USES_ALLOCATED_SIZE macros are not enabled.
Are they required to be enabled?
5) Any other mandatory macros which i missed enabling.

Request help in fixing the issue.
Thanks
Chowdary

mfrsi
Posts: 2
Joined: Tue Jan 10, 2017 8:10 pm

Re: Memory Leak on Micropython Port

Post by mfrsi » Mon Jan 16, 2017 10:41 am

Hi
The team is working on the MycroPython project ported on to Cortex M4.
The following is the detailed description of the problem faced by the team.
Debug trace of memory allocation and freeing was kept , and found that some of the
addresses allocated are not freed.

These allocations are happening in mp_parse, mp_compile and mp_call_function_0 in do_str()
function and some addresses allocated are not freed.

The following clarifications are required from the team.
1) gc_collect function in mycropython is not triggered(In the debug print don't see
any invocation of this function.)while running
Is it a problem? or some thing i have not been enabled in mpconfig.h or mpconfigport.h
files or in malloc.c file ?
2) Is the memory freeing problem due to gc_collect not triggered?
3) MICROPY_PY_GC in mpconfigport.h file to be enabled or not?
4) in malloc.c file MICROPY_ENABLE_FINALISER,
MICROPY_MEM_STATS,MICROPY_MALLOC_USES_ALLOCATED_SIZE macros are not enabled.
Are they required to be enabled?
5) Any other mandatory macros missed enabling in configuration files.

Request help in fixing the issue.
Thanks
Mazar

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Memory Leak on Micropython Port

Post by pythoncoder » Mon Jan 16, 2017 11:59 am

If gc.collect() isn't executed then unused heap blocks aren't freed. My understanding is that gc.collect() is only called when user code attempts an allocation - and then only if either the allocation fails or the amount of free heap memory falls below a threshold. That threshold can be set in user code (gc.threshold()).

So, depending on your user code, gc.collect() may never be called, and unused heap blocks may remain un-freed. As far as I can see this is not a bug.
Peter Hinch
Index to my micropython libraries.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Memory Leak on Micropython Port

Post by stijn » Mon Jan 16, 2017 12:35 pm

1) As pythoncoder suggests: yes if no memory is never freed then you'll run out of memory after enough allocations. Is gc_alloc triggered?
2) Likely
3) MICROPY_PY_GC enables the gc module so gc functions can be called from python code. It is not required for actual gc functionality in the C code (that would be the MICROPY_ENABLE_GC macro).
4) All depends on your usage scenarion. Too much to explain here, look in mpconfig.h for a short explanation of what these maros enable. For more explanation you'll hav to go through the code.
5) Not that I know of, except you definitely need MICROPY_ENABLE_GC and MICROPY_GC_ALLOC_THRESHOLD.

For the rest, if you don't have a live debugger, enabling DEBUG_printf macors in files like malloc.c/gc.c is the way to see what goes on.

chitturi123
Posts: 7
Joined: Mon Jan 16, 2017 6:11 am

Re: Memory Leak on Micropython Port

Post by chitturi123 » Mon Jan 30, 2017 7:18 am

Hi,
When executing python code we have observed memory leaks as was posted earlier.
To work around this problem, mp_init(), do_str() function calls of python interpreter are invoked whenever code is executed.
The above solution is working for python code without loops.
When loops are added in the python code memory leaks are happening under each iteration in the loop.
We ran for 2 sample codes with loops and captured the logs.
One of the logs , it was observed that vstr_print is allocating memory in the loop but not freeing
it is possible for different sample codes malloc may be called from different function calls.
Is there a way to fix this problem?
I need help for the above problem

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Memory Leak on Micropython Port

Post by dhylands » Mon Jan 30, 2017 4:44 pm

MicroPython uses a garbage collector. So yes it "leaks" memory. Eventually, you'll run out of memory, and the garbage collector will run and free up all of the unused memory.

You can invoke the garbage collector manually by called

Code: Select all

gc.collect()
So if you wanted to see the freeing of the memory, then put a gc.collect() call at the bottom of your loop.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Memory Leak on Micropython Port

Post by dhylands » Mon Jan 30, 2017 4:52 pm

You make sure that the GC is enabled via MICROPY_ENABLE_GC.

Post Reply