GC and Heap

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Cyrille de Brébisson
Posts: 8
Joined: Wed May 30, 2018 7:46 am

GC and Heap

Post by Cyrille de Brébisson » Mon Jun 04, 2018 9:06 am

Hello,

I appologize if my question is naive, but I am not sure that I am understanding things properly here.

Is my understanding correct here:
If I enable MICROPY_ENABLE_GC, then the system will allocate an area of RAM at startup and use this ram for storing all the data. Doing it's own malloc in that are and managing the RAM as needed.
If I do NOT enable it, will the system use the normal, OS provided malloc and free?

Am I correct in assuming that using MICROPY_ENABLE_GC, the ram is allocated once and can not be increased, thus leading to potential out of memory condition even if more RAM is available from the main OS? If not, how can one increase the heap size at run time, while the python machine is running?

If I do NOT use MICROPY_ENABLE_GC, will the system still do Garbage collection? or will ram be leaked?

If I do NOT use MICROPY_ENABLE_GC, then can I save the pythin state to flash and reload it later?

Thanks,
Cyrille

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: GC and Heap

Post by jickster » Tue Jun 05, 2018 6:26 pm

Cyrille de Brébisson wrote:
Mon Jun 04, 2018 9:06 am
If I enable MICROPY_ENABLE_GC, then the system will allocate an area of RAM at startup and use this ram for storing all the data.
No. You have to allocate RAM as heap and pass it to gc_init()
This could be done as a global allocation i.e. byte theHeap[8192] and then gc_init(theHeap, theHeap+8192)

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: GC and Heap

Post by jickster » Tue Jun 05, 2018 6:28 pm

Cyrille de Brébisson wrote:
Mon Jun 04, 2018 9:06 am
If I do NOT enable it, will the system use the normal, OS provided malloc and free?
Yes. And you do not have to provide a heap because it will use the OS' heap.

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: GC and Heap

Post by jickster » Tue Jun 05, 2018 6:35 pm

Cyrille de Brébisson wrote:
Mon Jun 04, 2018 9:06 am
Am I correct in assuming that using MICROPY_ENABLE_GC, the ram is allocated once and can not be increased, thus leading to potential out of memory condition even if more RAM is available from the main OS? If not, how can one increase the heap size at run time, while the python machine is running?
For now it's not possible to increase heap size at runtime but it is being worked on.

https://github.com/micropython/micropython/pull/3580

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: GC and Heap

Post by jickster » Tue Jun 05, 2018 6:39 pm

Cyrille de Brébisson wrote:
Mon Jun 04, 2018 9:06 am
If I do NOT use MICROPY_ENABLE_GC, will the system still do Garbage collection? or will ram be leaked?
If you do NOT use MICROPY_ENABLE_GC, it'll use your system's heap which is not garbage collected because your system heap doesn't know you're allocated Python objects that need to be collected.

RAM will absolutely be leaked.

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: GC and Heap

Post by jickster » Tue Jun 05, 2018 6:42 pm

Cyrille de Brébisson wrote:
Mon Jun 04, 2018 9:06 am
If I do NOT use MICROPY_ENABLE_GC, then can I save the pythin state to flash and reload it later?
You can only do this in very specific case.
Restoring the heap to a different location is basically the same problem as supporting a precise GC. So basically I think it would only work if being restored to same location and size.

The other big issue is the state of the hardware and all of the peripherals.

And you'd probably need to save the stack as well, although that may not be required if saving the state is only allowed from certain locations.

The heap may also contain pointers back into flash, so you wouldn't be able to save the state, update some firmware and restore the state.
https://github.com/micropython/micropyt ... -394005772
This is somewhat related to #3619 -- if you could pause execution and return to the top level of the C code, then you'd have some chance of saving the VM state.
https://github.com/micropython/micropyt ... -394046605

Conditions to save/restore heap
1. heap must be same size
2. heap must be in same location in memory
3. you're ok with hw peripherals' state not matching with your heap object i.e. you configured SPI to be 100khz but the default is 1MHz so now you're out of sync with the hw.
4. you cannot update the flash the heap points to; to be safe, you shouldn't update flash at all in between save/restore
5. you must return to top level of C-code: if you're doing REPL, you'd be satisfying this but I think you could even execute a script and as long as it exits the vm and goes all the way back up to the "start" you're good.
Last edited by jickster on Tue Jun 05, 2018 9:36 pm, edited 1 time in total.

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

Re: GC and Heap

Post by jickster » Tue Jun 05, 2018 6:49 pm

Cyrille de Brébisson wrote:
Mon Jun 04, 2018 9:06 am
Hello,

I appologize if my question is naive, but I am not sure that I am understanding things properly here.

Is my understanding correct here:
If I enable MICROPY_ENABLE_GC, then the system will allocate an area of RAM at startup and use this ram for storing all the data. Doing it's own malloc in that are and managing the RAM as needed.
If I do NOT enable it, will the system use the normal, OS provided malloc and free?

Am I correct in assuming that using MICROPY_ENABLE_GC, the ram is allocated once and can not be increased, thus leading to potential out of memory condition even if more RAM is available from the main OS? If not, how can one increase the heap size at run time, while the python machine is running?

If I do NOT use MICROPY_ENABLE_GC, will the system still do Garbage collection? or will ram be leaked?

If I do NOT use MICROPY_ENABLE_GC, then can I save the pythin state to flash and reload it later?

Thanks,
Cyrille
My opinion: saving/restoring heap is unnecessary and if you tell us exactly why you're trying to do this, we can help you find a better way that doesn't involve this risky maneuver

Cyrille de Brébisson
Posts: 8
Joined: Wed May 30, 2018 7:46 am

Re: GC and Heap

Post by Cyrille de Brébisson » Wed Jun 06, 2018 7:07 am

Hello,

Sorry, this is a LONG post! To answer such a short question.
The first part of the post is a high level description of the existing system that I am trying to integrate µPython in. This (relatively large) system is pre-existing and non-negotiable. I was given the task to make µPython work in the system and do not have the freedom to change the system to make it work with µPython. This part is, in some way, a justification of my needs for the various
The second part describes the various technical issues that comes from the fact that I need to integrate the systems together. You can jump straight to the second part if you want to, but you will miss some of the context/justification for the asks.

--------------------- Part 1: The system that I am integrating µPython in
This is an embedded OS (single thread) with a call back driven GUI for interaction with the user.

The user facing GUI has 2 important concepts:
1) The "screen" concept
- This is a full screen GUI.
- There is always one and only one Screen active view at a time.
- The user can seamlessly switch from one screen to another. The Screen state needs to be preserved when entering/exiting a screen (so that you return to the same state when returning to the same screen).
- They are 2 types of Screens. System Screens (not context dependent) and "app" Screens which depend on the active app.
2) The "App" concept
- There is always ONE single active app
- “App” group functionality, “Screens” and user data
- An “App” is at the same time comparable to a class and an object instance. With the functionally and “screens” part of the class and the user data being an object instance. The user can have as many copies (object instances) of the app as he wants.
- The user can switch app whenever he wants. The app states must be preserved.
- Python will be an “App” of the system.

---------------------------------- Part 2: The issues that I am trying to fix/work around

The Python "app" in my system will present the user with an interactive "screen" (REPL loop).
-----
This interactive screen has to work through a call back system. So I need to break the current REPL into 3 parts: repl_init, repl_step(user entered text) and repl_end.
I have actually succeeded in doing so (at least I think so). The only issue that I am still facing is on the use of the “mp_stack_ctrl_init” calls.
Doing the init only at the top of the repl_init function does not work because the stack size/pos when executing repl_init does not match the stack size/pos when executing repl_step. I have therefore placed a call to mp_stack_ctrl_init at the top of each of the 3 repl functions.
How much trouble will that cause?
-----
When the user exits the interactive python area, I need to save the state and restore it when he returns for the user to feel that there is continuity of operation.
One issue is that I am memory limited so, if I allocate 1MB for the heap, but that only 5KB are used, I will waste a LOT of ram that I do not have the luxury to waste.
- Is there anyway to "compress" the RAM through the garbage?
- Could the garbage "clear" unused area (set them to 0) so that a zip could be applied to it to avoid needless use of memory?
----
Python is not the only thing running on the system, and my users can switch from one screen to another.
The "fixed size" heap allocation is therefore problematic; It would be great if the load/save state and GC system allowed memory compress (regroup?) and resize as this would allow a much nicer interaction with the rest of the system.
----
Question on re-entrance.
The system already has a native programming language (well 2 of them to be precise). And is highly extensible through user programs.
I am planning to add some python to native system hooks to allow python to call on existing system functionally to allow python and the native system to interact and to allow python programmers to use existing stuff (native, or user enhanced).
One issue is. If the user has created a native user program that in turn calls a Python sub-program. What will happen if said user was to create a python program calling the native user program? Ie: Python calls native that calls python again. Will it work? Does µPython store data in global variables that will cause things to break?
----
And debugging in all that?
It would be nice if µPython had a way to debug code :-)

I think that this is it for the moment. Thanks for taking the time to try to help me. I am in the early stages of the integration process, so I am having a hard time, but hopefully, it will get better!
Cyrille

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

GC and Heap

Post by jickster » Wed Jun 06, 2018 1:39 pm

There’s no way to compact the heap currently. This feature is in early stages of discussion.
Implementation will require new “pointer” model which will require all existing code to be transitioned to new pointer API.

Do not count on this to be implemented anytime soon.



Sent from my iPhone using Tapatalk
Last edited by jickster on Wed Jun 06, 2018 1:39 pm, edited 1 time in total.

jickster
Posts: 284
Joined: Thu Sep 07, 2017 8:57 pm

GC and Heap

Post by jickster » Wed Jun 06, 2018 1:43 pm

There is no debugger; it’s in progress.

The closest thing is a PyCharm plug-in to easily execute from PC but you cannot single step, see variables etc

Sent from my iPhone using Tapatalk

Post Reply