M_new in external C Module, How to protect from GC.collect()

[Codingfragments]

Hi

I currently try to develop a C/C++ based extension Module which needs to create some dynamic memory Buffers. After switching to the master branch which allowed me to use some C++ Code I’m making good progress but I currently seek the best way to link my dynamic buffers (char[] with flexible sizes) to the registered Object to allow the GC Algorithm to collect and keep the memory.

My Current Code uses some wrappers to map a C++ class to a Python object, this works great. But in the constructor I create a memory block (using m_new(char,1024) ) and I can see the memory been taken from Heap. But after a GC.collect() the Memory is freed which is expected because the Python Object seems to not be able to follow the char* pointer within the struct.

I think I need to link it to the Python structure somehow, But I couldn’t find a good example on how to do this.


Any Hints ?


Stefan

[stijn]

My Current Code uses some wrappers to map a C++ class to a Python object, this works great. But in the constructor I create a memory block (using m_new(char,1024) )

Has been a while since I looked into the details, but IIRC if the pointer to that memory is a member of the MicroPython object should be treated correctly by the GC automatically.

However: which constructor? If you mean the C++ one: that's wrong, all C++ functions and classes should use the C++ heap. If you mean the Python one: is that really needed, what do you do with that memory? Given the question you ask it sounds you have a problem with mixing the MicroPython and C++ heaps (not sure though). You say you're wrapping a class, then can't that class allocate what it needs?

[Codingfragments]

Thanks for your reply.

In my case I'm developing for the ESP32 port which doesn't have a nativ malloc function and all memory is taking from the combined heap, which includes all python heap as well.

I don't think the C++ part is the main issue here, more importantly, I like to understand how to store the pointer into the python structure so that it can be traversed correctly.

I tried to learn from framebuffer but in this case the buffer itself is created in python world and therefore it's already a valid python object that is given to the c-wrapper for the __init__ function. This object follows the buffer protocol as far as i know.


After some more digging, I "think' I need to wrap the raw pointer with mp_obj_new_bytearray_by_ref to create a proper mp_obj_t which can be traversed. But I'm not 100% sure.

[Codingfragments]

My Current Code uses some wrappers to map a C++ class to a Python object, this works great. But in the constructor I create a memory block (using m_new(char,1024) )

Has been a while since I looked into the details, but IIRC if the pointer to that memory is a member of the MicroPython object should be treated correctly by the GC automatically.

However: which constructor? If you mean the C++ one: that's wrong, all C++ functions and classes should use the C++ heap. If you mean the Python one: is that really needed, what do you do with that memory? Given the question you ask it sounds you have a problem with mixing the MicroPython and C++ heaps (not sure though). You say you're wrapping a class, then can't that class allocate what it needs?

BTW, I think you are the creator of the micropython-wrap utility ?

I couldn't find how you handle mp_object_t references in the class-wrapping. is it safe to just include them in the class itself or do I need to do anything special?



Stefan

[stijn]

In my case I'm developing for the ESP32 port which doesn't have a nativ malloc function and all memory is taking from the combined heap, which includes all python heap as well.

Possibly, but that is not the point wrt gc: the MicroPython heap is one big chunk of memory which is allocated once only. From then on, all allocations (e.g. m_new) in MicroPython are done from that heap. Garbage collection only works on that heap. So functionally it's separate from e.g. any C++ allocations those still uses the rest of the heap as usual, except that big MicroPython chunk.

I don't think the C++ part is the main issue here, more importantly, I like to understand how to store the pointer into the python structure so that it can be traversed correctly.

With 'python structure' I assume you mean the struct with mp_obj_base_t as first member? All other members there should be traversed by the gc. So something like

Code: Select all

typedef struct _my_obj_t {
    mp_obj_base_t base;
    char* ptr;
    mp_obj_t some_obj;
} my_obj_t;

leads to both ptr and some_obj being treated correctly (i.e. scanned recursively).

...the buffer itself is created in python world and therefore it's already a valid python object that is given to the c-wrapper for the __init__ function...

After some more digging, I "think' I need to wrap the raw pointer with mp_obj_new_bytearray_by_ref to create a proper mp_obj_t which can be traversed. But I'm not 100% sure.

Which raw pointer? What's passed to your __init__ is already an mp_obj_t, just store it in the struct then access it using the bytearray API?

BTW, I think you are the creator of the micropython-wrap utility ?

Indeed.

I couldn't find how you handle mp_object_t references in the class-wrapping. is it safe to just include them in the class itself or do I need to do anything special?

I'm not completely sure what you mean: you want to store an mp_obj_t in a C++ class? My initial reaction would be: why? There might be better ways to do that. But in any case: good question. Answer is: the layout of ClassWrapper is just like my_obj_t above and next to the base it stores a shared_ptr to the C++ class being wrapped. As such it is likely you need to do something special to make sure the gc can find the mp_obj_t stored in that C++ class because it depends on the memory layout of std::shared_ptr whether the gc will traverse and reach the stored mp_obj_t. It's exactly for that reason I wrote PinPyObj (https://github.com/stinos/micropython-w ... hon.h#L426) instead of storing bare mp_obj_t instances.

[Codingfragments]

Thanks a lot for your Answer.

I also spend some time last night (actualy 6 hours straight) digging into the Micropython src code. That exercise and your answer have been great fun and i learned a lot about the actual GC implementation in Micropython.

My current approach was to ditch the C++ implementation and start with a plain C-Module. You're right that the key was to store a wrapped reference of my byte-buffer into the returned mp_obj_t reference pointer. That way it will become part of the stored object and traced by the GC run.

Just having the object as part of the C++ definition didn't help, so I will also look into the PinPyObj way to have it stored in the modules globals dict.

On the heap spaces. in the unix port the c++ alloc is different from the python heap and malloc will allow you to create outside buffers with no interaction on the GC needed. But in most ports an OS level malloc doesn't exists so in ESP32 (which my modul is targeting on) those objects will be created in the python heap space. Without storing a reference that is accessible to the python GC it will be collect and potentially reuse the memory area later which will lead into problems.

Storing this in mp_obj_t is preventing this and was the missing step.


Unfortunately there wasn't anything real about this in the docs so it took a while to identify the problem, on the other hand this seems to be a specific use case that only a fraction of micropython developer will run into and the source structure is really easy to read once you learned about the easier ways to find stuff and follow references so i don't think it's a real problem.

And after all, it took only a day to find a solution AND have a reliable ws2812b RMT driver prototype with 300+ LEDs running. So i guess that's not to bad either.

Thanks for your help. I will look a bit more into the -wrap macros eventually because after returning to pure C/C++ after 20 years of Java, Objectiv-C, Go and Rust I feel C++ is a bit easier to write then C for myself

Stefan

[stijn]

On the heap spaces. in the unix port the c++ alloc is different from the python heap and malloc will allow you to create outside buffers with no interaction on the GC needed. But in most ports an OS level malloc doesn't exists so in ESP32 (which my modul is targeting on) those objects will be created in the python heap space. Without storing a reference that is accessible to the python GC it will be collect and potentially reuse the memory area later which will lead into problems.

I actually looked at esp32/main.c to figure out which memory is used for the MicroPython heap, and it does look like it is a separate piece of memory? I.e. are you saying that if on an ESP32 calling C++ new() or make_shared() will result in allocating a piece of memory which is in the same memory area that is passed to gc_init? That would quickly result in a mess?

[Codingfragments]

Good one, and i was slightly off

I did some experiments on the esp32 heap

Heap summary for capabilities 0x00000004:
At 0x3ffaff10 len 240 free 4 allocated 164 min_free 4
largest_free_block 4 alloc_blocks 9 free_blocks 1 total_blocks 10
At 0x3ffb6388 len 7288 free 0 allocated 7168 min_free 0
largest_free_block 0 alloc_blocks 22 free_blocks 0 total_blocks 22
At 0x3ffb9a20 len 16648 free 4 allocated 16368 min_free 4
largest_free_block 4 alloc_blocks 60 free_blocks 1 total_blocks 61
At 0x3ffcac00 len 87040 free 57464 allocated 29524 min_free 57464
largest_free_block 57464 alloc_blocks 4 free_blocks 1 total_blocks 5
At 0x3ffe0440 len 15072 free 15036 allocated 0 min_free 15036
largest_free_block 15036 alloc_blocks 0 free_blocks 1 total_blocks 1
At 0x3ffe4350 len 113840 free 0 allocated 113804 min_free 0
largest_free_block 0 alloc_blocks 1 free_blocks 0 total_blocks 1
Totals:
free 72508 allocated 167028 min_free 72508 largest_free_block 57464

On initial startup the python kernel tries to get the longest possible chunk which is at 0x3ffe4350 in my case (1 Block of Size 113840). This becomes the heap for GC.

any C/C++ storage requirements are filled in by malloc calls and they are spreaded based on most available space. My testcall did a 10K malloc which was given by 0x3ffcac00.

After a couple more execution this Area is depleted:

At 0x3ffcac00 len 87040 free 6244 allocated 80724 min_free 6244
largest_free_block 6244 alloc_blocks 9 free_blocks 1 total_blocks 10


So, fundamentally, I learned something else. IF your memory requirements do not need to be in the biggest block (which would end up taking memory from the python heap) you can get Memory outside of the python heap and simply use it as you would do with normal C/C++.

Also Be aware that these assignments are not freed upon soft reset, so if memory problems happen based on not enough memory available a hard reset will likely fix those.

In addition to this, I also learned ho i got trapped by this. My Memory need was higher then the 60k initial available and malloc ended up failing, so i switched to m_malloc which worked. I didn't understand the reason initially but saw the memory be within the python heap and tried to find a way to protect this memory. Also i read some docs that made me believe that esp32 will not have a separate memory area for the python heap which explained the symptoms at that time but it was wrong

With that journey, it's also quite easy to explain how the heap is secured and why it's only one Block of Memory to RTOS

// Allocate the uPy heap using malloc and get the largest available region
size_t mp_task_heap_size = heap_caps_get_largest_free_block(MALLOC_CAP_8BIT);
void *mp_task_heap = malloc(mp_task_heap_size);

This Code in the esp32 Port simply takes the biggest available block which is 8bit aligned and secure this block to the heap.

Stefan

[dhylands]

In order to “protect” memory allocated from the Python heap, the allocated object needs to reachable by tracking from the root pointers or the stack.

There are several threads in the forum discussing this like this one: viewtopic.php?f=2&t=8877&p=51142

[Codingfragments]

Yeah,

I search for some, didn't find that one.

After all, it was for good, the last 2 Days told me a lot about the memory management details, c-modules and the c++ wrapper .

So it was very well invested time.

I also found out that my C Skills are a bit Rusty ( ) So I'm not sure how good the code quality is at the end but I plan to develop a fast-led inspired neopixel model that is able to reliably get the timing right for up to 300 LED and support RGB and RGBW LEDs.

My Test code already solves the harder Part (RMT Controller of 300leds with reliable timing that is not taking up to much of python heap).


Thanks for the help in this Forum, that was great.