Advice for handling external memory in C modules

[wavey]

Hi all,

Newbie here, sorry if this is basic!

I am writing some C modules to integrate with some other pre-written libraries I have for my board, and would be grateful for any advice for how best to approach this.

I have external libraries which produce and consume a large amount of data - so one library produces a large chunk of data, which other libraries can consume, and in C++ this is just done by passing around pointers. I'm trying to write wrappers for these, so the process can be directed by a MicroPython script, something along the lines of:

Code: Select all

chunk = lib1.create_data()
lib2.process_data(chunk)

I guess I have two main questions:

1) If the Python script doesn't need to access the individual bytes of the data chunk, can I avoid copying the data into and out of MicroPython's heap? Could I maybe package up the data chunk as a "bytes" object while it is outside of MicroPython's heap, or will this cause complications with the garbage collector? Or is there a common way to have a "reference" to an external chunk of memory I should be looking at?

2) If I do have a way to reference this external chunk of memory, I'd like to be able to free it as soon as the Python script is done with it. Is there any way I can do this without waiting for a garbage collection sweep? (I was thinking of something like watching the reference count of the object, but I'm not sure that's how MicroPython manages memory..?) Or is the best solution to add an explicit API to the chunk object which the script could use to signal it is finished with it?

Again sorry if this is a common scenario I haven't yet found in the docs / forums! Many thanks for any advice.

[tannewt]

1) If the Python script doesn't need to access the individual bytes of the data chunk, can I avoid copying the data into and out of MicroPython's heap? Could I maybe package up the data chunk as a "bytes" object while it is outside of MicroPython's heap, or will this cause complications with the garbage collector? Or is there a common way to have a "reference" to an external chunk of memory I should be looking at?

The best thing to do is to allocate the memory on the MicroPython heap with a bytearray object and then pass the bytearray into the object that takes it. That way the memory can be used again if you get more data.

You can reference external memory from a MicroPython allocated object without an issue. The GC will ignore pointers outside its range.

2) If I do have a way to reference this external chunk of memory, I'd like to be able to free it as soon as the Python script is done with it. Is there any way I can do this without waiting for a garbage collection sweep? (I was thinking of something like watching the reference count of the object, but I'm not sure that's how MicroPython manages memory..?) Or is the best solution to add an explicit API to the chunk object which the script could use to signal it is finished with it?

"as soon as the python script is done with it" isn't something Python does that well. Memory isn't actually freed until "some time later". In MicroPython this is either when a manual collect is triggered or an allocation fails. There is a good video on the GC from jimmo here: https://www.youtube.com/watch?v=H_xq8IYjh2w

You can have MicroPython call a function when the memory is freed "some time later" by allocating with a finalizer and providing __del__ iirc.

Hope that helps!

[wavey]

Many thanks for this, much food for thought.

The best thing to do is to allocate the memory on the MicroPython heap with a bytearray object and then pass the bytearray into the object that takes it. That way the memory can be used again if you get more data.

Do you mean ask the initial library to write into an area allocated on the MicroPython heap? That would solve a lot of issues, but the library I'm using doesn't support that unfortunately.

You can reference external memory from a MicroPython allocated object without an issue. The GC will ignore pointers outside its range.

This is good to know, thankyou. I'm wondering about the different lifetimes of the allocations, and the case where I want to free the external memory before the MicroPython object is deleted - I guess if I create my own MicroPython native object I can guard against access after the external memory is freed, and raise an exception if anything then tries to use it.

"as soon as the python script is done with it" isn't something Python does that well. Memory isn't actually freed until "some time later". In MicroPython this is either when a manual collect is triggered or an allocation fails. There is a good video on the GC from jimmo here: https://www.youtube.com/watch?v=H_xq8IYjh2w

You can have MicroPython call a function when the memory is freed "some time later" by allocating with a finalizer and providing __del__ iirc.

That's a great video, thank you for sharing. I learned a lot, and it's also answered why I need to add in CPU register reads to the GC implementation (I'd been meaning to look into that for a while!), and perfectly explained a severe slowdown issue I had with a benchmark a few weeks ago, which turned out to be due to it allocating a lot of double-precision complex numbers.

After some more digging around, I wonder if the nicest way to deal with letting me free the external memory before a GC sweep is wrapping it up with context managers? So instead of the code I posted above, something like

Code: Select all

with lib1.create_data() as chunk:
	lib2.process_data(chunk)

should let me free it when the 'with' block is done. Does that seem reasonable, or would it be a misuse of context managers?

[tannewt]

Many thanks for this, much food for thought.

The best thing to do is to allocate the memory on the MicroPython heap with a bytearray object and then pass the bytearray into the object that takes it. That way the memory can be used again if you get more data.

Do you mean ask the initial library to write into an area allocated on the MicroPython heap? That would solve a lot of issues, but the library I'm using doesn't support that unfortunately.

You're welcome! I'd suggest allocating the memory before the first library.

You can reference external memory from a MicroPython allocated object without an issue. The GC will ignore pointers outside its range.

This is good to know, thankyou. I'm wondering about the different lifetimes of the allocations, and the case where I want to free the external memory before the MicroPython object is deleted - I guess if I create my own MicroPython native object I can guard against access after the external memory is freed, and raise an exception if anything then tries to use it.

"as soon as the python script is done with it" isn't something Python does that well. Memory isn't actually freed until "some time later". In MicroPython this is either when a manual collect is triggered or an allocation fails. There is a good video on the GC from jimmo here: https://www.youtube.com/watch?v=H_xq8IYjh2w

You can have MicroPython call a function when the memory is freed "some time later" by allocating with a finalizer and providing __del__ iirc.

That's a great video, thank you for sharing. I learned a lot, and it's also answered why I need to add in CPU register reads to the GC implementation (I'd been meaning to look into that for a while!), and perfectly explained a severe slowdown issue I had with a benchmark a few weeks ago, which turned out to be due to it allocating a lot of double-precision complex numbers.

After some more digging around, I wonder if the nicest way to deal with letting me free the external memory before a GC sweep is wrapping it up with context managers? So instead of the code I posted above, something like

Code: Select all

with lib1.create_data() as chunk:
	lib2.process_data(chunk)

should let me free it when the 'with' block is done. Does that seem reasonable, or would it be a misuse of context managers?

I really like context managers myself because they are harder to mess up. In CircuitPython we do something similar. Our native objects have a `deinit` method to release hardware resources and the objects can be used in a context manager to have `deinit` automatically called.

[jimmo]

Sorry I missed this thread earlier!

You can have MicroPython call a function when the memory is freed "some time later" by allocating with a finalizer and providing __del__ iirc.

Yep, that's exactly right.

The main gotcha with finalisers in MicroPython is that:
- They're not supported for types defined in Python (i.e. you can't just add `def __del__(self):` to a type). Only native types.
- You have to tell the GC at allocation time that the type has a finaliser. Usually this means using m_new_obj_with_finaliser() to allocate it instead of m_new_obj (or m_malloc_with_finaliser). This means that your native types need to do this in their `make_new` function. There's a few examples of this.

should let me free it when the 'with' block is done. Does that seem reasonable, or would it be a misuse of context managers?

I agree with tannewt, seems totally reasonable.

And just to recap on the other bits -- yes it's totally fine to reference non-heap memory from your objects. From MicroPython's perspective, a non-heap pointer is just like any other field in the struct. However, the thing to be careful of is that if something in the non-heap memory points back into the heap, then the GC won't find it.

An example of that might be a native Python type that points to some struct in non-heap memory, which contains a callback, with an argument pointer that points to a heap object. The GC won't know not to free the argument.

[wavey]

Many thanks to both of you for the feedback, a lot of good information here.

- You have to tell the GC at allocation time that the type has a finaliser.

That's a gotcha that got me! Cheers, will fix this in my code.

Btw, as I was implementing this, I noticed references to a "buffer protocol" which is a really neat feature: by supporting this, my wrapper object now plays a lot more nicely with the rest of Python, and lets the user pass it to other modules which expect a buffer (like writing to a file, constructing a Bytes object etc.) Just wanted to note this in case anyone else is working with a similar problem to mine!

[wavey]

One last question regarding memory usage - I'm currently writing a module API implementation in C, which will need to allocate several buffers (to then fill and pass into some native libraries). I would also like to be able to raise exceptions at various points, if e.g. while I'm filling these buffers, if I discover a problem while iterating through an input list parameter.

The last thing I want to do is leak memory, so would it be OK to allocate the memory on the MicroPython heap with m_malloc and trust the garbage collector to clean up in the case of errors? (I guess it just feels a bit odd to use the MP heap for something MP doesn't need to access.) If so, is it best practice to m_free() the memory in the normal no-exception case, or is it best to leave it to the GC?

Alternatively I guess I could use the normal malloc and free, and try to be careful not to call anything which could raise an exception.

[jimmo]

The last thing I want to do is leak memory, so would it be OK to allocate the memory on the MicroPython heap with m_malloc and trust the garbage collector to clean up in the case of errors? (I guess it just feels a bit odd to use the MP heap for something MP doesn't need to access.)

It's totally fine to leave it up to the GC. But you probably do want to use m_free in all cases (exception and normal).

The MP heap doesn't care what you use the memory for, as long as it knows whether a given allocation needs to be marked during the collection process. So that means that the allocation needs to be pointed to by:
- The stack
- CPU Registers
- Root pointers
- In-scope python variables

So if you m_malloc() then raise, then the allocation will no longer be pointed to by anything, so it will be available for collection.

The reason you would explicitly m_free() is as an optimisation -- if it's a big buffer then it might be better to explicitly get that memory back ASAP rather than having to wait for the next collection (and collections can take several milliseconds).

Additionally, this really helps with fragmentation. In a simple example, imagine you have a program that does the following sequence in a loop:

- Allocate large buffer
- Do processing with large buffer
- Allocate a small result object
- Save result

The resulting memory layout will be

| <big buffer (unreferenced)> <small object> <big buffer (unreferenced)> <small object> <big buffer (unreferenced)> <small object>...|

then after collection

| <unused space> <small object> <unused space> <small object> <unused space> <small object> ... |

so it will be extremely fragmented. Despite having lots of ram free, you can never make an allocation larger than one of the unused blocks.

Instead if you do:

- Allocate a small result object
- Allocate large buffer
- Do processing with large buffer
- Free large buffer
- Save result

then all the small objects will be compacted at the start of the program. Note that this is actually quite difficult to achieve in practice as it requires that no objects are allocated while the big buffer is still in used.

If so, is it best practice to m_free() the memory in the normal no-exception case, or is it best to leave it to the GC?

It's kind of the same thing, except at least in the exception case you have a much better chance of getting back to the original heap state.

(Sorry I know this is a kind of vague answer, but the specifics really depend a lot on your program... In summary, explicitly freeing unused large buffers is a good thing, but may not actually make that much difference to fragmentation).

Alternatively I guess I could use the normal malloc and free, and try to be careful not to call anything which could raise an exception.

On bare metal there is no "normal malloc/free" but yeah on ESP32 or other custom ports on an RTOS this might be an option (assuming there's some RAM left over that isn't used by the MicroPython heap).

[wavey]

Many thanks again, I appreciate the clear explanation. Your advice is well taken - I'll ensure that the results objects (and any other small buffers) are allocated before the large ones. Cheers!