How to allocate a buffer that won't move around in memory?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

How to allocate a buffer that won't move around in memory?

Post by WhiteHare » Wed Oct 17, 2018 11:03 pm

Notionally, what I want to do is something like:

Code: Select all

ba=bytearray[100]
base_address = uctypes.addressof(ba)
However, I want to do it in a way that guarantees the garbage collector won't subsequently move the ba object around, leading to a changed base address.

Why do I want this? Well, on the nRF52 series, the radio uses DMA to read and write directly from the same memory used by the microcontroller. I need to allocate a block of memory for the radio to use to store the payload of a received packet, and I do that by telling the radio the address of the first byte (the base address) of the block of memory. Once I tell the radio that address, I can't have that block of memory moved somewhere else by the GC, or else the radio will be writing bytes into the wrong parts of memory.

Is a byte array naturally locked down (can't be moved in memory), or do I have to do something explicit to lock a byte array down so that it doesn't move?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: How to allocate a buffer that won't move around in memory?

Post by dhylands » Wed Oct 17, 2018 11:40 pm

The garbage collector doesn't move anything around.

The only way that the bytearray will move is if you try to add an additional element to the array (i.e. make it be 101 or more elements).

And I'm assuming you mean to use parenthesis rather than square brackets. i.e. ba = bytearray(100)

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: How to allocate a buffer that won't move around in memory?

Post by WhiteHare » Thu Oct 18, 2018 12:05 am

You mean the micropython gc doesn't consolidate the heap after the heap becomes fragmented so that larger objects can be allocated?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: How to allocate a buffer that won't move around in memory?

Post by dhylands » Thu Oct 18, 2018 4:56 am

No it does not do any consolidation. There are lots of discussions in the forum about heap fragmentation.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: How to allocate a buffer that won't move around in memory?

Post by WhiteHare » Thu Oct 18, 2018 4:45 pm

:shock:

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: How to allocate a buffer that won't move around in memory?

Post by jickster » Thu Oct 18, 2018 8:09 pm

WhiteHare wrote::shock:
That’s about the reaction I’d expect lol


Sent from my iPhone using Tapatalk Pro

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Consider the intended use cases

Post by pythoncoder » Fri Oct 19, 2018 8:48 am

I think some consideration should be given to the principal use cases for MicroPython: embedded systems.

Such systems typically run a do-forever loop. They are often highly time critical. They run on bare metal and are written by programmers with an understanding of hardware; they are prepared to make hardware-orientated optimisations.

This is a very different environment to that offered by a general purpose language like CPython. Applications often have unpredictable runtime behaviour. Their programmers have a reasonable expectation of an environment which abstracts hardware away. And, while overall performance matters, millisecond level timing is rarely an issue. With a pre-emptive OS lurking beneath such precision is hard to achieve.

MicroPython GC typically takes times measured in single digit milliseconds. This is already problematic in some applications. I'm no expert on GC, but I'd guess that a compacting GC would take at least an order of magnitude longer. Heap fragmentation is an issue, but it can usually be mitigated with careful programming. In my view it is a lesser evil than slow GC.

I'd prefer to see scoped allocation rather than a compacting GC.
Peter Hinch
Index to my micropython libraries.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: How to allocate a buffer that won't move around in memory?

Post by WhiteHare » Fri Oct 19, 2018 2:50 pm

What is "scoped allocation"?

There are different ways to do it (like incremental garbage collection), but in most GC's the compaction doesn't happen unless there is a failure to allocate a block of memory due to heap fragmentation. So, you can think of it as an option of last resort. Yes, it does take at least some time, but the trade-off is that the program can continue running after the compaction. Without that option, I guess you'd have to somehow prove to yourself that it will never be needed, or else maybe be prepared to reboot at any moment if that situation were to occur.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

How to allocate a buffer that won't move around in memory?

Post by jickster » Fri Oct 19, 2018 6:58 pm

Scoped allocation is not implemented but would be highly useful especially in interrupts.

Currently you cannot allocate in an interrupt because you have no way of knowing if the gc is executing. Some python operations allocate but it’s not obvious so it’s a messy case because you’d have to have a c debugger to detect an allocation takes place.

Scoped allocation would create a temporary heap on the stack of the ISR and all allocations would happen on that heap. When isr goes away, that heap goes away.



Sent from my iPhone using Tapatalk Pro

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: How to allocate a buffer that won't move around in memory?

Post by pythoncoder » Sat Oct 20, 2018 8:01 am

Peter Hinch
Index to my micropython libraries.

Post Reply