MicroPython Forum (Archive)

I often get memory allocation exceptions when there is quite a bit of free memory and the requested amount is substantially smaller than the current supply. For example, I might see something like this:

Free memory: 49616
MemoryError: memory allocation failed, allocating 4084 bytes

I know I can fiddle with calling the garbage collector manually, and I know I can try to make best practices of preallocating and reusing buffers, etc. etc. I know there's lots of tactics I can try to apply to insure good use of memory, but all that aside, why am I getting allocation failures of the sort shown above, where the request is less than a tenth of the available amount? This is a pretty frequent occurrence.

The only implication I can think of is that there aren't 4084 contiguous bytes available even though 49616 are free in total, but that is a rather staggering prospect at such low proportions. It suggests the memory is seriously fragged, even after explicitly calling the garbage collector. Frankly, I'm rather incredulous that's the explanation here. I suspect I'm not understanding something about how memory is being managed...or there is low-level bug in the memory management system (the former explanation being more likely I readily admit).

Any ideas?

Much thanks.

You can print the heap by using:Its best to do this just after doing a gc.collect(). Each byte corresponds to 8 bytes of memory. To allocate 4084 bytes would require 510 contiguous blocks.

Caveat: I know the above works on the pyboard, I assume that it works on the esp8266, but haven't tried it myself.

It's really hard to respond without seeing any code. However, I think it's fairly simple to imagine a procedure which systematically fragments memory in the way you describe and if your code were shared, it might even be possible to pin it down.

The way this happens is well documented here...
https://docs.micropython.org/en/latest/ ... ained.html

Especially...
https://docs.micropython.org/en/latest/ ... gmentation
and the section called "String concatenation"

@kwiley As the docs referenced by @cefn point out, heap fragmentation is an issue, and allocating buffers as big as 4KB can be problematic. The key is to allocate them early and to leave them in place for as long as they are required (usually the duration of the application). My approach is to import necessary modules, perform a GC, then allocate any required large buffers. Then run the rest of the application.

Thanks everyone. I'll take a closer look at that. One place that it is hard to avoid reallocating memory is in the rolling buffer coming in from a serial line. I can preallocate the receive buffer so as to use usb.readinto() instead of usb.read(), but I still have to chomp my way through the accumulating byte buffer to process the received data. Continually "chomping" the buffer obviously incurs reallocations. Even if I leave the buffer alone and only reference it via indexed subranges, won't those subranges then be copied out to temporary arrays when I reference them?

I don't know. I'll go through the code carefully and seek out all the memory allocations I can find.

Thanks.

you can use memoryview to avoid that chomping. See http://docs.micropython.org/en/latest/p ... memoryview

As long as your chomping is interspersed by a gc.collect() before other allocations take place, you should be OK as the same region of memory should be allocated and cleared each time.The problem comes if those temporary allocations are interspersed with allocations which are likely to remain set aside from the heap.

Also take note of memoryview as a means of working with subranges of re-existing buffers without doing substantial allocation. http://docs.micropython.org/en/v1.9.3/p ... iew#arrays

Edit: OK, Roberthh beat me to it!

That memorview tool looks very useful. However, I am not seeing the expected behavior. Consider the following:
After allocating a, I see a large jump in used memory. After allocating b, I also see an a large jump, as expected. However, after allocating c I still see a third large jump in used memory. That doesn't make sense. I expected it to allocate little more than a reference.

If I do a gc.collect() before each mp.mem_info(1), the memoryview add 64 bytes used memory. Tested with micropython@windows. I'm not at a micro at the moment.

A permanent fix to fragmentation requires a breaking change to usage of malloc.

It’s in initial stages of discussion.