Memory allocation errors with plenty of space

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
kwiley
Posts: 102
Joined: Wed May 16, 2018 5:53 pm
Contact:

Memory allocation errors with plenty of space

Post by kwiley » Mon Jun 18, 2018 10:09 pm

I often get memory allocation exceptions when there is quite a bit of free memory and the requested amount is substantially smaller than the current supply. For example, I might see something like this:

Free memory: 49616
MemoryError: memory allocation failed, allocating 4084 bytes

I know I can fiddle with calling the garbage collector manually, and I know I can try to make best practices of preallocating and reusing buffers, etc. etc. I know there's lots of tactics I can try to apply to insure good use of memory, but all that aside, why am I getting allocation failures of the sort shown above, where the request is less than a tenth of the available amount? This is a pretty frequent occurrence.

The only implication I can think of is that there aren't 4084 contiguous bytes available even though 49616 are free in total, but that is a rather staggering prospect at such low proportions. It suggests the memory is seriously fragged, even after explicitly calling the garbage collector. Frankly, I'm rather incredulous that's the explanation here. I suspect I'm not understanding something about how memory is being managed...or there is low-level bug in the memory management system (the former explanation being more likely I readily admit).

Any ideas?

Much thanks.

User avatar
dhylands
Posts: 3415
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Memory allocation errors with plenty of space

Post by dhylands » Mon Jun 18, 2018 11:47 pm

You can print the heap by using:

Code: Select all

import micropython
micropython.mem_info(1)
Its best to do this just after doing a gc.collect(). Each byte corresponds to 8 bytes of memory. To allocate 4084 bytes would require 510 contiguous blocks.

Caveat: I know the above works on the pyboard, I assume that it works on the esp8266, but haven't tried it myself.

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Memory allocation errors with plenty of space

Post by cefn » Mon Jun 18, 2018 11:48 pm

It's really hard to respond without seeing any code. However, I think it's fairly simple to imagine a procedure which systematically fragments memory in the way you describe and if your code were shared, it might even be possible to pin it down.

The way this happens is well documented here...
https://docs.micropython.org/en/latest/ ... ained.html

Especially...
https://docs.micropython.org/en/latest/ ... gmentation
and the section called "String concatenation"

User avatar
pythoncoder
Posts: 4351
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Memory allocation errors with plenty of space

Post by pythoncoder » Tue Jun 19, 2018 6:06 am

@kwiley As the docs referenced by @cefn point out, heap fragmentation is an issue, and allocating buffers as big as 4KB can be problematic. The key is to allocate them early and to leave them in place for as long as they are required (usually the duration of the application). My approach is to import necessary modules, perform a GC, then allocate any required large buffers. Then run the rest of the application.
Peter Hinch

kwiley
Posts: 102
Joined: Wed May 16, 2018 5:53 pm
Contact:

Re: Memory allocation errors with plenty of space

Post by kwiley » Tue Jun 19, 2018 4:19 pm

Thanks everyone. I'll take a closer look at that. One place that it is hard to avoid reallocating memory is in the rolling buffer coming in from a serial line. I can preallocate the receive buffer so as to use usb.readinto() instead of usb.read(), but I still have to chomp my way through the accumulating byte buffer to process the received data. Continually "chomping" the buffer obviously incurs reallocations. Even if I leave the buffer alone and only reference it via indexed subranges, won't those subranges then be copied out to temporary arrays when I reference them?

I don't know. I'll go through the code carefully and seek out all the memory allocations I can find.

Thanks.

User avatar
Roberthh
Posts: 2054
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Memory allocation errors with plenty of space

Post by Roberthh » Tue Jun 19, 2018 4:48 pm

you can use memoryview to avoid that chomping. See http://docs.micropython.org/en/latest/p ... memoryview

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Memory allocation errors with plenty of space

Post by cefn » Tue Jun 19, 2018 4:50 pm

As long as your chomping is interspersed by a gc.collect() before other allocations take place, you should be OK as the same region of memory should be allocated and cleared each time.The problem comes if those temporary allocations are interspersed with allocations which are likely to remain set aside from the heap.

Also take note of memoryview as a means of working with subranges of re-existing buffers without doing substantial allocation. http://docs.micropython.org/en/v1.9.3/p ... iew#arrays

Edit: OK, Roberthh beat me to it!

kwiley
Posts: 102
Joined: Wed May 16, 2018 5:53 pm
Contact:

Re: Memory allocation errors with plenty of space

Post by kwiley » Tue Jun 19, 2018 5:53 pm

That memorview tool looks very useful. However, I am not seeing the expected behavior. Consider the following:

Code: Select all

mp.mem_info(1)
a = bytearray(10000)
mp.mem_info(1)
b = a[:]
mp.mem_info(1)
c = memoryview(a)
mp.mem_info(1)
After allocating a, I see a large jump in used memory. After allocating b, I also see an a large jump, as expected. However, after allocating c I still see a third large jump in used memory. That doesn't make sense. I expected it to allocate little more than a reference.

User avatar
Roberthh
Posts: 2054
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Memory allocation errors with plenty of space

Post by Roberthh » Tue Jun 19, 2018 7:03 pm

If I do a gc.collect() before each mp.mem_info(1), the memoryview add 64 bytes used memory. Tested with micropython@windows. I'm not at a micro at the moment.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Memory allocation errors with plenty of space

Post by jickster » Tue Jun 19, 2018 7:18 pm

A permanent fix to fragmentation requires a breaking change to usage of malloc.

It’s in initial stages of discussion.

Post Reply