More detail for MemoryError exceptions

kjw · Post by **kjw** » Wed Oct 28, 2020 3:42 pm

I've had a few troublesome MemoryError exceptions in CircuitPython recently and from what I've seen on MicroPython on micro:bit the issue is the same. The tricky ones for me fall into four categories:

feels like there's enough memory around but still fails, sometimes explicity gc.mem_free() "profiling" in the code shows there is enough memory around;
allocation occurs within the interpreter so the line number of Python code is not specific enough;
no obvious reason;
infrequent and probably just exhaustion.

My assumption is that fragmentation is the likely cause of the first class of problem. Here it could be useful to print some extra detail about total free and biggest lump free and/or a memory map.

For the second one, if C code is doing multiple allocations in the interpreter then it's not always obvious which one caused the problem and that's important for pinpointing bugs including ones relating to over zealous use of memory. Some sort of C stack trace might help here to supplement the reporting of the Python line number. Even a primitive one with just memory addresses would be beneficial for the cognoscenti. It would help https://github.com/adafruit/circuitpython/issues/3119

The "no obvious reason" ones for me have been related to growth of dict-like objects, I think. I was using a fair number of variables in a sprawling, "pre-pylint", memory heavy program and that was getting MemoryError exceptions on harmless looking (non-allocating?) statements like var = 1. These would move around as I experimented and fiddled with the code. The likely problem here was the underlying hash map for the symbol table (locals()) expanding as a new variable appeared in the code. This one was very confusing but I did find a similar issue and there are some magic size values that pop out here that can help to identify these. 1784 bytes is one of these magic values as seen in https://github.com/jczic/MicroWebSrv/issues/5 where it's a bit more obvious on the cause.

The infrequent ones for me tend to be when a program is running very close to the limit. I just tend to run the code for a few hours to see how robust it is and tweak it so these aren't as problematic as they sound.

A hardware debugger could be used in some cases but I don't regard it as a complete solution as not everyone has one, not everyone knows how to use them, not all boards have the socket so it's soldering time and for one of my cases I couldn't predict which of six boards the exceptionally infrequent MemoryError was going to happen on.