MemoryError when importing a custom module

[Llwy]

Hello,
I am working on a module for the MFRC522 RFID reader. The module size is 11KB
I managed to copy the module on the micropython filesystem with the help of webrepl_cli.py.

When I try to import the module through the UART REPL, I get the following error:

Code: Select all

MemoryError: memory allocation failed, allocating 136 bytes

If I try to import a second time the module, no exception is raised but the module is clearly not loaded properly (the only property which is visible is __name__ and none of the defined classes are there).

I tried to open the file and read it but I also got a memory error when trying to read the whole file in a single string.
I stepped line by line in the file and could display all 380 lines of code: there did not seem to be any corrupted data from the transfer.

11KB does not seem excessively large for a module, is this really a limitation of the available memory or could it be some other issue?
Does it mean that the module should be "frozen" into the hardware to optimize the memory usage?

I am working with Wemos D1 mini boards which have allegedly 4MB of memory.
I am using the official 1.8 version released this week.

Any pointers welcome.

Best regards,

Llwy

[deshipu]

First of all, the "4M of memory" on Wemos refers to the flash memory, not the RAM that is needed to actually import and run the modules.
The operational memory available for your programs is right now quite limited. You can free some of it by not running the webrepl, and maybe by also deactivating the AP/STA interfaces -- whichever you are not using. But if you want this library to be general purpose, you should probably think about limiting the amount of memory it needs, by not creating too many objects and strings.

[pythoncoder]

In my experience on the Pyboard these problems can occur when the compiler runs out of memory. When you import a Python source module the compiler converts it to bytecode which can then be run by the Python VM. If the module is large, and especially if you've already imported other modules, the compiler can fail. My experience on the ESP8266 is limited but on the Pyboard there are several solutions:

• Soft-reset the board before each run.
• Import the large module before importing any others.
• Split the module into smaller ones.
• Precompile it using the cross compiler.
• Precompile it and freeze the bytecode into Flash.

The last is supremely effective. I appreciate that not all these options are (yet) available on the ESP8266 but frozen bytecode is, I gather, planned once other stretch goals have been achieved.

[Llwy]

Thank you Deshipu and Pythoncoder.

I had kind of hoped that it was some sort of bug instead of a real memory issue.

I do not have much experience in low-memory programming so there is possibly room for improvement.

I will try with the webrepl deactivated and cross my fingers for it to be sufficient.

Otherwise, I will have to wait for the availability of frozen bytecode as the whole purpose of using an ESP8266 would be defeated if I had to turn off STA mode.

Does anyone know about a good tutorial on how to optimize code to reduce memory usage for micropython?
For instance, would editing out the comments in the module source code lower the memory burden or are those discarded automatically during the module parsing?


Thanks again,

Llwy

[deshipu]

There was recently some discussion here: http://forum.micropython.org/viewtopic.php?f=2&t=1766

[Llwy]

OK so I removed webrepl from boot.py but I still do not have enough memory to load my module.
I guess I will simply have to shelve the whole project until I the frozen bytecode becomes usable on the ESP.

Best regards,

Llwy

[jwissing]

OK so I removed webrepl from boot.py but I still do not have enough memory to load my module.
I guess I will simply have to shelve the whole project until I the frozen bytecode becomes usable on the ESP.

Best regards,

Llwy

I had the same issue with another module. I got it to import with increased heap size.
If you have a build environment you could give it a try.
http://forum.micropython.org/viewtopic.php?f=16&t=1878

[Llwy]

I tried to split my module like Pythoncoder suggested and stored all the static values in a separate module:

Code: Select all

MAX_LEN = const(16)

PCD_IDLE = const(0x00)
PCD_AUTHENT = const(0x0E)
PCD_RECEIVE = const(0x08)
PCD_TRANSMIT = const(0x04)
PCD_TRANSCEIVE = const(0x0C)
PCD_RESETPHASE = const(0x0F)
PCD_CALCCRC = const(0x03)

PICC_REQIDL = const(0x26)
PICC_REQALL = const(0x52)
PICC_ANTICOLL = const(0x93)
PICC_SElECTTAG = const(0x93)
PICC_AUTHENT1A = const(0x60)
PICC_AUTHENT1B = const(0x61)
PICC_READ = const(0x30)
PICC_WRITE = const(0xA0)
PICC_DECREMENT = const(0xC0)
PICC_INCREMENT = const(0xC1)
PICC_RESTORE = const(0xC2)
PICC_TRANSFER = const(0xB0)
PICC_HALT = const(0x50)

MI_OK = const(0)
MI_NOTAGERR = const(1)
MI_ERR = const(2)

Reserved00 = const(0x00)
CommandReg = const(0x01)
CommIEnReg = const(0x02)
DivlEnReg = const(0x03)
CommIrqReg = const(0x04)
DivIrqReg = const(0x05)
ErrorReg = const(0x06)
Status1Reg = const(0x07)
Status2Reg = const(0x08)
FIFODataReg = const(0x09)
FIFOLevelReg = const(0x0A)
WaterLevelReg = const(0x0B)
ControlReg = const(0x0C)
BitFramingReg = const(0x0D)
CollReg = const(0x0E)
Reserved01 = const(0x0F)

Reserved10 = const(0x10)
ModeReg = const(0x11)
TxModeReg = const(0x12)
RxModeReg = const(0x13)
TxControlReg = const(0x14)
TxAutoReg = const(0x15)
TxSelReg = const(0x16)
RxSelReg = const(0x17)
RxThresholdReg = const(0x18)
DemodReg = const(0x19)
Reserved11 = const(0x1A)
Reserved12 = const(0x1B)
MifareReg = const(0x1C)
Reserved13 = const(0x1D)
Reserved14 = const(0x1E)
SerialSpeedReg = const(0x1F)

Reserved20 = const(0x20)
CRCResultRegM = const(0x21)
CRCResultRegL = const(0x22)
Reserved21 = const(0x23)
ModWidthReg = const(0x24)
Reserved22 = const(0x25)
RFCfgReg = const(0x26)
GsNReg = const(0x27)
CWGsPReg = const(0x28)
ModGsPReg = const(0x29)
TModeReg = const(0x2A)
TPrescalerReg = const(0x2B)
TReloadRegH = const(0x2C)
TReloadRegL = const(0x2D)
TCounterValueRegH = const(0x2E)
TCounterValueRegL = const(0x2F)

Reserved30 = const(0x30)
TestSel1Reg = const(0x31)
TestSel2Reg = const(0x32)
TestPinEnReg = const(0x33)
TestPinValueReg = const(0x34)
TestBusReg = const(0x35)
AutoTestReg = const(0x36)
VersionReg = const(0x37)
AnalogTestReg = const(0x38)
TestDAC1Reg = const(0x39)
TestDAC2Reg = const(0x3A)
TestADCReg = const(0x3B)
Reserved31 = const(0x3C)
Reserved32 = const(0x3D)
Reserved33 = const(0x3E)
Reserved34 = const(0x3F)

I used the "const" function as the documentation mentioned that it helped reducing the RAM usage

This module however with 88 static values of 1 byte each still cannot get imported and I get a MemoryError.

I read the threads mentioning frozen bytecode for fonts and it looks like this module would be a good candidate for that feature when it is available but I am still surprised at how quickly I ran into memory issues when it did not feel that I was dealing with very complicated code. I still have a lot of things to learn.

edit: I tried to paste directly the content of the above module into the REPL and it could parse it without issue. All the constants could be defined without a memory error.
What is the difference between pasting directly into the REPL and importing the same instructions as a module when it comes to memory?

I had the same issue with another module. I got it to import with increased heap size.
If you have a build environment you could give it a try.
http://forum.micropython.org/viewtopic.php?f=16&t=1878

I will look into the possibility of increasing the heap size. Maybe it is a stupid question but if the heap can be increased in this way, why is it not set to its maximum size by default? Does increasing the heap size come to the detriment of some other features?


Thank you once again everyone,

Llwy

[deshipu]

I'm afraid const() will only save memory when used within the same module. If you use it from another module, it has to live in memory.

[Llwy]

I see,
So that might explain why I can paste the code directly in the REPL but it fails to import in a module if it cannot benefit from the const() optimization.

Llwy