MicroPython Forum (Archive)

Hello!

I have a MicroPython program consisting of multiple modules. The main.py file is currently at 304 lines and 9KB.

Recently I started getting MemoryErrors upon booting. If I comment out some lines (which are not even reached in the program flow), it works. This leads me to believe that main.py is getting too large for the interpreter to handle. Currently it is hard to modularize it further because it represents a rather large state machine.

First idea: use mpy-cross to create a main.mpy file, that should be much smaller and hog less RAM, right?
However, the resulting file is actually larger (.py=9224 bytes, .mpy=10407 bytes) and a lot of it looks like clear text. How is this possible?

Here is the offending file, in case it helps:
https://gist.github.com/formatc1702/0d8 ... 9f44d13f40

Any hints on why this is happening? Thanks a lot!

Hello @formatc1702, i compiled your file and the result is 9353 bytes, still a little bit larger than the source file, but small enough for the ESP8266. Are you compiling on a Windows machine? Then an old phenomenon may have hit you, in that NL (newline) is replaced by CR-NL, which would explain the increased size. I recall something like that was fixed in the past.
The benefit of cross-compiling list in the compile phase, in which MP typically runs out of memory.

P.S.: I have added a compiled copy of your file
Edit: I compiled your file on both Linux and Windows, and they are identical. So no CR-NL trap.

Compiled .mpy files can sometimes be large on disk, but usually when you load them into your program (ie import them) they take up a lot less RAM than the size of their file.

There are a few things to look out for:

• .mpy files store the source file name of the original .py file, including the path, so it can give accurate traceback errors. If you compile a file with a long name then it'll take up more room in the file because every function needs to store the file name. Eg if you do "mpy-cross /path/to/my/script/which/is/very/long.py" then it stores that long path. To get around this you can use the -s option to mpy-cross which allows you to specify the filename to store, eg "mpy-cross -s main.py <path>".
• .mpy files will store line number info by default. To disable this use the -O3 option to mpy-cross. This also disables debugging so the __debug__ constant is now false (-O3 saves about 300 bytes in your case).
• .mpy files store a lot of redundant qstr info which is not actually loaded into RAM when they are imported. They do this to make the loading of .mpy files simple and use minimal RAM. A 10k .mpy file might only take 5k when loaded.

that's interesting.

So mpy-cross may be good for your RAM usage but not necessarily for your Flash storage. I guess it is a different story with the frozen bytecode (since that should not have the qstrs overhead)?

It adds another view when deciding betwen mpy-cross, frozen bytecode or plain text.

Could that information be added to the docs (copy&paste of damiens list is fine)?

Another option where flash space is at a premium is pyminifier https://liftoff.github.io/pyminifier/.

My experience is that if I have a source file with normal (4 spaces) indentation, whitespace according to PEP-8 and medium amount of comments and docstrings, pyminifier will shrink the source substantially, but usually the resulting file from running mpy-cross on the same source is about the same size. Running mpy-cross on the minified file doesn't give any further file size reduction, the resulting file may even be a few bytes larger. So minified files can be helpful if you want to save flash space but still want to have the ability to quickly fix things by editing the file directly on the device.

Here are some examples where I have a normal and a minified version of each file:

https://github.com/SpotlightKid/micropy ... /midi/midi

Hi everybody,

thanks for the insightful replies.
However, I failed to post the main issue I'm having: Once I compile my .py file and put the resulting .mpy file in its place (on the ESP8266), I receive a
ValueError: invalid .mpy file

No further information is provided, unfortunately. Has this happened to anyone else?

@Roberthh: I shall try your file later today and will report back!

I am compiling on OS X BTW, so the LF -> CRLF problem should not be the issue.

There was a change in the mpy file format recently. Please ensure, that both mpy-cross and the firmware come from the same build.

I'd still recommend burning your code to an image, if you want to really keep the memory image down. For large applications, I have found that I need to do that, in order to not run out of memory at runtime.

There was a change in the mpy file format recently. Please ensure, that both mpy-cross and the firmware come from the same build.

Thanks! This did it for me!

I'd still recommend burning your code to an image, if you want to really keep the memory image down. For large applications, I have found that I need to do that, in order to not run out of memory at runtime.

I guess that's true.. but for now, cross-compiling the individual .py files to .mpy seems to be a good compromise between saving memory and iterating quickly, as the code is still changing quite a lot.

Thank you everyone for the awesome support!