mpy-cross not making files smaller

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
formatc1702
Posts: 4
Joined: Fri Mar 10, 2017 3:17 pm

mpy-cross not making files smaller

Post by formatc1702 » Fri Mar 10, 2017 3:28 pm

Hello!

I have a MicroPython program consisting of multiple modules. The main.py file is currently at 304 lines and 9KB.

Recently I started getting MemoryErrors upon booting. If I comment out some lines (which are not even reached in the program flow), it works. This leads me to believe that main.py is getting too large for the interpreter to handle. Currently it is hard to modularize it further because it represents a rather large state machine.

First idea: use mpy-cross to create a main.mpy file, that should be much smaller and hog less RAM, right?
However, the resulting file is actually larger (.py=9224 bytes, .mpy=10407 bytes) and a lot of it looks like clear text. How is this possible?

Here is the offending file, in case it helps:
https://gist.github.com/formatc1702/0d8 ... 9f44d13f40

Any hints on why this is happening? Thanks a lot!

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: mpy-cross not making files smaller

Post by Roberthh » Fri Mar 10, 2017 7:28 pm

Hello @formatc1702, i compiled your file and the result is 9353 bytes, still a little bit larger than the source file, but small enough for the ESP8266. Are you compiling on a Windows machine? Then an old phenomenon may have hit you, in that NL (newline) is replaced by CR-NL, which would explain the increased size. I recall something like that was fixed in the past.
The benefit of cross-compiling list in the compile phase, in which MP typically runs out of memory.

P.S.: I have added a compiled copy of your file
main.mpy.zip
(2.98 KiB) Downloaded 309 times
Edit: I compiled your file on both Linux and Windows, and they are identical. So no CR-NL trap.

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: mpy-cross not making files smaller

Post by Damien » Fri Mar 10, 2017 10:27 pm

Compiled .mpy files can sometimes be large on disk, but usually when you load them into your program (ie import them) they take up a lot less RAM than the size of their file.

There are a few things to look out for:
  • .mpy files store the source file name of the original .py file, including the path, so it can give accurate traceback errors. If you compile a file with a long name then it'll take up more room in the file because every function needs to store the file name. Eg if you do "mpy-cross /path/to/my/script/which/is/very/long.py" then it stores that long path. To get around this you can use the -s option to mpy-cross which allows you to specify the filename to store, eg "mpy-cross -s main.py <path>".
  • .mpy files will store line number info by default. To disable this use the -O3 option to mpy-cross. This also disables debugging so the __debug__ constant is now false (-O3 saves about 300 bytes in your case).
  • .mpy files store a lot of redundant qstr info which is not actually loaded into RAM when they are imported. They do this to make the loading of .mpy files simple and use minimal RAM. A 10k .mpy file might only take 5k when loaded.

User avatar
marfis
Posts: 215
Joined: Fri Oct 31, 2014 10:29 am
Location: Zurich / Switzerland

Re: mpy-cross not making files smaller

Post by marfis » Sun Mar 12, 2017 8:43 pm

that's interesting.

So mpy-cross may be good for your RAM usage but not necessarily for your Flash storage. I guess it is a different story with the frozen bytecode (since that should not have the qstrs overhead)?

It adds another view when deciding betwen mpy-cross, frozen bytecode or plain text.

Could that information be added to the docs (copy&paste of damiens list is fine)?

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: mpy-cross not making files smaller

Post by pythoncoder » Mon Mar 13, 2017 8:51 am

Another option where flash space is at a premium is pyminifier https://liftoff.github.io/pyminifier/.
Peter Hinch
Index to my micropython libraries.

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: mpy-cross not making files smaller

Post by SpotlightKid » Mon Mar 13, 2017 10:42 am

My experience is that if I have a source file with normal (4 spaces) indentation, whitespace according to PEP-8 and medium amount of comments and docstrings, pyminifier will shrink the source substantially, but usually the resulting file from running mpy-cross on the same source is about the same size. Running mpy-cross on the minified file doesn't give any further file size reduction, the resulting file may even be a few bytes larger. So minified files can be helpful if you want to save flash space but still want to have the ability to quickly fix things by editing the file directly on the device.

Here are some examples where I have a normal and a minified version of each file:

https://github.com/SpotlightKid/micropy ... /midi/midi

formatc1702
Posts: 4
Joined: Fri Mar 10, 2017 3:17 pm

Re: mpy-cross not making files smaller

Post by formatc1702 » Mon Mar 13, 2017 3:00 pm

Hi everybody,

thanks for the insightful replies.
However, I failed to post the main issue I'm having: Once I compile my .py file and put the resulting .mpy file in its place (on the ESP8266), I receive a
ValueError: invalid .mpy file

No further information is provided, unfortunately. Has this happened to anyone else?

@Roberthh: I shall try your file later today and will report back!

I am compiling on OS X BTW, so the LF -> CRLF problem should not be the issue.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: mpy-cross not making files smaller

Post by Roberthh » Tue Mar 14, 2017 7:14 am

There was a change in the mpy file format recently. Please ensure, that both mpy-cross and the firmware come from the same build.

User avatar
fdushin
Posts: 32
Joined: Thu Jul 21, 2016 5:38 pm

Re: mpy-cross not making files smaller

Post by fdushin » Thu Mar 16, 2017 8:31 pm

I'd still recommend burning your code to an image, if you want to really keep the memory image down. For large applications, I have found that I need to do that, in order to not run out of memory at runtime.

formatc1702
Posts: 4
Joined: Fri Mar 10, 2017 3:17 pm

Re: mpy-cross not making files smaller

Post by formatc1702 » Sat Apr 15, 2017 10:13 am

There was a change in the mpy file format recently. Please ensure, that both mpy-cross and the firmware come from the same build.
Thanks! This did it for me!
I'd still recommend burning your code to an image, if you want to really keep the memory image down. For large applications, I have found that I need to do that, in order to not run out of memory at runtime.
I guess that's true.. but for now, cross-compiling the individual .py files to .mpy seems to be a good compromise between saving memory and iterating quickly, as the code is still changing quite a lot.

Thank you everyone for the awesome support!

Post Reply