python-flavin - minimizing imports

[Neon22]

Python-Flavin is designed to take your final .py file and create a new copy of it containing only the minimal set of classes/functions/variables used.
I.e. its specifically useful for microcontrollers where unused code takes up valuable room.
Its designed to follow all nested imports as deeply as possible and prune away unused code.
It only works on modules(imports) written in python. Modules implemented in C do not get pruned.

Its not designed to do this perfectly but instead to attempt to remove all the unused aspects of imported modules.
I started this project a while back. I'm getting back into it and writing about it here.

E.g. if you imported the collection module because you wanted ordered dictionaries - then you want to remove all the other functions in that module that you aren't using. You also want to traverse the entire set of nested imports to prune away functions that collections (say) might import so it can do its work.

The process works like this:
- A modified version of vulture traverses all import hierarchies and then processes each file determining the unused functions, classes, and variables. (Vulture uses the wonderful ast (abstract syntax trees) module to parse the python code directly.)
- A modified version of Rope (a python refactoring library) will then be used to extract/move and rename the imports to minimize the code.

Currently the implementation is just starting and can:
- identify the unused functions and variables in the nested import tree.
- create copies of the imports ready for minimizing.

No code submitted yet on the refactoring, and not all import syntaxes can be processed yet.

I will update info here as time progresses.
https://github.com/Neon22/python-flavin

[torwag]

Hi,

this really sounds interesting esp. for micropython. If I understand it right, this will minimise the final bytecode for the modules to a minimum instead of providing the entire bytecode for a complete module which might contain 99% of unused code?!
Does that also means, that we might be a step further to use (tiny parts of) big modules like numpy or scipy on a microcontroller?
If you say it can't work on C (or any other) code (which is understandable reading about the approach), does it still only include "foreigner" libs, which are finally used?
I also wondering is it possible to create the bytecode on a hostmachine and copy it over to the uC once it is finished? I just try to figure out what is the most convenient workflow. Maybe a pyboard VM would be great (e.g., to simulate I/Os and peripherals), developing your code on a PC, using profiling tools and things like your python-flavin to analyse and optimise it and finally run it on the pyboard.
Maybe a debug mode for all modules which access uC functions and which "just" simulate responses (and if necessary timing) from those functions if running on a PC.
I did something like this for a I2C connection and it greatly helped me to debug my code.

This reminds me, we should really establish a good documentation about python code optimization (speed and memory footprint). Nowadays, running on several GB of RAM and with GHZ clocks, it seems not to be that much important (unlike you deal with a lot of data or make billions of calls), but going back to a uC this will be very important even for "trivial" tasks.
This should not only cover advanced methods like python-flavin, however, even basics and common pitfalls for beginners should be covered. I assume even python veterans, might need to relearn resp. reconsider the one or other thing, if you have to count suddenly in byte and kB again.

I will try to play around with python-flavin and try to get some numbers to support the stated features. Thanks

[Neon22]

These sound like great ideas.
I see there is a qemu folder appearing - maybe it can do some emulation.
As for bytecodes on the PC then transfer to the pyboard - I'm not sure of size advantage. A lot of the code ends up in flash.
I think you can spit out the bytecodes and thereby determine relative sizes. Maybe QEMU can help wih determinign if one is faster than the other - or you could just run it on the board.

As for python-flavin -new update soon.

[pfalcon]

> I also wondering is it possible to create the bytecode on a hostmachine and copy it over to the uC once it is finished?

No, see https://github.com/micropython/micropython/issues/222

> As for bytecodes on the PC then transfer to the pyboard - I'm not sure of size advantage. A lot of the code ends up in flash.

Not sure what is meant here, but uPy currently does not allow to store bytecode in flash (well, there's no persistent bytecode, that's why). So, everything is in RAM, and with some 60K Python module, you get out of Pyboard port heap just compiling it (unix port has by default heap size of almost the same size, and you can try by importing some CPython stdlib modules).


Otherwise, bytecode optimization is great idea, I have lotsa (well, at least some) ideas on that, but not enough time to even dump them. Great you guys started to discuss them.