Page 1 of 2
Persistent bytecode
Posted: Thu Apr 14, 2016 10:35 am
by pythoncoder
@Damien has now merged this awesome feature. It enables Python modules to be cross-compiled to bytecode on a PC, with the resultant bytecode being incorporated into a firmware build. You can then import the module as if it were stored on disk.
It enables modules containing large constant data structures to be accessed with minimal RAM usage. By avoiding the need for runtime compilation it also reduces the time required to import a module.
Limitations (as I understand it): constant data such as floats and tuples can't be cross-compiled, nor can native and Viper code.
To use it, first ensure you can successfully clone the Micropython repo and build and deploy a standard firmware build (the README files in the project root and in stmhal are your friend).
The following instructions assume Linux on a PC and source dated 15th April 2016 or later. Navigate to the micropython source directory. Build the cross compiler with
Navigate to stmhal. Create a subdirectory 'scripts' and copy the Python modules to be cross compiled there. First check that your files will successfully cross-compile (assumes we're compiling fonts.py):
Code: Select all
../mpy-cross/mpy-cross scripts/fonts.py
ls -l scripts
If successful the listing will show a compiled file with a .mpy extension.
Then issue (adapt the following for your board type):
Code: Select all
make BOARD=PYBV10 clean
make BOARD=PYBV10 FROZEN_MPY_DIR=scripts
Put the Pyboard into DFU mode and issue
Re: Persistent bytecode
Posted: Sat Apr 16, 2016 11:33 am
by Roberthh
Thanks @pythoncoder. These are good instructions. By not following them literally, I ran into an issue.
mpy_cross creates symbols form the source filenames. If these contains '.' characters, like ../sample.py, these dots will go into the mpy file, and further on into the frozen_mpy.c file, which then does not compile.
The problem was, that if after the trial run of mpy_cross the file still exist, the will not be created again, and then .. boom.
Regards, Robert
Re: Persistent bytecode
Posted: Sat Apr 16, 2016 12:02 pm
by kfricke
Is this feature pyboard specific?
Re: Persistent bytecode
Posted: Sun Apr 17, 2016 9:06 am
by pythoncoder
@Roberthh Perhaps you should raise an issue?
@kfricke It's STM specific, as I understand it. I don't know if it's Pyboard specific - perhaps someone with different STM hardware could test, or @Damien might elucidate?
Re: Persistent bytecode
Posted: Mon Nov 14, 2016 2:15 pm
by subutai6
Hi, I tried the procedure as explained in the tutorial.
If I try to cross-compile using
Code: Select all
../mpy-cross/mpy-cross scripts/myscript.py
ls -l scripts
It works.
Then when I issue:
Code: Select all
make BOARD=PYBV11 clean
make BOARD=PYBV11 FROZEN_MPY_DIR=scripts
I get the following error:
Code: Select all
find: -printf: unknown primary or operator
Creating build-PYBV11/frozen_mpy.c
usage: mpy-tool.py [-h] [-d] [-f] [-q QSTR_HEADER]
[-mlongint-impl {none,longlong,mpz}] [-mmpz-dig-size N]
files [files ...]
mpy-tool.py: error: too few arguments
make: *** [build-PYBV11/frozen_mpy.c] Error 2
make: *** Deleting file `build-PYBV11/frozen_mpy.c'
What's the missing argument?
Thanks
Re: Persistent bytecode
Posted: Mon Nov 14, 2016 4:53 pm
by dhylands
It would be useful to know a few additional things:
1 - What host are you using? (windows, Mac, linux)
2 - You can get more verbose output from make by adding V=1 and then it should show us the command it was trying to execute.
Re: Persistent bytecode
Posted: Mon Nov 14, 2016 6:02 pm
by subutai6
host: Mac
Code: Select all
make V=1 BOARD=PYBV11 FROZEN_MPY_DIR=scripts/
find: -printf: unknown primary or operator
python ../py/makeversionhdr.py build-PYBV11/genhdr/mpversion.h
Creating build-PYBV11/frozen_mpy.c
python ../tools/mpy-tool.py -f -q build-PYBV11/genhdr/qstrdefs.preprocessed.h > build-PYBV11/frozen_mpy.c
usage: mpy-tool.py [-h] [-d] [-f] [-q QSTR_HEADER]
[-mlongint-impl {none,longlong,mpz}] [-mmpz-dig-size N]
files [files ...]
mpy-tool.py: error: too few arguments
make: *** [build-PYBV11/frozen_mpy.c] Error 2
make: *** Deleting file `build-PYBV11/frozen_mpy.c'
Re: Persistent bytecode
Posted: Mon Nov 14, 2016 6:51 pm
by dhylands
I was able to reproduce your problem on my Mac Mini.
Under linux, the command being run is:
Code: Select all
find -L scripts -type f -name '*.py' -printf '%P\n'
and if I run that on my mac, I get this:
Code: Select all
519 >find -L scripts -type f -name '*.py' -printf '%P\n'
find: -printf: unknown primary or operator
Re: Persistent bytecode
Posted: Mon Nov 14, 2016 6:58 pm
by dhylands
I was able to change my py/mkrules.py around line 111 and changed:
Code: Select all
FROZEN_MPY_PY_FILES := $(shell find -L $(FROZEN_MPY_DIR) -type f -name '*.py' -printf '%P\n')
to read (all on one line):
Code: Select all
FROZEN_MPY_PY_FILES := $(shell find -L $(FROZEN_MPY_DIR) -type f -name '*.py' | sed -e 's=^$(FROZEN_MPY_DIR)/==')
and that worked for me. Could you test this?
If it works for you as well, then I'll submit a PR to fix it.
Re: Persistent bytecode
Posted: Mon Nov 14, 2016 9:00 pm
by dhylands