Persistent bytecode

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
User avatar
pythoncoder
Posts: 3664
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Persistent bytecode

Post by pythoncoder » Thu Apr 14, 2016 10:35 am

@Damien has now merged this awesome feature. It enables Python modules to be cross-compiled to bytecode on a PC, with the resultant bytecode being incorporated into a firmware build. You can then import the module as if it were stored on disk.

It enables modules containing large constant data structures to be accessed with minimal RAM usage. By avoiding the need for runtime compilation it also reduces the time required to import a module.

Limitations (as I understand it): constant data such as floats and tuples can't be cross-compiled, nor can native and Viper code.

To use it, first ensure you can successfully clone the Micropython repo and build and deploy a standard firmware build (the README files in the project root and in stmhal are your friend).

The following instructions assume Linux on a PC and source dated 15th April 2016 or later. Navigate to the micropython source directory. Build the cross compiler with

Code: Select all

make -C mpy-cross
Navigate to stmhal. Create a subdirectory 'scripts' and copy the Python modules to be cross compiled there. First check that your files will successfully cross-compile (assumes we're compiling fonts.py):

Code: Select all

../mpy-cross/mpy-cross scripts/fonts.py
ls -l scripts
If successful the listing will show a compiled file with a .mpy extension.

Then issue (adapt the following for your board type):

Code: Select all

make BOARD=PYBV10 clean
make BOARD=PYBV10 FROZEN_MPY_DIR=scripts
Put the Pyboard into DFU mode and issue

Code: Select all

sudo make BOARD=PYBV10 deploy
Last edited by pythoncoder on Fri Apr 22, 2016 12:09 pm, edited 1 time in total.
Peter Hinch

User avatar
Roberthh
Posts: 1705
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Persistent bytecode

Post by Roberthh » Sat Apr 16, 2016 11:33 am

Thanks @pythoncoder. These are good instructions. By not following them literally, I ran into an issue.
mpy_cross creates symbols form the source filenames. If these contains '.' characters, like ../sample.py, these dots will go into the mpy file, and further on into the frozen_mpy.c file, which then does not compile.
The problem was, that if after the trial run of mpy_cross the file still exist, the will not be created again, and then .. boom.
Regards, Robert

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: Persistent bytecode

Post by kfricke » Sat Apr 16, 2016 12:02 pm

Is this feature pyboard specific?

User avatar
pythoncoder
Posts: 3664
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Persistent bytecode

Post by pythoncoder » Sun Apr 17, 2016 9:06 am

@Roberthh Perhaps you should raise an issue?

@kfricke It's STM specific, as I understand it. I don't know if it's Pyboard specific - perhaps someone with different STM hardware could test, or @Damien might elucidate?
Peter Hinch

subutai6
Posts: 5
Joined: Fri Nov 11, 2016 5:25 pm

Re: Persistent bytecode

Post by subutai6 » Mon Nov 14, 2016 2:15 pm

Hi, I tried the procedure as explained in the tutorial.
If I try to cross-compile using

Code: Select all

../mpy-cross/mpy-cross scripts/myscript.py
ls -l scripts
It works.
Then when I issue:

Code: Select all

make BOARD=PYBV11 clean
make BOARD=PYBV11 FROZEN_MPY_DIR=scripts

I get the following error:

Code: Select all

find: -printf: unknown primary or operator
Creating build-PYBV11/frozen_mpy.c
usage: mpy-tool.py [-h] [-d] [-f] [-q QSTR_HEADER]
                   [-mlongint-impl {none,longlong,mpz}] [-mmpz-dig-size N]
                   files [files ...]
mpy-tool.py: error: too few arguments
make: *** [build-PYBV11/frozen_mpy.c] Error 2
make: *** Deleting file `build-PYBV11/frozen_mpy.c'
What's the missing argument?

Thanks

User avatar
dhylands
Posts: 3241
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Persistent bytecode

Post by dhylands » Mon Nov 14, 2016 4:53 pm

It would be useful to know a few additional things:

1 - What host are you using? (windows, Mac, linux)

2 - You can get more verbose output from make by adding V=1 and then it should show us the command it was trying to execute.

subutai6
Posts: 5
Joined: Fri Nov 11, 2016 5:25 pm

Re: Persistent bytecode

Post by subutai6 » Mon Nov 14, 2016 6:02 pm

host: Mac

Code: Select all

make V=1 BOARD=PYBV11 FROZEN_MPY_DIR=scripts/
find: -printf: unknown primary or operator
python ../py/makeversionhdr.py build-PYBV11/genhdr/mpversion.h
Creating build-PYBV11/frozen_mpy.c
python ../tools/mpy-tool.py -f -q build-PYBV11/genhdr/qstrdefs.preprocessed.h  > build-PYBV11/frozen_mpy.c
usage: mpy-tool.py [-h] [-d] [-f] [-q QSTR_HEADER]
                   [-mlongint-impl {none,longlong,mpz}] [-mmpz-dig-size N]
                   files [files ...]
mpy-tool.py: error: too few arguments
make: *** [build-PYBV11/frozen_mpy.c] Error 2
make: *** Deleting file `build-PYBV11/frozen_mpy.c'

User avatar
dhylands
Posts: 3241
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Persistent bytecode

Post by dhylands » Mon Nov 14, 2016 6:51 pm

I was able to reproduce your problem on my Mac Mini.

Under linux, the command being run is:

Code: Select all

find -L scripts -type f -name '*.py' -printf '%P\n'
and if I run that on my mac, I get this:

Code: Select all

519 >find -L scripts -type f -name '*.py' -printf '%P\n'
find: -printf: unknown primary or operator

User avatar
dhylands
Posts: 3241
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Persistent bytecode

Post by dhylands » Mon Nov 14, 2016 6:58 pm

I was able to change my py/mkrules.py around line 111 and changed:

Code: Select all

FROZEN_MPY_PY_FILES := $(shell find -L $(FROZEN_MPY_DIR) -type f -name '*.py' -printf '%P\n')
to read (all on one line):

Code: Select all

FROZEN_MPY_PY_FILES := $(shell find -L $(FROZEN_MPY_DIR) -type f -name '*.py' | sed -e 's=^$(FROZEN_MPY_DIR)/==')
and that worked for me. Could you test this?

If it works for you as well, then I'll submit a PR to fix it.

User avatar
dhylands
Posts: 3241
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Persistent bytecode

Post by dhylands » Mon Nov 14, 2016 9:00 pm


Post Reply