Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Post by pfalcon » Mon Dec 31, 2018 5:42 pm

Bytecode is important part of MicroPython, just as of CPython. However, there was a limited infrastructure to deal with it, namely:
  • MicroPython Unix port can dump bytecode in assembly form with "-v -v" switches.
  • tools/mpy-tool.py from MicroPython distribution, despite its generic name, is quite adhoc tool to "freeze" bytecode files into executable.
I worked on making some generic, mostly CPython stdlib inspired modules for dealing with bytecode available via micropython-lib project:

  • opcode module which allows to convert from raw bytecode opcodes and their string names and vice versa, and means to classify bytecodes based on parameters they take. This module is based on CPython's opcode module, which is not documented in the official CPython documentation, but turns out it's part of the public API and used by various projects (e.g. PyPy). MicroPython's version is roughly, but not completely compatible with CPython.
  • mpylib is a module to load and write .mpy bytecode files. This module is adhoc for micropython-lib. But it tries to present result of loading by emulating Python's standard CodeType objects.
  • dis is a well-known CPython module allowing to disassemble bytecode (roughly, what "micropython -v -v" does). Currently, it works only with code objects loaded using the mpylib module above.

All this stuff is very experimental at the moment and subject to change. The idea is to make "opcode" and "dis" module as close as possible to to CPython's versions, and extend the feature set they cover (which will likely require changes to MicroPython itself, to be delivered via micropython-dev variant. This won't be fully possible, as MicroPython's bytecode and code object are different from CPython's. So, another idea is to provide suitable compatible, but expressive API for MicroPython-specific aspects.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Post by pfalcon » Mon Dec 31, 2018 6:01 pm

As an example of what's now possible, there's as an example script which can works both for MicroPython and CPython to dump/disassemble .mpy and .pyc files respectively. When run on .mpy produced from following source:

Code: Select all

print("loooooooooooooooong string")
gives following output:

Code: Select all

co_argcount 0
co_cellvars ()
co_code b'\x1b\x00\x00\x00\x17\x00d\x012\x11['
co_consts ('loooooooooooooooong string',)
co_filename a.py
co_flags 0
co_kwonlyargcount 0
co_name <module>
co_names ('print',)
co_stacksize 2
co_varnames ()
mpy_argnames ()
mpy_codeobjs ()
mpy_consts ('loooooooooooooooong string',)
mpy_def_pos_args 0
mpy_excstacksize 0
mpy_stacksize 2
              0 LOAD_NAME                0 (print)
              4 LOAD_CONST_OBJ           0 ('loooooooooooooooong string')
              6 CALL_FUNCTION            n=1 nkw=0
              8 POP_TOP                 
              9 LOAD_CONST_NONE         
             10 RETURN_VALUE            
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Post by pfalcon » Fri Feb 22, 2019 5:39 am

To mpylib, support for assembling bytecode instruction by instruction was added:

https://github.com/pfalcon/micropython- ... ite_mpy.py
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Post by pfalcon » Tue Mar 26, 2019 6:28 am

Work is under way to update these modules to .mpy format version 4, as recently pushed to MicroPython master.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Post by pfalcon » Tue Apr 16, 2019 7:24 am

Modules were updated for .mpy format version 4, at least in the part of "disassembling bytecode" part.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Post Reply