Page 1 of 1

Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Posted: Mon Dec 31, 2018 5:42 pm
by pfalcon
Bytecode is important part of MicroPython, just as of CPython. However, there was a limited infrastructure to deal with it, namely:
  • MicroPython Unix port can dump bytecode in assembly form with "-v -v" switches.
  • tools/mpy-tool.py from MicroPython distribution, despite its generic name, is quite adhoc tool to "freeze" bytecode files into executable.
I worked on making some generic, mostly CPython stdlib inspired modules for dealing with bytecode available via micropython-lib project:

  • opcode module which allows to convert from raw bytecode opcodes and their string names and vice versa, and means to classify bytecodes based on parameters they take. This module is based on CPython's opcode module, which is not documented in the official CPython documentation, but turns out it's part of the public API and used by various projects (e.g. PyPy). MicroPython's version is roughly, but not completely compatible with CPython.
  • mpylib is a module to load and write .mpy bytecode files. This module is adhoc for micropython-lib. But it tries to present result of loading by emulating Python's standard CodeType objects.
  • dis is a well-known CPython module allowing to disassemble bytecode (roughly, what "micropython -v -v" does). Currently, it works only with code objects loaded using the mpylib module above.

All this stuff is very experimental at the moment and subject to change. The idea is to make "opcode" and "dis" module as close as possible to to CPython's versions, and extend the feature set they cover (which will likely require changes to MicroPython itself, to be delivered via micropython-dev variant. This won't be fully possible, as MicroPython's bytecode and code object are different from CPython's. So, another idea is to provide suitable compatible, but expressive API for MicroPython-specific aspects.

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Posted: Mon Dec 31, 2018 6:01 pm
by pfalcon
As an example of what's now possible, there's as an example script which can works both for MicroPython and CPython to dump/disassemble .mpy and .pyc files respectively. When run on .mpy produced from following source:

Code: Select all

print("loooooooooooooooong string")
gives following output:

Code: Select all

co_argcount 0
co_cellvars ()
co_code b'\x1b\x00\x00\x00\x17\x00d\x012\x11['
co_consts ('loooooooooooooooong string',)
co_filename a.py
co_flags 0
co_kwonlyargcount 0
co_name <module>
co_names ('print',)
co_stacksize 2
co_varnames ()
mpy_argnames ()
mpy_codeobjs ()
mpy_consts ('loooooooooooooooong string',)
mpy_def_pos_args 0
mpy_excstacksize 0
mpy_stacksize 2
              0 LOAD_NAME                0 (print)
              4 LOAD_CONST_OBJ           0 ('loooooooooooooooong string')
              6 CALL_FUNCTION            n=1 nkw=0
              8 POP_TOP                 
              9 LOAD_CONST_NONE         
             10 RETURN_VALUE            

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Posted: Fri Feb 22, 2019 5:39 am
by pfalcon
To mpylib, support for assembling bytecode instruction by instruction was added:

https://github.com/pfalcon/micropython- ... ite_mpy.py

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Posted: Tue Mar 26, 2019 6:28 am
by pfalcon
Work is under way to update these modules to .mpy format version 4, as recently pushed to MicroPython master.

Re: Bytecode manipulation infrastructure for MicroPython - modules opcode, dis, mpylib

Posted: Tue Apr 16, 2019 7:24 am
by pfalcon
Modules were updated for .mpy format version 4, at least in the part of "disassembling bytecode" part.