Monkey patching built-in modules

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Monkey patching built-in modules

Post by cefn » Mon Feb 26, 2018 1:09 pm

Hi all,

I would like to figure out how to monkey-patch modules to help with this, but literally assigning to modules just leads to e.g.

Code: Select all

paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== from ucollections import namedtuple
=== _struct_time = namedtuple("struct_time", ("tm_year", "tm_mon", "tm_mday", "tm_hour", "tm_min", "tm_sec", "tm_wday", "tm_yday", "tm_isdst"))
=== def struct_time(tm_year, tm_mon, tm_mday, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=-1, tm_yday=-1, tm_isdst=-1):
===     return _struct_time(tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_wday, tm_yday, tm_isdst)
=== 
>>> import time
>>> setattr(time, "struct_time", struct_time)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'struct_time'
I've been trying to make it possible for CircuitPython modules to run on standard Micropython images, to give access to the increasing pool of consistently-managed and documented driver modules downstream e.g. at https://github.com/adafruit/Adafruit_Ci ... hon_Bundle

Seems like a compatibility layer is mostly feasible, but of course there is a huge value to this being seamless and not requiring a specialised bundle to be distributed with tweaked modules. The experiment is to see if people can be enabled to use the regular module source instead, but relying on extra attributes having been inserted beforehand.

To achieve this, I am wondering what I can do to insert functions into system modules where necessary (this would be done before importing libraries depending on them). For example if an implementation of struct_time could be made available to a module like https://github.com/adafruit/Adafruit_Ci ... uit_gps.py then it should be able to run OK, but I can't control how the module imports struct_time without modifying the module, (which is what I'm trying to avoid).

I acknowledge that even if a wrapper approach could be found, then this runtime monkey-patching approach eats away at limited RAM, (although this could end up being academic in the case of 4Meg PSRAM ESP32 modules which are on their way into our ecosystem).
Last edited by cefn on Mon Feb 26, 2018 2:31 pm, edited 1 time in total.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Monkey patching built-in modules

Post by stijn » Mon Feb 26, 2018 2:13 pm

Assigning to modules only works for modules written in python and custom modules written C depending on how they are constructed, but unfortunately for you it does not work for most (all?) modules implemented in C in the main repository. Not sure how familiar you are with the internals, but module definitions typically look like

Code: Select all

STATIC const mp_rom_map_elem_t mp_module_time_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_utime) },
    { MP_ROM_QSTR(MP_QSTR_clock), MP_ROM_PTR(&mod_time_clock_obj) }
    //etc
};

STATIC MP_DEFINE_CONST_DICT(mp_module_time_globals, mp_module_time_globals_table);

const mp_obj_module_t mp_module_time = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t*)&mp_module_time_globals,
};
and that MP_DEFINE_CONST_DICT creates a dict which cannot be modified later on, which is exactly the reason your setattr call fails. Modifying the source code to change that behaviour and for example optionally do allow all those dictionaries to be changed on the fly is not impossible but quite some work.

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Monkey patching built-in modules

Post by cefn » Mon Feb 26, 2018 2:43 pm

I presume there's no other tricks such as implementing a module which 'hides' the original module either. For example having a time.py in the local directory taking precedence over the system time module? Any import precedence rules which I could take advantage of here?

In fantasy-land I find myself wondering what is the least-worst way that Micropython could facilitate this without huge resource cost throughout. For example, allowing __import__ to be overridden, (at the cost of an extra lookup for every import line, allowing substitute module implementations to be provided, falling back to returning the fast module lookup and fast definitions). I speculate that import isn't hit very often and typically at startup. Of course, such a cost would only ever be considered if it opened up a lot of extension/wrapping possibilities, not just this nice-to-have. Perhaps there is a better place to intervene? However, this is basically off-topic as the aim is to build only on official micropython and I don't expect this cost/benefit would stack up, unless others have been considering it for more substantial reasons.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Monkey patching built-in modules

Post by stijn » Mon Feb 26, 2018 3:22 pm

For example having a time.py in the local directory taking precedence over the system time module?
not that, but you can do things like

Code: Select all

import sys
sys.modules['time'] = mymodule
import time  # won't actually import since it's considered imported already, points to mymodule
For example, allowing __import__ to be overridden
It is actually possible to override builtins, including __import__, but effectively using the overridden version is still on the TODO list it seems ("TODO lookup __import__ and call that instead of going straight to builtin implementation" in runtime.c). That doesn't seem like a major amount of work and probably worth it, cost of a coupl of extra statements vs amount of time it actually takes for most imports is negligible.

Post Reply