enable polymorphism for C-modules

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
jickster
Posts: 533
Joined: Thu Sep 07, 2017 8:57 pm

enable polymorphism for C-modules

Post by jickster » Sun Jul 01, 2018 8:44 pm

I'm creating a CAN API.

My hierarchy (in Python) looks like this:

Code: Select all

device.CAN.
    queue
    CAN1
    message
    CAN2
Also, I'm creating this hierarchy completely dynamically so that the API reflects the underlying capabilities so if a HW module has a SPI instead of CAN bus, then it would look like device.SPI.<SPI stuff>.

In C, "device" is "mp_obj_module_t" as is "CAN".

The problem is that currently, if the user says "device.CAN.queue = 55" then that'll replace "queue" with "55" . . . which is very bad because now they can no longer create queues to read from the CAN channel until they reboot the system.
Also, "del device.CAN.CAN1" will also "work" i.e. it'll delete "device.CAN.CAN1" which again, is very bad.

Code: Select all

const mp_obj_type_t mp_type_module = {
    { &mp_type_type },
    .name = MP_QSTR_module,
    .print = module_print,
    .attr = module_attr,
};
"module_attr" is called in all of these cases and that's what I need to override.

Is there a way for me to define a new type in C so that:
(1) I can override one or more functions but inherit the ones defined in the "base" type?
In my case, I'd like to override "mp_type_module.attr"
(2) MP_OBJ_IS_TYPE(obj, &mp_type_module) returns TRUE if object "obj" is "inherited" from &mp_type_module
I had hoped const void * struct _mp_obj_type_t::parent; would do what I need but it turns out this field is only used for types created in ".py" code.


I already know (2) is not possible using current definition of MP_OBJ_IS_TYPE(o, t) because
#define MP_OBJ_IS_TYPE(o, t) (MP_OBJ_IS_OBJ(o) && (((mp_obj_base_t*)MP_OBJ_TO_PTR(o))->type == (t)))
this macro simply compares the type-ptr without doing some kind of search in the hierarchy.

I know that for the case of mp_type_module.attr, I could "lock" it by setting dict->map.is_fixed=1 after I'm done creating the hierarchy but that's not a general solution.
Last edited by jickster on Thu Jul 05, 2018 3:11 pm, edited 1 time in total.

stijn
Posts: 274
Joined: Thu Apr 24, 2014 9:13 am

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by stijn » Mon Jul 02, 2018 6:43 pm

jickster wrote:
Sun Jul 01, 2018 8:44 pm
"module_attr" is called in all of these cases and that's what I need to override.
Even if you do that users can still do, for example, device.CAN.queue.pop = 0 and now they can't pop from the queue anymore. And there are probably other things which allow people to screw up in other ways as well. That's what you get with Python (and others). But maybe if your users do things like that there are more pertinent problems than trying to deal with that in code: how are those users going to write a basic script which doesn't mess up? After all you expose something rather low-level like CAN to them - if they're able to work with that correctly you can expect them to not assign 55 to a random name in a module. And would they do it anyway they won't do it twice. I hope. Whereas if they do that I wouldn't really expect them to be able to deal with a communication protocol. Like, making sure not to let the queue get full and whatnot.

Anyway I guess if you really wanted you could cast away the constness of mp_type_module and replace it's attr with your own. Defining a new type based on mp_type_module and using that whenever something gets imported is something else though.

jickster
Posts: 533
Joined: Thu Sep 07, 2017 8:57 pm

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by jickster » Mon Jul 02, 2018 6:45 pm

stijn wrote:
Mon Jul 02, 2018 6:43 pm
jickster wrote:
Sun Jul 01, 2018 8:44 pm
"module_attr" is called in all of these cases and that's what I need to override.
Even if you do that users can still do, for example, device.CAN.queue.pop = 0
That can't happen because the function can_message_type.attr does not allow delete at all so that's literally impossible.

jickster
Posts: 533
Joined: Thu Sep 07, 2017 8:57 pm

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by jickster » Mon Jul 02, 2018 6:47 pm

stijn wrote:
Mon Jul 02, 2018 6:43 pm
jickster wrote:
Sun Jul 01, 2018 8:44 pm
"module_attr" is called in all of these cases and that's what I need to override.

Anyway I guess if you really wanted you could cast away the constness of mp_type_module and replace it's attr with your own. Defining a new type based on mp_type_module and using that whenever something gets imported is something else though.
(A) I don't want to replace the definitions of original types.
(B) I already have a proposal which implements C-based inheritance and overriding without touching original types

https://github.com/micropython/micropython/issues/3910

stijn
Posts: 274
Joined: Thu Apr 24, 2014 9:13 am

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by stijn » Tue Jul 03, 2018 9:24 am

jickster wrote:
Mon Jul 02, 2018 6:45 pm
That can't happen because the function can_message_type.attr does not allow delete at all so that's literally impossible.
Are you sure? Firstly there's no deletion going on, an attribute is being stored here, that's not exactly the same. But that doesn't even matter as far as I know because your type's attr is not invoked here: the example I gave was device.CAN.queue.pop = 0. I'm assuming device.CAN is a module and queue is a class which you defined in C and pop is a function of that class. So what happens is uPy calls type_attr to store a 0 in the type's local_dict. Hereby overwriting whatever was there before.

jickster
Posts: 533
Joined: Thu Sep 07, 2017 8:57 pm

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by jickster » Tue Jul 03, 2018 3:18 pm

stijn wrote:
Tue Jul 03, 2018 9:24 am
jickster wrote:
Mon Jul 02, 2018 6:45 pm
That can't happen because the function can_message_type.attr does not allow delete at all so that's literally impossible.
Are you sure?
I'm the one who wrote can_message_type.attr so yes I'm sure I know what it's doing.
The .attr function is called whenever user attempts to add/modify/delete an attribute.
stijn wrote:
Mon Jul 02, 2018 6:43 pm
jickster wrote:
Sun Jul 01, 2018 8:44 pm
"module_attr" is called in all of these cases and that's what I need to override.
Even if you do that users can still do, for example, device.CAN.queue.pop = 0 and now they can't pop from the queue anymore.
Yes and no. The current code doesn't check if the .map is fixed which could lead to a processor exception when ROM is written to.
This is a bug.

stijn
Posts: 274
Joined: Thu Apr 24, 2014 9:13 am

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by stijn » Tue Jul 03, 2018 6:28 pm

jickster wrote:
Tue Jul 03, 2018 3:18 pm
I'm the one who wrote can_message_type.attr so yes I'm sure I know what it's doing.
The .attr function is called whenever user attempts to add/modify/delete an attribute.
Only for instance attributes (as in, you create an object of the type and assign/delete attributes), not class attributes (assigning attributes to the the type itself). My point is your attr function is not getting called in the example given. At least that's the behaviour I'm seeing here, for example Timer is a class with a function Elapsed, defined in C in a module timer:

Code: Select all

>>> import timer
>>> timer.Timer().Elapsed()  # create instance, call function
0.002
>>> timer.Timer().Elapsed = 0  # my attr function doesn't allow store so raises an exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Timer' object has no attribute 'Elapsed'
>>> timer.Timer.Elapsed = 0  # this overwrites my function with 0, attr is not called since there is no instance
>>> timer.Timer().Elapsed()  # fails because 0 is not callable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable

jickster
Posts: 533
Joined: Thu Sep 07, 2017 8:57 pm

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by jickster » Tue Jul 03, 2018 6:44 pm

stijn wrote:
Tue Jul 03, 2018 6:28 pm
jickster wrote:
Tue Jul 03, 2018 3:18 pm
I'm the one who wrote can_message_type.attr so yes I'm sure I know what it's doing.
The .attr function is called whenever user attempts to add/modify/delete an attribute.
Only for instance attributes (as in, you create an object of the type and assign/delete attributes), not class attributes (assigning attributes to the the type itself). My point is your attr function is not getting called in the example given. At least that's the behaviour I'm seeing here, for example Timer is a class with a function Elapsed, defined in C in a module timer:

Code: Select all

>>> import timer
>>> timer.Timer().Elapsed()  # create instance, call function
0.002
>>> timer.Timer().Elapsed = 0  # my attr function doesn't allow store so raises an exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Timer' object has no attribute 'Elapsed'
>>> timer.Timer.Elapsed = 0  # this overwrites my function with 0, attr is not called since there is no instance
>>> timer.Timer().Elapsed()  # fails because 0 is not callable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
The reason why timer.Timer.Elapsed can be modified is apparent in

Code: Select all

STATIC void type_attr(mp_obj_t self_in, qstr attr, mp_obj_t *dest)
: the .is_fixed flag is never checked. There's an assert in mp_map_elem_t *mp_map_lookup(mp_map_t *map, mp_obj_t index, mp_map_lookup_kind_t lookup_kind) but that assert only gets hit if you're using a debug build which is incorrect behavior: if a map is fixed, why would you only want to NOT modify it if you're NOT in debug mode???

Also, the only way your assignment even worked is because you did not put your timer.locals_dict in ROM using

Code: Select all

#define MP_DEFINE_CONST_DICT(dict_name, table_name)
When I try such an assignment - change the .map defined in ROM - I get a processor exception . . . which is expected because I'm modifying ROM but incorrect behavior: if .is_fixed == 1, why would you try to modify the .map ?????

jickster
Posts: 533
Joined: Thu Sep 07, 2017 8:57 pm

Re: how to inherit and override the C-methods of an mp_type_* in C

Post by jickster » Thu Jul 05, 2018 3:05 pm

the .is_fixed flag is fixed in a commit

https://github.com/micropython/micropyt ... -402323654

HOWEVER there is still no builtin way to disallow modules from overriding members while still keeping the same type as mp_type_module

BUT I do have a suggestion: https://github.com/micropython/micropython/issues/3910

Post Reply