implementing properties in user modules

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
v923z
Posts: 110
Joined: Mon Dec 28, 2015 6:19 pm

implementing properties in user modules

Post by v923z » Fri Feb 14, 2020 11:39 pm

Hi all,

What is the proper way of implementing properties in user modules? objproperty.c doesn't have a matching header file, so it is not clear, how I am supposed to user properties. Attributes won't do, because they override the locals dictionary.

I would appreciate any hints.

Cheers,

Zoltán

User avatar
jimmo
Posts: 1204
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: implementing properties in user modules

Post by jimmo » Sat Feb 15, 2020 1:09 am

MicroPython doesn't use properties anywhere in its built-in modules. objproperty.c exists only to provide the @property decorator for use in Python code.
v923z wrote:
Fri Feb 14, 2020 11:39 pm
objproperty.c doesn't have a matching header file, so it is not clear, how I am supposed to user properties.
All you need is the definition of the mp_obj_property_t struct, so just define it yourself:

Code: Select all

typedef struct _mp_obj_property_t {
    mp_obj_base_t base;
    mp_obj_t proxy[3]; // getter, setter, deleter
} mp_obj_property_t;
at the top of your C file (or just make your own objproperty.h), then you can define a property like this:

Code: Select all

STATIC mp_obj_t foo_obj_get_value(mp_obj_t self_in) {
    return ...;
}
MP_DEFINE_CONST_FUN_OBJ_1(foo_obj_get_value_obj, foo_obj_get_value);

STATIC mp_obj_t foo_obj_set_value(mp_obj_t self_in, mp_obj_t value) {
    ...
    return mp_const_none;
}
MP_DEFINE_CONST_FUN_OBJ_2(foo_obj_set_value_obj, foo_obj_set_value);

STATIC const mp_obj_property_t foo_value_obj = {
    .base.type = &mp_type_property,
    .proxy = {(mp_obj_t)&foo_obj_set_value,
              (mp_obj_t)&foo_obj_set_value_obj,
              (mp_obj_t)&mp_const_none_obj},
};
and in your locals dict

Code: Select all

  { MP_ROM_QSTR(MP_QSTR_value), MP_ROM_PTR(&foo_value_obj) },
(This is untested, but this is how properties created at runtime with @property work)

v923z
Posts: 110
Joined: Mon Dec 28, 2015 6:19 pm

Re: implementing properties in user modules

Post by v923z » Sat Feb 15, 2020 7:47 am

Hi Jim,

Thanks for the prompt reply! Unfortunately, that doesn't cut it. I should have posted my code at the very beginning, so we could've saved an extra round. Sorry for that.

In any case, here is my complete code

Code: Select all

#include <stdio.h>
#include "py/runtime.h"
#include "py/obj.h"

typedef struct _propertyclass_obj_t {
    mp_obj_base_t base;
    mp_float_t x;
} propertyclass_obj_t;

const mp_obj_type_t propertyclass_type;

// lifted from objproperty.c
typedef struct _mp_obj_property_t {
    mp_obj_base_t base;
    mp_obj_t proxy[3]; // getter, setter, deleter
} mp_obj_property_t;


STATIC mp_obj_t propertyclass_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
    mp_arg_check_num(n_args, n_kw, 1, 1, true);
    propertyclass_obj_t *self = m_new_obj(propertyclass_obj_t);
    self->base.type = &propertyclass_type;
    self->x = mp_obj_get_float(args[0]);
    return MP_OBJ_FROM_PTR(self);
}

STATIC mp_obj_t propertyclass_get_x(mp_obj_t self_in) {
    (void)self_in;
    printf("in propertyclass_get_x()\n");
    return mp_obj_new_float(1.0);
}

MP_DEFINE_CONST_FUN_OBJ_1(propertyclass_get_x_obj, propertyclass_get_x);

STATIC mp_obj_t propertyclass_set_x(mp_obj_t self_in, mp_obj_t value) {
    (void)self_in;
    (void)value;
    printf("in propertyclass_set_x()\n");
    return mp_const_none;
}

MP_DEFINE_CONST_FUN_OBJ_2(propertyclass_set_x_obj, propertyclass_set_x);

STATIC mp_obj_t propertyclass_del_x(mp_obj_t self_in) {
    (void)self_in;
    printf("in propertyclass_del_x()\n");
    return mp_const_none;
}

MP_DEFINE_CONST_FUN_OBJ_1(propertyclass_del_x_obj, propertyclass_del_x);

const mp_obj_property_t propertyclass_x_obj = {
    .base.type = &mp_type_property,
    .proxy = {(mp_obj_t)&propertyclass_get_x_obj,
              (mp_obj_t)&propertyclass_set_x_obj,
              (mp_obj_t)&propertyclass_del_x_obj},
};

STATIC mp_obj_t propertyclass_xsq(mp_obj_t self_in) {
    propertyclass_obj_t *self = MP_OBJ_TO_PTR(self_in);
    return mp_obj_new_float(self->x * self->x);
}

MP_DEFINE_CONST_FUN_OBJ_1(propertyclass_xsq_obj, propertyclass_xsq);

STATIC const mp_rom_map_elem_t propertyclass_locals_dict_table[] = {
    { MP_ROM_QSTR(MP_QSTR_xsq), MP_ROM_PTR(&propertyclass_xsq_obj) },
    { MP_ROM_QSTR(MP_QSTR_x), MP_ROM_PTR(&propertyclass_x_obj) },
};

STATIC MP_DEFINE_CONST_DICT(propertyclass_locals_dict, propertyclass_locals_dict_table);

const mp_obj_type_t propertyclass_type = {
    { &mp_type_type },
    .name = MP_QSTR_propertyclass,
    .make_new = propertyclass_make_new,
    .locals_dict = (mp_obj_dict_t*)&propertyclass_locals_dict,
};

STATIC const mp_map_elem_t propertyclass_globals_table[] = {
    { MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_propertyclass) },
    { MP_OBJ_NEW_QSTR(MP_QSTR_propertyclass), (mp_obj_t)&propertyclass_type },	
};

STATIC MP_DEFINE_CONST_DICT (
    mp_module_propertyclass_globals,
    propertyclass_globals_table
);

const mp_obj_module_t propertyclass_user_cmodule = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t*)&mp_module_propertyclass_globals,
};

MP_REGISTER_MODULE(MP_QSTR_propertyclass, propertyclass_user_cmodule, MODULE_PROPERTYCLASS_ENABLED);
and here is the output that it produces

Code: Select all

import propertyclass
a = propertyclass.propertyclass(12.3)

print(a.x)
print(a.xsq())

Code: Select all

<property>
151.29
It seems to me that the getter function is not even called (there is no printout). Either something is fundamentally wrong here, or there is a trivial error that I overlook...

Cheers,
Zoltán

stijn
Posts: 410
Joined: Thu Apr 24, 2014 9:13 am

Re: implementing properties in user modules

Post by stijn » Sat Feb 15, 2020 8:45 am

You do not need anything at all from objproperty.c, if you want a.x to get a value you can implement that in attr for instance (there might be more suitable ways, depends on specific use case). Pseudcode:

Code: Select all

static void attr(mp_obj_t self_in, qstr attr, mp_obj_t* dest) {
      if(dest[ 0 ] == MP_OBJ_NULL) { //it's a load
          if (attr == MP_QSTR_x) {
              return self_in->x;
          }
      }
}

propertyclasstype.attr = attr;

v923z
Posts: 110
Joined: Mon Dec 28, 2015 6:19 pm

Re: implementing properties in user modules

Post by v923z » Sat Feb 15, 2020 8:57 am

stijn wrote:
Sat Feb 15, 2020 8:45 am
You do not need anything at all from objproperty.c, if you want a.x to get a value you can implement that in attr for instance (there might be more suitable ways, depends on specific use case). Pseudcode:

Code: Select all

static void attr(mp_obj_t self_in, qstr attr, mp_obj_t* dest) {
      if(dest[ 0 ] == MP_OBJ_NULL) { //it's a load
          if (attr == MP_QSTR_x) {
              return self_in->x;
          }
      }
}

propertyclasstype.attr = attr;
I have tried this, though not in this way.

Code: Select all

STATIC mp_obj_t propertyclass_x(mp_obj_t self_in) {
    propertyclass_obj_t *self = MP_OBJ_TO_PTR(self_in);
    return mp_obj_new_float(self->x);
}

STATIC void propertyclass_attr(mp_obj_t self, qstr attribute, mp_obj_t *destination) {
    if(attribute == MP_QSTR_x) {
        destination[0] = propertyclass_x(self);
    } else if(attribute == MP_QSTR_y) {
        destination[0] = propertyclass_y(self);
    }
}

const mp_obj_type_t propertyclass_type = {
    { &mp_type_type },
    .name = MP_QSTR_propertyclass,
    .make_new = propertyclass_make_new,
    .attr = propertyclass_attr,
    .locals_dict = (mp_obj_dict_t*)&propertyclass_locals_dict,
};
That overrides the locals dictionary, so nothing defined in the locals will be available as a class method.

v923z
Posts: 110
Joined: Mon Dec 28, 2015 6:19 pm

Re: implementing properties in user modules

Post by v923z » Sat Feb 15, 2020 9:10 am

stijn wrote:
Sat Feb 15, 2020 8:45 am
You do not need anything at all from objproperty.c, if you want a.x to get a value you can implement that in attr for instance (there might be more suitable ways, depends on specific use case). Pseudcode:

Code: Select all

static void attr(mp_obj_t self_in, qstr attr, mp_obj_t* dest) {
      if(dest[ 0 ] == MP_OBJ_NULL) { //it's a load
          if (attr == MP_QSTR_x) {
              return self_in->x;
          }
      }
}

propertyclasstype.attr = attr;
I have tried this, though not in this way.

Code: Select all

STATIC mp_obj_t propertyclass_x(mp_obj_t self_in) {
    propertyclass_obj_t *self = MP_OBJ_TO_PTR(self_in);
    return mp_obj_new_float(self->x);
}

STATIC void propertyclass_attr(mp_obj_t self, qstr attribute, mp_obj_t *destination) {
    if(attribute == MP_QSTR_x) {
        destination[0] = propertyclass_x(self);
    } else if(attribute == MP_QSTR_y) {
        destination[0] = propertyclass_y(self);
    }
}

const mp_obj_type_t propertyclass_type = {
    { &mp_type_type },
    .name = MP_QSTR_propertyclass,
    .make_new = propertyclass_make_new,
    .attr = propertyclass_attr,
    .locals_dict = (mp_obj_dict_t*)&propertyclass_locals_dict,
};
That overrides the locals dictionary, so nothing defined in the locals will be available as a class method. Here is the complete implementation https://github.com/v923z/micropython-us ... ties.c#L11

User avatar
jimmo
Posts: 1204
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: implementing properties in user modules

Post by jimmo » Sat Feb 15, 2020 11:06 am

v923z wrote:
Sat Feb 15, 2020 7:47 am
Thanks for the prompt reply! Unfortunately, that doesn't cut it. I should have posted my code at the very beginning, so we could've saved an extra round. Sorry for that.
FWIW, CircuitPython is a very heavy user of properties (including for builtin types) so it's worth looking at how they do it.

The key is that they also have a patch to runtime.c (in mp_convert_member_lookup for get, and mp_store_attr for set) that makes this work for built-in types. (For Python types this already works via objtype.c mp_obj_instance_load_attr etc).

So, if you wanted to avoid needing to make core changes (I assume this is for ulab), then you could make a special "attr" function for all types that have property members, that would use the built-in attr lookup (mp_load_attr), then add the "is property -> use proxy" code from CircuitPython's modified mp_convert_member_lookup.

v923z
Posts: 110
Joined: Mon Dec 28, 2015 6:19 pm

Re: implementing properties in user modules

Post by v923z » Sat Feb 15, 2020 11:25 am

jimmo wrote:
Sat Feb 15, 2020 11:06 am
v923z wrote:
Sat Feb 15, 2020 7:47 am
Thanks for the prompt reply! Unfortunately, that doesn't cut it. I should have posted my code at the very beginning, so we could've saved an extra round. Sorry for that.
FWIW, CircuitPython is a very heavy user of properties (including for builtin types) so it's worth looking at how they do it.
Well, I am actually coming from the circuitpython side. You are correct in saying that this is for ulab. We were trying to establish a code base that could be compiled in both micropython, and circuitpython right away, because that would significantly reduce maintenance costs. They managed to implement properties, and I ran into difficulties, when I tried to merge their changes into micropython-ulab. Properties are the single outstanding issue, all others have been sorted out.
jimmo wrote:
Sat Feb 15, 2020 11:06 am

The key is that they also have a patch to runtime.c (in mp_convert_member_lookup for get, and mp_store_attr for set) that makes this work for built-in types. (For Python types this already works via objtype.c mp_obj_instance_load_attr etc).

So, if you wanted to avoid needing to make core changes (I assume this is for ulab), then you could make a special "attr" function for all types that have property members, that would use the built-in attr lookup (mp_load_attr), then add the "is property -> use proxy" code from CircuitPython's modified mp_convert_member_lookup.
Oh, that is interesting. I haven't expected that runtime.c was also modified to this end. So, I have been barking up the wrong tree :oops:
I will look into that, thanks for the pointer!

But I am actually wondering, was there a pressing reason for cutting back on this functionality in micropython? I think, it would be great, if this were solved in micropython proper in the long run. Such a workaround could be a show-stopper for external (user) modules that try to implement some sort of compatibility with CPython. Hacking around the language core is a pretty good way of writing fragile code ;)

User avatar
jimmo
Posts: 1204
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: implementing properties in user modules

Post by jimmo » Sat Feb 15, 2020 12:04 pm

v923z wrote:
Sat Feb 15, 2020 11:25 am
But I am actually wondering, was there a pressing reason for cutting back on this functionality in micropython? I think, it would be great, if this were solved in micropython proper in the long run. Such a workaround could be a show-stopper for external (user) modules that try to implement some sort of compatibility with CPython. Hacking around the language core is a pretty good way of writing fragile code
The overarching rule in the core MicroPython firmware is to save RAM and code size. Anything that can be done in Python (and is therefore opt-in by the user) is left to Python.

Properties are a perfect example -- they're less efficient than methods (due to the additional proxy object which costs both ROM and has a small runtime penalty), so they aren't used in the main firmware. But the minimum functionality is still there to support Python code that does use it. My understanding is that CircuitPython were really keen to build their API around properties, which makes sense, so they were willing to pay the cost to do so.

My recommendation would be that if you need ulab to support an API based on properties, then write a facade in Python that implements it, and only provide the very core low-level perf-critical stuff in a simple low-level API in C.

Alternatively, seeing as ulab is a user c module, it requires building the firmware anyway. So you could try adding the additional functionality to upstream MicroPython (based on the CircuitPython patches to runtime.c) but conditionally enabled based on another flag, (distinct to the existing MICROPY_PY_BUILTINS_PROPERTY which gives the @property decorator, e.g. MICROPY_PY_BUILTIN_PROPERTIES which would enable properties for builtin types... maybe... naming is hard...), and then ulab's micropython.mk can enable it with CFLAGS -D.

(But full disclaimer, I don't generally like properties, so it's easy for me to say that the cost isn't worth it. However, I understand that others would see it differently -- but that doesn't change the fact that they have a cost).

v923z
Posts: 110
Joined: Mon Dec 28, 2015 6:19 pm

Re: implementing properties in user modules

Post by v923z » Sat Feb 15, 2020 12:15 pm

jimmo wrote:
Sat Feb 15, 2020 12:04 pm
My recommendation would be that if you need ulab to support an API based on properties, then write a facade in Python that implements it, and only provide the very core low-level perf-critical stuff in a simple low-level API in C.
This is definitely a possible workaround, though, that would mean that the library is no longer monolithic. But this approach is definitely cheap (in development time).
jimmo wrote:
Sat Feb 15, 2020 12:04 pm
Alternatively, seeing as ulab is a user c module, it requires building the firmware anyway. So you could try adding the additional functionality to upstream MicroPython (based on the CircuitPython patches to runtime.c) but conditionally enabled based on another flag, (distinct to the existing MICROPY_PY_BUILTINS_PROPERTY which gives the @property decorator, e.g. MICROPY_PY_BUILTIN_PROPERTIES which would enable properties for builtin types... maybe... naming is hard...), and then ulab's micropython.mk can enable it with CFLAGS -D.
OK, so you are, in principle, willing to accept changes to the micropython core. If it is so, then I could use `ulab` as a testbed for implementing the required changes, and then push it upstream with a pre-processor flag. I actually think that this could be a clean and flexible solution.

Well, then it's time to get some work done.

Post Reply