How to implement __setattr__ without infinite recursion

[cdwilson]

I'm trying to implement a Register class which allows direct access to bitfields on a register. I need it to work identical to attribute access on a uctypes.struct bitfield, but I need to read/write to an I2C device behind the scenes whenever the bitfield attributes are accessed.

i.e. something like the following:

Code: Select all

r = Register(addr, layout)
r.FIELD1 = 0x2
r.FIELD2 = 0x1
(etc...)

I tried to implement this using composition where attribute access on the bitfield attributes would be delegated to the internal self.struct object:

Code: Select all

class Register:
    def __init__(self, addr, layout):
    	self.addr = addr
        self._buf = bytearray(sizeof(layout))
        self._struct = uctypes.struct(uctypes.addressof(self._buf), layout)

    def __getattr__(self, name):
    	# I2C bus read code goes here
        return getattr(self._struct, name)
        
    def __setattr__(self, name, value):
    	# this obviously doesn't work, but I'm not sure what to put here???
    	if hasattr(self.struct, name):
        	setattr(self._struct, name, value)
        	# I2C bus write code goes here
        else:
        	setattr(self, name, value)

I can't quite figure out how to implement a __setattr__ for something like this in MicroPython without ending up in an infinite recursion situation.

Here's a very simple example highlighting the issue. Instantiating Test1 leads to infinite recursion because the self._storage assignment calls __setattr__:

Code: Select all

class Storage:

    def __init__(self):
        self.x = 1
        self.y = 2
        self.z = 3

class Test1:

    def __init__(self):
        self._storage = Storage()

    def __getattr__(self, name):
        return getattr(self._storage, name)
    
    def __setattr__(self, name, value):
        setattr(self._storage, name, value)

t1 = Test1() # RuntimeError: maximum recursion depth exceeded

In CPython, I think this would normally be addressed using one of the two options below (Test2 or Test3). However, in MicroPython, objects don't support __setattr__ and obj.__dict__ is read-only, which prevents both of these from working.

Code: Select all

class Test2:

    def __init__(self):
        object.__setattr__(self, '_storage', Storage()) # avoids infinite recursion

    def __getattr__(self, name):
        return getattr(self._storage, name)
    
    def __setattr__(self, name, value):
        setattr(self._storage, name, value)

t2 = Test2()
t2.x = 10

Code: Select all

class Test3:

    def __init__(self):
        self.__dict__['_storage'] = Storage() # avoids infinite recursion

    def __getattr__(self, name):
        return getattr(self._storage, name)
    
    def __setattr__(self, name, value):
        setattr(self._storage, name, value)
        
t3 = Test3()
t2.x = 10

Is there a way to implement this in MicroPython?

[jimmo]

hi,

Wow great question...

I took a quick look at this, and without object.__setattr__ I don't see a good way to do this. However, I have a bad way to do this.

Note: __setattr__ is only supported when MICROPY_PY_DELATTR_SETATTR is enabled, which is only the STM32 port. Not even the Unix port. (!!!)

The bad way works like this:
The issue is that you need a way to access the members of the class from __setattr__ / __getattr__ but without using self. So one option is to make a closure over these variables instead.

Code: Select all

import uctypes

STRUCT1 = {
    "data1": 0 | uctypes.UINT8,
    "data2": 4 | uctypes.UINT32,
}

class Register():
    def __init__(self, addr, layout):
        _buf = bytearray(uctypes.sizeof(layout))
        _storage = uctypes.struct(uctypes.addressof(_buf), layout)
        def _get(name):
            if name == '_storage':
                return _storage
            if name == '_buf':
                return _buf
            if name == '_addr':
                return addr
            print('i2c read from', addr)
            return getattr(_storage, name)
        def _set(name, value):
            setattr(_storage, name, value)
            print('i2c write to', addr, _buf)
        self.__getattr__ = _get
        self.__setattr__ = _set

However, this doesn't quite work... MicroPython has an optimisation with __setattr__ where it won't try to use __setattr__ if it thinks the type doesn't have a __setattr__ (which it only checks when the type is defined, and stores in a flag TYPE_FLAG_HAS_SPECIAL_ACCESSORS). See mp_obj_instance_store_attr in objtype.c for details, it skips ahead to the skip_special_accessors label if the flag isn't set.

So, you could argue that special accessors is actually about properties and descriptors, and that __setattr__ should be checked unconditionally. And if you look at the corresponding read code path, in mp_obj_instance_load_attr, the flag is _only_ used for properties and descriptors. So if you move the definition of the skip_special_accessors: label in mp_obj_instance_store_attr up to above the line "#if MICROPY_PY_DELATTR_SETATTR", then the code above will work. (I don't think this is a strong argument though, if anything the read side should be fixed maybe).

Unfortunately you can't update the flags when __setattr__ is assigned, because the flags are for the type, not the instance. I've been trying to think of some way of combining this with a base class that has a "real" __setattr__, but haven't come up with anything that works yet. I also considered inheriting from uctypes.struct, but this is a great example of where inheriting from a built-in does not work well.

So I think the correct solution here is that object.__setattr__ needs to be implemented in MicroPython.

[jimmo]

Oh!! This was staring me in the face. To enable the special accessors flag, you just need _any_ type of special accessor, including a property!

Code: Select all

import uctypes

STRUCT1 = {
    "data1": 0 | uctypes.UINT8,
    "data2": 4 | uctypes.UINT32,
}

class Register:
    def __init__(self, addr, layout):
        _buf = bytearray(uctypes.sizeof(layout))
        _storage = uctypes.struct(uctypes.addressof(_buf), layout)
        def _get(name):
            if name == '_storage':
                return _storage
            if name == '_buf':
                return _buf
            if name == '_addr':
                return addr
            print('i2c read from', addr)
            return getattr(_storage, name)
        def _set(name, value):
            setattr(_storage, name, value)
            print('i2c write to', addr, _buf)
        self.__getattr__ = _get
        self.__setattr__ = _set

    @property
    def _special(self):
        return

r = Register(0x20, STRUCT1)
print(r.data1)
r.data1 = 3
print(r.data1)

The above works for me on unmodified firmware.

(Still bad and fragile, but probably not _that_ fragile... :p )

[cdwilson]

Great idea!

You actually don't need to do anything special for __getattr__ since it will only get called if the attribute is not found "normally" (https://docs.python.org/3/reference/dat ... .__getattr__). That saves you from having to type all the "if" checks for the attributes stored on Register. See the updated example below.

However, while "get" works correctly for all attributes, this still has the problem that it's not possible to "set" an attribute on the Register class (i.e. trying to change r.addr or create a new r.x attr both fail)... I'm not sure how to get that working without object.__setattr__ because Register.__setattr__ needs a way to set self.attr.

Code: Select all

import uctypes

STRUCT1 = {
    "data1": 0 | uctypes.UINT8,
    "data2": 4 | uctypes.UINT32,
}

class Register:
    def __init__(self, addr, layout):
        self.addr = addr
        self._buf = bytearray(uctypes.sizeof(layout))
        self._storage = uctypes.struct(uctypes.addressof(self._buf), layout)
            
        def _set(name, value):
            setattr(self._storage, name, value)
            print('i2c write to', self.addr, self._buf)
        self.__setattr__ = _set
    
    def __getattr__(self, name):
            print('i2c read from', self.addr)
            return getattr(self._storage, name)

    @property
    def _special(self):
        return


r = Register(0x20, STRUCT1)
print(r.data1)
r.data1 = 3
print(r.data1)
print(r.addr)
r.addr = 4 # KeyError: addr

Note: __setattr__ is only supported when MICROPY_PY_DELATTR_SETATTR is enabled, which is only the STM32 port. Not even the Unix port. (!!!)

This confused me up on the Unix port as well! Supposedly it's enabled on the ./micropython_coverage build (https://github.com/micropython/micropython/issues/4560), but "make coverage" fails to build for me on OSX (still need to debug...).

I've been trying to think of some way of combining this with a base class that has a "real" __setattr__, but haven't come up with anything that works yet. I also considered inheriting from uctypes.struct, but this is a great example of where inheriting from a built-in does not work well.

I had the same thought process (part of what inspired my asking viewtopic.php?f=2&t=6978 which you also answered )

[cdwilson]

FYI, looks like I'm not the first person to want to do something like this https://github.com/micropython/micropyt ... -466670063

[jimmo]

Ah sorry those "if" statements were left over from an earlier experiment.

Here's a variant that lets you additionally have arbitrary members on the Register class (i.e. DIY __dict__).

If you need to be able to change addr, you could easily adapt this with an `if name == 'addr'` in get/set.

Code: Select all

import uctypes

STRUCT1 = {
    "data1": 0 | uctypes.UINT8,
    "data2": 4 | uctypes.UINT32,
}

class Register:
    def __init__(self, addr, layout):
        _buf = bytearray(uctypes.sizeof(layout))
        _storage = uctypes.struct(uctypes.addressof(_buf), layout)
        _members = {}
        def _get(name):
            if name in _members:
                return _members[name]
            print('i2c read from', addr)
            return getattr(_storage, name)
        def _set(name, value):
            try:
                setattr(_storage, name, value)
                print('i2c write to', addr, _buf)
            except KeyError:
                _members[name] = value
        self.__getattr__ = _get
        self.__setattr__ = _set

    @property
    def _special(self):
        return

r = Register(0x20, STRUCT1)
print(r.data1)
r.data1 = 3
print(r.data1)

r.foo = 'bar'
print(r.foo)

[cdwilson]

Yeah, that seems like a good option.

In the closure approach, does each instance store a separate identical copy of _get/_set in memory vs. normal methods that are stored once on the class?

[cdwilson]

I just realized there is a gotcha if the struct layout is nested. In that case, the overloaded __getattr__ will be called with the top-level attribute name only whenever a nested attribute is being get or set. As a result, on a nested attribute 'get', the I2C logic won't have the correct information about which bits to read, and on a 'set' the I2C write will not happen at all.

For example:

Code: Select all

import uctypes

STRUCT1 = {
    "sub": (0, {
        "b0": 0 | uctypes.UINT8,
        "b1": 1 | uctypes.UINT8,
    })
}

class Register:
    def __init__(self, addr, layout):
        _buf = bytearray(uctypes.sizeof(layout))
        _storage = uctypes.struct(uctypes.addressof(_buf), layout)
        _members = {}
        def _get(name):
            print("Register.__getattr__(self, '%s')" % name)
            if name in _members:
                return _members[name]
            print('i2c read from', addr)
            return getattr(_storage, name)
        def _set(name, value):
            print("Register.__setattr__(self, '%s', %s)" % (name, value))
            try:
                setattr(_storage, name, value)
                print('i2c write to', addr, _buf)
            except KeyError:
                _members[name] = value
        self.__getattr__ = _get
        self.__setattr__ = _set

    @property
    def _special(self):
        return

Code: Select all

>>> r = Register(0x20, STRUCT1)
>>> print(r.sub.b0)
Register.__getattr__(self, 'sub')  <-- __getattr__ only gets 'sub' even though we're getting the field 'sub.b0'
i2c read from 32
0
>>> r.sub.b0 = 3
Register.__getattr__(self, 'sub')  <-- __getattr__ is called instead of __setattr__, so I2C write never happens
i2c read from 32
>>> print(r.sub.b0)
Register.__getattr__(self, 'sub')
i2c read from 32
3
>>> 

[pythoncoder]

@jimmo These are clever solutions but a naive Python programmer would expect to use object.__setattr__() to break the recursion. Would implementing this be a major problem?

[jimmo]

@jimmo These are clever solutions but a naive Python programmer would expect to use object.__setattr__() to break the recursion. Would implementing this be a major problem?

Oh yeah, wholeheartedly agree!! Just got distracted by the puzzle, but I have already started doing exactly that (implementing object.__setattr__).