dir and __getattr__ weirdness

[fstengel]

Here is a simple class (sorry no BBCode for now and indents are off):

[code]
class Test(object):
def __init__(self):
print("Test : init")
def __dir__(self):
return ["Test : dir"]
def __getattr__(self, name):
print("Test, getattr : {}".format(name))
[/code]

I would expect that using it like:
[code]
t = Test()
dir(t)
[/code]
would produce:
[code]
Test : init
['Test : dir']
[/code]
At least that's what the various flavours of Anaconda give me. If I try the same in micropython, I get:
[code]
Test, getattr :
Test, getattr : __abs__
Test, getattr : __add__
...
[/code]
and quite a lot of lines ending with a list of seemingly all the names known to micropython. Is that a feature or an issue?

[pythoncoder]

Yikes That is surely an issue, and should be raised on GitHub. If you aren't familiar with doing this please advise: I'm happy to do it on your behalf.

[stijn]

I don't think uPy ever supported a custom __dir__ method, and a couple of months ago dir got implemented as probing all possible attributes, which greatly improveds usability, so the output shown is basically expected.

I think it's a minor change to implement __dir__ support, though on the other hand it's maybe not 'micro' enough, also seeing that there probably won't be that much usecases. Alternatievely it could be listed in https://github.com/micropython/micropyt ... ifferences, but that doesn't contain a complete list of all methods not supported (there's more than just __dir__, I presume?) so just adding one item there seems a bit weird.

[fstengel]

I think I understand a bit more. I nonetheless opened a ticket (https://github.com/micropython/micropython/issues/3729) What bothers me is more the behaviour with __getattr__: it is as if dir forces __getattr__ to iterate over the list of known names...

[pythoncoder]

I don't think uPy ever supported a custom __dir__ method, and a couple of months ago dir got implemented as probing all possible attributes, which greatly improveds usability, so the output shown is basically expected...

Have you seen just how much output it generates?

Not implementing __dir__() is not a problem in my view; it's not the presence of __dir__() in the test class which breaks it.

It seems that the mere presence of __getattr__ causes dir(object) to issue reams of output. I routinely issue dir() against class instances and it's always worked until I saw this. This is surely a bug.

[dhylands]

It appears to be deliberate:
https://github.com/micropython/micropyt ... 67ca8ddd03

[stijn]

It seems that the mere presence of __getattr__ causes dir(object) to issue reams of output

Arguably it's the presence of the print statement which causes that. I think the point of __attr__ is to lookup attributes, and anybody can query that, so it's expected that one's custom __attr__ implementation (or the corresponding custom function in C) can get called with arbitrary values, I think? And so in the spirit of being micro, uPy (ab)uses it to implement dir(), and this current implementation surely is way more convenient than the previous one.

[pythoncoder]

...Arguably it's the presence of the print statement which causes that...

Take this reductio ad absurdum lacking any print statement and run in a freshly launched instance of MicroPython:

Code: Select all

MicroPython v1.9.3-547-g3d5d76f on 2018-04-11; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== class Foo:
===     def __getattr__(self, name):
===         return 42
=== 
>>> foo=Foo()
>>> foo.bar
42
>>> 

So far, so good. Surely this is wrong?

Code: Select all

dir(foo)
['', '__abs__', '__add__', '__aenter__', '__aexit__', '__aiter__', '__and__', '__anext__', '__bool__', '__build_class__', '__call__', '__class__', '__contains__', '__del__', '__delete__', '__delitem__', '__dict__', '__divmod__', '__enter__', '__eq__', '__exit__', '__file__', '__floordiv__', '__ge__', '__get__', '__getattr__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__import__', '__init__', '__invert__', '__isub__', '__iter__', '__le__', '__len__', '__lshift__', '__lt__', '__main__', '__mod__', '__module__', '__mul__', '__name__', '__neg__', '__new__', '__next__', '__or__', '__path__', '__pos__', '__pow__', '__qualname__', '__radd__', '__rand__', '__repl_print__', '__repr__', '__reversed__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__set__', '__setitem__', '__str__', '__sub__', '__traceback__', '__truediv__', '__xor__', '*', '_', '/', '%#o', '%#x', '{:#b}', ' ', '\n', 'maximum recursion depth exceeded', '<module>', '<lambda>', '<listcomp>', '<dictcomp>', '<setcomp>', '<genexpr>', '<string>', '<stdin>', 'utf-8', 'AF_INET', 'AF_INET6', 'AF_UNIX', 'ARRAY', 'ArithmeticError', 'AssertionError', 'AttributeError', 'B115200', 'B57600', 'B9600', 'BFINT16', 'BFINT32', 'BFINT8', 'BFUINT16', 'BFUINT32', 'BFUINT8', 'BF_LEN', 'BF_POS', 'BIG_ENDIAN', 'BaseException', 'BytesIO', 'DEBUG', 'DESC', 'DecompIO', 'EACCES', 'EADDRINUSE', 'EAGAIN', 'EALREADY', 'EBADF', 'ECONNABORTED', 'ECONNREFUSED', 'ECONNRESET', 'EEXIST', 'EHOSTUNREACH', 'EINPROGRESS', 'EINVAL', 'EIO', 'EISDIR', 'ENOBUFS', 'ENODEV', 'ENOENT', 'ENOMEM', 'ENOTCONN', 'EOFError', 'EOPNOTSUPP', 'EPERM', 'ETIMEDOUT', 'Ellipsis', 'Exception', 'FLOAT32', 'FLOAT64', 'FileIO', 'GeneratorExit', 'INCL', 'INT16', 'INT32', 'INT64', 'INT8', 'ImportError', 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LITTLE_ENDIAN', 'LockType', 'LookupError', 'MSG_DONTROUTE', 'MSG_DONTWAIT', 'MemoryError', 'NATIVE', 'NameError', 'None', 'NoneType', 'NotImplemented', 'NotImplementedError', 'OSError', 'OrderedDict', 'OverflowError', 'POLLERR', 'POLLHUP', 'POLLIN', 'POLLOUT', 'PTR', 'PinBase', 'RuntimeError', 'SOCK_DGRAM', 'SOCK_RAW', 'SOCK_STREAM', 'SOL_SOCKET', 'SO_BROADCAST', 'SO_ERROR', 'SO_KEEPALIVE', 'SO_LINGER', 'SO_REUSEADDR', 'Signal', 'StopAsyncIteration', 'StopIteration', 'StringIO', 'SyntaxError', 'SystemExit', 'TCSANOW', 'TextIOWrapper', 'TypeError', 'UINT16', 'UINT32', 'UINT64', 'UINT8', 'UnicodeError', 'VOID', 'ValueError', 'ViperTypeError', 'ZeroDivisionError', '_thread', 'a2b_base64', 'abs', 'accept', 'acos', 'acosh', 'acquire', 'add', 'addr', 'addressof', 'all', 'allocate_lock', 'any', 'append', 'args', 'argv', 'array', 'as_bytearray', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'b2a_base64', 'bin', 'bind', 'bool', 'bound_method', 'btree', 'buffering', 'builtins', 'bytearray', 'bytearray_at', 'bytecode', 'byteorder', 'bytes', 'bytes_at', 'cachesize', 'calcsize', 'callable', 'callback', 'ceil', 'center', 'cert', 'chr', 'classmethod', 'clear', 'clock', 'close', 'closure', 'cmath', 'code', 'collect', 'compile', 'complex', 'connect', 'const', 'copy', 'copysign', 'cos', 'cosh', 'count', 'crc32', 'decode', 'decompress', 'default', 'degrees', 'delattr', 'deleter', 'deque', 'dict', 'dict_view', 'difference', 'difference_update', 'digest', 'dir', 'disable', 'discard', 'divmod', 'doc', 'dump', 'dumps', 'e', 'enable', 'encode', 'encoding', 'end', 'endswith', 'enumerate', 'erf', 'erfc', 'errno', 'errorcode', 'eval', 'exc_info', 'exec', 'exit', 'exp', 'expm1', 'extend', 'fabs', 'ffi', 'fficallback', 'ffifunc', 'ffimod', 'ffivar', 'file', 'fileno', 'filter', 'find', 'flags', 'float', 'floor', 'flush', 'fmod', 'format', 'frexp', 'from_bytes', 'fromkeys', 'frozenset', 'func', 'function', 'gamma', 'gc', 'generator', 'get', 'get_ident', 'getaddrinfo', 'getattr', 'getenv', 'getrandbits', 'getter', 'getvalue', 'globals', 'group', 'hasattr', 'hash', 'heap_lock', 'heap_unlock', 'heapify', 'heappop', 'heappush', 'hex', 'hexlify', 'id', 'ilistdir', 'imag', 'implementation', 'index', 'inet_ntop', 'inet_pton', 'input', 'insert', 'int', 'intersection', 'intersection_update', 'invert', 'ioctl', 'ipoll', 'isalpha', 'isdigit', 'isdisjoint', 'isenabled', 'isfinite', 'isinf', 'isinstance', 'islower', 'isnan', 'isspace', 'issubclass', 'issubset', 'issuperset', 'isupper', 'items', 'iter', 'iterable', 'iterator', 'join', 'kbd_intr', 'keepends', 'key', 'keys', 'ldexp', 'len', 'lgamma', 'list', 'listen', 'little', 'load', 'loads', 'locals', 'localtime', 'lock', 'locked', 'log', 'log10', 'log2', 'lower', 'lstrip', 'makefile', 'map', 'match', 'math', 'max', 'maxsize', 'mem', 'mem16', 'mem32', 'mem8', 'mem_alloc', 'mem_current', 'mem_free', 'mem_info', 'mem_peak', 'mem_total', 'memoryview', 'micropython', 'min', 'minkeypage', 'mkdir', 'mode', 'modf', 'modify', 'module', 'modules', 'name', 'namedtuple', 'native', 'next', 'object', 'oct', 'off', 'on', 'open', 'opt_level', 'ord', 'pack', 'pack_into', 'pagesize', 'partition', 'path', 'peektime', 'pend_throw', 'phase', 'pi', 'platform', 'polar', 'poll', 'pop', 'popitem', 'popleft', 'pow', 'print', 'print_exception', 'property', 'ptr', 'ptr16', 'ptr32', 'ptr8', 'push', 'put', 'qstr_info', 'r', 'radians', 'range', 'read', 'readinto', 'readline', 'readlines', 'real', 'rect', 'recv', 'recvfrom', 'register', 'release', 'remove', 'replace', 'repr', 'reverse', 'reversed', 'rfind', 'rindex', 'round', 'rpartition', 'rsplit', 'rstrip', 'search', 'seed', 'seek', 'send', 'sendto', 'sep', 'seq', 'server_hostname', 'server_side', 'set', 'setattr', 'setblocking', 'setdefault', 'setraw', 'setsockopt', 'setter', 'sha1', 'sha256', 'sin', 'single', 'sinh', 'sizeof', 'sleep', 'sleep_ms', 'sleep_us', 'slice', 'sockaddr', 'socket', 'sort', 'sorted', 'split', 'splitlines', 'sqrt', 'stack_size', 'stack_use', 'start', 'start_new_thread', 'startswith', 'stat', 'staticmethod', 'statvfs', 'stderr', 'stdin', 'stdout', 'step', 'stop', 'str', 'strip', 'struct', 'sum', 'super', 'symmetric_difference', 'symmetric_difference_update', 'sys', 'system', 'tan', 'tanh', 'tcgetattr', 'tcsetattr', 'tell', 'termios', 'threshold', 'throw', 'ticks_add', 'ticks_cpu', 'ticks_diff', 'ticks_ms', 'ticks_us', 'time', 'time_pulse_us', 'to_bytes', 'trunc', 'tuple', 'type', 'ubinascii', 'ucollections', 'uctypes', 'uerrno', 'uhashlib', 'uheapq', 'uint', 'uio', 'ujson', 'umachine', 'unhexlify', 'union', 'unlink', 'unpack', 'unpack_from', 'unregister', 'uos', 'update', 'upper', 'urandom', 'ure', 'uselect', 'usocket', 'ussl', 'ustruct', 'utime', 'utimeq', 'uzlib', 'value', 'values', 'var', 'version', 'version_info', 'viper', 'websocket', 'wrap_socket', 'write', 'zip', 'upip.py', 'os', 'json', 'upip_utarfile', 'tarfile', 'debug', 'install_path', 'cleanup_files', 'gzdict_sz', 'file_buf', 'NotFoundError', 'op_split', 'op_basename', '_makedirs', 'save_file', 'install_tar', 'expandhome', 'warn_ussl', 'url_open', 'get_pkg_metadata', 'fatal', 'install_pkg', 'install', 'get_install_path', 'cleanup', 'help', 'main', 'wb', 'fname', 'subf', 'setup.', 'PKG-INFO', 'README', '.egg-info', 'extractfile', 'deps', 'Skipping', 'DIRTYPE', 'f', 'prefix', '~/', 'HOME', 's', 'https:', 'url', 'Error:', 'msg', 'exc', 'info', 'releases', 'TarFile', 'fileobj', 'pkg_spec', 'Queue:', 'to_install', '-h', '--help', '-p', '-r', '#', '--debug', '-', 'upip_utarfile.py', 'size', 'TAR_HEADER', 'REGTYPE', 'roundup', 'FileSection', 'TarInfo', 'val', 'align', 'skip', 'content_len', 'self', 'aligned_len', 'sz', 'buf', 'rb', '\x00', 'tarinfo', '/usr/lib/micropython', 'Foo', 'foo', 'bar']

Code: Select all

>>> len(dir(foo))
687

This only occurs if the class has a __getattr__ method.

[fstengel]

Whats more is that __getattr__ gets called properly: that is only if the name is not directly known to the object. Basically anything dir() would not normally return. A class accumulating the names sent as argument to __getattr__ would retain only those names not known to that class. The following:

Code: Select all

class Bar(object):
    def __init__(self):
        self.g = []
    def __getattr__(self, name):
        self.g.append(name)
b = Bar()
d = dir(b)
print([n for n in d if n not in b.g])

would result (as I would expect with this implementation of dir) in:

Code: Select all

['__class__', '__dict__', '__getattr__', '__init__', '__module__', '__qualname__', 'g']

My basic motivation for this was creating a singleton class to encapsulate a display object. I needed to be able to delegate methods to the display. I used something that, after paring down and changing the encapsulated object to a list, looks like that:

Code: Select all

class Foo(object):
    def __init__(self):
        self.a = []
    def __dir__(self):
        return dir(self.a)
    def __getattr__(self, name):
        return getattr(self.a, name)

That works quite well, except for dir. See:

Code: Select all

f = Foo()
f.append(1)
print(f.a)
print(dir(f))

which produces:

Code: Select all

[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute ''

As to why I use dir, well it is usually the fastest way to see what an object knows about. I cannot count the times it saved me from spelling mistakes when using obscure methods or magic constants.

[stijn]

So far, so good. Surely this is wrong?

Sorry my mistake, because of the way the first code sample was structured I thought the actual result of dir() was still correct, and it would ignore any names for which __getattr__ didn't return anything