Improvements to uctypes module/infrastructure

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
pfalcon
Posts: 1153
Joined: Fri Feb 28, 2014 2:05 pm

Improvements to uctypes module/infrastructure

Post by pfalcon » Wed Sep 26, 2018 9:46 pm

I was recently working on uctypes module fixing and improvements. The main improvement is ability to automatically calculate field offsets in structures - something which was envisioned from the beginning, but kept being backlogged. Other changes are described in https://github.com/pfalcon/micropython/issues/14 and in commits in my fork, https://github.com/pfalcon/micropython . I started to spool these to mainline, but patch processing in the mainline goes veeeery slow, so it make take long time until all changes are there.

Beyond work on uctypes modules itself, I also started some helper modules in the micropython-lib project. Those are work in progress and subject to change, but cover some ideas which were in thinking for some time too. Commit message should give info what they're about:
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

stijn
Posts: 321
Joined: Thu Apr 24, 2014 9:13 am

Re: Improvements to uctypes module/infrastructure

Post by stijn » Thu Sep 27, 2018 8:28 am

Looks good! Just needs examples but I see those are on the TODO list already :)

User avatar
marfis
Posts: 187
Joined: Fri Oct 31, 2014 10:29 am
Location: Zurich / Switzerland

Re: Improvements to uctypes module/infrastructure

Post by marfis » Thu Sep 27, 2018 7:26 pm

+1 from me too.
and +2 for examples and hints on using uctypes...

jickster
Posts: 628
Joined: Thu Sep 07, 2017 8:57 pm

Re: Improvements to uctypes module/infrastructure

Post by jickster » Thu Sep 27, 2018 10:03 pm

I'm a C, C++ programmer and I JUST realized "offsets" is there to perform the function of casting a void* to a struct in C.

Maybe I'm slow but it'd be very helpful to state the above fact in the documentation.

pfalcon
Posts: 1153
Joined: Fri Feb 28, 2014 2:05 pm

Re: Improvements to uctypes module/infrastructure

Post by pfalcon » Sat Oct 06, 2018 12:13 pm

A test-example for uctypeslib was posted: https://github.com/pfalcon/micropython- ... ib/test.py . It was roughly based on https://github.com/pfalcon/micropython/ ... ypes_le.py , so can be used for comparison.

Code: Select all

desc = OrderedDict((
    ("s0", uc.UINT16),
    ("sub", uc.STRUCT(od((
        ("b0", uc.UINT8),
        ("b1", uc.UINT8),
    )))),
    ("arr", uc.ARRAY(uctypes.UINT8, 2)),
    ("arr2", uc.ARRAY(uc.STRUCT(od([("b", uc.UINT8)])), 2)),

    ("bitf0", uc.BITFIELD(0, 8, type=uc.UINT16)),
    ("bitf1", uc.BITFIELD(8, 8)),

    ("bf0", uc.C_BITFIELD(4, type=uc.UINT16)),
    ("bf1", uc.C_BITFIELD(4)),
    ("bf2", uc.C_BITFIELD(4)),
    ("bf3", uc.C_BITFIELD(4)),
))
P.S. Actually, uctypeslib/test.py may look less pretty due to the fact that there's no support for OrderedDict literals (i.e. OrderedDict({...}) doesn't guarantee field order), so tuples of tuples have to be used instead, leading to the proliferation of parens. That's actually the reason why this work was postponed for so long - the whole idea of these additions is to make it easier and more beautiful, but "beautiful" part regresses without OrderedDict literals. Anyway, I decided to do all that to "finish" the uctypes modules, and OD literals are definitely on TODO too.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1153
Joined: Fri Feb 28, 2014 2:05 pm

Re: Improvements to uctypes module/infrastructure

Post by pfalcon » Sat Oct 20, 2018 10:14 am

Updated docs for the uctypes module are available at https://pycopy.readthedocs.io/en/latest ... types.html . (So far, these capture the state of the module before the changes described in this thread.)
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1153
Joined: Fri Feb 28, 2014 2:05 pm

Re: Improvements to uctypes module/infrastructure

Post by pfalcon » Sun Oct 21, 2018 11:56 am

The last thing in queue for this work was implementing the proper OrderedDict literals, which is now done too. So, it's possible to write:
OrderedDict({"foo": "bar"})
And it will be handled as efficient as just

Code: Select all

{"foo": "bar"}
. So, no compromises on both readability and efficiency are now required.

Initial patchset for all these features was posted for the mainline, but by now only 1-2 of patches were processed. So as usual, these and many other great features are available in my fork, https://github.com/pfalcon/micropython .
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1153
Joined: Fri Feb 28, 2014 2:05 pm

Re: Improvements to uctypes module/infrastructure

Post by pfalcon » Sun Oct 21, 2018 11:58 am

And now project which prompted this work was posted too - FFmpeg bindings: viewtopic.php?f=15&t=5419 .

FFmpeg uses data structures (not just functions) in its public API, so emphasized not implemented/missing features in uctypes quite well.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1153
Joined: Fri Feb 28, 2014 2:05 pm

Re: Improvements to uctypes module/infrastructure

Post by pfalcon » Fri Oct 26, 2018 10:32 am

Doc changes mentioned above are now in the mainline: http://docs.micropython.org/en/latest/l ... types.html
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Post Reply