msgpack "trimmed down" for MicroPython on small controllers.

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
User avatar
WZab
Posts: 25
Joined: Tue Jan 21, 2020 7:45 pm

msgpack "trimmed down" for MicroPython on small controllers.

Post by WZab » Wed Jul 14, 2021 4:56 pm

I needed to add the MessagePack module to the MicroPython running on one of my systems running on a STM32 microcontroller. I have found a repository https://github.com/sponster-au/micropython-msgpack . Unfortunately, it relies on datetime module that in turn requires tz-data, resulting with consumption of too much area in the FLASH.

As I do not plan to transmit date information (or it will be transmitted as a plain integer - number of microseconds after the start of the epoch) in my application, I've removed the part related to packing and unpacking of the date and time values. That has reduced the FLASH occupancy significantly. I have published the "reduced" msgpack in my repository https://github.com/wzab/micropython-msgpack in the branch "no-tz" https://github.com/wzab/micropython-msgpack/tree/no-tz. Maybe it will be useful for somebody else?

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by stijn » Wed Jul 14, 2021 7:34 pm

A tip: update the readme to reflect what you; it's not a huge problem the links point to the original but it really should state what the difference is with the original implementation. Now it's a fork of a fork and it is not easy to figure out what the difference with the original implementation is. Also it's already quite some commits behind that one.

User avatar
WZab
Posts: 25
Joined: Tue Jan 21, 2020 7:45 pm

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by WZab » Sun Jul 18, 2021 5:41 pm

I've started to test the trimmed down msgpack on a STM32 MicroPython board, and it appeared, that it is not so trivial.
First, it appeared that there is no sys.float_info. Then it appeared that the "collections" does not contain "Hashable".
Therefore, I needed to trim it down even more. The version available in https://github.com/wzab/micropython-msg ... 5346f44f6a seems to work correctly on STM32.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by pythoncoder » Wed Jul 21, 2021 6:51 am

I think this could be a very useful addition to the current MicroPython serialisation options but I agree with the comment of @stijn: it would benefit greatly by being presented as a complete project with installation instructions. Also docs that state any MicroPython limitations or differences.

I would update my serialisation FAQ to include this if it were easy to install and use.
Peter Hinch
Index to my micropython libraries.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by pythoncoder » Wed Jul 21, 2021 4:54 pm

I have tried your code and it does indeed seem OK on the Unix build and STM32. There is more to be done to make it "micro". The low hanging fruit are stripping out the Python2 stuff and maybe ditching compatibility mode, but it really needs a thorough review. It does strike me as worth doing as we haven't got a self-describing binary serialisation library (protobuf requires a schema). I'm willing to hack on this if you don't plan to pursue this further.

MP lacks a timestamp class so users would probably pack an integer from utime.time() or do something creative with utime.time_ns().
Peter Hinch
Index to my micropython libraries.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by pythoncoder » Fri Jul 23, 2021 6:23 am

I don't know if anyone is still interested in this - if not I'll pipe down ;)

After scrapping the Python 2 cruft and other easy trims the thing still weighs in at a hefty 26K. The authors' concept of "lightweight" is not mine. The function dispatch table weighs in at an elephantine 10K which I plan to reclaim by replacing it with a binary chop algorithm. So I can maybe get it down to 16K or so, which I think would be usable. Another weight loss idea is to separate load and dump into separate modules as some applications will only need one.

I studied the original sources and reinstated the ext_serializable decorator as I think this is a deeply cool feature. It means you can write a small, simple, optional module defining wrappers for unsupported Python types (I tried complex and set). So, provided the source and destination have access to the module, you can write:

Code: Select all

from umsgpack import dumps, loads
from umsgpack_extensions import ComplexSerialiser
c = 1.6 + 2.2j
obj = ComplexSerialiser(c)  # Simple wrapper class makes it serialisable
d = [obj, 1.23, 'rats']  # Put it in a Python object like any supported type
s = dumps(d)  # Usage is just like ujson
res = loads(s)
and the list is restored with res[0] being a complex. No schema required by the receiver.
Peter Hinch
Index to my micropython libraries.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by stijn » Fri Jul 23, 2021 7:02 am

Have you considered https://github.com/msgpack/msgpack-pyth ... allback.py as a starting point? I didn't actually test, but just from the looks it could be smaller than umsgpack.. You'll need to remove the multiple inheritance in its exceptions.py but for the rest works out of the box.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by pythoncoder » Fri Jul 23, 2021 2:08 pm

I'll take a look at that and try to figure out if it has a chance of being significantly smaller.

I've done a bit of optimisation and got it down to ~12.5KiB on STM. This is with dump and load functionality retained in one module. I think it would be hard to go much further with this codebase. The code seems OK to me, it just needed some attention in a microcontroller context.

Mods were:
Remove all Python 2 cruft.
Replace function despatch table with code. Discard __init() function.
Reinstate ext_serialiser.
New means of detecting platform's float size.
Remove most globals and aliases: it now supports dump, dumps, load and loads only, as per ujson.
A lot of minor optimisations: replacing comparisons of single element bytes objects with integer compares, using lookups instead of code, trimming exception messages and simplifying "logic error" debug code that should never actually run.
Peter Hinch
Index to my micropython libraries.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by stijn » Fri Jul 23, 2021 2:30 pm

pythoncoder wrote:
Fri Jul 23, 2021 2:08 pm
I've done a bit of optimisation and got it down to ~12.5KiB on STM.
Nice! Don't want to be nagging you, but since you mention you reinstated things does this mean you didn't start from the original project (https://github.com/vsergeev/u-msgpack-python) ? Not sure if it matters a lot but if this becomes the de-facto 'msgpack for micropython' it would be good starting from there.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: msgpack "trimmed down" for MicroPython on small controllers.

Post by pythoncoder » Sun Jul 25, 2021 1:37 pm

I started out looking for some quick fixes to help @WZab, so I did indeed begin with the wrong codebase. I have since done a line by line review against the vsergeev original. I have run its test suite (after adapting for the unsupported functionality mentioned above such as timestamps). I'm now pretty confident of the code. The task now is to update the docs.

Incidentally the test suite has no chance of running under MicroPython because it tests against huge data structures. Also MicroPython doesn't support unittest. Running a full set of tests under CPython seems like a good test of code correctness and conformity with the spec - and it did find a bug, proving my trimmed-down bug catching code in the process.

Regarding the code at https://github.com/msgpack/msgpack-pyth ... allback.py, the only way to do a meaningful comparison would be to perform a similar reduction exercise on it to get rid of globals, large dicts, and suchlike, and then compare RAM use. I'm not convinced there's much to be gained in doing this.
Peter Hinch
Index to my micropython libraries.

Post Reply