New MQTT client library for new uasyncio

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
Online
User avatar
tve
Posts: 88
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

New MQTT client library for new uasyncio

Post by tve » Thu Mar 26, 2020 5:50 am

In case you haven't noticed, the new uasyncio has finally landed! After a lot of effort Damien merged https://github.com/micropython/micropython/pull/5332 today, yay!
While I was tracking his progress I was working on a new MQTT client library that works in concert with the new uasyncio on the esp32. I started with Peter's great mqtt-as but ended up rewriting pretty much all of it so I could add streaming publish (not waiting for an ACK after every publish) and an automated test suite covering almost all the code. The library continues to automatically reconnect and retransmit under the hood and supports both QoS 0 and QoS 1.

I just published it as a beta version to pypi [ upip.install('micropython-mqtt') ] { Update: now renamed to [ upip.install('micropython-mqtt-async') ] } and github [ https://github.com/tve/micropython-mqtt ... mqtt-async ]. There is an extensive README which should help you get started. Note that this library will only work with the new built-in uasyncio and I have only used it on an esp32 so far.
Last edited by tve on Thu Mar 26, 2020 3:23 pm, edited 1 time in total.

kevinkk525
Posts: 636
Joined: Sat Feb 03, 2018 7:02 pm

Re: New MQTT client library for new uasyncio

Post by kevinkk525 » Thu Mar 26, 2020 7:49 am

I appreciate your efforts in making a new mqtt library!

However, I have to say, rewriting it and uploading it to pypi as "micropython-mqtt" seems like a hostile takeover, considering that Peter Hinch's library is the most popular mqtt library and is named "micropython-mqtt". So if people install micropython-mqtt from pypi they would actually expect Peter Hinch's version and not your rewritten fork that is not even a drop-in replacement.
If you rewrite something, why don't you name it uniquely and upload it using a different name?
Also if you call it "New MQTT client library" then please also make it new by using a "new" name. Because this way it looks like you are either just advertising another fork of the great work from Peter Hinch, or you are in fact advertising a new library but trying to take over an existing project.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Online
User avatar
tve
Posts: 88
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: New MQTT client library for new uasyncio

Post by tve » Thu Mar 26, 2020 8:18 am

kevinkk525 wrote:
Thu Mar 26, 2020 7:49 am
I appreciate your efforts in making a new mqtt library!

However, I have to say, rewriting it and uploading it to pypi as "micropython-mqtt" seems like a hostile takeover, considering that Peter Hinch's library is the most popular mqtt library and is named "micropython-mqtt". So if people install micropython-mqtt from pypi they would actually expect Peter Hinch's version and not your rewritten fork that is not even a drop-in replacement.
If you rewrite something, why don't you name it uniquely and upload it using a different name?
Also if you call it "New MQTT client library" then please also make it new by using a "new" name. Because this way it looks like you are either just advertising another fork of the great work from Peter Hinch, or you are in fact advertising a new library but trying to take over an existing project.
I'm sorry you feel so offended, I feel like I must have hit some hot button I didn't know about. In my mind I had associated the name mqtt_as with Peter's library, that's the name he uses all the time in all his posts on this forum, and so I picked a different name, specifically mqtt_async. When I uploaded to pypi I noticed that -mqtt was still available and didn't really think much further about it. I was mostly trying to find the shortest option given that micropython- as a prefix is already a mile long. I did not remember that his github repo has that name. Also, apparently Peter is not interested in pypi, or else he would have uploaded there long ago? I have to admit that so far I'm not thrilled by the pypi experience (first time I use it), it seems it causes more confusion than anything else in terms of versioning WRT what's in github.If Peter wants that pypi name he can have it, but then he better actually use it.

User avatar
pythoncoder
Posts: 4011
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: New MQTT client library for new uasyncio

Post by pythoncoder » Thu Mar 26, 2020 10:57 am

I don't use PyPi and have no plans to do so.

I think it's in both our interests - and more importantly, those of our users, to avoid name clashes. They cause confusion which you and I will have repeatedly to resolve. I don't know about you, but I could do without the hassle.

While I haven't studied your library, my "off the top of my head" response is that yours and mine serve different purposes. Having a coroutine wait for an ACK has benefits on low resource targets such as ESP8266. It also serves a purpose on certain projects. This general issue has been the subject of lengthy discussions between myself and Kevin; it was a deliberate design decision rather than a random oversight. However I do agree that streaming publications have a role on suitably resourced platforms.

So can I suggest you adopt a new, unique, name which identifies the purpose and merit of your library?
Peter Hinch

kevinkk525
Posts: 636
Joined: Sat Feb 03, 2018 7:02 pm

Re: New MQTT client library for new uasyncio

Post by kevinkk525 » Thu Mar 26, 2020 1:14 pm

tve wrote:
Thu Mar 26, 2020 8:18 am
I'm sorry you feel so offended, I feel like I must have hit some hot button I didn't know about. In my mind I had associated the name mqtt_as with Peter's library, that's the name he uses all the time in all his posts on this forum, and so I picked a different name, specifically mqtt_async.
I saw some possibilities and possible motives so I felt a little offended (although micropython-mqtt is not my project) but I was wrong to react that way. I should have just explained how it looks and could be interpreted without jumping to conclusions and I should have given you the chance to explain why you did it that way. I apologize for that.

As Peter explained, it would be good if you chose a different name for your implementation to prevent problems from name clashing.


I didn't get a chance yet to have a close look at your implementation but some features look interesting. The library seems to be only for esp32 (or only tested on that platform) if I read that correctly? As I use mostly esp8266, this library is "too powerful" anyway. But I'm interested to see how your library will be adopted and how well it'll work in all the edge-cases and long-term deployments that Peter's micropython-mqtt has already been tested in.

It was interesting to see that you used CPython for testing and not the unix port. I made my fork of micropython-mqtt compatible to the unix port for testing because that way I could test an entire project using the unix port because that project would probably not run on CPython and it would be too much work to always try to be compatible to CPython as well (just for testing..).
So might be worth making your library run on the unix port as well?
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Online
User avatar
tve
Posts: 88
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: New MQTT client library for new uasyncio

Post by tve » Thu Mar 26, 2020 2:56 pm

pythoncoder wrote:
Thu Mar 26, 2020 10:57 am
So can I suggest you adopt a new, unique, name which identifies the purpose and merit of your library?
I did, yours is called mqtt_as and mine is called mqtt_async, what am I missing?

I re-uploaded mine to pypi as micropython-mqtt-async, awkward long but thankfully one only types this once...

Online
User avatar
tve
Posts: 88
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: New MQTT client library for new uasyncio

Post by tve » Thu Mar 26, 2020 3:33 pm

pythoncoder wrote:
Thu Mar 26, 2020 10:57 am
Having a coroutine wait for an ACK has benefits on low resource targets such as ESP8266. It also serves a purpose on certain projects. This general issue has been the subject of lengthy discussions between myself and Kevin; it was a deliberate design decision rather than a random oversight. However I do agree that streaming publications have a role on suitably resourced platforms.
The asynchronous publishing is optional in mqtt_async, it's simply an sync=True/False KW parameter and the default is True, i.e. wait just like mqtt_as does. In terms of resource consumption the only real overhead when choosing sync=False is having to hang onto the previous message, whether that's a problem really depends on the message size. I also compared the code size, using mpy-cross -O3, mqtt_async is a bit less than 10% larger than mqtt_as (9168 bytes vs 8360). I have not compared data memory sizes.

Online
User avatar
tve
Posts: 88
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: New MQTT client library for new uasyncio

Post by tve » Thu Mar 26, 2020 4:08 pm

kevinkk525 wrote:
Thu Mar 26, 2020 1:14 pm
tve wrote:
Thu Mar 26, 2020 8:18 am
I'm sorry you feel so offended, I feel like I must have hit some hot button I didn't know about. In my mind I had associated the name mqtt_as with Peter's library, that's the name he uses all the time in all his posts on this forum, and so I picked a different name, specifically mqtt_async.
I saw some possibilities and possible motives so I felt a little offended (although micropython-mqtt is not my project) but I was wrong to react that way. I should have just explained how it looks and could be interpreted without jumping to conclusions and I should have given you the chance to explain why you did it that way. I apologize for that.
Thank you, no harm.
kevinkk525 wrote:
Thu Mar 26, 2020 1:14 pm
I didn't get a chance yet to have a close look at your implementation but some features look interesting. The library seems to be only for esp32 (or only tested on that platform) if I read that correctly? As I use mostly esp8266, this library is "too powerful" anyway.
I have left esp8266 code in but have not really tested it other than to notice that it has to be frozen into the firmware to be of any use. Given the hassle I thought I'd better leave that until I stop tweaking the code. I'm also not really interested in doing a ton of work myself for the esp8266 'cause I'm trying hard to only use encrypted communication. I've ordered a PYBD and I'll see what I need to do for things to run there. I'm curious about its Wifi low power functions.
kevinkk525 wrote:
Thu Mar 26, 2020 1:14 pm
But I'm interested to see how your library will be adopted and how well it'll work in all the edge-cases and long-term deployments that Peter's micropython-mqtt has already been tested in.
Yup, I know I still have work to do and things to improve to make mqtt_async behave better when the connectivity goes south, especially if one cares about power consumption. I've found it easy to end up with a system that eventually reconnects just fine but that is non-functional in the meantime because it's trying to reconnect like crazy.
Note that when mqtt_as gets ported to the new asyncio it pretty much resets all the "long-term deployments": socket reads/writes go from non-blocking to blocking which rippled further through the code that I had expected, but YMMV.
kevinkk525 wrote:
Thu Mar 26, 2020 1:14 pm
It was interesting to see that you used CPython for testing and not the unix port. I made my fork of micropython-mqtt compatible to the unix port for testing because that way I could test an entire project using the unix port because that project would probably not run on CPython and it would be too much work to always try to be compatible to CPython as well (just for testing..).
So might be worth making your library run on the unix port as well?
I did run the tests on unix micropython at some point, I see that I need to tweak the damn try-import-except stuff again for it to work, thanks for pointing it out. There are a whole bunch of reasons I went for CPython: I wanted to double-check against the "official" asyncio, I find that pytest provides vastly better error reports when an assertion fails, pytest provides additional nice powerful assertions (pytest.raises, etc), and last but not least, it gives me a test coverage report so I can see which fragments of code are not tested at all.

User avatar
pythoncoder
Posts: 4011
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: New MQTT client library for new uasyncio

Post by pythoncoder » Thu Mar 26, 2020 5:08 pm

tve wrote:
Thu Mar 26, 2020 2:56 pm
...
I re-uploaded mine to pypi as micropython-mqtt-async, awkward long but thankfully one only types this once...
Thank you, I think that will save both of us from needless support issues.
Peter Hinch

Post Reply