In case you haven't noticed, the new uasyncio has finally landed! After a lot of effort Damien merged https://github.com/micropython/micropython/pull/5332 today, yay!
While I was tracking his progress I was working on a new MQTT client library that works in concert with the new uasyncio on the esp32. I started with Peter's great mqtt-as but ended up rewriting pretty much all of it so I could add streaming publish (not waiting for an ACK after every publish) and an automated test suite covering almost all the code. The library continues to automatically reconnect and retransmit under the hood and supports both QoS 0 and QoS 1.
I just published it as a beta version to pypi [ upip.install('micropython-mqtt') ] { Update: now renamed to [ upip.install('micropython-mqtt-async') ] } and github [ https://github.com/tve/micropython-mqtt ... mqtt-async ]. There is an extensive README which should help you get started. Note that this library will only work with the new built-in uasyncio and I have only used it on an esp32 so far.
New MQTT client library for new uasyncio
New MQTT client library for new uasyncio
Last edited by tve on Thu Mar 26, 2020 3:23 pm, edited 1 time in total.
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: New MQTT client library for new uasyncio
I appreciate your efforts in making a new mqtt library!
However, I have to say, rewriting it and uploading it to pypi as "micropython-mqtt" seems like a hostile takeover, considering that Peter Hinch's library is the most popular mqtt library and is named "micropython-mqtt". So if people install micropython-mqtt from pypi they would actually expect Peter Hinch's version and not your rewritten fork that is not even a drop-in replacement.
If you rewrite something, why don't you name it uniquely and upload it using a different name?
Also if you call it "New MQTT client library" then please also make it new by using a "new" name. Because this way it looks like you are either just advertising another fork of the great work from Peter Hinch, or you are in fact advertising a new library but trying to take over an existing project.
However, I have to say, rewriting it and uploading it to pypi as "micropython-mqtt" seems like a hostile takeover, considering that Peter Hinch's library is the most popular mqtt library and is named "micropython-mqtt". So if people install micropython-mqtt from pypi they would actually expect Peter Hinch's version and not your rewritten fork that is not even a drop-in replacement.
If you rewrite something, why don't you name it uniquely and upload it using a different name?
Also if you call it "New MQTT client library" then please also make it new by using a "new" name. Because this way it looks like you are either just advertising another fork of the great work from Peter Hinch, or you are in fact advertising a new library but trying to take over an existing project.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Re: New MQTT client library for new uasyncio
I'm sorry you feel so offended, I feel like I must have hit some hot button I didn't know about. In my mind I had associated the name mqtt_as with Peter's library, that's the name he uses all the time in all his posts on this forum, and so I picked a different name, specifically mqtt_async. When I uploaded to pypi I noticed that -mqtt was still available and didn't really think much further about it. I was mostly trying to find the shortest option given that micropython- as a prefix is already a mile long. I did not remember that his github repo has that name. Also, apparently Peter is not interested in pypi, or else he would have uploaded there long ago? I have to admit that so far I'm not thrilled by the pypi experience (first time I use it), it seems it causes more confusion than anything else in terms of versioning WRT what's in github.If Peter wants that pypi name he can have it, but then he better actually use it.kevinkk525 wrote: ↑Thu Mar 26, 2020 7:49 amI appreciate your efforts in making a new mqtt library!
However, I have to say, rewriting it and uploading it to pypi as "micropython-mqtt" seems like a hostile takeover, considering that Peter Hinch's library is the most popular mqtt library and is named "micropython-mqtt". So if people install micropython-mqtt from pypi they would actually expect Peter Hinch's version and not your rewritten fork that is not even a drop-in replacement.
If you rewrite something, why don't you name it uniquely and upload it using a different name?
Also if you call it "New MQTT client library" then please also make it new by using a "new" name. Because this way it looks like you are either just advertising another fork of the great work from Peter Hinch, or you are in fact advertising a new library but trying to take over an existing project.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: New MQTT client library for new uasyncio
I don't use PyPi and have no plans to do so.
I think it's in both our interests - and more importantly, those of our users, to avoid name clashes. They cause confusion which you and I will have repeatedly to resolve. I don't know about you, but I could do without the hassle.
While I haven't studied your library, my "off the top of my head" response is that yours and mine serve different purposes. Having a coroutine wait for an ACK has benefits on low resource targets such as ESP8266. It also serves a purpose on certain projects. This general issue has been the subject of lengthy discussions between myself and Kevin; it was a deliberate design decision rather than a random oversight. However I do agree that streaming publications have a role on suitably resourced platforms.
So can I suggest you adopt a new, unique, name which identifies the purpose and merit of your library?
I think it's in both our interests - and more importantly, those of our users, to avoid name clashes. They cause confusion which you and I will have repeatedly to resolve. I don't know about you, but I could do without the hassle.
While I haven't studied your library, my "off the top of my head" response is that yours and mine serve different purposes. Having a coroutine wait for an ACK has benefits on low resource targets such as ESP8266. It also serves a purpose on certain projects. This general issue has been the subject of lengthy discussions between myself and Kevin; it was a deliberate design decision rather than a random oversight. However I do agree that streaming publications have a role on suitably resourced platforms.
So can I suggest you adopt a new, unique, name which identifies the purpose and merit of your library?
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: New MQTT client library for new uasyncio
I saw some possibilities and possible motives so I felt a little offended (although micropython-mqtt is not my project) but I was wrong to react that way. I should have just explained how it looks and could be interpreted without jumping to conclusions and I should have given you the chance to explain why you did it that way. I apologize for that.tve wrote: ↑Thu Mar 26, 2020 8:18 amI'm sorry you feel so offended, I feel like I must have hit some hot button I didn't know about. In my mind I had associated the name mqtt_as with Peter's library, that's the name he uses all the time in all his posts on this forum, and so I picked a different name, specifically mqtt_async.
As Peter explained, it would be good if you chose a different name for your implementation to prevent problems from name clashing.
I didn't get a chance yet to have a close look at your implementation but some features look interesting. The library seems to be only for esp32 (or only tested on that platform) if I read that correctly? As I use mostly esp8266, this library is "too powerful" anyway. But I'm interested to see how your library will be adopted and how well it'll work in all the edge-cases and long-term deployments that Peter's micropython-mqtt has already been tested in.
It was interesting to see that you used CPython for testing and not the unix port. I made my fork of micropython-mqtt compatible to the unix port for testing because that way I could test an entire project using the unix port because that project would probably not run on CPython and it would be too much work to always try to be compatible to CPython as well (just for testing..).
So might be worth making your library run on the unix port as well?
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Re: New MQTT client library for new uasyncio
I did, yours is called mqtt_as and mine is called mqtt_async, what am I missing?pythoncoder wrote: ↑Thu Mar 26, 2020 10:57 amSo can I suggest you adopt a new, unique, name which identifies the purpose and merit of your library?
I re-uploaded mine to pypi as micropython-mqtt-async, awkward long but thankfully one only types this once...
Re: New MQTT client library for new uasyncio
The asynchronous publishing is optional in mqtt_async, it's simply an sync=True/False KW parameter and the default is True, i.e. wait just like mqtt_as does. In terms of resource consumption the only real overhead when choosing sync=False is having to hang onto the previous message, whether that's a problem really depends on the message size. I also compared the code size, using mpy-cross -O3, mqtt_async is a bit less than 10% larger than mqtt_as (9168 bytes vs 8360). I have not compared data memory sizes.pythoncoder wrote: ↑Thu Mar 26, 2020 10:57 amHaving a coroutine wait for an ACK has benefits on low resource targets such as ESP8266. It also serves a purpose on certain projects. This general issue has been the subject of lengthy discussions between myself and Kevin; it was a deliberate design decision rather than a random oversight. However I do agree that streaming publications have a role on suitably resourced platforms.
Re: New MQTT client library for new uasyncio
Thank you, no harm.kevinkk525 wrote: ↑Thu Mar 26, 2020 1:14 pmI saw some possibilities and possible motives so I felt a little offended (although micropython-mqtt is not my project) but I was wrong to react that way. I should have just explained how it looks and could be interpreted without jumping to conclusions and I should have given you the chance to explain why you did it that way. I apologize for that.tve wrote: ↑Thu Mar 26, 2020 8:18 amI'm sorry you feel so offended, I feel like I must have hit some hot button I didn't know about. In my mind I had associated the name mqtt_as with Peter's library, that's the name he uses all the time in all his posts on this forum, and so I picked a different name, specifically mqtt_async.
I have left esp8266 code in but have not really tested it other than to notice that it has to be frozen into the firmware to be of any use. Given the hassle I thought I'd better leave that until I stop tweaking the code. I'm also not really interested in doing a ton of work myself for the esp8266 'cause I'm trying hard to only use encrypted communication. I've ordered a PYBD and I'll see what I need to do for things to run there. I'm curious about its Wifi low power functions.kevinkk525 wrote: ↑Thu Mar 26, 2020 1:14 pmI didn't get a chance yet to have a close look at your implementation but some features look interesting. The library seems to be only for esp32 (or only tested on that platform) if I read that correctly? As I use mostly esp8266, this library is "too powerful" anyway.
Yup, I know I still have work to do and things to improve to make mqtt_async behave better when the connectivity goes south, especially if one cares about power consumption. I've found it easy to end up with a system that eventually reconnects just fine but that is non-functional in the meantime because it's trying to reconnect like crazy.kevinkk525 wrote: ↑Thu Mar 26, 2020 1:14 pmBut I'm interested to see how your library will be adopted and how well it'll work in all the edge-cases and long-term deployments that Peter's micropython-mqtt has already been tested in.
Note that when mqtt_as gets ported to the new asyncio it pretty much resets all the "long-term deployments": socket reads/writes go from non-blocking to blocking which rippled further through the code that I had expected, but YMMV.
I did run the tests on unix micropython at some point, I see that I need to tweak the damn try-import-except stuff again for it to work, thanks for pointing it out. There are a whole bunch of reasons I went for CPython: I wanted to double-check against the "official" asyncio, I find that pytest provides vastly better error reports when an assertion fails, pytest provides additional nice powerful assertions (pytest.raises, etc), and last but not least, it gives me a test coverage report so I can see which fragments of code are not tested at all.kevinkk525 wrote: ↑Thu Mar 26, 2020 1:14 pmIt was interesting to see that you used CPython for testing and not the unix port. I made my fork of micropython-mqtt compatible to the unix port for testing because that way I could test an entire project using the unix port because that project would probably not run on CPython and it would be too much work to always try to be compatible to CPython as well (just for testing..).
So might be worth making your library run on the unix port as well?
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: New MQTT client library for new uasyncio
Thank you, I think that will save both of us from needless support issues.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.