paho-mqtt for MicroPython

[danielm]

I took a quick look at current implementation of paho-mqtt for Python. It has following dependencies:

errno - several error codes used in the code (following module could be extended with necessary error codes and imported https://github.com/micropython/micropyt ... o/errno.py)

platform - platform.system() used once to detect if running on Windows, can be omitted

random - used once to generate client_id, I guess that pyb.rng() could be used

select - available in MicroPython

socket - usocket available - not sure if all required functions are supported (setsockopt() seems to be supported; getfqdn() is used to determine local domain, seems to be missing but not mandatory; other unsupported function is create_connection() - it seems to be higher-level function based on connect(), bind() and settimeout() functions (https://docs.python.org/2/library/socke ... connection) - I guess that missing function create_connection() could be defined additionally or in better case implemented usocket direcly )

struct - available in MicroPython

sys - available in MicroPython (only values version and version_info are used)

time - available in MicroPython

threading - this is what I am afraid of, Lock() and Thread() functions used, could uasyncio module be used instead of threading?

Thanks a lot for your opinion.

[kfricke]

...
threading - this is what I am afraid of, Lock() and Thread() functions used, could uasyncio module be used instead of threading?

This should be modular and "skipable". The CPython paho-mqtt has several methods to handle the network connection. Threading is only one and the polling methods should be more practical on MicroPython.

[kfricke]

After checking the paho-mqtt library last night I' would suggest to at least fork it and strip all the threading code from it. I started by trying to import the paho-mqtt library in the Unix port of MicroPython and everything seemed to play nicely until encountering the limits and specialties of the usocket module. Synchronization during threaded network IO is done by self-referencing sockets and the suport code imho bloats the paho-mqtt module a little to far from being reasonable to cleanly port/fork to/for MicroPython.

I will have a look into the Arduino MQTT libraries and maybe borrow some coding-inspiration from there to implement a more lightweight MQTT module for MicroPython.

MicroPython will not ever really support threading and I would say that non-blocking IO should be good enough and reasonable because of being portable across all supported platforms.

Question: Is this correct, or does some of the major platforms not support non-blocking-/async-IO on the network socket?

[pfalcon]

After checking the paho-mqtt library last night I' would suggest to at least fork it and strip all the threading code from it. I started by trying to import the paho-mqtt library in the Unix port of MicroPython and everything seemed to play nicely until encountering the limits and specialties of the usocket module. Synchronization during threaded network IO is done by self-referencing sockets and the suport code imho bloats the paho-mqtt module a little to far from being reasonable to cleanly port/fork to/for MicroPython.

Reading danielm's questions, I find myself wanting to go to level up and answer more general questions about MicroPython. I refrained to not "scare away" a novice, but given that words like "bloat" were mentioned more than once already, I'd rather give a try. So, (and here I express only my personal PoV) the reason MicroPython exists is not because folks who develop it thought: "There're lot of libraries for Python, let's let people use them on microcontrollers". It was: "Python is nice language, in which you can quickly write quality software, let's run it on microcontrollers". Ok, then what about all the existing Python software and libraries? Sad news is that half of its is "bad". "Bad" defines to various criteria, and the curse of any software in high-level language is over-design and over-engineering. You see, high-level language allows to do useful things in dozens lines of code and complex things in hundreds, but people don't feel well until they churn out thousands and thousands of lines. No such code will run with MicroPython.

Take a spec of MQTT. See if it mentions that protocol should be done differently on Windows and non-Windows. If not, then a library which tries to query for Windows vs non-Windows doesn't really do MQTT protocol. It does something different, like trying to build a whole world on a needle's tip. Such approach doesn't work well with MicroPython.

I will have a look into the Arduino MQTT libraries and maybe borrow some coding-inspiration from there to implement a more lightweight MQTT module for MicroPython.

MQTT is another dead simple protocol. Reading a spec will produce max a hundred lines of Python code in an hour (we're speaking about client implementation here, right?). Looking around will definitely take more time. (And it already should be clear that MQTT is so simple, that people can't resist temptation to build a Jack's house while working on it.)

MicroPython will not ever really support threading and I would say that non-blocking IO should be good enough and reasonable because of being portable across all supported platforms.

Why, it will - as soon as someone who really needs it will implement it. Or otherwise who will sponsor its development. So far, the closest party is ESA, which sponsors Damien's work.

Question: Is this correct, or does some of the major platforms not support non-blocking-/async-IO on the network socket?

That's rather peculiarly formulated question - what "major platforms" do you mean?

[kfricke]

...but given that words like "bloat" were mentioned more than once already...

I do not want to pull this cite out of context too far, just to mention that i did not add "bloated" to MicroPython but that library mentioned. So we do agree that the paho-mqtt module is an example for this.

Take a spec of MQTT. See if it mentions that protocol should be done differently on Windows and non-Windows. If not, then a library which tries to query for Windows vs non-Windows doesn't really do MQTT protocol. It does something different, like trying to build a whole world on a needle's tip. Such approach doesn't work well with MicroPython.

In this particular case they need to take differences in the socket implementation into account. So not a needle tip approach, but a portability. That of course is good for the CPython world, but as mentioned nothing for the MicroPython cosmos.

MQTT is another dead simple protocol. Reading a spec will produce max a hundred lines of Python code in an hour (we're speaking about client implementation here, right?). Looking around will definitely take more time. (And it already should be clear that MQTT is so simple, that people can't resist temptation to build a Jack's house while working on it.)

Maybe I will try to do this. And yes, the MicroPython module should perfectly useable as a MQTT client with pub&sub support.

MicroPython will not ever really support threading and I would say that non-blocking IO should be good enough and reasonable because of being portable across all supported platforms.

Why, it will - as soon as someone who really needs it will implement it. Or otherwise who will sponsor its development. So far, the closest party is ESA, which sponsors Damien's work.

My fault then. That was my interpretation of various topics regarding this. My understanding was that is too complex for MCUs and coop async schedulers are the way to go.

Question: Is this correct, or does some of the major platforms not support non-blocking-/async-IO on the network socket?

That's rather peculiarly formulated question - what "major platforms" do you mean?

1. STM HAL
2. CC3200
3. Unix (and Android)
4. ESP8266
5. PIC 16-Bit

[danielm]

kfricke, I think that a lot of people from MicroPython community would really be grateful for efficient and resource-saving implementation of MQTT client for MicroPython. From my point of view, emulation of HTTP requests si the only way how to talk to the Internet taking into consideration current state of MicroPython implementation and available libraries. And this is definitely not suficient for developing IoT solutions on this platform. Or did I miss any other possible solution?

[danielm]

Yesterday I tried to switch protocol of backend -> browser UI update channel of my solution to mqtt and it works quite well, better than socket.io I used paho-mqtt for python backend (over TCP) and mqttws31.js (over WebSocket) for browser side running on test.mosquitto.org.

[kfricke]

Do not proceed too fast. I just began to read myself into the MQTT spec

[danielm]

I just wanted to get some experience with implementation of MQTT use-case:)
One fact is slightly dissapointing - the message can carry only single payload, at least it seems so. Anyway it was easy to encode several values to JSON string and decode it browser-side.

[kfricke]

MQTT is by design payload content agnostic. So encoding multiple payload parts is up to the application. As you did decide to JSON, I do also encode most of my valuable MQTT payloads in JSON strings.

As for a use-case...

I have an ESP8266 based sensor network spread across our house. The sensor nodes are publishing several sensor readings (temperature, humidity and supply voltage) to the "/raw_sensors/<sensor-ID>" topic on my MQTT broker. These sensor nodes do publish simple strings containing multiple tab separated sensor readings.
A python scripts is running on my Linux server, which is subscribed to that MQTT topic containing raw sensor payloads. Its only purpose is to enhance the payloads on "/raw_sensors/#" with the timestamp of the original MQTT message, recode them into JSON strings and publish the enhanced message onto another MQTT topic called "/sensors/<sensor-ID>".
The next step on my to-do list is to spread the sensor readings across multiple topics:

• "/sensors/temperature/<sensor-ID>"
• "/sensors/humidity/<sensor-ID>"
• "/sensors/vcc/<sensor-ID>"

And later collect the sensor readings in a "[url=http://graphite.readthedocs.org/en/latest/[/url] bucket".

The following is a small capture of MQTT messages on my broker

Code: Select all

$ mosquitto_sub -v -t "/sensors/#"
/raw_sensors/2A0EA7D7D Humidity=51.82, Temperature=20.23, VCC=3.80
/sensors/2A0EA7D7D {"humidity": "51.82", "temperature": "20.23", "timestamp": "2015-10-22T17:41:36.100254", "vcc": "3.80"}
/raw_sensors/2A0EA7FFD Humidity=49.68, Temperature=21.69, VCC=3.82
/sensors/2A0EA7FFD {"humidity": "49.68", "temperature": "21.69", "timestamp": "2015-10-22T17:41:58.422221", "vcc": "3.82"}
/sensors/2A0EA7DFD {"humidity": "56.15", "temperature": "21.29", "timestamp": "2015-10-22T17:42:19.353905"}

("/sensors/2A0EA7DFD" is a locally attached Arduino board with no local battery in need of power monitoring. Another stupid python script does publish sensor readings read from "/dev/ttyACM0")

The sensor messages are all marked with the "retain" flag. This is a very easy way to let the MQTT broker make the last message on a topic to be persistent. Subscribing to such a topics does always deliver the last retained message, even when the subscriber was not connected while the message got sent.

There are several other "home automation" projects doing this kind of stuff, but it is more fun to plug a few software components together, throw in a few hundred lines of python code and see such things evolve.

Sadly there is no MicroPython involved in this project yet. Another reason why I am waiting for my WiPys to arrive. The ESP8266 port has no I2C yet which I need for my sensors :-/
And i hope to get reasonable ultra-low power performance of the WiPy as well.