A "resilient" asynchronous MQTT client

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

A "resilient" asynchronous MQTT client

Post by pythoncoder » Mon Jul 24, 2017 10:59 am

An attempt to provide asynchronous operation and supply improved operation in the presence of unreliable WiFi, such as in the presence of interference or near the limit of range. It also offers true qos == 1 operation with duplicate message publication.

The official "robust" MQTT client is commendably lightweight but has the following limitations:
  • It uses blocking sockets which can cause execution to pause for arbitrary periods when accessing a slow broker. It can also block forever in the case of qos == 1 publications while it waits for a publication acknowledge which never arrives; this can occur if a WiFi outage occurs at this point in the sequence. This blocking behaviour limits compatibility with asynchronous applications.
  • It is unable reliably to resume operation after a WiFi outage.
  • Its support for qos == 1 is partial. It does not support retransmission in the event of a publication acknowledge being lost. This can occur on a WiFi network especially near the limit of range or in the presence of interference.
  • Its partial qos == 1 support and inability reliably to resume after a WiFi outage places a limit on the usable WiFi range. To achieve reliable operation the client must be well within range of the access point (AP).
This module aims to address these issues, at the cost of significant code size. It is too large to compile on the ESP8266 and works best as frozen bytecode.

Testing proved difficult because of the number of possible failure modes, however in my testing it is reliable and improves the usable range. If anyone fancies testing it on an ESP32 I'd be keen to hear the results; also with SSL/TLS which I've been unable to test.

[EDIT]
The URL:https://github.com/peterhinch/micropython-mqtt.git
Peter Hinch
Index to my micropython libraries.

tknp
Posts: 5
Joined: Sun Feb 19, 2017 6:26 am

Re: A "resilient" asynchronous MQTT client

Post by tknp » Wed Jul 26, 2017 6:24 am

This is looking great! I was able to get your client into a project and have it subscribe and publish perfectly after a little poking around. I had to change https://github.com/peterhinch/micropyth ... as.py#L162 to

Code: Select all

if e.args[0] not in [uerrno.EINPROGRESS, uerrno.ETIMEDOUT]:
in order for the client to connect. I haven't done much troubleshooting yet on why that is so but will take a look when I can.

Otherwise, great work and many thanks.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: A "resilient" asynchronous MQTT client

Post by pythoncoder » Wed Jul 26, 2017 6:40 am

Thanks for testing and reporting. As the code comment indicates my way of opening a nonblocking socket is borrowed from the uasyncio library. I wonder why such a socket would issue ETIMEDOUT?

In any event I can't see this change doing any harm, so I'll implement it and re-test. In the mean time if you get any insight as to why it happens I'd be glad to hear it.
Peter Hinch
Index to my micropython libraries.

torwag
Posts: 220
Joined: Fri Dec 13, 2013 9:25 am

Re: A "resilient" asynchronous MQTT client

Post by torwag » Fri Jul 28, 2017 9:08 am

Hi Peter,

as I understood your main project, you are going to use the ESP as a MQTT-WIFI module, as it was intended orignally (plus the MQTT). Enabling any other controller to speak to a MQTT broker via wifi. Do I get this right?

[EDIT] I got a bit confused until I reread your first statement in the README (maybe you could enhance this a bit to make it more clear). Just wondering, if the communication between host and esp could be reduced down to a protocol which is all about sending some strings, the topic could be switched to MQTT via WIFI for microcontrollers lacking WiFi connectivity...
You gave an example using a pyboard, but further enhancing the protocol and reducing the code requirements on the host, right now I can't see why it should not work with any other implementation in e.g. an arduino board.

For very simple applications, like sensor and actuator nodes, etc. Would it be possible to use the ESP8266 directly with your MQTT-as client implementation? E.g. would it work on those sonoff devices. Or do you see the code base as to big to add any extra functionality beside the mqtt stuff?

Thanks
Torsten

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: A "resilient" asynchronous MQTT client

Post by pythoncoder » Fri Jul 28, 2017 1:14 pm

Hi Torsten, sorry if my README is unclear. There are now two projects:
  • The "resilient" MQTT client. This is an alternative to the official client.
  • Using an ESP8266 as a peripheral to add MQTT to non-networked boards.
The first runs on an ESP8266 target (and hopefully ESP32). It's simply a more full-featured client than the official one. It's intended to be used with sensors and/or actuators with the whole application running on the target. It uses about 50% of the ESP8266 RAM so there is reasonable scope for actually using it ;). If your application is substantial you're better off with the lightweight official client or an ESP32.

The second uses the ESP8266 as a cheap peripheral. You flash it with my pre-built firmware image, wire it to the target board, and run the supplied driver on the target. The target then has MQTT access. This should work with any MicroPython target capable of supporting five Pin instances. It using no special hardware related features (e.g. UARTs, timers, I2C, SPI, special code emitters, Assembler...). It will work regardless of processor speed so should be highly portable. But it is written in MicroPython.

If you wanted to use it with a board which doesn't support MicroPython (e.g. AVR based Arduinos) you'd need to rewrite my client in a supported language (C in the case of Arduino). This wouldn't be particularly hard, but I have no plans actually to do it.
Peter Hinch
Index to my micropython libraries.

torwag
Posts: 220
Joined: Fri Dec 13, 2013 9:25 am

Re: A "resilient" asynchronous MQTT client

Post by torwag » Mon Jul 31, 2017 12:19 pm

Hi Peter,

I got your MQTT-as working on a standard sonoff device.
Even more, I was able to use your aswitch and Delay_ms functions to create a proper asynchronous function for the (on board) press button. Having a device as as small as sonoff and even three possible trigger with a single button is really nice.
With a little hardware hack, one can free another GPIO (albeit there is now already a two channel version). This enables one to switch another (lower power DC device) or read in a sensor.

Right now I am really happy with the solution. I need to give it some "real world" testing to see how it behaves in case of network failures but overall it looks very good. I added mqtt_as and the uasyncio libs as froozen bytecode to the firmeware.
That gives me still enough space for quite a bit of logic:

DONE:
long button press: turns on resp. off the device
double button press: turns the device on (or resets the timer) and set a timer to be off in 5 min
each of those are accompanied with a proper MQTT message
receiving MQTT messages and turn on or turn off the device

MAYBE:
I could go with a the idea to send a timer information in addition via MQTT, then I would not need to turn the device off, but receive it is switched off. That would save sending an additional mqtt message and might have some safety advantages as there wil be no situation where the device keeps on, due to a lost network in the meantime. I will investigate this.
A overall watchdog. The device should turn off, as soon as the watchdog runs out of time, e.g. due to some network problems

Just to give you an idea of the application:
I installed an irrigation system in my greenhouse. To operate it, I have to switch on a 230V pump and open a valve, as the pump is usually be used for a few other watering tasks. If this works out reliable with a single sonoff unit, and some wiring for the 12V valve, I am pretty happy. It will be added to my home assistant server and in the future maybe some humidity and water sensors will be added as well.

Finally one small challenge remains:
Starting the esp, I get the following error from time to time, dropping back in REPL.

Code: Select all

Waiting for network to come up
WebREPL daemon started on ws://192.168.4.1:8266
WebREPL daemon started on ws://0.0.0.0:8266
Started webrepl in normal mode
b'dd2e0700'
Connected to broker
Traceback (most recent call last):
  File "main.py", line 101, in <module>
  File "main.py", line 99, in <module>
  File "uasyncio/core.py", line 132, in run_until_complete
  File "uasyncio/core.py", line 121, in run_forever
  File "uasyncio/core.py", line 85, in run_forever
  File "uasyncio/core.py", line 129, in _run_and_stop
  File "main.py", line 70, in main
  File "mqtt_as.py", line 354, in connect
  File "mqtt_as.py", line 202, in _connect
  File "mqtt_as.py", line 118, in _as_read
OSError: [Errno 103] ECONNABORTED
As you can see, I took care already that the network is up and ready. Only then I go on and start the MQTT client.
I thought I have a race condition between some of your code and the time it takes to set-up the wifi network.
Calling import main from the REPL again, works always.
Any idea on that?

Finally also a big thanks to @pfalcon who worked on the underlying uasyncio function... GREAT JOB!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: A "resilient" asynchronous MQTT client

Post by pythoncoder » Tue Aug 01, 2017 7:46 am

Hi Torsten - as far as I can see you haven't provided a link to your code so I'm having to guess a little from the traceback. It seems it's failing in the initial connect(). This is because subsequent reconnection attempts should trap all OSErrors and handle them.

If this is the case, it's by design. I couldn't see any sensible way for the driver to respond if the network goes down or the broker fails while initially attempting the connection. I intended that the application would trap such an error and handle it in an application specific manner. For example an app might tell the user and re-try at user discretion once the problem was fixed.

You may find that a simple delay between the network appearing to be OK and attempting a broker connection will suffice. In my testing where the ESP8266 runs from a battery my main.py has a delay of five seconds before running the test program.

However if it's failing after working for a while then I've definitely made a mistake - please let me know.
Peter Hinch
Index to my micropython libraries.

torwag
Posts: 220
Joined: Fri Dec 13, 2013 9:25 am

Re: A "resilient" asynchronous MQTT client

Post by torwag » Tue Aug 01, 2017 9:59 am

Hi Peter,

yes no link to the code yet. A bit work in progress and a bit shy to upload it yet (yeah it is still quite messy).
I added a 2 seconds delay and I added a loop which is only left if the network is reported of being active.
Nevertheless, I get this error right after power-up.
import main
or
Ctrl-D
works all the time.
I will try to increase the time delay. Maybe this helps.
As for the initial phase as you described, I agree. Have to see how I could inform the user, on the sonoff, as all I have is a button and a LED. BTW. Right now I am scratching at the RAM limit. There is already some logic implemented (as written earlier) but yeah, it is definitely RAM restricted. I did not do any optimisations yet. I might can get a few more functions into it by optimising some code. However, it is enough to get simple user interactions and quite some program logic running. Thus, I would say sufficient for many IoT applications. Now it is time to work on a scheme for my MQTT protocol. I'm using ujson to send and receive json-style messages. Now I need to think about the structure in general. Did you investigate into this already?

Unfortunately, this is all at home... I can only test and continue it this evening... silly, some of us have still to go to work everyday :D
Thanks for the great code-base and the docs, it is really a pleasure to work with this.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Update

Post by pythoncoder » Tue Aug 22, 2017 4:16 pm

I have today posted an update to both of these projects. The "resilient" async client has a few fixes. It also has an improved way of dealing with the initialisation parameters - the module provides a dict of defaults which may optionally be modified on both a cross-project and per-project basis.

The project using an ESP8266 card as a peripheral to another MicroPython board has had a major revision. The ESP8266 firmware now uses the above asynchronous code rather than the official driver. This means that forced reboots of the ESP8266 only occur if the ESP8266 has crashed - previously they occurred under various error conditions. This should improve resilience and reduce latency caused by error conditions. I have also done work to improve the performance of the interface between the boards with a view to reducing latency when connectivity is good.

There is a significant API change in the way that application coroutines using the interface are launched and terminate, aimed at improving reliability under adverse conditions.

I have adapted the "resilient" async client to detect and run on the ESP32. However the ESP32 firmware appears to have issues so the outcome is at present not very resilient. See the docs for details.

My intention is that future updates will be backwards compatible - apologies to existing users for changing the API. Aside from bugfixes my future aim is to properly support the ESP32.

I would appreciate any assistance into understanding why SSL doesn't work. There is an example in mqtt_as/ssl.py - its failure to work is doubtless down to my inexperience with the technology.
Peter Hinch
Index to my micropython libraries.

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: A "resilient" asynchronous MQTT client

Post by cefn » Mon Dec 04, 2017 4:11 am

Do you have an intuition for the number of messages that an ESP8266 could receive each second using this library? Is it an order of magnitude around 1000 or around 10, for example. That's assuming the handler was a noop and messages contain e.g. 4 bytes.

How might this compare to a less resilient library?

I am trying to design a protocol which chains MQTT over wi-fi to multiple nodes, which relay the information locally over serial to a driver for WS2811 chains so timing and throughput is important.

Post Reply