UMQTT unofficial user FAQ

[pythoncoder]

FAQ suggested rules
For the FAQ to be useful it's best if comments are restricted to new FAQ entries. Where an entry has errors or omissions, or you'd like to expand the entry I suggest sending a PM to the author with the suggested amendments, who should then edit the post. By the same token I suggest authors check and respond to PM's.

The title should be the question e.g. 'UMQTT fails to connect to server' with the message containing the suggested solution.

[pythoncoder]

MQTT is a means of communicating between multiple clients. A single server, also known as a broker, manages the network. Clients may include ESP8266 modules or other networked computers. A typical server is a Raspberry Pi or similar small Linux machine which may be left running 24/7.

Packets are passed between clients using a publish/subscribe model. Packets consist of a topic and a message, each of which (in the MicroPython implementation) must be a bytes object. Clients subscribe to a topic and will receive all packets published by any client and having that topic.

Getting started
This guide assumes familiarity with using the ESP8266 module: how to connect to the network and how to copy and run Python programs.

The server
Setting up your server is simply a matter of installing mosquitto which will (under Linux at least) automatically run as a background task (daemon) requiring no user intervention. If you wish you can also run a client on the server (typically for testing). Install mosquitto-clients and check out the documentation links below for mosquitto_pub and mosquitto_sub.

You can write PC based clients in Python using the Paho module (see link below).

The client
Precompiled binaries include the necessary libraries (release builds only, not daily development builds). The library is divided into two, umqtt.simple and umqtt.robust - the latter is designed to work in the presence of network outages, and depends on the former. To get started you may want to try the test programs in the library. The Micropython library may be copied to your PC by issuing

Code: Select all

git clone https://github.com/micropython/micropython-lib.git

Test programs may be found in the umqtt.simple and umqtt.robust directories.

Advanced users wanting the latest source can use a daily development build or one compiled from source and acquire the package from the library micropython-lib. The preferred solution is to copy the packages (i.e. the directory structure) to the ESP8266. If you have problems creating the subdirectories copy simple.py and (if required) robust.py to the ESP8266 root. If doing this the test programs will require minor modification.

Further reading:

mqtt introduction http://mosquitto.org/man/mqtt-7.html
mosquitto server http://mosquitto.org/man/mosquitto-8.html
mosquitto client publish http://mosquitto.org/man/mosquitto_pub-1.html
mosquitto client subscribe http://mosquitto.org/man/mosquitto_sub-1.html
full MQTT spec http://docs.oasis-open.org/mqtt/mqtt/v3 ... c398718048
python client for PC's https://www.eclipse.org/paho/clients/python/

[pythoncoder]

Publish-only clients
Some applications publish without subscribing, for example an ESP8266 with a temperature sensor might publish on an hourly basis. In general it's best to close the connection after each publication. If you need to keep the connection open - presumably to check for occasional publications from other clients - it's essential to call client.check_msg() regularly. This is because the server tests for client disconnection by sending periodic ping packets. These are handled automatically by check_msg(), but a failure to respond is likely to lead to disconnection by the sever, typically after a minute.

[profra]

@pythoncoder Nice (clear & simple) introduction and start of useful MQTT FAQ stream... and also good example of use...

[pfalcon]

The best approach to acquiring the software is to clone the micropython library

"The best" approach still should be to use the latest release (like just released 1.8.3) which includes these modules out of the box.

Advanced users and developers who would like to get the latest fixes/improvements to the modules are welcome to install versions from micropython-lib of course.

[pythoncoder]

Speed on the Internet is moot, however the protocol can be benchmarked on a LAN. One approach is periodically to publish messages with a particular topic while subscribed to the same topic. The round-trip delay to the server/broker can then be measured. This should be roughly equivalent to the time to communicate between ESP8266 devices on the same network. If the remote client is on the PC running the server the delay should be roughly halved.

A script using this approach may be found here: https://github.com/peterhinch/micropyth ... nchmark.py. It prints the time for each message and after 100 messages it prints maximum and minimum delays.

Results were as follows. Testing was done with a Raspberry Pi on a wired network running mosquitto as a server. The benchmark runs on a Wemos D1 mini on WiFi running at standard clock rate. Speed is dependent on QOS (quality of service) setting.
At QOS == 0 (no guarantees of delivery) times were: max 154 ms min 27 ms.
With QOS == 1 (guaranteed delivery of at least one copy of a packet) times were: max 404 ms min 242 ms.

[pythoncoder]

A brief summary at the termination of three months (intermittent) work using MQTT. In the presence of a reliable, solid WiFi signal it may well suffice. But it does not deal robustly with exceptional conditions. The following issues may become apparent if you are trying to achieve 24/7 reliability.

The robust library deals with WiFi outages, but it does not seem robust in dealing with edge conditions such as radio interference or running near to the limits of WiFi range. An issue with the library is that it uses blocking sockets. This means that underlying read and write calls can take arbitrarily long times. Under very rare conditions they can block indefinitely. A further issue with the library is that the code which deals with a publication with a qos of 1 assumes that a PUBACK packet will eventually be received from the server. Again, under rare and exceptional conditions this does not occur, leading to indefinite blocking. I have raised these points as an issue.

An option is to modify the library to use sockets with a timeout. This begs the question of how long the timeout should be. On my LAN I have observed sockets blocking for up to 1.5s but if the server is on the internet a timeout of 60s seems to be widely used. The issue of qos == 1 could be worked round by using qos == 0 and handling reliability at the application level. Whether blocking for a minute is acceptable does of course depend on the application.

Another solution may be to use a watchdog timer to reset the ESP8266. I haven't tested this.

The long term solution is a nonblocking implementation of the mqtt libraries in conjunction with an enhanced version of uasyncio. I gather this is a work in progress. From a personal perspective I've concluded that my understanding of network programming is inadequate for dealing with the numerous edge conditions which can occur and am abandoning efforts with MQTT - at least until a nonblocking library emerges which handles the exceptional conditions. Sometimes you have to admit defeat and I've failed to achieve adequate reliability in my intended application.

A final observation. My ESP8266 code occasionally fails in a curious way suggestive of underlying data corruption. The code continues to run but issues garbled text strings. While this could be a bug in my code it isn't typical of a fault in a pure Python program; I suspect an underlying issue in the ESP8266 port. Alas it's a very rare failure and I haven't been able to write a reproducible test case.

[pythoncoder]

Before attempting to use MQTT on the ESP8266 it is wise to test your server (broker) against another client based on a PC or mobile device. Suitable clients for Linux are available in the mosquitto-clients repository. This provides mosquitto_pub and mosquitto_sub client utilities.

Note that some users of servers based on the Raspberry Pi and other Debian based distros have encountered a situation where clients other than ESP8266 devices connect OK, but the ESP8266 fails. This is because the version of the server in the Debian repositories is out of date. For users of Raspbian and Debian Wheezy it consists of issuing the following commands at the terminal:

Code: Select all

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key

cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-wheezy.list
sudo apt-get update
apt-get install mosquitto

Users of Debian Jessie should issue

Code: Select all

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key

cd /etc/apt/sources.list.d/wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key

cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
sudo apt-get update
apt-get install mosquitto

Sources for the above:
http://diy.viktak.com/2015/05/installin ... -with.html
https://mosquitto.org/2013/01/mosquitto ... epository/.

[pythoncoder]

A list of public MQTT brokers is available here https://github.com/mqtt/mqtt.github.io/ ... ic_brokers.

The following repo https://github.com/peterhinch/micropython-mqtt.git contains MQTT client solutions using asynchronous programming (uasyncio). User applications should be based on uasyncio. They are intended to be "resilient" supporting true qos==1 communications with automatic retransmission. They are designed to cope with and recover from conditions such as poor WiFi connectivity and broker outages. The repo contains:

• An asynchronous resilient client for ESP8266 and ESP32.
• A means of using an ESP8266 to enable an asynchronous resilient client to run on a Pyboard.

When running on an ESP32 the client has some non-ideal hacks which block for 20ms to yield to the underlying RTOS. This is because the ESP32 firmware is still under development. Hopefully these will be removed as the port matures.