Cannot get umqtt.robust to work (subscription)

[JeroenH]

Hello, for the past couple of days I've been googling, tinkering, trying, cursing, taking breaks etc. but I'm stuck.

I have a generic ESP32 board running the latest Micropython. I have installed umqtt.robust with upip, and very quickly and easily I got publishing a message to my Mosquitto broker to work. However, trying to get the subscription part working has me stumped. From several websites I have cobbled the following together, which I think should work:

Code: Select all

from umqtt.robust import MQTTClient

def sub_callback(topic,payload):
  if topic == "test/test":
    print("message received")
    print(payload)

mqtt = MQTTClient('esp32test', '192.168.178.84')
print("*** Connecting to MQTT broker...")
mqtt.connect()
print("*** MQTT broker connected")

mqtt.set_callback(sub_callback)
mqtt.subscribe("test/test")

while True:
  mqtt.check_msg()

... but it does not. What happens when I run it, is it just outputs this:

*** Connecting to MQTT broker...
*** MQTT broker connected

When I push a message to the test/test topic, the sub_callback does not appear to be called, because the text "message received" never appears.

The board is correctly connected to the wifi. I can ping it from my computer, and in any case, a script that publishes an mqtt message works fine. I push a message to the test/test topic with a python command on my Raspberry Pi, and on a mqtt client on my phone I see the message appear. I have other channels running on my broker which all work fine and fast, so my broker seems fine. I have tried different topic names, clientIDs, but it just doesn't work. Just to make sure it's not the board, I have tried a 2nd ESP32, same result.

I'm hoping (and expecting) it's just something I'm doing wrong that I'm not understanding. Can anyone point out my mistake?

[JeroenH]

I've gotten a little further. I've changed the script to print a short progress message between all the separate mqtt commands, to see how long things take, where it goes wrong, etc.

Code: Select all

print("Start script")
from umqtt.robust import MQTTClient

def sub_callback(topic,payload):
  print("test")
  print(payload)

print("Defining MQTT broker")
mqtt = MQTTClient('esp32test', '192.168.178.84')
print("Connecting")
mqtt.connect()
print("Setting callback function")
mqtt.set_callback(sub_callback)
print("Subscribing to topic")
mqtt.subscribe("test/test")
print("Starting check loop")
while True:
  mqtt.check_msg()

With this script I now have seen that the mqtt.subscribe command sometimes doesn't finish. When plugging the ESP32 in, this is the output I see most of the time:

Code: Select all

Start script
Defining MQTT broker
Connecting
Setting callback function
Subscribing to topic

So it doesn't finish the mqtt.subscribe, and never gets to the message check loop. However, sometimes (maybe 20% of the time I plug the board in or reset it), I get this output:

Code: Select all

Start script
Defining MQTT broker
Connecting
Setting callback function
Subscribing to topic
Starting check loop
test
b'Test message'

(The last two lines obviously after I've sent a message to test/test via the broker)

Occasionally the script stops after "connecting" and throws an error, I've seen that maybe once or twice in 50 starts. This is with the exact same code as above. The output then looks like this:

Code: Select all

Start script
Defining MQTT broker
Connecting
Traceback (most recent call last):
  File "main.py", line 14, in <module>
  File "umqtt/simple.py", line 97, in connect
IndexError: bytes index out of range
MicroPython v1.12 on 2019-12-20; ESP32 module with ESP32
Type "help()" for more information.

So, in principle, my code works, which is always nice However, it appears to be unreliable and I have the feeling that's going to be very hard to solve that. Is the mqtt.robust library maybe not so robust?

For this system to be useful I need it to be absolutely reliable.

[pythoncoder]

It works here on ESP32 with one change:

Code: Select all

from umqtt.robust import MQTTClient

def sub_callback(topic,payload):
  if topic == b"test/test":  # CHANGE: topic is a bytes instance not a string
    print("message received")
    print(payload)

mqtt = MQTTClient('esp32test', '192.168.0.10')
print("*** Connecting to MQTT broker...")
mqtt.connect()
print("*** MQTT broker connected")

mqtt.set_callback(sub_callback)
mqtt.subscribe("test/test")

while True:
  mqtt.check_msg()

I'm publishing using:

Code: Select all

mosquitto_pub -t test/test -m "this seems to work"

[JeroenH]

It works here on ESP32 with one change:

Yeah, I removed that if statement altogether in the code in my second post. Just to be sure I copy/pasted your code and ran it. I get the same hit and miss behaviour at startup, though It just won't start reliably. I'm pulling my hair out why this could be.

* Broker is fine. Messages arrive at other subscriber to same topic (and other topics)
* ESP32 board is fine. It's been running al my Micropython experiment code fine for months, and I have now tried a total of three boards, which all display the same behaviour with this code.
* The wifi/network part is fine. I can ping the board, and the code that publishes mqtt messages runs flawlessly.
* I have the latest micropython on the board, and the latest umqtt.robust library.
* Power (always a potential problem with ESP32s, especially when wifi is involved) is fine. I use a good quality USB cable, and one of my boards even has a 18650 battery which should easily be able to supply all the power peaks the wifi draws.

I don't want to give up and go back to Arduino

[JeroenH]

It's all different than I thought. It works 100% of the time, the startup is just very, very slow.

I've now done three tests where I've booted up the board, waited until it seemingly hung (or even got to the check loop), sent a message to test/test with a time stamp and watched some videos on YT. Then, after a full five minutes (!) the ESP32 printed that it had received the message. (Phone mqtt client had received the message in question directly after sending)

So far, I've done this experiment three times, on two boards, and it worked every time.

Does any one have idea what could cause this? Want I want to build could work with this odd quirk, I just would like to understand why this is happening.

[pythoncoder]

I'm baffled. I first ran my connect script, which takes perhaps 5s to connect to the LAN. Then I pasted the code at the REPL and switched to another terminal with an SSH link to my server. I ran mosquitto_pub and the ESP32 responded immediately.

I can only suggest putting in print statements to find out what's taking the time.

[JeroenH]

I'm baffled. I first ran my connect script, which takes perhaps 5s to connect to the LAN.

Yeah connecting to the WLAN is always quick on my boards too.

I can only suggest putting in print statements to find out what's taking the time.

I was afraid it might come to that. I'll print/timestamp the relevant portions of the library and see if it becomes clear where it slows down. I'll report back. Many thanks for your input so far.

[JeroenH]

By the way, this is what I'm building:



It's an ESP32 glued to the back of a 3.2" ILI9341 LCD. The small PCB has a BME280 sensor, the temperature value of which is shown on floor number two. The temperatures of the other rooms in my house will need to come from other ESP32s via MQTT (Mosquitto on a Rasberry Pi). Now working on the fun stuff related to time; things like leading zeroes and DST. When I've got that sorted out I'll return to the MQTT stuff.

If (sorry, when ) I've finished this functionality, I might extend it to showing the power produced by the solar panels, power drawn from/fed into the grid, etc. The possibilities are endless, really, it's just that my time isn't

[kevinkk525]

This is a very nice project!

The more features you add, the more difficult it might get with umqtt.robust.
I would suggest looking into uasyncio and an async mqtt client: https://github.com/peterhinch/micropython-mqtt

(or alternatively you can have a look at my framework that takes care of everything in the background, so you only have to program your sensors and lcd. But you'll learn more by doing it yourself from the ground up)