MQTT breaks infinite loop

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
dnoha
Posts: 7
Joined: Sat Jan 11, 2020 11:43 am

MQTT breaks infinite loop

Post by dnoha » Thu Oct 07, 2021 6:06 pm

I just can't figure out why this breaks run after few seconds or minutes?

Device is D1 Mini V3.0.0.
It works better when serial cable is not plugged into PC.
WiFi coverage is really good at test spot.

I'm pretty sure that there is something with MQTT connection, but don't know how to test and repair.
Serial monitor (within Thonny) thus not print out anything, except while fresh restart of ESP and Thonny.

Code: Select all

from umqtt.simple import MQTTClient
import machine
import utime
import ubinascii

from config import SERVER, COMMAND_TOPIC, STATE_TOPIC, AVAILABILITY_TOPIC

#sensor setup and read function
import sensor_DS18X20

import gc
gc.collect()

#onboard LED is on pin 2 
LED = machine.Pin(2, machine.Pin.OUT, value=1)

#disply module
import OLED
OLED.redraw("--", "starting", ".")


#unique ID
CLIENT = None
CLIENT_ID = ubinascii.hexlify(machine.unique_id())
 
#MQTT messages handling background function 
def new_msg(topic, msg):
    print("Received {}".format(msg))
    if msg == b"on":
        LED.value(0)
        CLIENT.publish(STATE_TOPIC, "on")
    elif msg == b"off":
        LED.value(1)
        CLIENT.publish(STATE_TOPIC, "off")
    gc.collect()


 
#main loop 
#def main():
    
#global CLIENT
CLIENT = MQTTClient(CLIENT_ID, SERVER)
CLIENT.set_callback(new_msg)
CLIENT.connect()
CLIENT.subscribe(COMMAND_TOPIC)
# Publish as available once connected
CLIENT.publish(AVAILABILITY_TOPIC, "online") 

print("Connected to {}, subscribed to {} topic".format(SERVER, COMMAND_TOPIC))


temp_current = 99
temp_min = 18
temp_max = 23
temp_offset = 1
rstate = "idle"


last_temp_time = utime.time()

gc.collect()


try:
    while 1:
                
        CLIENT.check_msg()
        
        dt = utime.time() - last_temp_time
        OLED.redraw(str(temp_current), rstate, str(dt))

        
        if dt > 5:
                           
            # read sensor
            temp_current = round(sensor_DS18X20.temperature,1)
            # after read, some time is needed for conversion (750 ms from used example)
            # therefore UI and states update is set in later time (next 'if')
            
        
        if dt > 6:
            
            # save time of this update
            last_temp_time = utime.time()

            # evalueate temperature and set state 
            print(temp_current)
            
            if temp_current < temp_min:
                print("set to heating")
                rstate = "heating"
            if temp_current > temp_max:
                print("set to cooling")
                rstate = 'cooling'
            if (temp_current > (temp_min + temp_offset)) and (temp_current < (temp_max - temp_offset)):
                print("set to idle")
                rstate = 'idle'
            
            # refresh displays
            #OLED.redraw(str(temp_current), rstate, str(dt))
            
except OSError as exc:
    print("EXCEPTION")
    print(exc.errno)

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: MQTT breaks infinite loop

Post by pythoncoder » Sat Oct 09, 2021 9:35 am

The official MQTT libraries are designed on the assumption of a perfect network connection and they can lock up in the event of disruption to the WiFi. A WiFi system can seem fine on PC's yet experience disruption which breaks these libraries.

This library includes mqtt_as, an asynchronous MQTT client, which has undergone long periods of testing. On an ESP8266 perfect reliability seems unattainable but run times of many days can be expected and it can reboot on failure with machine.WDT.
Peter Hinch
Index to my micropython libraries.

dnoha
Posts: 7
Joined: Sat Jan 11, 2020 11:43 am

Re: MQTT breaks infinite loop

Post by dnoha » Sat Oct 09, 2021 12:05 pm

Thank You for this input, peterhinch/micropython-mqtt looks good, but this might be obsticle for user like me:
on the ESP8266. It must either be cross compiled or (preferably) built as frozen bytecode
I have choosen Micropython hoping for simplicity. Compilig firmware probably is not too hard if you have all tools for that. I have nothing, and seting-up machine and tools for that now looks to much hustle. Maybe there is already precomplied Micropython firmware with this library (mqtt and uasyncio) for ESP8266 board?

Or maybe there is solution to copy some kind of pre-compiled library directly to Micropython filesystem?
(like *.dll or *.bin :?: )

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: MQTT breaks infinite loop

Post by pythoncoder » Mon Oct 11, 2021 7:10 am

I've put a precompiled version here. Download the file (mqtt_as.mpy), put it in your filesystem, and it should work.

The reason I don't include this in the library is that the bytecode format changes from time to time, so .mpy files need to broadly match firmware builds. So you'll need to run recent firmware.
Peter Hinch
Index to my micropython libraries.

dnoha
Posts: 7
Joined: Sat Jan 11, 2020 11:43 am

Re: MQTT breaks infinite loop

Post by dnoha » Mon Oct 11, 2021 6:36 pm

It works !

Thank you for Your time compiling this. Actually, until now I was'nt aware of *.mpy binaries :oops: I thought that only option is to build custom firmware. Btw, now I need to learn also uasyncio ..

One thing to note is that "stop/restart backend" (within Thonny) thus not start program, but "send EOF / soft reboot" seams to work ever time. And yes, I did put time.sleep(5) at start of main.py.

In https://github.com/peterhinch/micropyth ... er/mqtt_as this "from config import config" probably should be "from mqtt_as import config".

Once again, thank you for help and awsome library !!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: MQTT breaks infinite loop

Post by pythoncoder » Tue Oct 12, 2021 9:52 am

See this doc for information on how to create .mpy files.

Thanks for pointing out the error in my doc: will fix.

Re learning uasyncio have you seen my tutorial?
Peter Hinch
Index to my micropython libraries.

beetle
Posts: 51
Joined: Sat Oct 16, 2021 11:35 am

Re: MQTT breaks infinite loop

Post by beetle » Sat Oct 16, 2021 11:51 am

pythoncoder wrote:
Tue Oct 12, 2021 9:52 am
See this doc for information on how to create .mpy files.
Thanks for the mqtt_as.mpy file you linked to, most useful.

Also a thanks for the link above to the info on how to create .mpy files. But for a rpi python user, coming to micropython, the docs leave me lost. The doc says 'build it as usual - $make'. So I download all the files in the directory mpy-cross to a directory on my mac. I go to the directory and I type 'make' and just get a few lines about xxx - No such file or directory.

So what does 'build as usual - $make' mean. Obviously I have more to learn. But where to start? Any pointers to some 'using make for Dummies' type of doc's.

beetle
Posts: 51
Joined: Sat Oct 16, 2021 11:35 am

Re: MQTT breaks infinite loop

Post by beetle » Sat Oct 16, 2021 9:59 pm

And to answer my own question on creating mpy files:
I don't know why theres all this 'do make' malarky. After faffing about I found the following which I give for anyone else looking for an easy answer:
1. import mpy_cross into python running on your computer - well it worked on my mac computer.
pip3 install mpy-cross

2. run the mpy_cross (yes now an underscore) to create a .mpy file from a micropython .py file
python -m mpy_cross yourfile.py

And out pops a yourfile.mpy file Blooming marvellous, but I had to go around the houses to find this info. There must be some simpler documentation on all this somewhere I guess, but if it does exist its not easy to find.

dnoha
Posts: 7
Joined: Sat Jan 11, 2020 11:43 am

Re: MQTT breaks infinite loop

Post by dnoha » Mon Nov 01, 2021 5:58 pm

I'm probably not using mqtt_as properly?

Probably problematic part is connection section:
# START CONNECTION TO MQTT (ASYNC)
asyncio.run(main(client))
Main use of this program is controlling fridge temeprature. But main try while true loop thus not start until mqtt connection is established. That's not good. I should have temperature controll function independently of wifi or mqtt part working or not!

How to reconfigure structure of this code?

Code: Select all

import time
time.sleep(5)  # Could probably be shorter

from config_mqtt import SERVER, COMMAND_TOPIC, STATE_TOPIC, AVAILABILITY_TOPIC

import sensor_DS18X20

import utime
temp_current = 99
temp_min = 18
temp_max = 23
rstate = "idle"


from mqtt_as import MQTTClient
from mqtt_as import config
import uasyncio as asyncio


import OLED
OLED.redraw("--", "--", "--", "starting", ".")


def subs_cb(topic, msg, retained):
    print((topic, msg, retained))

async def connect_coro(client):
    await client.subscribe(COMMAND_TOPIC, 1)


async def main(client):
    await client.connect()
    print("fridge1 connected to mqtt")
    # If WiFi is down the following will pause for the duration.
    await client.publish(STATE_TOPIC, '{}'.format(temp_current), qos = 1)
      

async def read_sensor():
    global temp_current
    temp_current = round(sensor_DS18X20.temperature,1)    
    await asyncio.sleep(1)  # crash without this line, why?
       

# CONFIGURE MQTT OBJECT TOPIC AND CONNECTION PARAMETERS
config['subs_cb'] = subs_cb
config['connect_coro'] = connect_coro
config['server'] = SERVER
MQTTClient.DEBUG = True  # Optional: print diagnostic messages
client = MQTTClient(config)
print('mqtt parameters configured')


# START CONNECTION TO MQTT (ASYNC)
asyncio.run(main(client))
print('started async connection to MQTT')


# LOOP
print('started infinit try loop')
try:
    while True:
        # READ SENSOR OBJECT
        asyncio.run(read_sensor())
            #temp_current = round(sensor_DS18X20.temperature,1) ??
        # THERMOSTAT LOGIC
        if temp_current < temp_min: rstate = 'heating'
        if temp_current > temp_max: rstate = 'cooling'
        if (rstate == 'cooling'):
            if temp_current < temp_min :
                rstate = 'idle'
        if (rstate == 'heating'):
            if temp_current > temp_max:
                rstate = 'idle'
        # DRAW TO OLED
        OLED.redraw(str(temp_current), str(temp_min), str(temp_max), rstate, ":")
        
finally:
    print('got into try/finally part')
    print('start reset')
    OLED.redraw(str(temp_current), str(temp_min), str(temp_max), 'RESET', "!")
    machine.reset()
    client.close()  # Prevent LmacRxBlk:1 errors


User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: MQTT breaks infinite loop

Post by pythoncoder » Tue Nov 02, 2021 9:00 am

I think you need to review the design from scratch. There should only be one asyncio.run() statement, which launches a task which runs forever. Code after the .run() statement only runs if an error occurs or if you interrupt the application with ctrl-c.

You need (at least) two concurrent tasks. One runs the MQTT connection and the other runs the control loop. That way if the MQTT link goes down for a period, the control loop still runs.

My approach is to think through the overall design before writing code. Determine what tasks need to run and how they will communicate with each other and how errors or disruption will be handled.
Peter Hinch
Index to my micropython libraries.

Post Reply