Help needed: My loop is ending prematurely

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
jdts
Posts: 21
Joined: Mon Dec 23, 2019 2:55 pm

Help needed: My loop is ending prematurely

Post by jdts » Fri Oct 09, 2020 2:04 am

I have a project running MicroPython v1.13 on an ESP32 controlling a 24V motor controller, with a loop that looks like:

Code: Select all

        while True:
            mqtt.check_msg()
            self.checkSwitches()
            self.checkReboot()
            self.stepCount+=1
            utime.sleep_ms(self.loopDelay)
        print("Exiting Main Loop")
which randomly stops looping after a day or so. I have a standard loop delay of 80ms. It tends to run for 1e5 or more iterations (I print an update ever 200 iterations), but then silently exits the loop. The stoppage doesn't correlate with running the motors or any MQTT messages. The two function calls just examine some GPIOs, look for a file named "REBOOT". It tends to stop like this:

Code: Select all

* 2020-10-06 21:45:34 Stepping to:  165400
* 2020-10-06 21:45:51 Stepping to:  165600
* 2020-10-06 21:46:07 Stepping to:  165800
* 2020-10-06 21:46:24Connection closed by foreign host.
It's not ever the same number of steps unit failure. The final "Exiting Main Loop" does not print, and no error message of any kind is thrown. But the serial connection is still there (via utelnetserver), and I can upload files (via uftpd). I am logging the boot time via main.py, and it is not rebooting. So same MP session, just no longer running an infinite loop.

I really am stuck without a good idea of how to debug this. Any thoughts?? This device is in the wall and so pulling it out to debug is not ideal.

kevinkk525
Posts: 781
Joined: Sat Feb 03, 2018 7:02 pm

Re: Help needed: My loop is ending prematurely

Post by kevinkk525 » Fri Oct 09, 2020 7:56 am

My guess is on the mqtt.check_message() being stuck but I haven't worked with the synchronous umqtt.robust version in years to help with it.. I always recommend switching to uasyncio if one is willing to learn it.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jdts
Posts: 21
Joined: Mon Dec 23, 2019 2:55 pm

Re: Help needed: My loop is ending prematurely

Post by jdts » Sat Oct 10, 2020 1:12 pm

Thanks. I'll trying running it a day or two without checking for MQTT messages and report back. It's not getting stuck though, but instead just ending the infinite loop somehow. I'm not clear how failures in check_message() would break out of a running loop without throwing an error though. Any thoughts on additional debug strategies? Should I be flushing the serial print buffer somehow? Are telnetserver or uftpd suspect? It's a challenge to know what to try without any sort of errors!

User avatar
pythoncoder
Posts: 4646
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Help needed: My loop is ending prematurely

Post by pythoncoder » Sat Oct 10, 2020 2:21 pm

ESP32's do occasionally spontaneously reboot: this seems to result from the FreeRTOS. You might also like to read this. While originally written for the ESP8266, most of the observations also apply to ESP32.
Peter Hinch

jdts
Posts: 21
Joined: Mon Dec 23, 2019 2:55 pm

Re: Help needed: My loop is ending prematurely

Post by jdts » Mon Oct 26, 2020 8:05 pm

kevinkk's instincts appear to have been correct. I removed the MQTT message checking mqtt.check_message() from the loop, and it ran quite a bit longer (about a week). Until, that is, I controlled the device via local switches, and caught (via telnet) an EHOSTUNREACHABLE error which caused MQTT publish to abort the loop. So presumably this was what was happening with check_message() as well (though never logged via telnet).

I upgraded to uPy v1.13, porting over to uasyncio and Peter's fantastic mqtt_as super-resilient MQTT library, using qos=1. I also eliminated the IRQ's for the 4 input Pins (which were just setting values being checked every 50ms anyway). The result has been super-stable and far more responsive.

Once you get used to it, uasyncio is really straightforward. In particular, setTimeout type logic with callbacks can be replaced with much more obvious-looking calls like:

do_stuff()
await asyncio.sleep(self.waitTime)
do_more_stuff()

or similar.

If you are using MQTT, I definitely recommend giving uasyncio and Peter's library a try. The only catches I found:
  • To avoid performance bottlenecks, the MQTT receiving callback is run synchronously, so you need to setup an asyncio task (create_task) quickly and return. I ended up saving such tasks for later cancellation, which worked out well. There are also queues and Events which can be used for more complex control.
  • mqtt.simple takes ints and other non-strings as publish values, but mqtt_as doesn't. Just convert these first.
Other than that, and not really a catch, once you start thinking async, you will wish everything had an async interface: file system (file existence), GPIO Pins, etc. Honestly I think it's absolutely the right paradigm for embedded programming with various network/hardware inputs/outputs.

Thanks all for the helpful suggestions.

Post Reply