How to turn off WiFi in async MQTT?

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
User avatar
Mike Teachman
Posts: 155
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

How to turn off WiFi in async MQTT?

Post by Mike Teachman » Thu Mar 28, 2019 4:58 pm

I'm hoping to get some help on a thorny problem I can't easily solve...

background:
I am using Async MQTT with the Lobo port to publish sensor data to a remote broker, every 15 minutes. This asyncio MQTT implementation solves a major requirement in my application - publishing sensor data without blocking other performance-critical coros.

the challenge:
There is one requirement that is not being met (or at least I don't see a solution in mqqt_as.py)
  • when the application is not publishing to the broker I want to turn off WiFi to reduce the ESP32 power consumption

Here is pseudo-code for the application:
1. wait for 15 minute signal
2. connect to MQTT broker
3. publish sensor data to broker
4. disconnect from broker and turn off wifi
5. back to 1.

I'm struggling with step 4 above. The challenge is that Async MQTT appears designed to maintain a broker and wifi connection at all times. Any thoughts on a way to turn off Wifi when it is not needed? I'm happy to fork-and-adapt the async mqtt repo to my own needs, but first wanted to see if there is a built-in solution that I am missing.

thanks !

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: How to turn off WiFi in async MQTT?

Post by kevinkk525 » Thu Mar 28, 2019 5:14 pm

This feature was never intended.
The library provides a resilient link (online/connected all the time) but has a disconnect feature, I just never tried it.
Even our new library micropython-iot doesn't have that feature implemented (this lirbary however is for communication between the client and a server, not mqtt).
These libraries are typcially not meant to be stopped.

But you raise a good point here, you might need a resilient link for a time and then safe some energy and pause the library/disconnect. I will think about how to implement this but I doubt that I'll be implementing it in my micropython-mqtt fork but maybe into my micropython_iot fork.

but as you are using loboris fork, why don't you use his own mqtt library implementation that is using threads?
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
Mike Teachman
Posts: 155
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

Re: How to turn off WiFi in async MQTT?

Post by Mike Teachman » Thu Mar 28, 2019 7:27 pm

kevinkk525 wrote:
Thu Mar 28, 2019 5:14 pm
This feature was never intended
thanks for the background design motivations for async mqtt. That totally makes sense - it does a really good job at keeping broker and wifi connections.
kevinkk525 wrote:
Thu Mar 28, 2019 5:14 pm
but as you are using loboris fork, why don't you use his own mqtt library implementation that is using threads?
I had considered this approach, but I'm somewhat reluctant to use the LoBo MQTT implementation because it creates a new FreeRTOS task. It's somewhat hard to predict how this MQTT task will take cycles from the uPy task (which runs a time-critical audio streaming operation in my code). In the worst case, the uPy task will see a reduction of 50% performance, assuming an equal-priority, round-robin situation. For this reason, I tend to prefer asyncio implementations rather than adding new tasks. Asyncio seems more deterministic.
kevinkk525 wrote:
Thu Mar 28, 2019 5:14 pm
my micropython-mqtt fork
thanks for mentioning your fork. Some of the changes you made will definitely improve the performance of my application, e.g. elimination of the periodic 20ms blocking sleeps. I'll try it out.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: How to turn off WiFi in async MQTT?

Post by kevinkk525 » Thu Mar 28, 2019 8:28 pm

I don't know the background of micropython-mqtt as I was not involved in the development but the feature of temporarily disconnecting was probably not on Peter Hinch's scope as the disconnect method only closes the socket but doesn't stop all running tasks.
Implementing a real stop function would be easier for the micropython_iot code but even there this feature was not discussed or thought of. However all running tasks are stopped on CTRL+C interrupt.

I understand your concerns about the performance when using loboris mqtt, I followed your audio thread.


I hope my fork will prove resilient on your device (Be aware of the small changes that prevent my fork from being a drop-in replacement), I didn't test the removal of all workarounds to the same extend as I did with micropython_iot (but we only support mainline esp32 there as there is no update on loboris fork since 7 month), therefore these workarounds are definitely not needed on mainline esp32 but my tests when removing them were on loboris fork and I couldn't see any problems but was not running 24/7.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: How to turn off WiFi in async MQTT?

Post by pythoncoder » Fri Mar 29, 2019 9:01 am

Mike Teachman wrote:
Thu Mar 28, 2019 7:27 pm
...elimination of the periodic 20ms blocking sleeps...
My resilient MQTT library requires updating to reflect recent improvements in the ESP32 firmware. When running on ESP32 there are a number of such hacks which were necessary at the time but should now be redundant. This is based on the experience of myself and Kevin developing micropython-iot: we found that all ESP32-specific code could now be removed. I also need to remove support for useless hardware such as the Sonoff. It's on my list of things to do.

As for turning off WiFi I'd need to think through the implications. keepalive packets will stop: the broker will assume client failure and react accordingly. Is this a reasonable way to operate?
Peter Hinch
Index to my micropython libraries.

User avatar
Mike Teachman
Posts: 155
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

Re: How to turn off WiFi in async MQTT?

Post by Mike Teachman » Sat Mar 30, 2019 3:50 am

pythoncoder wrote:
Fri Mar 29, 2019 9:01 am
Is this a reasonable way to operate?
Good point. Just turning of the WiFi radio is quite harsh. Turning off the WiFi radio to save some mA is the ultimate end goal. Would it be possible to somehow signal one of the mqtt_as tasks to first disconnect from the broker, then turn off WiFi. Perhaps a new "suspend()" method that allows on-the-fly disconnect from the broker and wifi. Paired with a resume() method to continue with the resilient re-connection behaviour. Ignorance is bliss. If I delved deeper into the implementation no doubt I'd see the challenges to pull this off.

Aside: A good bit of my current uPy project is made possible because of your initiatives - asyn, fast_io, mqtt_as. It's pretty cool to be able to read audio samples from a mic and save them to SD Card wav file, for hours, without missing a sample ... while at the same time reading sensors, updating a display, and publishing data to a cloud DB... many thanks !!

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: How to turn off WiFi in async MQTT?

Post by kevinkk525 » Sat Mar 30, 2019 7:34 am

I implemented the pause/resume feature on a different branch (and only for the esp32) and did test it with the esp32 mainline. Worked well in my short test. You'd have to test it for yourself with the lobo fork and in full use: https://github.com/kevinkk525/micropyth ... use_resume

It also deactivates the wifi completely to save the max amount of power.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: How to turn off WiFi in async MQTT?

Post by pythoncoder » Sat Mar 30, 2019 8:26 am

@Mike Teachman It's worth noting that uasyncio is not well suited to low power operation. On the Pyboard this can be mitigated by replacing the uasyncio timebase with one derived from the RTC, and I have written a solution documented here. This needs testing with the Pyboard D, but it's no help for ESPx boards. I also think that most MQTT applications will want to subscribe (with a timely response) as well as publish.

For these reasons my enthusiasm for implementing a low power mode is limited, but I'll be interested to see the outcome of measurements on the Pyboard D. Damien has quoted very low figures in server applications. Perhaps running the above uasyncio version will deliver results.

I have started work on the asynchronous MQTT library to improve ESP32 support and to provide Pyboard D support.

I'm glad you're finding my efforts useful. Continuously recording audio while performing other tasks is impressive: what sample rate are you using?
Peter Hinch
Index to my micropython libraries.

User avatar
Mike Teachman
Posts: 155
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

Re: How to turn off WiFi in async MQTT?

Post by Mike Teachman » Mon Apr 01, 2019 7:57 pm

kevinkk525 wrote:
Sat Mar 30, 2019 7:34 am
I implemented the pause/resume feature on a different branch
Great ! I'll get a chance to try this later today. Thanks so much for your time to extend this library.

User avatar
Mike Teachman
Posts: 155
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

Re: How to turn off WiFi in async MQTT?

Post by Mike Teachman » Mon Apr 01, 2019 8:43 pm

pythoncoder wrote:
Sat Mar 30, 2019 8:26 am
It's worth noting that uasyncio is not well suited to low power operation. On the Pyboard this can be mitigated by replacing the uasyncio timebase with one derived from the RTC, and I have written a solution documented here. This needs testing with the Pyboard D, but it's no help for ESPx boards. I also think that most MQTT applications will want to subscribe (with a timely response) as well as publish.
Interesting. I had assumed that uasyncio designs would never be possible when a device is placed into an ultra low power sleep mode (like ESP deepsleep). I'll check out your link.
pythoncoder wrote:
Sat Mar 30, 2019 8:26 am
I have started work on the asynchronous MQTT library to improve ESP32 support and to provide Pyboard D support.
Sweet. I'll stay tuned in for that work :D
pythoncoder wrote:
Sat Mar 30, 2019 8:26 am
what sample rate are you using?
Here is what is possible...

1. 20kHz sampling for reading an I2S microphone, with continuous, gapless, writing of 16bit audio samples to SD card:
- limiting factors are variations in SDCard sector write times. Sometimes an SD Card may write a sector in say 3ms, other times I've seen back-to-back sector writes in excess of 50ms. Maybe I need to find a better SD Card?
- garbage collection is also a bit of a wildcard. Here, an adequate DMA buffering of samples is your friend. In a worst case, a secondary FIFO RAM buffer can be implemented to further extend the DMA buffering - the psRAM version of the ESP32 is needed for this.
- assumes resolution reducing (X bits -> 16bits) routine is a uPy module written in C.
- using 128kB of DMA memory. This provides a buffer sample window of 371ms which is enough to "absorb" blocking interruptions such as garbage collection and slow SD sector writes.
- uses uasyncio with fast_io implementation.
- I2S loop only yields to the scheduler when read requests to the DMA buffer come back empty. This point is key.

2. 44.1kHz sampling for reading of an I2S mic and doing C or ASM based processing of the samples.
- for example, it takes 5.8ms to accumulate 256 audio samples from a typical I2S microphone at 44.1kHz sample rate
- the accumulation of samples happens entirely in the background using DMA. Processing of samples happens in the foreground. The ESP32 does this well.
- in practice, the application loop has about 4ms of processing time out of the 5.8ms (on average). uPy will be too slow for most, if not all stream processing demands. But, it should easily possible to process the samples using a C-based uPy module implementation. e.g C-based FFT module.

Above is based on the ESP I2S module I submitted as a PR. C-based, using the ESP-IDF.
https://github.com/miketeachman/micropy ... hine_i2s.c
https://github.com/miketeachman/micropy ... c292e622f2
https://github.com/miketeachman/micropy ... s-examples

note: When my pyboard d arrives I'll work an I2S PR for it.

alright... likely more than you wanted for this post... ;)

Post Reply