MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
User avatar
tve
Posts: 214
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by tve » Tue May 26, 2020 7:30 am

My projects tend to be on the infrastructure side of things resulting in microcontrollers stuck somewhere doing some thing for a long time, sometimes a decade. At the same time, a lot of these projects are experiments that I keep tweaking even once they're "deployed" and "in production". So I need a robust framework that allows me to see what's happening on the board remotely, sometimes perform interactive troubleshooting, and update software, all remote. And have high confidence that I'm not going to loose connectivity or management access to the board even if I make a mistake. Out of this context and need comes MQBoard.

MQBoard contains the following:
- an MQTT client library that uses the new uasyncio, that deals with disconnection and retransmission, and that supports the data rates needed for an OTA upgrade (1.5MB to transfer...)
- a set of python files that boot a board, provide a safe-mode when things go wrong, log over MQTT, provide Repl access, and launch a modular application
- a MQRepl task that provides access to the flash filesystem, to the Repl, and to OTA upgrades all over MQTT
- a mqboard commandline tool that has commands similar to pyboard.py, plus OTA update, log viewing, and file synchronization, all over MQTT
- a simple blinky demo app

Right now MQBoard only works on the esp32. I'd like to make it work on PYBD, but have not had the time yet. It will not work on the esp8266, there's just not enough memory there for this to make sense.

I won't copy the README here: there is quite some documentation in the repository, if you're interested in the issues addressed by MQBoard please check out https://github.com/tve/mqboard

kevinkk525
Posts: 765
Joined: Sat Feb 03, 2018 7:02 pm

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by kevinkk525 » Tue May 26, 2020 10:00 am

A very nice project!

I especially like the OTA over mqtt option, something my project is definitely missing on the esp32.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

uCTRL
Posts: 42
Joined: Fri Oct 12, 2018 11:50 pm

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by uCTRL » Tue May 26, 2020 10:23 am

Impressive development with emphases on reliable remote access. Well done and keep up a good work.

I have been avoiding MQTT due to network reliability risk of having to relay on single WiFi AP and single non redundant broker.

I guess it should be possible to implement MQTT network with a backup WiFi AP and broker but am not sure of the complexity, also beyond my ability?

Link to multiple brokers in bridge mode.

http://www.steves-internet-guide.com/mo ... iguration/

kinno
Posts: 32
Joined: Mon Oct 21, 2019 2:06 pm

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by kinno » Tue May 26, 2020 10:55 am

You did it! Congratulations TVE! I will be putting this through the paces this week for sure. This is something I really wanted for a long time. I've been scrapping together other things in order to make it work but now, can't wait to try this out. Again, thank you! :D

How would you like contributions bugs etc to be handled? Here or on the github? I'd love to participate in this project in anyway I can to help out.

Very excited about this.

User avatar
tve
Posts: 214
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by tve » Tue May 26, 2020 3:48 pm

uCTRL wrote:
Tue May 26, 2020 10:23 am
I have been avoiding MQTT due to network reliability risk of having to relay on single WiFi AP and single non redundant broker.
You shouldn't be conflating MQTT and WiFi. Yes, a single Wifi access point is a single point of failure, but with the esp32, unless you add bluetooth, that's regardless of what type of comm protocol you use.
uCTRL wrote:
Tue May 26, 2020 10:23 am
I guess it should be possible to implement MQTT network with a backup WiFi AP and broker but am not sure of the complexity, also beyond my ability?

Link to multiple brokers in bridge mode.

http://www.steves-internet-guide.com/mo ... iguration/
I have not tried the bridge stuff and don't know what the gotachas are, so I can't really comment.

I have been running an ODroid-XU4 w/eMMC (and previously a -U3) with mosquitto, postgres, grafana, and node-red for several years without a hitch. I've been really impressed by the reliability of it all. I've kept the U3 as a spare but it's time I order something more performant 'cause it wouldn't hold up to the load anymore.

WRT MQBoard, I want to separate out the Wifi management from the MQTT client. This should make it relatively easy to drop in a manager that does some scanning and can choose either between two SSIDs or between two APs offering the same SSID. The MQTT client could be enhanced as well to reconnect to one of two brokers, although some reading of the fine print in the standard would be required to make sure packet retransmission is done appropriately. I think this is where you loose the most with a broker bridge: QoS=1 no longer guarantees at-least-once.

Hope this helps.

User avatar
tve
Posts: 214
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by tve » Tue May 26, 2020 3:55 pm

kinno wrote:
Tue May 26, 2020 10:55 am
You did it! Congratulations TVE! I will be putting this through the paces this week for sure. This is something I really wanted for a long time. I've been scrapping together other things in order to make it work but now, can't wait to try this out. Again, thank you! :D
Thanks, let me know about the inevitable road bumps you encounter!
kinno wrote:
Tue May 26, 2020 10:55 am
How would you like contributions bugs etc to be handled? Here or on the github? I'd love to participate in this project in anyway I can to help out.
Very excited about this.
Github. I need to do a fresh round of adding unit tests so I have better coverage. Then I can try to enforce tests for everything new... Painful in the moment but essential in the long run.

Thanks for the excitement! I've been running my weather station with MQBoard for the past two weeks and have updated the code almost daily and so far I haven't had to go there physically to press the reset button, that's what has me excited :-).

User avatar
tve
Posts: 214
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by tve » Tue May 26, 2020 4:16 pm

kevinkk525 wrote:
Tue May 26, 2020 10:00 am
A very nice project!

I especially like the OTA over mqtt option, something my project is definitely missing on the esp32.
Thanks! The OTA puts a lot of stress on everything, in particular memory. It ends up being close to 1000 1400-byte messages... There's a definite reliability/speed tradeoff with the current shitty memory fragmenter, er, allocator. Fortunately there's safe-mode to reboot into in order to free up memory, haha.

kevinkk525
Posts: 765
Joined: Sat Feb 03, 2018 7:02 pm

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by kevinkk525 » Tue May 26, 2020 8:59 pm

Maybe one day I'll try to build my pysmartnode project (which is a very similar framework but without OTA and mostly built around esp8266 and more smarthome focused) on top of your framework ;)
Will have to take a closer look at the code and how to hook into it.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Tialm
Posts: 2
Joined: Fri May 15, 2020 3:12 pm

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by Tialm » Fri Jun 19, 2020 12:41 am

Hi tve! First of all let me thank you for sharing your project! It's exactly what I am looking for and I'll give it a try.

I am trying to compile your version of Micropython with the necessary PR https://github.com/tve/micropython/tree/tve together with lvgl https://github.com/lvgl/lv_binding_micr ... /README.md which I am using in my project. But it's proving frustrating..

I already tried different ways and I always end with the same errors.

The last try I did the following.

1) Cloned https://github.com/lvgl/lv_micropython
2) Merged with https://github.com/tve/micropython/tree/tve

Then tried to compile.

Got a lot of errors in the portnativeglue.c compiling fase. Any idea of what is happening? How can I address this issue?

Thank you once again for the great work! And thank you in advance for the help and time!

PS: More info and the errors txt file available in https://forum.lvgl.io/t/problem-compili ... ython/2525

kevinkk525
Posts: 765
Joined: Sat Feb 03, 2018 7:02 pm

Re: MQBoard - Micro-Framework for MicroPython Boards Managed via MQTT

Post by kevinkk525 » Tue Sep 15, 2020 7:40 am

I had a closer look again at your project because I would like to get some of the functionality into my project (like mqtt-repl and mqtt-client) but I realized that you use your own micropython fork with multiple changes and I don't want to use another fork that is even specific for esp32 because I want to implement that functionality for esp8266, esp32, pyboard and unix port because my project supports all those ports.
Now I'm not sure it's worth the effort to look into the code if your whole project relies on changes to the micropython firmware and port specific features like rtc memory (which unix doesn't have).

So is it possible to port at least the mqtt-repl and the mqtt-client without those changes? (without considerable effort, I don't want to rewrite half the code)
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Post Reply