How to run an mqtt-repl?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
User avatar
tve
Posts: 94
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

How to run an mqtt-repl?

Post by tve » Wed Jan 15, 2020 10:45 pm

I'm trying to write an MQTT Repl that accepts input by subscribing to a topic (the message payload is simply repl input) and that writes output to another topic (again, the message payload is simply repl output). I want to use Peter's mqtt_as, i.e. asyncio. I wrote an MQTTRepl class that inherits from IOBase and implements the readinto/write/ioctl functions that os.dupterm requires, and that fundamentally works.

The issue I'm running into is that right now I have main.py start the event loop (very much like the range.py example) and that's all good and dandy, except that the repl is now executing that event loop and it doesn't listen to input, so feeding it characters has no effect. How do I solve this? Do I need to create a separate thread? If so, what should it run and how do I avoid having MQTTRepl's method being called from the wrong thread by the repl?

I looked at the webrepl implementation and it calls setsockopt(socket.SOL_SOCKET, 20, accept_handler) on the listen socket, which presumably registers a callback with the bowels of the networking code. There's no similar thing available here...

BTW, looking at the synchronization primitives in https://github.com/peterhinch/micropyth ... er/asyn.py I was struck by the fact that they all end up busy waiting, i.e. calling await asyncio.sleep_ms(<some small number>) in a loop :-(. Is that going to be fixed with Paul's new uasyncio?

NB: I"m running on an esp32, in case that makes a difference.

kevinkk525
Posts: 643
Joined: Sat Feb 03, 2018 7:02 pm

Re: How to run an mqtt-repl?

Post by kevinkk525 » Thu Jan 16, 2020 6:36 am

tve wrote:
Wed Jan 15, 2020 10:45 pm
BTW, looking at the synchronization primitives in https://github.com/peterhinch/micropyth ... er/asyn.py I was struck by the fact that they all end up busy waiting, i.e. calling await asyncio.sleep_ms(<some small number>) in a loop :-(. Is that going to be fixed with Paul's new uasyncio?
I can only answer this part:

You think about it wrong, calling "await asyncio.sleep_ms(x)" is not busy waiting, it is returning control to the uasyncio task handler.
Also there's no new "Paul's uasyncio" version, there is a new "Damien's uasyncio" version, which handles waiting in the same way, because that's how asyncio works on both CPython and Micropython.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
tve
Posts: 94
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: How to run an mqtt-repl?

Post by tve » Thu Jan 16, 2020 7:55 am

Thanks for the name correction - brain fart... I still call it busy-waiting and find it horrible because: (a) it prevents the uC from sleeping when there is no activity 'cause it has to wake up all the time for the plethora of sleep_ms that are going on, and (b) it creates an awful tradeoff between making those sleep_ms pauses short to reduce latency but chew up power and cycles for nothing, vs. making them long to save power at the expense of latency. I don't understand why a coro can't block on something "hard" and be made runnable immediately when the event it's waiting on happens. But then, looking into the details of asyncio is still on my to-do list...

kevinkk525
Posts: 643
Joined: Sat Feb 03, 2018 7:02 pm

Re: How to run an mqtt-repl?

Post by kevinkk525 » Thu Jan 16, 2020 8:12 am

Yes that part is "fixed" in the new uasyncio version from Damien. Events won't need to "poll" and will only be woken up when the Event is set.
But in most applications this doesn't make much difference in power consumption as uasyncio doesn't integrate any sleeping/idle states anyway (except for one low-power implementation specific to the pyboard using RTC).
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

cgglzpy
Posts: 33
Joined: Thu Jul 18, 2019 4:20 pm

Re: How to run an mqtt-repl?

Post by cgglzpy » Thu Jan 16, 2020 7:49 pm

I'm trying to write an MQTT Repl that accepts input by subscribing to a topic (the message payload is simply repl input) and that writes output to another topic (again, the message payload is simply repl output)
This would be a very nice one to have :)

I don't know about the uasyncio part, but to make "os.dupterm" work with esp32 you need "os.dupterm_notify", something like:

Code: Select all

if hasattr(os, 'dupterm_notify'):
	client_socket.setsockopt(socket.SOL_SOCKET, 20, os.dupterm_notify)
Which I think is the callback that tells the 'os' that there is data available in the stream to read.

You may also want to have a look at the ble_uart_repl.py example that jimmo wrote.

User avatar
tve
Posts: 94
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: How to run an mqtt-repl?

Post by tve » Thu Jan 16, 2020 9:11 pm

@cgglzpy: thanks for the pointers... jimmo had already pointed me to the BLE repl and I patterned my Stream after his. There is a more fundamental issue, though. In the WebRepl and the BleRepl the network input/output is a socket and that can be put into callback-mode (i.e. `setsockopt(SOL_SOCKET, 20, callback)`). The result of this is that in the main interpreter loop the socket is checked and if a command arrives on the socket it's effectively injected into the main thread whether that is sitting at the repl prompt or is deep into executing an app.

With MQTTRepl things are different. An application needs to run to talk to the mqtt server and it presumably sends and receives many messages that do not pertain to the repl. When it receives a command for the repl it can inject it into the repl's input, but the repl never regains control 'cause the app is running and as soon as the app stops there's no MQTT connection anymore. So an MQTT repl is really something a bit different from a websocket repl or a BleRepl.

Options I'm toying with:
1. fuhget the repl, instead implement "run-statement" for mqtt: basically `eval(compile(...)).repr()` what comes in via mqtt and send the return value back, probably need to capture tty-output somehow too
2. run the application in a second thread so the main thread is idle sitting at the repl then "it all works" except that interacting with the application becomes tricky 'cause it's running in a different thread from the executed commands, so for example anything that touches asyncio becomes a potential bomb
3. try to instantiate a second repl, not sure that's a good idea, has the same issue as #2
4. similar to #1 but look into how the repl is implemented and use that instead of re-implementing the compile-eval-repr stuff...

I'm mostly interested in remote troubleshooting and app updating aspects and less in remote development (which I believe is better served by a serial-ssh bridge attached to the main uart/usb) so I'm most inclined to go with #1.

I haven't thought much about how to organize MQTT topics. Perhaps something like this:

MQTTrepl subscribes to:
- .../repl/cmd_in: incoming message body is passed to eval(compile(message)).repr()
- .../repl/eval: incoming message is json, in=<string to eval>, res=<topic to send result to>, out=<topic to send stdout to> -- this would feed the string into eval(), capture the result, convert it to json, and send that to the topic names by res; it would also redirect output to a buffer to send/stream to the topic named by out

MQTTrepl publishes to:
- .../repl/stdout: MQTTrepl hooks into stdout/stderr (haven't looked whether there's one or two streams in MP) and publishes blocks of text

I'm not clear on what I really want until I try it for a while :-)

cgglzpy
Posts: 33
Joined: Thu Jul 18, 2019 4:20 pm

Re: How to run an mqtt-repl?

Post by cgglzpy » Thu Jan 16, 2020 10:09 pm

Hmm, I see...
About #1
In that case you may want to have a look at some experimental work I did before the ssl repl : upysecrets.py

I think that the 'CRYPTOGRAPHER' class method 'crepl' can be adapted to work with MQTT with some changes (ignoring the decryption/encryption steps and instead of return the result, publish the result).

This could be implemented in the callback of a really simple mqtt-client like this mqtt_client.py
And the run 'check_msg' in a loop/uasyncio maybe? :|

User avatar
tve
Posts: 94
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: How to run an mqtt-repl?

Post by tve » Fri Jan 17, 2020 1:23 am

That looks really helpful! I have a couple of questions:
- why do you first try to eval and then exec? why not eval(compile(self.rec_msg)) ?
- why do you call gc.collect() when there is no output?

Using something like uqmtt.simple looks intersting, especially if modified to use the socket callback instead of check_msg/wait_msg... The main issue is that I will be using TLS (with the PSK key exchange I just created a PR for) and I only want one TLS connection due to memory usage. Hence I need one MQTT client for everything: repl and app. Hmmm, some tradeoffs to consider...

User avatar
tve
Posts: 94
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: How to run an mqtt-repl?

Post by tve » Fri Jan 17, 2020 7:09 am

Well, who says it can't be done... I ended up putting together a proof of concept for option #5 :lol: : implementing mqtt in a socket callback! Just like the webrepl works. I took umqtt.simple and threw in a setsockopt call so the check_msg method gets called when data comes in. Works even over SSL! At least as a proof of concept I can send an MQTT message with print('hello') and get back {"r":null,"o":"hello\r\n","e":null} (r=result, o=output, e=error), all while having the std repl prompt on the uart.

I now wonder whether this can be made reliable and whether I can add a shim with queues so pub/sub from the app can be made through asyncio...
In case anyone is curious: https://github.com/tve/mpy-mqttrepl/tree/master/src

cgglzpy
Posts: 33
Joined: Thu Jul 18, 2019 4:20 pm

Re: How to run an mqtt-repl?

Post by cgglzpy » Fri Jan 17, 2020 3:33 pm

That looks good! :)
tve wrote:
Fri Jan 17, 2020 1:23 am
That looks really helpful! I have a couple of questions:
- why do you first try to eval and then exec? why not eval(compile(self.rec_msg)) ?
- why do you call gc.collect() when there is no output?
As I said it was experimental (aka "I have the ideas but I don't know how to implement them properly") :roll: .
The eval/exec blocks its because some commands (like assigning a value to variables) throws an exception when using eval.
If so, exec runs the command and if this throws an exception too, its redirected to the error buffer...
The "gc.collect()" I don't know why I left it there, probably just forgot to delete it.

Anyway I'm glad you figured things out, I will try this when I find some time. ;)

Post Reply