ESP32 port + uasyncio + mqtt_as issue

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

ESP32 port + uasyncio + mqtt_as issue

Post by ChrisO » Mon Apr 06, 2020 7:10 pm

Hi,

As far as I know, the esp32 port of micropython does not contain uasyncio
(https://github.com/micropython/micropython is my source for micropython)

I have added and developed an application using https://github.com/micropython/micropython-lib.
So far, I was under the impression that these sources are both part of "official build"

I'm starting to guess my assumptions are wrong because I am running into the following issue:
When trying to import and use Peter Hinch's mqtt_as.py an attempt to use uasyncio.Lock() is made... My version of uasyncio does not have Lock. I do have a lock as part of uasyncio/synchro.py but then of course the default import is different.

Is one of the correct ways of solving this to simply to do

Code: Select all

from uasyncio.synchro import *
inside my __init__.py considering I have a folder structure as such: ports/esp32/modules/uasyncio/[__init__.py,core.py,queues.py,synchro.py]

I mean... it works.... but I don't want to entangle myself into silly situations with weird, unique errors because nobody does it this way 8-)

(edit: typo's and grammar)

kind regards,
Chris
Last edited by ChrisO on Tue Apr 07, 2020 12:08 pm, edited 1 time in total.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: ESP32 port + uasyncio + mqtt_as issue

Post by kevinkk525 » Mon Apr 06, 2020 9:28 pm

The mqtt repo got updated to use the new uasyncio version that got merged into micropython recently. The daily builds of esp32 firmware has that new version. You should use that. The uasyncio modules from micropython-lib are not supported anymore.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: ESP32 port + uasyncio + mqtt_as issue

Post by ChrisO » Tue Apr 07, 2020 11:48 am

Thank you for for your reply Kevin.

I was reading through the issues of the official build and it seems like Queues have not been implemented there yet.
(https://github.com/micropython/micropython/issues/5828)
My implementation relies on this, so I don't think using the daily build is going to solve anything for now.

At least I know it is (being) integrated thanks to you.
I'll keep an eye on the daily builds!

regards,
Chris

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 port + uasyncio + mqtt_as issue

Post by pythoncoder » Tue Apr 07, 2020 11:53 am

You might like to try this. I have tested it under new uasyncio. It isn't particularly efficient, but it does work. I should point out that it is based on the work of Paul Sokolovsky namely his implementation in uasyncio V2.

Code: Select all

from ucollections import deque
import uasyncio as asyncio

# Exception raised by get_nowait().
class QueueEmpty(Exception):
    pass

# Exception raised by put_nowait().
class QueueFull(Exception):
    pass

class Queue:

    def __init__(self, maxsize=0):
        self.maxsize = maxsize
        self._queue = deque((), maxsize)

    def _get(self):
        return self._queue.popleft()

    async def get(self):  #  Usage: item = await queue.get()
        while not self._queue:
            # Queue is empty, put the calling Task on the waiting queue
            await asyncio.sleep_ms(0)
        return self._get()

    def get_nowait(self):  # Remove and return an item from the queue.
        # Return an item if one is immediately available, else raise QueueEmpty.
        if not self._queue:
            raise QueueEmpty()
        return self._get()

    def _put(self, val):
        self._queue.append(val)

    async def put(self, val):  # Usage: await queue.put(item)
        while self.qsize() >= self.maxsize and self.maxsize:
            # Queue full
            await asyncio.sleep_ms(0)
            # Task(s) waiting to get from queue, schedule first Task
        self._put(val)

    def put_nowait(self, val):  # Put an item into the queue without blocking.
        if self.qsize() >= self.maxsize and self.maxsize:
            raise QueueFull()
        self._put(val)

    def qsize(self):  # Number of items in the queue.
        return len(self._queue)

    def empty(self):  # Return True if the queue is empty, False otherwise.
        return not self._queue

    def full(self):  # Return True if there are maxsize items in the queue.
        # Note: if the Queue was initialized with maxsize=0 (the default),
        # then full() is never True.

        if self.maxsize <= 0:
            return False
        else:
            return self.qsize() >= self.maxsize
[EDIT]
Removed error in the above pointed out by Kevin. Oddly it did work, but there was a definite mistake...
Peter Hinch
Index to my micropython libraries.

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: ESP32 port + uasyncio + mqtt_as issue

Post by ChrisO » Tue Apr 07, 2020 1:38 pm

Thank you Peter,

Using the official build + one or two extra's seems manageable.
(I forget a lot, so keeping things simple helps me)

I'm noticing that there's actually differences in micropython-lib classes and the official build.
This means I have to get back to the drawing board. (i.e. Event.set() no longer accepts an optional data argument)

Can topics be closed? I'll ask my follow-up question in a new thread :)
Chris

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 port + uasyncio + mqtt_as issue

Post by pythoncoder » Tue Apr 07, 2020 5:02 pm

Event.set() no longer accepts an optional data argument
The official Event class is CPython compatible. The version in my asyn.py library had three differences from the current official one:
  • It is an awaitable class.
  • The .set() method can be called from an ISR.
  • It has an optional data arg.
I have renamed this to Message while retaining this functionality. Note that the official Event is more efficient: mine relies on polling to achieve its ISR friendliness. If Event becomes ISR friendly I will make Message a subclass of Event to inherit its efficiency.

I've made these synchronisation primitives available here. All work with V3 but many are stopgap solutions while we wait for official versions. There's no documentation yet, but their API is as per my existing docs. They are now implemented as a Python package so the import statements will change - see the test programs.

I'm working on a substantial revision of the micropython-async repo to provide for V3, but it may be a couple of weeks before it's ready.
Peter Hinch
Index to my micropython libraries.

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: ESP32 port + uasyncio + mqtt_as issue

Post by ChrisO » Wed Apr 08, 2020 12:52 pm

pythoncoder wrote:
Tue Apr 07, 2020 5:02 pm
Event.set() no longer accepts an optional data argument
...Note that the official Event is more efficient: mine relies on polling to achieve its ISR friendliness. If Event becomes ISR friendly I will make Message a subclass of Event to inherit its efficiency....
If I understand the code correctly, the Message class you have implemented only sets flags on itself.
The async wait method will loop until the flag becomes True. is this the polling you speak of?

Code: Select all

# micropython-samples/uasyncio_iostream/v3/primitives/message.py
class Message():
    def __init__(self, delay_ms=0):
        self.delay_ms = delay_ms
        self.clear()

    def clear(self):
        self._flag = False
        self._data = None

    async def wait(self):  # CPython comptaibility
        while not self._flag:
            await asyncio.sleep_ms(self.delay_ms)

    def __await__(self):
        while not self._flag:
            await asyncio.sleep_ms(self.delay_ms)

    __iter__ = __await__

    def is_set(self):
        return self._flag

    def set(self, data=None):
        self._flag = True
        self._data = data

    def value(self):
        return self._data
        
At first glance the official Event class does not look much different.
The only difference I can see, which is likely to be of great significance, is that the wait method is no loop at all.
It simple pushes the current task on a queue and returns true.
Later on, when an event is set, this pops the queue, returning a value or starting a new coroutine.

for completion, here's the Event class I was looking at:

Code: Select all

# MicroPython uasyncio module
# MIT license; Copyright (c) 2019-2020 Damien P. George

from . import core

# Event class for primitive events that can be waited on, set, and cleared
class Event:
    def __init__(self):
        self.state = False  # False=unset; True=set
        self.waiting = core.TaskQueue()  # Queue of Tasks waiting on completion of this event

    def is_set(self):
        return self.state

    def set(self):
        # Event becomes set, schedule any tasks waiting on it
        while self.waiting.peek():
            core._task_queue.push_head(self.waiting.pop_head())
        self.state = True

    def clear(self):
        self.state = False

    async def wait(self):
        if not self.state:
            # Event not set, put the calling task on the event's waiting queue
            self.waiting.push_head(core.cur_task)
            # Set calling task's data to the event's queue so it can be removed if needed
            core.cur_task.data = self.waiting
            yield
        return True
        

Now, I know absolutely nothing about ISR's besides from what I read in 5 minutes looking at the docs... But you've peeked my interest.
What makes Event ISR unfriendly? As far as I can tell, the main loop gets interrupted only briefly and it will always wait for object creations to complete so no coro is left with any half updated object. What kind of use cases are ISR unfriendly?

regards,
Chris
Chris

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 port + uasyncio + mqtt_as issue

Post by pythoncoder » Wed Apr 08, 2020 5:16 pm

Kevin produced proof that, if the interrupt occurs while the asyncio pairing heap is being updated, a failure occurs. Note that using micropython.schedule() will not fix that fundamental concurrency issue.
Peter Hinch
Index to my micropython libraries.

uxhamby
Posts: 34
Joined: Thu Nov 14, 2019 9:47 pm

Re: ESP32 port + uasyncio + mqtt_as issue

Post by uxhamby » Sun Jun 28, 2020 8:37 pm

Was there ever a resolution to this issue?

Thanks,

Brian H.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: ESP32 port + uasyncio + mqtt_as issue

Post by kevinkk525 » Sun Jun 28, 2020 9:12 pm

To Event not being ISR safe with uasyncio v3? There's currently a PR from Damien (https://github.com/micropython/micropython/pull/6106) but nothing merged yet, might change too.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Post Reply