Async generators and async for loops

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Async generators and async for loops

Post by cefn » Sat Jan 06, 2018 4:15 pm

Hi all,

I was hitting an issue related to the use of 'async for', which I simplified into a minimal test case which I couldn't resolve, and then checked against python3.6, where I found there was no error in the approach, although I can't see which of the differences with Micropython uasyncio I have failed to keep track of.

I understood from the comment about "async for" in https://github.com/micropython/micropyt ... ifferences that it should be equivalent to the adoption of PEP492, lined up with Python3.6 but I guess the coroutine itself doesn't end up with the 'duck typing' of an asynchronous iterator itself, somehow, (since coroutines are generators in Micropython?).

Is there anything I should be doing differently to make asynchronous generators work with async for?

The test case is viewable at https://gist.github.com/cefn/e32496544a ... 9f4b59b6e3 and I get the results below. This is all based on Unix Micropython straight from Github head at the time of writing, and with uasyncio installed via micropython -m upip install micropython-uasyncio after trashing ~/.micropython

Code: Select all

gist$ python3.6
Python 3.6.3 (default, Oct  3 2017, 21:45:48) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from async_for_tester import results; print(results)
[0, 1, 2, 3, 4, 5, 6, 7]
>>> 
gist$ micropython
MicroPython v1.9.3-238-g42c4dd09 on 2018-01-03; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> from async_for_tester import results; print(results)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "async_for_tester.py", line 27, in <module>
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 149, in run_until_complete
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 138, in run_forever
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 97, in run_forever
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 146, in _run_and_stop
  File "async_for_tester.py", line 15, in aggregate
AttributeError: 'generator' object has no attribute '__aiter__'
I attempted to resolve this by creating a wrapper object which was compliant with PEP492...

Code: Select all

class CoroWrapper:
    def __init__(self, coro):
        self._it = iter(coro)

    def __aiter__(self):
        return self

    async def __anext__(self):
        try:
            next(self._it) # skip spurious entries
            value = next(self._it)
        except StopIteration:
            raise StopAsyncIteration
        return value
But this results in an error which suggests it needs to be an ordinary iterable within async for.

Code: Select all

gist$ micropython
MicroPython v1.9.3-238-g42c4dd09 on 2018-01-03; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> from async_for_tester import results; print(results)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "async_for_tester.py", line 43, in <module>
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 149, in run_until_complete
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 138, in run_forever
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 97, in run_forever
  File "/home/cefn/.micropython/lib/uasyncio/core.py", line 146, in _run_and_stop
  File "async_for_tester.py", line 31, in aggregate
TypeError: 'CoroWrapper' object is not iterable
Does anyone know is there something obvious that I have missed?

User avatar
pythoncoder
Posts: 4949
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Async generators and async for loops

Post by pythoncoder » Sun Jan 07, 2018 12:07 pm

I think the problem may be that __anext__, though declared with async def, does not have an await statement. It is a requirement that __anext__ is a coroutine. I suggest you look at the example in my tutorial.
Peter Hinch

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Async generators and async for loops

Post by cefn » Mon Jan 08, 2018 4:59 pm

Thanks for spotting a possible issue there. However, my attempt to build an async iterator object and explicitly define __anext__() was a workaround for the original error, which was that count() didn't apparently create an asynchronous iterator in the first place, which could be consumed by the async for in aggregate(). I believe the test case showed that Python3.6 does this, but Micropython does not.

Here's the link to the test case which revealed the original problem...
https://gist.github.com/cefn/e32496544a ... 9f4b59b6e3

Seemed fairly fundamental to async for, as per core examples like this...
https://www.blog.pythonlibrary.org/2017 ... enerators/

People can probably ignore my attempt to create a CoroWrapper (and explicitly implement __anext__()). I had just copy-pasted directly from the AsyncIterator reference example in PEP492, (only adding a line to skip the alternating entries "100" which were unexpectedly being returned by iter(coro)), as I was wondering if a simple workaround could resolve the issue.

So regards the original issue, should the count() function indeed create an asynchronous iterator which can be used by async for? This seems to be the behaviour of Cpython following PEP492 as per the same code running in Python3.6.

Alternatively are you saying an Object with the asynchronous iterator signature is the only valid Micropython way to achieve something to satisfy async for and that's deliberate?

I was considering raising an issue about how count() was being interpreted (incorrectly?) but don't want to fill the github tracker with stuff which is invalid stupid on my part.

User avatar
pythoncoder
Posts: 4949
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Async generators and async for loops

Post by pythoncoder » Tue Jan 09, 2018 6:40 am

Your second ref is for Python 3.6. In general MicroPython supports Python 3.4 with very limited Python 3.5 extensions, notably async def. So any comparisons with CPython behaviour should be done using CPython V3.4, declaring coroutines using V3.4 syntax. I believe this syntax works on MicroPython although I haven't tested it.

Further uasyncio is a lightweight subset of asyncio. I have taken considerable trouble to document the subset in my tutorial, yet you persistently complain when you try to use unsupported features or attempt bizarre approaches like throwing exceptions to coroutines. CPython would disallow this. I'm increasingly baffled as to what you're actually trying to achieve.

If you're trying to write practical code I suggest you follow the practice in my tutorial, the examples in which have been tested against the MicroPython asyncio subset.
Peter Hinch

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: Async generators and async for loops

Post by cefn » Tue Jan 09, 2018 11:00 am

Thanks for the clarification I should be assuming 3.4 plus the handful of additional features listed at... https://github.com/micropython/micropyt ... ifferences
I thought I was indeed doing that, but may well have lost track of the matching versions for a particular language feature. According to https://www.python.org/dev/peps/pep-0492/ we are looking at a 3.5 feature, and I believe one indicated to be supported by Micropython by https://github.com/micropython/micropyt ... ifferences which includes the statement "async for should be equivalent to PEP492 description."
I'm increasingly baffled as to what you're actually trying to achieve.
In each case I am starting out trying to achieve practical results and hitting apparent discrepancies vs. CPython. I then attempt to strip back to minimal examples which don't include a lot of distracting and application-specific cruft, then checking they run in CPython, then checking if there is meant to be support or not. The aim is to make them minimal and easy to pick apart. If support is indeed intended, my aim is to arrive at test cases which can improve support for core python language features mirroring CPython asyncio (with the usual Micropython resource-limitation caveats). Before raising something wrongly as an issue on github, I am documenting it here on the forum first.

My expectation has been that adopting emerging features then helping to identify, and either document or fix, discrepancies in the execution of emerging language/library features in Micropython compared to the same in CPython would be an asset for future adopters. This is especially the case for uasyncio which has taken a different path from asyncio for good reasons. Unless all implementations of Micropython interpreter logic are error-free first time :) I am genuinely sorry if at any point my failure to understand the extent of support means I am wasting your time. One reason for investing time in creating minimal cases is to make it as easy as possible to dismiss mistakenly-raised issues.

I am absolutely willing to accept at any point that async for won't ever extend support to asynchronous iterators as defined through async def and yield, and instead requires an object with the proper methods in each case. However, if this is the case, given the Cpython support, it would be worth documenting the difference. Currently it seems from the documentation that this would be supported, but I may certainly have mismatched the versions in which this support was introduced if PEP492 isn't the right vehicle.

I have deliberately been focusing on the use of core language features, coroutines with returns, generators with yields, raises rather than the inclusion of concepts from asyn which are worthy, but relatively confusing for adopters (including myself) as all code paths enter extensive well-designed and -maintained but hard-to-follow eventing code, compared to being core python language features with a minimal mental model. My target audience includes learners who I would like to benefit from the use of async, await, return and throw as simple developments of the python syntax introduced with def when I am teaching function definitions, but going further to introduce cooperative scheduling. Ideally their learning would transfer back to CPython with minimal changes, normally possible given Micropython normally offers a subset of Python.

Hope this goes some way to clarifying my posts on the forum and genuinely ready to get feedback on how I should do things differently, or if I should damn well stop trying to help in such an unhelpful way :)

User avatar
pythoncoder
Posts: 4949
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Async generators and async for loops

Post by pythoncoder » Tue Jan 09, 2018 12:36 pm

I spent a considerable length of time discovering and documenting the uasyncio subset and providing some hopefully useful extensions. If you wish to duplicate this effort that is your choice, and you may yet find things I've missed or got wrong - in which case please raise an issue. As a firmware writer I think the current subset is eminently useful and has exceptional performance. If there are absent features which prevent you from writing practical code an RFC with a simple use case is the way to proceed.

I wouldn't say that uasyncio will never support any feature: uasyncio is not my responsibility, nor is the future development of MicroPython. But in terms of any use-case I can envisage, based on many years' experience of writing asynchronous firmware, now that it supports timeouts and cancellation it is unlikely I'll be pushing for further enhancements.
Peter Hinch

Post Reply