Learning uasyncio and futures

[apois]

Ah - many thanks pythoncoder - that would explain it. I wasn't aware of that subtlety of python closures, I'm more used to Javascript.

I don't know why it worked at all without the nonlocal keyword - I was running on a linux 1.9.4 build.

[HermannSW]

Code: Select all

Note that

Code: Select all

   results = [None for i in range(count)]

could be written

Code: Select all

   results = [None] * count

Cool, can be done on both sides:

Code: Select all

>>> [None]*2
[None, None]
>>> 3*[None]
[None, None, None]
>>> 

While 'list'*'list' is not possible directly, creating list of lists is surprisingly easy:

Code: Select all

>>> ([None]*2)*([None]*3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported types for __mul__: 'list', 'list'
>>> [[None]*2]*3
[[None, None], [None, None], [None, None]]
>>> 

Although this surprises me a bit:

Code: Select all

>>> a=[[None]*2]*3
>>> a
[[None, None], [None, None], [None, None]]
>>> a[1][0]=7
>>> a
[[7, None], [7, None], [7, None]]
>>> 

This works as I would expect it:

Code: Select all

>>> a=[[1,2],[3,4],[5,6]]
>>> a
[[1, 2], [3, 4], [5, 6]]
>>> a[1]
[3, 4]
>>> a[1][0]=7
>>> a
[[1, 2], [7, 4], [5, 6]]
>>> 

[pythoncoder]

This is because Python copies lists by reference. So your list of lists is actually a list of list, IYSWIM. To reduce your example to its simplest:

Code: Select all

>>> z = [1,]
>>> q=[z]*5
>>> q
[[1], [1], [1], [1], [1]]
>>> q[0][0]=999
>>> q
[[999], [999], [999], [999], [999]]
>>> 

The "pass by reference" method is efficient but this is one of a number of well known cases where the outcome causes puzzlement to those unfamiliar with the quirks of the language.

[HermannSW]

Thanks for the explanation.

This allows for an evil thimblerig game

Code: Select all

>>> thimbles=[['pea',]]*3
>>> thimbles
[['pea'], ['pea'], ['pea']]
>>> thimbles[0][0]=None
>>> thimbles[2][0]=None
>>> thimbles[0][0],thimbles[1][0]=thimbles[1][0],thimbles[0][0]
>>> thimbles[0][0],thimbles[2][0]=thimbles[2][0],thimbles[0][0]
>>> 
>>> thimbles[2][0]
>>> 

[HermannSW]

Unfortunately importing takes so much RAM that it runs oom on ESP-01s with 28KB free RAM.

Back to topic, and sorry, was my fault.
Compiling uasyncio modules was not possible on ESP8266.
I just replaced the two "uasyncio/*.py" files with *.mpy files.
Now poorfutures.py just runs, even without calling gc.collect() free memory after powering ESP_01s module of 26096 bytes only reduces to 9456, and poorfutures.py (and uasyncio) works perfectly on ESP8266:

Code: Select all

$ webrepl_client.py -p abcd 192.168.4.1
Password: 
WebREPL connected
>>> 
>>> 
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> uos.listdir('uasyncio')
['core.mpy', '__init__.mpy']
>>> gc.mem_free()
26096
>>> import poorfutures
0 of 3
0 of 5
0 of 1
[False, False, False]
[False, False, False]
[False, False, False]
[False, False, False]
[False, False, False]
[False, False, False]
[False, False, False]
[False, False, False]
[False, False, False]
1 of 3
1 of 5
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
2 of 3
2 of 5
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
[False, False, True]
3 of 5
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
4 of 5
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, False, True]
[True, True, True]
results: ['banana', 'satsuma', 'mango']
>>> gc.mem_free()
9456
>>> 

[pythoncoder]

I use frozen bytecode for uasyncio on ESP8266. This avoids having to cross compile applications (unless they are large).

[apois]

I've been doing a bit of reading around futures to learn a bit more and found a couple of interesting references:

Firstly, Javascript has the Promises/A+ specification - https://promisesaplus.com - which has been widely adopted and is largely seen as a step-up from the previous "callback hell". I quite like promises and find them fairly intuitive to work with, but they appear to have a number of shortcomings:

• difficult/complex to implement correctly
• difficult to cancel operations

There is another project - https://github.com/fluture-js/Fluture - that is a bit lower level and tries to address some of these shortcomings. It is based on a "monadic" API for futures - monads are seen by some as scary, irrelevant abstract algebra, but the algebraic monad laws boil down to widely-applicable sensible rules for nesting/sequencing/flattening operations. These rules have been usefully applied to all sorts of computational structures - lists, exceptions, state, IO, parsers, nullable types, etc. - so it is not surprising to see them applied to futures.

So - a couple of the deficiencies of Promises that Flutures tries to address:

• simpler to implement/reason about - whereas Promises allow arbitrary listeners to subscribe to the result of a Promise, Flutures explicitly plumb in a single "reject" continuation/callback (for error handling) and single "resolve" continuation/callback (for proceeding). This makes the implementation simpler, because they essentially "unicast" their result, but a bit more work is required to "broadcast" results to multiple subsequent Flutures.
• designed to support cancellation - when you "fork" a Fluture (ie. kick off its computation) it can (optionally) return a callback to cancel that specific Fluture. When called, that cancel callback should, in turn, propagate the cancel message to any Flutures it is depending on.

This seems quite neat - and as it is based on explicitly passing continuations, I suspect it could be implemented fairly efficiently in something like micropython. However, I don't know enough about these, or (u)asyncio, to know if it provides significant benefit/difference over the existing uasyncio task implementation. I assume Flutures could be implemented on top of the uasyncio event loop, but I'm not sure how good a fit it would be. But it has been interesting reading about this stuff

[pfalcon]

A couple of words, how I, the author of uasyncio learned it. Well, I of course started with "big" Python version, asyncio. Then I just reimplemente a subset of each which made sense for very small-memory systems, leaving out everything which could be left out (out of the core, with the idea that other things can be implemented on top of it as needed).

I personally can't see any other way to start, except by following Python in general - the whole MicroPython project was originally intended for people who know Python and would like to use it in more constrained environments (disclaimer: intended by me, as a second largest contributor to the MicroPython, other parties may had other intentions). MicroPython docs, while fairly detailed in comparison with similar projects, still usually describe the differences to "big" Python, etc.

It turns out that MicroPython has a unique challenge that a lot of people coming to it who aren't familiar (enough or at all) with Python. My personal position in that regard is firm - learn Python, if you're interested in MicroPython, it's fully worth it. But then, I'm very happy that the forum has enough people who patiently explain things to novices, which are well described in the Python docs .

But it has been interesting reading about this stuff

That's the right way to do it - read about this stuff. Reading is exciting! Read docs, read code, read everything.

[pythoncoder]

@apois uasyncio is highly efficient. Scheduling is performed without RAM allocation: this ensures that context switches are always fast (156μs on the Pyboard) as they never trigger garbage collection. Non-allocation also avoids contributing to heap fragmentation. It is possible to run 1000 coroutines on a Pyboard.

The uasyncio API is heavily based on the standard Python asyncio library so it is easy to write asynchronous code which is portable between CPython and MicroPython. I regard it as a major achievement by @pfalcon.

An alternative library using a different paradigm could be written, and it might be a thoroughly worthwhile exercise. But achieving comparable performance in a microcontroller context might prove challenging.

[apois]

@pfalcon - thanks for your advice. Yes - I agree - compatibility with (a subset of) Python 3 is a crucial goal. That's kind of how I ended up down the rabbit-hole of this thread - I wanted to do what I thought would be a simple asyncio.gather() but realised that this functionality wasn't available in uasyncio, so started look for options to achieve it. But I'm certainly not qualified to try reimplementing a production-ready async library in micropython! Just curious about options for achieving my use case, which I initially thought would be straightforward.

Regarding your comment about python novices - don't discount the fact that there is another group - developers like myself who are familiar with python but have never had reason to use asyncio until trying micropython, where it becomes much more compelling than on the desktop/server. My async experience is all node or client-side javascript - hence my attempts to understand uasyncio in terms of those.

@pythoncoder - yes - an efficient implementation is key, as this enables close to system-level functionality. For my particular use case though, it is more about higher-level programming convenience - I'm not worried about wasting a few hundred milliseconds if my sensor reading takes 5 seconds, and on an ESP32 I have room to be a bit "flabby" on memory usage . But yes - the underlying uasyncio is an amazing achievement for device-level programming.