uasyncio - asyncio-like cooperative multitasking framework for uPy

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
pfalcon
Posts: 1154
Joined: Fri Feb 28, 2014 2:05 pm

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by pfalcon » Wed Dec 26, 2018 9:47 pm

This thread is now back to the "announcements of new uasyncio releases" mode. Please ask technical questions in separate threads, properly titled - after you searched the forum for existing discussions. Please ask non-technical questions in dedicated threads - after you read the existing material on the matter.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

kevinkk525
Posts: 391
Joined: Sat Feb 03, 2018 7:02 pm

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by kevinkk525 » Thu Dec 27, 2018 5:31 am

pfalcon wrote:
Wed Dec 26, 2018 9:31 pm
make the default upip install of micropython-asyncio work with mainline MicroPython
mattyt, are you sure you knock on the right door? Did you read the post above at all? Did you read the links in it? There's a patch which needs to be merged to "make the default upip install of micropython-asyncio work with mainline MicroPython". You should implore to the maintainer of the mainline. And heck, the mainline has fallen into subpar maintenance, as discussed by the links above. I'm doing all what I can to continue development and improve the situation with the mainline too (raising questions, offering help with the maintenance, submitting patches). What are you doing to improve the situation?
At least we are not doing anything to make the situation worse. Putting a version on pypi that can't work on the mainline micropython (yet) is a very unhealthy and hostile move, further scattering the micropython environment and harming the community.
You can't demand mainline developers to implement the patch that you want to see by leveraging that the typical newbie to micropython will try to use your new uasyncio module from pypi which won't work on his mainline firmware build. So he gets frustrated and leaves micropython in the worst case. So why do that? It does not matter what you think about mainline, it is still the entry point to micropython.
What exactly are you doing that improves the micropython environment with this move?
You could have published your new version as micropython-uasyncio-dev and everyone would have been happy.

User avatar
mattyt
Posts: 222
Joined: Mon Jan 23, 2017 6:39 am

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by mattyt » Tue Jan 08, 2019 10:58 am

pfalcon wrote:
Wed Dec 26, 2018 9:31 pm
mattyt, are you sure you knock on the right door? Did you read the post above at all? Did you read the links in it?
Yes, I had read the entire sordid mess. I'm well aware that I'm "knocking on the door" of the person who just willfully broke mainline MicroPython. Again.

I'm going to have to take another group of fledgling MicroPython programmers and explain to them how to work around the issue you've created. I fail to understand how you think this is good for the community? Particularly since it would have taken little effort to avoid (by renaming the 'development' package of asyncio as discussed above).
pfalcon wrote:
Wed Dec 26, 2018 9:31 pm
There's a patch which needs to be merged to "make the default upip install of micropython-asyncio work with mainline MicroPython". You should implore to the maintainer of the mainline.
Interesting point of view. I read that PR and Damien raised a reasonable objection to the change. Your responses since weren't particularly compelling.

Shouldn't the onus be on the person raising the PR to make a convincing case to integrate a change?
pfalcon wrote:
Wed Dec 26, 2018 9:31 pm
And heck, the mainline has fallen into subpar maintenance, as discussed by the links above.
Just because the project isn't moving at the pace - or in the direction - that you want doesn't mean it's 'fallen into subpar maintenance'.
pfalcon wrote:
Wed Dec 26, 2018 9:31 pm
I'm doing all what I can to continue development and improve the situation with the mainline too (raising questions, offering help with the maintenance, submitting patches).
I'm honestly and genuinely grateful for your contributions. Seriously; thanks Paul!
pfalcon wrote:
Wed Dec 26, 2018 9:31 pm
What are you doing to improve the situation?
What does that have to do with anything? I do what I can. I apologise if you feel it's not enough. I would dearly love to contribute more.

In any case, I wasn't going to respond because doing so is exhausting and feels futile. You've made up your mind and seem unlikely to change. The only purpose of my response is that I think you should be aware that at least I feel that you've not acted in the best interest of this community with this decision. But c'est la vie.

pfalcon
Posts: 1154
Joined: Fri Feb 28, 2014 2:05 pm

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by pfalcon » Sat Feb 02, 2019 8:33 am

A major evolutionary step for uasyncio - a long-standing bug with cancellation of uasyncio.sleep() call was fixed. That bug didn't allow to write reliable networking applications (with timeouts, etc.), and rendered uasyncio practically useless for applications outside its initial domain (which was again, writing networking servers). Now uasyncio becomes a truly generic async scheduling framework. This is released as uasyncio.core 2.1 .

The history of solving this bug is "truly dramatic" - the work started in the framework of ESP8266 kickstarter campaign, one of the stretch goal of which was bringing uasyncio to ESP8266 and let it shine there. Cancellation support was at all implemented then, with a known issue of not being able to cancel uasyncio.sleep(). Solving that issue required considerable architectural refactors to uasyncio and its support modules in MicroPython core. All in all, by December 2017, I had a solution for uasyncio.sleep() problem too. But out of sudden, the owner of project, and my companion in the kickstarter, rose against the next batch of uasyncio-related improvements in MicroPython, and I had my commit access removed. While I continues work in my fork, there was an outcry of "don't break our loved bugs with the mainline", so related patches sat for a year in my backlog. Last December I brought up issue of the need of maintenance of mainline again, with just as disappointing results. So, the decision to go and fix the pending issues and continue active development in Pycopy, my fork of MicroPython, solidified. Even then a couple of "well-wishers" from the Cult of Unfixed Bug started a sh%tstorm in a teacup, as can be beheld in the messages above.

So, all in all, bugs are being fixed and intended to be fixed further. The active development of MicroPython happens in the Pycopy fork, linked above. Per the previous announcements, it is required for the latest versions of uasyncio. There're more bugs to fix and features to add, stay tuned.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

eltomek
Posts: 11
Joined: Sat Feb 02, 2019 8:32 am

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by eltomek » Sat Mar 02, 2019 8:38 pm

For posterity, as I've spend several days investigating this case, maybe this is going to save somebody else's time.

While running my picoweb server and some mqtt reporting to Thingspeak executed within uasyncio loop on my Wemos D1 mini (ESP8266) after some time I was getting a vague

Code: Select all

ValueError: generator already executing
error message reported for

Code: Select all

await asyncio.sleep(delay_secs)
line of code of the coro definition.

More context:
Error message:

Code: Select all

Traceback (most recent call last):
  File "main.py", line 233, in <module>
  File "picoweb/__init__.py", line 298, in run
  File "uasyncio/core.py", line 154, in run_forever
  File "uasyncio/core.py", line 109, in run_forever
  File "main.py", line 223, in mem
ValueError: generator already executing
where "main.py", line 223, in mem is:

Code: Select all

await asyncio.sleep(delay_secs)
"main.py", line 233, in is:

Code: Select all

loop.create_task(mem(1))
As it was happening after dozens of minutes but in the same loop of iteration every time I thought it might be related to memory.
Quick check on gc.mem_free() every loop iteration and it turned out the the heap was going down to as low as ~10kB while the script encountered an interruption with the above error message.

Now, with gc.collect() executed every Thingspeak upload iteration of the loop the script really runs forever ;-)
Alternatively you can set the gc.threshold(2000) at the top of your script to make sure gc does the collection with 2kB advance, 2000 is an empirical value confirmed to work for my Wemos D1 mini board.

Anyway, it's interesting why does the uasyncio return this error message and why it fails allocating the memory.

Interestingly, as a follow up, I checked the gc.threshold(-1) to disable allocation threshold and then heap goes to as low as 400 bytes even not causing the issue described above. So, not sure if I nailed this problem right unfortunately.

pfalcon
Posts: 1154
Joined: Fri Feb 28, 2014 2:05 pm

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by pfalcon » Tue Mar 26, 2019 6:38 am

eltomek, for there to be any chance to do anything about this report, you would need to provide minimal reproduction testcase running on unix port of MicroPython. See http://sscce.org/ for basic info about such reproduction testcases.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
pythoncoder
Posts: 3404
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by pythoncoder » Tue Mar 26, 2019 7:03 am

As a very general point unrelated to uasyncio I have found that periodically issuing gc.collect() is a wise precaution in MicroPython programs running on baremetal platforms. This is especially true of the ESP8266 with its limited RAM. As you probably know, the heap can become fragmented: regular collections can pre-empt this. If the heap is fragmented, the exact line of code where failure occurs is not necessarily very revealing as it can be a symptom rather than the cause.
Peter Hinch

pfalcon
Posts: 1154
Joined: Fri Feb 28, 2014 2:05 pm

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by pfalcon » Thu May 09, 2019 9:51 am

A few releases were made in the meantime:

* uasyncio.core/uasyncio 2.3, with a cancellation fix for I/O-blocked coroutines.
* uasyncio 3.0, with long-planned optimization of I/O-blocked coroutine tracking.
* uasyncio 3.0.1, with fixes for better wrapper streams support.

As all the recent versions of uasyncio, it requires Pycopy project, viewtopic.php?f=15&t=5699 . uasyncio 3.0+ is tested with Unix port primarily, help with testing it on bare-metal targets is welcome.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Nezra
Posts: 3
Joined: Mon May 13, 2019 10:36 am

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by Nezra » Mon May 13, 2019 10:45 am

Here's my problem. I would love to use asyncio. I need to use it. But. I'm no super compilation genius. I thinker, as do allot of people coming in with the promise of python on the esp32.

Picoweb has effectively cornered the market on async. Picoweb, and asyncio, only run on the pycopy of the library. Pycopy has 0 binaries for the esp32 with psram support.

The main branch has binaries but doesn't work with either. I've searched for days, tried all of the workarounds, and it's effectively useless without what is a difficult process of figuring out how to recompile a binary to work on the esp32. I don't have Unix. And making a vm for the sole purpose of making the firmware is a pain. Most folks who are all new to this just wants something that "works".

I'm afraid to upip anything since asyncio and picoweb demonstrate the pitfalls created. I'm afraid to delve down the pycopy train since it's not "newb" friendly.

So what options or directions are left to help people like me who will get turned off any port of micro python because of this? Anyone have a better way or decent instructions for dummies to do all of this to just get something that "works"?

User avatar
pythoncoder
Posts: 3404
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: uasyncio - asyncio-like cooperative multitasking framework for uPy

Post by pythoncoder » Mon May 13, 2019 1:00 pm

You can use uasyncio with official MicroPython - on many platforms it is pre-installed as frozen bytecode. You might like to look at my uasyncio repo which contains various resources for running uasyncio under official MicroPython, including a tutorial. This has an installation guide for any platform which doesn't include it as standard.

The version of uasyncio which requires Pycopy is version 3.0. The official version is V2.0.

It is also possible to run Picoweb under official firmware with a little simple hacking. See my post in this thread which explains how to run one of the Picoweb demos under V2.0 and official firmware.
Peter Hinch

Post Reply