Some problems with asyn library

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
User avatar
on4aa
Posts: 70
Joined: Sat Nov 11, 2017 8:41 pm
Location: Europe
Contact:

Re: uasyncio — basic event loop

Post by on4aa » Sat Jan 27, 2018 8:02 pm

@cefn
All your remarks make perfectly sense.
cefn wrote:
Sat Jan 27, 2018 3:47 pm
Simply create an 'uncooperative' coroutine function which is scheduled in the round-robin and will block the thread with a utime.sleep() call. Then you have your behaviour, but without lodging it deep in a library where it isn't appropriate.
However, the problem lies not with the loop round robin.
When one leaves out loop.create_task(update_lcd(data_ready)), only leaving loop.create_task(sample(data_ready)), the round robin behaves perfectly fine.

It is only when starting to await an Event in update_lcd(data_ready) that things get out of control.

I only started meddling with MicroPython for Unix because it happens to be the recommended route to install uasyncio on MCU platforms that are lacking internet connectivity.
I figured it would equally be a nice way to run a few quick experiments before implementing these on the Pyboard or ESP32. Honestly, I am a bit shocked that even the simplest of uasyncio experiments got me, as a beginner, caught up into this.

On the other hand, code portability to other platforms has its merits besides quick prototyping; it is a sign of language maturity and gives the developer confidence that his/her code is performing well.

I do not wish to influence one way or another, but I have to admit that some of that confidence has gone now. I just would like to implement uasyncio with eventually a GUI and a number of background processes, but right now, I am not sure that what I am doing is right.
Serge

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: uasyncio — basic event loop

Post by cefn » Sun Jan 28, 2018 12:41 am

Let me be more precise.

Here is your code, with an extra coroutine function make_idle() added, and a commented call which passes the coroutine constructed by make_idle() to loop.create_task()

Code: Select all

import uasyncio as asyncio
import asyn    # https://github.com/peterhinch/micropython-async/blob/master/asyn.py
import sys
import utime

async def sample(data_ready):
    while True:
        print('Sample taken')
        data_ready.set()
        await asyncio.sleep(3)    # in s

async def update_lcd(data_ready):
    while True:
        await data_ready
        print('LCD updated')
        data_ready.clear()

async def make_idle():
    while True:
        utime.sleep(0.050)
        await asyncio.sleep(0)

loop = asyncio.get_event_loop()
data_ready = asyn.Event()
loop.create_task(sample(data_ready))
loop.create_task(update_lcd(data_ready))
#loop.create_task(make_idle())

try:
    loop.run_forever()
except KeyboardInterrupt or SystemExit:
    sys.exit()
I ran this on Ubuntu with...

Code: Select all

micropython forum_test.py & top -p "$!" && kill "$!"
...and it demonstrated the issue you described - 100% CPU usage.

However, when I uncomment the call to create_task, instead of the Event causing a repeated rescheduling of itself, they are interleaved by a blocking delay (as you requested) but without any need to manipulate asyn.py defaults. Running the same command line then results in a CPU usage of 0.0%

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: uasyncio — basic event loop

Post by pythoncoder » Sun Jan 28, 2018 9:27 am

@cefn I don't think this will have any effect on STM power consumption. A call to utime.sleep(0.050) on systems with an underlying OS causes the underlying OS to deschedule the MicroPython VM for that period, reducing CPU utilisation.

On STM the processor consumes a fairly constant current unless you specifically invoke a low power mode. In my micropower library I have a low power version of pyb.delay() which does this. Standard delay routines do not conserve power.

I haven't studied this on ESPx chips.

@on4aa
I have to admit that some of that confidence has gone now
It seems to me you're complicating this with your concerns about power consumption. As designed uasyncio doesn't address this issue. If I were you I'd get your application working without worrying about calls to sleep(0). Then, when the basic functionality is there, consider what can be done to reduce power. The situation is platform dependent: techniques which work to reduce CPU utilisation on Unix won't have any effect on STM. You may wish to modify uasyncio to invoke a low power mode. You'll be in a better position to attempt this when you have experience of making a working uasyncio application.
Peter Hinch
Index to my micropython libraries.

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: uasyncio — basic event loop

Post by cefn » Sun Jan 28, 2018 10:14 am

@cefn I don't think this will have any effect on STM power consumption. A call to utime.sleep(0.050) on systems with an underlying OS causes the underlying OS to deschedule the MicroPython VM for that period, reducing CPU utilisation.
Absolutely. Wasn't claiming it would help with power consumption except in Unix. Hence my comment to the OP...
the issue you raise arises for the desktop ports of Micropython, where sleep actually translates into a low-power mode, which is not the development target for Micropython
The approach I shared is relevant if you want a long-running test of code within the unix port to verify stability and eliminate the possibility of memory leaks for example. (I have done code fuzzing this way in the past). Without the time.sleep(0.05) task, your CPU will run hot at 100%, your fan will spin up and your laptop run out of battery in a fraction of the time. With that task, it runs around 0.0% in top.

My observation was that any need for idleness for a given platform could be achieved by scheduling an idle task without going under the hood in asyn, hence supporting your point of view that Event shouldn't be polluted with arbitrary delay numbers and should remain with an inline asyncio.sleep(0), despite the fact this creates a busy-wait. The busy-wait is the right thing to do, and can be trivially neutralised for desktop platforms as per the code example I shared.

However, I am still a bit curious whether scheduling a time.sleep(0.05) actually gives up CPU time to other threads in Micropython-derived platforms having threading , compared to the use of only asyncio.sleep(0). Not curious enough to actually acquire such a platform and test it though :)

User avatar
on4aa
Posts: 70
Joined: Sat Nov 11, 2017 8:41 pm
Location: Europe
Contact:

Re: uasyncio — basic event loop

Post by on4aa » Mon Jan 29, 2018 4:35 pm

@cefn Thank you very much for amending my code to the point where it behaves itself on CPU.

Bear with me for a moment, I would like to recapitulate one step earlier to make sure all is well understood. Correct me if I am wrong.

Code: Select all

loop.create_task(sample(data_ready))
#loop.create_task(update_lcd(data_ready))
#loop.create_task(make_idle())
Above code runs with almost 0% CPU usage, whereas adding the LCD coroutine that needs to wait on an event, makes CPU usage soar to 100%:

Code: Select all

loop.create_task(sample(data_ready))
loop.create_task(update_lcd(data_ready))
#loop.create_task(make_idle())
Hence, an additional coroutine is introduced that behaves better at waiting thanks to utime.
Every 50ms, control is returned to the round robin be means of await asyncio.sleep(0). This gives other coroutines an opportunity to do something else. The whole time, CPU usage remains at almost 0%.

Code: Select all

async def make_idle():
    while True:
        utime.sleep(0.050)
        await asyncio.sleep(0)

loop.create_task(sample(data_ready))
loop.create_task(update_lcd(data_ready))
loop.create_task(make_idle())
Don't shoot the messenger, but having to add a coroutine because of the behaviour of another coroutine does not feel like the obvious or intuitive thing to do. More so because the sampling coroutine solely on its own behaves just fine. This is just my own personal opinion; nothing else. I will do a test to see whether this also happens with plain CPython.
The approach I shared is relevant if you want a long-running test of code within the unix port to verify stability and eliminate the possibility of memory leaks for example. (I have done code fuzzing this way in the past).
This is exactly why MicroPython Unix is relevant to me.
Serge

cefn
Posts: 230
Joined: Tue Aug 09, 2016 10:58 am

Re: uasyncio — basic event loop

Post by cefn » Mon Jan 29, 2018 6:02 pm

I can understand your concern, but the implementation of asyn's Event simply is a busy-wait polling a value. If you simply add a delay into the poll operation, this has an impact on responsiveness, and would not be appropriate for all applications, so shouldn't be baked into a library.

The asyn logic permits multiple 'simultaneously active' polling busy-waits, which means for most micropython microcontroller contexts, (without power-saving sleep, and without threading) this provides a suitable implementation for application logic which is waiting for one or more values to change. They can all be monitored simultaneously, with a cooperative scheduling strategy.

The convenience of the asyn classes can be omitted if you have a more opinionated strategy you can try, just use async and await directly.

If you were awaiting something which can actually be evented at a hardware/os level (I believe things like socket events, timer events, interrupts) then there is the opportunity to avoid polling busy-waits, and the event which awakens and reschedules into the loop can be deeply integrated into uasyncio rather than being just application-level values which are repeatedly polled. If I understand the situation, in these circumstances, uasyncio is handing over responsibility to one or more other subsystems to cause the CPU to re-enter its 'thread' based on some outside event, (often with a timeout meaning it will be rescheduled eventually anyway). That means it doesn't have to spend all its time in a busy-wait. For example, uasyncio.sleep() on unix can hand over responsibility to the OS, hence the low CPU.

Perhaps you are arguing for asyn events to be backed by some kind of hardware or os eventing, rather than being implemented as polling busy-waits. If so, what candidate hardware/os eventing semaphore are you thinking of which could be employed to provide this facility for asyn Event on Unix?

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Some problems with asyn library

Post by pfalcon » Tue Jan 30, 2018 9:35 am

Kind request - if you experience problems with library X, please name your topics correspondingly.

uasyncio library currently doesn't have an "Event" class, nor it possibly could have it with the behavior described here. As an uasyncio author I do not endorse usage of any 3rd-party library together with uasyncio (exactly because of the problems described in topics like this), so if you experience problems with library X, please treat it as problems with library X, unless you can reproduce it with the pristine uasyncio.

I've renamed topics to avoid confusion, please feel free to re-rename them to be more descriptive (which is good practice for topic authors anyway - to name topics well, and then rename as discussion goes on, to keep the title correspond to the content.)

Thanks.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
on4aa
Posts: 70
Joined: Sat Nov 11, 2017 8:41 pm
Location: Europe
Contact:

Re: asyn events — in basic uasyncio event loop; CPU usage

Post by on4aa » Tue Jan 30, 2018 7:54 pm

@pfalcon I understand your concern. I tend to chose my subject wording very carefully. However, this thread indeed took another turn along the way, which may give the wrong impressions about uasyncio on its own. I now changed the subject to attend to your concerns, whilst remaining as correctly informative as is humanly possible.

However, now that we have you here, perhaps you could lead us the way to some official documentation/tutorial about micropython-uasyncio, micropython-uasyncio.queues and micropython-uasyncio.synchro. Having had that would have saved some grief.
So far, I only found this.
Serge

User avatar
on4aa
Posts: 70
Joined: Sat Nov 11, 2017 8:41 pm
Location: Europe
Contact:

Re: asyn events — in basic uasyncio event loop; CPU usage

Post by on4aa » Tue Jan 30, 2018 8:02 pm

on4aa wrote:
Tue Jan 30, 2018 7:54 pm
I now changed the subject to attend to your concerns, whilst remaining as correctly informative as is humanly possible.
@pfalcon It indeed looks like I do not share your godlike powers to change thread titles! :mrgreen:
Serge

jonaslorander
Posts: 11
Joined: Tue Nov 08, 2016 12:33 pm

Re: asyn events — in basic uasyncio event loop; CPU usage

Post by jonaslorander » Tue Jan 30, 2018 8:08 pm

on4aa wrote:@pfalcon I understand your concern. I tend to chose my subject wording very carefully. However, this thread indeed took another turn along the way, which may give the wrong impressions about uasyncio on its own. I now changed the subject to attend to your concerns, whilst remaining as correctly informative as is humanly possible.

However, now that we have you here, perhaps you could lead us the way to some official documentation/tutorial about micropython-uasyncio, micropython-uasyncio.queues and micropython-uasyncio.synchro. Having had that would have saved some grief.
So far, I only found this.
Not official (?) but still a good tutorial https://github.com/peterhinch/micropyth ... UTORIAL.md

Post Reply