(u)asyncio task cancellation spreading like a virus

[kevinkk525]

(u)asyncio is great and you can easily write concurrent programs with it, but the deeper I get into asyncio, the more strange it can get.
This is not an issue with neither uasyncio nor Cpython asyncio. It's just a general "library problem".
One particluarly frustrating problem is this:

The new uasyncio offers way better task cancellation and there are so many use-cases for task cancellation.
However, cancelling a task can behave like a virus, killing everything connected to that task.

Example:

Code: Select all

try:
    import uasyncio as asyncio
except:
    import asyncio


async def wait():
    print("waiting")
    try:
        await asyncio.sleep(1)
    except asyncio.CancelledError:
        print("wait cancelled")
        raise


async def awaiter(t):
    print("awaiting t")
    try:
        await t
    except asyncio.CancelledError:
        print("awaiting cancelled")


async def test_cancel_wait():
    aw = asyncio.create_task(wait())
    t = []
    for _ in range(5):
        t.append(asyncio.create_task(awaiter(aw)))
    await asyncio.sleep(0.1)
    aw.cancel()
    await asyncio.sleep(0.1)


async def test_cancel_single_awaiter():
    aw = asyncio.create_task(wait())
    t = []
    for _ in range(5):
        t.append(asyncio.create_task(awaiter(aw)))
    await asyncio.sleep(0.1)
    t[0].cancel()
    await asyncio.sleep(0.1)


print("----------------------------\ntest_cancel_wait")
asyncio.run(test_cancel_wait())
print("----------------------------\ntest_cancel_single_awaiter")
asyncio.run(test_cancel_single_awaiter())

To summarize the tests:
In both tests a wait() task "t" is created and 5 "awaiter()" tasks which will wait for "t" to finish.
In the 1. test the task "t" will be cancelled, which therefore propagates the cancellation to all tasks awaiting it. So far absolutely understandable and logical.
In the 2. test one of the "awaiter()" task is cancelled. This now spreads like a virus, killing task "t" and therefore all other "awaiter()" tasks.

The behaviour in Test 2 ("test_cancel_single_awaiter") I absolutely can't understand.. What's even the point of this behaviour? Task "t" with coroutine "wait()" is an independent task, it has no ties to "awaiter()" but still, if an "awaiter()" gets cancelled, the "awaiter()" also cancels "t".
So killing one task awaiting some completely independent task can make your whole program get killed..

Real world example:
Let's make task "t" with coroutine "wait()" a task that subscribes all mqtt topics after a (re)connect.
Also you have 5 "awaiter()" tasks that are going to wait for the subscriptions to be done so they can continue with some other initialization code.
Now let's say one of these "awaiter()" can't wait until everything is susbcribed and has to move on after a short time so it uses wait_for() which behaves the same way as cancelling that task.
So when this wait_for() times out (or the task gets cancelled), our task "t" which subscribes all mqtt topics gets cancelled! (This of course causes all other tasks to receive a CancelledError but that is expected behaviour).


Is there anything I missed in asyncio that would prevent this?

I know I could use an Event, because then cancellations are no problem anymore. But then there's only a limited use in awaiting other Tasks except if you really have only a single task awaiting another task.

[pythoncoder]

For the benefit of others following this thread, the behaviour of CPython 3.8 is identical, so this is not a uasyncio bug.

Perhaps I'm missing something, but this behaviour does not seem surprising. If task foo awaits task bar, and foo is cancelled, bar is also cancelled otherwise it would be an orphan:

Code: Select all

try:
    import uasyncio as asyncio
except:
    import asyncio


async def bar():
    print("waiting")
    try:
        await asyncio.sleep(1)
    except asyncio.CancelledError:
        print("bar cancelled")
        raise


async def foo():
    print("awaiting bar")
    try:
        await bar()
    except asyncio.CancelledError:
        print("foo cancelled")

async def main():
    t = asyncio.create_task(foo())
    await asyncio.sleep(1)
    t.cancel()
    await asyncio.sleep(1)

asyncio.run(main())

In your example you have a single instance of aw, with multiple tasks waiting on it. When you cancel one of those tasks, aw gets cancelled as above. But if you cancel a task on which another awaits the cancellation propagates up the call stack to the awaiting task. That too has to happen, as your first test demonstrates well.

I can imagine that preventing this from occurring would be complex, and I'm unconvinced that the resulting behaviour would be more logically "correct".

I think there are some scenarios where task cancellation is not the right solution and a system of Event flags would be better.

[pythoncoder]

On further thought, you can fix this by swallowing the cancellation rather than issuing raise. I'm undecided whether this is a good idea, but the following works in MicroPython and CPython:

Code: Select all

try:
    import uasyncio as asyncio
except:
    import asyncio

async def wait():
    print("waiting")
    try:
        await asyncio.sleep(1)
    except asyncio.CancelledError:
        print("wait cancelled")  # Swallow the exception!
#        raise

async def awaiter(t):
    print("awaiting t")
    try:
        await t
    except asyncio.CancelledError:
        print("awaiting cancelled")
        raise  # I added this as it's usual


async def test_cancel_wait():
    aw = asyncio.create_task(wait())
    t = []
    for _ in range(5):
        t.append(asyncio.create_task(awaiter(aw)))
    await asyncio.sleep(0.1)
    aw.cancel()
    await asyncio.sleep(0.1)

async def test_cancel_single_awaiter():
    aw = asyncio.create_task(wait())
    t = []
    for _ in range(5):
        t.append(asyncio.create_task(awaiter(aw)))
    await asyncio.sleep(0.1)
    t[0].cancel()
    await asyncio.sleep(0.1)


print("----------------------------\ntest_cancel_wait")
asyncio.run(test_cancel_wait())
print("----------------------------\ntest_cancel_single_awaiter")
asyncio.run(test_cancel_single_awaiter())

[kevinkk525]

Perhaps I'm missing something, but this behaviour does not seem surprising. If task foo awaits task bar, and foo is cancelled, bar is also cancelled otherwise it would be an orphan:

In your example you have a single task foo() that awaits another coroutine (not Task!) bar(). Then you cancel the task foo(). In this scenario it is absolutely logical and needed that bar() gets canceled because it is the same task.
However, you can't compare that behaviour to having multiple tasks running because it is (theroretically) quite different. (Well it may behave the same way but that is where it gets illogical in my opinion)

My example has independent tasks:
Task 1: foo(): do something that depends on nobody, e.g. subscribe mqtt topics
Task 2-5: bar(): await Task 1 (the already independently running task, not the coroutine)

So Task 2-5 await a result from Task 1.
Cancelling Task 1 consequently pushes the CancelledError up the chain to all Tasks awaiting it.
Cancelling one of Task 2-5 pushes the CancelledError up the chain (which is nothing because nothing is awaiting those Task) but apparently also "down the chain" to Task 1. And this is illogical because Task 1 is an independently spawned Task that won't be an orphan if Tasks 2-5 get cancelled because it was independently started and supposed to run by itself.

On further thought, you can fix this by swallowing the cancellation rather than issuing raise.

No it wouldn't fix the issue because Task 1 would still get canceled, which is the main issue. The remaining Tasks 2-5 would then not get canceled but receive "None" as a return value because Task 1 still gets canceled and returns something.

I noticed that CPython provides "asyncio.shield(task)" which would protect Task 1 from being cancelled and consequently also the remaining Tasks 2-5 from being cancelled. While the behaviour is still illogical to me, it means the problem could be solved by having asyncio.shield implemented in uasyncio.

Why does this bother me in the first place? I could use an Event to provide similar functionality but it would make the code a lot more complex and more RAM hungry, so I try to be as efficient as possible.

[pythoncoder]

OK, thanks. I now see what you're trying to do but can't see a solution other than an Event-type mechanism.

I also see why this behaviour is arguably incorrect, but given that CPython works this way I think we'll have to accept it. I must admit that until recently it never occurred to me that you could await an already-running Task.

[kevinkk525]

Yeah there are some more interesting scenarios with asyncio. You could e.g. take the same code as above and cancel an awaiter(). That throws a CancelledError inside the awaiter() in "await t" but how do you know if Task t got cancelled or awaiter() got cancelled? You can't... In the end both get cancelled anyway but you can't determine if the reason for the cancellation was task t or awaiter()...

Awaiting tasks seems like a perfect recipe to cause confusion and strange scenarios.. Might be easier to never await other task, only await coroutines.

[tve]

Awaiting tasks seems like a perfect recipe to cause confusion and strange scenarios.. Might be easier to never await other task, only await coroutines.

The semantics of await and cancel seem rather unintuitive! I looked at the implementation of futures in CPython for a few minutes and I can't tell whether they also exhibit this issue, i.., if two tasks await a future and one gets canceled does the future get canceled too such that the second task is left high and dry? I know futures aren't implemented in MP yet, I was just wondering how much these semantics find their way into other abstractions in a sneaky way.

[kevinkk525]

Yeah Futures and Tasks are another perfect example of looking for trouble.. To be honest I haven't really done much with futures and the last time I looked at these was a few years ago. There have been so many changes in CPython asyncio since, I don't know if it even works the same anymore.

If I had the time (and the option in micropython) I would probably learn an asyncio alternative like trio or curio. They seem to be better in a lot of aspects (no future/task confusion, better timeout handling, no callback hell, ...)

[pythoncoder]

@kevinkk525 What do we conclude from this? I see one conclusion and a question.

My tutorial should warn against cancellation of Task instances.

Should we advocate for (or try to implement) .shield?

In all the code I've written for uasyncio I've never found the need to await a Task so I wonder how common this is in microcontroller applications. But if implementation of .shield is easy, perhaps that's the way to go?

[kevinkk525]

My tutorial should warn against cancellation of Task instances.

No, cancelling Tasks is a crucial part of working with uasyncio and works just fine. It's good to use it.

@kevinkk525 What do we conclude from this? I see one conclusion and a question.

I would advise against awaiting other tasks unless you know the implications. It could easily mess up things and confuse especially people new to uasyncio. Only exception might be a function that starts mutliple tasks and then wants to wait until one task has finished before proceeding (or maybe even until all tasks have finished using gather()).

Should we advocate for (or try to implement) .shield?

If I have time over the next days, I might try an implementation. I already have something in mind. Might only be "proof-of-concept" for Damien but should be enough for now.

In all the code I've written for uasyncio I've never found the need to await a Task so I wonder how common this is in microcontroller applications.

I wonder about this myself.. But there are a few use-cases like synchronization of multiple tasks or returning a result to multiple other tasks. But this could be done less confusing (but with more code and RAM) by using an Event/Message instance
Only other exception might be a function that starts mutliple tasks and then wants to wait until one task has finished before proceeding (or maybe even until all tasks have finished using gather()).
On the other hand we never had this feature on microcontrollers because the old uasyncio didn't offer any of these features. So there might be more use-cases we just didn't encounter yet.
I always find a new way of solving my solutions while trying to port and optimize all my modules to the new uasyncio.