Managing asyncio tasks

[p_j]

What are the best practices for managing asyncio tasks, for example let's assume I have an event loop and three infinite tasks:
1. Streamreader capturing UART sensor data
2. Task updating display
3. Tasks sending data over modem

If a task crashes, what would be the best way to detect this and correct the behaviour. Should a 4th task be created that monitors the first 3 tasks and restarts failed tasks? What if the 4th task crashes, how deep should one go..

Currently I create the 3 tasks, append them to a loop and await all tasks, if any task has an unhandled error I reboot the mcu. Works but is pretty crude.

Code: Select all

async def main_loop():
    set_global_exception()

    tasks = []

    # Sensor task
    m = asyncio.create_task(sensor_process())
    await asyncio.sleep(0)
    tasks.append(m)

    # Connect modem and send MQTT
    s = asyncio.create_task(gsm_run())
    tasks.append(s)
    
    # LCD update
    m = asyncio.create_task(screen_update())
    tasks.append(m)

    await asyncio.sleep(0)
    for p in tasks:
        await p


try:
    logger.debug('starting event loop')
    asyncio.run(main_loop())

except KeyboardInterrupt:
    print('Interrupted')
except Exception as e:
    print("caught")
    print_exception(e)
finally:
    asyncio.new_event_loop()
    logger.warning('ended event loop')


logger.error('WE SHOULD NEVER GET HERE')
time.sleep(5)
logger.error('REBOOTING')
machine.reset()

[jimmo]

If a task crashes, what would be the best way to detect this and correct the behaviour. Should a 4th task be created that monitors the first 3 tasks and restarts failed tasks?

Honestly this isn't as bad as it sounds.

If a task crashes it becomes very hard to reason about the state of your program. Restoring to a known good state (i.e. reboot) is very sensible.

Perhaps I'm missing some context, but a task _shouldn't_ ever crash (i.e. have an unhandled exception). i.e. by definition if a task is crashing then there's a logic error or validation error or something wrong with the code, so all bets are off anyway.

If rebooting isn't an option, then the other way to have tasks mange themselves -- i.e. there's a top-level catch-all exception handler around the task entry function. Then at least a task can know how to resume itself.

e.g.

Code: Select all

async def display_task():
  while True:
    try:
      ... task code
    except:
      ... clean up

[p_j]

If a task crashes it becomes very hard to reason about the state of your program. Restoring to a known good state (i.e. reboot) is very sensible.

Perhaps I'm missing some context, but a task _shouldn't_ ever crash (i.e. have an unhandled exception). i.e. by definition if a task is crashing then there's a logic error or validation error or something wrong with the code, so all bets are off anyway.

Thanks jimmo this is a logical approach, if all obvious/known error conditions can be handled, then crashes should only be limited to obscure cases where a reboot would be best.

Yes in theory a task shouldn't ever crash however things sometimes crop up! I'm working on some devices that are being deployed remotely offshore and trying to develop the most robust approach. These devices are only accessible every 12 months so it is challenging. The first two I sent out lasted 5 weeks before a bug in my code put them permanently to sleep

[pythoncoder]

Where hardware is inaccessible the best solution is often to run a watchdog timer. This needs careful design: you need to ensure that, if any task fails, the task feeding the WDT is cancelled. You might want to study the global exception handler.

[p_j]

Where hardware is inaccessible the best solution is often to run a watchdog timer. This needs careful design: you need to ensure that, if any task fails, the task feeding the WDT is cancelled. You might want to study the global exception handler.

Thanks Peter I will have a look at using watchdog timers, this seems like a sensible solution. Interfacing to external hardware can be a pain, I'm finding modems to be the most annoying!