_thread on esp32

[pumelo]

Sorry to raise this again:

I found threads on the esp32 to be not reliable and I would like to know if anyone is using theme with success? Am I the only one to experience issues with theme?

First things first:
I use threads to do blocking IO operations like socket.getaddrinfo and ussl.wrap_socket.
At first I implemented a long running thread which pops functions from a list and executes theme. -> This did not work as in the presence of exceptions (or something else which I am not understanding) the thread somehow deadlocked/ceased operation.

Next I tried with "one blocking function per thread". So, for every function to execute in a thread, a new thread is started which will exit once the function is processed. Even with that approach some threads never will return ... In the end this manifests in that no new threads can be started due to some limit (memory for stack or task count?, I don't know).

The BAD:
I only experience this behavior of deadlocking thread in a fairly complete application. Until now I was not able to reproduce this issue with a "simple" case.

The speculations:
At first glance I thought, that the _thread deadlocks on some lock like the GIL. Investigating all MP_THREAD_GIL_EXIT(); / MP_THREAD_GIL_ENTER(); in the codebase I don't think the problem is here.

Reading the comments in modthread.c (see bleow) and my observation that in the presence of exceptions the thread dies, I am asking myself is: What happens if a pending exception is handled by another thread (can this even happen? soft interrupts?)? Will it jump to the stack of the thread which raised the exception?

Code: Select all

    // TODO set more thread-specific state here:
    //  mp_pending_exception? (root pointer)
    //  cur_exception (root pointer)

Most likely I am all wrong and the error is in my application code, still I am not able to find it :-/

Cheers

This

[pythoncoder]

I don't know if anyone else has used threading as a way to deal with blocking I/O - it has the potential of being an excellent idea.

I lack experience of threading in MicroPython - so far I've managed successfully to avoid it But regarding exceptions I'd try some test cases. Write scripts which throw after a period and see what happens. I would expect an exception to be contained within its own thread. If you can demonstrate that it propagates elsewhere I'd question whether this is expected behaviour or a firmware bug.

[pumelo]

Well inspecting this more I found that at some point the task priority of the main micropython task is raised to the task priority of the lwip task. And it will never revert back to it's original priority ... Which means, that the mp_thread did not actually die, but it does not get a chance to run anymore as the main micropython task has higher priority and hogs all the cpu.

Not sure why the task priority of the main micropython task does not get reset tough ...?

Task priorities can get increased if a task holds a mutexes which a higher priority task is waiting for ...
https://www.freertos.org/Real-time-embe ... texes.html

Here is the output of the task list as it should NOT be: (as output by https://docs.espressif.com/projects/esp ... TaskListPc)

Third column is the task priority

Code: Select all

mp_task        	R	18	9972	12
mp_thread      	R	1	27800	16
IDLE0          	R	0	916	6
IDLE1          	R	0	1048	7
isr_task       	B	24	14788	19
tiT            	B	18	2176	14
Tmr Svc        	B	1	1548	8
ipc0           	B	24	492	2
sys_evt        	B	20	1084	15
wifi           	B	23	4148	17
esp_timer      	B	22	3384	1
mdns           	B	1	2948	18
ipc1           	B	24	492	3

Task list as expected:

Code: Select all

mp_task        	R	1	9972	12
IDLE0          	R	0	916	6
IDLE1          	R	0	1048	7
isr_task       	B	24	14788	18
tiT            	B	18	2176	14
Tmr Svc        	B	1	1548	8
sys_evt        	B	20	1084	15
mdns           	B	1	2556	19
mp_thread      	B	1	27976	16
wifi           	B	23	4148	17
esp_timer      	B	22	3336	1
ipc1           	B	24	444	3
ipc0           	B	24	492	2

[pumelo]

Just in case somebody is still wondering about the problems I had with the thread:

This problem might be due to the omission of mine to read from the socket.

I opened a socked and did write to it but I never read from it.
I could not reproduce the problem described above when I do read from the socket.

So I think, that the problem is due to the recv buffer to fill up which under some condition will trigger to raise the task priority to the level of the lwip task or even the priority of the wifi task. This raising of the task priority made the thread (which is still running at task priority 1) not responding as it was never scheduled by freertos to run again. Basically the main micropython task will consume all the cpu ...

Once I started to read from the socket everything is fine and I did not see this issue again.