Right way of synchronizing C and Micropython

[agonnen]

Hi,
I'm writing a Micropython module on a multi-threaded platform (ESP32 in this case, but the question is relevant to any other multi-threaded platform).
My module maintains a certain state (a C struct). This state can be changed from *both* Micropython extension functions (written in C), and another thread.
I would like to synchronize changes on my module's state (prevent two threads change the state on the same time). I could use a mutex to protect every Micropython extension.

Instead of doing that, since Micropython is running on a single thread, perhaps I could synchronize with Micropython itself?
I can think of the following options:

- Use `mp_sched_schedule` to schedule any access from another thread. This unfortunately requires me to define extension functions which will only be used by mp_sched_schedule and not the user, since `mp_sched_schedule` is designed to schedule Micropython functions and not C functions .

- Use `MICROPY_BEGIN_ATOMIC_SECTION` (or some other macro/funcion?) on any code that changes my state from another thread. Will that ensure Micropython and my code are synchronized?

- Use a mutex on every Micropython extension

Any suggestions for the right way of doing that?

[Maksym Galemin]

I don't fully understand your use case, but usually the right way to protect a C/C++ structure from concurrent access is by using a mutex in your C/C++ code.

[agonnen]

@

I don't fully understand your use case, but usually the right way to protect a C/C++ structure from concurrent access is by using a mutex in your C/C++ code.

Use case is any library with MicroPython interface that performs some async work, creates threads, handles interrupts etc. I would expect this to be rather common.
Of course mutex is the most straightforward way. But not the most efficient way. It consumes cycles and program ram if we protect every Micropython function that access that modules state (assuming we have lots of such Micropython functions).
So the other option is taking into advantage the fact that Micropython runs in a single thread. By using `mp_sched_schedule`, we can avoid mutex entirely. This is what I meant by synchronizing with Micropython itself.

My question was what is a good practice of doing that. Using mutex is one possible answer, but probably not the only one and possibly not the best one.

[BramPeeters]

What makes you say micropython is running a single thread ? I don't know the esp platform, but on eg an stm with freertos under it i have several micropython threads. Wether or not they are capable of interrupting each other depends on your use of the GIL. (I enabled it so that my python threads do not preempt each other and the persons writing micropython code do not have to worry about thread safey (but at a price of course) )

And if i understand correctly you also have some kind of OS next to micropython that has other non micropython threads ?
So apart from mutexes, you can also use the functions of that OS to create a critical section (usually their are various flavours, from disabling all interrupts to prevent preemtive scheduling) around the code that needs to be protected. (I assume the python thread(s) is/are scheduled by that OS too)

Or if your CPU has it (or something similar), and the criticial part of the struct fits in an unsigned long you can also use the strex/ldrex functions to guarantee atomic modifications without costly mutexes.

[Maksym Galemin]

Use case is any library with MicroPython interface that performs some async work, creates threads, handles interrupts etc. I would expect this to be rather common.

I would say more common is avoiding any undocumented, highly experimental features with unstable API like preemptive multitasking in MicroPython and using something more stable instead like cooperative multitasking with uasyncio.

I would still use a mutex in you case. Even if you have millions of functions you need just one mutex to protect your resource.