heap allocation in interrupts

[ttmetro]

@loboris @dhylands Am I interpreting this correctly:

On the ESP32 interrupts are handled in two stages:

• Actual hardware interrupt calls c-code that effectively "sets a flag" to tell the RTOS to run the MicroPython interrupt handler code at some later time
• The RTOS, whenever it gets around to it, schedules the interrupt handler code written in MicroPython. That scheduling never happens when the GC is busy. Hence, on the ESP32, it is safe to allocate heap in interrupt service routines. Correct?
• (That's the good thing - the downside presumably is that this is one of the reasons for the very long interrupt latencies we are seeing on the ESP32. Would be interesting to see if there is a way around this, since presently they are comparable to what's seen under e.g. Linux - why not just use an rpi or something then, aside from power, size, cost.)

Sorry to be so stuck on this - I want to be sure to say the right thing in my class.

[loboris]

@ttmetro

That is partially right.

• Actual hardware interrupt calls c-code that effectively puts the Python interrupt function pointer and arguments pointer to the MicroPython scheduler stack. That is all from the hw interrupt point of view. Scheduling the Python function execution has nothing to do with FreeRTOS.
• At that point the Python interrupt handler function is scheduled to run by the MPy runtime as soon as possible.
• “As soon as possible” means that the MicroPython runtime will do its best to execute the function at the earliest possible time, given that it is also trying to be efficient, and that the following conditions hold:
  • A scheduled function will never preempt another scheduled function.
  • Scheduled functions are always executed “between opcodes” which means that all fundamental Python operations (such as appending to a list) are guaranteed to be atomic.
  • A given port may define “critical regions” within which scheduled functions will never be executed. Functions may be scheduled within a critical region but they will not be executed until that region is exited. An example of a critical region is a preempting interrupt handler (an IRQ).

Yes, this is one of the reasons for the (very) long interrupt latencies we are seeing on the ESP32 MicroPython.

The method I'm using when I need very fast interrupt response is to write the whole interrupt function in C and run it directly from hw interrupt handler (if short enough) or start it as a FreeRTOS task (if more complex). Of course, it is not at all user frendly nor a Python way to do it.
There are some better ways to handle the interrupts in MicroPython on ESP32. I'm working on some, but it will not be available soon.

Some explanation quoted from MicroPython documentation

[ttmetro]

@loboris Many thanks for the detailed background!

There are some better ways to handle the interrupts in MicroPython on ESP32. I'm working on some, but it will not be available soon

Great to know that eventually we can expect better solutions.

My guess is that a few years from now MicroPython (or similar solutions) will replace C the way C replaced assembly for most micro controller programming. For that to happen it is important that MicroPython covers most use cases (without necessarily completely replacing C, the way C did not completely replace assembly). I understand that the ESP32 memory architecture is partially the culprit here (and that fast interrupts are possible e.g. on the pyboard), but some of these shortcomings of the ESP32 may appear also in other hardware and hence workarounds will be more important.

Thanks to all involved for the incredible work! What looks pioneering now may well be the norm a few years from now. I bet I will have students doing internships that will blow the minds their mentors when they demo a solution in a week what they thought would take month'. Then they will use it, too!

[pythoncoder]

So am I right in saying that in essence MicroPython ESP32 interrupt handlers are soft ISR's and GC is a critical region?

If so, allocation is permitted but it does imply worst-case latencies of multiple ms. Typical measured GC durations on the Pyboard are 1-10ms.

[dhylands]

So am I right in saying that in essence MicroPython ESP32 interrupt handlers are soft ISR's and GC is a critical region?

If so, allocation is permitted but it does imply worst-case latencies of multiple ms. Typical measured GC durations on the Pyboard are 1-10ms.

That sounds about right (at least the way things are coded right now).