Run more than one micropython instance

[jockm]

I can tell you why I would like to have multiple micropython instances. I am developing a system that can have multiple user supplied "applets" and I would like to have process isolation. I don't want the code in one applet to be able to see (or conflict with) the code in another applet.

As far as I can tell there is no way to do this with uPy. There is with eLua (though it is a PITA to build it as a library), or with JerryScript (though a pain for other reasons).

[pythoncoder]

True process isolation needs to cope with the case where someone runs

Code: Select all

while True:
    pass

thereby locking everyone else out. This is only feasible with an underlying OS arbitrating between the MicroPython instances with pre-emptive multitasking. You could doubtless readily do this on a Raspberry Pi running Linux, but I don't think that's the answer you were looking for.

[jickster]

True process isolation needs to cope with the case where someone runs

Code: Select all

while True:
    pass

thereby locking everyone else out. This is only feasible with an underlying OS arbitrating between the MicroPython instances with pre-emptive multitasking. You could doubtless readily do this on a Raspberry Pi running Linux, but I don't think that's the answer you were looking for.

You could do cooperative multitasking and OS.yield() in between the op-codes.

[pythoncoder]

Enforcing that would require a modification to the compiler. But there are numerous ways in which arbitrary apps could conflict. Cooperative multi-tasking is just that: it assumes a set of tasks designed to cooperate in their use of resources.

Regardless of the cooperative/pre-emptive solution, MicroPython is designed primarily for bare metal environments. If two tasks start independently accessing the same peripheral, there will not be a happy outcome. Likewise if two tasks issue uctypes.bytearray_at(addr, size) with the same addr each can scribble over the other's data. Interrupts, the RTC - the sources of potential conflict are legion.

It seems to me that the only hope of task isolation is to run the Unix version and hope that the OS will save you from all but the worst excesses. My knowledge of security issues is limited but I doubt even Linux could protect a "good" MicroPython script from one deliberately crafted to subvert it.

[oclyke]

Hello everyone - interesting thread. I wish I had seen it earlier this year.

Summary: if the op (tarun2121) is still pursuing this project it is definitely within reach and I'd like to work together on it. If you can use an underlying scheduler such as FreeRTOS then the rest of the work is minimal thanks to the MP_STATE_XXX(x) macros.

Might as well throw in my two cents, and hopefully learn a little more. Earlier this year I started working toward the same goal (https://github.com/micropython/micropython/issues/4480) and now I have the basics in place. Fortunately for me the ESP32 port is already built within a FreeRTOS task so this problem was reduced to changing references to the mp_state_ctx structure to pointers and making sure that the correct pointer was always selected.

I'm still working on this and I would really love to get it pulled into MicroPython, but there are quite a few questions to answer first. The ones that I can think of off the top of my head are:

- How does this play with the rest of MicroPython?
- What is the common interface that any port can share and what needs to be reduced to a port-specific implementation?
- How can we update/improve the garbage collector and general heap management to be efficient and dynamically extendable?
- How much effort might it take to ensure that hardware peripherals all have some sort of lock to prevent simultaneous access?

At least for this first point it's tempting to say that we could relatively easily install the foundation (pointer access to the state) into MicroPython without impacting others. Also I think it's fair to say that this would not step on the toes of uasyncio because it serves a different audience and simply putting in the foundation hardly bogs down the codebase at all (IMO that is someone who might want to add additional code to execute without worrying about knowing what code is currently on the board.)

The other points will be significantly more work and I surely would not come up with the best solution on my own. Anyone interested and willing to collaborate? I'll be working on this here so please join me! https://github.com/oclyke/micropython

[imbearr]

Hi @oclyke, great start!
I'm think same direction - you can isolate py tasks by changing MP_STATE_XXX(x) macroses.

There some ideas:
1) Exist another way to have task context available from execution point - you can use Thread Local Storage Pointers provided by FreeRTOS.
2) Easiest way to improve heap usage - create isolated heap space for each task as they all already have each own stack, so when task end it's life you don't need to free memory.
3) If you implement (2), garbage collector will work isolated too.

I don't know how your changes will play with rest part of upy, and what's common or specific interfaces.
And I have big doupts to how you use global dicts because don't fully understand how it's work .
But argee with your idea that need root task that work with repl and another global things.
What about simultaneous access, there exist mutexes (in Micropython and in FreeFTOS too) and as I understand: you must implement some kind of it for each interface if it already not exist.

[quark]

https://github.com/micropython/micropython/issues/4480

Interesting thread to read. I'm coming from a Lua background, where it is designed to support multiple lua states without global locks. The application can make sure a single lua state is only used by a single thread (but a thread can have multiple lua states) to avoid the use of locking. That is actually an appealing API to me.

I don't know if MicroPython wants to have a similar mode where the vm states are not global variables, which I think has values. If so, I'd imagine the change will be having some macro magic prepending a "state" parameter to the majority of APIs.

[pythoncoder]

...
- How much effort might it take to ensure that hardware peripherals all have some sort of lock to prevent simultaneous access?
...

This is decidedly nontrivial. Consider peripherals like UARTs and Pins which raise IRQ's. If multiple instances are using the same device, the only way to ensure that the hardware interrupts get directed to the correct instance is for that instance to lock the device for the duration of its use (because incoming data can arrive at any time).

It would also require code-breaking API changes: applications would have to check for locks of each peripheral in turn, and implement some strategy for waiting for, asserting and de-asserting locks. Further, locking a device for the entire runtime of an application would cattle the competing instances for the duration. Possibly for hours.

As a very general point you need to be wary of conflicts between usage of MicroPython under an OS and baremetal use. In the latter case, applications assume sole access to devices. Many applications require microsecond level responses to interrupts. Any change which compromises this would not be well received: many users (myself included) regard baremetal usage as MicroPython's core raison d'etre.

[stijn]

the only way to ensure that the hardware interrupts get directed to the correct instance is for that instance to lock the device for the duration of its use

Depending on the type of interrupt isn't there another way, namely routing each interrupt to the different instances in software?

[pythoncoder]

I can't see how this would work. As far as I can see only one instance can use a device at any one time. If data arrives on a UART, the device driver can't "know" where to send it unless the data itself contains that information.

Consider Linux. My understanding is that if a process opens a serial device, the device is locked until it is closed. I don't think a serial device can be shared between processes.