Process isolation

[jockm]

I am developing a system where there can be one or more user applets — code that will be supplied by unknown sources and loaded off of external storage. I would like to make sure the applets don't conflict with each other, and aren't visible to one another.

Or in other words, I would like process isolation.

Normally I would solve this kind of problem by having multiple interpreters, one for each applet.

Unfortunately there doesn't appear to be any way to do this using uPy.

So is there any other way to achieve process isolation, or some way to have multiple interpreters that I am not aware of?

[pfalcon]

I am developing a system where there can be one or more user applets — code that will be supplied by unknown sources and loaded off of external storage. I would like to make sure the applets don't conflict with each other, and aren't visible to one another.

Or in other words, I would like process isolation.

Normally I would solve this kind of problem by having multiple interpreters, one for each applet.

There was discussion about supporting this, but nobody needed it really enough to actually implement it (at least in the mainline).

So is there any other way to achieve process isolation, or some way to have multiple interpreters that I am not aware of?

One which you're definitely aware of - just use a multiprocessing OS.

[jockm]

So is there any other way to achieve process isolation, or some way to have multiple interpreters that I am not aware of?

One which you're definitely aware of - just use a multiprocessing OS.

Which would be lovely if my RTOS supported heavyweight processes. Most that I am aware of have threads, so code that relies on global variables is going to be a problem.

Given how resource lean uPy is, it is an ideal choice for user scripting on larger microcontrollers (were JerryScript is being used right now); but without some means of isolating the system from user scripts, or user scripts from one another, it isn't an option.

[pfalcon]

jockm, you seem to be in good position to contract @dpgeorge to implement it then.

[jickster]

I am developing a system where there can be one or more user applets — code that will be supplied by unknown sources and loaded off of external storage. I would like to make sure the applets don't conflict with each other, and aren't visible to one another.

Or in other words, I would like process isolation.

Normally I would solve this kind of problem by having multiple interpreters, one for each applet.

Unfortunately there doesn't appear to be any way to do this using uPy.

So is there any other way to achieve process isolation, or some way to have multiple interpreters that I am not aware of?

Is the issue in this post different from that in your other post?

[jockm]

In my other post I explained a use case for wanting multiple interpreters — to achieve isolation.

Here I was asking if there was some other way to do this, and was told to use multiple processes to run multiple interpreters; and you know my response to that.

I thought I would ask in case there were some python based approach I was unaware of that might work in uPy.

This need is going to show up. Wanting to isolate user scripts from one another, isolating system code from user code, etc all need some way to make sure code in one script cannot access variables or make calls into another script. Also to make sure that collisions of class and variable names don't occur.

The moment uPy opened up the possibility of building the code as a library and embedding it in other projects (as opposed to just being the OS) people started wanting this feature. The question is if enough do to justify adding it.

In my case the effort to do so may not justify the reward. There are other scripting languages that can be used on microcontrollers, it would just be nice if I could use uPy

[SpotlightKid]

There was an effort to port MicroPython to mbed, but unfortunately it seems to have stalled shortly after it began in 2016. And anyway it was based on the obsolete old mbed not on the current mbed OS. But, AFAICS, this shows that in theory it should be possible to run MicroPython as a thread under an multitasking/threading microprocessor OS and from there it can't be that far to having multiple instances of it, right?

[jockm]

There was an effort to port MicroPython to mbed, but unfortunately it seems to have stalled shortly after it began in 2016. And anyway it was based on the obsolete old mbed not on the current mbed OS. But, AFAICS, this shows that in theory it should be possible to run MicroPython as a thread under an multitasking/threading microprocessor OS and from there it can't be that far to having multiple instances of it, right?

It wouldn't be hard to build uPy as a library and link it in a mbed project, but that doesn't mean you are any further along in getting multiple instances.

The issue with multiple instances and threads is that threads only have a separate stack (i.e. local variables), the heap (global variables) are shared. To allow for multiple instances without heavyweight processes, you would have to move all global variables to a structure (call it context) and then pass that context to every function that would need it. It can be done, but it isn't a small refactor. Also unless the comitters are behind the need and the overhead it imposes then it will die on the vine or live only as a fork.

[jickster]

There was an effort to port MicroPython to mbed, but unfortunately it seems to have stalled shortly after it began in 2016. And anyway it was based on the obsolete old mbed not on the current mbed OS. But, AFAICS, this shows that in theory it should be possible to run MicroPython as a thread under an multitasking/threading microprocessor OS and from there it can't be that far to having multiple instances of it, right?

It wouldn't be hard to build uPy as a library and link it in a mbed project, but that doesn't mean you are any further along in getting multiple instances.

The issue with multiple instances and threads is that threads only have a separate stack (i.e. local variables), the heap (global variables) are shared. To allow for multiple instances without heavyweight processes, you would have to move all global variables to a structure (call it context) and then pass that context to every function that would need it. It can be done, but it isn't a small refactor. Also unless the comitters are behind the need and the overhead it imposes then it will die on the vine or live only as a fork.

Why isn’t it just a small refactor?

Everywhere the macros MP_STATE_* are used is what you’d have to modify.

@pythoncoder mentioned another issue: scripts that allow direct access to physical resources - peripherals, memoryview of RAM - are going to have to be rewritten to use synchronization mechanisms to multiplex. Sounds very messy.

Even if code is refactored to pass in CONTEXT object, I guarantee you peripheral modules will NOT be rewritten (anytime soon) to support multiplexing. An entire new abstraction layer would have to be written and peripheral code updated to support multiplexing access to peripherals.

Example is SPI. There is one SPI peripheral and two apps with multiple slaves. How does one app know when the other is finished with a transaction so it doesn’t change the chip-select prematurely?
Mutex? If the apps have different CONTEXT and heaps, mutex is only possible with an underlying OS-type mechanism otherwise how would two apps share ANY information?

If you require an external OS, that limits the usefulness since, as someone here said, many people run micropython on bare-metal.
So that means a micropython mini-OS would have to be created to facilitate the synchronization across independent processes. Hardware-driver scripts would have to run in the uPy-OS kernel to be able to synchronize.

Sounds complicated but necessary if we want to enable process isolation AND synchronize access to hardware resources.

Can you run useful scripts without hw resource synchronization? Doubtful. Even networking requires synchronization: the ip stack cannot run in both of the apps because only one thing can control the physical interface at once.

Without synchronization, one app must be the master - and running the networking - and the others the slaves. The master downloads the `.py` files and kicks off new isolated instances then ???somehow??? detect when the slaves are done and copy the results.
This still doesn’t synchronize other hw resources like SPI: if two or more scripts use same bus simultaneously you’re still screwed.


I finally get what @pythoncoder meant in his `while True: pass` example: running on bare-metal with no OS, the uPy compiler would have to switch execution between multiple isolated processes otherwise without explicit “yield” statements in the `.py` files, you’d never switch execution between them … but since they’re independent why should you need to “yield”?

IM NOT SURE IF THIS PARAGRAPH IS TRUE
So if you’re running bare-metal, no OS then I think you have to run the slaves `.py` within the heap of the master by allocating slave heap from master heap.

Sent from my iPhone using Tapatalk Pro

[pythoncoder]

An excellent explanation and summary.

many people run micropython on bare-metal.

I'd go further than that and say that running on bare-metal is the primary mission of MicroPython. I would oppose any change that compromised the performance of the language in that role.