Process isolation

[jockm]

Besides that and everywhere MP_STATE* macros are used, you also have to add code to vm.c to switch the execution between the applets.

I see exactly no need for this. Please explain why you think it would be required.

If you have multiple VMs, you want them to execute simultaneously from the perspective of each .py.

If you don’t have an OS to do that, it must be done in the uPy code somehow: vm.c must switch between applets otherwise one would hog 100% CPU until (or if) it ever finishes.


Sent from my iPhone using Tapatalk Pro

Well you may not want them executing simultaneously. You may simply want them idle until an event occurs and then they are called to respond to them. This is my use case for example. I think you are perceiving a requirement where one doesn't exist.

If I spawn more than one DukTape (mJS, eLua, etc) interpreter there is no expectation that they run independently. There if you want parallel execution then you spin them up in seperate threads — if we go back to the beginning of this thread I didn't have a problem with different threads, just not different heavyweight processes.

If uPy is to have multiple instances I suggest the most basic implementation until you can actually prove the need. I mean that is why uPy doesn't have multiple instances now, right?

[jockm]

The bigger reason why this’ll get rejected is the still unjustified requirement for spawning to occur at the C-level. This means more infrastructure will need to be added to manage the applets with one C-API.

Why is this unjustified? If it can be done at C-Level then you can make a module that does it

[jickster]

The bigger reason why this’ll get rejected is the still unjustified requirement for spawning to occur at the C-level. This means more infrastructure will need to be added to manage the applets with one C-API.

Why is this unjustified?

Just because it can be done doesn't mean you've provided a reason for it!

What is the reason why spawning a .py from within a .py is not sufficient for your needs?

The bigger reason why this’ll get rejected is the still unjustified requirement for spawning to occur at the C-level. This means more infrastructure will need to be added to manage the applets with one C-API.

If it can be done at C-Level then you can make a module that does it

I think there's a misunderstanding. We may have to write some additional C-code to implement this which would be exposed to .py code; I'm not against that.

I thought you are against doing the entire "launch the new .py in the execution of an existing .py"

[jickster]

Damien's excellent plan for scoped allocations was to allocate on the C stack. Would that be compatible with context switching?

The Py-stack size is actually pre-computed and allocated on the py-heap (see fun_bc_call()) so allocating py-heap on C-stack is irrelevant.

After some thought, I don't think it makes sense at all to implement context-switching within micropython.
It's obvious that all we're doing is compensating for lack of an OS and even OP has stated he's not against using a lightweight OS.

If you want to execute two .py in parallel there are options, though not exactly implemented all the way:
(1) scoped-heap - spawn a .py within a .py but still have separate heaps
(2) use an OS + change MP_STATE*() macros to switch mp_state_ctx_t instance - you'll get complete memory isolation since you can instantiate separate heaps and mp_state_ctx_t instances

Both of these must be implemented but (2) is definitely easier.

[jickster]

I see exactly no need for this. Please explain why you think it would be required.

If you have multiple VMs, you want them to execute simultaneously from the perspective of each .py.

If you don’t have an OS to do that, it must be done in the uPy code somehow: vm.c must switch between applets otherwise one would hog 100% CPU until (or if) it ever finishes.


Sent from my iPhone using Tapatalk Pro

You may simply want them idle until an event occurs and then they are called to respond to them.

And THAT line is confirmation that you need at least a lightweight OS with threads.
Only an OS can provide "events".

Otherwise, a bunch of code has to be added to uPy to basically create an OS which implements threads.

[jockm]

The bigger reason why this’ll get rejected is the still unjustified requirement for spawning to occur at the C-level. This means more infrastructure will need to be added to manage the applets with one C-API.

Why is this unjustified?

Just because it can be done doesn't mean you've provided a reason for it!

What is the reason why spawning a .py from within a .py is not sufficient for your needs?

Because my entire codebase isn't written in python? I have a C++ application that is using uPy as an the way user applets are written.

I want to do exactly what I would do with any other extension language (including CPython), spin up an interpreter from my application when a user applet is invoked.

In my case applets won't have access to anything other than the Python standard library and an API supplied to the applet. No GPIO, UART, SPI, etc.

I suspect you will find this will be a common use case of multiple interpreters

The bigger reason why this’ll get rejected is the still unjustified requirement for spawning to occur at the C-level. This means more infrastructure will need to be added to manage the applets with one C-API.

If it can be done at C-Level then you can make a module that does it

I think there's a misunderstanding. We may have to write some additional C-code to implement this which would be exposed to .py code; I'm not against that.

I thought you are against doing the entire "launch the new .py in the execution of an existing .py"

I am not against it, but it can't be the only way to do it. There are use cases that don't need it and shouldn't be forced to do it that way.

[jickster]

Since you're OK with having lightweight OS, the changes you're looking to be made are pretty trivial.
The state of an interpreter aka VM is contained in two things:

1. mp_state_ctx_t mp_state_ctx;
2. the heap passed to gc_init()

The following changes need to be made:
The MP_STATE*() macros all reference a global mp_state_ctx_t mp_state_ctx defined in mpstate.c
1. You need to change that variable to be a mp_state_ctx_t * mp_state_ctx . You'd assign that pointer manually to whichever interpreter you want to be executed and you need to create the mp_state_ctx_t manually.
2. Change the MP_STATE*() macros to use pointers:

Code: Select all

#define MP_STATE_VM(x) (mp_state_ctx.vm.x)

to

Code: Select all

#define MP_STATE_VM(x) (mp_state_ctx->vm.x)

If you look in gc_alloc(), you'll see it's littered with MP_STATE_MEM() references which means it correctly only references the heap bound to this interpreter so once you've made the above (2) changes, you should be good to go.

I am not against it, but it can't be the only way to do it. There are use cases that don't need it and shouldn't be forced to do it that way.

I was pushing this before because I thought you were against any kind of OS.
But if you're ok with a lightweight thread-OS - which means the C/C++ code all shares the same memory space - then this is code to launch a interpreter WITH THE CURRENT UN-MODIFIED CODE:

Code: Select all

mp_stack_ctrl_init();
mp_stack_set_limit(stack_size);
gc_init(ptr_heap, ptr_heap + heap_size);
mp_init();

mp_parse_input_kind_t parse_type = MP_PARSE_SINGLE_INPUT;
if (repl == true)
{
	parse_type = MP_PARSE_SINGLE_INPUT;
}
else
{
	parse_type = MP_PARSE_FILE_INPUT;
}

mp_lexer_t * lex = mp_lexer_new_from_str_len(0, ptr_code, (size_t)code_length, (size_t)0);
mp_parse_tree_t parse_tree = mp_parse(lex, parse_type);
mp_obj_t module_func = mp_compile(&parse_tree, lex->source_name, MP_EMIT_OPT_NONE, repl);
mp_call_function_0(module_func);

Once you make those two changes from above, before you call any of those functions you have to
(1) allocate an mp_state_ctx_t
(2) set the global mp_state_ctx_t pointer to that instance

[jockm]

Since you're OK with having lightweight OS, the changes you're looking to be made are pretty trivial.
The state of an interpreter aka VM is contained in two things:

1. mp_state_ctx_t mp_state_ctx;
2. the heap passed to gc_init()

The following changes need to be made:
The MP_STATE*() macros all reference a global mp_state_ctx_t mp_state_ctx defined in mpstate.c
1. You need to change that variable to be a mp_state_ctx_t * mp_state_ctx . You'd assign that pointer manually to whichever interpreter you want to be executed and you need to create the mp_state_ctx_t manually.
2. Change the MP_STATE*() macros to use pointers:

Code: Select all

#define MP_STATE_VM(x) (mp_state_ctx.vm.x)

to

Code: Select all

#define MP_STATE_VM(x) (mp_state_ctx->vm.x)

If you look in gc_alloc(), you'll see it's littered with MP_STATE_MEM() references which means it correctly only references the heap bound to this interpreter so once you've made the above (2) changes, you should be good to go.

I will take a look at that

[jickster]

I created an issue in Github

https://github.com/micropython/micropython/issues/4080

Process Isolation: enable multiple Virtual Machines in same memory space

[jickster]

@jockm I know you care about this but this does fit under the "Process Isolation" theme if you want to launch a .py within a .py and have them be isolated.

https://github.com/micropython/micropython/issues/4081

Scoped allocation