Saving and loading "state"
-
- Posts: 8
- Joined: Wed May 30, 2018 7:46 am
Saving and loading "state"
Hello,
Is there a way to load/save the current python machine state?
I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?
Thanks,
Cyrille
Is there a way to load/save the current python machine state?
I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?
Thanks,
Cyrille
Re: Saving and loading "state"
The state is composed of two things: the heap andCyrille de Brébisson wrote: ↑Fri Jun 01, 2018 11:50 amHello,
Is there a way to load/save the current python machine state?
I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?
Thanks,
Cyrille
Code: Select all
mp_state_ctx_t mp_state_ctx;
In the case that the location of the uPy heap is not the same location (or same size) as when it was saved, you have to modify some internal pointers from mp_state_ctx_t mp_state_ctx;:
Code: Select all
MP_STATE_MEM(gc_alloc_table_start), MP_STATE_MEM(gc_finaliser_table_start), MP_STATE_MEM(gc_pool_start) = (byte*)end, MP_STATE_MEM(gc_pool_end) = end
Very messy.
There should definitely be an API that abstracts all this messiness but if you want to try, see gc_init()
I created an enhancement request https://github.com/micropython/micropython/issues/3833
Simple case - size of heap doesn't change
If you're ok keeping the heap size the same after you restore, it becomes much simpler:
Save (to flash) steps
1. save the heap to flash
2. save mp_state_ctx_t mp_state_ctx to flash
Restore steps
1. allocate heap in RAM
2. restore mp_state_ctx_t mp_state_ctx from flash
3. call gc_init(void *start, void *end) with pointer ranges for the heap from (1)
mp_state_ctx is updated with correct addresses for new heap-in-RAM
and heap-in-RAM is reset
4. copy heap in flash to heap in RAM
Restores the heap as it was before
Re: Saving and loading "state"
Looks like it's only possible in very narrow circumstance.Cyrille de Brébisson wrote: ↑Fri Jun 01, 2018 11:50 amHello,
Is there a way to load/save the current python machine state?
I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?
Thanks,
Cyrille
https://github.com/micropython/micropython/issues/3833Restoring the heap to a different location is basically the same problem as supporting a precise GC. So basically I think it would only work if being restored to same location and size.
The other big issue is the state of the hardware and all of the peripherals.
And you'd probably need to save the stack as well, although that may not be required if saving the state is only allowed from certain locations.
The heap may also contain pointers back into flash, so you wouldn't be able to save the state, update some firmware and restore the state.
-
- Posts: 8
- Joined: Wed May 30, 2018 7:46 am
Re: Saving and loading "state"
Hello,
How dependent is mp_state_ctx_t to the CPU architecture?
I am not too woried about endianess as little endian seems prety dead, however, I am assuming that it is NOT 32/64 bit agnostic.
Meaning that if I save the state on a 32 bit machine, I can not reload it on a 64 bit machine...
Cyrille
How dependent is mp_state_ctx_t to the CPU architecture?
I am not too woried about endianess as little endian seems prety dead, however, I am assuming that it is NOT 32/64 bit agnostic.
Meaning that if I save the state on a 32 bit machine, I can not reload it on a 64 bit machine...
Cyrille
Re: Saving and loading "state"
This is a C-coding problem. This all depends on whether the size of data types in the code for mp_state_ctx_t vary depending on 32v64 bit; it appears the answer is yes.Cyrille de Brébisson wrote: ↑Tue Jun 26, 2018 1:12 pmHello,
How dependent is mp_state_ctx_t to the CPU architecture?
I am not too woried about endianess as little endian seems prety dead, however, I am assuming that it is NOT 32/64 bit agnostic.
Meaning that if I save the state on a 32 bit machine, I can not reload it on a 64 bit machine...
Cyrille
Code: Select all
mp_uint_t mp_optimise_value;
// size of the emergency exception buf, if it's dynamically allocated
#if MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF && MICROPY_EMERGENCY_EXCEPTION_BUF_SIZE == 0
mp_int_t mp_emergency_exception_buf_size;
#endif
#if MICROPY_PY_THREAD_GIL
// This is a global mutex used to make the VM/runtime thread-safe.
mp_thread_mutex_t gil_mutex;
#endif
} mp_state_vm_t;
Code: Select all
typedef int32_t mp_int_t; // must be pointer size
typedef uint32_t mp_uint_t; // must be pointer size
-
- Posts: 8
- Joined: Wed May 30, 2018 7:46 am
Re: Saving and loading "state"
Hello,
But I am assuming that the same problem will exist within the heap with incompatible data formats...
What I am trying to acheive is the ability to save a state file on one of my implementation of my program, transfert it to another device, and reload it there.
Since my program can run both on 32 and 64 bit systems, I have made all my file format 32/64 compatible.
The underlying question (and the answer seems to be no), is: Can I save/load state in an architecture agnostic way...
Cyrille
But I am assuming that the same problem will exist within the heap with incompatible data formats...
What I am trying to acheive is the ability to save a state file on one of my implementation of my program, transfert it to another device, and reload it there.
Since my program can run both on 32 and 64 bit systems, I have made all my file format 32/64 compatible.
The underlying question (and the answer seems to be no), is: Can I save/load state in an architecture agnostic way...
Cyrille
Re: Saving and loading "state"
That's a good point. If you're on 32-bit and you do mp_uint_t * p = m_new_obj(mp_uint_t), it will take 4 bytes in the heap.Cyrille de Brébisson wrote: ↑Thu Jun 28, 2018 9:48 amHello,
But I am assuming that the same problem will exist within the heap with incompatible data formats...
If you save the heap, reload on 64-bit then mp_uint_t will be 64-bit and *p will load 8-bytes from heap which will be incorrect behavior.
I agree. Seems like you cannot do what you want.Cyrille de Brébisson wrote: ↑Thu Jun 28, 2018 9:48 amHello,
Since my program can run both on 32 and 64 bit systems, I have made all my file format 32/64 compatible.
The underlying question (and the answer seems to be no), is: Can I save/load state in an architecture agnostic way...
Cyrille
-
- Posts: 8
- Joined: Wed May 30, 2018 7:46 am
Re: Saving and loading "state"
Hello,
> I agree. Seems like you cannot do what you want.
Thanks for confirming.
Cyrille
> I agree. Seems like you cannot do what you want.
Thanks for confirming.
Cyrille