Saving and loading "state"

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
Cyrille de Brébisson
Posts: 8
Joined: Wed May 30, 2018 7:46 am

Saving and loading "state"

Post by Cyrille de Brébisson » Fri Jun 01, 2018 11:50 am

Hello,

Is there a way to load/save the current python machine state?

I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?

Thanks,
Cyrille

jickster
Posts: 505
Joined: Thu Sep 07, 2017 8:57 pm

Re: Saving and loading "state"

Post by jickster » Fri Jun 01, 2018 4:32 pm

Cyrille de Brébisson wrote:
Fri Jun 01, 2018 11:50 am
Hello,

Is there a way to load/save the current python machine state?

I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?

Thanks,
Cyrille
The state is composed of two things: the heap and

Code: Select all

mp_state_ctx_t mp_state_ctx;
You'd have to save/restore those two items but it's actually more complicated because the state contains pointers to the heap.
In the case that the location of the uPy heap is not the same location (or same size) as when it was saved, you have to modify some internal pointers from mp_state_ctx_t mp_state_ctx;:

Code: Select all

MP_STATE_MEM(gc_alloc_table_start), MP_STATE_MEM(gc_finaliser_table_start), MP_STATE_MEM(gc_pool_start) = (byte*)end, MP_STATE_MEM(gc_pool_end) = end


Very messy.

There should definitely be an API that abstracts all this messiness but if you want to try, see gc_init()

I created an enhancement request https://github.com/micropython/micropython/issues/3833


Simple case - size of heap doesn't change
If you're ok keeping the heap size the same after you restore, it becomes much simpler:

Save (to flash) steps
1. save the heap to flash
2. save mp_state_ctx_t mp_state_ctx to flash

Restore steps
1. allocate heap in RAM
2. restore mp_state_ctx_t mp_state_ctx from flash
3. call gc_init(void *start, void *end) with pointer ranges for the heap from (1)
mp_state_ctx is updated with correct addresses for new heap-in-RAM
and heap-in-RAM is reset
4. copy heap in flash to heap in RAM
Restores the heap as it was before

jickster
Posts: 505
Joined: Thu Sep 07, 2017 8:57 pm

Re: Saving and loading "state"

Post by jickster » Fri Jun 01, 2018 8:59 pm

Cyrille de Brébisson wrote:
Fri Jun 01, 2018 11:50 am
Hello,

Is there a way to load/save the current python machine state?

I am in a system where I have an interractive evaluation loop (repl)...
But I might need to interrupt and restart said evaluation (basically save state to stream, load state from strem).
How would I do that?

Thanks,
Cyrille
Looks like it's only possible in very narrow circumstance.
Restoring the heap to a different location is basically the same problem as supporting a precise GC. So basically I think it would only work if being restored to same location and size.

The other big issue is the state of the hardware and all of the peripherals.

And you'd probably need to save the stack as well, although that may not be required if saving the state is only allowed from certain locations.

The heap may also contain pointers back into flash, so you wouldn't be able to save the state, update some firmware and restore the state.
https://github.com/micropython/micropython/issues/3833

Cyrille de Brébisson
Posts: 8
Joined: Wed May 30, 2018 7:46 am

Re: Saving and loading "state"

Post by Cyrille de Brébisson » Tue Jun 26, 2018 1:12 pm

Hello,

How dependent is mp_state_ctx_t to the CPU architecture?
I am not too woried about endianess as little endian seems prety dead, however, I am assuming that it is NOT 32/64 bit agnostic.
Meaning that if I save the state on a 32 bit machine, I can not reload it on a 64 bit machine...

Cyrille

jickster
Posts: 505
Joined: Thu Sep 07, 2017 8:57 pm

Re: Saving and loading "state"

Post by jickster » Wed Jun 27, 2018 9:50 pm

Cyrille de Brébisson wrote:
Tue Jun 26, 2018 1:12 pm
Hello,

How dependent is mp_state_ctx_t to the CPU architecture?
I am not too woried about endianess as little endian seems prety dead, however, I am assuming that it is NOT 32/64 bit agnostic.
Meaning that if I save the state on a 32 bit machine, I can not reload it on a 64 bit machine...

Cyrille
This is a C-coding problem. This all depends on whether the size of data types in the code for mp_state_ctx_t vary depending on 32v64 bit; it appears the answer is yes.

Code: Select all

   mp_uint_t mp_optimise_value;

    // size of the emergency exception buf, if it's dynamically allocated
    #if MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF && MICROPY_EMERGENCY_EXCEPTION_BUF_SIZE == 0
    mp_int_t mp_emergency_exception_buf_size;
    #endif

    #if MICROPY_PY_THREAD_GIL
    // This is a global mutex used to make the VM/runtime thread-safe.
    mp_thread_mutex_t gil_mutex;
    #endif
} mp_state_vm_t;

Code: Select all

typedef int32_t mp_int_t; // must be pointer size
typedef uint32_t mp_uint_t; // must be pointer size
You could load it but you'd have to write your own code to deal with going from 32 to 64 bit.

Cyrille de Brébisson
Posts: 8
Joined: Wed May 30, 2018 7:46 am

Re: Saving and loading "state"

Post by Cyrille de Brébisson » Thu Jun 28, 2018 9:48 am

Hello,

But I am assuming that the same problem will exist within the heap with incompatible data formats...

What I am trying to acheive is the ability to save a state file on one of my implementation of my program, transfert it to another device, and reload it there.
Since my program can run both on 32 and 64 bit systems, I have made all my file format 32/64 compatible.
The underlying question (and the answer seems to be no), is: Can I save/load state in an architecture agnostic way...

Cyrille

jickster
Posts: 505
Joined: Thu Sep 07, 2017 8:57 pm

Re: Saving and loading "state"

Post by jickster » Thu Jun 28, 2018 2:49 pm

Cyrille de Brébisson wrote:
Thu Jun 28, 2018 9:48 am
Hello,

But I am assuming that the same problem will exist within the heap with incompatible data formats...
That's a good point. If you're on 32-bit and you do mp_uint_t * p = m_new_obj(mp_uint_t), it will take 4 bytes in the heap.
If you save the heap, reload on 64-bit then mp_uint_t will be 64-bit and *p will load 8-bytes from heap which will be incorrect behavior.

Cyrille de Brébisson wrote:
Thu Jun 28, 2018 9:48 am
Hello,

Since my program can run both on 32 and 64 bit systems, I have made all my file format 32/64 compatible.
The underlying question (and the answer seems to be no), is: Can I save/load state in an architecture agnostic way...

Cyrille
I agree. Seems like you cannot do what you want.

Cyrille de Brébisson
Posts: 8
Joined: Wed May 30, 2018 7:46 am

Re: Saving and loading "state"

Post by Cyrille de Brébisson » Fri Jun 29, 2018 6:39 am

Hello,

> I agree. Seems like you cannot do what you want.

Thanks for confirming.

Cyrille

Post Reply