Micropython data stored in Heap

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Chinkal
Posts: 6
Joined: Sun Oct 02, 2016 7:12 pm

Micropython data stored in Heap

Post by Chinkal » Sun Oct 02, 2016 7:15 pm

Does Micropython stores all local and global variables in Heap?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Micropython data stored in Heap

Post by dhylands » Sun Oct 02, 2016 8:45 pm

MicroPython stores most python objects in the heap. There are a number of objects which are stored in flash (like all of the pin objects).

Small ints (anything upto 31 bits) don't take up any space in the heap.

Chinkal
Posts: 6
Joined: Sun Oct 02, 2016 7:12 pm

Re: Micropython data stored in Heap

Post by Chinkal » Fri Oct 14, 2016 3:29 am

@dhylands: Thank you for clearing my doubt. I will really appreciate it if you could help me to visualize this thing.

I wants to store all local and global variable in the flash so if I exit from my current program and rerun it than it should have all the locals and global variable from my previous session.

Anything on this topic would be a great help.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Micropython data stored in Heap

Post by dhylands » Fri Oct 14, 2016 6:20 am

Objects stored in flash never change (i.e. are read-only).

locals lose their value as soon as the function they're defined in exit.

If you want to persist values, then you'll need to write the code to do that yourself, and store the data someplace (like on the filesystem, or in some battery backed RAM).

Chinkal
Posts: 6
Joined: Sun Oct 02, 2016 7:12 pm

Re: Micropython data stored in Heap

Post by Chinkal » Sun Oct 16, 2016 3:22 am

dhylands wrote:Objects stored in flash never change (i.e. are read-only).

locals lose their value as soon as the function they're defined in exit.

If you want to persist values, then you'll need to write the code to do that yourself, and store the data someplace (like on the filesystem, or in some battery backed RAM).
Correct me if I am wrong in this, I understood that I have to store the local and global dictionary that has the local and global variables information into the file system and later retrieve it from the same.
I think in micropython it stores the local and global variable information in Hash map data structure so if I could access this data structure than I can save the variable information into file system.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Micropython data stored in Heap

Post by pythoncoder » Sun Oct 16, 2016 6:52 am

It might help us understand what you're trying to achieve if you could clarify a few points. What platform are you considering, Unix or a microcontroller? Are you considering normal program termination or a program interrupted by an error condition or ctrl-C? Resuming the latter in a "new session" would require significant application-specific coding. I doubt a "generic" solution would be feasible.
Peter Hinch
Index to my micropython libraries.

Chinkal
Posts: 6
Joined: Sun Oct 02, 2016 7:12 pm

Re: Micropython data stored in Heap

Post by Chinkal » Sun Oct 16, 2016 9:15 pm

pythoncoder wrote:It might help us understand what you're trying to achieve if you could clarify a few points. What platform are you considering, Unix or a microcontroller? Are you considering normal program termination or a program interrupted by an error condition or ctrl-C? Resuming the latter in a "new session" would require significant application-specific coding. I doubt a "generic" solution would be feasible.
I am right now considering Unix platform for this implementation and a program interrupted by ctrl-c or external factor like power interruption. I know that it requires application specific coding and generic solution would not work.

I wants to start with a simple implementation and later will modify it for specific application.

Python has inbuilt function called Locals() and Globals() that displays all the locals and global variable and value. I am not sure If I could access it or modify it externally from the program.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Micropython data stored in Heap

Post by pythoncoder » Mon Oct 17, 2016 6:52 am

The ctrl-C interrupt can occur at arbitrary points in your code, so you'll need to trap the exception in the outermost scope of the program. As @dhylands has pointed out, there is little point in saving local variables: the ones that were current when the exception occurred will be out of scope in the exception handler. Further, if your exception handler saves data and returns you to the REPL there is no way to resume execution at the point where the exception occurred; this would be the only way you could use the locals if there were a way of storing them. If you can follow that rather convoluted sentence ;)

The conventional way to do persistent programming in Python is along the following lines. I would store the data which needs to be persistent in some kind of Python object, let's assume a dictionary. The objects in the dictionary could be arbitrary Python objects - integers, floats, strings, lists, sets, dicts or whatever. The application uses and maintains the data in this dict. The exception handler would use the pickle module to save the dict to a file. On startup the application would load the dict from file using pickle. (An implementation detail is that you need a way to handle the initial case where the file does not yet exist.)

You can then interrupt the program which will store the partial results. When you re-start it, it will start at the beginning using the data which was current when it was interrupted. In practice this simplistic approach usually has a drawback.

This is because you need to ensure that the dataset is consistent in such circumstances: an update could have been partially complete at the time of the interrupt. The solution to this can vary from trivial to difficult depending on the application. Often the best approach is to have the interrupt handler set a "stop" flag before allowing execution to re-commence at the point where the interrupt occurred. The application would test the flag at a suitable point, save the data and exit in an orderly fashion.

An alternative to pickle is ujson. This trades generality for cross-platform compatibility: the JSON standard does not support all possible Python data types.
Peter Hinch
Index to my micropython libraries.

Chinkal
Posts: 6
Joined: Sun Oct 02, 2016 7:12 pm

Re: Micropython data stored in Heap

Post by Chinkal » Wed Oct 19, 2016 3:04 pm

pythoncoder wrote:The ctrl-C interrupt can occur at arbitrary points in your code, so you'll need to trap the exception in the outermost scope of the program. As @dhylands has pointed out, there is little point in saving local variables: the ones that were current when the exception occurred will be out of scope in the exception handler. Further, if your exception handler saves data and returns you to the REPL there is no way to resume execution at the point where the exception occurred; this would be the only way you could use the locals if there were a way of storing them. If you can follow that rather convoluted sentence ;)

The conventional way to do persistent programming in Python is along the following lines. I would store the data which needs to be persistent in some kind of Python object, let's assume a dictionary. The objects in the dictionary could be arbitrary Python objects - integers, floats, strings, lists, sets, dicts or whatever. The application uses and maintains the data in this dict. The exception handler would use the pickle module to save the dict to a file. On startup the application would load the dict from file using pickle. (An implementation detail is that you need a way to handle the initial case where the file does not yet exist.)

You can then interrupt the program which will store the partial results. When you re-start it, it will start at the beginning using the data which was current when it was interrupted. In practice this simplistic approach usually has a drawback.

This is because you need to ensure that the dataset is consistent in such circumstances: an update could have been partially complete at the time of the interrupt. The solution to this can vary from trivial to difficult depending on the application. Often the best approach is to have the interrupt handler set a "stop" flag before allowing execution to re-commence at the point where the interrupt occurred. The application would test the flag at a suitable point, save the data and exit in an orderly fashion.

An alternative to pickle is ujson. This trades generality for cross-platform compatibility: the JSON standard does not support all possible Python data types.
Thanks for the information. I really appreciates the detailed explanation. I thinks it cleared my all doubts as before I was trying to implement this with low level c implementation but using python it will be easy to implement.

Chinkal
Posts: 6
Joined: Sun Oct 02, 2016 7:12 pm

Re: Micropython data stored in Heap

Post by Chinkal » Mon Oct 24, 2016 3:35 am

I was able to save the locals and global dict data into the file system. So my program has a function called checkpoint(), whenever called it will save the locals and global dict data and version into the file.

I also want to implement restart function so whenever the system reboots, It will check the file whether it has any previously stored data and version information. If yes then it should start executing the program after that checkpoint().

Is there any way I could implement this?

Post Reply