Storing data into RP2040 flash
Storing data into RP2040 flash
Hi,
I would like to save variables/flags to a RP2040 non-volatile storage (flash), in case the RP2040 is restarted and thus losing the variables in RAM.
Pycom, using the ESP32 and their own libraries, has a method called nvs_set to store the data and a method called nvs_get to retrieve the data. It uses a key-value pair to store data. Ex: pycom.nvs_set(var, 0xDEAD); pycom.nvs_get(var).
I have a few thoughts of how to do it, one is using just a simple file, maybe even a json and a similar key-value to store it.
However it seems pretty rudimentar, so I would like to get suggestions of how to tackle this issue.
Thanks!
I would like to save variables/flags to a RP2040 non-volatile storage (flash), in case the RP2040 is restarted and thus losing the variables in RAM.
Pycom, using the ESP32 and their own libraries, has a method called nvs_set to store the data and a method called nvs_get to retrieve the data. It uses a key-value pair to store data. Ex: pycom.nvs_set(var, 0xDEAD); pycom.nvs_get(var).
I have a few thoughts of how to do it, one is using just a simple file, maybe even a json and a similar key-value to store it.
However it seems pretty rudimentar, so I would like to get suggestions of how to tackle this issue.
Thanks!
Re: Storing data into RP2040 flash
A text file is what I would use. Probably simple name,value pairs, one per line. That's easy to save from a dictionary, read and parse into one.
Maybe something JSON-ish if the data is complex or nested.
Maybe something JSON-ish if the data is complex or nested.
Code: Select all
def SaveIniFile(filename, dictionary):
with open(filename, "w") as f:
for key in dictionary:
f.write("{},{}\n".format(key, dictionary[key]))
Code: Select all
def LoadIniFile(filename):
dictionary = {}
with open(filename, "r") as f:
for s in f:
lst = s.strip().split(",")
dictionary[lst[0]] = lst[1]
return dictionary
Code: Select all
data = {}
data["Name"] = "Hippy"
data["Year"] = "2021"
SaveIniFile("/MyData.ini", data)
data = LoadIniFile("/MyData.ini")
print(data)
Re: Storing data into RP2040 flash
ujson is already included:
Code: Select all
.
>>> import ujson
>>> data = {}
data["Name"] = "Hippy"
data["Year"] = "2021"
>>>
>>> data
{'Name': 'Hippy', 'Year': '2021'}
>>> s=ujson.dumps(data)
>>> del data
>>> data
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'data' isn't defined
>>> data=ujson.loads(s)
>>> data
{'Name': 'Hippy', 'Year': '2021'}
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Data serialisation
The ujson library is one of four ways to do data serialisation in MicroPython. There are many other approaches, but those are the ones where I'm aware of specific MP variants.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Storing data into RP2040 flash
The NVS feature on ESP32 is basically a very lightweight filesystem (i.e. a way to implement key/value storage on a block device). MicroPython instead just provides the regular filesystems, so even though these two different approaches feel different, they're really the same thing.
ALso worth noting that on RP2040, MicroPython uses LittleFS which does automatic wear-levelling etc.
Others have commented about using json, which I think is a good idea, but the other approach I've seen is to write out the data as a .py file, which you can just load directly using "import".
Re: Storing data into RP2040 flash
Thank you for sharing your notes, that gave a nice overview of the ways I could deal with it.pythoncoder wrote: ↑Sun Jul 18, 2021 6:32 amThe ujson library is one of four ways to do data serialisation in MicroPython. There are many other approaches, but those are the ones where I'm aware of specific MP variants.
In the short term using ujson seems faster to implement. Do you see a big improvement by using ustruct or minipb?
Re: Storing data into RP2040 flash
That's good to know, simplicity is key.jimmo wrote: ↑Mon Jul 19, 2021 2:08 amThe NVS feature on ESP32 is basically a very lightweight filesystem (i.e. a way to implement key/value storage on a block device). MicroPython instead just provides the regular filesystems, so even though these two different approaches feel different, they're really the same thing.
I've thought about that, importing makes it very simple to have the variables available. However rewriting that file with formatting and to change constants values in the file might be too cumbersome.
Re: Storing data into RP2040 flash
After playing a little bit with it, it seems wasteful the way MicroPython organizes the files in memory.
Every time I rewrite the file, instead of rewriting it to the same memory location, it takes more memory.
Example:
Writing a json to a file and checking its place in memory using the rp2.Flash:
Then overwriting the file and looking the block memory:
We can see that the data was first stored in the Memory Block 1, offset 3328. When rewritten, it was stored in Memory Block 1, offset 3840, 512 bytes away from where the original memory was located. Is this expected?
On the other hand, if I want to try to read/write the memory through the flash.writeblocks method, I feel that's dangerous and can eventually corrupt the File System. Is that correct?
Every time I rewrite the file, instead of rewriting it to the same memory location, it takes more memory.
Example:
Writing a json to a file and checking its place in memory using the rp2.Flash:
Code: Select all
import ujson
data = {"id": 48879}
def write_file(data):
with open("/store.ini", "w") as f:
s = ujson.dumps(data)
f.write(s)
write_file(data)
flash = rp2.Flash()
buf = bytearray(256)
flash.readblocks(1, buf, 3328) # I was able to manually find the block in the memory
buf
>>> bytearray(b'\xf8\xfa{"id": 48879}\x10\x00\x00\x15......)
Code: Select all
#Changing the data and rewriting file
data['id'] = 0
write_file(data)
flash.readblocks(1, buf, 3840) # Shifted 512
buf
>>> bytearray(b'\xf8\xfa{"id": 0}\x10\x00\x00\x15......)
On the other hand, if I want to try to read/write the memory through the flash.writeblocks method, I feel that's dangerous and can eventually corrupt the File System. Is that correct?
Re: Storing data into RP2040 flash
This is what I was talking about earlier with the wear levelling. So the writes get distributed through the flash, but the unused blocks are reclaimed when they're needed. Basically the filesystem is avoiding ever writing (or erasing) a block unless it has to, and always choosing new blocks over recently used blocks. (But if you need to write a big file those blocks will be immediately reclaimed as needed -- they're not really "taking up space").
That's right, if you read/write blocks that are shared with the filesystem.
But if you want to implement your own storage somewhere else on the flash in a distinct region to the filesystem (and not sharing an erase page) then that's fine.
Re: Storing data into RP2040 flash
Okay, now I understand. Thanks for the clarification.jimmo wrote: ↑Tue Jul 20, 2021 12:16 am
Basically the filesystem is avoiding ever writing (or erasing) a block unless it has to, and always choosing new blocks over recently used blocks. (But if you need to write a big file those blocks will be immediately reclaimed as needed -- they're not really "taking up space").
In case I follow this, how do I make sure to put my memory block aside and make sure the filesystem is not expecting to use it?