Storing data into RP2040 flash

RP2040 based microcontroller boards running MicroPython.
Target audience: MicroPython users with an RP2040 boards.
This does not include conventional Linux-based Raspberry Pi boards.
User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Storing data into RP2040 flash

Post by dbrazil » Fri Jul 16, 2021 9:27 pm

Hi,

I would like to save variables/flags to a RP2040 non-volatile storage (flash), in case the RP2040 is restarted and thus losing the variables in RAM.

Pycom, using the ESP32 and their own libraries, has a method called nvs_set to store the data and a method called nvs_get to retrieve the data. It uses a key-value pair to store data. Ex: pycom.nvs_set(var, 0xDEAD); pycom.nvs_get(var).

I have a few thoughts of how to do it, one is using just a simple file, maybe even a json and a similar key-value to store it.
However it seems pretty rudimentar, so I would like to get suggestions of how to tackle this issue.

Thanks!

hippy
Posts: 130
Joined: Sat Feb 20, 2021 2:46 pm
Location: UK

Re: Storing data into RP2040 flash

Post by hippy » Sat Jul 17, 2021 8:32 pm

A text file is what I would use. Probably simple name,value pairs, one per line. That's easy to save from a dictionary, read and parse into one.

Maybe something JSON-ish if the data is complex or nested.

Code: Select all

def SaveIniFile(filename, dictionary):
    with open(filename, "w") as f:
        for key in dictionary:
            f.write("{},{}\n".format(key, dictionary[key]))

Code: Select all

def LoadIniFile(filename):
    dictionary = {}
    with open(filename, "r") as f:
        for s in f:
            lst = s.strip().split(",")
            dictionary[lst[0]] = lst[1]
    return dictionary

Code: Select all

data = {}
data["Name"] = "Hippy"
data["Year"] = "2021"

SaveIniFile("/MyData.ini", data)

data = LoadIniFile("/MyData.ini")

print(data)

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: Storing data into RP2040 flash

Post by scruss » Sat Jul 17, 2021 9:32 pm

ujson is already included:

Code: Select all

.
>>> import ujson
>>> data = {}
data["Name"] = "Hippy"
data["Year"] = "2021"
>>> 
>>> data
{'Name': 'Hippy', 'Year': '2021'}
>>> s=ujson.dumps(data)
>>> del data
>>> data
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'data' isn't defined
>>> data=ujson.loads(s)
>>> data
{'Name': 'Hippy', 'Year': '2021'}

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Data serialisation

Post by pythoncoder » Sun Jul 18, 2021 6:32 am

The ujson library is one of four ways to do data serialisation in MicroPython. There are many other approaches, but those are the ones where I'm aware of specific MP variants.
Peter Hinch
Index to my micropython libraries.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Storing data into RP2040 flash

Post by jimmo » Mon Jul 19, 2021 2:08 am

dbrazil wrote:
Fri Jul 16, 2021 9:27 pm
Pycom, using the ESP32 and their own libraries, has a method called nvs_set to store the data and a method called nvs_get to retrieve the data. It uses a key-value pair to store data. Ex: pycom.nvs_set(var, 0xDEAD); pycom.nvs_get(var).
The NVS feature on ESP32 is basically a very lightweight filesystem (i.e. a way to implement key/value storage on a block device). MicroPython instead just provides the regular filesystems, so even though these two different approaches feel different, they're really the same thing.

ALso worth noting that on RP2040, MicroPython uses LittleFS which does automatic wear-levelling etc.

Others have commented about using json, which I think is a good idea, but the other approach I've seen is to write out the data as a .py file, which you can just load directly using "import".

User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Re: Storing data into RP2040 flash

Post by dbrazil » Mon Jul 19, 2021 6:23 pm

pythoncoder wrote:
Sun Jul 18, 2021 6:32 am
The ujson library is one of four ways to do data serialisation in MicroPython. There are many other approaches, but those are the ones where I'm aware of specific MP variants.
Thank you for sharing your notes, that gave a nice overview of the ways I could deal with it.
In the short term using ujson seems faster to implement. Do you see a big improvement by using ustruct or minipb?

User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Re: Storing data into RP2040 flash

Post by dbrazil » Mon Jul 19, 2021 6:25 pm

jimmo wrote:
Mon Jul 19, 2021 2:08 am
The NVS feature on ESP32 is basically a very lightweight filesystem (i.e. a way to implement key/value storage on a block device). MicroPython instead just provides the regular filesystems, so even though these two different approaches feel different, they're really the same thing.
That's good to know, simplicity is key.
jimmo wrote:
Mon Jul 19, 2021 2:08 am
Others have commented about using json, which I think is a good idea, but the other approach I've seen is to write out the data as a .py file, which you can just load directly using "import".
I've thought about that, importing makes it very simple to have the variables available. However rewriting that file with formatting and to change constants values in the file might be too cumbersome.

User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Re: Storing data into RP2040 flash

Post by dbrazil » Mon Jul 19, 2021 10:46 pm

After playing a little bit with it, it seems wasteful the way MicroPython organizes the files in memory.
Every time I rewrite the file, instead of rewriting it to the same memory location, it takes more memory.

Example:
Writing a json to a file and checking its place in memory using the rp2.Flash:

Code: Select all

import ujson
data = {"id": 48879}

def write_file(data):
    with open("/store.ini", "w") as f:
        s = ujson.dumps(data)
        f.write(s)

write_file(data)

flash = rp2.Flash()
buf = bytearray(256)
flash.readblocks(1, buf, 3328) # I was able to manually find the block in the memory
buf
>>> bytearray(b'\xf8\xfa{"id": 48879}\x10\x00\x00\x15......)
Then overwriting the file and looking the block memory:

Code: Select all

#Changing the data and rewriting file
data['id'] = 0
write_file(data)
    
flash.readblocks(1, buf, 3840) # Shifted 512
buf
>>> bytearray(b'\xf8\xfa{"id": 0}\x10\x00\x00\x15......)

We can see that the data was first stored in the Memory Block 1, offset 3328. When rewritten, it was stored in Memory Block 1, offset 3840, 512 bytes away from where the original memory was located. Is this expected?

On the other hand, if I want to try to read/write the memory through the flash.writeblocks method, I feel that's dangerous and can eventually corrupt the File System. Is that correct?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Storing data into RP2040 flash

Post by jimmo » Tue Jul 20, 2021 12:16 am

dbrazil wrote:
Mon Jul 19, 2021 10:46 pm
We can see that the data was first stored in the Memory Block 1, offset 3328. When rewritten, it was stored in Memory Block 1, offset 3840, 512 bytes away from where the original memory was located. Is this expected?
This is what I was talking about earlier with the wear levelling. So the writes get distributed through the flash, but the unused blocks are reclaimed when they're needed. Basically the filesystem is avoiding ever writing (or erasing) a block unless it has to, and always choosing new blocks over recently used blocks. (But if you need to write a big file those blocks will be immediately reclaimed as needed -- they're not really "taking up space").
dbrazil wrote:
Mon Jul 19, 2021 10:46 pm
At the same time, if I want to try to read/write the memory through the flash.writeblocks method, I feel that it's dangerous and can eventually corrupt the File System. Is that correct?
That's right, if you read/write blocks that are shared with the filesystem.

But if you want to implement your own storage somewhere else on the flash in a distinct region to the filesystem (and not sharing an erase page) then that's fine.

User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Re: Storing data into RP2040 flash

Post by dbrazil » Tue Jul 20, 2021 12:28 am

jimmo wrote:
Tue Jul 20, 2021 12:16 am

Basically the filesystem is avoiding ever writing (or erasing) a block unless it has to, and always choosing new blocks over recently used blocks. (But if you need to write a big file those blocks will be immediately reclaimed as needed -- they're not really "taking up space").
Okay, now I understand. Thanks for the clarification.
jimmo wrote:
Tue Jul 20, 2021 12:16 am

That's right, if you read/write blocks that are shared with the filesystem.

But if you want to implement your own storage somewhere else on the flash in a distinct region to the filesystem (and not sharing an erase page) then that's fine.
In case I follow this, how do I make sure to put my memory block aside and make sure the filesystem is not expecting to use it?

Post Reply