Storing data into RP2040 flash

RP2040 based microcontroller boards running MicroPython.
Target audience: MicroPython users with an RP2040 boards.
This does not include conventional Linux-based Raspberry Pi boards.
User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Storing data into RP2040 flash

Post by jimmo » Tue Jul 20, 2021 1:09 am

dbrazil wrote:
Tue Jul 20, 2021 12:28 am
In case I follow this, how do I make sure to put my memory block aside and make sure the filesystem is not expecting to use it?
The way the filesystem works is:

- In ports/rp2/boards/PICO/mpconfigboard.h we define MICROPY_HW_FLASH_STORAGE_BYTES which is how many bytes of flash to reserve for the "user accessible flash" (i.e. the rp2.Flash object). This is always at the end of the flash, i.e. offset from PICO_FLASH_SIZE_BYTES (which is 2MiB).
- In ports/rp2/rp2_flash.c we define the rp2.Flash object using those two macros.
- In ports/rp2/modules/_boot.py we construct a LittleFS filesystem using the entire rp2.Flash object. The flash object (a block device) provides an ioctl that returns the size, and the filesystem uses that to figure out how much space it has available.

So there are a couple of ways to do this:
- Extend rp2_flash.c to make it work like pyb.Flash (where rather than returning a singleton, you can construct it with any arbitrary offset and length). Then in _boot.py you could just create the filesystem with a smaller Flash object.
- In _boot.py, you could define a class that wraps rp2.Flash and provides the block device interface, mostly passing straight through to rp2.Flash but providing a modified offset and length. Then pass an instance of that to the filesystem instead. (Effectively you're defining a "partition" class).

I like the second approach more I think, and not just because it doesn't involve writing C :)

The other option would be to just make MICROPY_HW_FLASH_STORAGE_BYTES smaller, and you'd know that there was some reserved space. You could modify rp2_flash.c to offset MICROPY_HW_FLASH_STORAGE_BASE by the amount you shrunk it by, which will force your reserved region to be at the end of flash (and a known size).

More info at https://docs.micropython.org/en/latest/ ... ystem.html

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Storing data into RP2040 flash

Post by pythoncoder » Tue Jul 20, 2021 4:53 am

dbrazil wrote:
Mon Jul 19, 2021 6:23 pm
...
In the short term using ujson seems faster to implement. Do you see a big improvement by using ustruct or minipb?
I use ujson a lot, because it is so easy. However ustruct - at the opposite end of the ease/efficiency tradeoff - should be understood.

The efficiency gain with the fixed length binary records of ustruct can be dramatic when storing data in a file because you can use random access. Accessing an 8 byte record from the middle of an N MB file can be fast as the filesystem only needs to retrieve one block. With ujson you have to read the entire file. Consider reading glyphs from a font file. With ujson you'd have to store the entire set in RAM whereas with ustruct you only need to store the current glyph. In this instance storing them in RAM would be much faster (once the initial load was complete). But you might not be able to spare the RAM.

Another technology worth considering for file storage is btree - but that solves a different problem than object serialisation.
Peter Hinch
Index to my micropython libraries.

User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Re: Storing data into RP2040 flash

Post by dbrazil » Tue Jul 20, 2021 6:47 pm

jimmo wrote:
Tue Jul 20, 2021 1:09 am
- In ports/rp2/boards/PICO/mpconfigboard.h we define MICROPY_HW_FLASH_STORAGE_BYTES which is how many bytes of flash to reserve for the "user accessible flash" (i.e. the rp2.Flash object). This is always at the end of the flash, i.e. offset from PICO_FLASH_SIZE_BYTES (which is 2MiB).
The board I'm using is the Adafruit Feather RP2040, and because it has the wrong flash size definition, I had to change the mpconfigboard.h, changing MICROPY_HW_FLASH_STORAGE_BYTES and redefining PICO_FLASH_SIZE_BYTES.
However, I can't find where the PICO_FLASH_SIZE_BYTES is normally defined.

So in terms of flash allocation, it first allocates the MICROPY_HW_FLASH_STORAGE_BASE (where the MicroPython itself is located) in the beginning of the flash and then it allocates the MICROPY_HW_FLASH_STORAGE_BYTES. Is that accurate?
If this is correct, I don't understand why when I was writing my file, it was using the first blocks (block 0 and 1). Maybe these first blocks are already offset from the MICROPY_HW_FLASH_STORAGE_BASE? So I wouldn't even have access to where the MicroPython internals are located.

I like both the second approach and the extra one (shrinking the MICROPY_HW_FLASH_STORAGE_BASE). Thanks for your thoughts.

User avatar
dbrazil
Posts: 14
Joined: Mon Apr 26, 2021 5:52 pm

Re: Storing data into RP2040 flash

Post by dbrazil » Tue Jul 20, 2021 6:51 pm

pythoncoder wrote:
Tue Jul 20, 2021 4:53 am
The efficiency gain with the fixed length binary records of ustruct can be dramatic when storing data in a file because you can use random access. Accessing an 8 byte record from the middle of an N MB file can be fast as the filesystem only needs to retrieve one block.

Another technology worth considering for file storage is btree - but that solves a different problem than object serialisation.
That makes sense. In this case, the size of the file I will be using would be very small. However, in the future, when storing real data I will consider using ustruct.
In the mean time, I will also take a look at btree, thanks for the heads up.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Storing data into RP2040 flash

Post by jimmo » Wed Jul 21, 2021 1:16 am

dbrazil wrote:
Tue Jul 20, 2021 6:47 pm
The board I'm using is the Adafruit Feather RP2040, and because it has the wrong flash size definition, I had to change the mpconfigboard.h, changing MICROPY_HW_FLASH_STORAGE_BYTES and redefining PICO_FLASH_SIZE_BYTES.
However, I can't find where the PICO_FLASH_SIZE_BYTES is normally defined.
Ahh I missed that thread. There's a different board definition for ADAFRUIT_FEATHER_RP2040 for exactly this reason.

I don't understand why this isn't doing the right thing... PICO_FLASH_SIZE_BYTES comes from pico-sdk/src/boards/include/boards/<name>.h (and <name> comes from the BOARD= you specify on the micropython build).

The only part that's a bit weird is that we set MICROPY_HW_FLASH_STORAGE_BYTES to 3072kiB for the feather (when like you say, it could be 7.5kiB).
dbrazil wrote:
Tue Jul 20, 2021 6:47 pm
So in terms of flash allocation, it first allocates the MICROPY_HW_FLASH_STORAGE_BASE (where the MicroPython itself is located) in the beginning of the flash and then it allocates the MICROPY_HW_FLASH_STORAGE_BYTES. Is that accurate?
If this is correct, I don't understand why when I was writing my file, it was using the first blocks (block 0 and 1). Maybe these first blocks are already offset from the MICROPY_HW_FLASH_STORAGE_BASE? So I wouldn't even have access to where the MicroPython internals are located.
Yep, that's right. You can't access the part of flash used to store the firmware from the rp2.Flash object.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Storing data into RP2040 flash

Post by Roberthh » Wed Jul 21, 2021 6:00 am

According to Adafruit their rp2040 device come with 8 MB flash. The initial plans were for 4 MB, but they changed it due to better allocation for 8 MB chips. There was a note in the forum about that fact.

Post Reply