Flash file system best practices (or: how to not destroy the flash / NVS by logging data)

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
Ricardo
Posts: 11
Joined: Mon Dec 09, 2019 9:49 pm

Flash file system best practices (or: how to not destroy the flash / NVS by logging data)

Post by Ricardo » Tue Apr 21, 2020 8:42 pm

Hi!

I'm about to write a simple humidity / temperature logger using an ESP32. Sensor data will be collected at least once per minute, and the device should run 24/7, for several days, powered by a phone charger. There is no Wi-Fi nearby most of the time, so I have to store data at least momentarily in RAM or flash.

My first obvious guess is that the less often I write to the flash file system, the better. AFAIK, if a file is not closed and the ESP32 resets for any reason (e.g., bug in code, power outage), the file becomes corrupted, so I should close it right after writing.

So I'm thinking about pre-allocating some RAM buffer to store the data, and now and them check if Wi-Fi is available. If there is Wi-Fi, I send any data from RAM and/or flash to the server and clean them up. If the RAM buffer gets full, I dump it to the file system and clean it. Rinse and repeat.

What else could / should I do to spare the flash in the device? Is there a way to know the sector / block / page size of a MicroPython device, so I can create a buffer that is a multiple of that (if that's possible)? Is there any wear-leveling (triggered either manually or automatically)?

Your input is greatly appreciated! Thanks!

User avatar
tve
Posts: 216
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)

Post by tve » Wed Apr 22, 2020 1:33 am

Lots of questions...
Use LFS not FAT, then you get wear leveling and better protection. I have not checked what the guarantees are for unclosed files, but their docs are pretty good, so you should be able to find the info (please report). I don't know what happens when you append, say, 100 bytes to a file. The flash sectors are 512 bytes, right?
One of my to-do items is to write a memory logger that uses some statically allocated buffer and recovers the contents after a reset (as long as the power doesn't go out, of course). You could use the RTC memory for that too from python. Sadly there is no way to power just the RTC memory from a coin cell...
Hopefully someone else who has already investigated all this can chime in!

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)

Post by jimmo » Wed Apr 22, 2020 7:22 am

Definitely recommend to use LittleFS (which will be the default on ESP32 in the next release of MicroPython).
tve wrote:
Wed Apr 22, 2020 1:33 am
I don't know what happens when you append, say, 100 bytes to a file. The flash sectors are 512 bytes, right?
It depends on the flash part, but generally they're larger than this (and this is why we have to have a big buffer for the FAT filesystem so we can map fat's tiny 512B sectors to the (e.g.) 4kiB flash pages).

However, (and caveat: it's been a little while since I looked into this in detail) but one part of the design of littlefs is that when you write, you only set the "new" bits. i.e. you do a once-off erase of the block, then all subsequent writes only affect the bits that need to change. And in general the design is append-only. There's a lot more complexity, but in general, littlefs solves wear in two ways:
- It spreads the blocks across the device
- Erases happen infrequently.

(Also flash parts these days have erase/write cycle limits measured in the hundreds of thousands or more, so in many cases it's just not an issue).
Ricardo wrote:
Tue Apr 21, 2020 8:42 pm
if a file is not closed
Yep, you want to close() or flush() the file. Depending on your app, keeping the file open and flushing it might be more convenient, rather than continuously opening in append mode.

Anything else involving buffering in RAM is likely to be more complicated and lead to more possibility of data loss.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)

Post by pythoncoder » Wed Apr 22, 2020 8:15 am

Another option is to use an external device like an Adafruit FRAM module. See this doc for options and drivers.
Peter Hinch
Index to my micropython libraries.

Ricardo
Posts: 11
Joined: Mon Dec 09, 2019 9:49 pm

Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)

Post by Ricardo » Tue May 12, 2020 5:16 pm

Thank you all for your information and tips!

Post Reply