Hi!
I'm about to write a simple humidity / temperature logger using an ESP32. Sensor data will be collected at least once per minute, and the device should run 24/7, for several days, powered by a phone charger. There is no Wi-Fi nearby most of the time, so I have to store data at least momentarily in RAM or flash.
My first obvious guess is that the less often I write to the flash file system, the better. AFAIK, if a file is not closed and the ESP32 resets for any reason (e.g., bug in code, power outage), the file becomes corrupted, so I should close it right after writing.
So I'm thinking about pre-allocating some RAM buffer to store the data, and now and them check if Wi-Fi is available. If there is Wi-Fi, I send any data from RAM and/or flash to the server and clean them up. If the RAM buffer gets full, I dump it to the file system and clean it. Rinse and repeat.
What else could / should I do to spare the flash in the device? Is there a way to know the sector / block / page size of a MicroPython device, so I can create a buffer that is a multiple of that (if that's possible)? Is there any wear-leveling (triggered either manually or automatically)?
Your input is greatly appreciated! Thanks!
Flash file system best practices (or: how to not destroy the flash / NVS by logging data)
Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)
Lots of questions...
Use LFS not FAT, then you get wear leveling and better protection. I have not checked what the guarantees are for unclosed files, but their docs are pretty good, so you should be able to find the info (please report). I don't know what happens when you append, say, 100 bytes to a file. The flash sectors are 512 bytes, right?
One of my to-do items is to write a memory logger that uses some statically allocated buffer and recovers the contents after a reset (as long as the power doesn't go out, of course). You could use the RTC memory for that too from python. Sadly there is no way to power just the RTC memory from a coin cell...
Hopefully someone else who has already investigated all this can chime in!
Use LFS not FAT, then you get wear leveling and better protection. I have not checked what the guarantees are for unclosed files, but their docs are pretty good, so you should be able to find the info (please report). I don't know what happens when you append, say, 100 bytes to a file. The flash sectors are 512 bytes, right?
One of my to-do items is to write a memory logger that uses some statically allocated buffer and recovers the contents after a reset (as long as the power doesn't go out, of course). You could use the RTC memory for that too from python. Sadly there is no way to power just the RTC memory from a coin cell...
Hopefully someone else who has already investigated all this can chime in!
Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)
Definitely recommend to use LittleFS (which will be the default on ESP32 in the next release of MicroPython).
However, (and caveat: it's been a little while since I looked into this in detail) but one part of the design of littlefs is that when you write, you only set the "new" bits. i.e. you do a once-off erase of the block, then all subsequent writes only affect the bits that need to change. And in general the design is append-only. There's a lot more complexity, but in general, littlefs solves wear in two ways:
- It spreads the blocks across the device
- Erases happen infrequently.
(Also flash parts these days have erase/write cycle limits measured in the hundreds of thousands or more, so in many cases it's just not an issue).
Anything else involving buffering in RAM is likely to be more complicated and lead to more possibility of data loss.
It depends on the flash part, but generally they're larger than this (and this is why we have to have a big buffer for the FAT filesystem so we can map fat's tiny 512B sectors to the (e.g.) 4kiB flash pages).
However, (and caveat: it's been a little while since I looked into this in detail) but one part of the design of littlefs is that when you write, you only set the "new" bits. i.e. you do a once-off erase of the block, then all subsequent writes only affect the bits that need to change. And in general the design is append-only. There's a lot more complexity, but in general, littlefs solves wear in two ways:
- It spreads the blocks across the device
- Erases happen infrequently.
(Also flash parts these days have erase/write cycle limits measured in the hundreds of thousands or more, so in many cases it's just not an issue).
Yep, you want to close() or flush() the file. Depending on your app, keeping the file open and flushing it might be more convenient, rather than continuously opening in append mode.
Anything else involving buffering in RAM is likely to be more complicated and lead to more possibility of data loss.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)
Another option is to use an external device like an Adafruit FRAM module. See this doc for options and drivers.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Flash file system best practices (or: how to not destroy the flash / NVS by logging data)
Thank you all for your information and tips!