This is my first post here! I started coding in micropython this weekend and it's been a real blast. Small disclaimer before explaining my project/issues: I'm not YET using a real ESP32, for now I'm coding in Wokwi's esp32 simulator (https://wokwi.com/projects/new/micropython-esp32). My controlers should arrive during the week.
So, the project is pretty simple, I have many biosensors collecting body data at a high rate (max around 64 acq/sec for one of the probe). I have no problem running all the async loops and collect the data. In theory, the device should be connected to a smartphone and stream data every second, then clear the data lists and repeat until turne off.
However, implementing a local mode is crucial. If the device is disconnected, it should be able to log all the data in the flash, either until it connects back to the phone or until the end of the session. However, I can't find a good, optimized and clean way to save the data locally.
I thought of/tried:
- Appending values to several csv files, however, because i would be writing an unknown quantity of data every second, I feel like it is going to be very slow and/or fragment the data on the flash too much. Also, csv are quite heavy because of the encoding.
- Using json or Tinydb but both are worse than csv files in term of encoding.
- I tried to open, write into, and close binary files every_second for every sensor and write packed lists that i would then clear. The idea is to have a timestamp in the file names that would allow me to reconstruct the data later. However, after just a few seconds, the simulation slows down and stops, with no way to know if it's because i reached the flash limit or not, which seems unlikely.
- Using the btree database to append sensor data to keys defined at the beginning instead of creating new keys for each entry. This does not seem to work properly and I have no way/don't know how to verify if the data is flushed to the stream correctly. I also believe calling the key/value pair to append my bytes to the value is a problem.
I already red a lot of things on the forum that helped me start but I couldn't really find related topics. I thought this would be the easy part of the project lmao
Thanks in advance for your advices!