internal flash write speed with packed data

[cederom]

Hello world

I am creating a logger on ESP32 that needs to write measurements to internal flash memory as fast as possible (1ms would be enough but 10ms in worst case).

Each measurement will be a set of floating point values with an integer timestamp.

What would be the most efficient and quick way to store data?

Should I use Partition.writeblocks(block_num, buf, offset) or btree object?

I am planning to struct.pack_into(fmt, buffer, offset, v1, v2, ...) in order to conserve flash space.

There is a LittleFS on / already so writeblocks would probably corrupt existing data/code? On the other hand btree intuitively seems to be slower than binary write with writeblocks at selected offset but that would allow easy write/read bytes addressed by a timestamp.

Any hints welcome

[jimmo]

> There is a LittleFS on / already so writeblocks would probably corrupt existing data/code?

If you go the writeblocks approach then you should create a separate partition away from your littlefs filesystem.

My instinct is that you should just append your packed bytes to a regular file on the LittleFS filesystem using plain-old open() and write() from python, and the LittleFS filesystem will do everything you need. It's optimised to work with partial block updates, and the filesystem knows exactly what bytes you're writing and there will be the minimum overhead.

If it's important that you flush to the disk after every write (i.e via f.flush()), then that will come with a performance cost, but if it's OK to lose the last few writes on power outage then just using append directly will be very efficient.

Using a btree is also a good option if you want random access (e.g. by timestamp) to your data.

I would caution against any approach using writeblocks directly, as you're effectively trying to beat LittleFS at its own game.

> write measurements to internal flash memory as fast as possible

Just to clarify (but not really relevant here), ESP32 has no internal flash, it's external spiflash.