Real-life application using micropython

[ajie_dirgantara]

Hi all,

We are planning to use micropython for our real-life consumer-end project. Some doubt arise when we are talking about micropython are, our project require power/time/storage--constrained requirement, but still using the usual files system access.

1. we need to store critical data (max 64 byte) to a file, to non volatile memory (in this case internal flash) in maximum 200ms window period after power loss. Is this possible? what I saw from the file system is that it works in a multiple 512byte block, which might be too long to be stored in 200ms period.

2. To avoid data corruption, we must prevent any read/write to file, or if the power loss event is concurrent with the file access process, somehow data corruption should be avoided. Can we make sure a file in file system is already update/not updated at all in an atomic way if we are using file system-or in this case micropython file system?

Rgds,

[dhylands]

There's a 4K block of SRAM which can be battery backed up using a small coin cell.

Perhaps you could store your data there rather that on the file system?

Even though the filesystem uses 512 byte blocks, the underlying flash uses either 16K or 64K blocks. So writing a 512 byte block goes through the sequence:
1 - read the 16K or 64K flash sector containing the block into RAM
2 - make modifications to the 16k/64k sector
3 - Erase the flash sector
4 - write the flash sector back to flash.

[ajie_dirgantara]

There's a 4K block of SRAM which can be battery backed up using a small coin cell.

Perhaps you could store your data there rather that on the file system?

Even though the filesystem uses 512 byte blocks, the underlying flash uses either 16K or 64K blocks. So writing a 512 byte block goes through the sequence:
1 - read the 16K or 64K flash sector containing the block into RAM
2 - make modifications to the 16k/64k sector
3 - Erase the flash sector
4 - write the flash sector back to flash.

unfortunately SRAM won't be an options because the battery is limited only for rtc.

ok, actually I've wrote block device driver for 2MB spi nor flash m25pe16, and it has 256byte per page. Still, this must be wrote two times in sequence to be able to supported by file system.

for critical data (my first question), we might use eeprom and i2c. But for my 2nd questions, how do we guarantee file integrity in such system?

Thanks,

[markxr]

200ms sounds like quite a long time to write a small file, I think it should be doable.

Robustness is a bigger problem. Writing a file to the (DOS) filesystem needs at least to update the directory entry, the fat and the file itself. This is three writes which might not all succeed if the power fails half way.

Moreover, the way the flash works is by erase/write, so if power fails at precisely the wrong time, I suppose we end up with an empty block instead of the previous contents.

Probably if we want very robust operation, the dos fat filesystem is not an option. My guess is that it's possible to bypass the dos fs, or at least use it in a way less prone to corruption.

There is a lower level API for reading / erasing / writing individual blocks that you can use. If you are prepared to manage the way records are stored, then use that instead of the DOS FAT.

[torwag]

I do not understand the time limitation.
It is by all means an electronic problem and not a micropython problem. You could build your device to last years on battery in case power fails. Or you make sure you can still operate for X ms whereas X is the time you need to run your clean-shutdown interrupt routine. All you need to do now is to enable an IRQ for power-loss, which might or might not be already implemented in the uC of your choice. If not it is rather trivial to add an external circuitry which triggers the IRQ as soon as the power supply fails.
Again this is more an electrical rather then a programming problem.

[dhylands]

unfortunately SRAM won't be an options because the battery is limited only for rtc.

There is 4K of SRAM which is backed up by the same battery which is used for the RTC. It's referred to as "backup SRAM" in the datasheet.

[ajie_dirgantara]

200ms sounds like quite a long time to write a small file, I think it should be doable.

Robustness is a bigger problem. Writing a file to the (DOS) filesystem needs at least to update the directory entry, the fat and the file itself. This is three writes which might not all succeed if the power fails half way.

Moreover, the way the flash works is by erase/write, so if power fails at precisely the wrong time, I suppose we end up with an empty block instead of the previous contents.

Probably if we want very robust operation, the dos fat filesystem is not an option. My guess is that it's possible to bypass the dos fs, or at least use it in a way less prone to corruption.

There is a lower level API for reading / erasing / writing individual blocks that you can use. If you are prepared to manage the way records are stored, then use that instead of the DOS FAT.

agree. If we choose not to use DOS FAT for data storage, we had a full blown development environment to develop our product in a traditional way (KEIL-ARM, RTX RTOS + it's library), but another requirement is Over-The-Air firmware/application update, which I think it will be major advantage if we are using scripted firmware logic, in this case, micropython.

Well, I guess we won't use a DOS FAT for any data storage, as we can't really depend on it robustness. As for firmware files (python script), I think it doable because it only written/rewrite when there are updates, and mainly read only.

[ajie_dirgantara]

I do not understand the time limitation.
It is by all means an electronic problem and not a micropython problem. You could build your device to last years on battery in case power fails. Or you make sure you can still operate for X ms whereas X is the time you need to run your clean-shutdown interrupt routine. All you need to do now is to enable an IRQ for power-loss, which might or might not be already implemented in the uC of your choice. If not it is rather trivial to add an external circuitry which triggers the IRQ as soon as the power supply fails.
Again this is more an electrical rather then a programming problem.

here, time limitation is about how long do the file system operation is processed. What we had is 200ms window period to do the clean shutdown. More than that, we need more "electrical" effort which mean cost more money, and we can't afford that as our product will be produce in large qty, so cost must be kept at minimum. Again, we might consider other options than the file system for data storage.

[ajie_dirgantara]

unfortunately SRAM won't be an options because the battery is limited only for rtc.

There is 4K of SRAM which is backed up by the same battery which is used for the RTC. It's referred to as "backup SRAM" in the datasheet.

Currently we are using STM32L476 chip. I don't think it has SRAM is it?. gotta check the datasheet first.

[Roberthh]

According to the data sheet, there is no SRAM, but 32x32 bit = 128 bytes backup register space in the RTC unit, which is kept by the battery and not cleared at reset. This data is cleared at an tamper event.