I'd be interested to know if people have experienced file system corruption as a result of simply executing some code. This is as opposed to a system crash, or rebooting it when it is anyway doing some file system work.
The latter case is to be expected - many OS can damage the file system if they are interrupted at critical moments. The ESP chips do sometimes restart for whatever reason and in my experience, this is when I get file system corruption.
But I am interested specifically whether people have discovered any particular cases when the file system gets trashed NOT for reasons of resets or system crashes...
As an experiment I've lashed up an ESP chip with some code which keeps writing/reading/deleting a variable length file (with varying file names) to RAM.
It then checks to see if the file system is still intact and sends data to thingspeak. It's doing one cycle every ~1.5 seconds. It sends a count of the total number of file operations and the amount of free memory available. So far it has passed the 5000 operations mark and it is still climbing without apparent damage.
I don't manually invoke the garbage collection...
If you are interested, you can see it at http://thingspeak.com/channels/139274
I'll leave it running until it falls over!
Robustness of file system?
Re: Robustness of file system?
It's FAT on NOR so will get broken (or rather the block(s) containing the file allocation table) after some thousands of any write operation. So you now have a small number of broken blocks. You can live with it by creating a special build that skips those placing the filesystem a few blocks higher.
As you can see from this forum filesystem breakages are common.
Desperately needs a filesystem that is designed for NOR that would be several orders of magnitude better. I have started to look at both Micropython itself and SPIFFS this evening. Right now both baffle me (though it's more about build and configuration than the workings).
As you can see from this forum filesystem breakages are common.
Desperately needs a filesystem that is designed for NOR that would be several orders of magnitude better. I have started to look at both Micropython itself and SPIFFS this evening. Right now both baffle me (though it's more about build and configuration than the workings).
Re: Robustness of file system?
I think most of the filesystem breakage reported on this forum so far has been traced to either bad power (the flash chips seem to be more sensitive to voltage fluctuations than the microcontroller), leftover data from the previous content, or bad command used for flashing.
I think there was one person who broke their esp8266 by reconnecting to networks a lot of times (the network config gets saved each time), but that's not the filesystem.
I think there was one person who broke their esp8266 by reconnecting to networks a lot of times (the network config gets saved each time), but that's not the filesystem.
-
pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Robustness of file system?
Nevertheless a Flash friendly filesystem to replace FAT on all MicroPython platforms would be awesome - if it keeps things 'micro' and supports the block protocol.
FAT sucks. It thrashes the low blocks mercilessly and (as far as I know) has no means of detecting and mapping out defective blocks.
@jms Bear in mind that drivers exist for various devices which use the block protocol as the interface. Conforming to the block protocol enables the drivers to automatically be compatible with MicroPython filesystem operations. Devices include other types of flash devices and ferroelectric RAM chips (which are small in capacity). There is the SD card on the Pyboard and the official SD card driver for external SD card devices. Any change to the filesystem needs to accommodate this fact - are you envisioning a mechanism where the filesystem is selectable?
SD cards must clearly support a filesystem accessible from PC's which seems to imply FAT (do SD cards have an internal flash management system?).
[edit]After a quick look at SPIFFS I think the lack of support for directories could be a deal breaker in terms of getting a driver incorporated into MicroPython mainstream. Python packages require directories.
FAT sucks. It thrashes the low blocks mercilessly and (as far as I know) has no means of detecting and mapping out defective blocks.
@jms Bear in mind that drivers exist for various devices which use the block protocol as the interface. Conforming to the block protocol enables the drivers to automatically be compatible with MicroPython filesystem operations. Devices include other types of flash devices and ferroelectric RAM chips (which are small in capacity). There is the SD card on the Pyboard and the official SD card driver for external SD card devices. Any change to the filesystem needs to accommodate this fact - are you envisioning a mechanism where the filesystem is selectable?
SD cards must clearly support a filesystem accessible from PC's which seems to imply FAT (do SD cards have an internal flash management system?).
[edit]After a quick look at SPIFFS I think the lack of support for directories could be a deal breaker in terms of getting a driver incorporated into MicroPython mainstream. Python packages require directories.
Last edited by pythoncoder on Tue Jul 26, 2016 6:55 am, edited 1 time in total.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Robustness of file system?
SD cards have their own wear levelling etc. and are well suited being a block level device.
Proper flash file systems (JFFS2, YAFFS etc) are not really suited to being block level devices. You can get a description of JFFS (the predecessor of JFFS2) which will give you some insight to how a proper flash file system works.
Proper flash file systems (JFFS2, YAFFS etc) are not really suited to being block level devices. You can get a description of JFFS (the predecessor of JFFS2) which will give you some insight to how a proper flash file system works.
Re: Robustness of file system?
It's always nice to have more options, but there is only one way to get more features and support for more stuff -- someone has to write it. For that to happen, we need people who are both interested in using and extending MicroPython and able and willing to learn how to do it. In other words, we need to grow the community.
There are several ways to help with that. Of course the most direct way is to just write the code yourself, and it's a great way to learn -- but it doesn't scale. You can show MicroPython to people you meet at maker spaces or conferences, and show them how to start with MicroPython -- possibly on the ESP8266, because it doesn't require a big investment. You can also encourage people to start using it by organizing events and by building awesome things with it and showing them off. Finally -- and I think it's the most productive way currently -- you can make it easier to get started by writing tutorials and improving the documentation.
There are several ways to help with that. Of course the most direct way is to just write the code yourself, and it's a great way to learn -- but it doesn't scale. You can show MicroPython to people you meet at maker spaces or conferences, and show them how to start with MicroPython -- possibly on the ESP8266, because it doesn't require a big investment. You can also encourage people to start using it by organizing events and by building awesome things with it and showing them off. Finally -- and I think it's the most productive way currently -- you can make it easier to get started by writing tutorials and improving the documentation.
If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
-- Antoine de Saint-Exupery
Re: Robustness of file system?
Does this mean that FAT format is used on STM32F4 Micropython platforms, including 405 pyboards?Nevertheless a Flash friendly filesystem to replace FAT on all MicroPython platforms would be awesome - if it keeps things 'micro' and supports the block protocol.
Re: Robustness of file system?
YesEasyRider wrote:Does this mean that FAT format is used on STM32F4 Micropython platforms, including 405 pyboards?
Re: Robustness of file system?
To clarify few obvious (or maybe not so obvious) things:
1. FAT is the most used and well-known filesystem in the world. It's so well known that everyone and their grandma knows how "bad" it is.
2. All other filesystems, much less is known about them. Much few people known how "bad" those are. A cautious person would expect the worst then.
3. "FatFs" library, which MicroPython uses, is the most well-known liberally licensed open-source FAT FS. It is used by millions (likely, tens of millions) of devices. Majority of these devices uses flash. As a recent bit of trivia, FatFs was the first and so far the only file system imported into Zephyr RTOS, a "critically acclaimed" RTOS for IoT run by Linux Foundation, with majority of contributions coming from Intel.
So, while I'd personally love to run more advanced FS with MicroPython, like ext2, it wouldn't help anyone. In the end, there would be the same filesystem, just less familiar to people, so handling issues, error recovery, etc. would be more complicated.
I'd be specifically wary dealing with "out of nowhere" filesystems. One good example is SPIFFS. I stumbled upon it when there was no esp8266 or NodeMCU, more than 2 years ago, exactly when considering an alternative FS for MicroPython. It was completely unknown project 4 stars then on github, and I just shivered at the thought of how much effort it may take to make it work well and reliable. Nodemcu gave it some kick, likely due to the name - if esp8266 uses SPI flash, then "SPIFFS" is what you've got to use with it, right? It's still several years and couple of millions users short to treat it is as well-known and reliable.
So, all the above is to present the landscape and to explain why there're no immediate plans from the maintainers to work on other filesystems integration. However, we encourage MicroPython community to experiment with that, share results, when proven useful and well done, contribute patches. That can be EXT2, SPIFFS, or something else. My personal favorite to try is CoffeeFS from ContikiOS. Being advertised as flash-friendly filesystem, CoffeeFS is around for more time than SPIFFS, written for well-known RTOS by its maintainers, has academic publications behind it, etc.
We, the maintainers, from our side, working to make that - integrating different, and even multiple, filesystems - easier (work on ESP8266 port literally started with that).
1. FAT is the most used and well-known filesystem in the world. It's so well known that everyone and their grandma knows how "bad" it is.
2. All other filesystems, much less is known about them. Much few people known how "bad" those are. A cautious person would expect the worst then.
3. "FatFs" library, which MicroPython uses, is the most well-known liberally licensed open-source FAT FS. It is used by millions (likely, tens of millions) of devices. Majority of these devices uses flash. As a recent bit of trivia, FatFs was the first and so far the only file system imported into Zephyr RTOS, a "critically acclaimed" RTOS for IoT run by Linux Foundation, with majority of contributions coming from Intel.
So, while I'd personally love to run more advanced FS with MicroPython, like ext2, it wouldn't help anyone. In the end, there would be the same filesystem, just less familiar to people, so handling issues, error recovery, etc. would be more complicated.
I'd be specifically wary dealing with "out of nowhere" filesystems. One good example is SPIFFS. I stumbled upon it when there was no esp8266 or NodeMCU, more than 2 years ago, exactly when considering an alternative FS for MicroPython. It was completely unknown project 4 stars then on github, and I just shivered at the thought of how much effort it may take to make it work well and reliable. Nodemcu gave it some kick, likely due to the name - if esp8266 uses SPI flash, then "SPIFFS" is what you've got to use with it, right? It's still several years and couple of millions users short to treat it is as well-known and reliable.
So, all the above is to present the landscape and to explain why there're no immediate plans from the maintainers to work on other filesystems integration. However, we encourage MicroPython community to experiment with that, share results, when proven useful and well done, contribute patches. That can be EXT2, SPIFFS, or something else. My personal favorite to try is CoffeeFS from ContikiOS. Being advertised as flash-friendly filesystem, CoffeeFS is around for more time than SPIFFS, written for well-known RTOS by its maintainers, has academic publications behind it, etc.
We, the maintainers, from our side, working to make that - integrating different, and even multiple, filesystems - easier (work on ESP8266 port literally started with that).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/
Re: Robustness of file system?
I've been looking at SPIFFS and most of the work seems to be in creating a nice interface from MicroPython C API to a roughly POSIX-alike file C API and to a lesser extent the glue to a flash device below.
Much of this exists but it's tangled with the FAT and also native (unix for example) filesystem code which is a great shame.
From what I can see the work to make the filesystem interface more general is called fsusermount give or take a bit of punctuation and capitalisation.
I wonder to what extent I am duplicating effort.
Much of this exists but it's tangled with the FAT and also native (unix for example) filesystem code which is a great shame.
From what I can see the work to make the filesystem interface more general is called fsusermount give or take a bit of punctuation and capitalisation.
I wonder to what extent I am duplicating effort.