Filesystem: Maximum number of files/directories

[ZKDMun]

How many files and/or directories are possible on the internal filesystem of MicroPython?

Code: Select all

>>> import os
>>> for i in range(1000):
>>>    dirname = str(i)
>>>    os.mkdir(dirname)
>>>    print(i)

After ~500, sometimes ~510, new directories, I always get this:

Code: Select all

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
OSError: [Errno 13] EACCES

for longer names of the files/directories, there will be less files/directories possible...
How can I create more files/directories?

I try to implement a log-system, with ~2KB logfiles... it should be theoretically feasible to create ~1000 logfiles on the flash, but with this problem there are only (longer names of the logfiles) ~180 logfiles possible.

(I use MicroPython esp8266-20171101-v1.9.3.bin on ESP8266)
(with esp32-20180419-v1.9.3-548-gd12483d9 on ESP32 there is the same problem)

edit:
think about a SD-Card:
On an ESP8266, the free RAM size is ~20-25KB? With 25KB-Files (because you cannot open or read larger files with 25KB RAM size?), and the abovementioned limit of ~510 files => only 12,7MB of data will be possible in MicroPython on an ESP8266?

edit2:
with free RAM size of ~20-25KB, there should be more files/directories possible... these should not be the limitator/the reason for this problem... there should be ten times more files/directories possible (I guess ~5000 files with really short names will be the limit in one directory, because if there are much more files/directories you will get a memory exception if you try os.listdir() or something else?)

[dhylands]

Not sure why you would be limited to a filesize of 25K. In general there is no need to read the entire file into memory.
You can open a file in append mode and just keep writing to it.

Creating files consumes flash blocks since the directory is also stored on the filesystem.

[akronmacon]

This smells like a problem we run into in scientific logging equipment, specifically with FAT file systems.

Often, the default formatting for a FAT file system (FAT12, FAT16) only allows for 512 root entries in the root directory. This restriction doesn't exist outside the root directory.

Both directories and filenames count as an entry. I believe long filenames count as multiple entries each too.

Later versions of FAT move recording the root directory outside of the File Allocation Table I believe, removing this limitation. I also think you could theoretically reformat the drive with the option for provisioning more root entries, but I'd personally shy away from that due to the special nature of the storage directly on the pyboard.

Perhaps you might try moving your main operations inside a directory itself, off of the root. I suspect that should get around this problem.

For a deep dive into this see: https://docs.microsoft.com/en-us/previo ... 0(v=ws.10)

There's also a really well written .doc file from Microsoft that floats around the internet on this, but I can't find it right now.

[pythoncoder]

There is work in progress to implement an alternative filesystem.