[nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Discussion and questions about boards that can run MicroPython but don't have a dedicated forum.
Target audience: Everyone interested in running MicroPython on other hardware.
Post Reply
User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

[nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by WhiteHare » Tue Oct 30, 2018 6:25 pm

As an example, on an nRF52840 by default there's enough space in the micro file system to store two ~12K micropython (.py) files, but apparently not enough space for three or more. Yet, the nRF52840 comes standard with 512K of flash memory, so there should be plenty more space that can be allocated and made available. Is the unallocated flash serving some other purpose, or is it just going to waste if it isn't allocated?

I was going to use uos.statvfs to try to find out more, but it doesn't seem to be implemented:

Code: Select all

>>> MicroPython v1.9.4-651-g0f6f86ca4-dirty on 2018-10-28; PCA10056 with NRF52840
Type "help()" for more information.
>>> import uos
>>> uos.listdir()
['main.py', 'update.txt']
>>> uos.statvfs()
Traceback (most recent call last):
  File "<stdin>", in <module>
AttributeError: 'module' object has no attribute 'statvfs'
>>>
Any suggestions or advice would be appreciated.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by dhylands » Tue Oct 30, 2018 10:16 pm

From a quick glance at the code. the microbit filesystem is limited to about 31.5kB regardless of flash size.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by WhiteHare » Thu Nov 01, 2018 11:19 pm

Sadly, I'm now up against this limit. I'm more or less halted until it can get resolved.

Who knows how to fix it? You? Glenn Rub? Somebody else?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by dhylands » Fri Nov 02, 2018 12:15 am

You can use frozen files, or you'll need to use a different filesystem.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by WhiteHare » Fri Nov 02, 2018 12:19 am

dhylands wrote:
Fri Nov 02, 2018 12:15 am
You can use frozen files, or you'll need to use a different filesystem.
What other file system is there that I can use?

c45713
Posts: 51
Joined: Fri Dec 09, 2016 7:09 pm

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by c45713 » Sat Nov 03, 2018 12:12 am

In order to scale the size set of for internal file system you can add a custom linker script like for microbit:
https://github.com/micropython/micropyt ... icrobit.ld
Point to your custom linker script holding the size you want to override in the mpconfigboard.mk instead of the default linker script.

It is possible to swap out the microbitfs with FATFS, or add it in addition for external storage. There is a discussion with experiments and findings in the old repository for the port:
https://github.com/tralamazza/micropython/issues/96

The the experiements it was basically found that FATFS on internal flash used a bit to much in overhead, at least for nrf51 and in practice also for nrf52. The findings were that 50 pages had to be reserved in order to even make the FATFS structures, then the file storage came on top of this. If i recall correctly, if you set of 64k for filesystem on nrf51 (page size 1k) 50 pages was to be reserved for the FS, and the last 14 pages of 1k will be available for storage. For nrf52 it was even worse, 50 pages resulted in only using the top 1k of each 4k page in physical flash, needing 200k only for the FS itself. Then data on top of this.

The conclusion was to go for the microbitfs as default, with its limitations, for internal storage as it utilizes flash much better when the flash storage is limited, and use the possibility to plug in FATFS for external usage (SD cards etc.) using VFS. It should be possible to run the microbitfs and FATFS in paralell using VFS, pointing internal to microbitfs and external to FATFS based on mounting point.

Personally, when i need space, i enable the VFS and FATFS and run this on SPI to a SD card. Storing py files on the SD card. From my experience, loading big .py files from SD often triggers micropython compiler out of ram issues. So maybe, cross compiling them first is better.
https://github.com/micropython/micropyt ... mountsd.py

You can try to play with the MICROPY_VFS/MICROPY_VFS_FAT in mpconfigport.h in combo with the make flag in MICROPY_FATFS=1.
For more detailed description of this:
https://github.com/micropython/micropyt ... ropy_fatfs
https://github.com/micropython/micropyt ... 8be14587ec

Hope this was to some help

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by WhiteHare » Tue Nov 06, 2018 6:04 am

WhiteHare wrote:
Tue Oct 30, 2018 6:25 pm
Yet, the nRF52840 comes standard with 512K of flash memory
Correction: I misquoted the spec. Actually, the nRF52840 comes with 1 megabyte of flash memory, not 512K of flash.

User avatar
marfis
Posts: 215
Joined: Fri Oct 31, 2014 10:29 am
Location: Zurich / Switzerland

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by marfis » Fri Nov 16, 2018 3:31 pm

@c45713
The the experiements it was basically found that FATFS on internal flash used a bit to much in overhead, at least for nrf51 and in practice also for nrf52. The findings were that 50 pages had to be reserved in order to even make the FATFS structures, then the file storage came on top of this. If i recall correctly, if you set of 64k for filesystem on nrf51 (page size 1k) 50 pages was to be reserved for the FS, and the last 14 pages of 1k will be available for storage. For nrf52 it was even worse, 50 pages resulted in only using the top 1k of each 4k page in physical flash, needing 200k only for the FS itself. Then data on top of this.


I was able to succesfully mount a Flash FS system on my nRF52832 with app 20k overhead, leaving me with roughly 100k free space for file system.

The limitation of 50 is in my opinion a bug in oofatfs (i filed a bug report here https://github.com/micropython/micropython/issues/4305) since it doesn't include the block size (it should be set to 4096 Bytes and therefore 5 blocks should be ok). Also the minimum block size should be set to 4096 bytes too.

Once I changed that I wrote a simple C-wrapper module around your flash module that exposes functions for read and write . Once that was made it was simply a matter of defining a flash block device in micropython (basically copy&pasted from the esp8266) and mounting the blockdevice as explained https://docs.micropython.org/en/latest/ ... ck-devices :). I have also added the same logic as on the 8266 (mount it within a frozen _boot.py)

I can provide a PR if interested..

User avatar
rcolistete
Posts: 352
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: [nRF52] How do I increase the amount of flash allocated to micropython's micro file system?

Post by rcolistete » Tue Aug 18, 2020 1:45 am

Updating some info that may be useful about microbitFS on nRF boards with MicroPython :
- by default, ports/nrf uses microbitFS as file system, see :
* issue #6194 "nrf: internal flash filesystem/storage not implemented with VFS" :
Today, microbitfs (MBFS) is default internal flash filesystem for all nrf-targets.
* MicroPython Forum topic "Questions about the nRF port" :
After testing out the different solutions, we decided mostly on IRC to enable microbitfs as default on all targets.
- "micropython/ports/nrf/boards/memory.ld", line 9 :

Code: Select all

_fs_size       = DEFINED(_fs_size) ? _fs_size : 64K; /* TODO: set to 0 if not using the filesystem */
cites 64 kB of flash size for file system. But microbitFS is configured with 32 kB, see line 39 of "micropython/ports/nrf/modules/uos/microbitfs", with 31752 usable bytes. So it is possible to :
* decrease '_fs_size' to '32K', such that it increases in 32 kB the 'FLASH_TEXT'/'_app_size' (space for firmware);
* or use this _fs_size' of '64K' for microbitFS configured with the maximum of 64 kB, by changing line 39 to "MBFS_LOG_CHUNK_SIZE 8".
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

Post Reply