littlefs - advantages & disadvantages

[jedie]

I now see, that LittleFS will be the default in MicroPython v2.0, see: https://github.com/micropython/micropython/issues/4821

Question: It there a way to detect, if the filesystem is already converted to LittleFS?

[jedie]

Question: It there a way to detect, if the filesystem is already converted to LittleFS?

Think i found a way:

Code: Select all

import flashbdev
import uos as os

uname = os.uname()

print(uname.machine, uname.release)
print('MicroPython', uname.version)
print()

print('flashbdev.size....:', flashbdev.size)
print('reserved sectors..:', flashbdev.bdev.RESERVED_SECS)
print('start sector......:', flashbdev.bdev.START_SEC)
print('sector size.......:', flashbdev.bdev.SEC_SIZE)
print('num blocks........:', flashbdev.bdev.NUM_BLK)


def filesystem_hex_dump(line_count=10, chunk_size=16):
    buf = bytearray(chunk_size)
    for block_num in range(line_count):
        offset = block_num * chunk_size
        print('%04x - %04x' % (offset, offset + chunk_size - 1), end=' - ')
        flashbdev.bdev.readblocks(block_num, buf)
        print(' '.join('%02x' % char for char in buf), end=' - ')
        print(''.join(chr(char) if 32 < char < 177 else '.' for char in buf))


filesystem_hex_dump(line_count=10, chunk_size=16)

Code: Select all

ESP module with ESP8266 2.2.0-dev(9422289)
MicroPython v1.11-8-g48dcbbe60 on 2019-05-29

flashbdev.size....: 1048576
reserved sectors..: 1
start sector......: 153
sector size.......: 4096
num blocks........: 106
0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff 00 f0 ff ff 0f 00 ff 0f 00 ff ff ff ff - ................
0020 - 002f - 42 4f 4f 54 20 20 20 20 50 59 20 20 18 00 51 b6 - BOOT....PY....Q.
0030 - 003f - e5 41 54 43 48 44 4f 47 4d 50 59 20 18 00 19 86 - .ATCHDOGMPY....†
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0050 - 005f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0060 - 006f - 4d 04 02 1f 20 82 00 03 00 30 00 00 00 07 07 00 - M....‚...0......
0070 - 007f - 4d 04 02 1f 20 84 28 08 00 00 00 00 00 07 07 00 - M....„(.........
0080 - 008f - 4d 04 02 1f 20 81 04 02 00 30 00 00 00 07 07 00 - M.......0......
0090 - 009f - 4d 04 02 1f 20 81 14 02 00 30 00 00 00 07 07 00 - M.......0......

MSDOS5.0 -> FAT, isn't it?

So, it's possible to made this:

Code: Select all

def detect_filesystem():
    buf = bytearray(8)
    flashbdev.bdev.readblocks(0, buf)
    if buf[3:8] == b'MSDOS':
        return 'FAT'
    return 'unknown'
print(detect_filesystem())

Is it maybe sufficient to check only the first one or two bytes?
EDIT: No: The first 3 bytes are the "jump address" -> https://en.wikipedia.org/wiki/Design_of ... oot_Sector


EDIT: Boring: I can't convert to littlefs1 or littlefs2:

os.VfsLfs.mkfs(flashbdev.bdev) # AttributeError: 'module' object has no attribute 'VfsLfs'
os.VfsLfs2.mkfs(flashbdev.bdev) # AttributeError: 'module' object has no attribute 'VfsLfs2'

EDIT2: Create a issue -> https://github.com/micropython/micropython/issues/5387

[kevinkk525]

MicroPython v1.11-8-g48dcbbe60 on 2019-05-29

You are using an old firmware build, pull the latest changes and build it again. littleFS will be included.

[jedie]

MicroPython v1.11-8-g48dcbbe60 on 2019-05-29

You are using an old firmware build, pull the latest changes and build it again. littleFS will be included.

I used the last stable build from: http://micropython.org/download#esp8266
So the feature does only exists in the "daily builds" ?
Are daily build "stable" ?
Why contains the official docs stuff that are in last stable builds?!?

[kevinkk525]

you can use the daily builds, it's basically what I use when building the firmware myself. They are stable.

[jedie]

I flash the initial OTA esp8266-ota-20191206-v1.11-624-g210d05328.bin and the filesystem looks strange:

Code: Select all

ESP module with ESP8266 2.0.0(5a875ba)
MicroPython v1.11-624-g210d05328 on 2019-12-06

flashbdev.size....: 1048576
reserved sectors..: 1
start sector......: 213
sector size.......: 4096
num blocks........: 106
0000 - 000f - 4d 04 02 1f 20 82 58 03 00 30 00 00 00 07 07 00 - M....‚X..0......
0010 - 001f - 4d 04 02 1f 20 84 64 04 00 30 00 00 00 07 07 00 - M....„d..0......
0020 - 002f - 4d 04 02 1f 20 83 40 04 00 30 00 00 00 07 07 00 - M....ƒ@..0......
0030 - 003f - 4d 04 02 1f 20 81 4c 03 00 30 00 00 00 07 07 00 - M....L..0......
0040 - 004f - 4d 04 02 1f 20 84 2c 03 00 30 00 00 00 07 07 00 - M....„,..0......

After os.VfsFat.mkfs(flashbdev.bdev)

Code: Select all

0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0020 - 002f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0030 - 003f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................

But again: AttributeError: 'module' object has no attribute 'VfsLfs2'

EDIT: I used the latest non-OTA build: esp8266-20191206-v1.11-624-g210d05328.bin
Get interesting message after boot:
The FAT filesystem starting at sector 153 with size 98 sectors appears to
be corrupted. If you had important data there, you may want to make a flash
snapshot to try to recover it. Otherwise, perform factory reprogramming
of MicroPython firmware (completely erase flash, followed by firmware
programming).

I run my script again:

Code: Select all

ESP module with ESP8266 2.0.0(5a875ba)
MicroPython v1.11-624-g210d05328 on 2019-12-06

flashbdev.size....: 1048576
reserved sectors..: 1
start sector......: 153
sector size.......: 4096
num blocks........: 106
0000 - 000f - 01 00 00 00 30 34 c0 39 12 0d f0 00 ee 07 2c 40 - ....04.9......,@
0010 - 001f - 39 12 3c f3 37 92 02 86 ad 00 5c b3 37 92 02 86 - 9.<.7’.†­.\.7’.†
0020 - 002f - 12 c1 10 0d f0 00 00 00 c6 31 2c 40 8a 31 2c 40 - .........1,@Š1,@
0030 - 003f - 7a e1 da f7 d8 0f e8 0e 00 46 40 30 f3 81 da ee - z........F@0...
0040 - 004f - c7 f8 30 30 f4 d8 51 88 81 98 91 c0 c0 f4 0c 04 - ..00..Qˆ˜‘.....
Detected filesystem: unknown

convert to FAT...

0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0020 - 002f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0030 - 003f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
Detected filesystem: FAT

convert to littlefs2...

Traceback (most recent call last):
  File "<stdin>", line 58, in <module>
AttributeError: 'module' object has no attribute 'VfsLfs2'

EDIT: Just for completeness. With a "erase_flash" and fresh flash, it looks:

Code: Select all

0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff ff 0f 00 00 00 00 00 00 00 00 00 00 00 - ................
0020 - 002f - 42 4f 4f 54 20 20 20 20 50 59 20 20 18 00 01 00 - BOOT....PY......
0030 - 003f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................

[jedie]

you can use the daily builds, it's basically what I use when building the firmware myself. They are stable.

Is that right? I guess I'm out of luck.

In the last daily build esp8266-20191206-v1.11-624-g210d05328.bin is no 'uasyncio' -> https://github.com/micropython/micropython/issues/5390

[Christian Walther]

As far as I can see, littlefs is not enabled by default for ESP8266, you need to explicitly build with MICROPY_VFS_LFS2=1.

Apparently the daily builds are made with the default manifest rather than the release manifest and therefore don’t have as many frozen modules as the stable releases. I’m not sure what the standard way of changing that is, I’m guessing FROZEN_MANIFEST=boards/manifest_release.py.

Both of these together don’t seem to fit with the current linker script, I get

Code: Select all

xtensa-lx106-elf-ld: build-GENERIC/firmware.elf section `.irom0.text' will not fit in region `irom0_0_seg'
xtensa-lx106-elf-ld: region `irom0_0_seg' overflowed by 13344 bytes

[pythoncoder]

There are a few "wrinkles" in doing this stuff. If you have too many frozen modules you need to allocate more space. Modify esp8266/boards/esp8266.ld so that line 8 reads:

Code: Select all

    irom0_0_seg :  org = 0x40209000, len = 0xa7000

You can specify any location for frozen files. The manifest file does this. Mine reads

Code: Select all

freeze('/mnt/qnap2/Scripts/modules/esp8266_modules')
freeze('$(MPY_DIR)/tools', ('upip.py', 'upip_utarfile.py'))
freeze('$(MPY_DIR)/drivers/dht', 'dht.py')
freeze('$(MPY_DIR)/drivers/onewire')

The file /mnt/qnap2/Scripts/modules/esp8266_modules contains symlinks to modules for freezing.

I build ESP8266 firmware with this script, which points to the above manifest file and specifies littlefs2. Obviously directories will need changing:

Code: Select all

#! /bin/bash
cd /mnt/qnap2/data/Projects/MicroPython/micropython/ports/esp8266
MANIFEST='/mnt/qnap2/Scripts/manifests/esp8266_manifest.py'

if [ $# -eq 1 ] && [ $1 = "--clean" ]
then
    make clean
fi

if [ $# -eq 1 ] && [ $1 = "--erase" ]
then
    make clean
    esptool.py  --port /dev/ttyUSB0 erase_flash
fi

if make -j 8 FROZEN_MANIFEST=$MANIFEST MICROPY_VFS_LFS2=1
then
    sleep 1
    esptool.py --port /dev/ttyUSB0 --baud 115200 write_flash --flash_size=detect -fm dio 0 build-GENERIC/firmware-combined.bin
    cd -
    sleep 4
    rshell -p /dev/ttyUSB0 --editor nano --buffer-size=30
else
    echo Build failure
fi
cd -

Lastly, as @kevinkk525 has explained, to force a format to littlefs you need to edit inisetup.py (in your frozen directory):

Code: Select all

    uos.VfsLfs2.mkfs(bdev)
    vfs = uos.VfsLfs2(bdev)
#    uos.VfsFat.mkfs(bdev)
#    vfs = uos.VfsFat(bdev)

Erase flash, compile, load and enjoy.

I now have a ~600LOC program running with 25.1KiB of free RAM...

[jedie]

There are a few "wrinkles" in doing this stuff.

It all seems to me that it's not finished yet.
I still shy away from the effort to compile the firmware myself.

I hope the next stable version will be released soon and has littlefs2 directly on board. (Because daily builds lacking 'uasyncio', too, see: https://github.com/micropython/micropython/issues/5390 )