littlefs - advantages & disadvantages

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: littlefs - advantages & disadvantages

Post by jedie » Fri Dec 06, 2019 12:58 pm

I now see, that LittleFS will be the default in MicroPython v2.0, see: https://github.com/micropython/micropython/issues/4821 :D

Question: It there a way to detect, if the filesystem is already converted to LittleFS?

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: littlefs - advantages & disadvantages

Post by jedie » Fri Dec 06, 2019 2:34 pm

jedie wrote:
Fri Dec 06, 2019 12:58 pm
Question: It there a way to detect, if the filesystem is already converted to LittleFS?
Think i found a way:

Code: Select all

import flashbdev
import uos as os

uname = os.uname()

print(uname.machine, uname.release)
print('MicroPython', uname.version)
print()

print('flashbdev.size....:', flashbdev.size)
print('reserved sectors..:', flashbdev.bdev.RESERVED_SECS)
print('start sector......:', flashbdev.bdev.START_SEC)
print('sector size.......:', flashbdev.bdev.SEC_SIZE)
print('num blocks........:', flashbdev.bdev.NUM_BLK)


def filesystem_hex_dump(line_count=10, chunk_size=16):
    buf = bytearray(chunk_size)
    for block_num in range(line_count):
        offset = block_num * chunk_size
        print('%04x - %04x' % (offset, offset + chunk_size - 1), end=' - ')
        flashbdev.bdev.readblocks(block_num, buf)
        print(' '.join('%02x' % char for char in buf), end=' - ')
        print(''.join(chr(char) if 32 < char < 177 else '.' for char in buf))


filesystem_hex_dump(line_count=10, chunk_size=16)

Code: Select all

ESP module with ESP8266 2.2.0-dev(9422289)
MicroPython v1.11-8-g48dcbbe60 on 2019-05-29

flashbdev.size....: 1048576
reserved sectors..: 1
start sector......: 153
sector size.......: 4096
num blocks........: 106
0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff 00 f0 ff ff 0f 00 ff 0f 00 ff ff ff ff - ................
0020 - 002f - 42 4f 4f 54 20 20 20 20 50 59 20 20 18 00 51 b6 - BOOT....PY....Q.
0030 - 003f - e5 41 54 43 48 44 4f 47 4d 50 59 20 18 00 19 86 - .ATCHDOGMPY....†
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0050 - 005f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0060 - 006f - 4d 04 02 1f 20 82 00 03 00 30 00 00 00 07 07 00 - M....‚...0......
0070 - 007f - 4d 04 02 1f 20 84 28 08 00 00 00 00 00 07 07 00 - M....„(.........
0080 - 008f - 4d 04 02 1f 20 81 04 02 00 30 00 00 00 07 07 00 - M.......0......
0090 - 009f - 4d 04 02 1f 20 81 14 02 00 30 00 00 00 07 07 00 - M.......0......
MSDOS5.0 -> FAT, isn't it?

So, it's possible to made this:

Code: Select all

def detect_filesystem():
    buf = bytearray(8)
    flashbdev.bdev.readblocks(0, buf)
    if buf[3:8] == b'MSDOS':
        return 'FAT'
    return 'unknown'
print(detect_filesystem())
Is it maybe sufficient to check only the first one or two bytes?
EDIT: No: The first 3 bytes are the "jump address" -> https://en.wikipedia.org/wiki/Design_of ... oot_Sector


EDIT: Boring: I can't convert to littlefs1 or littlefs2:
os.VfsLfs.mkfs(flashbdev.bdev) # AttributeError: 'module' object has no attribute 'VfsLfs'
os.VfsLfs2.mkfs(flashbdev.bdev) # AttributeError: 'module' object has no attribute 'VfsLfs2'
EDIT2: Create a issue -> https://github.com/micropython/micropython/issues/5387

kevinkk525
Posts: 654
Joined: Sat Feb 03, 2018 7:02 pm

Re: littlefs - advantages & disadvantages

Post by kevinkk525 » Fri Dec 06, 2019 3:15 pm

MicroPython v1.11-8-g48dcbbe60 on 2019-05-29

You are using an old firmware build, pull the latest changes and build it again. littleFS will be included.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: littlefs - advantages & disadvantages

Post by jedie » Fri Dec 06, 2019 3:50 pm

kevinkk525 wrote:
Fri Dec 06, 2019 3:15 pm
MicroPython v1.11-8-g48dcbbe60 on 2019-05-29

You are using an old firmware build, pull the latest changes and build it again. littleFS will be included.
I used the last stable build from: http://micropython.org/download#esp8266
So the feature does only exists in the "daily builds" ?
Are daily build "stable" ?
Why contains the official docs stuff that are in last stable builds?!?

kevinkk525
Posts: 654
Joined: Sat Feb 03, 2018 7:02 pm

Re: littlefs - advantages & disadvantages

Post by kevinkk525 » Fri Dec 06, 2019 4:22 pm

you can use the daily builds, it's basically what I use when building the firmware myself. They are stable.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: littlefs - advantages & disadvantages

Post by jedie » Fri Dec 06, 2019 6:37 pm

I flash the initial OTA esp8266-ota-20191206-v1.11-624-g210d05328.bin and the filesystem looks strange:

Code: Select all

ESP module with ESP8266 2.0.0(5a875ba)
MicroPython v1.11-624-g210d05328 on 2019-12-06

flashbdev.size....: 1048576
reserved sectors..: 1
start sector......: 213
sector size.......: 4096
num blocks........: 106
0000 - 000f - 4d 04 02 1f 20 82 58 03 00 30 00 00 00 07 07 00 - M....‚X..0......
0010 - 001f - 4d 04 02 1f 20 84 64 04 00 30 00 00 00 07 07 00 - M....„d..0......
0020 - 002f - 4d 04 02 1f 20 83 40 04 00 30 00 00 00 07 07 00 - M....ƒ@..0......
0030 - 003f - 4d 04 02 1f 20 81 4c 03 00 30 00 00 00 07 07 00 - M....L..0......
0040 - 004f - 4d 04 02 1f 20 84 2c 03 00 30 00 00 00 07 07 00 - M....„,..0......
After os.VfsFat.mkfs(flashbdev.bdev)

Code: Select all

0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0020 - 002f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0030 - 003f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
But again: AttributeError: 'module' object has no attribute 'VfsLfs2'

EDIT: I used the latest non-OTA build: esp8266-20191206-v1.11-624-g210d05328.bin
Get interesting message after boot:
The FAT filesystem starting at sector 153 with size 98 sectors appears to
be corrupted. If you had important data there, you may want to make a flash
snapshot to try to recover it. Otherwise, perform factory reprogramming
of MicroPython firmware (completely erase flash, followed by firmware
programming).


I run my script again:

Code: Select all

ESP module with ESP8266 2.0.0(5a875ba)
MicroPython v1.11-624-g210d05328 on 2019-12-06

flashbdev.size....: 1048576
reserved sectors..: 1
start sector......: 153
sector size.......: 4096
num blocks........: 106
0000 - 000f - 01 00 00 00 30 34 c0 39 12 0d f0 00 ee 07 2c 40 - ....04.9......,@
0010 - 001f - 39 12 3c f3 37 92 02 86 ad 00 5c b3 37 92 02 86 - 9.<.7’.†­.\.7’.†
0020 - 002f - 12 c1 10 0d f0 00 00 00 c6 31 2c 40 8a 31 2c 40 - .........1,@Š1,@
0030 - 003f - 7a e1 da f7 d8 0f e8 0e 00 46 40 30 f3 81 da ee - z........F@0...
0040 - 004f - c7 f8 30 30 f4 d8 51 88 81 98 91 c0 c0 f4 0c 04 - ..00..Qˆ˜‘.....
Detected filesystem: unknown

convert to FAT...

0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0020 - 002f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0030 - 003f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
Detected filesystem: FAT

convert to littlefs2...

Traceback (most recent call last):
  File "<stdin>", line 58, in <module>
AttributeError: 'module' object has no attribute 'VfsLfs2'

EDIT: Just for completeness. With a "erase_flash" and fresh flash, it looks:

Code: Select all

0000 - 000f - eb fe 90 4d 53 44 4f 53 35 2e 30 00 10 01 01 00 - ..MSDOS5.0.....
0010 - 001f - f8 ff ff ff 0f 00 00 00 00 00 00 00 00 00 00 00 - ................
0020 - 002f - 42 4f 4f 54 20 20 20 20 50 59 20 20 18 00 01 00 - BOOT....PY......
0030 - 003f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
0040 - 004f - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: littlefs - advantages & disadvantages

Post by jedie » Fri Dec 06, 2019 7:15 pm

kevinkk525 wrote:
Fri Dec 06, 2019 4:22 pm
you can use the daily builds, it's basically what I use when building the firmware myself. They are stable.
Is that right? I guess I'm out of luck.

In the last daily build esp8266-20191206-v1.11-624-g210d05328.bin is no 'uasyncio' -> https://github.com/micropython/micropython/issues/5390

Christian Walther
Posts: 91
Joined: Fri Aug 19, 2016 11:55 am

Re: littlefs - advantages & disadvantages

Post by Christian Walther » Fri Dec 06, 2019 8:56 pm

As far as I can see, littlefs is not enabled by default for ESP8266, you need to explicitly build with MICROPY_VFS_LFS2=1.

Apparently the daily builds are made with the default manifest rather than the release manifest and therefore don’t have as many frozen modules as the stable releases. I’m not sure what the standard way of changing that is, I’m guessing FROZEN_MANIFEST=boards/manifest_release.py.

Both of these together don’t seem to fit with the current linker script, I get

Code: Select all

xtensa-lx106-elf-ld: build-GENERIC/firmware.elf section `.irom0.text' will not fit in region `irom0_0_seg'
xtensa-lx106-elf-ld: region `irom0_0_seg' overflowed by 13344 bytes

User avatar
pythoncoder
Posts: 4037
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: littlefs - advantages & disadvantages

Post by pythoncoder » Sat Dec 07, 2019 8:52 am

There are a few "wrinkles" in doing this stuff. If you have too many frozen modules you need to allocate more space. Modify esp8266/boards/esp8266.ld so that line 8 reads:

Code: Select all

    irom0_0_seg :  org = 0x40209000, len = 0xa7000
You can specify any location for frozen files. The manifest file does this. Mine reads

Code: Select all

freeze('/mnt/qnap2/Scripts/modules/esp8266_modules')
freeze('$(MPY_DIR)/tools', ('upip.py', 'upip_utarfile.py'))
freeze('$(MPY_DIR)/drivers/dht', 'dht.py')
freeze('$(MPY_DIR)/drivers/onewire')
The file /mnt/qnap2/Scripts/modules/esp8266_modules contains symlinks to modules for freezing.

I build ESP8266 firmware with this script, which points to the above manifest file and specifies littlefs2. Obviously directories will need changing:

Code: Select all

#! /bin/bash
cd /mnt/qnap2/data/Projects/MicroPython/micropython/ports/esp8266
MANIFEST='/mnt/qnap2/Scripts/manifests/esp8266_manifest.py'

if [ $# -eq 1 ] && [ $1 = "--clean" ]
then
    make clean
fi

if [ $# -eq 1 ] && [ $1 = "--erase" ]
then
    make clean
    esptool.py  --port /dev/ttyUSB0 erase_flash
fi

if make -j 8 FROZEN_MANIFEST=$MANIFEST MICROPY_VFS_LFS2=1
then
    sleep 1
    esptool.py --port /dev/ttyUSB0 --baud 115200 write_flash --flash_size=detect -fm dio 0 build-GENERIC/firmware-combined.bin
    cd -
    sleep 4
    rshell -p /dev/ttyUSB0 --editor nano --buffer-size=30
else
    echo Build failure
fi
cd -
Lastly, as @kevinkk525 has explained, to force a format to littlefs you need to edit inisetup.py (in your frozen directory):

Code: Select all

    uos.VfsLfs2.mkfs(bdev)
    vfs = uos.VfsLfs2(bdev)
#    uos.VfsFat.mkfs(bdev)
#    vfs = uos.VfsFat(bdev)
Erase flash, compile, load and enjoy.

I now have a ~600LOC program running with 25.1KiB of free RAM...
Peter Hinch

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: littlefs - advantages & disadvantages

Post by jedie » Sat Dec 07, 2019 9:00 am

pythoncoder wrote:
Sat Dec 07, 2019 8:52 am
There are a few "wrinkles" in doing this stuff.
It all seems to me that it's not finished yet.
I still shy away from the effort to compile the firmware myself.

I hope the next stable version will be released soon and has littlefs2 directly on board. (Because daily builds lacking 'uasyncio', too, see: https://github.com/micropython/micropython/issues/5390 )

Post Reply