os.statvfs() returns negative numbers in tuple

[MicroRichard]

In the boot.py i use the following:

Code: Select all

p2 = pyb.Flash(start=512*1024)
os.VfsLfs2.mkfs(p2)
os.mount(p2, '/data')

This creates a littleFS data partition. Now i have created 135 files and when i run

Code: Select all

info = os.statvfs("/data")
logging.info("Free file system: {:d} bytes".format(info[0]*info[3]))

then the reported file system information contains negative numbers.

Code: Select all

2015-1-21 12:39:7  [DEBUG]  (4096, 4096, 384, -174, -174, 0, 0, 0, 0, 255)
2015-1-21 12:39:7  [INFO ]  Free file system: -712704 bytes

What does the negative numbers mean, no space or other problem?

[jimmo]

What does the negative numbers mean, no space or other problem?

Thats...bad. My guess would be filesystem corruption. Is there another fileystem in the first 512kiB? Is it definitely being constrained not to go past 512kiB?

Which device are you using? Do you have a full example that I can try and repro with here?

[MicroRichard]

The device is a pyboard d, the micropython version is 1.15.
This is the complete boot.py:

Code: Select all

import os, pyb
os.umount('/flash')
p1 = pyb.Flash(start=0, len=512*1024)
p2 = pyb.Flash(start=512*1024)
os.VfsFat.mkfs(p1)
os.VfsLfs2.mkfs(p2)
os.mount(p1, '/flash')
os.mount(p2, '/data')
os.chdir('/flash')

The first 512 KB is configured as FAT32 and the rest as littlefs.

In my main code i have the following lines to mount the data partition:

Code: Select all

data_partition = pyb.Flash(start=512*1024)
os.mount(data_partition, '/data')

Then the code to write a file with the config being a dictionary is:

Code: Select all

def write(config, file):
    try:
        config_file = open(file, 'w')
        config_file.write(json.dumps(config))
        config_file.close()
    except OSError as e:
        logging.error("Failed to write config file: {}".format(e))

There were already 35 files on the flash and i called write() another 100 times. The following code is not the actual code but what it basically does is:

Code: Select all

self.config_file = None
self.config = {}
self.config['id'] = test
for x in range(100):
  self.config_file = '/data/' + str(x)  # create new file name
  write(self.config, self.config_file)

[jimmo]

The device is a pyboard d, the micropython version is 1.15.

It would be interesting to know if this still happens on 1.19.1. There's been a bit of work done in this area since 1.15 I think.

I will try and replicate this and raise a bug if possible.

[jimmo]

The device is a pyboard d, the micropython version is 1.15.

Sorry I tried to replicate this but I wasn't sure exactly how your application works.

In particular I was a bit confused by your boot.py example which appears to always unconditionally format both partitions.

Is there any chance you can make a script that I can run on a pyboard d that replicates the problem? Thanks

[MicroRichard]

The boot.py is used once to format the flash. After that the boot.py is removed from the pyboard and the rest of the files are copied to the pyboard. Otherwise the boot.py would remove all contents.

[jimmo]

The boot.py is used once to format the flash. After that the boot.py is removed from the pyboard and the rest of the files are copied to the pyboard. Otherwise the boot.py would remove all contents.

OK I figured that might be the case. (I was confused why this specifically needed to be boot.py though)

I did spent some time trying to replicate this (on v1.19.1 on a pyboard-d) but did not manage to repro the problem. If there's any way you can send me a script (or scripts) that I can run that triggers this, that would be extremely useful.

(Feel free to re-open this at https://github.com/orgs/micropython/discussions too, would be good to get to the bottom of what's happening)

[MicroRichard]

Just tested this problem with MicroPython v1.19.1-2-g8e3f8fb on PYBD-SF2W with STM32F722IEK and the os.statvfs() still reports a negative number which gives me reason to conclude the file system corruption still occurs.
The following script might reproduce the problem, perhaps increase the number of files to create. No exception is reported when flash boundary is exceeded.

Code: Select all

from os import mount, statvfs
import json
import uasyncio as asyncio
import sys
import gc
from pyb import Flash
gc.collect()

def set_global_exception():
    def handle_exception(loop, context):
        sys.print_exception(context["exception"])
    loop = asyncio.get_event_loop()
    loop.set_exception_handler(handle_exception)

def mount_data_partition():
    try:
        data_partition = Flash(start=512*1024)
        mount(data_partition, '/data')

        # Free littleFS space
        info = statvfs("/data")

        # Print the file system free bytes
        print("Free file system: {:d} bytes".format(info[0]*info[3])) 
        return True
    except Exception as err:
        print('Mounting data partition failed: {}'.format(err))
        return False

def write(config, file):
    try:
        config_file = open(file, 'w')
        config_file.write(json.dumps(config))
        config_file.close()
    except OSError as e:
        print("Failed to write config file: {}".format(e))

async def test_create_number_of_files(number_of_files_to_create):
    import os
    config_file = None
    config = {}
    config['id'] = 'test'
    print('Creating files...')
    for x in range(number_of_files_to_create):
      config_file = '/data/' + str(x)  # create new file name
      write(config, config_file)

async def main():
    try:
        second_counter = 0
        set_global_exception()
        
        mount_data_partition()
        
        await test_create_number_of_files(100)
        
        while True:
            gc.collect()

            if second_counter < 30:
                second_counter += 1
            else:
                print("Free memory: {:d} bytes".format(gc.mem_free()))
                second_counter = 0
            
            await asyncio.sleep(1)

    except asyncio.CancelledError:
        # Task cancelled
        pass


asyncio.run(main())

[jimmo]

MicroPython v1.19.1-2-g8e3f8fb on PYBD-SF2W

Also can't find this commit in the history...

The following script might reproduce the problem, perhaps increase the number of files to create. No exception is reported when flash boundary is exceeded.

I'm not sure I understand how to run this script.

Here's what I did:
I ran your boot.py from earlier to create a 512k FAT and the remainder as LittleFS.
Then I ran your script from the previous message a few times. (With the number of files increased to 1000)
What am I looking for?

[MicroRichard]

The script is saved as main.py on the board. Perhaps after the creation of files you should add

Code: Select all

info = statvfs("/data")

# Print the file system free bytes
print("Free file system: {:d} bytes".format(info[0]*info[3]))

Then, if you create a lot of files, you should see a negative number as system free bytes returned from os.statvfs().