MicroPython and flash memory issues

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
manos
Posts: 27
Joined: Mon Sep 28, 2020 9:03 am

MicroPython and flash memory issues

Post by manos » Tue Sep 29, 2020 7:39 am

Hey everybody!

Being a total newbie, i am sorry in advance for the hard time I am giving you :roll:
Let me start quoting the following from the MicroPython Documentation:

"The file system is typically backed by internal flash memory on the device, but can also use external flash, RAM, or a custom block device."

So, if i understand correctly MicroPython provides an on-device file system which usually "resides" in flash memory.
My first question is: are there any built-in functions of any module that can show me what is the size of the flash memory (when the filesystem "resides" in the flash)?

Secondly, what does it mean that the file system can be backed in a custom block device? Custom block devices are something like a custom file system, isn't? Is that stored in the flash memory too?

And something irrelevant. According to what i have understood, MicroPython can be implemented in bare-metal environment (no OS) or even when there is some underlying OS. Is there any way i can know that, given that when accessing the MicroPython environment, it acts as a small OS?

Now, in a bit different aspect, but again with memory issues in mind.
I have a Digi XBee 3 Zigbee and according to its specs, it has the following memory:
1 MB / 128 KB RAM (32 KB are available for MicroPython)

Well, correct me if I am wrong, but:
the 1MB is flash memory and 128MB are RAM, of which 32KB are available for MicroPython.
Now, I tried to use the uos.statvfs function from the uos module. The result I got was a tuple which indicated that I have 2048 free blocks of memory.
Since the name of the function indicates that it is about the virtual file system (VFS), which as mentioned before, I expected it to "reside" in flash memory, would that mean that that is my free flash memory? This doesn't really make any sense though...(not to mention that through XCTU GUI, the file system is said to have a total of 382KB)
So my guess was that this is referring to the available RAM memory. In specific, taken into consideration that a block is 16bits (not sure if that is true though) then by multiplying it to the number of free blocks, then i would have about 4KB of memory, which could correspond to the amount of stack memory (in the RAM) - which is actually what Xbee allocates, as Xbee's documentation states.
So, is that true? Does uos.statvfs() show info about stack memory? (doubt it, but I got no better idea for the time being)

Hopefully, I didn't confuse you all.
Thank you in advance for any help!!!

manos
Posts: 27
Joined: Mon Sep 28, 2020 9:03 am

Re: MicroPython and flash memory issues

Post by manos » Tue Sep 29, 2020 8:18 am

Oh, well regarding the block device, I have found that:
MicroPython creates the VFS and offers the ability to mount other file systems(i.e LFS) within this VFS. So, a block device is a custom class (a template of which should be inspired by the uos.AbstractBlockDev class) which can provide me access to the VFS and/or the extra file system mounted within the VFS. Is that right? Am i getting this right?

Also, regarding the uos.statvfs function:
I still have some questions but I should definitely had to be more careful...
uos.statvfs returns: (2048, 128, 3056, 2448, 2448, 0, 0, 0, 0, 64)
which means that the file system block size is 2048. How is that translated to actual flash memory?
Now fragment size is 128 -- this I guess refers to the whole RAM memory or am I completely mistaken?
Well, I guess that the values 3056 and 2048 will be explained (how they were derived) after the aforementioned questions are somehow cleared out..

By the way, while sys module can be imported and function properly, I am not able to do the same with usys, although the implementation is MicroPython in my Xbee...Why is not usys built in MicroPython environment?

Thank you again in advance

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: MicroPython and flash memory issues

Post by Roberthh » Tue Sep 29, 2020 11:46 am

uos.statvfs returns: (2048, 128, 3056, 2448, 2448, 0, 0, 0, 0, 64)
statvfs() talks about the file system and tells nothing about the RAM. The result of the statvfs call says, that the total file system size is 6130 kbyte. Since the CPU is told to have 1 MByte flash, there is obviously some more flash memory on the board, which may only be usable as data memory. I have a unit here, but I do not like to open the package just to look for that.

The XBEE3 uses an own version of MicroPython, which does only implement a subset of MP's functions. In order to use that, better refer to Digi's excellent MicroPython documentation (https://www.digi.com/resources/document ... /90002219/)

manos
Posts: 27
Joined: Mon Sep 28, 2020 9:03 am

Re: MicroPython and flash memory issues

Post by manos » Tue Sep 29, 2020 2:03 pm

Hi Roberthh! Thanks a lot for your response!

Thank you for clearing up that statvfs tells nothing about the RAM.
May I ask how did you get to the conclusion of the file system size (you said it is 6130 KB?), just from statvfs's result?
Well, maybe it's not of great interest to you, but from what I have found out, half of the flash size is deserved for the OTA updates (and some extra is excluded due to some overhead).
So, there remains something less than 512KB of memory. The XCTU GUI program actually lets me know that the total file system is about 385KB. So you are definitely right saying that obviously there is some more flash memory laying there as data memory.
    The question is how am I able to access it?
      Won't every write() method of let's say uio.BytesIO class be able to write only in the flash that MP "sees"?
        Also, I am interested in filing most of the flash memory with data, so I would like to have some checking of the file size. For that reason, the size of the flash would be useful to be returned by some MP function. And that's why I am insisting in pre-knowing the exact size of the flash.

        Regarding the usys, thank you for pointing out that XBEE3 uses it's own version of MP. Good to know that.

        Thanks again.

        User avatar
        Roberthh
        Posts: 3667
        Joined: Sat May 09, 2015 4:13 pm
        Location: Rhineland, Europe

        Re: MicroPython and flash memory issues

        Post by Roberthh » Tue Sep 29, 2020 3:34 pm

        I have noticed the discrepancy between XCTU and statvfs. My experience is, that statvfs is right, because I have written definittely more than 385k to the device, using pyboard.py resp. the new mpr.py tool of Damien. You can use the normal Python file operations for reading and writing. But there are some back-draws, as I had to notice:

        - uos.rename() is not supported
        - open(name, "w") on an existing file is not supported. You have to delete a file first to be able to write to it,
        - deleting a file does NOT reclaim the space used by it. So even if you delete files, writing will fill up the file system. The only way to reclaim it is using os.format() which will erase everything.

        statvfs is the method to tell the space of the file system, like

        import uos
        data=uos.statvfs("")
        fs_free = data[0]*data[3]

        Whatever other memory is on the device is hard to tell. I do not use XCTU a lot. Just for initial set-up, like setting the REPL speed to 115200 baud. Besides that I use picocom as terminal emulator and the mpr.py tool for file transfer.

        User avatar
        Roberthh
        Posts: 3667
        Joined: Sat May 09, 2015 4:13 pm
        Location: Rhineland, Europe

        Re: MicroPython and flash memory issues

        Post by Roberthh » Wed Sep 30, 2020 10:11 am

        I mst have been sleeping. Actually, the free space on the file system is counted in fragment sizes,

        import uos
        data=uos.statvfs("")
        fs_free = data[1]*data[3]

        And then the free space reported by XCTU and statvfs() match. And also the file system fits into the 1MByte flash size of the CPU.

        manos
        Posts: 27
        Joined: Mon Sep 28, 2020 9:03 am

        Re: MicroPython and flash memory issues

        Post by manos » Wed Sep 30, 2020 11:57 am

        Hahaha, don't worry I feel the same...Especially now cause I still don't get it...

        First of all the calculation data[1]*data[3] = 313344.
        [This imply that you multiply fragment size(the what?) times # of free blocks]
        So, you mean that the result, 313344, is bytes (?) or approximately 314KB which is actually close to what the XCTU shows (385KB)...
          Do I get it right?
          But again, what are then the other values returned by statvfs? The documentation says that the "0" element of the tuple is the file system block size. Meaning what?

          Moreover, I would like to ask (please tell me if I need to open new topic about this, if it is irrelevant):
            How can I create files in the flash memory through MP. My first trials were around uio.BytesIO() and the file doesn't seem to exist somewhere inside the filesystem.

            Thank you in advance!

            User avatar
            Roberthh
            Posts: 3667
            Joined: Sat May 09, 2015 4:13 pm
            Location: Rhineland, Europe

            Re: MicroPython and flash memory issues

            Post by Roberthh » Wed Sep 30, 2020 12:43 pm

            stat[1]*stat[2] would then be the total size of the file system. You create a file in the standard python (and C) mode:

            Code: Select all

            f = open(filename, "w")
            and the you can write with.

            Code: Select all

            f.write(data)
            and close the file with

            Code: Select all

            f.close().
            For reading, open with:

            Code: Select all

            f = open(filename, "r")
            and then:

            Code: Select all

            data = f.read(number of bytes)
            or

            Code: Select all

            data = f.readline()
            etc...
            It is considered bette style to us context managers, e.g.

            Code: Select all

            with open(filename, "w") as f:
                f.write()
            
            That will ensure that the close() cannot be forgotten. and the data is actually written. If you want to write/read binary data, use "wb" or "rb" as open mode.

            manos
            Posts: 27
            Joined: Mon Sep 28, 2020 9:03 am

            Re: MicroPython and flash memory issues

            Post by manos » Wed Sep 30, 2020 3:38 pm

            Once again thank you for your precious help!

            The open("filename.txt", ..) seems to work ok. At least I can see it in the /flash directory.
            Now a couple of questions (it seems like it's never ending with me...I'm sorry):
              Let's say that I have a file stored in flash memory which has a size bigger than that of RAM. Will I be able to read/write to it? My question lies on the suspicion that each time I open it for further processing, I "drag" the file in RAM, and afterwards back to flash. So, how will it be handled if there is enough RAM?
              Maybe I am totally wrong and this is a stupid question (forgive me in advance), but please spare the noob and his concerns:)
                Stubborn as I am, I would like to insist on the BytesIO class a little bit more. My first approach was something like this:
                import uio
                f = uio.BytesIO(b"42")
                f.getvalue() # returns >> 42
                f.write(b"99")
                f.close()

                or using the getbuffer() method.
                The problem is obvious (although it took me some time..)
                There is nowhere that I name a file name and/or location to store the above data! So, my question is if there is some way I can do it like this.

                Thanks in advance!

                User avatar
                Roberthh
                Posts: 3667
                Joined: Sat May 09, 2015 4:13 pm
                Location: Rhineland, Europe

                Re: MicroPython and flash memory issues

                Post by Roberthh » Wed Sep 30, 2020 4:19 pm

                BytesIO is the in memory binary stream, open a file in binary mode is working on the file system. Both are parts of the io module. So it is just two sides of the same coin.
                Obviously, you cannot store more data in RAM than the RAM size is. With files, being it also an io type set of methods, you can read & write partial files, you can position the point of reading/writing with seek, etc. Just consider it a a large buffer.

                Note: the XBEE micropython has some restriction in file handling. For details, look into their manual.

                About the io modules in general, just look into the Python manual for details (https://docs.python.org/3/library/io.html).

                Post Reply