MicroPython and flash memory issues

[manos]

Dear Roberthh, thanks a lot!

What you said about BytesIO and open really cleared up my mind. Thanks!

Regarding files and RAM, if I understood correctly, you suggest that I exploit the advantages of IO type set of methods and i.e. you suggest that I break the file into many partial ones or use methods like seek, in order to position the point of reading/writing (does that mean that by positioning those points I manage not to load the whole file in RAM?)

Lastly, regarding XBEE and its file handling restrictions/limitations: are you referring to things like that when a file on the file system is deleted, the space it was using is only reclaimed if it is found at the end of the file system?

Thank you again for your help!

[Roberthh]

suggest that I break the file into many partial ones or use methods like seek, in order to position the point of reading/writing (does that mean that by positioning those points I manage not to load the whole file in RAM?)

Yes, you can do so. Please note, that on XBEE write always writes to the end of the file, regardless of the seek position, and the '+' option is not supported on the XBEE modules. (DIGI Micropython manual, page 73). That makes it a little bit tedious to work with it.

Lastly, regarding XBEE and its file handling restrictions/limitations: are you referring to things like that when a file on the file system is deleted, the space it was using is only reclaimed if it is found at the end of the file system?

The space is definitely reclaimed if you reinitialize the file system, deleting all files and bundled code. Whether space at the End of the file system is reclaimed has to be tested. A few tests I made seem to confirm that assumption.

[jimmo]

So, if i understand correctly MicroPython provides an on-device file system which usually "resides" in flash memory.
My first question is: are there any built-in functions of any module that can show me what is the size of the flash memory (when the filesystem "resides" in the flash)?

I don't know much about the XBee port, but in upstream MicroPython a block device (such as pyb.Flash) is expected to support a block size and a number of blocks ioctl.

#define MP_BLOCKDEV_IOCTL_BLOCK_COUNT (4)
#define MP_BLOCKDEV_IOCTL_BLOCK_SIZE (5)

So for example:

Code: Select all

IOCTL_BLOCK_COUNT = const(4)
IOCTL_BLOCK_SIZE = const(5)
f = pyb.Flash(start=0)
total_bytes = f.ioctl(IOCTL_BLOCK_COUNT, 0) * f.ioctl(IOCTL_BLOCK_SIZE, 0)

However I have no idea if that will work on XBee.

Secondly, what does it mean that the file system can be backed in a custom block device? Custom block devices are something like a custom file system, isn't? Is that stored in the flash memory too?

It's anything that can read and write bytes. So you could for example use RAM (I think that's the example given on that page) but also some sort of network device, or external storage (such as an external spiflash chip).

And something irrelevant. According to what i have understood, MicroPython can be implemented in bare-metal environment (no OS) or even when there is some underlying OS. Is there any way i can know that, given that when accessing the MicroPython environment, it acts as a small OS?

I'm not quite sure I understand the question, but yes MicroPython does provide some of the functionality of an OS (e.g. drivers, scheduling, memory management).

However, like you say, a given port can be implemented on top of a small (or large) OS such as FreeRTOS (or Linux). There should be very little difference to the user of MicroPython though... and from the MicroPython environment I'm not sure of a general-purpose way to know the difference.

I have a Digi XBee 3 Zigbee and according to its specs, it has the following memory:
1 MB / 128 KB RAM (32 KB are available for MicroPython)

Looks like Robert has already very thoroughly answered your main question here but it's worth pointing out that very little is known about the XBee port -- I'm not aware that Digi have ever published any source code or details other than the PDF manual.

Also worth knowing about

Code: Select all

>>> import micropython
>>> micropython.mem_info()

to learn about heap and stack RAM usage.

By the way, while sys module can be imported and function properly, I am not able to do the same with usys, although the implementation is MicroPython in my Xbee...Why is not usys built in MicroPython environment?

Until quite recently, the "sys" module was special and unlike the other modules, it was actually called "sys" (whereas uos, utime, etc). It has only been recently renamed to usys in upstream MicroPython. It sounds like the XBee port hasn't made this change yet.

[manos]

Thank you both for your replies!!!

It takes me some time to review and...process all the info you share with me (so maybe I will have some further questons in the future), but I really appreciate it!

[manos]

Hey,

sorry for bothering you again, but I am having some tough time here...
As you said here:

The space is definitely reclaimed if you reinitialize the file system, deleting all files and bundled code. Whether space at the End of the file system is reclaimed has to be tested. A few tests I made seem to confirm that assumption.

It seems like I've got 2 options. The first one, reinitialization of the file system is kind of a format in the flash, thus it doesn't serve my cause.
As far as the second option is concerned - reclaiming end of the file system - I would appreciate it, if you could please assist me on how I can achieve this. In the manual, I have found that claim, but no indication of how to achieve it...So, since you said that you actually managed to make some (successful) tests, could you lend a helping hand?

Thanks in advance!

[Roberthh]

Whether space at the End of the file system is reclaimed has to be tested. A few tests I made seem to confirm that assumption.

That seems to be the case, if you remove the files at the end of the list given by os.listdir(), os.statvfs() reports an increase in free file space.

[manos]

Hey Roberthh,
Thanks for your reply!

I am not sure about what is it that I am doing wrong but this is not working for me.
I have created a 30KB text file, put it in flash using ampy and then I accessed the Micropython REPL of my Xbee.
There, I imported the os module and noticed that the result of os.statvfs('') is the same even after deleting the imported file (by using the os.remove). Also, the same result (that the 30KB flash memory is not freed after os.remove(30KBfile)) is confirmed by XCTU (and specifically by the file manager of XCTU).

Do you have any ideas or suggestions?

Is there any chance that ampy is creating that problem? I remember you suggested the pyboard.py resp. the new mpr.py tool of Damien.
Could you please indicate me any guide of how to install and use it, if there is one?

Thanks in advance

[Roberthh]

I'm not sure how the file system works and whether the mechanism I have seen is dependable. Was the files you removed the last one listed by uos.listdir()?

[manos]

Yes, my file was listed in os.listdir and it was the last file.
After os.remove it got removed from os.listdir too, but the bytes were not freed as statvfs showed.

[manos]

Well, at last some good progress!

I managed to delete the last file completely from flash (releasing the occupied flash memory) after upgrading the firmware.
In case someone else faces the same problem, update the firmware and then use:
import uos
uos.remove("filename")
as Roberthh has also claimed

Thanks again for all the help