Flash write stutter

[ajvperth]

I experiences stuttering of up to 900 mSec while writing to the flash on ESP-01s with a Esp8266. Reading the forum it could be due to flash management. I appreciate if this is hardware related, but I can not accept that python is unavailable for 900 mSec. If this is the cause a async mechanism would be very nice. Version used = 1.19.1
The program I use to demonstrate this behavior is;

Code: Select all

import time, os
import gc, sys

machine.freq(160000000)
gc.collect()

times = [0]*500
t1    = 0
t2    = 0
n     = 0
f     = None

while True:
    t1 = time.ticks_ms()

    f = open('a.py', 'w')
    f.close()
            
    t2 = time.ticks_ms()
    times[n] = t2 - t1
    n += 1
    if n == 500: break
    gc.collect
print(times)
print(sys.version)

The output is a list with the duration of 500 tests. Most tests require approximately 70 mSec, a couple of samples require more then 600 mSec. I wrote this program with pre-allocated memory locations to exclude garbage collection

My output is:

Code: Select all

[6, 6, 6, 6, 5, 6, 6, 7, 7, 7, 6, 7, 7, 7, 7, 8, 8, 10, 8, 9, 9, 9, 8, 10, 9, 10, 10, 11, 12, 13, 12, 12, 12, 11, 13, 13, 443, 8, 8, 8, 9, 9, 68, 53, 55, 52, 55, 53, 54, 55, 55, 53, 53, 55, 55, 53, 56, 55, 52, 57, 56, 60, 54, 55, 54, 54, 56, 54, 54, 55, 55, 56, 55, 57, 54, 55, 55, 56, 54, 55, 55, 58, 56, 59, 57, 57, 58, 57, 59, 57, 59, 60, 59, 59, 58, 59, 59, 59, 63, 60, 62, 58, 59, 55, 60, 60, 60, 60, 58, 63, 62, 58, 61, 57, 58, 61, 59, 60, 59, 60, 62, 58, 63, 62, 62, 59, 60, 61, 61, 62, 61, 127, 591, 58, 55, 57, 56, 56, 56, 57, 56, 55, 58, 59, 55, 59, 57, 56, 58, 59, 60, 57, 60, 57, 57, 59, 60, 57, 58, 59, 59, 58, 58, 60, 59, 59, 60, 57, 59, 58, 61, 61, 60, 62, 59, 62, 60, 61, 61, 60, 63, 61, 60, 62, 60, 63, 60, 63, 64, 64, 61, 62, 61, 65, 62, 63, 62, 61, 65, 63, 63, 62, 59, 63, 65, 62, 63, 63, 64, 65, 62, 66, 64, 66, 65, 64, 66, 66, 64, 121, 66, 67, 64, 65, 65, 67, 66, 68, 67, 65, 68, 66, 66, 69, 66, 69, 67, 67, 70, 66, 68, 68, 66, 66, 69, 66, 67, 69, 68, 66, 68, 668, 59, 63, 61, 61, 62, 64, 64, 60, 64, 64, 62, 63, 62, 63, 63, 62, 64, 63, 63, 64, 62, 65, 64, 66, 66, 65, 63, 66, 64, 65, 67, 64, 65, 65, 66, 66, 67, 65, 63, 65, 66, 65, 65, 66, 66, 67, 68, 70, 67, 68, 70, 66, 70, 67, 67, 103, 69, 71, 68, 70, 68, 70, 68, 70, 69, 68, 69, 69, 67, 73, 70, 67, 71, 70, 72, 70, 72, 68, 70, 72, 71, 69, 71, 69, 70, 70, 71, 70, 70, 73, 71, 74, 72, 71, 74, 73, 74, 73, 72, 74, 73, 73, 72, 74, 74, 72, 73, 73, 72, 75, 75, 76, 74, 77, 73, 74, 73, 75, 672, 69, 67, 69, 71, 69, 69, 68, 67, 68, 69, 70, 67, 67, 69, 72, 68, 73, 71, 69, 70, 70, 70, 70, 69, 88, 72, 71, 70, 72, 69, 74, 71, 71, 70, 70, 72, 72, 70, 76, 71, 71, 74, 72, 78, 73, 74, 71, 71, 73, 72, 72, 74, 73, 72, 73, 75, 75, 74, 76, 75, 74, 74, 76, 76, 75, 76, 77, 74, 73, 77, 78, 75, 74, 77, 76, 76, 79, 77, 77, 77, 76, 78, 79, 75, 77, 76, 78, 78, 77, 76, 78, 79, 78, 77, 77, 77, 77, 79, 83, 78, 80, 79, 82, 79, 82, 79, 81, 80, 79, 83, 80, 82, 148, 82, 83, 81, 82, 79, 680, 75, 74, 74, 73, 74, 74, 72, 79, 75, 75]
3.4.0; MicroPython v1.19.1 on 2022-06-18

It is remarkable than the first 50 tests require 10 mSec, after this 60 mSec.
Is this resolvable?

[Roberthh]

According to the data sheet of e.g. a typical 4 MB flash chip, the sector erase times are 90 - 500ms, the block erase times are 300 - 1200ms, and the full erase take 30 - 60s. Normally, the actual times are at the shorter end of the range, but extended times may happen. if just a sector has to be written to an erased area, the write takes 0.7 - 2.4 ms.

[ajvperth]

Hello Roberthh,
thank you for your answer, as these erase times are hardware design related I have to accept this.
But what I can not accept is that micropython is stuttering, as you say up to 1200 mSec. I would like these erase times to be integrated in the asyncio module so my processing can continue while the flash is busy in the background, I assume this is technically possible. Or at least, I would like a function to check the availability of the flash. Maybe this is already available?

[ajvperth]

I do not think a feature to check the availability of the flash is implemented.
I looked at the class pyb.Flash method Flash.ioctl implementing os.AbstractBlockDev
Possibly op=3 can be used if sync returns True or False on busy

[Roberthh]

I do not expect that you can make a reliable estimate on how long a write will take. The best a code can do is knowing, whether a block erase is requested. That's in the block device driver modules/flashbdev.py and modesp.c. And while writing to flash happens, it cannot be read, meaning that no code can can be accessed on flash.
If you need faster write times, you have to use a different media, like a SD card, or external F-RAM devices.

[ajvperth]

My ESP-01s with 1 MB x 8 storage through SPI is of type 25S80.
Erase time, depending on sector size is between 40 and 250 mSec, the page programming time is 1 mSec.
During the write and erase cycle its status register can be read where bit 0 indicates ready / busy.
So, it is possible to implement reading the busy status using ioctl provided that the driver is not written in blocking mode.

thank you for sending me a pointer to the source of the driver. I am not a C programmer but I will have a look at it.

[Roberthh]

Are these the typical or the worst case times?
It's clear that you can read the status of the chip to tell, whether it is busy. But that's only one piece of the story, Again: While erase is ongoing, you cannot read the flash, and thus you cannot run code from flash. To be able to run other code during erase you have to suspend erase, run that code, and resume erase again. spi_flash_erase_sector() is in an espressif library, provided as binary file. So you cannot change it.

[jimmo]

(Looks like I just raced with Robert to write a very similar answer... will post this anyway)

I would like these erase times to be integrated in the asyncio module so my processing can continue while the flash is busy in the background, I assume this is technically possible.

As Robert said, the issue is that sometimes we need to do a page erase.

It would be really good to make this asynchronous from Python, however there are two fundamental issues:

1. First is that the filesystem drivers are written in C and are completely synchronous -- i.e. they call the block erase function on MicroPython block device synchronously. This is not an easy thing to solve without a significant re-write of the (third-party) filesystem driver. (See https://github.com/littlefs-project/littlefs) On some ports we can solve this with threading (unlikely to happen on ESP8266).

2. The more important issue though is that the ESP8266 is itself executing code (the actual firmware, not Python) from flash. So while the flash is busy, the ESP8266 cannot execute code at all and the CPU stalls. For firmware written in C, it is possible to put small bits of code into RAM such that it can continue executing, but this is not feasible for MicroPython. This is fairly much a universal limitation on microcontrollers -- you cannot execute while erasing flash. The only solution is to have a separate flash chip for the filesystem (but you still need to solve (1) above).

See also viewtopic.php?f=18&t=4725&hilit=filesys ... =10#p68540

[ajvperth]

Dear Roberthh and Jimmo,
thank you both very much for your answers, it increased certainly my insight in micropython.
the solution I will try is a external I2C flash to store my log data and using a async driver to communicate with it.

Thanks.