SD card vs flash drive, speed?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
rankor
Posts: 38
Joined: Sun Nov 30, 2014 12:38 am

SD card vs flash drive, speed?

Post by rankor » Fri Jan 02, 2015 3:15 am

Is it much slower to have your libraries on the SD card than on the flash drive?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card vs flash drive, speed?

Post by dhylands » Fri Jan 02, 2015 5:05 am

I'm not sure, but alot will depend on the speed rating of the sdcard.

writing is usually the bottleneck with flash devices, not reading, so I imagine that reading from the internal flash will be much faster than reading from an sdcard, but you'd really need to benchmark it to be sure.

manitou
Posts: 73
Joined: Wed Feb 25, 2015 12:15 am

Re: SD card vs flash drive, speed?

Post by manitou » Mon May 25, 2015 12:59 am

IO performance of SD, SPI SD, and FLASH: (pyboard)
(taking advantage of the thread subject)
Here are some anecdotal results comparing write/read speeds of SD, SPI SD, and FLASH on pyboard. In the following thread,
http://forum.micropython.org/viewtopic. ... 3395#p3665
it was noted that the SD firmware (sdcard.c) disables IRQ's so pyb.millis() and pyb.micros() will not be accurate, so I have used timer2 to count microseconds. Comparing performance with pyb.micros() vs timer2, it also appears that "time is lost" when doing FLASH IO, perhaps too much being done in HAL_FLASH_IRQHandler() (further analysis required, or insights from developers). The following code was used to time IO operations.
https://github.com/manitou48/pyboard/bl ... sdperf2.py

SD performance varies by the make and model of the SD card. For these tests, I was using a SanDisk Ultra Plus 16GB microSD. The SPI SD tests used the same microSD in a SD carrier on a SD card breakout board (on a breadboard with jumpers to pyboard). The pyboard SPI SD driver (sdcard.py) uses a baudrate of 100000 hz for initial configuration, and then a default data baudrate of 1320000 hz (for SPI 1, max rate is 42mhz with power-of-2 prescale values, so default rate is actually 1.31250 MHz). Measurements were made writing/reading one or more 512-byte blocks, and timing a full open/write/close and open/read/close file operations.

Some nice graphs would be best, but the tabular data will have to do for now. Data rate is in megabits/second (mbs) based on timer2. Block is 512 bytes.

Code: Select all

                   data rates (mbs)
  test                            SDIO        SPI SD(@42mhz)       FLASH
-----------------------------------------------------------------------------------------------------
block write                       0.05            2.0                  0.06  mbs
block read                       11.6             5.8                 52 
40-block file create              0.05            1.3                  0.11
40-block file read                9.7             4.5                 45      mbs
The 40-block file data rates(write/read in megabits/sec mbs) for other SPI clocks are
.47/.91 mbs @1.3mhz, .38/2.4 mbs @5.25mhz, .84/3.3 mbs @10.5mhz, .92/4 mbs @21mhz

-------------------------------------------
NOTES:

1. The FLASH uses a 64Kbyte RAM cache so some block times can be quite fast (memcpy() speeds).

2. The firmware sdcard.c uses 4-bit parallel SDIO interface to the microSD. I might have expected higher block data rates ....

3. The pyb.micros() loss of time is evident on file/block write's. For example, pyb.micros() measure the 40-block FLASH file create at 744904 us, where timer2 reports 1258763 us.

4. Often at high SPI clock rates and breadboard connectors, errors may occur in the SPI data stream. I did verify some of the blocks read back and observed 0 errors (YMMV).

5. raw SPI peformance (unconnected)
Timing 1000 spi.send(buffer) (where buffer is a 1000-byte array), the pyboard's, SPI 1 performance at various SPI CLK speeds is

Code: Select all

       6115 ms 1.308258 mbs @ 2000000 Hz
       3067 ms 2.608412 mbs @ 3000000
       1544 ms 5.181347 mbs @ 6000000
        782 ms 10.23018 mbs @ 11000000
        403 ms 19.85112 mbs @ 21000000
        212 ms 37.73585 mbs @ 42000000 Hz
(mbs is megabits/second) The max speed for SPI 2 is 24 MHz.

-----------------------
EDITS:
1. instrumented HAL_FLASH_IRQHandler() with timer2 to measure time spent in handler -- 0. time is being lost elsewhere...

2. time appears to be lost in flash.c doing flash_erase(), part of the write sequence. sys_tick_get_microseconds() measures 1072 us, timer2 measures 156072 us for a 16KB sector. Perhaps, not surprisingly, during flash-erase any code running out of flash will be stalled, micros/millis updates will be lost during erase. RAM-based code could run ... For a 64KB-sector erase, sys_tick_get_microseconds() reports 1261 us, timer2 reports 576261 us.
Last edited by manitou on Fri Jun 26, 2015 12:14 am, edited 3 times in total.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: SD card vs flash drive, speed?

Post by pythoncoder » Tue May 26, 2015 6:05 am

Is it much slower to have your libraries on the SD card than on the flash drive?
In terms of running Python code I think this only has an impact when you execute the import statement: once the library is imported the code runs at full speed. I hope someone corrects me if I'm wrong but I believe the code is compiled to bytecode at this point. If so, the time for performing the import will also depend on factors such as compilation time and runtime of any initialisation code in the library.
Peter Hinch
Index to my micropython libraries.

hasham12
Posts: 1
Joined: Fri Jun 05, 2015 12:50 pm

Re: SD card vs flash drive, speed?

Post by hasham12 » Fri Jun 05, 2015 12:52 pm

yup i also face the same problem this must be sort out

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card vs flash drive, speed?

Post by dhylands » Fri Jun 05, 2015 3:25 pm

The speed of the flash writes will also vary depending on the block size and the location in the filesystem.

The internal flash file system is 112K comprised of 3x16K blocks and 1x64K block.

There is some caching which occurs. I believe that there is a timer IRQ which is used to trigger writing out of the cache, and you can force it by using the pyb.sync()

Each time a write to flash actually occurs, then the following is performed: the flash block (16 or 64K) is read into RAM, the RAM is modified, the flash page is erased, and then the modified memory is written out.

Erasing flash can only erase an entire 16K or 64K page.

So if you're just timing the write operation, you might get a wide variety of timings depending on the position of the blocks in the file system.
You might be measuring the time to cache the information in RAM, or you write may include some write operations if it crosses page boundaries.

So just pointing out that profiling the internal flash is a bit more involved.

bradstew
Posts: 41
Joined: Thu Nov 29, 2018 9:29 pm

Re: SD card vs flash drive, speed?

Post by bradstew » Fri Jan 04, 2019 4:58 pm

Why is the internal flash file system limited to 112K? The STM32F405RG has 1MB of flash memory. Looking at the linker file, it places end of memory at 512K (0x80000) Is there a reason why this wasn't set to 1M (0x100000)? This should add another 512K to the internal flash file size.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card vs flash drive, speed?

Post by dhylands » Fri Jan 04, 2019 9:06 pm

In order to make a change to a flash block, you need to read the flash block into memory, erase the flash block in the flash, make the change to your copy in RAM, and then write the flash block back out to flash.

The pyboard reserves the 64K CCM memory for this buffer which means it can only modify 64K or smaller flash blocks.

You can increase the filesystem size by 64K by changing this #if 0 to a #if 1 and using 64K from the last flash block:
https://github.com/micropython/micropyt ... .c#L51-L54

On devices with 1Mb of flash, I believe that its possible to double the filesystem size since there is an addition set of 3x16K+1x64K flash blocks.

But you need to make sure that the code doesn't expand across the 512K mark. This can easily happen if you start using frozen mpy files.

bradstew
Posts: 41
Joined: Thu Nov 29, 2018 9:29 pm

Re: SD card vs flash drive, speed?

Post by bradstew » Fri Jan 04, 2019 10:42 pm

Doubling of size is good. It's too small as it is. Frozen But it seems to me that the upper 512K could be used as a virtual flash file system. It would be a great place to store modules, most of which are under 64K. I understand the block sizes are 64K, but that shouldn't be a show stopper.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card vs flash drive, speed?

Post by dhylands » Sat Jan 05, 2019 4:04 pm

The problem is that the flask block sizes are 128K for most of the flash.

Post Reply