SD card vs flash drive, speed?
SD card vs flash drive, speed?
Is it much slower to have your libraries on the SD card than on the flash drive?
Re: SD card vs flash drive, speed?
I'm not sure, but alot will depend on the speed rating of the sdcard.
writing is usually the bottleneck with flash devices, not reading, so I imagine that reading from the internal flash will be much faster than reading from an sdcard, but you'd really need to benchmark it to be sure.
writing is usually the bottleneck with flash devices, not reading, so I imagine that reading from the internal flash will be much faster than reading from an sdcard, but you'd really need to benchmark it to be sure.
Re: SD card vs flash drive, speed?
IO performance of SD, SPI SD, and FLASH: (pyboard)
(taking advantage of the thread subject)
Here are some anecdotal results comparing write/read speeds of SD, SPI SD, and FLASH on pyboard. In the following thread,
http://forum.micropython.org/viewtopic. ... 3395#p3665
it was noted that the SD firmware (sdcard.c) disables IRQ's so pyb.millis() and pyb.micros() will not be accurate, so I have used timer2 to count microseconds. Comparing performance with pyb.micros() vs timer2, it also appears that "time is lost" when doing FLASH IO, perhaps too much being done in HAL_FLASH_IRQHandler() (further analysis required, or insights from developers). The following code was used to time IO operations.
https://github.com/manitou48/pyboard/bl ... sdperf2.py
SD performance varies by the make and model of the SD card. For these tests, I was using a SanDisk Ultra Plus 16GB microSD. The SPI SD tests used the same microSD in a SD carrier on a SD card breakout board (on a breadboard with jumpers to pyboard). The pyboard SPI SD driver (sdcard.py) uses a baudrate of 100000 hz for initial configuration, and then a default data baudrate of 1320000 hz (for SPI 1, max rate is 42mhz with power-of-2 prescale values, so default rate is actually 1.31250 MHz). Measurements were made writing/reading one or more 512-byte blocks, and timing a full open/write/close and open/read/close file operations.
Some nice graphs would be best, but the tabular data will have to do for now. Data rate is in megabits/second (mbs) based on timer2. Block is 512 bytes.
The 40-block file data rates(write/read in megabits/sec mbs) for other SPI clocks are
.47/.91 mbs @1.3mhz, .38/2.4 mbs @5.25mhz, .84/3.3 mbs @10.5mhz, .92/4 mbs @21mhz
-------------------------------------------
NOTES:
1. The FLASH uses a 64Kbyte RAM cache so some block times can be quite fast (memcpy() speeds).
2. The firmware sdcard.c uses 4-bit parallel SDIO interface to the microSD. I might have expected higher block data rates ....
3. The pyb.micros() loss of time is evident on file/block write's. For example, pyb.micros() measure the 40-block FLASH file create at 744904 us, where timer2 reports 1258763 us.
4. Often at high SPI clock rates and breadboard connectors, errors may occur in the SPI data stream. I did verify some of the blocks read back and observed 0 errors (YMMV).
5. raw SPI peformance (unconnected)
Timing 1000 spi.send(buffer) (where buffer is a 1000-byte array), the pyboard's, SPI 1 performance at various SPI CLK speeds is(mbs is megabits/second) The max speed for SPI 2 is 24 MHz.
-----------------------
EDITS:
1. instrumented HAL_FLASH_IRQHandler() with timer2 to measure time spent in handler -- 0. time is being lost elsewhere...
2. time appears to be lost in flash.c doing flash_erase(), part of the write sequence. sys_tick_get_microseconds() measures 1072 us, timer2 measures 156072 us for a 16KB sector. Perhaps, not surprisingly, during flash-erase any code running out of flash will be stalled, micros/millis updates will be lost during erase. RAM-based code could run ... For a 64KB-sector erase, sys_tick_get_microseconds() reports 1261 us, timer2 reports 576261 us.
(taking advantage of the thread subject)
Here are some anecdotal results comparing write/read speeds of SD, SPI SD, and FLASH on pyboard. In the following thread,
http://forum.micropython.org/viewtopic. ... 3395#p3665
it was noted that the SD firmware (sdcard.c) disables IRQ's so pyb.millis() and pyb.micros() will not be accurate, so I have used timer2 to count microseconds. Comparing performance with pyb.micros() vs timer2, it also appears that "time is lost" when doing FLASH IO, perhaps too much being done in HAL_FLASH_IRQHandler() (further analysis required, or insights from developers). The following code was used to time IO operations.
https://github.com/manitou48/pyboard/bl ... sdperf2.py
SD performance varies by the make and model of the SD card. For these tests, I was using a SanDisk Ultra Plus 16GB microSD. The SPI SD tests used the same microSD in a SD carrier on a SD card breakout board (on a breadboard with jumpers to pyboard). The pyboard SPI SD driver (sdcard.py) uses a baudrate of 100000 hz for initial configuration, and then a default data baudrate of 1320000 hz (for SPI 1, max rate is 42mhz with power-of-2 prescale values, so default rate is actually 1.31250 MHz). Measurements were made writing/reading one or more 512-byte blocks, and timing a full open/write/close and open/read/close file operations.
Some nice graphs would be best, but the tabular data will have to do for now. Data rate is in megabits/second (mbs) based on timer2. Block is 512 bytes.
Code: Select all
data rates (mbs)
test SDIO SPI SD(@42mhz) FLASH
-----------------------------------------------------------------------------------------------------
block write 0.05 2.0 0.06 mbs
block read 11.6 5.8 52
40-block file create 0.05 1.3 0.11
40-block file read 9.7 4.5 45 mbs
.47/.91 mbs @1.3mhz, .38/2.4 mbs @5.25mhz, .84/3.3 mbs @10.5mhz, .92/4 mbs @21mhz
-------------------------------------------
NOTES:
1. The FLASH uses a 64Kbyte RAM cache so some block times can be quite fast (memcpy() speeds).
2. The firmware sdcard.c uses 4-bit parallel SDIO interface to the microSD. I might have expected higher block data rates ....
3. The pyb.micros() loss of time is evident on file/block write's. For example, pyb.micros() measure the 40-block FLASH file create at 744904 us, where timer2 reports 1258763 us.
4. Often at high SPI clock rates and breadboard connectors, errors may occur in the SPI data stream. I did verify some of the blocks read back and observed 0 errors (YMMV).
5. raw SPI peformance (unconnected)
Timing 1000 spi.send(buffer) (where buffer is a 1000-byte array), the pyboard's, SPI 1 performance at various SPI CLK speeds is
Code: Select all
6115 ms 1.308258 mbs @ 2000000 Hz
3067 ms 2.608412 mbs @ 3000000
1544 ms 5.181347 mbs @ 6000000
782 ms 10.23018 mbs @ 11000000
403 ms 19.85112 mbs @ 21000000
212 ms 37.73585 mbs @ 42000000 Hz
-----------------------
EDITS:
1. instrumented HAL_FLASH_IRQHandler() with timer2 to measure time spent in handler -- 0. time is being lost elsewhere...
2. time appears to be lost in flash.c doing flash_erase(), part of the write sequence. sys_tick_get_microseconds() measures 1072 us, timer2 measures 156072 us for a 16KB sector. Perhaps, not surprisingly, during flash-erase any code running out of flash will be stalled, micros/millis updates will be lost during erase. RAM-based code could run ... For a 64KB-sector erase, sys_tick_get_microseconds() reports 1261 us, timer2 reports 576261 us.
Last edited by manitou on Fri Jun 26, 2015 12:14 am, edited 3 times in total.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: SD card vs flash drive, speed?
In terms of running Python code I think this only has an impact when you execute the import statement: once the library is imported the code runs at full speed. I hope someone corrects me if I'm wrong but I believe the code is compiled to bytecode at this point. If so, the time for performing the import will also depend on factors such as compilation time and runtime of any initialisation code in the library.Is it much slower to have your libraries on the SD card than on the flash drive?
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: SD card vs flash drive, speed?
yup i also face the same problem this must be sort out
Re: SD card vs flash drive, speed?
The speed of the flash writes will also vary depending on the block size and the location in the filesystem.
The internal flash file system is 112K comprised of 3x16K blocks and 1x64K block.
There is some caching which occurs. I believe that there is a timer IRQ which is used to trigger writing out of the cache, and you can force it by using the pyb.sync()
Each time a write to flash actually occurs, then the following is performed: the flash block (16 or 64K) is read into RAM, the RAM is modified, the flash page is erased, and then the modified memory is written out.
Erasing flash can only erase an entire 16K or 64K page.
So if you're just timing the write operation, you might get a wide variety of timings depending on the position of the blocks in the file system.
You might be measuring the time to cache the information in RAM, or you write may include some write operations if it crosses page boundaries.
So just pointing out that profiling the internal flash is a bit more involved.
The internal flash file system is 112K comprised of 3x16K blocks and 1x64K block.
There is some caching which occurs. I believe that there is a timer IRQ which is used to trigger writing out of the cache, and you can force it by using the pyb.sync()
Each time a write to flash actually occurs, then the following is performed: the flash block (16 or 64K) is read into RAM, the RAM is modified, the flash page is erased, and then the modified memory is written out.
Erasing flash can only erase an entire 16K or 64K page.
So if you're just timing the write operation, you might get a wide variety of timings depending on the position of the blocks in the file system.
You might be measuring the time to cache the information in RAM, or you write may include some write operations if it crosses page boundaries.
So just pointing out that profiling the internal flash is a bit more involved.
Re: SD card vs flash drive, speed?
Why is the internal flash file system limited to 112K? The STM32F405RG has 1MB of flash memory. Looking at the linker file, it places end of memory at 512K (0x80000) Is there a reason why this wasn't set to 1M (0x100000)? This should add another 512K to the internal flash file size.
Re: SD card vs flash drive, speed?
In order to make a change to a flash block, you need to read the flash block into memory, erase the flash block in the flash, make the change to your copy in RAM, and then write the flash block back out to flash.
The pyboard reserves the 64K CCM memory for this buffer which means it can only modify 64K or smaller flash blocks.
You can increase the filesystem size by 64K by changing this #if 0 to a #if 1 and using 64K from the last flash block:
https://github.com/micropython/micropyt ... .c#L51-L54
On devices with 1Mb of flash, I believe that its possible to double the filesystem size since there is an addition set of 3x16K+1x64K flash blocks.
But you need to make sure that the code doesn't expand across the 512K mark. This can easily happen if you start using frozen mpy files.
The pyboard reserves the 64K CCM memory for this buffer which means it can only modify 64K or smaller flash blocks.
You can increase the filesystem size by 64K by changing this #if 0 to a #if 1 and using 64K from the last flash block:
https://github.com/micropython/micropyt ... .c#L51-L54
On devices with 1Mb of flash, I believe that its possible to double the filesystem size since there is an addition set of 3x16K+1x64K flash blocks.
But you need to make sure that the code doesn't expand across the 512K mark. This can easily happen if you start using frozen mpy files.
Re: SD card vs flash drive, speed?
Doubling of size is good. It's too small as it is. Frozen But it seems to me that the upper 512K could be used as a virtual flash file system. It would be a great place to store modules, most of which are under 64K. I understand the block sizes are 64K, but that shouldn't be a show stopper.
Re: SD card vs flash drive, speed?
The problem is that the flask block sizes are 128K for most of the flash.