data write speeds and broken SD cards

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

data write speeds and broken SD cards

Post by Turbinenreiter » Thu Sep 11, 2014 12:54 pm

I just wan't to vent my frustration over the SD card connector. I just broke an SD card, again, losing some data, again.
I guess my usecase of the board isn't that common, but it sometimes involves crashing to the ground from 100 feet (because f*cking parachute's not deploying).
SD cards don't surive that. And sadly you can't read cracked SD cards.
I get there is a pretty good reason to use the type of connector that is used - board space, but still, I hope the next revision will use the other kind, where you plug in the card all the way.

However, with that experience, I'm thinking about what I can do.
I ordered an microSD extension cable - this will allow me to mount everyhing differently and move the card to a saver place. Also the female on this encloses the card completly.

The other thing is to use the flash as storage for the data logs. But the flash is much slower. With the sd card I get about 30Hz (measuring with 100Hz, but only storing every third measurement) - the same code storing to flash gets about 15Hz.

The code I use for writting to the file looks somewhat like this (but with 20 different variables instead of just 3):

Code: Select all

log = open('file.csv', 'w')
while condition:
    a,b,c = get_measurements()
    log.write('{},{},{}'.format(a,b,c))
log.close()
Now, how do I make this faster? I just want to store fast, the format is completley irrelevant - I have all the time in the world to read it later. Would it be faster to store in a binary format? And how? I guess the string formatting eats cycles, but file.write() won't take a list of floats as arg. It would also be ok to just store ints instead of floats.
What I also tried is to append each new measurment to a list, and then write the whole list to the file, but the length of the list is limited (I remeber about 50 measurments of 5 variables) and then you have a gap between every block of measurements, which only gets longer the longer you make your buffer.

The thing is that it's not python who limits my speed. It's the writing to the file.

So, any ideas are welcome.

Oh, and as more positive sidenote:
The pyboard handles crashing from 100 feet without any damage. Like a boss. Which is awesome.

blmorris
Posts: 348
Joined: Fri May 02, 2014 3:43 pm
Location: Massachusetts, USA

Re: data write speeds and broken SD cards

Post by blmorris » Thu Sep 11, 2014 2:07 pm

Have you updated your uPy firmware recently? I'm not aware of any major changes in writing to flash or SD filesystems, (there may have been improvements that I'm not aware of), but the changes to the garbage collector have tremendously improved memory allocation performance - this might speed up your string formatting functions.
You could try preemptively calling the garbage collector if that is making you miss measurements.

Also, binary read and write works on the pyboard, you need to use options 'rb' and 'wb'.
I just tried the following commands on mine:

Code: Select all

Micro Python v1.3.1-26-gbad2df3 on 2014-09-02; PYBv1.0 with STM32F405RG
Type "help()" for more information.
>>> log = open('file.bin', 'wb')
>>> b = bytes([1,2,3,4,5,6,7,8,9])
>>> log.write(b)
9
>>> log.close()
>>> log = open('file.bin', 'rb')
>>> l = log.read()
>>> l
b'\x01\x02\x03\x04\x05\x06\x07\x08\t'
>>> 
Not sure why the '9' is returned and not written to the file, maybe someone else has an explanation.

I imagine that implementing DMA on the SD card driver would help a lot, but I don't know how much of a priority this is for the developers yet.
And I love knowing that your pyboard is getting launched by a rocket. Awesome!

User avatar
andrew
Posts: 22
Joined: Sun Aug 10, 2014 9:22 am

Re: data write speeds and broken SD cards

Post by andrew » Thu Sep 11, 2014 2:16 pm

blmorris wrote:...

>>> l
b'\x01\x02\x03\x04\x05\x06\x07\x08\t'
>>>
[/code]
Not sure why the '9' is returned and not written to the file, maybe someone else has an explanation.
...
I imagine the 9 that's returned from the log.write command is the number of bytes written.
Your byte 9 is written to the file, it's just being displayed as a tab character ('\t') when printed out.

Andrew

blmorris
Posts: 348
Joined: Fri May 02, 2014 3:43 pm
Location: Massachusetts, USA

Re: data write speeds and broken SD cards

Post by blmorris » Thu Sep 11, 2014 2:40 pm

Andrew- good catch, thanks. Yes, binary read and write does seem to work as expected.
-Bryan

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Thu Sep 11, 2014 3:18 pm

aaaah, 'rb' - obviously.

I call gc.collect() in every iteration.
Will update and experiment a bit.

And hey - I never said anything about a rocket!
It's true, however. I will post more about that to the forums once it's done.

User avatar
dhylands
Posts: 3228
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: data write speeds and broken SD cards

Post by dhylands » Thu Sep 11, 2014 3:38 pm

As far as broken sdcatds go, can't you just add mechanical support? I.e create a small board which puts some support under the portion of the sdcard that is over-hanging?

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Thu Sep 11, 2014 3:57 pm

Generally yes, in my case - nah.
I'm going to place everything a bit differently and use a extension cable. That way the card is save AND accessible.

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Fri Sep 12, 2014 8:44 am

/edit
WARNING:
Writting floats doesn't write the values correctly. See posts below.


Ok, I ran some tests:

Code: Select all

file type, storage type, data type
binary, flash, int:     1000 lines 1.88 s 529 Hz
binary, flash, float:   1000 lines 1.89 s 528 Hz

string, flash, int:      1000 lines 4.72 s 211 Hz
string, flash, float:    1000 lines 5.79 s 172 Hz

binary, sd, int:         1000 lines 0.05 s 21256 Hz
binary, sd, float:       1000 lines 0.05 s 20810 Hz

string, sd, int:          1000 lines 0.48 s  2055 Hz
string, sd, float:        1000 lines 0.37 s  2700 Hz
Ran them a couple of times, the results didn't vary much.
  • int is little faster than float, but not much
  • binary is way faster than text, 2x faster on flash, 10x faster on sd
  • sd is way faster than flash, 20x faster in binary, 10x faster in string
So, if you need to store fast:
  • store to sd
  • store in binary format
  • store int or float, ints are not faster than floats, so converting is a waste of cycles
I also ran the fastest option (binary sd int) with the native emitter:

Code: Select all

binary sd int native:           1000 lines 0.03796387 s 26314.49 Hz
A nice 20% speed bump for free.

Viper however didn't run, unless I removed the file operations, so it seems like viper can't write to files. Also for viper you typically have to make some changes. 1+1.0 doesn't work, only 1+1 or 1.0+1.0. When the function without file writing in native and viper, viper gave a speed pump of about 7%.

test code:

Code: Select all

# main.py -- put your code here!
import pyb
import gc

def write_binary_flash(inp, no):
    t_start = pyb.millis()/1000
    log = open('/flash/testlog.bin', 'wb')
    for i in range(no):
        b = bytes([i, inp, '\n'])
        log.write(b)
    log.close()
    dt = pyb.millis()/1000 - t_start
    print(i+1, 'lines', dt, 's', i/dt, 'Hz')

def write_str_flash(inp, no):
    t_start = pyb.millis()/1000
    log = open('/flash/testlog.txt', 'w')
    for i in range(no):
        string = '{}{}\n'.format(i, inp)
        log.write(string)
    log.close()
    dt = pyb.millis()/1000 - t_start
    print(i+1, 'lines', dt, 's', i/dt, 'Hz')

def write_binary_sd(inp, no):
    t_start = pyb.millis()/1000
    log = open('/sd/testlog.bin', 'wb')
    for i in range(no):
        b = bytes([i, inp, '\n'])
        log.write(b)
    log.close()
    dt = pyb.millis()/1000 - t_start
    print(i+1, 'lines', dt, 's', i/dt, 'Hz')

def write_str_sd(inp, no):
    t_start = pyb.millis()/1000
    log = open('/sd/testlog.txt', 'w')
    for i in range(no):
        string = '{}{}\n'.format(i, inp)
        log.write(string)
    log.close()
    dt = pyb.millis()/1000 - t_start
    print(i+1, 'lines', dt, 's', i/dt, 'Hz')

@micropython.native
def write_binary_sd_native(inp, no):
    t_start = pyb.millis()/1000
    log = open('/sd/testlog.bin', 'wb')
    for i in range(no):
        b = bytes([i, inp, '\n'])
        log.write(b)
    log.close()
    dt = pyb.millis()/1000 - t_start
    print(i+1, 'lines', dt, 's', i/dt, 'Hz')

@micropython.viper
def write_binary_sd_viper(inp:int, no:int):
    t_start = pyb.millis()/1000.0
#    log = open('/sd/testlog.bin', 'wb')
    for i in range(no):
        b = bytes([i, inp, '\n'])
#        log.write(b)
#    log.close()
    dt = pyb.millis()/1000.0 - t_start
    ifl = float(i)
    print(i+1, 'lines', dt, 's', ifl/dt, 'Hz')

print('binary flash:', end='\t\t')
gc.collect()
write_binary_flash(973, 1000)
print('string flash:', end='\t\t')
gc.collect()
write_str_flash(973, 1000)
print('binary flash float:', end='\t')
gc.collect()
write_binary_flash(973.1, 1000)
print('string flash float:', end='\t')
gc.collect()
write_str_flash(973.1, 1000)
print('binary sd:', end='\t\t')
gc.collect()
write_binary_sd(973, 1000)
print('string sd:', end='\t\t')
gc.collect()
write_str_sd(973, 1000)
print('binary sd float:', end='\t')
gc.collect()
write_binary_sd(973.1, 1000)
print('string sd float:', end='\t')
gc.collect()
write_str_sd(973.1, 1000)
print('binary sd int native:', end='\t')
gc.collect()
write_binary_sd_native(973, 1000)
print('binary sd int viper:', end='\t')
gc.collect()
write_binary_sd_viper(973, 1000)
print('but viper can\'t write files')
Last edited by Turbinenreiter on Mon Sep 15, 2014 8:57 am, edited 1 time in total.

blmorris
Posts: 348
Joined: Fri May 02, 2014 3:43 pm
Location: Massachusetts, USA

Re: data write speeds and broken SD cards

Post by blmorris » Fri Sep 12, 2014 3:47 pm

Looks like you have been able to speed things up quite a bit - great!
However, there is a problem passing floating point objects to 'bytes()' which I found after my last post; I tried to pass a list of floats to bytes() the same way that you did and the file just didn't look right. It seems that bytes() really only accepts list of 8-bit integers (either decimal or hex strings) or ASCII characters and escape sequences (I'm probably not getting this exactly right…)
When another type of object is passed to bytes(), only a single byte gets stored; I suspect that this single byte may be the low-order byte of the pointer to that object, but I'm not certain of this.
You may need to figure out a way to convert your floats to integers or just go ahead with storing strings anyway. There must be a way to convert the 32-bit value from a float to a set of 4 bytes, but I haven't found it yet.
Good luck, sorry if I have accidentally sent you down a blind alley, I'm just starting to figure out the more subtle aspects of this myself. I'd hate to be responsible for losing a set of launch data.
Cheers, Bryan

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Mon Sep 15, 2014 8:55 am

Yeah, I never looked at the files the tests wrote.
But it doesn't really matter. The sensor-data I read over I2C comes in as Bytes. So I will just store those Bytes - and save even more cycles because I don't have to unpack them.

Post Reply