data write speeds and broken SD cards
Posted: Thu Sep 11, 2014 12:54 pm
I just wan't to vent my frustration over the SD card connector. I just broke an SD card, again, losing some data, again.
I guess my usecase of the board isn't that common, but it sometimes involves crashing to the ground from 100 feet (because f*cking parachute's not deploying).
SD cards don't surive that. And sadly you can't read cracked SD cards.
I get there is a pretty good reason to use the type of connector that is used - board space, but still, I hope the next revision will use the other kind, where you plug in the card all the way.
However, with that experience, I'm thinking about what I can do.
I ordered an microSD extension cable - this will allow me to mount everyhing differently and move the card to a saver place. Also the female on this encloses the card completly.
The other thing is to use the flash as storage for the data logs. But the flash is much slower. With the sd card I get about 30Hz (measuring with 100Hz, but only storing every third measurement) - the same code storing to flash gets about 15Hz.
The code I use for writting to the file looks somewhat like this (but with 20 different variables instead of just 3):
Now, how do I make this faster? I just want to store fast, the format is completley irrelevant - I have all the time in the world to read it later. Would it be faster to store in a binary format? And how? I guess the string formatting eats cycles, but file.write() won't take a list of floats as arg. It would also be ok to just store ints instead of floats.
What I also tried is to append each new measurment to a list, and then write the whole list to the file, but the length of the list is limited (I remeber about 50 measurments of 5 variables) and then you have a gap between every block of measurements, which only gets longer the longer you make your buffer.
The thing is that it's not python who limits my speed. It's the writing to the file.
So, any ideas are welcome.
Oh, and as more positive sidenote:
The pyboard handles crashing from 100 feet without any damage. Like a boss. Which is awesome.
I guess my usecase of the board isn't that common, but it sometimes involves crashing to the ground from 100 feet (because f*cking parachute's not deploying).
SD cards don't surive that. And sadly you can't read cracked SD cards.
I get there is a pretty good reason to use the type of connector that is used - board space, but still, I hope the next revision will use the other kind, where you plug in the card all the way.
However, with that experience, I'm thinking about what I can do.
I ordered an microSD extension cable - this will allow me to mount everyhing differently and move the card to a saver place. Also the female on this encloses the card completly.
The other thing is to use the flash as storage for the data logs. But the flash is much slower. With the sd card I get about 30Hz (measuring with 100Hz, but only storing every third measurement) - the same code storing to flash gets about 15Hz.
The code I use for writting to the file looks somewhat like this (but with 20 different variables instead of just 3):
Code: Select all
log = open('file.csv', 'w')
while condition:
a,b,c = get_measurements()
log.write('{},{},{}'.format(a,b,c))
log.close()
What I also tried is to append each new measurment to a list, and then write the whole list to the file, but the length of the list is limited (I remeber about 50 measurments of 5 variables) and then you have a gap between every block of measurements, which only gets longer the longer you make your buffer.
The thing is that it's not python who limits my speed. It's the writing to the file.
So, any ideas are welcome.
Oh, and as more positive sidenote:
The pyboard handles crashing from 100 feet without any damage. Like a boss. Which is awesome.