data write speeds and broken SD cards

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Thu Oct 30, 2014 11:12 am

24 Bytes per cycle, 1k cycles per second - I would fill this up in less than two seconds.

Last thing I worked on: http://fovs.de/category/general/
Rockets are awesome.
Solid fuels motors have very high g-loads (tough water rockets have higher loads - but only very short, about 0.2 seconds) and pretty nasty vibrations. That said, in our experiment that wasn't to big of a problem.

User avatar
pythoncoder
Posts: 3655
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: data write speeds and broken SD cards

Post by pythoncoder » Thu Oct 30, 2014 5:38 pm

Awesome! I had no idea your rockets were on that scale.

Regards, Pete
Peter Hinch

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Thu Oct 30, 2014 6:11 pm

Oh, no they aren't.
What I'm using the pyboard for are water rockets. On the big one we had something like an Atmeg8 - but as mechanical engineer didn't work with that, I just build the structure.

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Tue Dec 09, 2014 8:52 pm

Ok, so on the issue of the breaking SD cards and breaking SD card connectors (#2 today :( )

Is it possible to remap the SD card to other pins, so I can make a little daugtherboard to stack on top of the pyboard, and have the sd card connector there?

https://github.com/micropython/micropyt ... l/sdcard.c

Code: Select all

void sdcard_init(void) {
    GPIO_InitTypeDef GPIO_Init_Structure;

    // invalidate the sd_handle
    sd_handle.Instance = NULL;

    // configure SD GPIO
    // we do this here an not in HAL_SD_MspInit because it apparently
    // makes it more robust to have the pins always pulled high
    GPIO_Init_Structure.Mode = GPIO_MODE_AF_PP;
    GPIO_Init_Structure.Pull = GPIO_PULLUP;
    GPIO_Init_Structure.Speed = GPIO_SPEED_HIGH;
    GPIO_Init_Structure.Alternate = GPIO_AF12_SDIO;
    GPIO_Init_Structure.Pin = GPIO_PIN_8 | GPIO_PIN_9 | GPIO_PIN_10 | GPIO_PIN_11 | GPIO_PIN_12;
    HAL_GPIO_Init(GPIOC, &GPIO_Init_Structure);
    GPIO_Init_Structure.Pin = GPIO_PIN_2;
    HAL_GPIO_Init(GPIOD, &GPIO_Init_Structure);

    // configure the SD card detect pin
    // we do this here so we can detect if the SD card is inserted before powering it on
    GPIO_Init_Structure.Mode = GPIO_MODE_INPUT;
    GPIO_Init_Structure.Pull = MICROPY_HW_SDCARD_DETECT_PULL;
    GPIO_Init_Structure.Speed = GPIO_SPEED_HIGH;
    GPIO_Init_Structure.Pin = MICROPY_HW_SDCARD_DETECT_PIN.pin_mask;
    HAL_GPIO_Init(MICROPY_HW_SDCARD_DETECT_PIN.gpio, &GPIO_Init_Structure);
}
Am I rigth in thinking that this is the part I'd need to change to remap?

edit:
Answer: No, as this uses SDIO, SD card on the header pins would have to use SPI - SD card over SPI hast yet to be implemented.

Damien
Site Admin
Posts: 622
Joined: Mon Dec 09, 2013 5:02 pm

Re: data write speeds and broken SD cards

Post by Damien » Tue Dec 09, 2014 10:49 pm

Answer: No, as this uses SDIO, SD card on the header pins would have to use SPI - SD card over SPI hast yet to be implemented.
Right. The hardware has builtin SDIO support and the driver is designed around that. The pins are hardwired.

It's not a priority for me, but if someone wants to submit a driver for SD over SPI, then we can incorporate that into master. With such a driver, we could "mount" SD cards over SPI in the root directory, something like:

Code: Select all

>>> pyb.mount_sd(pyb.SPI(1))
>>> os.listdir('/')
['flash', 'sd', 'spi1']

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Mon Jan 12, 2015 7:43 pm

Thanks to the sheer awesomeness of our beloved devs, sd via spi is now implemented.

It's also been 16 days since this was implemented, but I only discovered it today, despite following the project on Github, which makes me think we have to find a way to better communicate such stuff WITHOUT putting that strain on the devs. Maybe I will try to make a 'what's new' post on the forum once a week or something like that. I think if you sit down once a week and go through the project, closed pulls and issues, you can get that info quiet easily. In fact, I will start right now.

Martin
Posts: 7
Joined: Tue May 05, 2015 5:11 pm

Re: data write speeds and broken SD cards

Post by Martin » Tue May 05, 2015 5:36 pm

@Turbinenreiter: I pretty much want to do the same thing, high speed logging of accelerometer data onto the SD card. Would you share your code? Would help me quite a bit - I'm new to pyboard!

BTW I want to measure the effectiveness of my bike suspension fork. I intend to place the accelerometer at the top of the fork and see how resulting acceleration changes depending on air pressure and fork type. Without having tried, I assume the acceleration is beyond the +-8 or 16 g typical for most sensors, so I ordered a few ADXL375 (+-200g) instead. Here http://www.veloplus.ch/pdf/veloplus/wei ... sattel.pdf is an interesting measurement of acceleration - on your back, depending on seatpost suspension. I just purchased an ADXL345 breakout board for starters, it is pin (and maybe software) compatible with the ADXL375. For connecting the part, I'm not sure whether I just fix the ADXL375 upside down on a small PCB and solder thin wires directly at the part or remove the ADXL345 from the breakout board and solder the ADXL375 instead - I have equipment for both, I'll see...
If I would do this in C on the STM32, I'd have the following structure:
- Initialize the sensor for 3200 Hz output rate, use FiFo, generate Interrupt when FiFo is full
- interface via SPI (> 2 MHz)
- In the interrupt routine, do 32 times an SPI read (6 bytes each for XYZ 16 bit data), maybe DMA (but not necessarily), unfortunately you can't do a single read for reading out the FiFo and unfortunately2 SPI reads at 2 MHz require an additional delay between reads as the FiFo is so slow.
- Write data in a buffer, when 512 bytes are full, write to SD card (SPI) via DMA.

Not sure if this can be done on the pyboard. Of course a few missed samples would't be the end of the world in this application, but most elegant would be if a seamless recording would work. I am open to suggestions here...

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Wed May 06, 2015 11:19 am

@Martin

I managed to get about 900Hz at one time. I since moved to using a Teensy for other reasons.
The main trick to log fast is quite simple - store in binary format and know the buffer size of the sd card.

I don't have much code to share - it's basically a loop with a single

Code: Select all

log.write(b'')
in it. Actually, the SDDatalogger in the example section is the first version of my logger.

I'm also going to use the ADXL375 in the next version.

The only challenge in doing this the way you describe is that you can't allocate memory on the heap in an interrupt routine, but I think it should be possible to do all that without allocating.

When I started with logging, I only got about 50Hz. I managed to speed it up to about 900Hz quickly, but what I was missing the most was profiling tools, like profile or line_profiler.

Code: Select all

# datalogger.py
# Logs the data from the acceleromter to a file on the SD-card

import struct
import pyb
from bmp180 import BMP180
from mpu9150 import MPU9150
from rcv import Reciever
import os
import gc
import sys

def main():

    # create objects
    accel = pyb.Accel()
    blue = pyb.LED(4)
    switch = pyb.Switch()
    cam = pyb.Servo(2)
    pyb.delay(10)
    baro = BMP180('X')
    pyb.delay(10)
    imu = MPU9150('X', 1)
    rcv = Reciever()

    # settings
    baro.oversample_sett = 3
    #servo.calibration(860, 2160, 1500)

    # test i2c
    if len(baro._bmp_i2c.scan()) != 4:
        pyb.LED(2).on()
        print('sensors failed')
    else:
        print('sensors ready')

    # create dir
    if 'log' not in os.listdir('/sd/'):
        os.mkdir('/sd/log')
        print('created log dir')
    else:
        print('log dir already there')

    config = open('/sd/log/config.txt', 'w')
    config.write('AC1, AC2, AC3, AC4, AC5, AC6, B1, B2, MB, MC, MD, oversample_sett, accel_range, gyro_range\n')
    con = baro.compvaldump() + [imu.accel_range(), imu.gyro_range()]
    config.write('{},{},{},{},{},{},{},{},{},{},{},{},{},{}'.format(*con))
    config.close()
    print('wrote config file')

    # loop
    while True:

        # wait for interrupt
        pyb.wfi()

        # start if switch is pressed
        if (rcv.ch[1][0] < -45) or switch():                            # switch(): rcv.ch[1][0] < -45
            pyb.delay(300)                      # delay avoids detection of multiple presses
            blue.on()                           # blue LED indicates file open
            pyb.LED(3).off()
            filename = '/sd/log/log'+str(len(os.listdir('log'))+1)+'.bin'
            print('opening '+filename)
            log = open(filename, 'wb')
            i = 0
            trigger = b'\x00'
            cam.angle(90)
            pyb.delay(100)
            cam.angle(0)
            gc.collect()
            t0 = pyb.millis()/1000
            # until switch is pressed again
            while (rcv.ch[1][0] < 45) and (not switch()):                      #not switch(): rcv.ch[1][0] < 45

                t_start_ms = pyb.millis()
                next(baro.gauge())
                log.write(b''.join([struct.pack('I', t_start_ms),
                                    bytes(accel.filtered_xyz()),
                                    imu.get_accel_raw(),
                                    imu.get_gyro_raw(),
                                    baro.UT_raw,
                                    baro.MSB_raw,
                                    baro.LSB_raw,
                                    baro.XLSB_raw,
                                    trigger]))
                i = i+1
                if i%100 == 0:
                    trigger = para(baro.altitude)
                    dt = t_start_ms/1000-t0
                    sys.stdout.write('writing '+
                                     filename+
                                     ' %d lines, %d s, %d Hz \r'
                                     % (i, dt, i/dt))
                    pyb.LED(1).toggle()

            # end after switch is pressed again
            log.close()                         # close file
            blue.off()                          # blue LED indicates file closed
            print('\nclosed '+filename)
            cam.angle(90)
            pyb.delay(100)
            cam.angle(0)
            pyb.delay(300)                      # delay avoids detection of multiple presses

main()

Martin
Posts: 7
Joined: Tue May 05, 2015 5:11 pm

Re: data write speeds and broken SD cards

Post by Martin » Mon May 25, 2015 2:29 pm

Actually it turned out to be much simpler than I thought. Instead of using interrupts and DMA, I just enabled the ADXL345 FiFo and configured the INT1 output for "watermark" mode with 20 entries (fifo is 32 entries). I set the ADXL345 for 3200 Hz sampling and monitor the INT1 line. If it is 1, I read 20 samples from the FiFo and write them to the SD card. This is pretty simple and works out quite well:

Code: Select all

f=open('/sd/test.bin','wb')
for k in range(160*5):
    # wait for FiFo to be filled
    while int1pin.value()==False:
        pass
    # read 20 values and write to SD card
    for i in range(20):
        nCS.low();spi.send_recv(CMD_RD,b7);nCS.high()
        f.write(b7)

f.close()
I just record 5 seconds here, I'll have to add stopping with a switch and incrementing the file name.

What was a great help (and what I recommend to anyone) is hooking up a logic analyzer to the SPI pins. I used the Saleae logic, quite affordable and with SPI (and others) protocol analyzers. Here's a screenshot:
Unbenannt.JPG
Unbenannt.JPG (154.08 KiB) Viewed 3599 times
The ADXL345 is connected via SPI at 5 MHz clock rate (I2C is too slow and I hate the protocol). At 3200 Hz sampling rate, the INT1 goes high every 20/3200 = 6.25 ms. The readout/file write takes 1.3 ms, so there's plenty of time. If the write buffer has filled up to 512, a sector is written to the SD card, which takes about 1.2 ms (427 kB/s).

I wrote a small import function in Matlab (I use it daily and I'm a total Python newbie, so this was quicker), here's a sample figure:
The attachment Unbenannt.JPG is no longer available
(somehow this image inlining doesn't seem to work in the preview)
Next I'll have to replace the ADXL345 on the board with the ADXL375 (which I got as samples from Analog), QFN rework...
Attachments
untitled.jpg
untitled.jpg (71.55 KiB) Viewed 3599 times

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: data write speeds and broken SD cards

Post by Turbinenreiter » Tue May 26, 2015 2:55 pm

Nice!
I'm gonna have good use for this soon.

Post Reply